[Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets

[Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets

[Philippe Mathieu-Daudé]

This patchset add 2 more architectures to the cross-build farm.

- Reorganize Dockerfiles to use less layers, resulting in smaller images.
  This also reduce time of image transfer, for example while using:
  `docker save qemu:debian | ssh remote docker load`
- Install more debian packages so the configure script enable more features and
  more code can be compiled/covered.
- There are still some incorrect multiarch packages on debian/jessie used in
  the docker images, add a script to generate fake packages and avoid
  dependencies issues.
- Modify the docker.py script to include an extra file while building images.

There is still some issue trying to cross-build mips64el-softmmu, it seems the
cross tools use the system outdated libfdt instead of the one checkouted in the
dtc submodule. I disabled this target for now.

I had some issues in some jobs, git reporting "Problem with the SSL CA cert"
trying to clone from github.com. I enabled GIT_CURL_VERBOSE but didn't want to
spend more time on this right now so I disabled the verification by setting
GIT_SSL_NO_VERIFY. Since we use Shippable services to compile the codebase and
not to release binaries it seemed Ok to me.

The branch https://github.com/philmd/qemu/tree/docker_shippable can be checked
here at Shippable:
  https://app.shippable.com/github/philmd/qemu/status/dashboard

With the free open source projects service, each arch builds in around ~9min,
the 5 jobs take ~44min in total. It seems each docker image is rebuilt from
scratch, maybe we could save a lot pushing/pulling to a registry.

Regards,

Phil.

Philippe Mathieu-Daudé (23):
  shippable: use C locale to simplify console output
  shippable: build using all available cpus
  shippable: ignore SSL verification
  docker: compact debian base
  docker: compact debian armhf
  docker: compact debian arm64
  docker: add extra libs to armhf target to extend codebase coverage
  docker: add extra libs to arm64 target to extend codebase coverage
  docker: add extra libs to s390x target to extend codebase coverage
  docker: add mipsel build target
  shippable: add mipsel target
  docker: let _copy_with_mkdir() sub_path argument be optional
  docker: add --include-file argument to 'build' command
  docker: add 'apt-fake' script which generate fake debian packages
  docker: fix mipsel debian/stable dependencies problem
  shippable: do not initialize submodules automatically
  shippable: use dtc submodule if distrib packages are too old
  shippable: temporary disable mips64el-softmmu build
  docker: add powerpc build target
  shippable: add powerpc target
  docker: fix powerpc debian/stable dependencies problem
  MAINTAINERS: add Shippable automation platform URL
  MAINTAINERS: self-appoint me as reviewer in build/test automation

 .shippable.yml                                     | 23 ++++++++++-
 MAINTAINERS                                        |  2 +
 tests/docker/Makefile.include                      |  7 +++-
 tests/docker/docker.py                             |  7 +++-
 tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
 tests/docker/dockerfiles/debian-arm64-cross.docker | 16 ++++++--
 tests/docker/dockerfiles/debian-armhf-cross.docker | 16 ++++++--
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 30 ++++++++++++++
 .../docker/dockerfiles/debian-powerpc-cross.docker | 30 ++++++++++++++
 tests/docker/dockerfiles/debian-s390x-cross.docker |  6 +++
 tests/docker/dockerfiles/debian.docker             | 24 ++++++-----
 11 files changed, 186 insertions(+), 21 deletions(-)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

-- 
2.11.0

[Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output

[Philippe Mathieu-Daudé]

remove this noise:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = "en_US.UTF-8",
    LC_CTYPE = "en_US.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 653bd750fe..231c29b620 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,5 +1,7 @@
 language: c
 env:
+  global:
+    - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
       TARGET_LIST=arm-softmmu,arm-linux-user
-- 
2.11.0

[Qemu-devel] [PATCH 02/23] shippable: build using all available cpus

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index 231c29b620..5170486ff9 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -20,4 +20,4 @@ build:
   ci:
     - unset CC
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
-    - make -j2
+    - make -j$(getconf _NPROCESSORS_ONLN)
-- 
2.11.0

[Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification

[Philippe Mathieu-Daudé]

shippable logs:
--------------
git_sync
- ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
Cloning into '/root/src/github.com/philmd/qemu'...
fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
retrying 1 of 3 times...

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 5170486ff9..b661e667b3 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -2,6 +2,10 @@ language: c
 env:
   global:
     - LC_ALL=C
+    # sometimes Shippable fails to clone from github (git_sync stage):
+    # "Problem with the SSL CA cert (path? access rights?)"
+    # for now disable SSL verification.
+    - GIT_SSL_NO_VERIFY=1
   matrix:
     - IMAGE=debian-armhf-cross
       TARGET_LIST=arm-softmmu,arm-linux-user
-- 
2.11.0

[Qemu-devel] [PATCH 04/23] docker: compact debian base

[Philippe Mathieu-Daudé]

- install common/basic tools at once
- one-line Emdebian setup
- use eatmydata and remove apt cache to save space
- add bison and flex and git

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian.docker | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 52bd79938e..694f8d3b63 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -9,17 +9,19 @@
 #
 FROM debian:stable-slim
 
-# Setup some basic tools we need
-RUN apt update
-RUN apt install -yy curl aptitude
-
-# Setup Emdebian
-RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
+# Install some basic tools and common build utilities
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
+        eatmydata && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        curl aptitude \
+        build-essential clang git \
+        bison flex && \
+    rm -rf /var/lib/apt
 
 # Duplicate deb line as deb-src
 RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
 
-# Install common build utilities
-RUN apt update
-RUN apt install -yy build-essential clang
+# Setup Emdebian
+RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
+    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-- 
2.11.0

[Qemu-devel] [PATCH 05/23] docker: compact debian armhf

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-armhf-cross.docker | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 668d60aeb3..533e525971 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -6,10 +6,12 @@
 FROM qemu:debian
 
 # Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture armhf
-RUN apt update
-RUN apt install -yy crossbuild-essential-armhf
-RUN apt-get build-dep -yy -a armhf qemu
+RUN dpkg --add-architecture armhf && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        crossbuild-essential-armhf
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu
-- 
2.11.0

[Qemu-devel] [PATCH 06/23] docker: compact debian arm64

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-arm64-cross.docker | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 592b5d7055..12b30aae54 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -6,10 +6,12 @@
 FROM qemu:debian
 
 # Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture arm64
-RUN apt update
-RUN apt install -yy crossbuild-essential-arm64
-RUN apt-get build-dep -yy -a arm64 qemu
+RUN dpkg --add-architecture arm64 && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        crossbuild-essential-arm64
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu
-- 
2.11.0

[Qemu-devel] [PATCH 07/23] docker: add extra libs to armhf target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-armhf-cross.docker | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 533e525971..01d9fcb7f2 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -14,4 +14,10 @@ RUN dpkg --add-architecture armhf && \
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
 
-RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        glusterfs-common:armhf \
+        libncursesw5-dev:armhf libnfs-dev:armhf \
+        libbz2-dev:armhf liblzo2-dev:armhf \
+        libsnappy-dev:armhf librdmacm-dev:armhf \
+        libxen-dev:armhf
-- 
2.11.0

[Qemu-devel] [PATCH 08/23] docker: add extra libs to arm64 target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-arm64-cross.docker | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 12b30aae54..5e0036917b 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -14,4 +14,10 @@ RUN dpkg --add-architecture arm64 && \
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
 
-RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        glusterfs-common:arm64 \
+        libncursesw5-dev:arm64 libnfs-dev:arm64 \
+        libbz2-dev:arm64 liblzo2-dev:arm64 \
+        libsnappy-dev:arm64 librdmacm-dev:arm64 \
+        libxen-dev:arm64
-- 
2.11.0

[Qemu-devel] [PATCH 09/23] docker: add extra libs to s390x target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-s390x-cross.docker | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 3a687feda0..119b152971 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -20,3 +20,9 @@ RUN apt install -yy gcc-multilib-s390x-linux-gnu binutils-multiarch
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=s390x-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    glusterfs-common:s390x \
+    libncursesw5-dev:s390x libnfs-dev:s390x \
+    libbz2-dev:s390x liblzo2-dev:s390x \
+    libsnappy-dev:s390x librdmacm-dev:s390x
-- 
2.11.0

[Qemu-devel] [PATCH 10/23] docker: add mipsel build target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include                      |  1 +
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 23 ++++++++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 03eda37bf4..ceff2c1654 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -57,6 +57,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
+docker-image-debian-mipsel-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
new file mode 100644
index 0000000000..c569a2e1b4
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -0,0 +1,23 @@
+#
+# Docker mipsel cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture mipsel && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        crossbuild-essential-mipsel
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a mipsel qemu && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        glusterfs-common:mipsel \
+        libncursesw5-dev:mipsel libnfs-dev:mipsel \
+        libbz2-dev:mipsel liblzo2-dev:mipsel \
+        libsnappy-dev:mipsel librdmacm-dev:mipsel
-- 
2.11.0

[Qemu-devel] [PATCH 11/23] shippable: add mipsel target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index b661e667b3..fa2d6a4e9b 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,6 +13,8 @@ env:
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
       TARGET_LIST=s390x-softmmu,s390x-linux-user
+    - IMAGE=debian-mipsel-cross
+      TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE}
-- 
2.11.0

[Qemu-devel] [PATCH 12/23] docker: let _copy_with_mkdir() sub_path argument be optional

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/docker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 8747f6a440..6ddc6e4c2a 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -52,7 +52,7 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
-def _copy_with_mkdir(src, root_dir, sub_path):
+def _copy_with_mkdir(src, root_dir, sub_path='.'):
     """Copy src into root_dir, creating sub_path as needed."""
     dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
     try:
-- 
2.11.0

[Qemu-devel] [RFC PATCH 13/23] docker: add --include-file argument to 'build' command

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 3 ++-
 tests/docker/docker.py        | 5 +++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index ceff2c1654..47978fb56c 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -51,7 +51,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
 		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
 		$(if $(NOUSER),,--add-current-user) \
-		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
+		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE))\
+		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
 		"BUILD","$*")
 
 # Enforce dependancies for composite images
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 6ddc6e4c2a..4c096a8178 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -237,6 +237,9 @@ class BuildCommand(SubCommand):
                             help="""Specify a binary that will be copied to the
                             container together with all its dependent
                             libraries""")
+        parser.add_argument("--include-file", "-f",
+                            help="""Specify a binary that will be copied to the
+                            container""")
         parser.add_argument("--add-current-user", "-u", dest="user",
                             action="store_true",
                             help="Add the current user to image's passwd")
@@ -274,6 +277,8 @@ class BuildCommand(SubCommand):
             if args.include_executable:
                 _copy_binary_with_libs(args.include_executable,
                                        docker_dir)
+            if args.include_file:
+                _copy_with_mkdir(args.include_file, docker_dir)
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
-- 
2.11.0

[Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include               |  2 ++
 tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
 tests/docker/dockerfiles/debian.docker      |  2 ++
 3 files changed, 50 insertions(+)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 47978fb56c..ca6f57a292 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -55,6 +55,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
 		"BUILD","$*")
 
+docker-image-debian: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
+
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
new file mode 100755
index 0000000000..387522c174
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-apt-fake.sh
@@ -0,0 +1,46 @@
+#! /bin/sh
+#
+# Generate fake debian package to resolve unimportant unmet dependencies held
+# by upstream multiarch broken packages.
+#
+# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+test $1 = "install" && shift 1
+
+fake_install()
+{
+    echo "Generating fake $2 $1 $3 ..."
+    (cd /var/cache/apt/archives
+        (cat << 'EOF'
+Section: misc
+Priority: optional
+Standards-Version: 3.9.2
+
+Package: NAME
+Version: VERSION
+Maintainer: qemu-devel@nongnu.org
+Architecture: any
+Multi-Arch: same
+Description: fake NAME
+EOF
+        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
+        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
+        dpkg -i $2_$3_$1.deb
+    )
+}
+
+try_install()
+{
+    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
+    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
+    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
+    apt-get install -q -yy $1 || fake_install $arch $name $vers
+}
+
+for package in $*; do
+    try_install $package
+done
diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 694f8d3b63..a99396ae8f 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -22,6 +22,8 @@ RUN apt-get update && \
 # Duplicate deb line as deb-src
 RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
 
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+
 # Setup Emdebian
 RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
     curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-- 
2.11.0

[Qemu-devel] [PATCH 15/23] docker: fix mipsel debian/stable dependencies problem

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-mipsel-cross.docker | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
index c569a2e1b4..3c4945d78e 100644
--- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -12,6 +12,13 @@ RUN dpkg --add-architecture mipsel && \
     DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
         crossbuild-essential-mipsel
 
+# <kludge> to fix "following packages have unmet dependencies" ...
+RUN apt-get install -y --no-install-recommends equivs pkg-config && \
+    apt-fake install pkgconf:mipsel=0.9.7-fake pkg-config:mipsel=0.28-1.1-fake && \
+    ln -s pkg-config /usr/bin/mipsel-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/mipsel-linux-gnu/pkgconfig
+# </kludge>
+
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
 
-- 
2.11.0

[Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically

[Philippe Mathieu-Daudé]

do it in the 'ci' target when needed.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index fa2d6a4e9b..b3cbca458c 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,4 +1,6 @@
 language: c
+git:
+   submodules: false
 env:
   global:
     - LC_ALL=C
-- 
2.11.0

[Qemu-devel] [PATCH 17/23] shippable: use dtc submodule if distrib packages are too old

[Philippe Mathieu-Daudé]

shippable output:
----------------
  LINK    mips64el-softmmu/qemu-system-mips64el
../hw/core/loader-fit.o: In function `load_fit':
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
collect2: error: ld returned 1 exit status
Makefile:201: recipe for target 'qemu-system-mips64el' failed
make[1]: *** [qemu-system-mips64el] Error 1
Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
make: *** [subdir-mips64el-softmmu] Error 2

having this too old libfdt version (required version >= 1.4.2):
 # dpkg-query --showformat='${Version}\n' --show libfdt-dev
1.4.0+dfsg-1

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index b3cbca458c..6601243060 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -27,5 +27,13 @@ build:
     options: "-e HOME=/root"
   ci:
     - unset CC
+    # some targets require newer up to date packages, for example TARGET_LIST matching
+    # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
+    # see the configure script:
+    #    error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
+    #    "  (1) Preferred: Install the DTC (libfdt) devel package"
+    #    "  (2) Fetch the DTC submodule, using:"
+    #    "      git submodule update --init dtc"
+    - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
     - make -j$(getconf _NPROCESSORS_ONLN)
-- 
2.11.0

[Qemu-devel] [PATCH 19/23] docker: add powerpc build target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include                      |  1 +
 .../docker/dockerfiles/debian-powerpc-cross.docker | 23 ++++++++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index ca6f57a292..704b74bfbd 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -61,6 +61,7 @@ docker-image-debian: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
 docker-image-debian-mipsel-cross: docker-image-debian
+docker-image-debian-powerpc-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
new file mode 100644
index 0000000000..9107ebc2d3
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -0,0 +1,23 @@
+#
+# Docker powerpc cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture powerpc && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        crossbuild-essential-powerpc
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a powerpc qemu && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+        glusterfs-common:powerpc \
+        libncursesw5-dev:powerpc libnfs-dev:powerpc \
+        libbz2-dev:powerpc liblzo2-dev:powerpc \
+        libsnappy-dev:powerpc librdmacm-dev:powerpc
-- 
2.11.0

[Qemu-devel] [PATCH 20/23] shippable: add powerpc target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 7e714991f9..3a80165b6a 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -18,6 +18,8 @@ env:
     # mips64el-softmmu disabled due to libfdt problem
     - IMAGE=debian-mipsel-cross
       TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
+    - IMAGE=debian-powerpc-cross
+      TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE}
-- 
2.11.0

[Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-powerpc-cross.docker | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
index 9107ebc2d3..d6e4d1e977 100644
--- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -12,6 +12,13 @@ RUN dpkg --add-architecture powerpc && \
     DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
         crossbuild-essential-powerpc
 
+# <kludge> to fix "following packages have unmet dependencies" ...
+RUN apt-get install -y --no-install-recommends equivs pkg-config && \
+    apt-fake install pkg-config:powerpc=0.28-1.1-fake && \
+    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
+# </kludge>
+
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
 
-- 
2.11.0

[Qemu-devel] [PATCH 22/23] MAINTAINERS: add Shippable automation platform URL

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index a1d2b3a4d3..9251f575d8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1833,6 +1833,7 @@ F: .travis.yml
 F: .shippable.yml
 F: tests/docker/
 W: https://travis-ci.org/qemu/qemu
+W: https://app.shippable.com/github/qemu/qemu
 W: http://patchew.org/QEMU/
 
 Documentation
-- 
2.11.0

[Qemu-devel] [PATCH 23/23] MAINTAINERS: self-appoint me as reviewer in build/test automation

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 9251f575d8..c7520b76db 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1827,6 +1827,7 @@ Build and test automation
 -------------------------
 M: Alex Bennée <alex.bennee@linaro.org>
 M: Fam Zheng <famz@redhat.com>
+R: Philippe Mathieu-Daudé <f4bug@amsat.org>
 L: qemu-devel@nongnu.org
 S: Maintained
 F: .travis.yml
-- 
2.11.0

[Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index 6601243060..7e714991f9 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -15,8 +15,9 @@ env:
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
       TARGET_LIST=s390x-softmmu,s390x-linux-user
+    # mips64el-softmmu disabled due to libfdt problem
     - IMAGE=debian-mipsel-cross
-      TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
+      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE}
-- 
2.11.0

Re: [Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> remove this noise:
>
> perl: warning: Setting locale failed.
> perl: warning: Please check that your locale settings:
>     LANGUAGE = (unset),
>     LC_ALL = "en_US.UTF-8",
>     LC_CTYPE = "en_US.UTF-8",
>     LANG = "en_US.UTF-8"
>     are supported and installed on your system.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .shippable.yml | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 653bd750fe..231c29b620 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -1,5 +1,7 @@
>  language: c
>  env:
> +  global:
> +    - LC_ALL=C
>    matrix:
>      - IMAGE=debian-armhf-cross
>        TARGET_LIST=arm-softmmu,arm-linux-user

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

--
Alex Bennée

Re: [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .shippable.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 231c29b620..5170486ff9 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -20,4 +20,4 @@ build:
>    ci:
>      - unset CC
>      - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
> -    - make -j2
> +    - make -j$(getconf _NPROCESSORS_ONLN)

Did you ever check what this generates on Shippable? AFAICT they only
have one core for each docker image. I following the core+1 pattern
although that may not make as much sense for low core counts.

--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> shippable logs:
> --------------
> git_sync
> - ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
> Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
> Cloning into '/root/src/github.com/philmd/qemu'...
> fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
> retrying 1 of 3 times...

I've not seen this before but which git_sync stage is it. The host or
the container? If it is the container that probably just means we are
missing the CA Cert package.

>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .shippable.yml | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 5170486ff9..b661e667b3 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -2,6 +2,10 @@ language: c
>  env:
>    global:
>      - LC_ALL=C
> +    # sometimes Shippable fails to clone from github (git_sync stage):
> +    # "Problem with the SSL CA cert (path? access rights?)"
> +    # for now disable SSL verification.
> +    - GIT_SSL_NO_VERIFY=1
>    matrix:
>      - IMAGE=debian-armhf-cross
>        TARGET_LIST=arm-softmmu,arm-linux-user


--
Alex Bennée

Re: [Qemu-devel] [PATCH 04/23] docker: compact debian base

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> - install common/basic tools at once
> - one-line Emdebian setup
> - use eatmydata and remove apt cache to save space
> - add bison and flex and git
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/dockerfiles/debian.docker | 22 ++++++++++++----------
>  1 file changed, 12 insertions(+), 10 deletions(-)
>
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index 52bd79938e..694f8d3b63 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -9,17 +9,19 @@
>  #
>  FROM debian:stable-slim
>
> -# Setup some basic tools we need
> -RUN apt update
> -RUN apt install -yy curl aptitude
> -
> -# Setup Emdebian
> -RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
> -RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
> +# Install some basic tools and common build utilities
> +RUN apt-get update && \
> +    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
> +        eatmydata && \

I wonder if we should just use ENV instead to set DEBIAN_FRONTEND?

> +    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
> +        curl aptitude \
> +        build-essential clang git \
> +        bison flex && \
> +    rm -rf /var/lib/apt

Use apt clean to do this.

>
>  # Duplicate deb line as deb-src
>  RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>
> -# Install common build utilities
> -RUN apt update
> -RUN apt install -yy build-essential clang
> +# Setup Emdebian
> +RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
> +    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -

Why move this to the end? Once we have curl installed it would be nice
to do just one update step and have emdebian synced up in the base
image.


--
Alex Bennée

Re: [Qemu-devel] [PATCH 11/23] shippable: add mipsel target

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .shippable.yml | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index b661e667b3..fa2d6a4e9b 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -13,6 +13,8 @@ env:
>        TARGET_LIST=aarch64-softmmu,aarch64-linux-user
>      - IMAGE=debian-s390x-cross
>        TARGET_LIST=s390x-softmmu,s390x-linux-user
> +    - IMAGE=debian-mipsel-cross
> +      TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
>  build:
>    pre_ci:
>      - make docker-image-${IMAGE}

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 13/23] docker: add --include-file argument to 'build' command

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include | 3 ++-
>  tests/docker/docker.py        | 5 +++++
>  2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index ceff2c1654..47978fb56c 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -51,7 +51,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
>  		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
>  		$(if $(NOUSER),,--add-current-user) \
> -		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
> +		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE))\
> +		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
>  		"BUILD","$*")
>
>  # Enforce dependancies for composite images
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 6ddc6e4c2a..4c096a8178 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -237,6 +237,9 @@ class BuildCommand(SubCommand):
>                              help="""Specify a binary that will be copied to the
>                              container together with all its dependent
>                              libraries""")
> +        parser.add_argument("--include-file", "-f",
> +                            help="""Specify a binary that will be copied to the
> +                            container""")

It's not a binary, just an additional file (which might be a binary).

>          parser.add_argument("--add-current-user", "-u", dest="user",
>                              action="store_true",
>                              help="Add the current user to image's passwd")
> @@ -274,6 +277,8 @@ class BuildCommand(SubCommand):
>              if args.include_executable:
>                  _copy_binary_with_libs(args.include_executable,
>                                         docker_dir)
> +            if args.include_file:
> +                _copy_with_mkdir(args.include_file, docker_dir)
>
>              argv += ["--build-arg=" + k.lower() + "=" + v
>                          for k, v in os.environ.iteritems()


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include               |  2 ++
>  tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
>  tests/docker/dockerfiles/debian.docker      |  2 ++
>  3 files changed, 50 insertions(+)
>  create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 47978fb56c..ca6f57a292 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -55,6 +55,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
>  		"BUILD","$*")
>
> +docker-image-debian: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
> +

Can we only include debian-apt-fake on the debian images that require it please?

>  # Enforce dependancies for composite images
>  docker-image-debian-armhf-cross: docker-image-debian
>  docker-image-debian-arm64-cross: docker-image-debian
> diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
> new file mode 100755
> index 0000000000..387522c174
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-apt-fake.sh
> @@ -0,0 +1,46 @@
> +#! /bin/sh
> +#
> +# Generate fake debian package to resolve unimportant unmet dependencies held
> +# by upstream multiarch broken packages.
> +#
> +# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +test $1 = "install" && shift 1
> +
> +fake_install()
> +{
> +    echo "Generating fake $2 $1 $3 ..."
> +    (cd /var/cache/apt/archives
> +        (cat << 'EOF'
> +Section: misc
> +Priority: optional
> +Standards-Version: 3.9.2
> +
> +Package: NAME
> +Version: VERSION
> +Maintainer: qemu-devel@nongnu.org
> +Architecture: any
> +Multi-Arch: same
> +Description: fake NAME
> +EOF
> +        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
> +        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
> +        dpkg -i $2_$3_$1.deb
> +    )
> +}
> +
> +try_install()
> +{
> +    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
> +    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
> +    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
> +    apt-get install -q -yy $1 || fake_install $arch $name $vers
> +}
> +
> +for package in $*; do
> +    try_install $package
> +done
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index 694f8d3b63..a99396ae8f 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -22,6 +22,8 @@ RUN apt-get update && \
>  # Duplicate deb line as deb-src
>  RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>
> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
> +

And only add this in the images that need it.

>  # Setup Emdebian
>  RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
>      curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -


--
Alex Bennée

Re: [Qemu-devel] [PATCH 15/23] docker: fix mipsel debian/stable dependencies problem

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/dockerfiles/debian-mipsel-cross.docker | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
> index c569a2e1b4..3c4945d78e 100644
> --- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
> +++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
> @@ -12,6 +12,13 @@ RUN dpkg --add-architecture mipsel && \
>      DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
>          crossbuild-essential-mipsel
>
> +# <kludge> to fix "following packages have unmet dependencies" ...
> +RUN apt-get install -y --no-install-recommends equivs pkg-config && \
> +    apt-fake install pkgconf:mipsel=0.9.7-fake
> pkg-config:mipsel=0.28-1.1-fake && \

I'll take your word that this fixes the problem for broken packages.
When I ran into these I just started using the linux-user approach
because it seemed fragile otherwise.

I assume a dot release of any of these packages might break the image
build?

> +    ln -s pkg-config /usr/bin/mipsel-linux-gnu-pkg-config
> +ENV PKG_CONFIG_PATH /usr/lib/mipsel-linux-gnu/pkgconfig
> +
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

>  # Specify the cross prefix for this image (see tests/docker/common.rc)
>  ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-


--
Alex Bennée

Re: [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> do it in the 'ci' target when needed.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .shippable.yml | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index fa2d6a4e9b..b3cbca458c 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -1,4 +1,6 @@
>  language: c
> +git:
> +   submodules: false
>  env:
>    global:
>      - LC_ALL=C

Maybe this should be merged with the next patch?

--
Alex Bennée

Re: [Qemu-devel] [PATCH 17/23] shippable: use dtc submodule if distrib packages are too old

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> shippable output:
> ----------------
>   LINK    mips64el-softmmu/qemu-system-mips64el
> ../hw/core/loader-fit.o: In function `load_fit':
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
> collect2: error: ld returned 1 exit status
> Makefile:201: recipe for target 'qemu-system-mips64el' failed
> make[1]: *** [qemu-system-mips64el] Error 1
> Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
> make: *** [subdir-mips64el-softmmu] Error 2
>
> having this too old libfdt version (required version >= 1.4.2):
>  # dpkg-query --showformat='${Version}\n' --show libfdt-dev
> 1.4.0+dfsg-1
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .shippable.yml | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index b3cbca458c..6601243060 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -27,5 +27,13 @@ build:
>      options: "-e HOME=/root"
>    ci:
>      - unset CC
> +    # some targets require newer up to date packages, for example TARGET_LIST matching
> +    # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
> +    # see the configure script:
> +    #    error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
> +    #    "  (1) Preferred: Install the DTC (libfdt) devel package"
> +    #    "  (2) Fetch the DTC submodule, using:"
> +    #    "      git submodule update --init dtc"
> +    - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
>      - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
>      - make -j$(getconf _NPROCESSORS_ONLN)

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>



--
Alex Bennée

Re: [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/dockerfiles/debian-powerpc-cross.docker | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
> index 9107ebc2d3..d6e4d1e977 100644
> --- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
> @@ -12,6 +12,13 @@ RUN dpkg --add-architecture powerpc && \
>      DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
>          crossbuild-essential-powerpc
>
> +# <kludge> to fix "following packages have unmet dependencies" ...
> +RUN apt-get install -y --no-install-recommends equivs pkg-config && \
> +    apt-fake install pkg-config:powerpc=0.28-1.1-fake && \
> +    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
> +# </kludge>
> +
>  # Specify the cross prefix for this image (see tests/docker/common.rc)
>  ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-

Given the image needs these package hacks it is probably best folded
into the commit adding the target.

--
Alex Bennée

Re: [Qemu-devel] [PATCH 22/23] MAINTAINERS: add Shippable automation platform URL

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  MAINTAINERS | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index a1d2b3a4d3..9251f575d8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1833,6 +1833,7 @@ F: .travis.yml
>  F: .shippable.yml
>  F: tests/docker/
>  W: https://travis-ci.org/qemu/qemu
> +W: https://app.shippable.com/github/qemu/qemu
>  W: http://patchew.org/QEMU/
>
>  Documentation

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

--
Alex Bennée

Re: [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .shippable.yml | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 6601243060..7e714991f9 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -15,8 +15,9 @@ env:
>        TARGET_LIST=aarch64-softmmu,aarch64-linux-user
>      - IMAGE=debian-s390x-cross
>        TARGET_LIST=s390x-softmmu,s390x-linux-user
> +    # mips64el-softmmu disabled due to libfdt problem
>      - IMAGE=debian-mipsel-cross
> -      TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
> +      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
>  build:
>    pre_ci:
>      - make docker-image-${IMAGE}


If you fix the order of your patches you can get rid of this noise.

--
Alex Bennée

Re: [Qemu-devel] [PATCH 23/23] MAINTAINERS: self-appoint me as reviewer in build/test automation

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  MAINTAINERS | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 9251f575d8..c7520b76db 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1827,6 +1827,7 @@ Build and test automation
>  -------------------------
>  M: Alex Bennée <alex.bennee@linaro.org>
>  M: Fam Zheng <famz@redhat.com>
> +R: Philippe Mathieu-Daudé <f4bug@amsat.org>
>  L: qemu-devel@nongnu.org
>  S: Maintained
>  F: .travis.yml

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification

[Philippe Mathieu-Daudé]

Hi Alex,

On 05/08/2017 07:58 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> shippable logs:
>> --------------
>> git_sync
>> - ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
>> Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
>> Cloning into '/root/src/github.com/philmd/qemu'...
>> fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
>> retrying 1 of 3 times...
>
> I've not seen this before but which git_sync stage is it. The host or
> the container? If it is the container that probably just means we are
> missing the CA Cert package.

You are right! Adding the ca-certificates package in the docker image 
solves this issue I had in container's git_sync stage :)

Thank for the review,

Phil.

>
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  .shippable.yml | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index 5170486ff9..b661e667b3 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -2,6 +2,10 @@ language: c
>>  env:
>>    global:
>>      - LC_ALL=C
>> +    # sometimes Shippable fails to clone from github (git_sync stage):
>> +    # "Problem with the SSL CA cert (path? access rights?)"
>> +    # for now disable SSL verification.
>> +    - GIT_SSL_NO_VERIFY=1
>>    matrix:
>>      - IMAGE=debian-armhf-cross
>>        TARGET_LIST=arm-softmmu,arm-linux-user
>
>
> --
> Alex Bennée
>

Re: [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus

[Philippe Mathieu-Daudé]

On 05/08/2017 07:55 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  .shippable.yml | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index 231c29b620..5170486ff9 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -20,4 +20,4 @@ build:
>>    ci:
>>      - unset CC
>>      - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
>> -    - make -j2
>> +    - make -j$(getconf _NPROCESSORS_ONLN)
>
> Did you ever check what this generates on Shippable? AFAICT they only
> have one core for each docker image. I following the core+1 pattern
> although that may not make as much sense for low core counts.

 From the 'Job node info':

$ sudo docker info
Architecture: x86_64
CPUs: 2

Checking at 'ci' stage:

$ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf 
_NPROCESSORS_CONF`
container proc: 2 2

I think the correct cores+1 pattern should instead be:

make -j$(($(getconf _NPROCESSORS_ONLN) + 1))

So we stay building at full speed if Shippable eventually upgrade/allow 
to use more cores.

Phil.

> --
> Alex Bennée
>

Re: [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus

[Philippe Mathieu-Daudé]

>>> diff --git a/.shippable.yml b/.shippable.yml
>>> index 231c29b620..5170486ff9 100644
>>> --- a/.shippable.yml
>>> +++ b/.shippable.yml
>>> @@ -20,4 +20,4 @@ build:
>>>    ci:
>>>      - unset CC
>>>      - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
>>> -    - make -j2
>>> +    - make -j$(getconf _NPROCESSORS_ONLN)
>>
>> Did you ever check what this generates on Shippable? AFAICT they only
>> have one core for each docker image. I following the core+1 pattern
>> although that may not make as much sense for low core counts.
>
> From the 'Job node info':
>
> $ sudo docker info
> Architecture: x86_64
> CPUs: 2
>
> Checking at 'ci' stage:
>
> $ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf
> _NPROCESSORS_CONF`
> container proc: 2 2
>
> I think the correct cores+1 pattern should instead be:
>
> make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
>
> So we stay building at full speed if Shippable eventually upgrade/allow
> to use more cores.

For what it's worth using this pattern builds last on average <8min vs 
 >9min previously :)

Re: [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages

[Philippe Mathieu-Daudé]

On 05/08/2017 08:12 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/Makefile.include               |  2 ++
>>  tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
>>  tests/docker/dockerfiles/debian.docker      |  2 ++
>>  3 files changed, 50 insertions(+)
>>  create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 47978fb56c..ca6f57a292 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -55,6 +55,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>>  		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
>>  		"BUILD","$*")
>>
>> +docker-image-debian: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
>> +
>
> Can we only include debian-apt-fake on the debian images that require it please?

Sure.

>>  # Enforce dependancies for composite images
>>  docker-image-debian-armhf-cross: docker-image-debian
>>  docker-image-debian-arm64-cross: docker-image-debian
>> diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
>> new file mode 100755
>> index 0000000000..387522c174
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/debian-apt-fake.sh
>> @@ -0,0 +1,46 @@
>> +#! /bin/sh
>> +#
>> +# Generate fake debian package to resolve unimportant unmet dependencies held
>> +# by upstream multiarch broken packages.
>> +#
>> +# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2
>> +# or (at your option) any later version. See the COPYING file in
>> +# the top-level directory.
>> +
>> +test $1 = "install" && shift 1
>> +
>> +fake_install()
>> +{
>> +    echo "Generating fake $2 $1 $3 ..."
>> +    (cd /var/cache/apt/archives
>> +        (cat << 'EOF'
>> +Section: misc
>> +Priority: optional
>> +Standards-Version: 3.9.2
>> +
>> +Package: NAME
>> +Version: VERSION
>> +Maintainer: qemu-devel@nongnu.org
>> +Architecture: any
>> +Multi-Arch: same
>> +Description: fake NAME
>> +EOF
>> +        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
>> +        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
>> +        dpkg -i $2_$3_$1.deb
>> +    )
>> +}
>> +
>> +try_install()
>> +{
>> +    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
>> +    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
>> +    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
>> +    apt-get install -q -yy $1 || fake_install $arch $name $vers
>> +}
>> +
>> +for package in $*; do
>> +    try_install $package
>> +done
>> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
>> index 694f8d3b63..a99396ae8f 100644
>> --- a/tests/docker/dockerfiles/debian.docker
>> +++ b/tests/docker/dockerfiles/debian.docker
>> @@ -22,6 +22,8 @@ RUN apt-get update && \
>>  # Duplicate deb line as deb-src
>>  RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>>
>> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
>> +
>
> And only add this in the images that need it.
>
>>  # Setup Emdebian
>>  RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
>>      curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
>
>
> --
> Alex Bennée
>

Re: [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically

[Philippe Mathieu-Daudé]

On 05/08/2017 08:15 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> do it in the 'ci' target when needed.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  .shippable.yml | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index fa2d6a4e9b..b3cbca458c 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -1,4 +1,6 @@
>>  language: c
>> +git:
>> +   submodules: false
>>  env:
>>    global:
>>      - LC_ALL=C
>
> Maybe this should be merged with the next patch?

Ok.

> --
> Alex Bennée
>

Re: [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem

[Philippe Mathieu-Daudé]

On 05/08/2017 08:19 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/dockerfiles/debian-powerpc-cross.docker | 7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> index 9107ebc2d3..d6e4d1e977 100644
>> --- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> @@ -12,6 +12,13 @@ RUN dpkg --add-architecture powerpc && \
>>      DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
>>          crossbuild-essential-powerpc
>>
>> +# <kludge> to fix "following packages have unmet dependencies" ...
>> +RUN apt-get install -y --no-install-recommends equivs pkg-config && \
>> +    apt-fake install pkg-config:powerpc=0.28-1.1-fake && \
>> +    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
>> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
>> +# </kludge>
>> +
>>  # Specify the cross prefix for this image (see tests/docker/common.rc)
>>  ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
>
> Given the image needs these package hacks it is probably best folded
> into the commit adding the target.

Ok, will squash with the "ADD debian-apt-fake.sh 
/usr/local/bin/apt-fake" line.

> --
> Alex Bennée
>

Re: [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build

[Philippe Mathieu-Daudé]

On 05/08/2017 08:22 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  .shippable.yml | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index 6601243060..7e714991f9 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -15,8 +15,9 @@ env:
>>        TARGET_LIST=aarch64-softmmu,aarch64-linux-user
>>      - IMAGE=debian-s390x-cross
>>        TARGET_LIST=s390x-softmmu,s390x-linux-user
>> +    # mips64el-softmmu disabled due to libfdt problem
>>      - IMAGE=debian-mipsel-cross
>> -      TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
>> +      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
>>  build:
>>    pre_ci:
>>      - make docker-image-${IMAGE}
>
>
> If you fix the order of your patches you can get rid of this noise.

I'll squash with patch 11 (removing your R-B).

> --
> Alex Bennée
>

Re: [Qemu-devel] [PATCH 04/23] docker: compact debian base

[Philippe Mathieu-Daudé]

On 05/08/2017 08:04 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> - install common/basic tools at once
>> - one-line Emdebian setup
>> - use eatmydata and remove apt cache to save space
>> - add bison and flex and git
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/dockerfiles/debian.docker | 22 ++++++++++++----------
>>  1 file changed, 12 insertions(+), 10 deletions(-)
>>
>> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
>> index 52bd79938e..694f8d3b63 100644
>> --- a/tests/docker/dockerfiles/debian.docker
>> +++ b/tests/docker/dockerfiles/debian.docker
>> @@ -9,17 +9,19 @@
>>  #
>>  FROM debian:stable-slim
>>
>> -# Setup some basic tools we need
>> -RUN apt update
>> -RUN apt install -yy curl aptitude
>> -
>> -# Setup Emdebian
>> -RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
>> -RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
>> +# Install some basic tools and common build utilities
>> +RUN apt-get update && \
>> +    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
>> +        eatmydata && \
>
> I wonder if we should just use ENV instead to set DEBIAN_FRONTEND?

In https://github.com/moby/moby/issues/4032#issuecomment-34597177 Tianon 
Gravi commented:

[...] setting it via ENV should be actively discouraged. [...]
The reason it's not a good default is that if I docker run -i -t ... 
bash, I'm now interactive, and this variable being set is very, very wrong.

>
>> +    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
>> +        curl aptitude \
>> +        build-essential clang git \
>> +        bison flex && \
>> +    rm -rf /var/lib/apt
>
> Use apt clean to do this.

Ok.

>>
>>  # Duplicate deb line as deb-src
>>  RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>>
>> -# Install common build utilities
>> -RUN apt update
>> -RUN apt install -yy build-essential clang
>> +# Setup Emdebian
>> +RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
>> +    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
>
> Why move this to the end? Once we have curl installed it would be nice
> to do just one update step and have emdebian synced up in the base
> image.

In the first apt-get can install curl + build-essential clang in the 
same call, and there is no need to fetch emdebian.org index.

Using 'apt clean' at the end of a layer keeps smaller layer but the next 
layer have to call `apt-get update` again. Since this base image is used 
to cross-compile, the child image will have to call `dpkg 
--add-architecture ...` before fetching emdebian.org arch index again 
(apt-get update) to be able to install the `crossbuild-essential-$arch` 
package.
So there is no need to update right after adding this repo, the 'child' 
will do it (and have to do it).

>
> --
> Alex Bennée
>