[Qemu-devel] [PATCH v3 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Qemu-devel] [PATCH v3 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Philippe Mathieu-Daudé]

This patchset add 2 more architectures to the cross-build farm.

There is still some issue trying to cross-build mips64el-softmmu, it seems the
cross tools use the system outdated libfdt instead of the one checkouted in the
dtc submodule. I disabled this target for now.

The branch https://github.com/philmd/qemu/tree/docker_shippable_v3 can be
checked at Shippable:
  https://app.shippable.com/github/philmd/qemu/status/dashboard

Each arch builds in around ~9min. Using Shippable free open source projects
service, the 5 jobs take ~38-44min in total.

v3:
- Addressed review feedbacks from Fam:
- Keep building images in various layers, but use DEBIAN_FRONTEND=noninteractive
- Document '--extra-files', now it supports adding various files at once
- Checksum extra files to trigger a docker image rebuild if modified
- Use better regex to generate deb-src entries
- Reordered extra libs, to ease further add/remove diffs

v2:
- Addressed review feedback from Alex, added his R-B
- Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
  installing the ca-certificates package.
- Squashed/split some commits
- use 'apt-get clean' instead of brutal 'rm -rf'
- checked how many cores are available on Shippable and use them fully
  (reduced total time from 44min to 38min)
- build armeb-linux-user target

v1:
- Reorganize Dockerfiles to use less layers, resulting in smaller images.
  This also reduce time of image transfer, for example while using:
  `docker save qemu:debian | ssh remote docker load`
- Install more debian packages so the configure script enable more features and
  more code can be compiled/covered.
- There are still some incorrect multiarch packages on debian/jessie used in
  the docker images, add a script to generate fake packages and avoid
  dependencies issues.
- Modify the docker.py script to include an extra file while building images.

Regards,

Phil.

Philippe Mathieu-Daudé (24):
  docker: let _copy_with_mkdir() sub_path argument be optional
  docker: add --include-files argument to 'build' command
  docker: rebuild image if 'extra files' checksum does not match
  docker: add 'apt-fake' script which generate fake debian packages
  docker: install ca-certificates package in base image
  docker: create deb-src entry and setup Emdebian in the same layer
  docker: use better regex to generate deb-src entries
  docker: use eatmydata, install common build packages in base image
  docker: use eatmydata in debian armhf image
  docker: use eatmydata in debian arm64 image
  docker: add extra libs to armhf target to extend codebase coverage
  docker: add extra libs to arm64 target to extend codebase coverage
  docker: add extra libs to s390x target to extend codebase coverage
  docker: add mipsel build target
  docker: add powerpc build target
  shippable: use C locale to simplify console output
  shippable: build using all available cpus
  shippable: do not initialize submodules automatically
  shippable: be verbose while building docker images
  shippable: add armeb-linux-user target
  shippable: add powerpc target
  shippable: add mipsel target
  MAINTAINERS: add Shippable automation platform URL
  MAINTAINERS: self-appoint me as reviewer in build/test automation

 .shippable.yml                                     | 23 +++++++++--
 MAINTAINERS                                        |  2 +
 tests/docker/Makefile.include                      |  8 +++-
 tests/docker/docker.py                             | 29 ++++++++++----
 tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
 tests/docker/dockerfiles/debian-arm64-cross.docker | 18 ++++++++-
 tests/docker/dockerfiles/debian-armhf-cross.docker | 18 ++++++++-
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 41 +++++++++++++++++++
 .../docker/dockerfiles/debian-powerpc-cross.docker | 40 +++++++++++++++++++
 tests/docker/dockerfiles/debian-s390x-cross.docker | 10 +++++
 tests/docker/dockerfiles/debian.docker             | 18 +++++----
 11 files changed, 230 insertions(+), 23 deletions(-)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

-- 
2.11.0

[Qemu-devel] [PATCH v3 01/24] docker: let _copy_with_mkdir() sub_path argument be optional

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/docker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 8747f6a440..6ddc6e4c2a 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -52,7 +52,7 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
-def _copy_with_mkdir(src, root_dir, sub_path):
+def _copy_with_mkdir(src, root_dir, sub_path='.'):
     """Copy src into root_dir, creating sub_path as needed."""
     dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
     try:
-- 
2.11.0

[Qemu-devel] [PATCH v3 02/24] docker: add --include-files argument to 'build' command

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include |  5 ++++-
 tests/docker/docker.py        | 12 +++++++++---
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 03eda37bf4..fe1a9a53ff 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -51,6 +51,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
 		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
 		$(if $(NOUSER),,--add-current-user) \
+		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
 		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
 		"BUILD","$*")
 
@@ -107,6 +108,8 @@ docker:
 	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
 	@echo '    NOCACHE=1            Ignore cache when build images.'
 	@echo '    EXECUTABLE=<path>    Include executable in image.'
+	@echo '    EXTRA_FILES="<path> [... <path>]"'
+	@echo '                         Include extra files in image.'
 
 # This rule if for directly running against an arbitrary docker target.
 # It is called by the expanded docker targets (e.g. make
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 6ddc6e4c2a..68cca25f89 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -237,6 +237,11 @@ class BuildCommand(SubCommand):
                             help="""Specify a binary that will be copied to the
                             container together with all its dependent
                             libraries""")
+        parser.add_argument("--extra-files", "-f", nargs='*',
+                            help="""Specify files that will be sent to the
+                            Docker daemon. The daemon will copy those files into
+                            the built image. The ADD directive of the Dockerfile
+                            specify where a file is placed into the image""")
         parser.add_argument("--add-current-user", "-u", dest="user",
                             action="store_true",
                             help="Add the current user to image's passwd")
@@ -270,10 +275,11 @@ class BuildCommand(SubCommand):
                     print "%s exited with code %d" % (docker_pre, rc)
                     return 1
 
-            # Do we include a extra binary?
+            # Include files used by ADD directives found within the Dockerfile.
             if args.include_executable:
-                _copy_binary_with_libs(args.include_executable,
-                                       docker_dir)
+                _copy_binary_with_libs(args.include_executable, docker_dir)
+            for filename in args.extra_files or []:
+                _copy_with_mkdir(filename, docker_dir)
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
-- 
2.11.0

[Qemu-devel] [PATCH v3 03/24] docker: rebuild image if 'extra files' checksum does not match

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/docker.py | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index eef92a54f1..0dd6a3304f 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -38,6 +38,9 @@ def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
     return hashlib.sha1(text).hexdigest()
 
+def _file_checksum(filename):
+    return _text_checksum(open(filename, 'rb').read())
+
 def _guess_docker_command():
     """ Guess a working docker command or raise exception if not found"""
     commands = [["docker"], ["sudo", "-n", "docker"]]
@@ -154,7 +157,7 @@ class Docker(object):
         return labels.get("com.qemu.dockerfile-checksum", "")
 
     def build_image(self, tag, docker_dir, dockerfile,
-                    quiet=True, user=False, argv=None):
+                    quiet=True, user=False, argv=None, extra_files_cksum=[]):
         if argv == None:
             argv = []
 
@@ -170,7 +173,7 @@ class Docker(object):
 
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
-                     _text_checksum(dockerfile))
+                     _text_checksum(dockerfile + "-".join(extra_files_cksum)))
         tmp_df.flush()
 
         self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
@@ -276,16 +279,22 @@ class BuildCommand(SubCommand):
                     return 1
 
             # Include files used by ADD directives found within the Dockerfile.
+            cksum = []
             if args.include_executable:
+                # FIXME: there is no checksum of this executable and the linked
+                # libraries, once the image built any change of this executable
+                # or any library won't trigger another build.
                 _copy_binary_with_libs(args.include_executable, docker_dir)
             for filename in args.include_files or []:
                 _copy_with_mkdir(filename, docker_dir)
+                cksum += [_file_checksum(filename)]
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
                         if k.lower() in FILTERED_ENV_NAMES]
             dkr.build_image(tag, docker_dir, dockerfile,
-                            quiet=args.quiet, user=args.user, argv=argv)
+                            quiet=args.quiet, user=args.user, argv=argv,
+                            extra_files_cksum=cksum)
 
             rmtree(docker_dir)
 
-- 
2.11.0

[Qemu-devel] [PATCH v3 04/24] docker: add 'apt-fake' script which generate fake debian packages

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh

diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
new file mode 100755
index 0000000000..387522c174
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-apt-fake.sh
@@ -0,0 +1,46 @@
+#! /bin/sh
+#
+# Generate fake debian package to resolve unimportant unmet dependencies held
+# by upstream multiarch broken packages.
+#
+# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+test $1 = "install" && shift 1
+
+fake_install()
+{
+    echo "Generating fake $2 $1 $3 ..."
+    (cd /var/cache/apt/archives
+        (cat << 'EOF'
+Section: misc
+Priority: optional
+Standards-Version: 3.9.2
+
+Package: NAME
+Version: VERSION
+Maintainer: qemu-devel@nongnu.org
+Architecture: any
+Multi-Arch: same
+Description: fake NAME
+EOF
+        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
+        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
+        dpkg -i $2_$3_$1.deb
+    )
+}
+
+try_install()
+{
+    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
+    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
+    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
+    apt-get install -q -yy $1 || fake_install $arch $name $vers
+}
+
+for package in $*; do
+    try_install $package
+done
-- 
2.11.0

[Qemu-devel] [PATCH v3 05/24] docker: install ca-certificates package in base image

[Philippe Mathieu-Daudé]

Resolve SSL verification issue at shippable container's git_sync stage:

shippable logs:
--------------
git_sync
- ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
Cloning into '/root/src/github.com/philmd/qemu'...
fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
retrying 1 of 3 times...

Suggested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 52bd79938e..d08def6a8d 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -11,7 +11,7 @@ FROM debian:stable-slim
 
 # Setup some basic tools we need
 RUN apt update
-RUN apt install -yy curl aptitude
+RUN apt install -yy aptitude ca-certificates curl
 
 # Setup Emdebian
 RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-- 
2.11.0

[Qemu-devel] [PATCH v3 06/24] docker: create deb-src entry and setup Emdebian in the same layer

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian.docker | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index d08def6a8d..33ba686892 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -13,13 +13,11 @@ FROM debian:stable-slim
 RUN apt update
 RUN apt install -yy aptitude ca-certificates curl
 
-# Setup Emdebian
-RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
-
 # Install common build utilities
 RUN apt update
 RUN apt install -yy build-essential clang
+
+# Duplicate deb line as deb-src, setup Emdebian
+RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
+    echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
+    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-- 
2.11.0

[Qemu-devel] [PATCH v3 07/24] docker: use better regex to generate deb-src entries

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 33ba686892..e5a10a5875 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -18,6 +18,6 @@ RUN apt update
 RUN apt install -yy build-essential clang
 
 # Duplicate deb line as deb-src, setup Emdebian
-RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
+RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list && \
     echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
     curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-- 
2.11.0

[Qemu-devel] [PATCH v3 08/24] docker: use eatmydata, install common build packages in base image

[Philippe Mathieu-Daudé]

The common build packages are: build-essential clang git bison flex

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian.docker | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index e5a10a5875..c62c87f205 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -11,7 +11,11 @@ FROM debian:stable-slim
 
 # Setup some basic tools we need
 RUN apt update
-RUN apt install -yy aptitude ca-certificates curl
+RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
+RUN DEBIAN_FRONTEND=noninteractive \
+    eatmydata apt install -y --no-install-recommends \
+        aptitude ca-certificates curl \
+        build-essential clang git bison flex
 
 # Install common build utilities
 RUN apt update
-- 
2.11.0

[Qemu-devel] [PATCH v3 09/24] docker: use eatmydata in debian armhf image

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-armhf-cross.docker | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 668d60aeb3..ba8d5a5167 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -8,8 +8,11 @@ FROM qemu:debian
 # Add the foreign architecture we want and install dependencies
 RUN dpkg --add-architecture armhf
 RUN apt update
-RUN apt install -yy crossbuild-essential-armhf
-RUN apt-get build-dep -yy -a armhf qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-armhf
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a armhf qemu
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
-- 
2.11.0

[Qemu-devel] [PATCH v3 10/24] docker: use eatmydata in debian arm64 image

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-arm64-cross.docker | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 592b5d7055..f3db7f4f55 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -8,8 +8,11 @@ FROM qemu:debian
 # Add the foreign architecture we want and install dependencies
 RUN dpkg --add-architecture arm64
 RUN apt update
-RUN apt install -yy crossbuild-essential-arm64
-RUN apt-get build-dep -yy -a arm64 qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-arm64
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a arm64 qemu
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
-- 
2.11.0

[Qemu-devel] [PATCH v3 11/24] docker: add extra libs to armhf target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-armhf-cross.docker | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index ba8d5a5167..e67dfdccc5 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -16,3 +16,14 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:armhf \
+        libbz2-dev:armhf \
+        liblzo2-dev:armhf \
+        libncursesw5-dev:armhf \
+        libnfs-dev:armhf \
+        librdmacm-dev:armhf \
+        libsnappy-dev:armhf \
+        libxen-dev:armhf
-- 
2.11.0

[Qemu-devel] [PATCH v3 12/24] docker: add extra libs to arm64 target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-arm64-cross.docker | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index f3db7f4f55..45b891d57a 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -16,3 +16,14 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:arm64 \
+        libbz2-dev:arm64 \
+        liblzo2-dev:arm64 \
+        libncursesw5-dev:arm64 \
+        libnfs-dev:arm64 \
+        librdmacm-dev:arm64 \
+        libsnappy-dev:arm64 \
+        libxen-dev:arm64
-- 
2.11.0

[Qemu-devel] [PATCH v3 13/24] docker: add extra libs to s390x target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-s390x-cross.docker | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 3a687feda0..cfc354ce5d 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -20,3 +20,13 @@ RUN apt install -yy gcc-multilib-s390x-linux-gnu binutils-multiarch
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=s390x-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:s390x \
+        libbz2-dev:s390x \
+        liblzo2-dev:s390x \
+        libncursesw5-dev:s390x \
+        libnfs-dev:s390x \
+        librdmacm-dev:s390x \
+        libsnappy-dev:s390x
-- 
2.11.0

[Qemu-devel] [PATCH v3 14/24] docker: add mipsel build target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include                      |  3 ++
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 41 ++++++++++++++++++++++
 2 files changed, 44 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 58bf249f02..111b8090b2 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -55,9 +55,12 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(if $(EXTRA_FILES),--include-files $(EXTRA_FILES)),\
 		"BUILD","$*")
 
+docker-image-debian-mipsel-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
+
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
+docker-image-debian-mipsel-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
new file mode 100644
index 0000000000..bda9fac9d9
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -0,0 +1,41 @@
+#
+# Docker mipsel cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture mipsel
+RUN apt-get update
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-mipsel
+
+# <kludge> to fix "following packages have unmet dependencies" ...
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+RUN apt-get install -y --no-install-recommends \
+        equivs \
+        pkg-config
+RUN apt-fake install \
+        pkgconf:mipsel=0.9.7-fake \
+        pkg-config:mipsel=0.28-1.1-fake && \
+    ln -s pkg-config /usr/bin/mipsel-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/mipsel-linux-gnu/pkgconfig
+# </kludge>
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a mipsel qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:mipsel \
+        libbz2-dev:mipsel \
+        liblzo2-dev:mipsel \
+        libncursesw5-dev:mipsel \
+        libnfs-dev:mipsel \
+        librdmacm-dev:mipsel \
+        libsnappy-dev:mipsel
-- 
2.11.0

[Qemu-devel] [PATCH v3 15/24] docker: add powerpc build target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include                      |  4 +--
 .../docker/dockerfiles/debian-powerpc-cross.docker | 40 ++++++++++++++++++++++
 2 files changed, 42 insertions(+), 2 deletions(-)
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 111b8090b2..9815976486 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -56,11 +56,13 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		"BUILD","$*")
 
 docker-image-debian-mipsel-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
+docker-image-debian-powerpc-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
 
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
 docker-image-debian-mipsel-cross: docker-image-debian
+docker-image-debian-powerpc-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
@@ -111,8 +113,6 @@ docker:
 	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
 	@echo '    NOCACHE=1            Ignore cache when build images.'
 	@echo '    EXECUTABLE=<path>    Include executable in image.'
-	@echo '    EXTRA_FILES="<path> [... <path>]"'
-	@echo '                         Include extra files in image.'
 
 # This rule if for directly running against an arbitrary docker target.
 # It is called by the expanded docker targets (e.g. make
diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
new file mode 100644
index 0000000000..fa2cc7a657
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -0,0 +1,40 @@
+#
+# Docker powerpc cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture powerpc
+RUN apt-get update
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-powerpc
+
+# <kludge> to fix "following packages have unmet dependencies" ...
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+RUN apt-get install -y --no-install-recommends \
+        equivs \
+        pkg-config
+RUN apt-fake install \
+        pkg-config:powerpc=0.28-1.1-fake && \
+    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
+# </kludge>
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a powerpc qemu
+RUN DEBIAN_FRONTEND=noninteractive \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:powerpc \
+        libbz2-dev:powerpc \
+        liblzo2-dev:powerpc \
+        libncursesw5-dev:powerpc \
+        libnfs-dev:powerpc \
+        librdmacm-dev:powerpc \
+        libsnappy-dev:powerpc
-- 
2.11.0

[Qemu-devel] [PATCH v3 16/24] shippable: use C locale to simplify console output

[Philippe Mathieu-Daudé]

remove this noise:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = "en_US.UTF-8",
    LC_CTYPE = "en_US.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 653bd750fe..231c29b620 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,5 +1,7 @@
 language: c
 env:
+  global:
+    - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
       TARGET_LIST=arm-softmmu,arm-linux-user
-- 
2.11.0

[Qemu-devel] [PATCH v3 17/24] shippable: build using all available cpus

[Philippe Mathieu-Daudé]

As of this commit:

$ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf _NPROCESSORS_CONF`
container proc: 2 2

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index 231c29b620..1e3ae35dd9 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -20,4 +20,4 @@ build:
   ci:
     - unset CC
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
-    - make -j2
+    - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
-- 
2.11.0

[Qemu-devel] [PATCH v3 18/24] shippable: do not initialize submodules automatically

[Philippe Mathieu-Daudé]

instead do it in the 'ci' target when needed.

for mips64el-softmmu target:
use dtc submodule if distrib packages are too old.

example with outdated libfdt on mips64el-softmmu target (required is >= 1.4.2):
 # dpkg-query --showformat='${Version}\n' --show libfdt-dev
 1.4.0+dfsg-1

shippable output:
----------------
  LINK    mips64el-softmmu/qemu-system-mips64el
../hw/core/loader-fit.o: In function `load_fit':
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
collect2: error: ld returned 1 exit status
Makefile:201: recipe for target 'qemu-system-mips64el' failed
make[1]: *** [qemu-system-mips64el] Error 1
Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
make: *** [subdir-mips64el-softmmu] Error 2

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 1e3ae35dd9..46adfa030f 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,4 +1,6 @@
 language: c
+git:
+   submodules: false
 env:
   global:
     - LC_ALL=C
@@ -19,5 +21,13 @@ build:
     options: "-e HOME=/root"
   ci:
     - unset CC
+    # some targets require newer up to date packages, for example TARGET_LIST matching
+    # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
+    # see the configure script:
+    #    error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
+    #    "  (1) Preferred: Install the DTC (libfdt) devel package"
+    #    "  (2) Fetch the DTC submodule, using:"
+    #    "      git submodule update --init dtc"
+    - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
     - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
-- 
2.11.0

[Qemu-devel] [PATCH v3 19/24] shippable: be verbose while building docker images

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index 46adfa030f..fe360f85cb 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,7 +13,7 @@ env:
       TARGET_LIST=s390x-softmmu,s390x-linux-user
 build:
   pre_ci:
-    - make docker-image-${IMAGE}
+    - make docker-image-${IMAGE} V=1
   pre_ci_boot:
     image_name: qemu
     image_tag: ${IMAGE}
-- 
2.11.0

[Qemu-devel] [PATCH v3 20/24] shippable: add armeb-linux-user target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index fe360f85cb..2070c4d827 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -6,7 +6,7 @@ env:
     - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
-      TARGET_LIST=arm-softmmu,arm-linux-user
+      TARGET_LIST=arm-softmmu,arm-linux-user,armeb-linux-user
     - IMAGE=debian-arm64-cross
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
-- 
2.11.0

[Qemu-devel] [PATCH v3 21/24] shippable: add powerpc target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 2070c4d827..aad66ec5ec 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -11,6 +11,8 @@ env:
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
       TARGET_LIST=s390x-softmmu,s390x-linux-user
+    - IMAGE=debian-powerpc-cross
+      TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE} V=1
-- 
2.11.0

[Qemu-devel] [PATCH v3 22/24] shippable: add mipsel target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index aad66ec5ec..75c2895a21 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,6 +13,9 @@ env:
       TARGET_LIST=s390x-softmmu,s390x-linux-user
     - IMAGE=debian-powerpc-cross
       TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
+    # mips64el-softmmu disabled due to libfdt problem
+    - IMAGE=debian-mipsel-cross
+      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE} V=1
-- 
2.11.0

[Qemu-devel] [PATCH v3 23/24] MAINTAINERS: add Shippable automation platform URL

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index ef2ec58a94..33f47a989b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1857,6 +1857,7 @@ F: .travis.yml
 F: .shippable.yml
 F: tests/docker/
 W: https://travis-ci.org/qemu/qemu
+W: https://app.shippable.com/github/qemu/qemu
 W: http://patchew.org/QEMU/
 
 Documentation
-- 
2.11.0

[Qemu-devel] [PATCH v3 24/24] MAINTAINERS: self-appoint me as reviewer in build/test automation

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 33f47a989b..24b1422231 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1851,6 +1851,7 @@ Build and test automation
 -------------------------
 M: Alex Bennée <alex.bennee@linaro.org>
 M: Fam Zheng <famz@redhat.com>
+R: Philippe Mathieu-Daudé <f4bug@amsat.org>
 L: qemu-devel@nongnu.org
 S: Maintained
 F: .travis.yml
-- 
2.11.0

Re: [Qemu-devel] [PATCH v3 03/24] docker: rebuild image if 'extra files' checksum does not match

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/docker.py | 15 ++++++++++++---
>  1 file changed, 12 insertions(+), 3 deletions(-)

Hmm odd what bracnh did you cut this from. git apply-mbox keeps falling
over:

Applying: docker: rebuild image if 'extra files' checksum does not match
error: sha1 information is lacking or useless (tests/docker/docker.py).
error: could not build fake ancestor
Patch failed at 0001 docker: rebuild image if 'extra files' checksum does not match
The copy of the patch that failed is found in: .git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".


>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index eef92a54f1..0dd6a3304f 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -38,6 +38,9 @@ def _text_checksum(text):
>      """Calculate a digest string unique to the text content"""
>      return hashlib.sha1(text).hexdigest()
>
> +def _file_checksum(filename):
> +    return _text_checksum(open(filename, 'rb').read())
> +
>  def _guess_docker_command():
>      """ Guess a working docker command or raise exception if not found"""
>      commands = [["docker"], ["sudo", "-n", "docker"]]
> @@ -154,7 +157,7 @@ class Docker(object):
>          return labels.get("com.qemu.dockerfile-checksum", "")
>
>      def build_image(self, tag, docker_dir, dockerfile,
> -                    quiet=True, user=False, argv=None):
> +                    quiet=True, user=False, argv=None, extra_files_cksum=[]):
>          if argv == None:
>              argv = []
>
> @@ -170,7 +173,7 @@ class Docker(object):
>
>          tmp_df.write("\n")
>          tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
> -                     _text_checksum(dockerfile))
> +                     _text_checksum(dockerfile + "-".join(extra_files_cksum)))
>          tmp_df.flush()
>
>          self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
> @@ -276,16 +279,22 @@ class BuildCommand(SubCommand):
>                      return 1
>
>              # Include files used by ADD directives found within the Dockerfile.
> +            cksum = []
>              if args.include_executable:
> +                # FIXME: there is no checksum of this executable and the linked
> +                # libraries, once the image built any change of this executable
> +                # or any library won't trigger another build.
>                  _copy_binary_with_libs(args.include_executable, docker_dir)
>              for filename in args.include_files or []:
>                  _copy_with_mkdir(filename, docker_dir)
> +                cksum += [_file_checksum(filename)]
>
>              argv += ["--build-arg=" + k.lower() + "=" + v
>                          for k, v in os.environ.iteritems()
>                          if k.lower() in FILTERED_ENV_NAMES]
>              dkr.build_image(tag, docker_dir, dockerfile,
> -                            quiet=args.quiet, user=args.user, argv=argv)
> +                            quiet=args.quiet, user=args.user, argv=argv,
> +                            extra_files_cksum=cksum)
>
>              rmtree(docker_dir)


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 01/24] docker: let _copy_with_mkdir() sub_path argument be optional

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/docker.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 8747f6a440..6ddc6e4c2a 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -52,7 +52,7 @@ def _guess_docker_command():
>      raise Exception("Cannot find working docker command. Tried:\n%s" % \
>                      commands_txt)
>
> -def _copy_with_mkdir(src, root_dir, sub_path):
> +def _copy_with_mkdir(src, root_dir, sub_path='.'):
>      """Copy src into root_dir, creating sub_path as needed."""
>      dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
>      try:


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 02/24] docker: add --include-files argument to 'build' command

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/Makefile.include |  5 ++++-
>  tests/docker/docker.py        | 12 +++++++++---
>  2 files changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 03eda37bf4..fe1a9a53ff 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -51,6 +51,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
>  		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
>  		$(if $(NOUSER),,--add-current-user) \
> +		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
>  		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
>  		"BUILD","$*")
>
> @@ -107,6 +108,8 @@ docker:
>  	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
>  	@echo '    NOCACHE=1            Ignore cache when build images.'
>  	@echo '    EXECUTABLE=<path>    Include executable in image.'
> +	@echo '    EXTRA_FILES="<path> [... <path>]"'
> +	@echo '                         Include extra files in image.'
>
>  # This rule if for directly running against an arbitrary docker target.
>  # It is called by the expanded docker targets (e.g. make
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 6ddc6e4c2a..68cca25f89 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -237,6 +237,11 @@ class BuildCommand(SubCommand):
>                              help="""Specify a binary that will be copied to the
>                              container together with all its dependent
>                              libraries""")
> +        parser.add_argument("--extra-files", "-f", nargs='*',
> +                            help="""Specify files that will be sent to the
> +                            Docker daemon. The daemon will copy those files into
> +                            the built image. The ADD directive of the Dockerfile
> +                            specify where a file is placed into the image""")
>          parser.add_argument("--add-current-user", "-u", dest="user",
>                              action="store_true",
>                              help="Add the current user to image's passwd")
> @@ -270,10 +275,11 @@ class BuildCommand(SubCommand):
>                      print "%s exited with code %d" % (docker_pre, rc)
>                      return 1
>
> -            # Do we include a extra binary?
> +            # Include files used by ADD directives found within the Dockerfile.
>              if args.include_executable:
> -                _copy_binary_with_libs(args.include_executable,
> -                                       docker_dir)
> +                _copy_binary_with_libs(args.include_executable, docker_dir)
> +            for filename in args.extra_files or []:
> +                _copy_with_mkdir(filename, docker_dir)
>
>              argv += ["--build-arg=" + k.lower() + "=" + v
>                          for k, v in os.environ.iteritems()


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 03/24] docker: rebuild image if 'extra files' checksum does not match

[Philippe Mathieu-Daudé]

Hi Alex,

On 05/22/2017 04:55 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/docker.py | 15 ++++++++++++---
>>  1 file changed, 12 insertions(+), 3 deletions(-)
>
> Hmm odd what bracnh did you cut this from. git apply-mbox keeps falling
> over:
>
> Applying: docker: rebuild image if 'extra files' checksum does not match
> error: sha1 information is lacking or useless (tests/docker/docker.py).
> error: could not build fake ancestor
> Patch failed at 0001 docker: rebuild image if 'extra files' checksum does not match
> The copy of the patch that failed is found in: .git/rebase-apply/patch
> When you have resolved this problem, run "git am --continue".
> If you prefer to skip this patch, run "git am --skip" instead.
> To restore the original branch and stop patching, run "git am --abort".

Weird...

I still have the command in my history, I used:
   $ git format-patch master --cover-letter -v 3 -n -s -o 
outgoing/shippable_v3 --to qemu-devel@nongnu.org --to 'Alex Bennée 
<alex.bennee@linaro.org>' --to 'Fam Zheng <famz@redhat.com>'

Having:
   $ git rev-parse master
56821559f0ba682fe6b367815572e6f974d329ab

You can also fetch this branch here:
   https://github.com/philmd/qemu/tree/docker_shippable_v3
Shippable output:
   https://app.shippable.com/github/philmd/qemu/status/dashboard

Regards,

Phil.

>
>
>>
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index eef92a54f1..0dd6a3304f 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -38,6 +38,9 @@ def _text_checksum(text):
>>      """Calculate a digest string unique to the text content"""
>>      return hashlib.sha1(text).hexdigest()
>>
>> +def _file_checksum(filename):
>> +    return _text_checksum(open(filename, 'rb').read())
>> +
>>  def _guess_docker_command():
>>      """ Guess a working docker command or raise exception if not found"""
>>      commands = [["docker"], ["sudo", "-n", "docker"]]
>> @@ -154,7 +157,7 @@ class Docker(object):
>>          return labels.get("com.qemu.dockerfile-checksum", "")
>>
>>      def build_image(self, tag, docker_dir, dockerfile,
>> -                    quiet=True, user=False, argv=None):
>> +                    quiet=True, user=False, argv=None, extra_files_cksum=[]):
>>          if argv == None:
>>              argv = []
>>
>> @@ -170,7 +173,7 @@ class Docker(object):
>>
>>          tmp_df.write("\n")
>>          tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
>> -                     _text_checksum(dockerfile))
>> +                     _text_checksum(dockerfile + "-".join(extra_files_cksum)))
>>          tmp_df.flush()
>>
>>          self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
>> @@ -276,16 +279,22 @@ class BuildCommand(SubCommand):
>>                      return 1
>>
>>              # Include files used by ADD directives found within the Dockerfile.
>> +            cksum = []
>>              if args.include_executable:
>> +                # FIXME: there is no checksum of this executable and the linked
>> +                # libraries, once the image built any change of this executable
>> +                # or any library won't trigger another build.
>>                  _copy_binary_with_libs(args.include_executable, docker_dir)
>>              for filename in args.include_files or []:
>>                  _copy_with_mkdir(filename, docker_dir)
>> +                cksum += [_file_checksum(filename)]
>>
>>              argv += ["--build-arg=" + k.lower() + "=" + v
>>                          for k, v in os.environ.iteritems()
>>                          if k.lower() in FILTERED_ENV_NAMES]
>>              dkr.build_image(tag, docker_dir, dockerfile,
>> -                            quiet=args.quiet, user=args.user, argv=argv)
>> +                            quiet=args.quiet, user=args.user, argv=argv,
>> +                            extra_files_cksum=cksum)
>>
>>              rmtree(docker_dir)
>
>
> --
> Alex Bennée
>

Re: [Qemu-devel] [PATCH v3 03/24] docker: rebuild image if 'extra files' checksum does not match

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi Alex,
>
> On 05/22/2017 04:55 AM, Alex Bennée wrote:
>>
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>  tests/docker/docker.py | 15 ++++++++++++---
>>>  1 file changed, 12 insertions(+), 3 deletions(-)
>>
>> Hmm odd what bracnh did you cut this from. git apply-mbox keeps falling
>> over:
>>
>> Applying: docker: rebuild image if 'extra files' checksum does not match
>> error: sha1 information is lacking or useless (tests/docker/docker.py).
>> error: could not build fake ancestor
>> Patch failed at 0001 docker: rebuild image if 'extra files' checksum does not match
>> The copy of the patch that failed is found in: .git/rebase-apply/patch
>> When you have resolved this problem, run "git am --continue".
>> If you prefer to skip this patch, run "git am --skip" instead.
>> To restore the original branch and stop patching, run "git am --abort".
>
> Weird...
>
> I still have the command in my history, I used:
>   $ git format-patch master --cover-letter -v 3 -n -s -o
> outgoing/shippable_v3 --to qemu-devel@nongnu.org --to 'Alex Bennée
> <alex.bennee@linaro.org>' --to 'Fam Zheng <famz@redhat.com>'
>
> Having:
>   $ git rev-parse master
> 56821559f0ba682fe6b367815572e6f974d329ab
>
> You can also fetch this branch here:
>   https://github.com/philmd/qemu/tree/docker_shippable_v3

Weird indeed. However my manually applied tree matches yours so I
haven't missed anything, *phew* ;-)

> Shippable output:
>   https://app.shippable.com/github/philmd/qemu/status/dashboard
>
> Regards,
>
> Phil.
>
>>
>>
>>>
>>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>>> index eef92a54f1..0dd6a3304f 100755
>>> --- a/tests/docker/docker.py
>>> +++ b/tests/docker/docker.py
>>> @@ -38,6 +38,9 @@ def _text_checksum(text):
>>>      """Calculate a digest string unique to the text content"""
>>>      return hashlib.sha1(text).hexdigest()
>>>
>>> +def _file_checksum(filename):
>>> +    return _text_checksum(open(filename, 'rb').read())
>>> +
>>>  def _guess_docker_command():
>>>      """ Guess a working docker command or raise exception if not found"""
>>>      commands = [["docker"], ["sudo", "-n", "docker"]]
>>> @@ -154,7 +157,7 @@ class Docker(object):
>>>          return labels.get("com.qemu.dockerfile-checksum", "")
>>>
>>>      def build_image(self, tag, docker_dir, dockerfile,
>>> -                    quiet=True, user=False, argv=None):
>>> +                    quiet=True, user=False, argv=None, extra_files_cksum=[]):
>>>          if argv == None:
>>>              argv = []
>>>
>>> @@ -170,7 +173,7 @@ class Docker(object):
>>>
>>>          tmp_df.write("\n")
>>>          tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
>>> -                     _text_checksum(dockerfile))
>>> +                     _text_checksum(dockerfile + "-".join(extra_files_cksum)))
>>>          tmp_df.flush()
>>>
>>>          self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
>>> @@ -276,16 +279,22 @@ class BuildCommand(SubCommand):
>>>                      return 1
>>>
>>>              # Include files used by ADD directives found within the Dockerfile.
>>> +            cksum = []
>>>              if args.include_executable:
>>> +                # FIXME: there is no checksum of this executable and the linked
>>> +                # libraries, once the image built any change of this executable
>>> +                # or any library won't trigger another build.
>>>                  _copy_binary_with_libs(args.include_executable, docker_dir)
>>>              for filename in args.include_files or []:
>>>                  _copy_with_mkdir(filename, docker_dir)
>>> +                cksum += [_file_checksum(filename)]
>>>
>>>              argv += ["--build-arg=" + k.lower() + "=" + v
>>>                          for k, v in os.environ.iteritems()
>>>                          if k.lower() in FILTERED_ENV_NAMES]
>>>              dkr.build_image(tag, docker_dir, dockerfile,
>>> -                            quiet=args.quiet, user=args.user, argv=argv)
>>> +                            quiet=args.quiet, user=args.user, argv=argv,
>>> +                            extra_files_cksum=cksum)
>>>
>>>              rmtree(docker_dir)
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 04/24] docker: add 'apt-fake' script which generate fake debian packages

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
>  create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>
> diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
> new file mode 100755
> index 0000000000..387522c174
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-apt-fake.sh
> @@ -0,0 +1,46 @@
> +#! /bin/sh
> +#
> +# Generate fake debian package to resolve unimportant unmet dependencies held
> +# by upstream multiarch broken packages.
> +#
> +# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +test $1 = "install" && shift 1
> +
> +fake_install()
> +{
> +    echo "Generating fake $2 $1 $3 ..."
> +    (cd /var/cache/apt/archives
> +        (cat << 'EOF'
> +Section: misc
> +Priority: optional
> +Standards-Version: 3.9.2
> +
> +Package: NAME
> +Version: VERSION
> +Maintainer: qemu-devel@nongnu.org
> +Architecture: any
> +Multi-Arch: same
> +Description: fake NAME
> +EOF
> +        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
> +        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
> +        dpkg -i $2_$3_$1.deb
> +    )
> +}
> +
> +try_install()
> +{
> +    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
> +    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
> +    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
> +    apt-get install -q -yy $1 || fake_install $arch $name $vers
> +}
> +
> +for package in $*; do
> +    try_install $package
> +done


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 05/24] docker: install ca-certificates package in base image

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Resolve SSL verification issue at shippable container's git_sync stage:
>
> shippable logs:
> --------------
> git_sync
> - ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
> Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
> Cloning into '/root/src/github.com/philmd/qemu'...
> fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
> retrying 1 of 3 times...
>
> Suggested-by: Alex Bennée <alex.bennee@linaro.org>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian.docker | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index 52bd79938e..d08def6a8d 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -11,7 +11,7 @@ FROM debian:stable-slim
>
>  # Setup some basic tools we need
>  RUN apt update
> -RUN apt install -yy curl aptitude
> +RUN apt install -yy aptitude ca-certificates curl
>
>  # Setup Emdebian
>  RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 06/24] docker: create deb-src entry and setup Emdebian in the same layer

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian.docker | 12 +++++-------
>  1 file changed, 5 insertions(+), 7 deletions(-)
>
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index d08def6a8d..33ba686892 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -13,13 +13,11 @@ FROM debian:stable-slim
>  RUN apt update
>  RUN apt install -yy aptitude ca-certificates curl
>
> -# Setup Emdebian
> -RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
> -RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
> -
> -# Duplicate deb line as deb-src
> -RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
> -
>  # Install common build utilities
>  RUN apt update
>  RUN apt install -yy build-essential clang
> +
> +# Duplicate deb line as deb-src, setup Emdebian
> +RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
> +    echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
> +    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 07/24] docker: use better regex to generate deb-src entries

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian.docker | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index 33ba686892..e5a10a5875 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -18,6 +18,6 @@ RUN apt update
>  RUN apt install -yy build-essential clang
>
>  # Duplicate deb line as deb-src, setup Emdebian
> -RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
> +RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list && \
>      echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
>      curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 08/24] docker: use eatmydata, install common build packages in base image

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> The common build packages are: build-essential clang git bison flex
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian.docker | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index e5a10a5875..c62c87f205 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -11,7 +11,11 @@ FROM debian:stable-slim
>
>  # Setup some basic tools we need
>  RUN apt update
> -RUN apt install -yy aptitude ca-certificates curl
> +RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
> +RUN DEBIAN_FRONTEND=noninteractive \
> +    eatmydata apt install -y --no-install-recommends \
> +        aptitude ca-certificates curl \
> +        build-essential clang git bison flex
>
>  # Install common build utilities
>  RUN apt update


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 09/24] docker: use eatmydata in debian armhf image

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian-armhf-cross.docker | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
> index 668d60aeb3..ba8d5a5167 100644
> --- a/tests/docker/dockerfiles/debian-armhf-cross.docker
> +++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
> @@ -8,8 +8,11 @@ FROM qemu:debian
>  # Add the foreign architecture we want and install dependencies
>  RUN dpkg --add-architecture armhf
>  RUN apt update
> -RUN apt install -yy crossbuild-essential-armhf
> -RUN apt-get build-dep -yy -a armhf qemu
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get install -y --no-install-recommends \
> +        crossbuild-essential-armhf
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get build-dep -yy -a armhf qemu
>
>  # Specify the cross prefix for this image (see tests/docker/common.rc)
>  ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 10/24] docker: use eatmydata in debian arm64 image

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian-arm64-cross.docker | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
> index 592b5d7055..f3db7f4f55 100644
> --- a/tests/docker/dockerfiles/debian-arm64-cross.docker
> +++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
> @@ -8,8 +8,11 @@ FROM qemu:debian
>  # Add the foreign architecture we want and install dependencies
>  RUN dpkg --add-architecture arm64
>  RUN apt update
> -RUN apt install -yy crossbuild-essential-arm64
> -RUN apt-get build-dep -yy -a arm64 qemu
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get install -y --no-install-recommends \
> +        crossbuild-essential-arm64
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get build-dep -yy -a arm64 qemu
>
>  # Specify the cross prefix for this image (see tests/docker/common.rc)
>  ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 11/24] docker: add extra libs to armhf target to extend codebase coverage

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian-armhf-cross.docker | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
> index ba8d5a5167..e67dfdccc5 100644
> --- a/tests/docker/dockerfiles/debian-armhf-cross.docker
> +++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
> @@ -16,3 +16,14 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>
>  # Specify the cross prefix for this image (see tests/docker/common.rc)
>  ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
> +
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get install -y --no-install-recommends \
> +        glusterfs-common:armhf \
> +        libbz2-dev:armhf \
> +        liblzo2-dev:armhf \
> +        libncursesw5-dev:armhf \
> +        libnfs-dev:armhf \
> +        librdmacm-dev:armhf \
> +        libsnappy-dev:armhf \
> +        libxen-dev:armhf


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 12/24] docker: add extra libs to arm64 target to extend codebase coverage

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian-arm64-cross.docker | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
> index f3db7f4f55..45b891d57a 100644
> --- a/tests/docker/dockerfiles/debian-arm64-cross.docker
> +++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
> @@ -16,3 +16,14 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>
>  # Specify the cross prefix for this image (see tests/docker/common.rc)
>  ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
> +
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get install -y --no-install-recommends \
> +        glusterfs-common:arm64 \
> +        libbz2-dev:arm64 \
> +        liblzo2-dev:arm64 \
> +        libncursesw5-dev:arm64 \
> +        libnfs-dev:arm64 \
> +        librdmacm-dev:arm64 \
> +        libsnappy-dev:arm64 \
> +        libxen-dev:arm64


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 14/24] docker: add mipsel build target

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/Makefile.include                      |  3 ++
>  .../docker/dockerfiles/debian-mipsel-cross.docker  | 41 ++++++++++++++++++++++
>  2 files changed, 44 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 58bf249f02..111b8090b2 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -55,9 +55,12 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		$(if $(EXTRA_FILES),--include-files $(EXTRA_FILES)),\
>  		"BUILD","$*")
>
> +docker-image-debian-mipsel-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
> +
>  # Enforce dependancies for composite images
>  docker-image-debian-armhf-cross: docker-image-debian
>  docker-image-debian-arm64-cross: docker-image-debian
> +docker-image-debian-mipsel-cross: docker-image-debian
>
>  # Expand all the pre-requistes for each docker image and test combination
>  $(foreach i,$(DOCKER_IMAGES), \
> diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
> new file mode 100644
> index 0000000000..bda9fac9d9
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
> @@ -0,0 +1,41 @@
> +#
> +# Docker mipsel cross-compiler target
> +#
> +# This docker target builds on the base debian image.
> +#
> +FROM qemu:debian
> +MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
> +
> +# Add the foreign architecture we want and install dependencies
> +RUN dpkg --add-architecture mipsel
> +RUN apt-get update
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get install -y --no-install-recommends \
> +        crossbuild-essential-mipsel
> +
> +# <kludge> to fix "following packages have unmet dependencies" ...
> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
> +RUN apt-get install -y --no-install-recommends \
> +        equivs \
> +        pkg-config
> +RUN apt-fake install \
> +        pkgconf:mipsel=0.9.7-fake \
> +        pkg-config:mipsel=0.28-1.1-fake && \
> +    ln -s pkg-config /usr/bin/mipsel-linux-gnu-pkg-config
> +ENV PKG_CONFIG_PATH /usr/lib/mipsel-linux-gnu/pkgconfig
> +# </kludge>
> +
> +# Specify the cross prefix for this image (see tests/docker/common.rc)
> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
> +
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get build-dep -yy -a mipsel qemu
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get install -y --no-install-recommends \
> +        glusterfs-common:mipsel \
> +        libbz2-dev:mipsel \
> +        liblzo2-dev:mipsel \
> +        libncursesw5-dev:mipsel \
> +        libnfs-dev:mipsel \
> +        librdmacm-dev:mipsel \
> +        libsnappy-dev:mipsel


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 13/24] docker: add extra libs to s390x target to extend codebase coverage

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian-s390x-cross.docker | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
> index 3a687feda0..cfc354ce5d 100644
> --- a/tests/docker/dockerfiles/debian-s390x-cross.docker
> +++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
> @@ -20,3 +20,13 @@ RUN apt install -yy gcc-multilib-s390x-linux-gnu binutils-multiarch
>
>  # Specify the cross prefix for this image (see tests/docker/common.rc)
>  ENV QEMU_CONFIGURE_OPTS --cross-prefix=s390x-linux-gnu-
> +
> +RUN DEBIAN_FRONTEND=noninteractive \
> +    apt-get install -y --no-install-recommends \
> +        glusterfs-common:s390x \
> +        libbz2-dev:s390x \
> +        liblzo2-dev:s390x \
> +        libncursesw5-dev:s390x \
> +        libnfs-dev:s390x \
> +        librdmacm-dev:s390x \
> +        libsnappy-dev:s390x


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 15/24] docker: add powerpc build target

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include                      |  4 +--
>  .../docker/dockerfiles/debian-powerpc-cross.docker | 40 ++++++++++++++++++++++
>  2 files changed, 42 insertions(+), 2 deletions(-)
>  create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 111b8090b2..9815976486 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -56,11 +56,13 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		"BUILD","$*")
>
>  docker-image-debian-mipsel-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
> +docker-image-debian-powerpc-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
>
>  # Enforce dependancies for composite images
>  docker-image-debian-armhf-cross: docker-image-debian
>  docker-image-debian-arm64-cross: docker-image-debian
>  docker-image-debian-mipsel-cross: docker-image-debian
> +docker-image-debian-powerpc-cross: docker-image-debian
>
>  # Expand all the pre-requistes for each docker image and test combination
>  $(foreach i,$(DOCKER_IMAGES), \
> @@ -111,8 +113,6 @@ docker:
>  	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
>  	@echo '    NOCACHE=1            Ignore cache when build images.'
>  	@echo '    EXECUTABLE=<path>    Include executable in image.'
> -	@echo '    EXTRA_FILES="<path> [... <path>]"'
> -	@echo '                         Include extra files in image.'

I'm fairly sure you didn't want to do this.

>
>  # This rule if for directly running against an arbitrary docker target.
>  # It is called by the expanded docker targets (e.g. make
> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
> new file mode 100644
> index 0000000000..fa2cc7a657
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
> @@ -0,0 +1,40 @@
> +#
> +# Docker powerpc cross-compiler target
> +#
> +# This docker target builds on the base debian image.
> +#
> +FROM qemu:debian
> +MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
> +
> +# Add the foreign architecture we want and install dependencies
> +RUN dpkg --add-architecture powerpc
> +RUN apt-get update
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get install -y --no-install-recommends \
> +        crossbuild-essential-powerpc
> +
> +# <kludge> to fix "following packages have unmet dependencies" ...
> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
> +RUN apt-get install -y --no-install-recommends \
> +        equivs \
> +        pkg-config
> +RUN apt-fake install \
> +        pkg-config:powerpc=0.28-1.1-fake && \
> +    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
> +# </kludge>
> +
> +# Specify the cross prefix for this image (see tests/docker/common.rc)
> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
> +
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get build-dep -yy -a powerpc qemu
> +RUN DEBIAN_FRONTEND=noninteractive \
> +    apt-get install -y --no-install-recommends \
> +        glusterfs-common:powerpc \
> +        libbz2-dev:powerpc \
> +        liblzo2-dev:powerpc \
> +        libncursesw5-dev:powerpc \
> +        libnfs-dev:powerpc \
> +        librdmacm-dev:powerpc \
> +        libsnappy-dev:powerpc


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 17/24] shippable: build using all available cpus

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> As of this commit:
>
> $ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf _NPROCESSORS_CONF`
> container proc: 2 2
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .shippable.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 231c29b620..1e3ae35dd9 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -20,4 +20,4 @@ build:
>    ci:
>      - unset CC
>      - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
> -    - make -j2
> +    - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 18/24] shippable: do not initialize submodules automatically

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> instead do it in the 'ci' target when needed.
>
> for mips64el-softmmu target:
> use dtc submodule if distrib packages are too old.
>
> example with outdated libfdt on mips64el-softmmu target (required is >= 1.4.2):
>  # dpkg-query --showformat='${Version}\n' --show libfdt-dev
>  1.4.0+dfsg-1
>
> shippable output:
> ----------------
>   LINK    mips64el-softmmu/qemu-system-mips64el
> ../hw/core/loader-fit.o: In function `load_fit':
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
> collect2: error: ld returned 1 exit status
> Makefile:201: recipe for target 'qemu-system-mips64el' failed
> make[1]: *** [qemu-system-mips64el] Error 1
> Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
> make: *** [subdir-mips64el-softmmu] Error 2
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .shippable.yml | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 1e3ae35dd9..46adfa030f 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -1,4 +1,6 @@
>  language: c
> +git:
> +   submodules: false
>  env:
>    global:
>      - LC_ALL=C
> @@ -19,5 +21,13 @@ build:
>      options: "-e HOME=/root"
>    ci:
>      - unset CC
> +    # some targets require newer up to date packages, for example TARGET_LIST matching
> +    # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
> +    # see the configure script:
> +    #    error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
> +    #    "  (1) Preferred: Install the DTC (libfdt) devel package"
> +    #    "  (2) Fetch the DTC submodule, using:"
> +    #    "      git submodule update --init dtc"
> +    - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
>      - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
>      - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 19/24] shippable: be verbose while building docker images

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .shippable.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 46adfa030f..fe360f85cb 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -13,7 +13,7 @@ env:
>        TARGET_LIST=s390x-softmmu,s390x-linux-user
>  build:
>    pre_ci:
> -    - make docker-image-${IMAGE}
> +    - make docker-image-${IMAGE} V=1
>    pre_ci_boot:
>      image_name: qemu
>      image_tag: ${IMAGE}


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 20/24] shippable: add armeb-linux-user target

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .shippable.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index fe360f85cb..2070c4d827 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -6,7 +6,7 @@ env:
>      - LC_ALL=C
>    matrix:
>      - IMAGE=debian-armhf-cross
> -      TARGET_LIST=arm-softmmu,arm-linux-user
> +      TARGET_LIST=arm-softmmu,arm-linux-user,armeb-linux-user
>      - IMAGE=debian-arm64-cross
>        TARGET_LIST=aarch64-softmmu,aarch64-linux-user
>      - IMAGE=debian-s390x-cross


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 21/24] shippable: add powerpc target

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .shippable.yml | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 2070c4d827..aad66ec5ec 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -11,6 +11,8 @@ env:
>        TARGET_LIST=aarch64-softmmu,aarch64-linux-user
>      - IMAGE=debian-s390x-cross
>        TARGET_LIST=s390x-softmmu,s390x-linux-user
> +    - IMAGE=debian-powerpc-cross
> +      TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
>  build:
>    pre_ci:
>      - make docker-image-${IMAGE} V=1


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 22/24] shippable: add mipsel target

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .shippable.yml | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index aad66ec5ec..75c2895a21 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -13,6 +13,9 @@ env:
>        TARGET_LIST=s390x-softmmu,s390x-linux-user
>      - IMAGE=debian-powerpc-cross
>        TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
> +    # mips64el-softmmu disabled due to libfdt problem
> +    - IMAGE=debian-mipsel-cross
> +      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
>  build:
>    pre_ci:
>      - make docker-image-${IMAGE} V=1


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 15/24] docker: add powerpc build target

[Philippe Mathieu-Daudé]

Hi Alex,

On 05/22/2017 11:08 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/Makefile.include                      |  4 +--
>>  .../docker/dockerfiles/debian-powerpc-cross.docker | 40 ++++++++++++++++++++++
>>  2 files changed, 42 insertions(+), 2 deletions(-)
>>  create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 111b8090b2..9815976486 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -56,11 +56,13 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>>  		"BUILD","$*")
>>
>>  docker-image-debian-mipsel-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
>> +docker-image-debian-powerpc-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
>>
>>  # Enforce dependancies for composite images
>>  docker-image-debian-armhf-cross: docker-image-debian
>>  docker-image-debian-arm64-cross: docker-image-debian
>>  docker-image-debian-mipsel-cross: docker-image-debian
>> +docker-image-debian-powerpc-cross: docker-image-debian
>>
>>  # Expand all the pre-requistes for each docker image and test combination
>>  $(foreach i,$(DOCKER_IMAGES), \
>> @@ -111,8 +113,6 @@ docker:
>>  	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
>>  	@echo '    NOCACHE=1            Ignore cache when build images.'
>>  	@echo '    EXECUTABLE=<path>    Include executable in image.'
>> -	@echo '    EXTRA_FILES="<path> [... <path>]"'
>> -	@echo '                         Include extra files in image.'
>
> I'm fairly sure you didn't want to do this.

Ups, good catch :) Rebase mistake.
Are you Ok with those 2 lines back? (I think since it is pretty much a 
copy/paste of mipsel Dockerfile).
I'll send fixed and your other r-b after Fam review, thanks!

>
>>
>>  # This rule if for directly running against an arbitrary docker target.
>>  # It is called by the expanded docker targets (e.g. make
>> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> new file mode 100644
>> index 0000000000..fa2cc7a657
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> @@ -0,0 +1,40 @@
>> +#
>> +# Docker powerpc cross-compiler target
>> +#
>> +# This docker target builds on the base debian image.
>> +#
>> +FROM qemu:debian
>> +MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
>> +
>> +# Add the foreign architecture we want and install dependencies
>> +RUN dpkg --add-architecture powerpc
>> +RUN apt-get update
>> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>> +    apt-get install -y --no-install-recommends \
>> +        crossbuild-essential-powerpc
>> +
>> +# <kludge> to fix "following packages have unmet dependencies" ...
>> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
>> +RUN apt-get install -y --no-install-recommends \
>> +        equivs \
>> +        pkg-config
>> +RUN apt-fake install \
>> +        pkg-config:powerpc=0.28-1.1-fake && \
>> +    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
>> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
>> +# </kludge>
>> +
>> +# Specify the cross prefix for this image (see tests/docker/common.rc)
>> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
>> +
>> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>> +    apt-get build-dep -yy -a powerpc qemu
>> +RUN DEBIAN_FRONTEND=noninteractive \
>> +    apt-get install -y --no-install-recommends \
>> +        glusterfs-common:powerpc \
>> +        libbz2-dev:powerpc \
>> +        liblzo2-dev:powerpc \
>> +        libncursesw5-dev:powerpc \
>> +        libnfs-dev:powerpc \
>> +        librdmacm-dev:powerpc \
>> +        libsnappy-dev:powerpc
>
>
> --
> Alex Bennée
>

Re: [Qemu-devel] [PATCH v3 02/24] docker: add --include-files argument to 'build' command

[Fam Zheng]

On Sun, 05/21 00:29, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include |  5 ++++-
>  tests/docker/docker.py        | 12 +++++++++---
>  2 files changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 03eda37bf4..fe1a9a53ff 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -51,6 +51,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
>  		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
>  		$(if $(NOUSER),,--add-current-user) \
> +		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
>  		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
>  		"BUILD","$*")
>  
> @@ -107,6 +108,8 @@ docker:
>  	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
>  	@echo '    NOCACHE=1            Ignore cache when build images.'
>  	@echo '    EXECUTABLE=<path>    Include executable in image.'
> +	@echo '    EXTRA_FILES="<path> [... <path>]"'
> +	@echo '                         Include extra files in image.'
>  
>  # This rule if for directly running against an arbitrary docker target.
>  # It is called by the expanded docker targets (e.g. make
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 6ddc6e4c2a..68cca25f89 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -237,6 +237,11 @@ class BuildCommand(SubCommand):
>                              help="""Specify a binary that will be copied to the
>                              container together with all its dependent
>                              libraries""")
> +        parser.add_argument("--extra-files", "-f", nargs='*',
> +                            help="""Specify files that will be sent to the
> +                            Docker daemon. The daemon will copy those files into
> +                            the built image. The ADD directive of the Dockerfile
> +                            specify where a file is placed into the image""")

Or more precisely, "The daemon will copy these files into the docker build
directory, which can be copied to the built image with ADD directives in
Dockerfile"?

>          parser.add_argument("--add-current-user", "-u", dest="user",
>                              action="store_true",
>                              help="Add the current user to image's passwd")
> @@ -270,10 +275,11 @@ class BuildCommand(SubCommand):
>                      print "%s exited with code %d" % (docker_pre, rc)
>                      return 1
>  
> -            # Do we include a extra binary?
> +            # Include files used by ADD directives found within the Dockerfile.
>              if args.include_executable:
> -                _copy_binary_with_libs(args.include_executable,
> -                                       docker_dir)
> +                _copy_binary_with_libs(args.include_executable, docker_dir)
> +            for filename in args.extra_files or []:
> +                _copy_with_mkdir(filename, docker_dir)
>  
>              argv += ["--build-arg=" + k.lower() + "=" + v
>                          for k, v in os.environ.iteritems()
> -- 
> 2.11.0
> 

Re: [Qemu-devel] [PATCH v3 03/24] docker: rebuild image if 'extra files' checksum does not match

[Fam Zheng]

On Sun, 05/21 00:29, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/docker.py | 15 ++++++++++++---
>  1 file changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index eef92a54f1..0dd6a3304f 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -38,6 +38,9 @@ def _text_checksum(text):
>      """Calculate a digest string unique to the text content"""
>      return hashlib.sha1(text).hexdigest()
>  
> +def _file_checksum(filename):
> +    return _text_checksum(open(filename, 'rb').read())
> +
>  def _guess_docker_command():
>      """ Guess a working docker command or raise exception if not found"""
>      commands = [["docker"], ["sudo", "-n", "docker"]]
> @@ -154,7 +157,7 @@ class Docker(object):
>          return labels.get("com.qemu.dockerfile-checksum", "")
>  
>      def build_image(self, tag, docker_dir, dockerfile,
> -                    quiet=True, user=False, argv=None):
> +                    quiet=True, user=False, argv=None, extra_files_cksum=[]):
>          if argv == None:
>              argv = []
>  
> @@ -170,7 +173,7 @@ class Docker(object):
>  
>          tmp_df.write("\n")
>          tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
> -                     _text_checksum(dockerfile))
> +                     _text_checksum(dockerfile + "-".join(extra_files_cksum)))

Since @dockerfile is the content of the the Dockerfile, how about concatenating
with "\n":

                        _text_checksum("\n".join([dockerfile] + extra_files_cksum))

?


>          tmp_df.flush()
>  
>          self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
> @@ -276,16 +279,22 @@ class BuildCommand(SubCommand):
>                      return 1
>  
>              # Include files used by ADD directives found within the Dockerfile.
> +            cksum = []
>              if args.include_executable:
> +                # FIXME: there is no checksum of this executable and the linked
> +                # libraries, once the image built any change of this executable
> +                # or any library won't trigger another build.
>                  _copy_binary_with_libs(args.include_executable, docker_dir)
>              for filename in args.include_files or []:
>                  _copy_with_mkdir(filename, docker_dir)
> +                cksum += [_file_checksum(filename)]
>  
>              argv += ["--build-arg=" + k.lower() + "=" + v
>                          for k, v in os.environ.iteritems()
>                          if k.lower() in FILTERED_ENV_NAMES]
>              dkr.build_image(tag, docker_dir, dockerfile,
> -                            quiet=args.quiet, user=args.user, argv=argv)
> +                            quiet=args.quiet, user=args.user, argv=argv,
> +                            extra_files_cksum=cksum)
>  
>              rmtree(docker_dir)
>  
> -- 
> 2.11.0
> 

Re: [Qemu-devel] [PATCH v3 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Fam Zheng]

On Sun, 05/21 00:29, Philippe Mathieu-Daudé wrote:
> This patchset add 2 more architectures to the cross-build farm.
> 
> There is still some issue trying to cross-build mips64el-softmmu, it seems the
> cross tools use the system outdated libfdt instead of the one checkouted in the
> dtc submodule. I disabled this target for now.
> 
> The branch https://github.com/philmd/qemu/tree/docker_shippable_v3 can be
> checked at Shippable:
>   https://app.shippable.com/github/philmd/qemu/status/dashboard
> 
> Each arch builds in around ~9min. Using Shippable free open source projects
> service, the 5 jobs take ~38-44min in total.
> 
> v3:
> - Addressed review feedbacks from Fam:
> - Keep building images in various layers, but use DEBIAN_FRONTEND=noninteractive
> - Document '--extra-files', now it supports adding various files at once
> - Checksum extra files to trigger a docker image rebuild if modified
> - Use better regex to generate deb-src entries
> - Reordered extra libs, to ease further add/remove diffs

Looks good in general, thanks! I've had two small questions, apart from that
it's ready to go.

Fam

Re: [Qemu-devel] [PATCH v3 02/24] docker: add --include-files argument to 'build' command

[Fam Zheng]

On Wed, 05/24 13:21, Fam Zheng wrote:
> > diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> > index 6ddc6e4c2a..68cca25f89 100755
> > --- a/tests/docker/docker.py
> > +++ b/tests/docker/docker.py
> > @@ -237,6 +237,11 @@ class BuildCommand(SubCommand):
> >                              help="""Specify a binary that will be copied to the
> >                              container together with all its dependent
> >                              libraries""")
> > +        parser.add_argument("--extra-files", "-f", nargs='*',
> > +                            help="""Specify files that will be sent to the
> > +                            Docker daemon. The daemon will copy those files into
> > +                            the built image. The ADD directive of the Dockerfile
> > +                            specify where a file is placed into the image""")
> 
> Or more precisely, "The daemon will copy these files into the docker build
> directory, which can be copied to the built image with ADD directives in
> Dockerfile"?

Hmm, not copied by daemon, maybe "The files will be sent...". But you have the
idea.

Fam

Re: [Qemu-devel] [PATCH v3 15/24] docker: add powerpc build target

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi Alex,
>
> On 05/22/2017 11:08 AM, Alex Bennée wrote:
>>
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>  tests/docker/Makefile.include                      |  4 +--
>>>  .../docker/dockerfiles/debian-powerpc-cross.docker | 40 ++++++++++++++++++++++
>>>  2 files changed, 42 insertions(+), 2 deletions(-)
>>>  create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker
>>>
>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>>> index 111b8090b2..9815976486 100644
>>> --- a/tests/docker/Makefile.include
>>> +++ b/tests/docker/Makefile.include
>>> @@ -56,11 +56,13 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>>>  		"BUILD","$*")
>>>
>>>  docker-image-debian-mipsel-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
>>> +docker-image-debian-powerpc-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
>>>
>>>  # Enforce dependancies for composite images
>>>  docker-image-debian-armhf-cross: docker-image-debian
>>>  docker-image-debian-arm64-cross: docker-image-debian
>>>  docker-image-debian-mipsel-cross: docker-image-debian
>>> +docker-image-debian-powerpc-cross: docker-image-debian
>>>
>>>  # Expand all the pre-requistes for each docker image and test combination
>>>  $(foreach i,$(DOCKER_IMAGES), \
>>> @@ -111,8 +113,6 @@ docker:
>>>  	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
>>>  	@echo '    NOCACHE=1            Ignore cache when build images.'
>>>  	@echo '    EXECUTABLE=<path>    Include executable in image.'
>>> -	@echo '    EXTRA_FILES="<path> [... <path>]"'
>>> -	@echo '                         Include extra files in image.'
>>
>> I'm fairly sure you didn't want to do this.
>
> Ups, good catch :) Rebase mistake.
> Are you Ok with those 2 lines back? (I think since it is pretty much a
> copy/paste of mipsel Dockerfile).
> I'll send fixed and your other r-b after Fam review, thanks!

Sure its fine with that fixed:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

>
>>
>>>
>>>  # This rule if for directly running against an arbitrary docker target.
>>>  # It is called by the expanded docker targets (e.g. make
>>> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>>> new file mode 100644
>>> index 0000000000..fa2cc7a657
>>> --- /dev/null
>>> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>>> @@ -0,0 +1,40 @@
>>> +#
>>> +# Docker powerpc cross-compiler target
>>> +#
>>> +# This docker target builds on the base debian image.
>>> +#
>>> +FROM qemu:debian
>>> +MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> +
>>> +# Add the foreign architecture we want and install dependencies
>>> +RUN dpkg --add-architecture powerpc
>>> +RUN apt-get update
>>> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>>> +    apt-get install -y --no-install-recommends \
>>> +        crossbuild-essential-powerpc
>>> +
>>> +# <kludge> to fix "following packages have unmet dependencies" ...
>>> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
>>> +RUN apt-get install -y --no-install-recommends \
>>> +        equivs \
>>> +        pkg-config
>>> +RUN apt-fake install \
>>> +        pkg-config:powerpc=0.28-1.1-fake && \
>>> +    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
>>> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
>>> +# </kludge>
>>> +
>>> +# Specify the cross prefix for this image (see tests/docker/common.rc)
>>> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
>>> +
>>> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>>> +    apt-get build-dep -yy -a powerpc qemu
>>> +RUN DEBIAN_FRONTEND=noninteractive \
>>> +    apt-get install -y --no-install-recommends \
>>> +        glusterfs-common:powerpc \
>>> +        libbz2-dev:powerpc \
>>> +        liblzo2-dev:powerpc \
>>> +        libncursesw5-dev:powerpc \
>>> +        libnfs-dev:powerpc \
>>> +        librdmacm-dev:powerpc \
>>> +        libsnappy-dev:powerpc
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 03/24] docker: rebuild image if 'extra files' checksum does not match

[Philippe Mathieu-Daudé]

On 05/24/2017 02:24 AM, Fam Zheng wrote:
> On Sun, 05/21 00:29, Philippe Mathieu-Daudé wrote:
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/docker.py | 15 ++++++++++++---
>>  1 file changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index eef92a54f1..0dd6a3304f 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -38,6 +38,9 @@ def _text_checksum(text):
>>      """Calculate a digest string unique to the text content"""
>>      return hashlib.sha1(text).hexdigest()
>>
>> +def _file_checksum(filename):
>> +    return _text_checksum(open(filename, 'rb').read())
>> +
>>  def _guess_docker_command():
>>      """ Guess a working docker command or raise exception if not found"""
>>      commands = [["docker"], ["sudo", "-n", "docker"]]
>> @@ -154,7 +157,7 @@ class Docker(object):
>>          return labels.get("com.qemu.dockerfile-checksum", "")
>>
>>      def build_image(self, tag, docker_dir, dockerfile,
>> -                    quiet=True, user=False, argv=None):
>> +                    quiet=True, user=False, argv=None, extra_files_cksum=[]):
>>          if argv == None:
>>              argv = []
>>
>> @@ -170,7 +173,7 @@ class Docker(object):
>>
>>          tmp_df.write("\n")
>>          tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
>> -                     _text_checksum(dockerfile))
>> +                     _text_checksum(dockerfile + "-".join(extra_files_cksum)))
>
> Since @dockerfile is the content of the the Dockerfile, how about concatenating
> with "\n":
>
>                         _text_checksum("\n".join([dockerfile] + extra_files_cksum))
>
> ?

This is only the text to hash, I don't think it will make a difference 
since it isn't useful even debugging the script, but it may be cleaner 
so I'll do it (nonetheless python ends up weirdly indented).

>
>
>>          tmp_df.flush()
>>
>>          self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
>> @@ -276,16 +279,22 @@ class BuildCommand(SubCommand):
>>                      return 1
>>
>>              # Include files used by ADD directives found within the Dockerfile.
>> +            cksum = []
>>              if args.include_executable:
>> +                # FIXME: there is no checksum of this executable and the linked
>> +                # libraries, once the image built any change of this executable
>> +                # or any library won't trigger another build.
>>                  _copy_binary_with_libs(args.include_executable, docker_dir)
>>              for filename in args.include_files or []:
>>                  _copy_with_mkdir(filename, docker_dir)
>> +                cksum += [_file_checksum(filename)]
>>
>>              argv += ["--build-arg=" + k.lower() + "=" + v
>>                          for k, v in os.environ.iteritems()
>>                          if k.lower() in FILTERED_ENV_NAMES]
>>              dkr.build_image(tag, docker_dir, dockerfile,
>> -                            quiet=args.quiet, user=args.user, argv=argv)
>> +                            quiet=args.quiet, user=args.user, argv=argv,
>> +                            extra_files_cksum=cksum)
>>
>>              rmtree(docker_dir)
>>
>> --
>> 2.11.0
>>

Re: [Qemu-devel] [PATCH v3 02/24] docker: add --include-files argument to 'build' command

[Philippe Mathieu-Daudé]

On 05/24/2017 02:34 AM, Fam Zheng wrote:
> On Wed, 05/24 13:21, Fam Zheng wrote:
>>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>>> index 6ddc6e4c2a..68cca25f89 100755
>>> --- a/tests/docker/docker.py
>>> +++ b/tests/docker/docker.py
>>> @@ -237,6 +237,11 @@ class BuildCommand(SubCommand):
>>>                              help="""Specify a binary that will be copied to the
>>>                              container together with all its dependent
>>>                              libraries""")
>>> +        parser.add_argument("--extra-files", "-f", nargs='*',
>>> +                            help="""Specify files that will be sent to the
>>> +                            Docker daemon. The daemon will copy those files into
>>> +                            the built image. The ADD directive of the Dockerfile
>>> +                            specify where a file is placed into the image""")
>>
>> Or more precisely, "The daemon will copy these files into the docker build
>> directory, which can be copied to the built image with ADD directives in
>> Dockerfile"?
>
> Hmm, not copied by daemon, maybe "The files will be sent...". But you have the
> idea.

I tried to keep it simple like this:
"Specify files that will be copied in the Docker image, fulfilling the 
ADD directives from the Dockerfile"