[Qemu-devel] [PATCH v5 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Qemu-devel] [PATCH v5 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Philippe Mathieu-Daudé]

This patchset add 2 more architectures to the cross-build farm.

There is still some issue trying to cross-build mips64el-softmmu, it seems the
cross tools use the system outdated libfdt instead of the one checkouted in the
dtc submodule. I disabled this target for now.

The branch https://github.com/philmd/qemu/tree/docker_shippable_v5 can be
checked at Shippable:
  https://app.shippable.com/github/philmd/qemu/status/dashboard

Each arch builds in around ~9min. Using Shippable free open source projects
service, the 5 jobs take ~38-44min in total.

v5:
- Addressed minor review feedback from Alex/Fam (improving a comment)
- Added Alex R-b

v4:
- Addressed review feedbacks from Fam (improving english, cleaner hashing)
- Added Alex R-b

v3:
- Addressed review feedbacks from Fam:
- Keep building images in various layers, but use DEBIAN_FRONTEND=noninteractive
- Document '--extra-files', now it supports adding various files at once
- Checksum extra files to trigger a docker image rebuild if modified
- Use better regex to generate deb-src entries
- Reordered extra libs, to ease further add/remove diffs

v2:
- Addressed review feedback from Alex, added his R-B
- Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
  installing the ca-certificates package.
- Squashed/split some commits
- use 'apt-get clean' instead of brutal 'rm -rf'
- checked how many cores are available on Shippable and use them fully
  (reduced total time from 44min to 38min)
- build armeb-linux-user target

v1:
- Reorganize Dockerfiles to use less layers, resulting in smaller images.
  This also reduce time of image transfer, for example while using:
  `docker save qemu:debian | ssh remote docker load`
- Install more debian packages so the configure script enable more features and
  more code can be compiled/covered.
- There are still some incorrect multiarch packages on debian/jessie used in
  the docker images, add a script to generate fake packages and avoid
  dependencies issues.
- Modify the docker.py script to include an extra file while building images.

Regards,

Phil.

Philippe Mathieu-Daudé (24):
  docker: let _copy_with_mkdir() sub_path argument be optional
  docker: add --include-files argument to 'build' command
  docker: rebuild image if 'extra files' checksum does not match
  docker: add 'apt-fake' script which generate fake debian packages
  docker: install ca-certificates package in base image
  docker: create deb-src entry and setup Emdebian in the same layer
  docker: use better regex to generate deb-src entries
  docker: use eatmydata, install common build packages in base image
  docker: use eatmydata in debian armhf image
  docker: use eatmydata in debian arm64 image
  docker: add extra libs to armhf target to extend codebase coverage
  docker: add extra libs to arm64 target to extend codebase coverage
  docker: add extra libs to s390x target to extend codebase coverage
  docker: add mipsel build target
  docker: add powerpc build target
  shippable: use C locale to simplify console output
  shippable: build using all available cpus
  shippable: do not initialize submodules automatically
  shippable: be verbose while building docker images
  shippable: add armeb-linux-user target
  shippable: add powerpc target
  shippable: add mipsel target
  MAINTAINERS: add Shippable automation platform URL
  MAINTAINERS: self-appoint me as reviewer in build/test automation

 .shippable.yml                                     | 23 +++++++++--
 MAINTAINERS                                        |  2 +
 tests/docker/Makefile.include                      |  8 ++++
 tests/docker/docker.py                             | 30 ++++++++++----
 tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
 tests/docker/dockerfiles/debian-arm64-cross.docker | 18 ++++++++-
 tests/docker/dockerfiles/debian-armhf-cross.docker | 18 ++++++++-
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 41 +++++++++++++++++++
 .../docker/dockerfiles/debian-powerpc-cross.docker | 40 +++++++++++++++++++
 tests/docker/dockerfiles/debian-s390x-cross.docker | 10 +++++
 tests/docker/dockerfiles/debian.docker             | 18 +++++----
 11 files changed, 232 insertions(+), 22 deletions(-)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

-- 
2.11.0

[Qemu-devel] [PATCH v5 01/24] docker: let _copy_with_mkdir() sub_path argument be optional

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 8747f6a440..6ddc6e4c2a 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -52,7 +52,7 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
-def _copy_with_mkdir(src, root_dir, sub_path):
+def _copy_with_mkdir(src, root_dir, sub_path='.'):
     """Copy src into root_dir, creating sub_path as needed."""
     dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
     try:
-- 
2.11.0

[Qemu-devel] [PATCH v5 02/24] docker: add --include-files argument to 'build' command

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include |  3 +++
 tests/docker/docker.py        | 12 +++++++++---
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 03eda37bf4..fe1a9a53ff 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -51,6 +51,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
 		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
 		$(if $(NOUSER),,--add-current-user) \
+		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
 		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
 		"BUILD","$*")
 
@@ -107,6 +108,8 @@ docker:
 	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
 	@echo '    NOCACHE=1            Ignore cache when build images.'
 	@echo '    EXECUTABLE=<path>    Include executable in image.'
+	@echo '    EXTRA_FILES="<path> [... <path>]"'
+	@echo '                         Include extra files in image.'
 
 # This rule if for directly running against an arbitrary docker target.
 # It is called by the expanded docker targets (e.g. make
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 6ddc6e4c2a..10fa907905 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -237,6 +237,10 @@ class BuildCommand(SubCommand):
                             help="""Specify a binary that will be copied to the
                             container together with all its dependent
                             libraries""")
+        parser.add_argument("--extra-files", "-f", nargs='*',
+                            help="""Specify files that will be copied in the
+                            Docker image, fulfilling the ADD directive from the
+                            Dockerfile""")
         parser.add_argument("--add-current-user", "-u", dest="user",
                             action="store_true",
                             help="Add the current user to image's passwd")
@@ -270,10 +274,12 @@ class BuildCommand(SubCommand):
                     print "%s exited with code %d" % (docker_pre, rc)
                     return 1
 
-            # Do we include a extra binary?
+            # Copy any extra files into the Docker context. These can be
+            # included by the use of the ADD directive in the Dockerfile.
             if args.include_executable:
-                _copy_binary_with_libs(args.include_executable,
-                                       docker_dir)
+                _copy_binary_with_libs(args.include_executable, docker_dir)
+            for filename in args.extra_files or []:
+                _copy_with_mkdir(filename, docker_dir)
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
-- 
2.11.0

[Qemu-devel] [PATCH v5 03/24] docker: rebuild image if 'extra files' checksum does not match

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 10fa907905..e707e5bcca 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -38,6 +38,9 @@ def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
     return hashlib.sha1(text).hexdigest()
 
+def _file_checksum(filename):
+    return _text_checksum(open(filename, 'rb').read())
+
 def _guess_docker_command():
     """ Guess a working docker command or raise exception if not found"""
     commands = [["docker"], ["sudo", "-n", "docker"]]
@@ -154,7 +157,7 @@ class Docker(object):
         return labels.get("com.qemu.dockerfile-checksum", "")
 
     def build_image(self, tag, docker_dir, dockerfile,
-                    quiet=True, user=False, argv=None):
+                    quiet=True, user=False, argv=None, extra_files_cksum=[]):
         if argv == None:
             argv = []
 
@@ -170,7 +173,8 @@ class Docker(object):
 
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
-                     _text_checksum(dockerfile))
+                     _text_checksum("\n".join([dockerfile] +
+                                    extra_files_cksum)))
         tmp_df.flush()
 
         self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
@@ -276,16 +280,22 @@ class BuildCommand(SubCommand):
 
             # Copy any extra files into the Docker context. These can be
             # included by the use of the ADD directive in the Dockerfile.
+            cksum = []
             if args.include_executable:
+                # FIXME: there is no checksum of this executable and the linked
+                # libraries, once the image built any change of this executable
+                # or any library won't trigger another build.
                 _copy_binary_with_libs(args.include_executable, docker_dir)
             for filename in args.extra_files or []:
                 _copy_with_mkdir(filename, docker_dir)
+                cksum += [_file_checksum(filename)]
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
                         if k.lower() in FILTERED_ENV_NAMES]
             dkr.build_image(tag, docker_dir, dockerfile,
-                            quiet=args.quiet, user=args.user, argv=argv)
+                            quiet=args.quiet, user=args.user, argv=argv,
+                            extra_files_cksum=cksum)
 
             rmtree(docker_dir)
 
-- 
2.11.0

[Qemu-devel] [PATCH v5 04/24] docker: add 'apt-fake' script which generate fake debian packages

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh

diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
new file mode 100755
index 0000000000..387522c174
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-apt-fake.sh
@@ -0,0 +1,46 @@
+#! /bin/sh
+#
+# Generate fake debian package to resolve unimportant unmet dependencies held
+# by upstream multiarch broken packages.
+#
+# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+test $1 = "install" && shift 1
+
+fake_install()
+{
+    echo "Generating fake $2 $1 $3 ..."
+    (cd /var/cache/apt/archives
+        (cat << 'EOF'
+Section: misc
+Priority: optional
+Standards-Version: 3.9.2
+
+Package: NAME
+Version: VERSION
+Maintainer: qemu-devel@nongnu.org
+Architecture: any
+Multi-Arch: same
+Description: fake NAME
+EOF
+        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
+        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
+        dpkg -i $2_$3_$1.deb
+    )
+}
+
+try_install()
+{
+    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
+    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
+    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
+    apt-get install -q -yy $1 || fake_install $arch $name $vers
+}
+
+for package in $*; do
+    try_install $package
+done
-- 
2.11.0

[Qemu-devel] [PATCH v5 05/24] docker: install ca-certificates package in base image

[Philippe Mathieu-Daudé]

Resolve SSL verification issue at shippable container's git_sync stage:

shippable logs:
--------------
git_sync
- ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
Cloning into '/root/src/github.com/philmd/qemu'...
fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
retrying 1 of 3 times...

Suggested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 52bd79938e..d08def6a8d 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -11,7 +11,7 @@ FROM debian:stable-slim
 
 # Setup some basic tools we need
 RUN apt update
-RUN apt install -yy curl aptitude
+RUN apt install -yy aptitude ca-certificates curl
 
 # Setup Emdebian
 RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-- 
2.11.0

[Qemu-devel] [PATCH v5 06/24] docker: create deb-src entry and setup Emdebian in the same layer

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian.docker | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index d08def6a8d..33ba686892 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -13,13 +13,11 @@ FROM debian:stable-slim
 RUN apt update
 RUN apt install -yy aptitude ca-certificates curl
 
-# Setup Emdebian
-RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
-
 # Install common build utilities
 RUN apt update
 RUN apt install -yy build-essential clang
+
+# Duplicate deb line as deb-src, setup Emdebian
+RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
+    echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
+    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-- 
2.11.0

[Qemu-devel] [PATCH v5 07/24] docker: use better regex to generate deb-src entries

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 33ba686892..e5a10a5875 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -18,6 +18,6 @@ RUN apt update
 RUN apt install -yy build-essential clang
 
 # Duplicate deb line as deb-src, setup Emdebian
-RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
+RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list && \
     echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
     curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-- 
2.11.0

[Qemu-devel] [PATCH v5 08/24] docker: use eatmydata, install common build packages in base image

[Philippe Mathieu-Daudé]

The common build packages are: build-essential clang git bison flex

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian.docker | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index e5a10a5875..c62c87f205 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -11,7 +11,11 @@ FROM debian:stable-slim
 
 # Setup some basic tools we need
 RUN apt update
-RUN apt install -yy aptitude ca-certificates curl
+RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
+RUN DEBIAN_FRONTEND=noninteractive \
+    eatmydata apt install -y --no-install-recommends \
+        aptitude ca-certificates curl \
+        build-essential clang git bison flex
 
 # Install common build utilities
 RUN apt update
-- 
2.11.0

[Qemu-devel] [PATCH v5 09/24] docker: use eatmydata in debian armhf image

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-armhf-cross.docker | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 668d60aeb3..ba8d5a5167 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -8,8 +8,11 @@ FROM qemu:debian
 # Add the foreign architecture we want and install dependencies
 RUN dpkg --add-architecture armhf
 RUN apt update
-RUN apt install -yy crossbuild-essential-armhf
-RUN apt-get build-dep -yy -a armhf qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-armhf
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a armhf qemu
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
-- 
2.11.0

[Qemu-devel] [PATCH v5 10/24] docker: use eatmydata in debian arm64 image

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-arm64-cross.docker | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 592b5d7055..f3db7f4f55 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -8,8 +8,11 @@ FROM qemu:debian
 # Add the foreign architecture we want and install dependencies
 RUN dpkg --add-architecture arm64
 RUN apt update
-RUN apt install -yy crossbuild-essential-arm64
-RUN apt-get build-dep -yy -a arm64 qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-arm64
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a arm64 qemu
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
-- 
2.11.0

[Qemu-devel] [PATCH v5 11/24] docker: add extra libs to armhf target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-armhf-cross.docker | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index ba8d5a5167..e67dfdccc5 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -16,3 +16,14 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:armhf \
+        libbz2-dev:armhf \
+        liblzo2-dev:armhf \
+        libncursesw5-dev:armhf \
+        libnfs-dev:armhf \
+        librdmacm-dev:armhf \
+        libsnappy-dev:armhf \
+        libxen-dev:armhf
-- 
2.11.0

[Qemu-devel] [PATCH v5 12/24] docker: add extra libs to arm64 target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-arm64-cross.docker | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index f3db7f4f55..45b891d57a 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -16,3 +16,14 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:arm64 \
+        libbz2-dev:arm64 \
+        liblzo2-dev:arm64 \
+        libncursesw5-dev:arm64 \
+        libnfs-dev:arm64 \
+        librdmacm-dev:arm64 \
+        libsnappy-dev:arm64 \
+        libxen-dev:arm64
-- 
2.11.0

[Qemu-devel] [PATCH v5 13/24] docker: add extra libs to s390x target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-s390x-cross.docker | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 3a687feda0..cfc354ce5d 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -20,3 +20,13 @@ RUN apt install -yy gcc-multilib-s390x-linux-gnu binutils-multiarch
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=s390x-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:s390x \
+        libbz2-dev:s390x \
+        liblzo2-dev:s390x \
+        libncursesw5-dev:s390x \
+        libnfs-dev:s390x \
+        librdmacm-dev:s390x \
+        libsnappy-dev:s390x
-- 
2.11.0

[Qemu-devel] [PATCH v5 14/24] docker: add mipsel build target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include                      |  3 ++
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 41 ++++++++++++++++++++++
 2 files changed, 44 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index fe1a9a53ff..1c2ab29303 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -55,9 +55,12 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
 		"BUILD","$*")
 
+docker-image-debian-mipsel-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
+
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
+docker-image-debian-mipsel-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
new file mode 100644
index 0000000000..bda9fac9d9
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -0,0 +1,41 @@
+#
+# Docker mipsel cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture mipsel
+RUN apt-get update
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-mipsel
+
+# <kludge> to fix "following packages have unmet dependencies" ...
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+RUN apt-get install -y --no-install-recommends \
+        equivs \
+        pkg-config
+RUN apt-fake install \
+        pkgconf:mipsel=0.9.7-fake \
+        pkg-config:mipsel=0.28-1.1-fake && \
+    ln -s pkg-config /usr/bin/mipsel-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/mipsel-linux-gnu/pkgconfig
+# </kludge>
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a mipsel qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:mipsel \
+        libbz2-dev:mipsel \
+        liblzo2-dev:mipsel \
+        libncursesw5-dev:mipsel \
+        libnfs-dev:mipsel \
+        librdmacm-dev:mipsel \
+        libsnappy-dev:mipsel
-- 
2.11.0

[Qemu-devel] [PATCH v5 15/24] docker: add powerpc build target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include                      |  2 ++
 .../docker/dockerfiles/debian-powerpc-cross.docker | 40 ++++++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 1c2ab29303..652deda21b 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -56,11 +56,13 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		"BUILD","$*")
 
 docker-image-debian-mipsel-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
+docker-image-debian-powerpc-cross: EXTRA_FILES:=tests/docker/dockerfiles/debian-apt-fake.sh
 
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
 docker-image-debian-mipsel-cross: docker-image-debian
+docker-image-debian-powerpc-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
new file mode 100644
index 0000000000..fa2cc7a657
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -0,0 +1,40 @@
+#
+# Docker powerpc cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture powerpc
+RUN apt-get update
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-powerpc
+
+# <kludge> to fix "following packages have unmet dependencies" ...
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+RUN apt-get install -y --no-install-recommends \
+        equivs \
+        pkg-config
+RUN apt-fake install \
+        pkg-config:powerpc=0.28-1.1-fake && \
+    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
+# </kludge>
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a powerpc qemu
+RUN DEBIAN_FRONTEND=noninteractive \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:powerpc \
+        libbz2-dev:powerpc \
+        liblzo2-dev:powerpc \
+        libncursesw5-dev:powerpc \
+        libnfs-dev:powerpc \
+        librdmacm-dev:powerpc \
+        libsnappy-dev:powerpc
-- 
2.11.0

[Qemu-devel] [PATCH v5 16/24] shippable: use C locale to simplify console output

[Philippe Mathieu-Daudé]

remove this noise:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = "en_US.UTF-8",
    LC_CTYPE = "en_US.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 653bd750fe..231c29b620 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,5 +1,7 @@
 language: c
 env:
+  global:
+    - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
       TARGET_LIST=arm-softmmu,arm-linux-user
-- 
2.11.0

[Qemu-devel] [PATCH v5 17/24] shippable: build using all available cpus

[Philippe Mathieu-Daudé]

As of this commit:

$ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf _NPROCESSORS_CONF`
container proc: 2 2

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index 231c29b620..1e3ae35dd9 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -20,4 +20,4 @@ build:
   ci:
     - unset CC
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
-    - make -j2
+    - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
-- 
2.11.0

[Qemu-devel] [PATCH v5 18/24] shippable: do not initialize submodules automatically

[Philippe Mathieu-Daudé]

instead do it in the 'ci' target when needed.

for mips64el-softmmu target:
use dtc submodule if distrib packages are too old.

example with outdated libfdt on mips64el-softmmu target (required is >= 1.4.2):
 # dpkg-query --showformat='${Version}\n' --show libfdt-dev
 1.4.0+dfsg-1

shippable output:
----------------
  LINK    mips64el-softmmu/qemu-system-mips64el
../hw/core/loader-fit.o: In function `load_fit':
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
collect2: error: ld returned 1 exit status
Makefile:201: recipe for target 'qemu-system-mips64el' failed
make[1]: *** [qemu-system-mips64el] Error 1
Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
make: *** [subdir-mips64el-softmmu] Error 2

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 1e3ae35dd9..46adfa030f 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,4 +1,6 @@
 language: c
+git:
+   submodules: false
 env:
   global:
     - LC_ALL=C
@@ -19,5 +21,13 @@ build:
     options: "-e HOME=/root"
   ci:
     - unset CC
+    # some targets require newer up to date packages, for example TARGET_LIST matching
+    # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
+    # see the configure script:
+    #    error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
+    #    "  (1) Preferred: Install the DTC (libfdt) devel package"
+    #    "  (2) Fetch the DTC submodule, using:"
+    #    "      git submodule update --init dtc"
+    - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
     - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
-- 
2.11.0

[Qemu-devel] [PATCH v5 19/24] shippable: be verbose while building docker images

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index 46adfa030f..fe360f85cb 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,7 +13,7 @@ env:
       TARGET_LIST=s390x-softmmu,s390x-linux-user
 build:
   pre_ci:
-    - make docker-image-${IMAGE}
+    - make docker-image-${IMAGE} V=1
   pre_ci_boot:
     image_name: qemu
     image_tag: ${IMAGE}
-- 
2.11.0

[Qemu-devel] [PATCH v5 20/24] shippable: add armeb-linux-user target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index fe360f85cb..2070c4d827 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -6,7 +6,7 @@ env:
     - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
-      TARGET_LIST=arm-softmmu,arm-linux-user
+      TARGET_LIST=arm-softmmu,arm-linux-user,armeb-linux-user
     - IMAGE=debian-arm64-cross
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
-- 
2.11.0

[Qemu-devel] [PATCH v5 21/24] shippable: add powerpc target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 2070c4d827..aad66ec5ec 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -11,6 +11,8 @@ env:
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
       TARGET_LIST=s390x-softmmu,s390x-linux-user
+    - IMAGE=debian-powerpc-cross
+      TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE} V=1
-- 
2.11.0

[Qemu-devel] [PATCH v5 22/24] shippable: add mipsel target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index aad66ec5ec..75c2895a21 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,6 +13,9 @@ env:
       TARGET_LIST=s390x-softmmu,s390x-linux-user
     - IMAGE=debian-powerpc-cross
       TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
+    # mips64el-softmmu disabled due to libfdt problem
+    - IMAGE=debian-mipsel-cross
+      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE} V=1
-- 
2.11.0

[Qemu-devel] [PATCH v5 23/24] MAINTAINERS: add Shippable automation platform URL

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 7df088259b..d578504294 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1865,6 +1865,7 @@ F: .travis.yml
 F: .shippable.yml
 F: tests/docker/
 W: https://travis-ci.org/qemu/qemu
+W: https://app.shippable.com/github/qemu/qemu
 W: http://patchew.org/QEMU/
 
 Documentation
-- 
2.11.0

[Qemu-devel] [PATCH v5 24/24] MAINTAINERS: self-appoint me as reviewer in build/test automation

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index d578504294..916c0e96cc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1859,6 +1859,7 @@ Build and test automation
 -------------------------
 M: Alex Bennée <alex.bennee@linaro.org>
 M: Fam Zheng <famz@redhat.com>
+R: Philippe Mathieu-Daudé <f4bug@amsat.org>
 L: qemu-devel@nongnu.org
 S: Maintained
 F: .travis.yml
-- 
2.11.0

Re: [Qemu-devel] [PATCH v5 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> This patchset add 2 more architectures to the cross-build farm.
>
> There is still some issue trying to cross-build mips64el-softmmu, it seems the
> cross tools use the system outdated libfdt instead of the one checkouted in the
> dtc submodule. I disabled this target for now.
>
> The branch https://github.com/philmd/qemu/tree/docker_shippable_v5 can be
> checked at Shippable:
>   https://app.shippable.com/github/philmd/qemu/status/dashboard
>
> Each arch builds in around ~9min. Using Shippable free open source projects
> service, the 5 jobs take ~38-44min in total.

Fam,

Are you picking this up or shall I roll a pull-req?

>
> v5:
> - Addressed minor review feedback from Alex/Fam (improving a comment)
> - Added Alex R-b
>
> v4:
> - Addressed review feedbacks from Fam (improving english, cleaner hashing)
> - Added Alex R-b
>
> v3:
> - Addressed review feedbacks from Fam:
> - Keep building images in various layers, but use DEBIAN_FRONTEND=noninteractive
> - Document '--extra-files', now it supports adding various files at once
> - Checksum extra files to trigger a docker image rebuild if modified
> - Use better regex to generate deb-src entries
> - Reordered extra libs, to ease further add/remove diffs
>
> v2:
> - Addressed review feedback from Alex, added his R-B
> - Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
>   installing the ca-certificates package.
> - Squashed/split some commits
> - use 'apt-get clean' instead of brutal 'rm -rf'
> - checked how many cores are available on Shippable and use them fully
>   (reduced total time from 44min to 38min)
> - build armeb-linux-user target
>
> v1:
> - Reorganize Dockerfiles to use less layers, resulting in smaller images.
>   This also reduce time of image transfer, for example while using:
>   `docker save qemu:debian | ssh remote docker load`
> - Install more debian packages so the configure script enable more features and
>   more code can be compiled/covered.
> - There are still some incorrect multiarch packages on debian/jessie used in
>   the docker images, add a script to generate fake packages and avoid
>   dependencies issues.
> - Modify the docker.py script to include an extra file while building images.
>
> Regards,
>
> Phil.
>
> Philippe Mathieu-Daudé (24):
>   docker: let _copy_with_mkdir() sub_path argument be optional
>   docker: add --include-files argument to 'build' command
>   docker: rebuild image if 'extra files' checksum does not match
>   docker: add 'apt-fake' script which generate fake debian packages
>   docker: install ca-certificates package in base image
>   docker: create deb-src entry and setup Emdebian in the same layer
>   docker: use better regex to generate deb-src entries
>   docker: use eatmydata, install common build packages in base image
>   docker: use eatmydata in debian armhf image
>   docker: use eatmydata in debian arm64 image
>   docker: add extra libs to armhf target to extend codebase coverage
>   docker: add extra libs to arm64 target to extend codebase coverage
>   docker: add extra libs to s390x target to extend codebase coverage
>   docker: add mipsel build target
>   docker: add powerpc build target
>   shippable: use C locale to simplify console output
>   shippable: build using all available cpus
>   shippable: do not initialize submodules automatically
>   shippable: be verbose while building docker images
>   shippable: add armeb-linux-user target
>   shippable: add powerpc target
>   shippable: add mipsel target
>   MAINTAINERS: add Shippable automation platform URL
>   MAINTAINERS: self-appoint me as reviewer in build/test automation
>
>  .shippable.yml                                     | 23 +++++++++--
>  MAINTAINERS                                        |  2 +
>  tests/docker/Makefile.include                      |  8 ++++
>  tests/docker/docker.py                             | 30 ++++++++++----
>  tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
>  tests/docker/dockerfiles/debian-arm64-cross.docker | 18 ++++++++-
>  tests/docker/dockerfiles/debian-armhf-cross.docker | 18 ++++++++-
>  .../docker/dockerfiles/debian-mipsel-cross.docker  | 41 +++++++++++++++++++
>  .../docker/dockerfiles/debian-powerpc-cross.docker | 40 +++++++++++++++++++
>  tests/docker/dockerfiles/debian-s390x-cross.docker | 10 +++++
>  tests/docker/dockerfiles/debian.docker             | 18 +++++----
>  11 files changed, 232 insertions(+), 22 deletions(-)
>  create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>  create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
>  create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker


--
Alex Bennée

Re: [Qemu-devel] [PATCH v5 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Fam Zheng]

On Tue, 06/20 16:54, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
> > This patchset add 2 more architectures to the cross-build farm.
> >
> > There is still some issue trying to cross-build mips64el-softmmu, it seems the
> > cross tools use the system outdated libfdt instead of the one checkouted in the
> > dtc submodule. I disabled this target for now.
> >
> > The branch https://github.com/philmd/qemu/tree/docker_shippable_v5 can be
> > checked at Shippable:
> >   https://app.shippable.com/github/philmd/qemu/status/dashboard
> >
> > Each arch builds in around ~9min. Using Shippable free open source projects
> > service, the 5 jobs take ~38-44min in total.
> 
> Fam,
> 
> Are you picking this up or shall I roll a pull-req?

Please do, thanks! I am occupied by something else for at least the rest of this
week (and my apologies for not responding to this series for days).

Fam

Re: [Qemu-devel] [PATCH v5 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> This patchset add 2 more architectures to the cross-build farm.
>
> There is still some issue trying to cross-build mips64el-softmmu, it seems the
> cross tools use the system outdated libfdt instead of the one checkouted in the
> dtc submodule. I disabled this target for now.

While I was rolling the pullreq I discovered the latest Debian release
Stretch had broken a bunch of things. Fortunately most things where
easily fixed.

> Philippe Mathieu-Daudé (24):
>   docker: let _copy_with_mkdir() sub_path argument be optional
>   docker: add --include-files argument to 'build' command
>   docker: rebuild image if 'extra files' checksum does not match
>   docker: add 'apt-fake' script which generate fake debian packages

dropped for now, no longer needed in mipsel

>   docker: install ca-certificates package in base image
>   docker: create deb-src entry and setup Emdebian in the same layer
>   docker: use better regex to generate deb-src entries
>   docker: use eatmydata, install common build packages in base image
>   docker: use eatmydata in debian armhf image
>   docker: use eatmydata in debian arm64 image
>   docker: add extra libs to armhf target to extend codebase coverage
>   docker: add extra libs to arm64 target to extend codebase coverage
>   docker: add extra libs to s390x target to extend codebase coverage
>   docker: add mipsel build target
>   docker: add powerpc build target

dropped as powerpc was dropped in stretch. This will need re-rolling
with its own jessie based dockerfile.

>   shippable: use C locale to simplify console output
>   shippable: build using all available cpus
>   shippable: do not initialize submodules automatically
>   shippable: be verbose while building docker images
>   shippable: add armeb-linux-user target
>   shippable: add powerpc target

dropped as above

>   shippable: add mipsel target
>   MAINTAINERS: add Shippable automation platform URL
>   MAINTAINERS: self-appoint me as reviewer in build/test automation
>
>  .shippable.yml                                     | 23 +++++++++--
>  MAINTAINERS                                        |  2 +
>  tests/docker/Makefile.include                      |  8 ++++
>  tests/docker/docker.py                             | 30 ++++++++++----
>  tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
>  tests/docker/dockerfiles/debian-arm64-cross.docker | 18 ++++++++-
>  tests/docker/dockerfiles/debian-armhf-cross.docker | 18 ++++++++-
>  .../docker/dockerfiles/debian-mipsel-cross.docker  | 41 +++++++++++++++++++
>  .../docker/dockerfiles/debian-powerpc-cross.docker | 40 +++++++++++++++++++
>  tests/docker/dockerfiles/debian-s390x-cross.docker | 10 +++++
>  tests/docker/dockerfiles/debian.docker             | 18 +++++----
>  11 files changed, 232 insertions(+), 22 deletions(-)
>  create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>  create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
>  create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

I'll send the pull req once I have a clean run on my branch.

--
Alex Bennée

Re: [Qemu-devel] [PATCH v5 00/24] docker/shippable: cross-build mipsel and powerpc targets

[Philippe Mathieu-Daudé]

On 06/21/2017 11:15 AM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> This patchset add 2 more architectures to the cross-build farm.
>>
>> There is still some issue trying to cross-build mips64el-softmmu, it seems the
>> cross tools use the system outdated libfdt instead of the one checkouted in the
>> dtc submodule. I disabled this target for now.
> 
> While I was rolling the pullreq I discovered the latest Debian release
> Stretch had broken a bunch of things. Fortunately most things where
> easily fixed.

I didn't notice Debian Stretch release day. Thank you to fix this series 
so quickly!

> 
>> Philippe Mathieu-Daudé (24):
>>    docker: let _copy_with_mkdir() sub_path argument be optional
>>    docker: add --include-files argument to 'build' command
>>    docker: rebuild image if 'extra files' checksum does not match
>>    docker: add 'apt-fake' script which generate fake debian packages
> 
> dropped for now, no longer needed in mipsel
> 
>>    docker: install ca-certificates package in base image
>>    docker: create deb-src entry and setup Emdebian in the same layer
>>    docker: use better regex to generate deb-src entries
>>    docker: use eatmydata, install common build packages in base image
>>    docker: use eatmydata in debian armhf image
>>    docker: use eatmydata in debian arm64 image
>>    docker: add extra libs to armhf target to extend codebase coverage
>>    docker: add extra libs to arm64 target to extend codebase coverage
>>    docker: add extra libs to s390x target to extend codebase coverage
>>    docker: add mipsel build target
>>    docker: add powerpc build target
> 
> dropped as powerpc was dropped in stretch. This will need re-rolling
> with its own jessie based dockerfile.

I think it is safer our images follow a specific release tag instead of 
'stable', this way we don't miss an upstream release until all our 
builds fail.

I'll purpose that in my next Docker series, changing current debian 
images to use debian:stretch base, then restore powerpc using debian:jessie.

>>    shippable: use C locale to simplify console output
>>    shippable: build using all available cpus
>>    shippable: do not initialize submodules automatically
>>    shippable: be verbose while building docker images
>>    shippable: add armeb-linux-user target
>>    shippable: add powerpc target
> 
> dropped as above
> 
>>    shippable: add mipsel target
>>    MAINTAINERS: add Shippable automation platform URL
>>    MAINTAINERS: self-appoint me as reviewer in build/test automation
>>
>>   .shippable.yml                                     | 23 +++++++++--
>>   MAINTAINERS                                        |  2 +
>>   tests/docker/Makefile.include                      |  8 ++++
>>   tests/docker/docker.py                             | 30 ++++++++++----
>>   tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
>>   tests/docker/dockerfiles/debian-arm64-cross.docker | 18 ++++++++-
>>   tests/docker/dockerfiles/debian-armhf-cross.docker | 18 ++++++++-
>>   .../docker/dockerfiles/debian-mipsel-cross.docker  | 41 +++++++++++++++++++
>>   .../docker/dockerfiles/debian-powerpc-cross.docker | 40 +++++++++++++++++++
>>   tests/docker/dockerfiles/debian-s390x-cross.docker | 10 +++++
>>   tests/docker/dockerfiles/debian.docker             | 18 +++++----
>>   11 files changed, 232 insertions(+), 22 deletions(-)
>>   create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>>   create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
>>   create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker
> 
> I'll send the pull req once I have a clean run on my branch.
> 
> --
> Alex Bennée
>