[Qemu-devel] [PULL 00/21] Docker and shippable updates

[Qemu-devel] [PULL 00/21] Docker and shippable updates

[Alex Bennée]

The following changes since commit 8dfaf23ae1f2273a9730a9b309cc8471269bb524:

  tcg/tci: fix tcg-interpreter build (2017-06-20 18:39:15 +0100)

are available in the git repository at:

  https://github.com/stsquad/qemu.git tags/pull-ci-updates-210617-2

for you to fetch changes up to 32b9ca986855a5d56daf47fdb516743008788b71:

  MAINTAINERS: self-appoint me as reviewer in build/test automation (2017-06-21 15:03:06 +0100)

----------------------------------------------------------------
This is mostly Philippe's updates

We add the following cross-compile targets:
  - mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
  - armeb-linux-user

While I was rolling I discovered we could also back out a bunch of the
emdebian hacks as the newly released stretch handles cross compilers
as first class citizens. Unfortunately this also meant I had to drop
the powerpc support as that is no longer in Debian stable.

----------------------------------------------------------------
Alex Bennée (1):
      docker: update qemu:debian base following stretch release

Philippe Mathieu-Daudé (20):
      docker: let _copy_with_mkdir() sub_path argument be optional
      docker: add --include-files argument to 'build' command
      docker: rebuild image if 'extra files' checksum does not match
      docker: install ca-certificates package in base image
      docker: use better regex to generate deb-src entries
      docker: use eatmydata, install common build packages in base image
      docker: use eatmydata in debian armhf image
      docker: use eatmydata in debian arm64 image
      docker: add extra libs to armhf target to extend codebase coverage
      docker: add extra libs to arm64 target to extend codebase coverage
      docker: add extra libs to s390x target to extend codebase coverage
      docker: add mipsel build target
      shippable: use C locale to simplify console output
      shippable: build using all available cpus
      shippable: do not initialize submodules automatically
      shippable: be verbose while building docker images
      shippable: add armeb-linux-user target
      shippable: add mipsel target
      MAINTAINERS: add Shippable automation platform URL
      MAINTAINERS: self-appoint me as reviewer in build/test automation

 .shippable.yml                                     | 21 ++++++++++++---
 MAINTAINERS                                        |  2 ++
 tests/docker/Makefile.include                      |  4 +++
 tests/docker/docker.py                             | 30 +++++++++++++++++-----
 tests/docker/dockerfiles/debian-arm64-cross.docker | 18 +++++++++++--
 tests/docker/dockerfiles/debian-armhf-cross.docker | 18 +++++++++++--
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 29 +++++++++++++++++++++
 tests/docker/dockerfiles/debian-s390x-cross.docker | 10 ++++++++
 tests/docker/dockerfiles/debian.docker             | 15 ++++-------
 9 files changed, 123 insertions(+), 24 deletions(-)
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker

[Qemu-devel] [PULL 01/21] docker: update qemu:debian base following stretch release

[Alex Bennée]

Debian has now released Stretch as its new stable. As we track
debian:stable-slim this has a few consequences. For one thing we can
now drop the emdebian hacks as cross compilers are part of the
official repositories now. However we do loose the ability to build
against powerpc (not ppc64) since that is no longer a release
architecture.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 52bd79938e..4ca6ecaad0 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -9,14 +9,6 @@
 #
 FROM debian:stable-slim
 
-# Setup some basic tools we need
-RUN apt update
-RUN apt install -yy curl aptitude
-
-# Setup Emdebian
-RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-
 # Duplicate deb line as deb-src
 RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
 
-- 
2.13.0

[Qemu-devel] [PULL 02/21] docker: let _copy_with_mkdir() sub_path argument be optional

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 8747f6a440..6ddc6e4c2a 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -52,7 +52,7 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
-def _copy_with_mkdir(src, root_dir, sub_path):
+def _copy_with_mkdir(src, root_dir, sub_path='.'):
     """Copy src into root_dir, creating sub_path as needed."""
     dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
     try:
-- 
2.13.0

[Qemu-devel] [PULL 03/21] docker: add --include-files argument to 'build' command

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 0ed8c3d323..61cc0d14e2 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -51,6 +51,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
 		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
 		$(if $(NOUSER),,--add-current-user) \
+		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
 		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
 		"BUILD","$*")
 
@@ -107,6 +108,8 @@ docker:
 	@echo '    NOUSER               Define to disable adding current user to containers passwd.'
 	@echo '    NOCACHE=1            Ignore cache when build images.'
 	@echo '    EXECUTABLE=<path>    Include executable in image.'
+	@echo '    EXTRA_FILES="<path> [... <path>]"'
+	@echo '                         Include extra files in image.'
 
 # This rule if for directly running against an arbitrary docker target.
 # It is called by the expanded docker targets (e.g. make
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 6ddc6e4c2a..10fa907905 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -237,6 +237,10 @@ class BuildCommand(SubCommand):
                             help="""Specify a binary that will be copied to the
                             container together with all its dependent
                             libraries""")
+        parser.add_argument("--extra-files", "-f", nargs='*',
+                            help="""Specify files that will be copied in the
+                            Docker image, fulfilling the ADD directive from the
+                            Dockerfile""")
         parser.add_argument("--add-current-user", "-u", dest="user",
                             action="store_true",
                             help="Add the current user to image's passwd")
@@ -270,10 +274,12 @@ class BuildCommand(SubCommand):
                     print "%s exited with code %d" % (docker_pre, rc)
                     return 1
 
-            # Do we include a extra binary?
+            # Copy any extra files into the Docker context. These can be
+            # included by the use of the ADD directive in the Dockerfile.
             if args.include_executable:
-                _copy_binary_with_libs(args.include_executable,
-                                       docker_dir)
+                _copy_binary_with_libs(args.include_executable, docker_dir)
+            for filename in args.extra_files or []:
+                _copy_with_mkdir(filename, docker_dir)
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
-- 
2.13.0

[Qemu-devel] [PULL 04/21] docker: rebuild image if 'extra files' checksum does not match

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 10fa907905..e707e5bcca 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -38,6 +38,9 @@ def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
     return hashlib.sha1(text).hexdigest()
 
+def _file_checksum(filename):
+    return _text_checksum(open(filename, 'rb').read())
+
 def _guess_docker_command():
     """ Guess a working docker command or raise exception if not found"""
     commands = [["docker"], ["sudo", "-n", "docker"]]
@@ -154,7 +157,7 @@ class Docker(object):
         return labels.get("com.qemu.dockerfile-checksum", "")
 
     def build_image(self, tag, docker_dir, dockerfile,
-                    quiet=True, user=False, argv=None):
+                    quiet=True, user=False, argv=None, extra_files_cksum=[]):
         if argv == None:
             argv = []
 
@@ -170,7 +173,8 @@ class Docker(object):
 
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
-                     _text_checksum(dockerfile))
+                     _text_checksum("\n".join([dockerfile] +
+                                    extra_files_cksum)))
         tmp_df.flush()
 
         self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
@@ -276,16 +280,22 @@ class BuildCommand(SubCommand):
 
             # Copy any extra files into the Docker context. These can be
             # included by the use of the ADD directive in the Dockerfile.
+            cksum = []
             if args.include_executable:
+                # FIXME: there is no checksum of this executable and the linked
+                # libraries, once the image built any change of this executable
+                # or any library won't trigger another build.
                 _copy_binary_with_libs(args.include_executable, docker_dir)
             for filename in args.extra_files or []:
                 _copy_with_mkdir(filename, docker_dir)
+                cksum += [_file_checksum(filename)]
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
                         if k.lower() in FILTERED_ENV_NAMES]
             dkr.build_image(tag, docker_dir, dockerfile,
-                            quiet=args.quiet, user=args.user, argv=argv)
+                            quiet=args.quiet, user=args.user, argv=argv,
+                            extra_files_cksum=cksum)
 
             rmtree(docker_dir)
 
-- 
2.13.0

[Qemu-devel] [PULL 05/21] docker: install ca-certificates package in base image

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Resolve SSL verification issue at shippable container's git_sync stage:

shippable logs:
--------------
git_sync
- ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
Cloning into '/root/src/github.com/philmd/qemu'...
fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
retrying 1 of 3 times...

Suggested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: fixed re-base conflict following stretch updates]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 4ca6ecaad0..e44ce2f1c0 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -14,4 +14,4 @@ RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
 
 # Install common build utilities
 RUN apt update
-RUN apt install -yy build-essential clang
+RUN apt install -yy build-essential clang ca-certificates
-- 
2.13.0

[Qemu-devel] [PULL 06/21] docker: use better regex to generate deb-src entries

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: fixed up following dropping emdebian]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index e44ce2f1c0..b0e1691beb 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -10,7 +10,7 @@
 FROM debian:stable-slim
 
 # Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
+RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
 
 # Install common build utilities
 RUN apt update
-- 
2.13.0

[Qemu-devel] [PULL 07/21] docker: use eatmydata, install common build packages in base image

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

The common build packages are: build-essential clang git bison flex

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: fixups following stretch update]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index b0e1691beb..10953b2425 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -14,4 +14,7 @@ RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.lis
 
 # Install common build utilities
 RUN apt update
-RUN apt install -yy build-essential clang ca-certificates
+RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt install -y --no-install-recommends \
+    ca-certificates build-essential clang git bison flex
-- 
2.13.0

[Qemu-devel] [PULL 08/21] docker: use eatmydata in debian armhf image

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 668d60aeb3..ba8d5a5167 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -8,8 +8,11 @@ FROM qemu:debian
 # Add the foreign architecture we want and install dependencies
 RUN dpkg --add-architecture armhf
 RUN apt update
-RUN apt install -yy crossbuild-essential-armhf
-RUN apt-get build-dep -yy -a armhf qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-armhf
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a armhf qemu
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
-- 
2.13.0

[Qemu-devel] [PULL 09/21] docker: use eatmydata in debian arm64 image

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 592b5d7055..f3db7f4f55 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -8,8 +8,11 @@ FROM qemu:debian
 # Add the foreign architecture we want and install dependencies
 RUN dpkg --add-architecture arm64
 RUN apt update
-RUN apt install -yy crossbuild-essential-arm64
-RUN apt-get build-dep -yy -a arm64 qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-arm64
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a arm64 qemu
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
-- 
2.13.0

[Qemu-devel] [PULL 10/21] docker: add extra libs to armhf target to extend codebase coverage

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index ba8d5a5167..e67dfdccc5 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -16,3 +16,14 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:armhf \
+        libbz2-dev:armhf \
+        liblzo2-dev:armhf \
+        libncursesw5-dev:armhf \
+        libnfs-dev:armhf \
+        librdmacm-dev:armhf \
+        libsnappy-dev:armhf \
+        libxen-dev:armhf
-- 
2.13.0

[Qemu-devel] [PULL 11/21] docker: add extra libs to arm64 target to extend codebase coverage

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index f3db7f4f55..45b891d57a 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -16,3 +16,14 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:arm64 \
+        libbz2-dev:arm64 \
+        liblzo2-dev:arm64 \
+        libncursesw5-dev:arm64 \
+        libnfs-dev:arm64 \
+        librdmacm-dev:arm64 \
+        libsnappy-dev:arm64 \
+        libxen-dev:arm64
-- 
2.13.0

[Qemu-devel] [PULL 12/21] docker: add extra libs to s390x target to extend codebase coverage

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 3a687feda0..cfc354ce5d 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -20,3 +20,13 @@ RUN apt install -yy gcc-multilib-s390x-linux-gnu binutils-multiarch
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=s390x-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:s390x \
+        libbz2-dev:s390x \
+        liblzo2-dev:s390x \
+        libncursesw5-dev:s390x \
+        libnfs-dev:s390x \
+        librdmacm-dev:s390x \
+        libsnappy-dev:s390x
-- 
2.13.0

[Qemu-devel] [PULL 13/21] docker: add mipsel build target

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: remove apt-fake kludge]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 61cc0d14e2..037cb9e9e7 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -58,6 +58,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
+docker-image-debian-mipsel-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
new file mode 100644
index 0000000000..2156bdb28d
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -0,0 +1,29 @@
+#
+# Docker mipsel cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture mipsel
+RUN apt-get update
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-mipsel
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a mipsel qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:mipsel \
+        libbz2-dev:mipsel \
+        liblzo2-dev:mipsel \
+        libncursesw5-dev:mipsel \
+        libnfs-dev:mipsel \
+        librdmacm-dev:mipsel \
+        libsnappy-dev:mipsel
-- 
2.13.0

[Qemu-devel] [PULL 14/21] shippable: use C locale to simplify console output

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

remove this noise:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = "en_US.UTF-8",
    LC_CTYPE = "en_US.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.shippable.yml b/.shippable.yml
index 653bd750fe..231c29b620 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,5 +1,7 @@
 language: c
 env:
+  global:
+    - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
       TARGET_LIST=arm-softmmu,arm-linux-user
-- 
2.13.0

[Qemu-devel] [PULL 15/21] shippable: build using all available cpus

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

As of this commit:

$ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf _NPROCESSORS_CONF`
container proc: 2 2

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.shippable.yml b/.shippable.yml
index 231c29b620..1e3ae35dd9 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -20,4 +20,4 @@ build:
   ci:
     - unset CC
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
-    - make -j2
+    - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
-- 
2.13.0

[Qemu-devel] [PULL 16/21] shippable: do not initialize submodules automatically

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

instead do it in the 'ci' target when needed.

for mips64el-softmmu target:
use dtc submodule if distrib packages are too old.

example with outdated libfdt on mips64el-softmmu target (required is >= 1.4.2):
 # dpkg-query --showformat='${Version}\n' --show libfdt-dev
 1.4.0+dfsg-1

shippable output:
----------------
  LINK    mips64el-softmmu/qemu-system-mips64el
../hw/core/loader-fit.o: In function `load_fit':
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
collect2: error: ld returned 1 exit status
Makefile:201: recipe for target 'qemu-system-mips64el' failed
make[1]: *** [qemu-system-mips64el] Error 1
Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
make: *** [subdir-mips64el-softmmu] Error 2

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.shippable.yml b/.shippable.yml
index 1e3ae35dd9..46adfa030f 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,4 +1,6 @@
 language: c
+git:
+   submodules: false
 env:
   global:
     - LC_ALL=C
@@ -19,5 +21,13 @@ build:
     options: "-e HOME=/root"
   ci:
     - unset CC
+    # some targets require newer up to date packages, for example TARGET_LIST matching
+    # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
+    # see the configure script:
+    #    error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
+    #    "  (1) Preferred: Install the DTC (libfdt) devel package"
+    #    "  (2) Fetch the DTC submodule, using:"
+    #    "      git submodule update --init dtc"
+    - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
     - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
-- 
2.13.0

[Qemu-devel] [PULL 17/21] shippable: be verbose while building docker images

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.shippable.yml b/.shippable.yml
index 46adfa030f..fe360f85cb 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,7 +13,7 @@ env:
       TARGET_LIST=s390x-softmmu,s390x-linux-user
 build:
   pre_ci:
-    - make docker-image-${IMAGE}
+    - make docker-image-${IMAGE} V=1
   pre_ci_boot:
     image_name: qemu
     image_tag: ${IMAGE}
-- 
2.13.0

[Qemu-devel] [PULL 18/21] shippable: add armeb-linux-user target

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.shippable.yml b/.shippable.yml
index fe360f85cb..2070c4d827 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -6,7 +6,7 @@ env:
     - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
-      TARGET_LIST=arm-softmmu,arm-linux-user
+      TARGET_LIST=arm-softmmu,arm-linux-user,armeb-linux-user
     - IMAGE=debian-arm64-cross
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
-- 
2.13.0

[Qemu-devel] [PULL 19/21] shippable: add mipsel target

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[AJB: fixups after dropping powerpc]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/.shippable.yml b/.shippable.yml
index 2070c4d827..5e0caa65c5 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -11,6 +11,9 @@ env:
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
       TARGET_LIST=s390x-softmmu,s390x-linux-user
+    # mips64el-softmmu disabled due to libfdt problem
+    - IMAGE=debian-mipsel-cross
+      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE} V=1
-- 
2.13.0

[Qemu-devel] [PULL 20/21] MAINTAINERS: add Shippable automation platform URL

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/MAINTAINERS b/MAINTAINERS
index 120788d8fb..b2cf06265e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1864,6 +1864,7 @@ F: .travis.yml
 F: .shippable.yml
 F: tests/docker/
 W: https://travis-ci.org/qemu/qemu
+W: https://app.shippable.com/github/qemu/qemu
 W: http://patchew.org/QEMU/
 
 Documentation
-- 
2.13.0

[Qemu-devel] [PULL 21/21] MAINTAINERS: self-appoint me as reviewer in build/test automation

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/MAINTAINERS b/MAINTAINERS
index b2cf06265e..5d9255e03c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1858,6 +1858,7 @@ Build and test automation
 -------------------------
 M: Alex Bennée <alex.bennee@linaro.org>
 M: Fam Zheng <famz@redhat.com>
+R: Philippe Mathieu-Daudé <f4bug@amsat.org>
 L: qemu-devel@nongnu.org
 S: Maintained
 F: .travis.yml
-- 
2.13.0

Re: [Qemu-devel] [PULL 00/21] Docker and shippable updates

[Peter Maydell]

On 21 June 2017 at 15:47, Alex Bennée <alex.bennee@linaro.org> wrote:
> The following changes since commit 8dfaf23ae1f2273a9730a9b309cc8471269bb524:
>
>   tcg/tci: fix tcg-interpreter build (2017-06-20 18:39:15 +0100)
>
> are available in the git repository at:
>
>   https://github.com/stsquad/qemu.git tags/pull-ci-updates-210617-2
>
> for you to fetch changes up to 32b9ca986855a5d56daf47fdb516743008788b71:
>
>   MAINTAINERS: self-appoint me as reviewer in build/test automation (2017-06-21 15:03:06 +0100)
>
> ----------------------------------------------------------------
> This is mostly Philippe's updates
>
> We add the following cross-compile targets:
>   - mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
>   - armeb-linux-user
>
> While I was rolling I discovered we could also back out a bunch of the
> emdebian hacks as the newly released stretch handles cross compilers
> as first class citizens. Unfortunately this also meant I had to drop
> the powerpc support as that is no longer in Debian stable.
>

Applied, thanks.

-- PMM

Re: [Qemu-devel] [PULL 00/21] Docker and shippable updates

[Philippe Mathieu-Daudé]

On 06/22/2017 11:09 AM, Peter Maydell wrote:
> On 21 June 2017 at 15:47, Alex Bennée <alex.bennee@linaro.org> wrote:
[...]>> We add the following cross-compile targets:
>>    - mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
>>    - armeb-linux-user
>>
>> While I was rolling I discovered we could also back out a bunch of the
>> emdebian hacks as the newly released stretch handles cross compilers
>> as first class citizens. Unfortunately this also meant I had to drop
>> the powerpc support as that is no longer in Debian stable.
>>
> 
> Applied, thanks.

Yay! Shippable happily working again \o/

https://app.shippable.com/github/qemu/qemu/runs/238/summary/console

   Overall Status: Success

Re: [Qemu-devel] [PULL 00/21] Docker and shippable updates

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 06/22/2017 11:09 AM, Peter Maydell wrote:
>> On 21 June 2017 at 15:47, Alex Bennée <alex.bennee@linaro.org> wrote:
> [...]>> We add the following cross-compile targets:
>>>    - mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
>>>    - armeb-linux-user
>>>
>>> While I was rolling I discovered we could also back out a bunch of the
>>> emdebian hacks as the newly released stretch handles cross compilers
>>> as first class citizens. Unfortunately this also meant I had to drop
>>> the powerpc support as that is no longer in Debian stable.
>>>
>>
>> Applied, thanks.
>
> Yay! Shippable happily working again \o/
>
> https://app.shippable.com/github/qemu/qemu/runs/238/summary/console
>
>   Overall Status: Success

Now we've been running shippable for a while is it worth turning on the
IRC notifications?

--
Alex Bennée

Re: [Qemu-devel] [PULL 00/21] Docker and shippable updates

[Philippe Mathieu-Daudé]

On Thu, Jun 22, 2017 at 12:13 PM, Alex Bennée <alex.bennee@linaro.org> wrote:
> Now we've been running shippable for a while is it worth turning on the
> IRC notifications?

What about moving such bot/scripts notifications in another channel
like #QEMU-notifications?
It is often hard to follow 3 concurrent topics while the travis-ci bot
disrupts entering/broadcasting/leaving...
I'd also like to see here checkpatch/patchew stripped output, and
eventually gcov/gprof reports.
I don't think full patchew reports are useful in a mailbox.

Re: [Qemu-devel] [PULL 00/21] Docker and shippable updates

[Philippe Mathieu-Daudé]

On Thu, Jun 22, 2017 at 12:42 PM, Philippe Mathieu-Daudé
<f4bug@amsat.org> wrote:
> On Thu, Jun 22, 2017 at 12:13 PM, Alex Bennée <alex.bennee@linaro.org> wrote:
>> Now we've been running shippable for a while is it worth turning on the
>> IRC notifications?
>
> What about moving such bot/scripts notifications in another channel
> like #QEMU-notifications?

ping?

> It is often hard to follow 3 concurrent topics while the travis-ci bot
> disrupts entering/broadcasting/leaving...
> I'd also like to see here checkpatch/patchew stripped output, and
> eventually gcov/gprof reports.
> I don't think full patchew reports are useful in a mailbox.

Re: [Qemu-devel] [PULL 00/21] Docker and shippable updates

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On Thu, Jun 22, 2017 at 12:42 PM, Philippe Mathieu-Daudé
> <f4bug@amsat.org> wrote:
>> On Thu, Jun 22, 2017 at 12:13 PM, Alex Bennée <alex.bennee@linaro.org> wrote:
>>> Now we've been running shippable for a while is it worth turning on the
>>> IRC notifications?
>>
>> What about moving such bot/scripts notifications in another channel
>> like #QEMU-notifications?
>
> ping?

I'm in two minds so I suspect this is something that needs to be taken
to the users of #qemu.

>> It is often hard to follow 3 concurrent topics while the travis-ci bot
>> disrupts entering/broadcasting/leaving...

I find it useful as its somewhere I look at a lot. That said it is not
exactly a trial to sub to yet another IRC channel.

>> I'd also like to see here checkpatch/patchew stripped output, and
>> eventually gcov/gprof reports.
>> I don't think full patchew reports are useful in a mailbox.

I disagree. They could certainly be made terser but I think the general
workflow of posting automated testing reports to patch series is a
useful one.

--
Alex Bennée

Re: [Qemu-devel] [PULL 00/21] Docker and shippable updates

[Peter Maydell]

On 10 July 2017 at 15:17, Alex Bennée <alex.bennee@linaro.org> wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>> I don't think full patchew reports are useful in a mailbox.
>
> I disagree. They could certainly be made terser but I think the general
> workflow of posting automated testing reports to patch series is a
> useful one.

Agreed. What we want is a mode for the makefile where
it doesn't even print the "   CC foo.o" lines (except
perhaps for the case where it's going to print an error
message, if possible, but that might not be doable), and
then we can use that for patchew emails and also maybe
for the Travis builds, whose logs are currently too long
for the web viewer to cope with (might run into issues with
travis thinking the build has stalled though.)

Is there a standard-ish convention for a really-silent
build we can borrow from the linux kernel or somewhere?

thanks
-- PMM

Re: [Qemu-devel] [PULL 00/21] Docker and shippable updates

[Philippe Mathieu-Daudé]

On Mon, Jul 10, 2017 at 11:22 AM, Peter Maydell
<peter.maydell@linaro.org> wrote:
> On 10 July 2017 at 15:17, Alex Bennée <alex.bennee@linaro.org> wrote:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>>> I don't think full patchew reports are useful in a mailbox.
>>
>> I disagree. They could certainly be made terser but I think the general
>> workflow of posting automated testing reports to patch series is a
>> useful one.
>
> Agreed. What we want is a mode for the makefile where
> it doesn't even print the "   CC foo.o" lines (except
> perhaps for the case where it's going to print an error
> message, if possible, but that might not be doable), and
> then we can use that for patchew emails and also maybe
> for the Travis builds, whose logs are currently too long
> for the web viewer to cope with (might run into issues with
> travis thinking the build has stalled though.)

Using webmail (gmail) you get:

---
Hi,

This series failed automatic build test. Please find the testing commands and
their output below. If you have docker installed, you can probably reproduce it
locally.
[...]
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC      arm-softmmu/hw/net/vhost_net.o
  CC      aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      arm-softmmu/hw/pcmcia/...

[Message clipped]  View entire message
---

also this one 132KB alone which makes thunderbird feature "filter
messages [Body]" slow to unresponsive:

---
Hi,

This series failed build test on s390x host. Please find the details below.

Type: series
Message-id: 1496820931-27416-10-git-send-email-a.perevalov@samsung.com
Subject: [Qemu-devel] [[PATCH V7] 09/11] migration: calculate vCPU
blocktime on dst side
---

I'm happy with the script output workflow, just say it could be more
useful if we can shorter it to focus on the problem, tailing last 500
stderr lines should be more than enough.