[Qemu-devel] [PATCH 0/8] Docker improvements

[Qemu-devel] [PATCH 0/8] Docker improvements

[Philippe Mathieu-Daudé]

Hi,

This is a set of various unrelated Docker patches, from my daily
docker-qemu usage.

Last RFC patches restore building for ppc32 hosts, which sometimes
triggers interesting 32-bit errors (atomic updates).

The other 32-bit host is the MXE MinGW32 port.

Regards,

Phil.

Philippe Mathieu-Daudé (8):
  docker: ubuntu: Update the package list before installing new ones
  docker: ubuntu: Use SDL2
  docker: Clean the MXE base image
  docker: Add packages required to build a Linux kernel
  docker: Restrict the 'travis' job to the Travis image
  docker: Do not run tests in 'intermediate' images
  docker: Cross build QEMU on ppc32 hosts
  .shippable.yml: Restore the powerpc-cross image

 .shippable.yml                                |  2 +
 tests/docker/Makefile.include                 | 16 +++++--
 tests/docker/dockerfiles/debian-apt-fake.sh   | 46 +++++++++++++++++++
 .../dockerfiles/debian-powerpc-cross.docker   | 43 +++++++++++++++++
 tests/docker/dockerfiles/debian-sid.docker    |  6 ++-
 tests/docker/dockerfiles/debian8-mxe.docker   |  2 +-
 tests/docker/dockerfiles/ubuntu.docker        |  8 ++--
 7 files changed, 114 insertions(+), 9 deletions(-)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh

-- 
2.18.0

[Qemu-devel] [PATCH 1/8] docker: ubuntu: Update the package list before installing new ones

[Philippe Mathieu-Daudé]

Since docker caches the different layers, updating the package
list does not invalidate the previous "apt-get update" layer,
and it is likely "apt-get install" hits an outdated repository.

See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#apt-get

This fixes:

  $ make docker-image-ubuntu V=1
  ./tests/docker/docker.py build qemu:ubuntu tests/docker/dockerfiles/ubuntu.docker   --add-current-user
  Sending build context to Docker daemon  3.072kB
  [...]
  E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/libgles2-mesa_17.0.7-0ubuntu0.16.04.2_amd64.deb  404  Not Found
  E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/libgles2-mesa-dev_17.0.7-0ubuntu0.16.04.2_amd64.deb  404  Not Found
  E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
  The command '/bin/sh -c apt-get -y install $PACKAGES' returned a non-zero code: 100
  tests/docker/Makefile.include:40: recipe for target 'docker-image-ubuntu' failed
  make: *** [docker-image-ubuntu] Error 1

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/ubuntu.docker | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index dabbf2a8a4..c03520ce3f 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -1,7 +1,6 @@
 FROM ubuntu:16.04
 RUN echo "deb http://archive.ubuntu.com/ubuntu/ trusty universe multiverse" >> \
     /etc/apt/sources.list
-RUN apt-get update
 ENV PACKAGES flex bison \
     libusb-1.0-0-dev libiscsi-dev librados-dev libncurses5-dev libncursesw5-dev \
     libseccomp-dev libgnutls-dev libssh2-1-dev  libspice-server-dev \
@@ -13,6 +12,7 @@ ENV PACKAGES flex bison \
     libjemalloc-dev libcacard-dev libusbredirhost-dev libnfs-dev libcap-dev libattr1-dev \
     texinfo \
     gettext git make ccache python-yaml gcc clang sparse
-RUN apt-get -y install $PACKAGES
+RUN apt-get update && \
+    apt-get -y install $PACKAGES
 RUN dpkg -l $PACKAGES | sort > /packages.txt
 ENV FEATURES clang pyyaml
-- 
2.18.0

[Qemu-devel] [PATCH 2/8] docker: ubuntu: Use SDL2

[Philippe Mathieu-Daudé]

Do not test the deprecated API versions (see cabd35840749d).
Debian MXE MinGW cross images are already using SDL2.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/ubuntu.docker | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index c03520ce3f..7d724e7f53 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -5,7 +5,7 @@ ENV PACKAGES flex bison \
     libusb-1.0-0-dev libiscsi-dev librados-dev libncurses5-dev libncursesw5-dev \
     libseccomp-dev libgnutls-dev libssh2-1-dev  libspice-server-dev \
     libspice-protocol-dev libnss3-dev libfdt-dev \
-    libgtk-3-dev libvte-2.91-dev libsdl1.2-dev libpng12-dev libpixman-1-dev \
+    libgtk-3-dev libvte-2.91-dev libsdl2-dev libpng12-dev libpixman-1-dev \
     libvdeplug-dev liblzo2-dev libsnappy-dev libbz2-dev libxen-dev librdmacm-dev libibverbs-dev \
     libsasl2-dev libjpeg-turbo8-dev xfslibs-dev libcap-ng-dev libbrlapi-dev libcurl4-gnutls-dev \
     libbluetooth-dev librbd-dev libaio-dev glusterfs-common libnuma-dev libepoxy-dev libdrm-dev libgbm-dev \
@@ -15,4 +15,4 @@ ENV PACKAGES flex bison \
 RUN apt-get update && \
     apt-get -y install $PACKAGES
 RUN dpkg -l $PACKAGES | sort > /packages.txt
-ENV FEATURES clang pyyaml
+ENV FEATURES clang pyyaml sdl2
-- 
2.18.0

[Qemu-devel] [PATCH 3/8] docker: Clean the MXE base image

[Philippe Mathieu-Daudé]

Using the duplicated same package is confusing.

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
It would be even better to figure out what I wanted to do with this command
and add a comment in the Dockerfile :)

I think the original idea was to share the packages common to both gw32/gw64
images in this intermediate common image.

 tests/docker/dockerfiles/debian8-mxe.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian8-mxe.docker b/tests/docker/dockerfiles/debian8-mxe.docker
index 9b8e577b03..2df4cc8c5c 100644
--- a/tests/docker/dockerfiles/debian8-mxe.docker
+++ b/tests/docker/dockerfiles/debian8-mxe.docker
@@ -14,6 +14,6 @@ RUN apt-get update
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
     apt-get install -y --no-install-recommends \
         libpython2.7-stdlib \
-        $(apt-get -s install -y --no-install-recommends gw32.shared-mingw-w64 gw32.shared-mingw-w64 | egrep "^Inst mxe-x86-64-unknown-" | cut -d\  -f2)
+        $(apt-get -s install -y --no-install-recommends gw32.shared-mingw-w64 | egrep "^Inst mxe-x86-64-unknown-" | cut -d\  -f2)
 
 ENV PATH $PATH:/usr/lib/mxe/usr/bin/ 
-- 
2.18.0

[Qemu-devel] [RFC PATCH 4/8] docker: Add packages required to build a Linux kernel

[Philippe Mathieu-Daudé]

Being able to also cross build Linux kernels with our docker cross
images allow us to add more complex qemu-system acceptance tests.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
This is not required to build QEMU, so I might resend it in the
acceptance-test series which actually requires this patch.

 tests/docker/dockerfiles/debian-sid.docker | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker
index 9a3d168705..846021b032 100644
--- a/tests/docker/dockerfiles/debian-sid.docker
+++ b/tests/docker/dockerfiles/debian-sid.docker
@@ -22,11 +22,15 @@ RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
     apt install -y --no-install-recommends \
         bison \
+        binutils-multiarch \
         build-essential \
         ca-certificates \
+        curl \
         flex \
+        gettext \
         git \
+        gnupg \
         pkg-config \
         psmisc \
-        python \
+        python-minimal \
         texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
-- 
2.18.0

[Qemu-devel] [RFC PATCH 5/8] docker: Restrict the 'travis' job to the Travis image

[Philippe Mathieu-Daudé]

We can still run any test in Travis.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 91d9665517..7d9d568eee 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -99,11 +99,17 @@ docker-image-tricore-cross: docker-image-debian9
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES), \
-	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
+	$(foreach t,$(DOCKER_TESTS), \
 		$(eval .PHONY: docker-$t@$i) \
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
+		$(eval docker-test: docker-$t@$i) \
 	) \
-	$(foreach t,$(DOCKER_TESTS), \
+)
+# we only run Travis tests on the Travis image
+$(foreach i,travis, \
+	$(foreach t,$(DOCKER_TOOLS), \
+		$(eval .PHONY: docker-$t@$i) \
+		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
 		$(eval docker-test: docker-$t@$i) \
 	) \
 )
-- 
2.18.0

[Qemu-devel] [PATCH 6/8] docker: Do not run tests in 'intermediate' images

[Philippe Mathieu-Daudé]

We can still build the DOCKER_INTERMEDIATE_IMAGES images,
but they won't appear in 'make test*@$IMAGE'.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 7d9d568eee..301ee68ae6 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -5,6 +5,8 @@
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 DOCKER_DEPRECATED_IMAGES := debian
+# we don't run tests on intermediate images (used as base by another image)
+DOCKER_INTERMEDIATE_IMAGES := debian8 debian9 debian8-mxe debian-ports
 DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
@@ -98,7 +100,7 @@ docker-image-travis: NOUSER=1
 docker-image-tricore-cross: docker-image-debian9
 
 # Expand all the pre-requistes for each docker image and test combination
-$(foreach i,$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES), \
+$(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS), \
 		$(eval .PHONY: docker-$t@$i) \
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
-- 
2.18.0

[Qemu-devel] [PATCH 7/8] docker: Cross build QEMU on ppc32 hosts

[Philippe Mathieu-Daudé]

Since 08f56d8c9c the powerpc-cross image is no more based on Emdebian
but on the Sid release.

This patch restore the ability to cross build QEMU on powerpc, which
is interesting since it is a 32-bit host.

The ugly apt-fake script is, however, still required.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include                 |  2 +
 tests/docker/dockerfiles/debian-apt-fake.sh   | 46 +++++++++++++++++++
 .../dockerfiles/debian-powerpc-cross.docker   | 43 +++++++++++++++++
 3 files changed, 91 insertions(+)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 301ee68ae6..d73d60cfbb 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -58,6 +58,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		"BUILD","$*")
 endif
 
+docker-image-debian-powerpc-cross: EXTRA_FILES:=$(SRC_PATH)/tests/docker/dockerfiles/debian-apt-fake.sh
+
 # Enforce dependencies for composite images
 docker-image-debian: docker-image-debian9
 docker-image-debian8-mxe: docker-image-debian8
diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
new file mode 100755
index 0000000000..2ec0fdf47a
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-apt-fake.sh
@@ -0,0 +1,46 @@
+#! /bin/sh
+#
+# Generate fake debian package to resolve unimportant unmet dependencies held
+# by upstream multiarch broken packages.
+#
+# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+test $1 = "install" && shift 1
+
+fake_install()
+{
+    echo "Generating fake $2 $1 $3 ..."
+    (cd /var/cache/apt/archives
+        (cat << 'EOF'
+Section: misc
+Priority: optional
+Standards-Version: 3.9.2
+
+Package: NAME
+Version: VERSION
+Maintainer: qemu-devel@nongnu.org
+Architecture: any
+Multi-Arch: same
+Description: fake NAME
+EOF
+        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
+        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
+        dpkg -i --force-overwrite $2_$3_$1.deb
+    )
+}
+
+try_install()
+{
+    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
+    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
+    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
+    apt-get install -q -yy $1 || fake_install $arch $name $vers
+}
+
+for package in $*; do
+    try_install $package
+done
diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
index 5e62ca0df1..4ccd17d9d5 100644
--- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -11,3 +11,46 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
     apt-get install -y --no-install-recommends \
         gcc-powerpc-linux-gnu \
         libc6-dev-powerpc-cross || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
+
+# Setup some basic tools we need
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        bison \
+        binutils-multiarch \
+        build-essential \
+        ca-certificates \
+        curl \
+        flex \
+        gettext \
+        git \
+        gnupg \
+        pkg-config \
+        python-minimal
+
+RUN dpkg --add-architecture powerpc && \
+    apt-get update
+
+ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
+
+# <kludge> to fix "following packages have unmet dependencies" ...
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+RUN apt-get install -y --no-install-recommends \
+        equivs
+RUN apt-fake install \
+        glusterfs-common:powerpc=4.0.2-fake
+# </kludge>
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get build-dep -yy -a powerpc qemu
+RUN DEBIAN_FRONTEND=noninteractive \
+    apt-get install -y --no-install-recommends \
+        glusterfs-common:powerpc \
+        libbz2-dev:powerpc \
+        liblzo2-dev:powerpc \
+        libncursesw5-dev:powerpc \
+        libnfs-dev:powerpc \
+        librdmacm-dev:powerpc \
+        libsnappy-dev:powerpc
-- 
2.18.0

[Qemu-devel] [PATCH 8/8] .shippable.yml: Restore the powerpc-cross image

[Philippe Mathieu-Daudé]

This reverts commit b2dba411cf5afc6771e0dd9900fa2b5031a686ac.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index f74a3de3ff..e907a4addd 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -23,6 +23,8 @@ env:
       TARGET_LIST=mips-softmmu,mipsel-linux-user
     - IMAGE=debian-mips64el-cross
       TARGET_LIST=mips64el-softmmu,mips64el-linux-user
+    - IMAGE=debian-powerpc-cross
+      TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
     - IMAGE=debian-ppc64el-cross
       TARGET_LIST=ppc64-softmmu,ppc64-linux-user,ppc64abi32-linux-user
 build:
-- 
2.18.0

Re: [Qemu-devel] [RFC PATCH 4/8] docker: Add packages required to build a Linux kernel

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Being able to also cross build Linux kernels with our docker cross
> images allow us to add more complex qemu-system acceptance tests.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> This is not required to build QEMU, so I might resend it in the
> acceptance-test series which actually requires this patch.
>
>  tests/docker/dockerfiles/debian-sid.docker | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker
> index 9a3d168705..846021b032 100644
> --- a/tests/docker/dockerfiles/debian-sid.docker
> +++ b/tests/docker/dockerfiles/debian-sid.docker
> @@ -22,11 +22,15 @@ RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
>  RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>      apt install -y --no-install-recommends \
>          bison \
> +        binutils-multiarch \
>          build-essential \
>          ca-certificates \
> +        curl \
>          flex \
> +        gettext \
>          git \
> +        gnupg \
>          pkg-config \
>          psmisc \
> -        python \
> +        python-minimal \

Hmm don't we need full python for QEMU though?

>          texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 5/8] docker: Restrict the 'travis' job to the Travis image

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> We can still run any test in Travis.


I don't know about this one. The travis job isn't a fill in for real
travis, but a way to work through the matrix. I don't think it needs to
be restricted to any particular image.

>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 91d9665517..7d9d568eee 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -99,11 +99,17 @@ docker-image-tricore-cross: docker-image-debian9
>
>  # Expand all the pre-requistes for each docker image and test combination
>  $(foreach i,$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES), \
> -	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
> +	$(foreach t,$(DOCKER_TESTS), \
>  		$(eval .PHONY: docker-$t@$i) \
>  		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
> +		$(eval docker-test: docker-$t@$i) \
>  	) \
> -	$(foreach t,$(DOCKER_TESTS), \
> +)
> +# we only run Travis tests on the Travis image
> +$(foreach i,travis, \
> +	$(foreach t,$(DOCKER_TOOLS), \
> +		$(eval .PHONY: docker-$t@$i) \
> +		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
>  		$(eval docker-test: docker-$t@$i) \
>  	) \
>  )


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 5/8] docker: Restrict the 'travis' job to the Travis image

[Fam Zheng]

On Thu, 06/28 13:46, Philippe Mathieu-Daudé wrote:
> We can still run any test in Travis.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 91d9665517..7d9d568eee 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -99,11 +99,17 @@ docker-image-tricore-cross: docker-image-debian9
>  
>  # Expand all the pre-requistes for each docker image and test combination
>  $(foreach i,$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES), \
> -	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
> +	$(foreach t,$(DOCKER_TESTS), \
>  		$(eval .PHONY: docker-$t@$i) \
>  		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
> +		$(eval docker-test: docker-$t@$i) \
>  	) \
> -	$(foreach t,$(DOCKER_TESTS), \
> +)
> +# we only run Travis tests on the Travis image
> +$(foreach i,travis, \
> +	$(foreach t,$(DOCKER_TOOLS), \
> +		$(eval .PHONY: docker-$t@$i) \
> +		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
>  		$(eval docker-test: docker-$t@$i) \
>  	) \
>  )
> -- 
> 2.18.0
> 

Apart from Alex's concern (which I don't have), I suppose a 'requires travis'
and 'ENV features travis' pair is the correct way to do it. See test-mingw for
example.

Fam

Re: [Qemu-devel] [PATCH 7/8] docker: Cross build QEMU on ppc32 hosts

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Since 08f56d8c9c the powerpc-cross image is no more based on Emdebian
> but on the Sid release.
>
> This patch restore the ability to cross build QEMU on powerpc, which
> is interesting since it is a 32-bit host.
>
> The ugly apt-fake script is, however, still required.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>


Hmm I couldn't get this to work with:

   make docker-test-build@debian-powerpc-cross J=9 V=1 NOCACHE=1


Step 9/13 : RUN apt-fake install         glusterfs-common:powerpc=4.0.2-fake
 ---> Running in 0bf7bf5e268a
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '4.0.2-fake' for 'glusterfs-common:powerpc' was not found
Generating fake glusterfs-common powerpc 4.0.2-fake ...
Selecting previously unselected package glusterfs-common:powerpc.
(Reading database ... 19461 files and directories currently installed.)
Preparing to unpack glusterfs-common_4.0.2-fake_powerpc.deb ...
Unpacking glusterfs-common:powerpc (4.0.2-fake) ...
Setting up glusterfs-common:powerpc (4.0.2-fake) ...
Removing intermediate container 0bf7bf5e268a
 ---> 9f903850b104
Step 10/13 : RUN DEBIAN_FRONTEND=noninteractive eatmydata     apt-get build-dep -yy -a powerpc qemu
 ---> Running in 973b5c17668d
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 builddeps:qemu:powerpc : Depends: libcapstone-dev:powerpc but it is not going to be installed
                          Depends: libaio-dev:powerpc but it is not going to be installed
                          Depends: libpulse-dev:powerpc but it is not going to be installed
                          Depends: libasound2-dev:powerpc but it is not going to be installed
                          Depends: libattr1-dev:powerpc but it is not going to be installed
                          Depends: libbluetooth-dev:powerpc but it is not going to be installed
                          Depends: libbrlapi-dev:powerpc but it is not going to be installed
                          Depends: libcap-dev:powerpc but it is not going to be installed
                          Depends: libcap-ng-dev:powerpc but it is not going to be installed
                          Depends: libcurl4-gnutls-dev:powerpc but it is not going to be installed
                          Depends: libfdt-dev:powerpc but it is not going to be installed
                          Depends: gnutls-dev:powerpc
                          Depends: libgtk-3-dev:powerpc but it is not going to be installed
                          Depends: libvte-2.91-dev:powerpc but it is not going to be installed
                          Depends: libiscsi-dev:powerpc (> 1.9.0~) but it is not going to be installed
                          Depends: libncursesw5-dev:powerpc
                          Depends: libnfs-dev:powerpc (> 1.9.3) but it is not going to be installed
                          Depends: libnuma-dev:powerpc but it is not going to be installed
                          Depends: libcacard-dev:powerpc but it is not going to be installed
                          Depends: libpixman-1-dev:powerpc but it is not going to be installed
                          Depends: librados-dev:powerpc but it is not going to be installed
                          Depends: librbd-dev:powerpc but it is not going to be installed
                          Depends: libsasl2-dev:powerpc but it is not going to be installed
                          Depends: libseccomp-dev:powerpc (>= 2.3.0) but it is not going to be installed
                          Depends: libusb-1.0-0-dev:powerpc (>= 2:1.0.13~) but it is not going to be installed
                          Depends: libusbredirparser-dev:powerpc (>= 0.6~) but it is not going to be installed
                          Depends: libssh2-1-dev:powerpc but it is not going to be installed
                          Depends: libvdeplug-dev:powerpc but it is not going to be installed
                          Depends: uuid-dev:powerpc but it is not going to be installed
                          Depends: xfslibs-dev:powerpc but it is not going to be installed
                          Depends: zlib1g-dev:powerpc but it is not going to be installed
                          Depends: libjpeg-dev:powerpc
                          Depends: libpng-dev:powerpc but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive eatmydata     apt-get build-dep -yy -a powerpc qemu' returned a non-zero code: 100
Traceback (most recent call last):
  File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 526, in <module>
    sys.exit(main())
  File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 523, in main
    return args.cmdobj.run(args, argv)
  File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 344, in run
    extra_files_cksum=cksum)
  File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 227, in build_image
    quiet=quiet)
  File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 154, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '-t', 'qemu:debian-powerpc-cross', '-f', '/tmp/docker_buildjXN8x4/tmpJASaT1.docker', '--no-cache', '/tmp/docker_buildjXN8x4']' returned non-zero exit status 100
/home/alex/lsrc/qemu/qemu.git/tests/docker/Makefile.include:50: recipe for target 'docker-image-debian-powerpc-cross' failed
make: *** [docker-image-debian-powerpc-cross] Error 1

> ---
>  tests/docker/Makefile.include                 |  2 +
>  tests/docker/dockerfiles/debian-apt-fake.sh   | 46 +++++++++++++++++++
>  .../dockerfiles/debian-powerpc-cross.docker   | 43 +++++++++++++++++
>  3 files changed, 91 insertions(+)
>  create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 301ee68ae6..d73d60cfbb 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -58,6 +58,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		"BUILD","$*")
>  endif
>
> +docker-image-debian-powerpc-cross: EXTRA_FILES:=$(SRC_PATH)/tests/docker/dockerfiles/debian-apt-fake.sh
> +
>  # Enforce dependencies for composite images
>  docker-image-debian: docker-image-debian9
>  docker-image-debian8-mxe: docker-image-debian8
> diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
> new file mode 100755
> index 0000000000..2ec0fdf47a
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-apt-fake.sh
> @@ -0,0 +1,46 @@
> +#! /bin/sh
> +#
> +# Generate fake debian package to resolve unimportant unmet dependencies held
> +# by upstream multiarch broken packages.
> +#
> +# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +test $1 = "install" && shift 1
> +
> +fake_install()
> +{
> +    echo "Generating fake $2 $1 $3 ..."
> +    (cd /var/cache/apt/archives
> +        (cat << 'EOF'
> +Section: misc
> +Priority: optional
> +Standards-Version: 3.9.2
> +
> +Package: NAME
> +Version: VERSION
> +Maintainer: qemu-devel@nongnu.org
> +Architecture: any
> +Multi-Arch: same
> +Description: fake NAME
> +EOF
> +        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
> +        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
> +        dpkg -i --force-overwrite $2_$3_$1.deb
> +    )
> +}
> +
> +try_install()
> +{
> +    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
> +    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
> +    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
> +    apt-get install -q -yy $1 || fake_install $arch $name $vers
> +}
> +
> +for package in $*; do
> +    try_install $package
> +done
> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
> index 5e62ca0df1..4ccd17d9d5 100644
> --- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
> @@ -11,3 +11,46 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>      apt-get install -y --no-install-recommends \
>          gcc-powerpc-linux-gnu \
>          libc6-dev-powerpc-cross || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
> +
> +# Setup some basic tools we need
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get install -y --no-install-recommends \
> +        bison \
> +        binutils-multiarch \
> +        build-essential \
> +        ca-certificates \
> +        curl \
> +        flex \
> +        gettext \
> +        git \
> +        gnupg \
> +        pkg-config \
> +        python-minimal
> +
> +RUN dpkg --add-architecture powerpc && \
> +    apt-get update
> +
> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
> +
> +# Specify the cross prefix for this image (see tests/docker/common.rc)
> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
> +
> +# <kludge> to fix "following packages have unmet dependencies" ...
> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
> +RUN apt-get install -y --no-install-recommends \
> +        equivs
> +RUN apt-fake install \
> +        glusterfs-common:powerpc=4.0.2-fake
> +# </kludge>
> +
> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> +    apt-get build-dep -yy -a powerpc qemu
> +RUN DEBIAN_FRONTEND=noninteractive \
> +    apt-get install -y --no-install-recommends \
> +        glusterfs-common:powerpc \
> +        libbz2-dev:powerpc \
> +        liblzo2-dev:powerpc \
> +        libncursesw5-dev:powerpc \
> +        libnfs-dev:powerpc \
> +        librdmacm-dev:powerpc \
> +        libsnappy-dev:powerpc


--
Alex Bennée

Re: [Qemu-devel] [PATCH 0/8] Docker improvements

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi,
>
Hi,

I've queued the following in my tree.

>   docker: ubuntu: Update the package list before installing new ones
>   docker: ubuntu: Use SDL2
>   docker: Clean the MXE base image
>   docker: Do not run tests in 'intermediate' images

--
Alex Bennée

Re: [Qemu-devel] [PATCH 7/8] docker: Cross build QEMU on ppc32 hosts

[Philippe Mathieu-Daudé]

On 06/29/2018 11:29 AM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Since 08f56d8c9c the powerpc-cross image is no more based on Emdebian
>> but on the Sid release.
>>
>> This patch restore the ability to cross build QEMU on powerpc, which
>> is interesting since it is a 32-bit host.
>>
>> The ugly apt-fake script is, however, still required.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> 
> Hmm I couldn't get this to work with:
> 
>    make docker-test-build@debian-powerpc-cross J=9 V=1 NOCACHE=1
> 
> 
> Step 9/13 : RUN apt-fake install         glusterfs-common:powerpc=4.0.2-fake
>  ---> Running in 0bf7bf5e268a
> Reading package lists...
> Building dependency tree...
> Reading state information...
> E: Version '4.0.2-fake' for 'glusterfs-common:powerpc' was not found
> Generating fake glusterfs-common powerpc 4.0.2-fake ...
> Selecting previously unselected package glusterfs-common:powerpc.
> (Reading database ... 19461 files and directories currently installed.)
> Preparing to unpack glusterfs-common_4.0.2-fake_powerpc.deb ...
> Unpacking glusterfs-common:powerpc (4.0.2-fake) ...
> Setting up glusterfs-common:powerpc (4.0.2-fake) ...
> Removing intermediate container 0bf7bf5e268a
>  ---> 9f903850b104
> Step 10/13 : RUN DEBIAN_FRONTEND=noninteractive eatmydata     apt-get build-dep -yy -a powerpc qemu
>  ---> Running in 973b5c17668d
> Reading package lists...
> Building dependency tree...
> Reading state information...
> Some packages could not be installed. This may mean that you have
> requested an impossible situation or if you are using the unstable
> distribution that some required packages have not yet been created
> or been moved out of Incoming.
> The following information may help to resolve the situation:
> 
> The following packages have unmet dependencies:
>  builddeps:qemu:powerpc : Depends: libcapstone-dev:powerpc but it is not going to be installed
>                           Depends: libaio-dev:powerpc but it is not going to be installed
>                           Depends: libpulse-dev:powerpc but it is not going to be installed
>                           Depends: libasound2-dev:powerpc but it is not going to be installed
>                           Depends: libattr1-dev:powerpc but it is not going to be installed
>                           Depends: libbluetooth-dev:powerpc but it is not going to be installed
>                           Depends: libbrlapi-dev:powerpc but it is not going to be installed
>                           Depends: libcap-dev:powerpc but it is not going to be installed
>                           Depends: libcap-ng-dev:powerpc but it is not going to be installed
>                           Depends: libcurl4-gnutls-dev:powerpc but it is not going to be installed
>                           Depends: libfdt-dev:powerpc but it is not going to be installed
>                           Depends: gnutls-dev:powerpc
>                           Depends: libgtk-3-dev:powerpc but it is not going to be installed
>                           Depends: libvte-2.91-dev:powerpc but it is not going to be installed
>                           Depends: libiscsi-dev:powerpc (> 1.9.0~) but it is not going to be installed
>                           Depends: libncursesw5-dev:powerpc
>                           Depends: libnfs-dev:powerpc (> 1.9.3) but it is not going to be installed
>                           Depends: libnuma-dev:powerpc but it is not going to be installed
>                           Depends: libcacard-dev:powerpc but it is not going to be installed
>                           Depends: libpixman-1-dev:powerpc but it is not going to be installed
>                           Depends: librados-dev:powerpc but it is not going to be installed
>                           Depends: librbd-dev:powerpc but it is not going to be installed
>                           Depends: libsasl2-dev:powerpc but it is not going to be installed
>                           Depends: libseccomp-dev:powerpc (>= 2.3.0) but it is not going to be installed
>                           Depends: libusb-1.0-0-dev:powerpc (>= 2:1.0.13~) but it is not going to be installed
>                           Depends: libusbredirparser-dev:powerpc (>= 0.6~) but it is not going to be installed
>                           Depends: libssh2-1-dev:powerpc but it is not going to be installed
>                           Depends: libvdeplug-dev:powerpc but it is not going to be installed
>                           Depends: uuid-dev:powerpc but it is not going to be installed
>                           Depends: xfslibs-dev:powerpc but it is not going to be installed
>                           Depends: zlib1g-dev:powerpc but it is not going to be installed
>                           Depends: libjpeg-dev:powerpc
>                           Depends: libpng-dev:powerpc but it is not going to be installed
> E: Unable to correct problems, you have held broken packages.
> The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive eatmydata     apt-get build-dep -yy -a powerpc qemu' returned a non-zero code: 100
> Traceback (most recent call last):
>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 526, in <module>
>     sys.exit(main())
>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 523, in main
>     return args.cmdobj.run(args, argv)
>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 344, in run
>     extra_files_cksum=cksum)
>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 227, in build_image
>     quiet=quiet)
>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 154, in _do_check
>     return subprocess.check_call(self._command + cmd, **kwargs)
>   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['docker', 'build', '-t', 'qemu:debian-powerpc-cross', '-f', '/tmp/docker_buildjXN8x4/tmpJASaT1.docker', '--no-cache', '/tmp/docker_buildjXN8x4']' returned non-zero exit status 100
> /home/alex/lsrc/qemu/qemu.git/tests/docker/Makefile.include:50: recipe for target 'docker-image-debian-powerpc-cross' failed
> make: *** [docker-image-debian-powerpc-cross] Error 1

Caveats of using Debian/Sid again :(

It worked for me 5 days ago, and I'v been using this image since, but
without rebuilding it, so did not notice the upstream packages change
again... whew :(

Let's postpone this for the next development window.

Thanks for giving it a try!

Phil.

> 
>> ---
>>  tests/docker/Makefile.include                 |  2 +
>>  tests/docker/dockerfiles/debian-apt-fake.sh   | 46 +++++++++++++++++++
>>  .../dockerfiles/debian-powerpc-cross.docker   | 43 +++++++++++++++++
>>  3 files changed, 91 insertions(+)
>>  create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 301ee68ae6..d73d60cfbb 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -58,6 +58,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>>  		"BUILD","$*")
>>  endif
>>
>> +docker-image-debian-powerpc-cross: EXTRA_FILES:=$(SRC_PATH)/tests/docker/dockerfiles/debian-apt-fake.sh
>> +
>>  # Enforce dependencies for composite images
>>  docker-image-debian: docker-image-debian9
>>  docker-image-debian8-mxe: docker-image-debian8
>> diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
>> new file mode 100755
>> index 0000000000..2ec0fdf47a
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/debian-apt-fake.sh
>> @@ -0,0 +1,46 @@
>> +#! /bin/sh
>> +#
>> +# Generate fake debian package to resolve unimportant unmet dependencies held
>> +# by upstream multiarch broken packages.
>> +#
>> +# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2
>> +# or (at your option) any later version. See the COPYING file in
>> +# the top-level directory.
>> +
>> +test $1 = "install" && shift 1
>> +
>> +fake_install()
>> +{
>> +    echo "Generating fake $2 $1 $3 ..."
>> +    (cd /var/cache/apt/archives
>> +        (cat << 'EOF'
>> +Section: misc
>> +Priority: optional
>> +Standards-Version: 3.9.2
>> +
>> +Package: NAME
>> +Version: VERSION
>> +Maintainer: qemu-devel@nongnu.org
>> +Architecture: any
>> +Multi-Arch: same
>> +Description: fake NAME
>> +EOF
>> +        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
>> +        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
>> +        dpkg -i --force-overwrite $2_$3_$1.deb
>> +    )
>> +}
>> +
>> +try_install()
>> +{
>> +    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
>> +    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
>> +    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
>> +    apt-get install -q -yy $1 || fake_install $arch $name $vers
>> +}
>> +
>> +for package in $*; do
>> +    try_install $package
>> +done
>> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> index 5e62ca0df1..4ccd17d9d5 100644
>> --- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> @@ -11,3 +11,46 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>>      apt-get install -y --no-install-recommends \
>>          gcc-powerpc-linux-gnu \
>>          libc6-dev-powerpc-cross || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
>> +
>> +# Setup some basic tools we need
>> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>> +    apt-get install -y --no-install-recommends \
>> +        bison \
>> +        binutils-multiarch \
>> +        build-essential \
>> +        ca-certificates \
>> +        curl \
>> +        flex \
>> +        gettext \
>> +        git \
>> +        gnupg \
>> +        pkg-config \
>> +        python-minimal
>> +
>> +RUN dpkg --add-architecture powerpc && \
>> +    apt-get update
>> +
>> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
>> +
>> +# Specify the cross prefix for this image (see tests/docker/common.rc)
>> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
>> +
>> +# <kludge> to fix "following packages have unmet dependencies" ...
>> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
>> +RUN apt-get install -y --no-install-recommends \
>> +        equivs
>> +RUN apt-fake install \
>> +        glusterfs-common:powerpc=4.0.2-fake
>> +# </kludge>
>> +
>> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>> +    apt-get build-dep -yy -a powerpc qemu
>> +RUN DEBIAN_FRONTEND=noninteractive \
>> +    apt-get install -y --no-install-recommends \
>> +        glusterfs-common:powerpc \
>> +        libbz2-dev:powerpc \
>> +        liblzo2-dev:powerpc \
>> +        libncursesw5-dev:powerpc \
>> +        libnfs-dev:powerpc \
>> +        librdmacm-dev:powerpc \
>> +        libsnappy-dev:powerpc
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [RFC PATCH 4/8] docker: Add packages required to build a Linux kernel

[Philippe Mathieu-Daudé]

On 06/29/2018 11:09 AM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Being able to also cross build Linux kernels with our docker cross
>> images allow us to add more complex qemu-system acceptance tests.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> This is not required to build QEMU, so I might resend it in the
>> acceptance-test series which actually requires this patch.
>>
>>  tests/docker/dockerfiles/debian-sid.docker | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/docker/dockerfiles/debian-sid.docker b/tests/docker/dockerfiles/debian-sid.docker
>> index 9a3d168705..846021b032 100644
>> --- a/tests/docker/dockerfiles/debian-sid.docker
>> +++ b/tests/docker/dockerfiles/debian-sid.docker
>> @@ -22,11 +22,15 @@ RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
>>  RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>>      apt install -y --no-install-recommends \
>>          bison \
>> +        binutils-multiarch \
>>          build-essential \
>>          ca-certificates \
>> +        curl \
>>          flex \
>> +        gettext \
>>          git \
>> +        gnupg \
>>          pkg-config \
>>          psmisc \
>> -        python \
>> +        python-minimal \
> 
> Hmm don't we need full python for QEMU though?

I'm not sure, but the goal of this image is to be useful, not tiny, so
I'm happy with the normal package (I cherry-picked this from an older
series and did not notice this particular change).

> 
>>          texinfo || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [PATCH 7/8] docker: Cross build QEMU on ppc32 hosts

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 06/29/2018 11:29 AM, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Since 08f56d8c9c the powerpc-cross image is no more based on Emdebian
>>> but on the Sid release.
>>>
>>> This patch restore the ability to cross build QEMU on powerpc, which
>>> is interesting since it is a 32-bit host.
>>>
>>> The ugly apt-fake script is, however, still required.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>
>>
>> Hmm I couldn't get this to work with:
>>
>>    make docker-test-build@debian-powerpc-cross J=9 V=1 NOCACHE=1
>>
>>
>> Step 9/13 : RUN apt-fake install         glusterfs-common:powerpc=4.0.2-fake
>>  ---> Running in 0bf7bf5e268a
>> Reading package lists...
>> Building dependency tree...
>> Reading state information...
>> E: Version '4.0.2-fake' for 'glusterfs-common:powerpc' was not found
>> Generating fake glusterfs-common powerpc 4.0.2-fake ...
>> Selecting previously unselected package glusterfs-common:powerpc.
>> (Reading database ... 19461 files and directories currently installed.)
>> Preparing to unpack glusterfs-common_4.0.2-fake_powerpc.deb ...
>> Unpacking glusterfs-common:powerpc (4.0.2-fake) ...
>> Setting up glusterfs-common:powerpc (4.0.2-fake) ...
>> Removing intermediate container 0bf7bf5e268a
>>  ---> 9f903850b104
>> Step 10/13 : RUN DEBIAN_FRONTEND=noninteractive eatmydata     apt-get build-dep -yy -a powerpc qemu
>>  ---> Running in 973b5c17668d
>> Reading package lists...
>> Building dependency tree...
>> Reading state information...
>> Some packages could not be installed. This may mean that you have
>> requested an impossible situation or if you are using the unstable
>> distribution that some required packages have not yet been created
>> or been moved out of Incoming.
>> The following information may help to resolve the situation:
>>
>> The following packages have unmet dependencies:
>>  builddeps:qemu:powerpc : Depends: libcapstone-dev:powerpc but it is not going to be installed
>>                           Depends: libaio-dev:powerpc but it is not going to be installed
>>                           Depends: libpulse-dev:powerpc but it is not going to be installed
>>                           Depends: libasound2-dev:powerpc but it is not going to be installed
>>                           Depends: libattr1-dev:powerpc but it is not going to be installed
>>                           Depends: libbluetooth-dev:powerpc but it is not going to be installed
>>                           Depends: libbrlapi-dev:powerpc but it is not going to be installed
>>                           Depends: libcap-dev:powerpc but it is not going to be installed
>>                           Depends: libcap-ng-dev:powerpc but it is not going to be installed
>>                           Depends: libcurl4-gnutls-dev:powerpc but it is not going to be installed
>>                           Depends: libfdt-dev:powerpc but it is not going to be installed
>>                           Depends: gnutls-dev:powerpc
>>                           Depends: libgtk-3-dev:powerpc but it is not going to be installed
>>                           Depends: libvte-2.91-dev:powerpc but it is not going to be installed
>>                           Depends: libiscsi-dev:powerpc (> 1.9.0~) but it is not going to be installed
>>                           Depends: libncursesw5-dev:powerpc
>>                           Depends: libnfs-dev:powerpc (> 1.9.3) but it is not going to be installed
>>                           Depends: libnuma-dev:powerpc but it is not going to be installed
>>                           Depends: libcacard-dev:powerpc but it is not going to be installed
>>                           Depends: libpixman-1-dev:powerpc but it is not going to be installed
>>                           Depends: librados-dev:powerpc but it is not going to be installed
>>                           Depends: librbd-dev:powerpc but it is not going to be installed
>>                           Depends: libsasl2-dev:powerpc but it is not going to be installed
>>                           Depends: libseccomp-dev:powerpc (>= 2.3.0) but it is not going to be installed
>>                           Depends: libusb-1.0-0-dev:powerpc (>= 2:1.0.13~) but it is not going to be installed
>>                           Depends: libusbredirparser-dev:powerpc (>= 0.6~) but it is not going to be installed
>>                           Depends: libssh2-1-dev:powerpc but it is not going to be installed
>>                           Depends: libvdeplug-dev:powerpc but it is not going to be installed
>>                           Depends: uuid-dev:powerpc but it is not going to be installed
>>                           Depends: xfslibs-dev:powerpc but it is not going to be installed
>>                           Depends: zlib1g-dev:powerpc but it is not going to be installed
>>                           Depends: libjpeg-dev:powerpc
>>                           Depends: libpng-dev:powerpc but it is not going to be installed
>> E: Unable to correct problems, you have held broken packages.
>> The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive eatmydata     apt-get build-dep -yy -a powerpc qemu' returned a non-zero code: 100
>> Traceback (most recent call last):
>>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 526, in <module>
>>     sys.exit(main())
>>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 523, in main
>>     return args.cmdobj.run(args, argv)
>>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 344, in run
>>     extra_files_cksum=cksum)
>>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 227, in build_image
>>     quiet=quiet)
>>   File "/home/alex/lsrc/qemu/qemu.git/tests/docker/docker.py", line 154, in _do_check
>>     return subprocess.check_call(self._command + cmd, **kwargs)
>>   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
>>     raise CalledProcessError(retcode, cmd)
>> subprocess.CalledProcessError: Command '['docker', 'build', '-t', 'qemu:debian-powerpc-cross', '-f', '/tmp/docker_buildjXN8x4/tmpJASaT1.docker', '--no-cache', '/tmp/docker_buildjXN8x4']' returned non-zero exit status 100
>> /home/alex/lsrc/qemu/qemu.git/tests/docker/Makefile.include:50: recipe for target 'docker-image-debian-powerpc-cross' failed
>> make: *** [docker-image-debian-powerpc-cross] Error 1
>
> Caveats of using Debian/Sid again :(
>
> It worked for me 5 days ago, and I'v been using this image since, but
> without rebuilding it, so did not notice the upstream packages change
> again... whew :(
>
> Let's postpone this for the next development window.

Yeah I think all we'll ever be doing is papering over the cracks with
this one. I think to support powerpc as a QEMU host we need to find a
stable host OS which actually supports it and cross compiling.

It's one thing hoping sid's gcc/libc stay functional but at soon as more
packages are involved it becomes a lot more dicey. Once gcc/libc bitrot
away for the platform then it is truly dead....

>
> Thanks for giving it a try!
>
> Phil.
>
>>
>>> ---
>>>  tests/docker/Makefile.include                 |  2 +
>>>  tests/docker/dockerfiles/debian-apt-fake.sh   | 46 +++++++++++++++++++
>>>  .../dockerfiles/debian-powerpc-cross.docker   | 43 +++++++++++++++++
>>>  3 files changed, 91 insertions(+)
>>>  create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>>>
>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>>> index 301ee68ae6..d73d60cfbb 100644
>>> --- a/tests/docker/Makefile.include
>>> +++ b/tests/docker/Makefile.include
>>> @@ -58,6 +58,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>>>  		"BUILD","$*")
>>>  endif
>>>
>>> +docker-image-debian-powerpc-cross: EXTRA_FILES:=$(SRC_PATH)/tests/docker/dockerfiles/debian-apt-fake.sh
>>> +
>>>  # Enforce dependencies for composite images
>>>  docker-image-debian: docker-image-debian9
>>>  docker-image-debian8-mxe: docker-image-debian8
>>> diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
>>> new file mode 100755
>>> index 0000000000..2ec0fdf47a
>>> --- /dev/null
>>> +++ b/tests/docker/dockerfiles/debian-apt-fake.sh
>>> @@ -0,0 +1,46 @@
>>> +#! /bin/sh
>>> +#
>>> +# Generate fake debian package to resolve unimportant unmet dependencies held
>>> +# by upstream multiarch broken packages.
>>> +#
>>> +# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> +#
>>> +# This work is licensed under the terms of the GNU GPL, version 2
>>> +# or (at your option) any later version. See the COPYING file in
>>> +# the top-level directory.
>>> +
>>> +test $1 = "install" && shift 1
>>> +
>>> +fake_install()
>>> +{
>>> +    echo "Generating fake $2 $1 $3 ..."
>>> +    (cd /var/cache/apt/archives
>>> +        (cat << 'EOF'
>>> +Section: misc
>>> +Priority: optional
>>> +Standards-Version: 3.9.2
>>> +
>>> +Package: NAME
>>> +Version: VERSION
>>> +Maintainer: qemu-devel@nongnu.org
>>> +Architecture: any
>>> +Multi-Arch: same
>>> +Description: fake NAME
>>> +EOF
>>> +        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
>>> +        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
>>> +        dpkg -i --force-overwrite $2_$3_$1.deb
>>> +    )
>>> +}
>>> +
>>> +try_install()
>>> +{
>>> +    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
>>> +    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
>>> +    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
>>> +    apt-get install -q -yy $1 || fake_install $arch $name $vers
>>> +}
>>> +
>>> +for package in $*; do
>>> +    try_install $package
>>> +done
>>> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>>> index 5e62ca0df1..4ccd17d9d5 100644
>>> --- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
>>> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>>> @@ -11,3 +11,46 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>>>      apt-get install -y --no-install-recommends \
>>>          gcc-powerpc-linux-gnu \
>>>          libc6-dev-powerpc-cross || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }
>>> +
>>> +# Setup some basic tools we need
>>> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>>> +    apt-get install -y --no-install-recommends \
>>> +        bison \
>>> +        binutils-multiarch \
>>> +        build-essential \
>>> +        ca-certificates \
>>> +        curl \
>>> +        flex \
>>> +        gettext \
>>> +        git \
>>> +        gnupg \
>>> +        pkg-config \
>>> +        python-minimal
>>> +
>>> +RUN dpkg --add-architecture powerpc && \
>>> +    apt-get update
>>> +
>>> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
>>> +
>>> +# Specify the cross prefix for this image (see tests/docker/common.rc)
>>> +ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
>>> +
>>> +# <kludge> to fix "following packages have unmet dependencies" ...
>>> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
>>> +RUN apt-get install -y --no-install-recommends \
>>> +        equivs
>>> +RUN apt-fake install \
>>> +        glusterfs-common:powerpc=4.0.2-fake
>>> +# </kludge>
>>> +
>>> +RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>>> +    apt-get build-dep -yy -a powerpc qemu
>>> +RUN DEBIAN_FRONTEND=noninteractive \
>>> +    apt-get install -y --no-install-recommends \
>>> +        glusterfs-common:powerpc \
>>> +        libbz2-dev:powerpc \
>>> +        liblzo2-dev:powerpc \
>>> +        libncursesw5-dev:powerpc \
>>> +        libnfs-dev:powerpc \
>>> +        librdmacm-dev:powerpc \
>>> +        libsnappy-dev:powerpc
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 5/8] docker: Restrict the 'travis' job to the Travis image

[Philippe Mathieu-Daudé]

On 06/29/2018 11:11 AM, Alex Bennée wrote:> Philippe Mathieu-Daudé
<f4bug@amsat.org> writes:
> 
>> We can still run any test in Travis.
> 
> 
> I don't know about this one. The travis job isn't a fill in for real
> travis, but a way to work through the matrix. I don't think it needs to
> be restricted to any particular image.

I had forgotten the context of this one.

IIRC, at some point the time expensive "make docker-test" try to run the
'travis' script in all docker images available, and failed with the
Debian ones. I now tested again with a single image and it did not fail.

As said Fam in the other reply to this tread, the script is protected by
the 'requires pyyaml' which isn't installed on the Debian images:

$ make docker-travis@debian-armhf-cross
  BUILD   debian9
  BUILD   debian-armhf-cross
  COPY    RUNNER
    RUN travis in qemu:debian-armhf-cross
Prerequisite 'pyyaml' not present, skip
$ echo $?
0

Anyway you are correct, this test isn't mean for a Travis builder but
for a developer to check what matrix will be run on Travis, so don't
need any image restriction.

(patch dropped).

Regards,

Phil.

>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/Makefile.include | 10 ++++++++--
>>  1 file changed, 8 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 91d9665517..7d9d568eee 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -99,11 +99,17 @@ docker-image-tricore-cross: docker-image-debian9
>>
>>  # Expand all the pre-requistes for each docker image and test combination
>>  $(foreach i,$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES), \
>> -	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
>> +	$(foreach t,$(DOCKER_TESTS), \
>>  		$(eval .PHONY: docker-$t@$i) \
>>  		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
>> +		$(eval docker-test: docker-$t@$i) \
>>  	) \
>> -	$(foreach t,$(DOCKER_TESTS), \
>> +)
>> +# we only run Travis tests on the Travis image
>> +$(foreach i,travis, \
>> +	$(foreach t,$(DOCKER_TOOLS), \
>> +		$(eval .PHONY: docker-$t@$i) \
>> +		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
>>  		$(eval docker-test: docker-$t@$i) \
>>  	) \
>>  )
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [PATCH 0/8] Docker improvements

[Philippe Mathieu-Daudé]

On 06/29/2018 11:37 AM, Alex Bennée wrote:
> I've queued the following in my tree.
> 
>>   docker: ubuntu: Update the package list before installing new ones
>>   docker: ubuntu: Use SDL2
>>   docker: Clean the MXE base image
>>   docker: Do not run tests in 'intermediate' images

Thanks!