[Qemu-devel] [PATCH v2 for 3.0 00/16] various docker fixes

[Qemu-devel] [PATCH v2 for 3.0 00/16] various docker fixes

[Alex Bennée]

Hi,

Unfortunately this series grew a little while I was re-basing as I
noticed a couple of bugs. An upstream change in debootstrap which I'd
been happily using for Ubuntu images caused a breakage which I've
hopefully now fixed:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903657

However since I upgraded by devbox I hadn't clocked to the fact I
should have been using the stable debootstrap anyway but our ugly
shell version check failed to pick it up. That's now fixed and I can
now bootstrap away.

I grabbed a couple of Philippe's patches while I was at it as they
were relevant to the other changes. It does mean the un-reviewed count
has gone up slightly:

  patch docker/base debian tricore on qemu debian9.patch
  patch docker/split configure_qemu from build_qemu.patch
  patch docker/move make check into check_qemu helper.patch
  patch docker/gracefully skip check_qemu.patch
  patch docker/Makefile.include don t include partial images.patch
  patch docker/disable debian powerpc user cross.patch
  patch docker/add test unit runner.patch
  patch docker/add expansion for docker test FOO to Makefile.patch
  patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
  patch docker/add hint to docker.py check.patch
  patch docker/add commentary to debian bootstrap.docker.patch
  patch docker/ignore distro versioning of debootstrap.patch


Alex Bennée (14):
  tests/.gitignore: don't ignore docker tests
  docker: base debian-tricore on qemu:debian9
  docker: fail more gracefully on docker.py check
  docker: split configure_qemu from build_qemu
  docker: move make check into check_qemu helper
  docker: gracefully skip check_qemu
  docker: Makefile.include don't include partial images
  docker: disable debian-powerpc-user-cross
  docker: add test-unit runner
  docker: add expansion for docker-test-FOO to Makefile.include
  docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set
  docker: add --hint to docker.py check
  docker: add commentary to debian-bootstrap.docker
  docker: ignore distro versioning of debootstrap

Philippe Mathieu-Daudé (2):
  shippable: Build the TriCore docker image
  docker: Update debootstrap script after Debian migration from Alioth
    to Salsa

 .shippable.yml                                |  2 ++
 tests/.gitignore                              |  1 +
 tests/docker/Makefile.include                 | 26 ++++++++++++-------
 tests/docker/common.rc                        | 23 +++++++++++++++-
 tests/docker/docker.py                        | 10 +++++--
 .../dockerfiles/debian-bootstrap.docker       |  1 +
 tests/docker/dockerfiles/debian-bootstrap.pre | 13 ++++++----
 .../dockerfiles/debian-tricore-cross.docker   |  2 +-
 tests/docker/test-clang                       |  2 +-
 tests/docker/test-debug                       |  2 +-
 tests/docker/test-full                        |  2 +-
 tests/docker/test-quick                       |  2 +-
 tests/docker/test-unit                        | 21 +++++++++++++++
 13 files changed, 85 insertions(+), 22 deletions(-)
 create mode 100755 tests/docker/test-unit

-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 01/16] tests/.gitignore: don't ignore docker tests

[Alex Bennée]

The .gitignore was being a little over enthusiastic hiding files.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

---
v2
  - minor wording tweak for more explicitness
---
 tests/.gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/.gitignore b/tests/.gitignore
index 08e2df1ce1..72c18aaab0 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -9,6 +9,7 @@ qht-bench
 rcutorture
 test-*
 !test-*.c
+!docker/test-*
 test-qapi-commands.[ch]
 test-qapi-events.[ch]
 test-qapi-types.[ch]
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 02/16] docker: base debian-tricore on qemu:debian9

[Alex Bennée]

We need both git and a working compiler to build the tools. Although
the qemu:debian9 image also has a bunch of extra dependencies it would
be fairly unusual for a user not to already have this layer available
for one of our many other docker images so lets not complicate things.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-tricore-cross.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 898b8dd511..180ca646c8 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -7,7 +7,7 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
-FROM debian:9
+FROM qemu:debian9
 
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 03/16] shippable: Build the TriCore docker image

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index f74a3de3ff..9670b13f8e 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -25,6 +25,8 @@ env:
       TARGET_LIST=mips64el-softmmu,mips64el-linux-user
     - IMAGE=debian-ppc64el-cross
       TARGET_LIST=ppc64-softmmu,ppc64-linux-user,ppc64abi32-linux-user
+    - IMAGE=debian-tricore-cross
+      TARGET_LIST=
 build:
   pre_ci:
     - make docker-image-${IMAGE} V=1
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 04/16] docker: fail more gracefully on docker.py check

[Alex Bennée]

As this is called directly from the Makefile while determining
dependencies and it is possible the user was configured in one window
but not have credentials in the other. Let's catch the Exceptions and
deal with it quietly.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/docker.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 69e7130db7..2f81c6b13b 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -479,7 +479,12 @@ class CheckCommand(SubCommand):
     def run(self, args, argv):
         tag = args.tag
 
-        dkr = Docker()
+        try:
+            dkr = Docker()
+        except:
+            print("Docker not set up")
+            return 1
+
         info = dkr.inspect_tag(tag)
         if info is None:
             print("Image does not exist")
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 05/16] docker: split configure_qemu from build_qemu

[Alex Bennée]

This allows some tests that just want to configure QEMU's source tree
to do so.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/common.rc | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index 046f8a5921..ba1f942328 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -21,7 +21,7 @@ requires()
     done
 }
 
-build_qemu()
+configure_qemu()
 {
     config_opts="--enable-werror \
                  ${TARGET_LIST:+--target-list=${TARGET_LIST}} \
@@ -32,6 +32,11 @@ build_qemu()
     echo $config_opts
     $QEMU_SRC/configure $config_opts || \
         { cat config.log && test_fail "Failed to run 'configure'"; }
+}
+
+build_qemu()
+{
+    configure_qemu $@
     make $MAKEFLAGS
 }
 
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 06/16] docker: move make check into check_qemu helper

[Alex Bennée]

Not all docker images can run the check step. Let's move everything
into a common helper so we don't need to replicate checks in the
future.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - add "move" to Let's everything in commit message
---
 tests/docker/common.rc  | 10 ++++++++++
 tests/docker/test-clang |  2 +-
 tests/docker/test-debug |  2 +-
 tests/docker/test-full  |  2 +-
 tests/docker/test-quick |  2 +-
 5 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index ba1f942328..cfc620d554 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -40,6 +40,16 @@ build_qemu()
     make $MAKEFLAGS
 }
 
+check_qemu()
+{
+    if test -z "$@"; then
+        TEST="check"
+    else
+        TEST="$@"
+    fi
+    make $MAKEFLAGS $TEST
+}
+
 test_fail()
 {
     echo "$@"
diff --git a/tests/docker/test-clang b/tests/docker/test-clang
index e90a793178..324e341cea 100755
--- a/tests/docker/test-clang
+++ b/tests/docker/test-clang
@@ -23,5 +23,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
 #OPTS="$OPTS --extra-cflags=-fsanitize=undefined \
     #--extra-cflags=-fno-sanitize=float-divide-by-zero"
 build_qemu $OPTS
-make $MAKEFLAGS check
+check_qemu
 install_qemu
diff --git a/tests/docker/test-debug b/tests/docker/test-debug
index d3f9f70d01..137f4f2ddc 100755
--- a/tests/docker/test-debug
+++ b/tests/docker/test-debug
@@ -22,5 +22,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
 OPTS="--enable-debug --enable-sanitizers $OPTS"
 
 build_qemu $OPTS
-make $MAKEFLAGS V=1 check
+check_qemu check V=1
 install_qemu
diff --git a/tests/docker/test-full b/tests/docker/test-full
index b4e42d25d7..aadc0f00a2 100755
--- a/tests/docker/test-full
+++ b/tests/docker/test-full
@@ -15,4 +15,4 @@
 
 cd "$BUILD_DIR"
 
-build_qemu && make check $MAKEFLAGS && install_qemu
+build_qemu && check_qemu && install_qemu
diff --git a/tests/docker/test-quick b/tests/docker/test-quick
index 3b7bce6105..eee59c55fb 100755
--- a/tests/docker/test-quick
+++ b/tests/docker/test-quick
@@ -18,5 +18,5 @@ cd "$BUILD_DIR"
 DEF_TARGET_LIST="x86_64-softmmu,aarch64-softmmu"
 TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
 build_qemu
-make check $MAKEFLAGS
+check_qemu
 install_qemu
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 07/16] docker: gracefully skip check_qemu

[Alex Bennée]

Not all our images are able to run the tests. Rather than use features
we can just check for the existence and run-ability of gtester. If the
image has been setup for binfmt_misc it will be able to run anyway.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/common.rc | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index cfc620d554..6df431eb72 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -47,7 +47,13 @@ check_qemu()
     else
         TEST="$@"
     fi
-    make $MAKEFLAGS $TEST
+
+    if type gtester > /dev/null 2>&1 && \
+           gtester --version > /dev/null 2>&1; then
+        make $MAKEFLAGS $TEST
+    else
+        echo "No working gtester, skipping make $TEST"
+    fi
 }
 
 test_fail()
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 08/16] docker: Makefile.include don't include partial images

[Alex Bennée]

Rename DOCKER_INTERMEDIATE_IMAGES to DOCKER_PARTIAL_IMAGES and add the
incomplete cross compiler images that can build tests but can't build
QEMU itself. We also add debian, debian-bootstrap and the tricode
images to the list.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index b2a7e761cc..09fb7db7fa 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -6,7 +6,7 @@ DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 DOCKER_DEPRECATED_IMAGES := debian
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_INTERMEDIATE_IMAGES := debian8 debian9 debian8-mxe debian-ports debian-sid
+DOCKER_PARTIAL_IMAGES := debian debian8 debian9 debian8-mxe debian-ports debian-sid debian-bootstrap
 DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
@@ -121,6 +121,11 @@ docker-image-travis: NOUSER=1
 # Specialist build images, sometimes very limited tools
 docker-image-tricore-cross: docker-image-debian9
 
+# These images may be good enough for building tests but not for test builds
+DOCKER_PARTIAL_IMAGES += debian-alpha-cross debian-hppa-cross debian-m68k-cross debian-sh4-cross
+DOCKER_PARTIAL_IMAGES += debian-sparc64-cross debian-mips64-cross debian-riscv64-cross
+DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-cross
+
 # Rules for building linux-user powered images
 #
 # These are slower than using native cross compiler setups but can
@@ -137,7 +142,7 @@ docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
 
 # Expand all the pre-requistes for each docker image and test combination
-$(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
+$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
 		$(eval .PHONY: docker-$t@$i) \
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 09/16] docker: disable debian-powerpc-user-cross

[Alex Bennée]

While we can build we can't run the unit tests due to a missing system
call (249 - swapcontext). This could be a code generation issue as
other architectures don't seem to have this system call.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Alexander Graf <agraf@suse.de>
Cc: qemu-ppc@nongnu.org
---
 tests/docker/Makefile.include | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 09fb7db7fa..fe63aacf69 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -140,6 +140,9 @@ docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
 docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
 docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
+# Unfortunalty while we can build with this image we can't run unit
+# tests due to missing system call: 249 which causes an abort
+DOCKER_PARTIAL_IMAGES += debian-powerpc-user-cross
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 10/16] docker: add test-unit runner

[Alex Bennée]

This test doesn't even build QEMU, it just builds and runs all the
unit tests. Intended to make checking unit tests on all docker images
easier.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - make script -e to error out on failure
  - add comment as to why the configure step is run
---
 tests/docker/test-unit | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100755 tests/docker/test-unit

diff --git a/tests/docker/test-unit b/tests/docker/test-unit
new file mode 100755
index 0000000000..8905d01150
--- /dev/null
+++ b/tests/docker/test-unit
@@ -0,0 +1,21 @@
+#!/bin/bash -e
+#
+# Build and run the unit tests
+#
+# Copyright (c) 2018 Linaro Ltd.
+#
+# Authors:
+#  Alex Bennée <alex.bennee@linaro.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+cd "$BUILD_DIR"
+
+# although we are not building QEMU itself we still need a configured
+# build for the unit tests to be built and run
+configure_qemu
+check_qemu check-unit
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 11/16] docker: add expansion for docker-test-FOO to Makefile.include

[Alex Bennée]

This allows us to run a particular test on all docker images. For
example:

  make docker-test-unit

Will run the unit tests on every supported image. At the same time
rename docker-test to docker-all-tests to be clearer.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - docker-test -> docker-all-tests
---
 tests/docker/Makefile.include | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index fe63aacf69..e32c35be0d 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -151,7 +151,8 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
 	) \
 	$(foreach t,$(DOCKER_TESTS), \
-		$(eval docker-test: docker-$t@$i) \
+		$(eval docker-all-tests: docker-$t@$i) \
+		$(eval docker-$t: docker-$t@$i) \
 	) \
 )
 
@@ -161,7 +162,8 @@ docker:
 	@echo 'Available targets:'
 	@echo
 	@echo '    docker:              Print this help.'
-	@echo '    docker-test:         Run all image/test combinations.'
+	@echo '    docker-all-tests:    Run all image/test combinations.'
+	@echo '    docker-TEST:         Run TEST on all image combinations.'
 	@echo '    docker-clean:        Kill and remove residual docker testing containers.'
 	@echo '    docker-TEST@IMAGE:   Run "TEST" in container "IMAGE".'
 	@echo '                         Note: "TEST" is one of the listed test name,'
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 12/16] docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set

[Alex Bennée]

The addition of QEMU_TARGET was intended to ensure we fall back to
checking for the existence of an image if the build system was not
currently configured to build it. However this breaks the direct use
of the rule for building custom binfmt_misc images. We already check
for EXECUTABLE so let us just use that as a proxy for deciding if we
are just going to check the image exits.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index e32c35be0d..ec23620153 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -58,13 +58,11 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 	$(if $(EXECUTABLE),,\
 		$(error EXECUTABLE not set, debootstrap of debian-$* would fail))
-	$(if $(wildcard $(EXECUTABLE)),,\
-		$(error Please build $(EXECUTABLE) first))
 	$(if $(DEB_ARCH),,\
 		$(error DEB_ARCH not set, debootstrap of debian-$* would fail))
 	$(if $(DEB_TYPE),,\
 		$(error DEB_TYPE not set, debootstrap of debian-$* would fail))
-	$(if $(filter $(QEMU_TARGET),$(TARGET_DIRS)),				\
+	$(if $(wildcard $(EXECUTABLE)),						\
 		$(call quiet-command,						\
 			DEB_ARCH=$(DEB_ARCH)					\
 			DEB_TYPE=$(DEB_TYPE)					\
@@ -136,7 +134,6 @@ DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-c
 # broken so we need a qemu-linux-user for this target
 docker-binfmt-image-debian-powerpc-user: DEB_ARCH = powerpc
 docker-binfmt-image-debian-powerpc-user: DEB_TYPE = jessie
-docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
 docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
 docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 13/16] docker: add --hint to docker.py check

[Alex Bennée]

When a check fails we currently just report why we failed. This is not
totally helpful to people who want to boot-strap a new image. Add a
--hint option which we can pass down to give a bit more information.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 3 ++-
 tests/docker/docker.py        | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index ec23620153..c9e412f9d0 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -73,7 +73,8 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 			$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
 			"BUILD","binfmt debian-$* (debootstrapped)"),		\
 		$(call quiet-command,						\
-			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<,	\
+			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<	\
+			--hint "you will need to build $(EXECUTABLE)",		\
 			"CHECK", "debian-$* exists"))
 
 endif
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 2f81c6b13b..523f4b95a2 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -475,6 +475,7 @@ class CheckCommand(SubCommand):
                             default="checksum", help="check type")
         parser.add_argument("--olderthan", default=60, type=int,
                             help="number of minutes")
+        parser.add_argument("--hint", default="", help="hint to user")
 
     def run(self, args, argv):
         tag = args.tag
@@ -487,7 +488,7 @@ class CheckCommand(SubCommand):
 
         info = dkr.inspect_tag(tag)
         if info is None:
-            print("Image does not exist")
+            print("Image does not exist %s" % (args.hint))
             return 1
 
         if args.checktype == "checksum":
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 14/16] docker: Update debootstrap script after Debian migration from Alioth to Salsa

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

This silents the following warning:

  Cloning into './debootstrap.git'...
  warning: redirecting to https://salsa.debian.org/installer-team/debootstrap.git/

See https://lists.debian.org/debian-devel-announce/2018/01/msg00004.html

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-bootstrap.pre | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index 56e1aa7a21..ea324d6e4a 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -62,7 +62,7 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
         NEED_DEBOOTSTRAP=true
     fi
     if $NEED_DEBOOTSTRAP; then
-        DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
+        DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
         git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
         export DEBOOTSTRAP_DIR=./debootstrap.git
         DEBOOTSTRAP=./debootstrap.git/debootstrap
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 15/16] docker: add commentary to debian-bootstrap.docker

[Alex Bennée]

This is just a note that later versions of debootstrap don't
technically need this hack.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-bootstrap.docker | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
index 14212b9cf4..e13c26a7ed 100644
--- a/tests/docker/dockerfiles/debian-bootstrap.docker
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -9,6 +9,7 @@ FROM scratch
 ADD . /
 
 # Patch all mounts as docker already has stuff set up
+# (this is not needed for later debootstraps but is harmless atm)
 RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
 
 # Run stage 2
-- 
2.17.1

[Qemu-devel] [PATCH v2 for 3.0 16/16] docker: ignore distro versioning of debootstrap

[Alex Bennée]

We do a minimum version check for the debootstrap but if the distro
has added their own minor version tick it would fail and fall-back to
the SCM version. This is sub-optimal as the latest/greatest version
may be broken at any one particular time. We fix that with a little
sed magic on the version string before passing to our ugly shell
versioning check.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-bootstrap.pre | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index ea324d6e4a..3b0ef95374 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -56,10 +56,13 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
     if [ -z $DEBOOTSTRAP ]; then
         echo "No debootstrap installed, attempting to install from SCM"
         NEED_DEBOOTSTRAP=true
-    elif ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; "${DEBOOTSTRAP}" --version \
-            | cut -d ' ' -f 2) | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -c &>/dev/null; then
-        echo "debootstrap too old, attempting to install from SCM"
-        NEED_DEBOOTSTRAP=true
+    else
+        INSTALLED_VERSION=$(${DEBOOTSTRAP} --version | sed 's/debootstrap \([0-9\.]*\)[^0-9\.]*.*/\1/')
+        if ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; echo "${INSTALLED_VERSION}") \
+                | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -C ; then
+            echo "debootstrap too old, attempting to install from SCM"
+            NEED_DEBOOTSTRAP=true
+        fi
     fi
     if $NEED_DEBOOTSTRAP; then
         DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
-- 
2.17.1

Re: [Qemu-devel] [PATCH v2 for 3.0 03/16] shippable: Build the TriCore docker image

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  .shippable.yml | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index f74a3de3ff..9670b13f8e 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -25,6 +25,8 @@ env:
>        TARGET_LIST=mips64el-softmmu,mips64el-linux-user
>      - IMAGE=debian-ppc64el-cross
>        TARGET_LIST=ppc64-softmmu,ppc64-linux-user,ppc64abi32-linux-user
> +    - IMAGE=debian-tricore-cross
> +      TARGET_LIST=
>  build:
>    pre_ci:
>      - make docker-image-${IMAGE} V=1

Serves me right for not waiting until the shippable build finished. This
needs to be dropped as while we can build the image we can't use it to
build anything.

--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 for 3.0 11/16] docker: add expansion for docker-test-FOO to Makefile.include

[Philippe Mathieu-Daudé]

Hi Alex,

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> This allows us to run a particular test on all docker images. For
> example:
> 
>   make docker-test-unit
> 
> Will run the unit tests on every supported image. At the same time
> rename docker-test to docker-all-tests to be clearer.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v2
>   - docker-test -> docker-all-tests
> ---
>  tests/docker/Makefile.include | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index fe63aacf69..e32c35be0d 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -151,7 +151,8 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
>  		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
>  	) \
>  	$(foreach t,$(DOCKER_TESTS), \
> -		$(eval docker-test: docker-$t@$i) \
> +		$(eval docker-all-tests: docker-$t@$i) \
> +		$(eval docker-$t: docker-$t@$i) \

Is this supposed to work this way?

$ make docker-test-quick@debian-alpha-cross
make: *** No rule to make target 'docker-test-quick@debian-alpha-cross'.
 Stop.

>  	) \
>  )
>  
> @@ -161,7 +162,8 @@ docker:
>  	@echo 'Available targets:'
>  	@echo
>  	@echo '    docker:              Print this help.'
> -	@echo '    docker-test:         Run all image/test combinations.'
> +	@echo '    docker-all-tests:    Run all image/test combinations.'
> +	@echo '    docker-TEST:         Run TEST on all image combinations.'
>  	@echo '    docker-clean:        Kill and remove residual docker testing containers.'
>  	@echo '    docker-TEST@IMAGE:   Run "TEST" in container "IMAGE".'
>  	@echo '                         Note: "TEST" is one of the listed test name,'
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 11/16] docker: add expansion for docker-test-FOO to Makefile.include

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi Alex,
>
> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>> This allows us to run a particular test on all docker images. For
>> example:
>>
>>   make docker-test-unit
>>
>> Will run the unit tests on every supported image. At the same time
>> rename docker-test to docker-all-tests to be clearer.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>
>> ---
>> v2
>>   - docker-test -> docker-all-tests
>> ---
>>  tests/docker/Makefile.include | 6 ++++--
>>  1 file changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index fe63aacf69..e32c35be0d 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -151,7 +151,8 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
>>  		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
>>  	) \
>>  	$(foreach t,$(DOCKER_TESTS), \
>> -		$(eval docker-test: docker-$t@$i) \
>> +		$(eval docker-all-tests: docker-$t@$i) \
>> +		$(eval docker-$t: docker-$t@$i) \
>
> Is this supposed to work this way?
>
> $ make docker-test-quick@debian-alpha-cross
> make: *** No rule to make target
> 'docker-test-quick@debian-alpha-cross'.

No

  make docker-test-quick

will make the test-quick on all images.

Your example fails because debian-alpha-cross is a PARTIAL image, good
for building test cases but not QEMU so it's not expanded in the above
bit: $(eval docker-$t@$i: docker-image-$i docker-run-$t@$i)

>  Stop.
>
>>  	) \
>>  )
>>
>> @@ -161,7 +162,8 @@ docker:
>>  	@echo 'Available targets:'
>>  	@echo
>>  	@echo '    docker:              Print this help.'
>> -	@echo '    docker-test:         Run all image/test combinations.'
>> +	@echo '    docker-all-tests:    Run all image/test combinations.'
>> +	@echo '    docker-TEST:         Run TEST on all image combinations.'
>>  	@echo '    docker-clean:        Kill and remove residual docker testing containers.'
>>  	@echo '    docker-TEST@IMAGE:   Run "TEST" in container "IMAGE".'
>>  	@echo '                         Note: "TEST" is one of the listed test name,'
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 for 3.0 05/16] docker: split configure_qemu from build_qemu

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> This allows some tests that just want to configure QEMU's source tree
> to do so.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  tests/docker/common.rc | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
> index 046f8a5921..ba1f942328 100755
> --- a/tests/docker/common.rc
> +++ b/tests/docker/common.rc
> @@ -21,7 +21,7 @@ requires()
>      done
>  }
>  
> -build_qemu()
> +configure_qemu()
>  {
>      config_opts="--enable-werror \
>                   ${TARGET_LIST:+--target-list=${TARGET_LIST}} \
> @@ -32,6 +32,11 @@ build_qemu()
>      echo $config_opts
>      $QEMU_SRC/configure $config_opts || \
>          { cat config.log && test_fail "Failed to run 'configure'"; }
> +}
> +
> +build_qemu()
> +{
> +    configure_qemu $@
>      make $MAKEFLAGS
>  }
>  
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 03/16] shippable: Build the TriCore docker image

[Philippe Mathieu-Daudé]

On 07/13/2018 10:27 AM, Alex Bennée wrote:
> 
> Alex Bennée <alex.bennee@linaro.org> writes:
> 
>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  .shippable.yml | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index f74a3de3ff..9670b13f8e 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -25,6 +25,8 @@ env:
>>        TARGET_LIST=mips64el-softmmu,mips64el-linux-user
>>      - IMAGE=debian-ppc64el-cross
>>        TARGET_LIST=ppc64-softmmu,ppc64-linux-user,ppc64abi32-linux-user
>> +    - IMAGE=debian-tricore-cross
>> +      TARGET_LIST=
>>  build:
>>    pre_ci:
>>      - make docker-image-${IMAGE} V=1
> 
> Serves me right for not waiting until the shippable build finished. This
> needs to be dropped as while we can build the image we can't use it to
> build anything.

It was tagged 'RFC' ;)

Re: [Qemu-devel] [PATCH v2 for 3.0 06/16] docker: move make check into check_qemu helper

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> Not all docker images can run the check step. Let's move everything
> into a common helper so we don't need to replicate checks in the
> future.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v2
>   - add "move" to Let's everything in commit message
> ---
>  tests/docker/common.rc  | 10 ++++++++++
>  tests/docker/test-clang |  2 +-
>  tests/docker/test-debug |  2 +-
>  tests/docker/test-full  |  2 +-
>  tests/docker/test-quick |  2 +-
>  5 files changed, 14 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
> index ba1f942328..cfc620d554 100755
> --- a/tests/docker/common.rc
> +++ b/tests/docker/common.rc
> @@ -40,6 +40,16 @@ build_qemu()
>      make $MAKEFLAGS
>  }
>  
> +check_qemu()
> +{
> +    if test -z "$@"; then
> +        TEST="check"
> +    else
> +        TEST="$@"

This is a bit confuse:

  check_qemu V=1 -> TEST=V=1

> +    fi
> +    make $MAKEFLAGS $TEST

    -> make V=1

> +}
> +
>  test_fail()
>  {
>      echo "$@"
> diff --git a/tests/docker/test-clang b/tests/docker/test-clang
> index e90a793178..324e341cea 100755
> --- a/tests/docker/test-clang
> +++ b/tests/docker/test-clang
> @@ -23,5 +23,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
>  #OPTS="$OPTS --extra-cflags=-fsanitize=undefined \
>      #--extra-cflags=-fno-sanitize=float-divide-by-zero"
>  build_qemu $OPTS
> -make $MAKEFLAGS check
> +check_qemu
>  install_qemu
> diff --git a/tests/docker/test-debug b/tests/docker/test-debug
> index d3f9f70d01..137f4f2ddc 100755
> --- a/tests/docker/test-debug
> +++ b/tests/docker/test-debug
> @@ -22,5 +22,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
>  OPTS="--enable-debug --enable-sanitizers $OPTS"
>  
>  build_qemu $OPTS
> -make $MAKEFLAGS V=1 check
> +check_qemu check V=1

Gotcha!

>  install_qemu
> diff --git a/tests/docker/test-full b/tests/docker/test-full
> index b4e42d25d7..aadc0f00a2 100755
> --- a/tests/docker/test-full
> +++ b/tests/docker/test-full
> @@ -15,4 +15,4 @@
>  
>  cd "$BUILD_DIR"
>  
> -build_qemu && make check $MAKEFLAGS && install_qemu
> +build_qemu && check_qemu && install_qemu
> diff --git a/tests/docker/test-quick b/tests/docker/test-quick
> index 3b7bce6105..eee59c55fb 100755
> --- a/tests/docker/test-quick
> +++ b/tests/docker/test-quick
> @@ -18,5 +18,5 @@ cd "$BUILD_DIR"
>  DEF_TARGET_LIST="x86_64-softmmu,aarch64-softmmu"
>  TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
>  build_qemu
> -make check $MAKEFLAGS
> +check_qemu
>  install_qemu
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 07/16] docker: gracefully skip check_qemu

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> Not all our images are able to run the tests. Rather than use features
> we can just check for the existence and run-ability of gtester. If the
> image has been setup for binfmt_misc it will be able to run anyway.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/common.rc | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
> index cfc620d554..6df431eb72 100755
> --- a/tests/docker/common.rc
> +++ b/tests/docker/common.rc
> @@ -47,7 +47,13 @@ check_qemu()
>      else
>          TEST="$@"
>      fi
> -    make $MAKEFLAGS $TEST
> +
> +    if type gtester > /dev/null 2>&1 && \

commit e465ce7d09939d631f1861e0bd8873417c1c0d65

    tests: Use "command -v" instead of which(1) in shell scripts

    When which(1) is not installed, we would complain "perl not found"
    because it's the first set_prog_path check. The error message is
    wrong.

    Fix it by using "command -v", a native way to query the existence
    of a command.

> +           gtester --version > /dev/null 2>&1; then
> +        make $MAKEFLAGS $TEST
> +    else
> +        echo "No working gtester, skipping make $TEST"
> +    fi
>  }
>  
>  test_fail()
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 08/16] docker: Makefile.include don't include partial images

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> Rename DOCKER_INTERMEDIATE_IMAGES to DOCKER_PARTIAL_IMAGES and add the
> incomplete cross compiler images that can build tests but can't build
> QEMU itself. We also add debian, debian-bootstrap and the tricode
> images to the list.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/Makefile.include | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index b2a7e761cc..09fb7db7fa 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -6,7 +6,7 @@ DOCKER_SUFFIX := .docker
>  DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
>  DOCKER_DEPRECATED_IMAGES := debian
>  # we don't run tests on intermediate images (used as base by another image)
> -DOCKER_INTERMEDIATE_IMAGES := debian8 debian9 debian8-mxe debian-ports debian-sid
> +DOCKER_PARTIAL_IMAGES := debian debian8 debian9 debian8-mxe debian-ports debian-sid debian-bootstrap
>  DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
>  DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
>  # Use a global constant ccache directory to speed up repetitive builds
> @@ -121,6 +121,11 @@ docker-image-travis: NOUSER=1
>  # Specialist build images, sometimes very limited tools
>  docker-image-tricore-cross: docker-image-debian9
>  
> +# These images may be good enough for building tests but not for test builds
> +DOCKER_PARTIAL_IMAGES += debian-alpha-cross debian-hppa-cross debian-m68k-cross debian-sh4-cross
> +DOCKER_PARTIAL_IMAGES += debian-sparc64-cross debian-mips64-cross debian-riscv64-cross
> +DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-cross

What about inverting the logic? Only keep track of 'full' images, and
use the $(filter ) function.

Anyway:
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> +
>  # Rules for building linux-user powered images
>  #
>  # These are slower than using native cross compiler setups but can
> @@ -137,7 +142,7 @@ docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
>  DOCKER_USER_IMAGES += debian-powerpc-user
>  
>  # Expand all the pre-requistes for each docker image and test combination
> -$(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
> +$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
>  	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
>  		$(eval .PHONY: docker-$t@$i) \
>  		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 13/16] docker: add --hint to docker.py check

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> When a check fails we currently just report why we failed. This is not
> totally helpful to people who want to boot-strap a new image. Add a
> --hint option which we can pass down to give a bit more information.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  tests/docker/Makefile.include | 3 ++-
>  tests/docker/docker.py        | 3 ++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index ec23620153..c9e412f9d0 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -73,7 +73,8 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
>  			$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
>  			"BUILD","binfmt debian-$* (debootstrapped)"),		\
>  		$(call quiet-command,						\
> -			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<,	\
> +			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<	\
> +			--hint "you will need to build $(EXECUTABLE)",		\
>  			"CHECK", "debian-$* exists"))
>  
>  endif
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 2f81c6b13b..523f4b95a2 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -475,6 +475,7 @@ class CheckCommand(SubCommand):
>                              default="checksum", help="check type")
>          parser.add_argument("--olderthan", default=60, type=int,
>                              help="number of minutes")
> +        parser.add_argument("--hint", default="", help="hint to user")
>  
>      def run(self, args, argv):
>          tag = args.tag
> @@ -487,7 +488,7 @@ class CheckCommand(SubCommand):
>  
>          info = dkr.inspect_tag(tag)
>          if info is None:
> -            print("Image does not exist")
> +            print("Image does not exist %s" % (args.hint))
>              return 1
>  
>          if args.checktype == "checksum":
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 10/16] docker: add test-unit runner

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> This test doesn't even build QEMU, it just builds and runs all the
> unit tests. Intended to make checking unit tests on all docker images
> easier.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> 
> ---
> v2
>   - make script -e to error out on failure
>   - add comment as to why the configure step is run
> ---
>  tests/docker/test-unit | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
>  create mode 100755 tests/docker/test-unit
> 
> diff --git a/tests/docker/test-unit b/tests/docker/test-unit
> new file mode 100755
> index 0000000000..8905d01150
> --- /dev/null
> +++ b/tests/docker/test-unit
> @@ -0,0 +1,21 @@
> +#!/bin/bash -e
> +#
> +# Build and run the unit tests
> +#
> +# Copyright (c) 2018 Linaro Ltd.
> +#
> +# Authors:
> +#  Alex Bennée <alex.bennee@linaro.org>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +. common.rc
> +
> +cd "$BUILD_DIR"
> +
> +# although we are not building QEMU itself we still need a configured
> +# build for the unit tests to be built and run
> +configure_qemu
> +check_qemu check-unit
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 06/16] docker: move make check into check_qemu helper

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>> Not all docker images can run the check step. Let's move everything
>> into a common helper so we don't need to replicate checks in the
>> future.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>
>> ---
>> v2
>>   - add "move" to Let's everything in commit message
>> ---
>>  tests/docker/common.rc  | 10 ++++++++++
>>  tests/docker/test-clang |  2 +-
>>  tests/docker/test-debug |  2 +-
>>  tests/docker/test-full  |  2 +-
>>  tests/docker/test-quick |  2 +-
>>  5 files changed, 14 insertions(+), 4 deletions(-)
>>
>> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
>> index ba1f942328..cfc620d554 100755
>> --- a/tests/docker/common.rc
>> +++ b/tests/docker/common.rc
>> @@ -40,6 +40,16 @@ build_qemu()
>>      make $MAKEFLAGS
>>  }
>>
>> +check_qemu()
>> +{
>> +    if test -z "$@"; then
>> +        TEST="check"
>> +    else
>> +        TEST="$@"
>
> This is a bit confuse:
>
>   check_qemu V=1 -> TEST=V=1
>
>> +    fi
>> +    make $MAKEFLAGS $TEST
>
>     -> make V=1

How about s/TEST/INVOCATION/?

>
>> +}
>> +
>>  test_fail()
>>  {
>>      echo "$@"
>> diff --git a/tests/docker/test-clang b/tests/docker/test-clang
>> index e90a793178..324e341cea 100755
>> --- a/tests/docker/test-clang
>> +++ b/tests/docker/test-clang
>> @@ -23,5 +23,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
>>  #OPTS="$OPTS --extra-cflags=-fsanitize=undefined \
>>      #--extra-cflags=-fno-sanitize=float-divide-by-zero"
>>  build_qemu $OPTS
>> -make $MAKEFLAGS check
>> +check_qemu
>>  install_qemu
>> diff --git a/tests/docker/test-debug b/tests/docker/test-debug
>> index d3f9f70d01..137f4f2ddc 100755
>> --- a/tests/docker/test-debug
>> +++ b/tests/docker/test-debug
>> @@ -22,5 +22,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
>>  OPTS="--enable-debug --enable-sanitizers $OPTS"
>>
>>  build_qemu $OPTS
>> -make $MAKEFLAGS V=1 check
>> +check_qemu check V=1
>
> Gotcha!
>
>>  install_qemu
>> diff --git a/tests/docker/test-full b/tests/docker/test-full
>> index b4e42d25d7..aadc0f00a2 100755
>> --- a/tests/docker/test-full
>> +++ b/tests/docker/test-full
>> @@ -15,4 +15,4 @@
>>
>>  cd "$BUILD_DIR"
>>
>> -build_qemu && make check $MAKEFLAGS && install_qemu
>> +build_qemu && check_qemu && install_qemu
>> diff --git a/tests/docker/test-quick b/tests/docker/test-quick
>> index 3b7bce6105..eee59c55fb 100755
>> --- a/tests/docker/test-quick
>> +++ b/tests/docker/test-quick
>> @@ -18,5 +18,5 @@ cd "$BUILD_DIR"
>>  DEF_TARGET_LIST="x86_64-softmmu,aarch64-softmmu"
>>  TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
>>  build_qemu
>> -make check $MAKEFLAGS
>> +check_qemu
>>  install_qemu
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 for 3.0 07/16] docker: gracefully skip check_qemu

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>> Not all our images are able to run the tests. Rather than use features
>> we can just check for the existence and run-ability of gtester. If the
>> image has been setup for binfmt_misc it will be able to run anyway.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  tests/docker/common.rc | 8 +++++++-
>>  1 file changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
>> index cfc620d554..6df431eb72 100755
>> --- a/tests/docker/common.rc
>> +++ b/tests/docker/common.rc
>> @@ -47,7 +47,13 @@ check_qemu()
>>      else
>>          TEST="$@"
>>      fi
>> -    make $MAKEFLAGS $TEST
>> +
>> +    if type gtester > /dev/null 2>&1 && \
>
> commit e465ce7d09939d631f1861e0bd8873417c1c0d65
>
>     tests: Use "command -v" instead of which(1) in shell scripts

type is also a shell built-in albeit an 1982 bashism. I'll switch to
command -v as tending towards POSIX is what we should aim for (although
our test-FOO/common.rc structure seems a little confused).

>
>     When which(1) is not installed, we would complain "perl not found"
>     because it's the first set_prog_path check. The error message is
>     wrong.
>
>     Fix it by using "command -v", a native way to query the existence
>     of a command.
>
>> +           gtester --version > /dev/null 2>&1; then
>> +        make $MAKEFLAGS $TEST
>> +    else
>> +        echo "No working gtester, skipping make $TEST"
>> +    fi
>>  }
>>
>>  test_fail()
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 for 3.0 02/16] docker: base debian-tricore on qemu:debian9

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> We need both git and a working compiler to build the tools. Although
> the qemu:debian9 image also has a bunch of extra dependencies it would
> be fairly unusual for a user not to already have this layer available
> for one of our many other docker images so lets not complicate things.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  tests/docker/dockerfiles/debian-tricore-cross.docker | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
> index 898b8dd511..180ca646c8 100644
> --- a/tests/docker/dockerfiles/debian-tricore-cross.docker
> +++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
> @@ -7,7 +7,7 @@
>  #
>  # SPDX-License-Identifier: GPL-2.0-or-later
>  #
> -FROM debian:9
> +FROM qemu:debian9
>  
>  MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
>  
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 06/16] docker: move make check into check_qemu helper

[Philippe Mathieu-Daudé]

On 07/17/2018 06:59 AM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>>> Not all docker images can run the check step. Let's move everything
>>> into a common helper so we don't need to replicate checks in the
>>> future.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>>   - add "move" to Let's everything in commit message
>>> ---
>>>  tests/docker/common.rc  | 10 ++++++++++
>>>  tests/docker/test-clang |  2 +-
>>>  tests/docker/test-debug |  2 +-
>>>  tests/docker/test-full  |  2 +-
>>>  tests/docker/test-quick |  2 +-
>>>  5 files changed, 14 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
>>> index ba1f942328..cfc620d554 100755
>>> --- a/tests/docker/common.rc
>>> +++ b/tests/docker/common.rc
>>> @@ -40,6 +40,16 @@ build_qemu()
>>>      make $MAKEFLAGS
>>>  }
>>>
>>> +check_qemu()
>>> +{
>>> +    if test -z "$@"; then
>>> +        TEST="check"
>>> +    else
>>> +        TEST="$@"
>>
>> This is a bit confuse:
>>
>>   check_qemu V=1 -> TEST=V=1
>>
>>> +    fi
>>> +    make $MAKEFLAGS $TEST
>>
>>     -> make V=1
> 
> How about s/TEST/INVOCATION/?

INVOCATION or make $MAKEFLAGS ${@:-check}:

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

>>
>>> +}
>>> +
>>>  test_fail()
>>>  {
>>>      echo "$@"
>>> diff --git a/tests/docker/test-clang b/tests/docker/test-clang
>>> index e90a793178..324e341cea 100755
>>> --- a/tests/docker/test-clang
>>> +++ b/tests/docker/test-clang
>>> @@ -23,5 +23,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
>>>  #OPTS="$OPTS --extra-cflags=-fsanitize=undefined \
>>>      #--extra-cflags=-fno-sanitize=float-divide-by-zero"
>>>  build_qemu $OPTS
>>> -make $MAKEFLAGS check
>>> +check_qemu
>>>  install_qemu
>>> diff --git a/tests/docker/test-debug b/tests/docker/test-debug
>>> index d3f9f70d01..137f4f2ddc 100755
>>> --- a/tests/docker/test-debug
>>> +++ b/tests/docker/test-debug
>>> @@ -22,5 +22,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
>>>  OPTS="--enable-debug --enable-sanitizers $OPTS"
>>>
>>>  build_qemu $OPTS
>>> -make $MAKEFLAGS V=1 check
>>> +check_qemu check V=1
>>
>> Gotcha!
>>
>>>  install_qemu
>>> diff --git a/tests/docker/test-full b/tests/docker/test-full
>>> index b4e42d25d7..aadc0f00a2 100755
>>> --- a/tests/docker/test-full
>>> +++ b/tests/docker/test-full
>>> @@ -15,4 +15,4 @@
>>>
>>>  cd "$BUILD_DIR"
>>>
>>> -build_qemu && make check $MAKEFLAGS && install_qemu
>>> +build_qemu && check_qemu && install_qemu
>>> diff --git a/tests/docker/test-quick b/tests/docker/test-quick
>>> index 3b7bce6105..eee59c55fb 100755
>>> --- a/tests/docker/test-quick
>>> +++ b/tests/docker/test-quick
>>> @@ -18,5 +18,5 @@ cd "$BUILD_DIR"
>>>  DEF_TARGET_LIST="x86_64-softmmu,aarch64-softmmu"
>>>  TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
>>>  build_qemu
>>> -make check $MAKEFLAGS
>>> +check_qemu
>>>  install_qemu
>>>
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 15/16] docker: add commentary to debian-bootstrap.docker

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> This is just a note that later versions of debootstrap don't
> technically need this hack.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  tests/docker/dockerfiles/debian-bootstrap.docker | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
> index 14212b9cf4..e13c26a7ed 100644
> --- a/tests/docker/dockerfiles/debian-bootstrap.docker
> +++ b/tests/docker/dockerfiles/debian-bootstrap.docker
> @@ -9,6 +9,7 @@ FROM scratch
>  ADD . /
>  
>  # Patch all mounts as docker already has stuff set up
> +# (this is not needed for later debootstraps but is harmless atm)
>  RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
>  
>  # Run stage 2
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 11/16] docker: add expansion for docker-test-FOO to Makefile.include

[Philippe Mathieu-Daudé]

On 07/16/2018 06:07 AM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Hi Alex,
>>
>> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>>> This allows us to run a particular test on all docker images. For
>>> example:
>>>
>>>   make docker-test-unit
>>>
>>> Will run the unit tests on every supported image. At the same time
>>> rename docker-test to docker-all-tests to be clearer.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>>   - docker-test -> docker-all-tests
>>> ---
>>>  tests/docker/Makefile.include | 6 ++++--
>>>  1 file changed, 4 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>>> index fe63aacf69..e32c35be0d 100644
>>> --- a/tests/docker/Makefile.include
>>> +++ b/tests/docker/Makefile.include
>>> @@ -151,7 +151,8 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
>>>  		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
>>>  	) \
>>>  	$(foreach t,$(DOCKER_TESTS), \
>>> -		$(eval docker-test: docker-$t@$i) \
>>> +		$(eval docker-all-tests: docker-$t@$i) \
>>> +		$(eval docker-$t: docker-$t@$i) \
>>
>> Is this supposed to work this way?
>>
>> $ make docker-test-quick@debian-alpha-cross
>> make: *** No rule to make target
>> 'docker-test-quick@debian-alpha-cross'.
> 
> No
> 
>   make docker-test-quick
> 
> will make the test-quick on all images.
> 
> Your example fails because debian-alpha-cross is a PARTIAL image, good
> for building test cases but not QEMU so it's not expanded in the above
> bit: $(eval docker-$t@$i: docker-image-$i docker-run-$t@$i)

I noticed too late ;)

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> 
>>  Stop.
>>
>>>  	) \
>>>  )
>>>
>>> @@ -161,7 +162,8 @@ docker:
>>>  	@echo 'Available targets:'
>>>  	@echo
>>>  	@echo '    docker:              Print this help.'
>>> -	@echo '    docker-test:         Run all image/test combinations.'
>>> +	@echo '    docker-all-tests:    Run all image/test combinations.'
>>> +	@echo '    docker-TEST:         Run TEST on all image combinations.'
>>>  	@echo '    docker-clean:        Kill and remove residual docker testing containers.'
>>>  	@echo '    docker-TEST@IMAGE:   Run "TEST" in container "IMAGE".'
>>>  	@echo '                         Note: "TEST" is one of the listed test name,'
>>>
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 07/16] docker: gracefully skip check_qemu

[Philippe Mathieu-Daudé]

On 07/17/2018 07:22 AM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>>> Not all our images are able to run the tests. Rather than use features
>>> we can just check for the existence and run-ability of gtester. If the
>>> image has been setup for binfmt_misc it will be able to run anyway.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>>  tests/docker/common.rc | 8 +++++++-
>>>  1 file changed, 7 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
>>> index cfc620d554..6df431eb72 100755
>>> --- a/tests/docker/common.rc
>>> +++ b/tests/docker/common.rc
>>> @@ -47,7 +47,13 @@ check_qemu()
>>>      else
>>>          TEST="$@"
>>>      fi
>>> -    make $MAKEFLAGS $TEST
>>> +
>>> +    if type gtester > /dev/null 2>&1 && \
>>
>> commit e465ce7d09939d631f1861e0bd8873417c1c0d65
>>
>>     tests: Use "command -v" instead of which(1) in shell scripts
> 
> type is also a shell built-in albeit an 1982 bashism. I'll switch to
> command -v as tending towards POSIX is what we should aim for (although
> our test-FOO/common.rc structure seems a little confused).

Thanks, with 'command -v':
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> 
>>
>>     When which(1) is not installed, we would complain "perl not found"
>>     because it's the first set_prog_path check. The error message is
>>     wrong.
>>
>>     Fix it by using "command -v", a native way to query the existence
>>     of a command.
>>
>>> +           gtester --version > /dev/null 2>&1; then
>>> +        make $MAKEFLAGS $TEST
>>> +    else
>>> +        echo "No working gtester, skipping make $TEST"
>>> +    fi
>>>  }
>>>
>>>  test_fail()
>>>
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 13/16] docker: add --hint to docker.py check

[Philippe Mathieu-Daudé]

On 07/17/2018 12:24 AM, Philippe Mathieu-Daudé wrote:
> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>> When a check fails we currently just report why we failed. This is not
>> totally helpful to people who want to boot-strap a new image. Add a
>> --hint option which we can pass down to give a bit more information.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Re: [Qemu-devel] [PATCH v2 for 3.0 12/16] docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> The addition of QEMU_TARGET was intended to ensure we fall back to
> checking for the existence of an image if the build system was not
> currently configured to build it. However this breaks the direct use
> of the rule for building custom binfmt_misc images. We already check
> for EXECUTABLE so let us just use that as a proxy for deciding if we
> are just going to check the image exits.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  tests/docker/Makefile.include | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index e32c35be0d..ec23620153 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -58,13 +58,11 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
>  	$(if $(EXECUTABLE),,\
>  		$(error EXECUTABLE not set, debootstrap of debian-$* would fail))
> -	$(if $(wildcard $(EXECUTABLE)),,\
> -		$(error Please build $(EXECUTABLE) first))
>  	$(if $(DEB_ARCH),,\
>  		$(error DEB_ARCH not set, debootstrap of debian-$* would fail))
>  	$(if $(DEB_TYPE),,\
>  		$(error DEB_TYPE not set, debootstrap of debian-$* would fail))
> -	$(if $(filter $(QEMU_TARGET),$(TARGET_DIRS)),				\
> +	$(if $(wildcard $(EXECUTABLE)),						\
>  		$(call quiet-command,						\
>  			DEB_ARCH=$(DEB_ARCH)					\
>  			DEB_TYPE=$(DEB_TYPE)					\
> @@ -136,7 +134,6 @@ DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-c
>  # broken so we need a qemu-linux-user for this target
>  docker-binfmt-image-debian-powerpc-user: DEB_ARCH = powerpc
>  docker-binfmt-image-debian-powerpc-user: DEB_TYPE = jessie
> -docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
>  docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
>  docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
>  DOCKER_USER_IMAGES += debian-powerpc-user
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 16/16] docker: ignore distro versioning of debootstrap

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> We do a minimum version check for the debootstrap but if the distro
> has added their own minor version tick it would fail and fall-back to
> the SCM version. This is sub-optimal as the latest/greatest version
> may be broken at any one particular time. We fix that with a little
> sed magic on the version string before passing to our ugly shell
> versioning check.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  tests/docker/dockerfiles/debian-bootstrap.pre | 11 +++++++----
>  1 file changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
> index ea324d6e4a..3b0ef95374 100755
> --- a/tests/docker/dockerfiles/debian-bootstrap.pre
> +++ b/tests/docker/dockerfiles/debian-bootstrap.pre
> @@ -56,10 +56,13 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
>      if [ -z $DEBOOTSTRAP ]; then
>          echo "No debootstrap installed, attempting to install from SCM"
>          NEED_DEBOOTSTRAP=true
> -    elif ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; "${DEBOOTSTRAP}" --version \
> -            | cut -d ' ' -f 2) | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -c &>/dev/null; then
> -        echo "debootstrap too old, attempting to install from SCM"
> -        NEED_DEBOOTSTRAP=true
> +    else
> +        INSTALLED_VERSION=$(${DEBOOTSTRAP} --version | sed 's/debootstrap \([0-9\.]*\)[^0-9\.]*.*/\1/')
> +        if ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; echo "${INSTALLED_VERSION}") \
> +                | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -C ; then
> +            echo "debootstrap too old, attempting to install from SCM"
> +            NEED_DEBOOTSTRAP=true
> +        fi
>      fi
>      if $NEED_DEBOOTSTRAP; then
>          DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 09/16] docker: disable debian-powerpc-user-cross

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> While we can build we can't run the unit tests due to a missing system
> call (249 - swapcontext). This could be a code generation issue as
> other architectures don't seem to have this system call.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> Cc: David Gibson <david@gibson.dropbear.id.au>
> Cc: Alexander Graf <agraf@suse.de>
> Cc: qemu-ppc@nongnu.org
> ---
>  tests/docker/Makefile.include | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 09fb7db7fa..fe63aacf69 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -140,6 +140,9 @@ docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
>  docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
>  docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
>  DOCKER_USER_IMAGES += debian-powerpc-user
> +# Unfortunalty while we can build with this image we can't run unit
> +# tests due to missing system call: 249 which causes an abort
> +DOCKER_PARTIAL_IMAGES += debian-powerpc-user-cross
>  
>  # Expand all the pre-requistes for each docker image and test combination
>  $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
> 

Re: [Qemu-devel] [PATCH v2 for 3.0 00/16] various docker fixes

[Philippe Mathieu-Daudé]

On 07/13/2018 09:17 AM, Alex Bennée wrote:
> Hi,
> 
> Unfortunately this series grew a little while I was re-basing as I
> noticed a couple of bugs. An upstream change in debootstrap which I'd
> been happily using for Ubuntu images caused a breakage which I've
> hopefully now fixed:
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903657
> 
> However since I upgraded by devbox I hadn't clocked to the fact I
> should have been using the stable debootstrap anyway but our ugly
> shell version check failed to pick it up. That's now fixed and I can
> now bootstrap away.
> 
> I grabbed a couple of Philippe's patches while I was at it as they
> were relevant to the other changes. It does mean the un-reviewed count
> has gone up slightly:
> 
>   patch docker/base debian tricore on qemu debian9.patch
>   patch docker/split configure_qemu from build_qemu.patch
>   patch docker/move make check into check_qemu helper.patch
>   patch docker/gracefully skip check_qemu.patch
>   patch docker/Makefile.include don t include partial images.patch
>   patch docker/disable debian powerpc user cross.patch
>   patch docker/add test unit runner.patch
>   patch docker/add expansion for docker test FOO to Makefile.patch
>   patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
>   patch docker/add hint to docker.py check.patch
>   patch docker/add commentary to debian bootstrap.docker.patch
>   patch docker/ignore distro versioning of debootstrap.patch

[Note]

I hit:

$ make docker-binfmt-image-debian-powerpc-user
  BUILD   binfmt debian-powerpc-user (debootstrapped)
Cloning into './debootstrap.git'...
remote: Enumerating objects: 3262, done.
remote: Counting objects: 100% (3262/3262), done.
remote: Compressing objects: 100% (902/902), done.
remote: Total 3262 (delta 2353), reused 3245 (delta 2339)
Receiving objects: 100% (3262/3262), 531.08 KiB | 335.00 KiB/s, done.
Resolving deltas: 100% (2353/2353), done.
The command '/bin/sh -c /debootstrap/debootstrap --second-stage'
returned a non-zero code: 139
Traceback (most recent call last):
  File "tests/docker/docker.py", line 536, in <module>
    sys.exit(main())
  File "tests/docker/docker.py", line 533, in main
    return args.cmdobj.run(args, argv)
  File "tests/docker/docker.py", line 348, in run
    extra_files_cksum=cksum)
  File "tests/docker/docker.py", line 231, in build_image
    quiet=quiet)
  File "tests/docker/docker.py", line 158, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '-t',
'qemu:debian-powerpc-user', '-f',
'/tmp/docker_buildPNDAtz/tmpIVpPn7.docker', '/tmp/docker_buildPNDAtz']'
returned non-zero exit status 139
make: *** [tests/docker/Makefile.include:62:
docker-binfmt-image-debian-powerpc-user] Error 1

"returned non-zero exit status 139" seems to be qemu-user crashing for
not being statistically linked.
Once rebuilt after ./configure --static, it worked:

$ make docker-binfmt-image-debian-powerpc-user V=1
[...]
I: Extracting zlib1g...
ppc-linux-user/qemu-ppc had no associated libraries (static build?)
Sending build context to Docker daemon  226.5MB
Step 1/7 : FROM scratch
Step 2/7 : ADD . /
Step 3/7 : RUN sed -i 's/in_target mount/echo not for docker in_target
mount/g' /debootstrap/functions
Step 4/7 : RUN /debootstrap/debootstrap --second-stage
I: Installing core packages...
[...]

Re: [Qemu-devel] [PATCH v2 for 3.0 00/16] various docker fixes

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>> Hi,
>>
>> Unfortunately this series grew a little while I was re-basing as I
>> noticed a couple of bugs. An upstream change in debootstrap which I'd
>> been happily using for Ubuntu images caused a breakage which I've
>> hopefully now fixed:
>>
>>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903657
>>
>> However since I upgraded by devbox I hadn't clocked to the fact I
>> should have been using the stable debootstrap anyway but our ugly
>> shell version check failed to pick it up. That's now fixed and I can
>> now bootstrap away.
>>
>> I grabbed a couple of Philippe's patches while I was at it as they
>> were relevant to the other changes. It does mean the un-reviewed count
>> has gone up slightly:
>>
>>   patch docker/base debian tricore on qemu debian9.patch
>>   patch docker/split configure_qemu from build_qemu.patch
>>   patch docker/move make check into check_qemu helper.patch
>>   patch docker/gracefully skip check_qemu.patch
>>   patch docker/Makefile.include don t include partial images.patch
>>   patch docker/disable debian powerpc user cross.patch
>>   patch docker/add test unit runner.patch
>>   patch docker/add expansion for docker test FOO to Makefile.patch
>>   patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
>>   patch docker/add hint to docker.py check.patch
>>   patch docker/add commentary to debian bootstrap.docker.patch
>>   patch docker/ignore distro versioning of debootstrap.patch
>
> [Note]
>
> I hit:
>
> $ make docker-binfmt-image-debian-powerpc-user
>   BUILD   binfmt debian-powerpc-user (debootstrapped)
> Cloning into './debootstrap.git'...
> remote: Enumerating objects: 3262, done.
> remote: Counting objects: 100% (3262/3262), done.
> remote: Compressing objects: 100% (902/902), done.
> remote: Total 3262 (delta 2353), reused 3245 (delta 2339)
> Receiving objects: 100% (3262/3262), 531.08 KiB | 335.00 KiB/s, done.
> Resolving deltas: 100% (2353/2353), done.
> The command '/bin/sh -c /debootstrap/debootstrap --second-stage'
> returned a non-zero code: 139
> Traceback (most recent call last):
>   File "tests/docker/docker.py", line 536, in <module>
>     sys.exit(main())
>   File "tests/docker/docker.py", line 533, in main
>     return args.cmdobj.run(args, argv)
>   File "tests/docker/docker.py", line 348, in run
>     extra_files_cksum=cksum)
>   File "tests/docker/docker.py", line 231, in build_image
>     quiet=quiet)
>   File "tests/docker/docker.py", line 158, in _do_check
>     return subprocess.check_call(self._command + cmd, **kwargs)
>   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['docker', 'build', '-t',
> 'qemu:debian-powerpc-user', '-f',
> '/tmp/docker_buildPNDAtz/tmpIVpPn7.docker', '/tmp/docker_buildPNDAtz']'
> returned non-zero exit status 139
> make: *** [tests/docker/Makefile.include:62:
> docker-binfmt-image-debian-powerpc-user] Error 1
>
> "returned non-zero exit status 139" seems to be qemu-user crashing for
> not being statistically linked.

What system are you building on? Basically --static is always safer but
as long as there isn't a clash between host/guest file system layouts it
can work which is why I don't enforce it (although earlier patches tried
to).

> Once rebuilt after ./configure --static, it worked:
>
> $ make docker-binfmt-image-debian-powerpc-user V=1
> [...]
> I: Extracting zlib1g...
> ppc-linux-user/qemu-ppc had no associated libraries (static build?)
> Sending build context to Docker daemon  226.5MB
> Step 1/7 : FROM scratch
> Step 2/7 : ADD . /
> Step 3/7 : RUN sed -i 's/in_target mount/echo not for docker in_target
> mount/g' /debootstrap/functions
> Step 4/7 : RUN /debootstrap/debootstrap --second-stage
> I: Installing core packages...
> [...]


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 for 3.0 00/16] various docker fixes

[Philippe Mathieu-Daudé]

On 07/17/2018 03:53 PM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>> On 07/13/2018 09:17 AM, Alex Bennée wrote:
>>> Hi,
>>>
>>> Unfortunately this series grew a little while I was re-basing as I
>>> noticed a couple of bugs. An upstream change in debootstrap which I'd
>>> been happily using for Ubuntu images caused a breakage which I've
>>> hopefully now fixed:
>>>
>>>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903657
>>>
>>> However since I upgraded by devbox I hadn't clocked to the fact I
>>> should have been using the stable debootstrap anyway but our ugly
>>> shell version check failed to pick it up. That's now fixed and I can
>>> now bootstrap away.
>>>
>>> I grabbed a couple of Philippe's patches while I was at it as they
>>> were relevant to the other changes. It does mean the un-reviewed count
>>> has gone up slightly:
>>>
>>>   patch docker/base debian tricore on qemu debian9.patch
>>>   patch docker/split configure_qemu from build_qemu.patch
>>>   patch docker/move make check into check_qemu helper.patch
>>>   patch docker/gracefully skip check_qemu.patch
>>>   patch docker/Makefile.include don t include partial images.patch
>>>   patch docker/disable debian powerpc user cross.patch
>>>   patch docker/add test unit runner.patch
>>>   patch docker/add expansion for docker test FOO to Makefile.patch
>>>   patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
>>>   patch docker/add hint to docker.py check.patch
>>>   patch docker/add commentary to debian bootstrap.docker.patch
>>>   patch docker/ignore distro versioning of debootstrap.patch
>>
>> [Note]
>>
>> I hit:
>>
>> $ make docker-binfmt-image-debian-powerpc-user
>>   BUILD   binfmt debian-powerpc-user (debootstrapped)
>> Cloning into './debootstrap.git'...
>> remote: Enumerating objects: 3262, done.
>> remote: Counting objects: 100% (3262/3262), done.
>> remote: Compressing objects: 100% (902/902), done.
>> remote: Total 3262 (delta 2353), reused 3245 (delta 2339)
>> Receiving objects: 100% (3262/3262), 531.08 KiB | 335.00 KiB/s, done.
>> Resolving deltas: 100% (2353/2353), done.
>> The command '/bin/sh -c /debootstrap/debootstrap --second-stage'
>> returned a non-zero code: 139
>> Traceback (most recent call last):
>>   File "tests/docker/docker.py", line 536, in <module>
>>     sys.exit(main())
>>   File "tests/docker/docker.py", line 533, in main
>>     return args.cmdobj.run(args, argv)
>>   File "tests/docker/docker.py", line 348, in run
>>     extra_files_cksum=cksum)
>>   File "tests/docker/docker.py", line 231, in build_image
>>     quiet=quiet)
>>   File "tests/docker/docker.py", line 158, in _do_check
>>     return subprocess.check_call(self._command + cmd, **kwargs)
>>   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
>>     raise CalledProcessError(retcode, cmd)
>> subprocess.CalledProcessError: Command '['docker', 'build', '-t',
>> 'qemu:debian-powerpc-user', '-f',
>> '/tmp/docker_buildPNDAtz/tmpIVpPn7.docker', '/tmp/docker_buildPNDAtz']'
>> returned non-zero exit status 139
>> make: *** [tests/docker/Makefile.include:62:
>> docker-binfmt-image-debian-powerpc-user] Error 1
>>
>> "returned non-zero exit status 139" seems to be qemu-user crashing for
>> not being statistically linked.
> 
> What system are you building on? Basically --static is always safer but
> as long as there isn't a clash between host/guest file system layouts it
> can work which is why I don't enforce it (although earlier patches tried
> to).

$ docker info
Server Version: 18.05.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Kernel Version: 4.16.0-2-amd64
Operating System: Debian GNU/Linux buster/sid
OSType: linux
Architecture: x86_64

>> Once rebuilt after ./configure --static, it worked:
>>
>> $ make docker-binfmt-image-debian-powerpc-user V=1
>> [...]
>> I: Extracting zlib1g...
>> ppc-linux-user/qemu-ppc had no associated libraries (static build?)
>> Sending build context to Docker daemon  226.5MB
>> Step 1/7 : FROM scratch
>> Step 2/7 : ADD . /
>> Step 3/7 : RUN sed -i 's/in_target mount/echo not for docker in_target
>> mount/g' /debootstrap/functions
>> Step 4/7 : RUN /debootstrap/debootstrap --second-stage
>> I: Installing core packages...
>> [...]