[Qemu-devel] [PATCH v3 for 3.0 00/18] docker fixes (and one tcg test tweak)

[Qemu-devel] [PATCH v3 for 3.0 00/18] docker fixes (and one tcg test tweak)

[Alex Bennée]

Hi,

I've missed the boat for today's rc1 but I'd like to get this merged
before rc2. The new docker.py change is technically new functionality
but I'm counting it as a usability bug fix as it replaces a random
back trace failure with a preemptive failure and message mentioning
binfmt_misc configuration. This would have saved Richard a lot of head
scratching as he tried to setup a powerpc-user setup to test his
setcontext fix (he had a custom binfmt_misc pointing to his src tree).

Finally we also drop the runcom test. It was cute that it got
resurrected but it is ultimately a pointless test for something I'm
sure no one actually uses.

There will be a follow-up RFC series after this that cleans-up some of
the rough edges when your host is not an x86_64 box but that series
won't be targeting the 3.0 release.

: The following patches need review
: patch docker/disable debian powerpc user cross.patch
: patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
: patch docker/Update debootstrap script after Debian migrat.patch
: patch docker/ignore distro versioning of debootstrap.patch
: patch docker/perform basic binfmt_misc validation in docke.patch
: patch tests/tcg remove runcom test.patch

Alex Bennée (17):
  tests/.gitignore: don't ignore docker tests
  docker: base debian-tricore on qemu:debian9
  docker: par down QEMU_CONFIGURE_OPTS in debian-tricore-cross
  docker: fail more gracefully on docker.py check
  docker: split configure_qemu from build_qemu
  docker: move make check into check_qemu helper
  docker: gracefully skip check_qemu
  docker: Makefile.include don't include partial images
  docker: disable debian-powerpc-user-cross
  docker: add test-unit runner
  docker: add expansion for docker-test-FOO to Makefile.include
  docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set
  docker: add --hint to docker.py check
  docker: add commentary to debian-bootstrap.docker
  docker: ignore distro versioning of debootstrap
  docker: perform basic binfmt_misc validation in docker.py
  tests/tcg: remove runcom test

Philippe Mathieu-Daudé (1):
  docker: Update debootstrap script after Debian migration from Alioth
    to Salsa

 tests/.gitignore                              |   1 +
 tests/docker/Makefile.include                 |  26 ++-
 tests/docker/common.rc                        |  24 ++-
 tests/docker/docker.py                        |  40 +++-
 .../dockerfiles/debian-bootstrap.docker       |   1 +
 tests/docker/dockerfiles/debian-bootstrap.pre |  13 +-
 .../dockerfiles/debian-tricore-cross.docker   |   6 +-
 tests/docker/test-clang                       |   2 +-
 tests/docker/test-debug                       |   2 +-
 tests/docker/test-full                        |   2 +-
 tests/docker/test-quick                       |   2 +-
 tests/docker/test-unit                        |  21 ++
 tests/tcg/i386/Makefile.target                |   5 -
 tests/tcg/i386/README                         |   3 -
 tests/tcg/i386/pi_10.com                      | Bin 54 -> 0 bytes
 tests/tcg/i386/runcom.c                       | 192 ------------------
 16 files changed, 116 insertions(+), 224 deletions(-)
 create mode 100755 tests/docker/test-unit
 delete mode 100644 tests/tcg/i386/pi_10.com
 delete mode 100644 tests/tcg/i386/runcom.c

-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 01/18] tests/.gitignore: don't ignore docker tests

[Alex Bennée]

The .gitignore was being a little over enthusiastic hiding files.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

---
v2
  - minor wording tweak for more explicitness
---
 tests/.gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/.gitignore b/tests/.gitignore
index 08e2df1ce1..72c18aaab0 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -9,6 +9,7 @@ qht-bench
 rcutorture
 test-*
 !test-*.c
+!docker/test-*
 test-qapi-commands.[ch]
 test-qapi-events.[ch]
 test-qapi-types.[ch]
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 02/18] docker: base debian-tricore on qemu:debian9

[Alex Bennée]

We need both git and a working compiler to build the tools. Although
the qemu:debian9 image also has a bunch of extra dependencies it would
be fairly unusual for a user not to already have this layer available
for one of our many other docker images so lets not complicate things.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-tricore-cross.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 898b8dd511..180ca646c8 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -7,7 +7,7 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
-FROM debian:9
+FROM qemu:debian9
 
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 03/18] docker: par down QEMU_CONFIGURE_OPTS in debian-tricore-cross

[Alex Bennée]

This image isn't going to build anything significant as it is just
intended for building test cases. In case it does end up getting
inadvertently included in a build lets aim for the minimal possible
product.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-tricore-cross.docker | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 180ca646c8..4a0f7706a3 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -19,5 +19,5 @@ RUN git clone --single-branch \
     make && make install && \
     rm -rf /usr/src/binutils
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
-ENV QEMU_CONFIGURE_OPTS --cross-prefix=tricore-
+# This image isn't designed for building QEMU but building tests
+ENV QEMU_CONFIGURE_OPTS --disable-system --disable-user
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 04/18] docker: fail more gracefully on docker.py check

[Alex Bennée]

As this is called directly from the Makefile while determining
dependencies and it is possible the user was configured in one window
but not have credentials in the other. Let's catch the Exceptions and
deal with it quietly.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/docker.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 69e7130db7..2f81c6b13b 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -479,7 +479,12 @@ class CheckCommand(SubCommand):
     def run(self, args, argv):
         tag = args.tag
 
-        dkr = Docker()
+        try:
+            dkr = Docker()
+        except:
+            print("Docker not set up")
+            return 1
+
         info = dkr.inspect_tag(tag)
         if info is None:
             print("Image does not exist")
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 05/18] docker: split configure_qemu from build_qemu

[Alex Bennée]

This allows some tests that just want to configure QEMU's source tree
to do so.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/common.rc | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index 046f8a5921..ba1f942328 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -21,7 +21,7 @@ requires()
     done
 }
 
-build_qemu()
+configure_qemu()
 {
     config_opts="--enable-werror \
                  ${TARGET_LIST:+--target-list=${TARGET_LIST}} \
@@ -32,6 +32,11 @@ build_qemu()
     echo $config_opts
     $QEMU_SRC/configure $config_opts || \
         { cat config.log && test_fail "Failed to run 'configure'"; }
+}
+
+build_qemu()
+{
+    configure_qemu $@
     make $MAKEFLAGS
 }
 
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 06/18] docker: move make check into check_qemu helper

[Alex Bennée]

Not all docker images can run the check step. Let's move everything
into a common helper so we don't need to replicate checks in the
future.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
v2
  - add "move" to Let's everything in commit message
v3
  - avoid confusion using IVOCATION, add comment
---
 tests/docker/common.rc  | 11 +++++++++++
 tests/docker/test-clang |  2 +-
 tests/docker/test-debug |  2 +-
 tests/docker/test-full  |  2 +-
 tests/docker/test-quick |  2 +-
 5 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index ba1f942328..4ff5974016 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -40,6 +40,17 @@ build_qemu()
     make $MAKEFLAGS
 }
 
+check_qemu()
+{
+    # default to make check unless the caller specifies
+    if test -z "$@"; then
+        INVOCATION="check"
+    else
+        INVOCATION="$@"
+    fi
+    make $MAKEFLAGS $INVOCATION
+}
+
 test_fail()
 {
     echo "$@"
diff --git a/tests/docker/test-clang b/tests/docker/test-clang
index e90a793178..324e341cea 100755
--- a/tests/docker/test-clang
+++ b/tests/docker/test-clang
@@ -23,5 +23,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
 #OPTS="$OPTS --extra-cflags=-fsanitize=undefined \
     #--extra-cflags=-fno-sanitize=float-divide-by-zero"
 build_qemu $OPTS
-make $MAKEFLAGS check
+check_qemu
 install_qemu
diff --git a/tests/docker/test-debug b/tests/docker/test-debug
index d3f9f70d01..137f4f2ddc 100755
--- a/tests/docker/test-debug
+++ b/tests/docker/test-debug
@@ -22,5 +22,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
 OPTS="--enable-debug --enable-sanitizers $OPTS"
 
 build_qemu $OPTS
-make $MAKEFLAGS V=1 check
+check_qemu check V=1
 install_qemu
diff --git a/tests/docker/test-full b/tests/docker/test-full
index b4e42d25d7..aadc0f00a2 100755
--- a/tests/docker/test-full
+++ b/tests/docker/test-full
@@ -15,4 +15,4 @@
 
 cd "$BUILD_DIR"
 
-build_qemu && make check $MAKEFLAGS && install_qemu
+build_qemu && check_qemu && install_qemu
diff --git a/tests/docker/test-quick b/tests/docker/test-quick
index 3b7bce6105..eee59c55fb 100755
--- a/tests/docker/test-quick
+++ b/tests/docker/test-quick
@@ -18,5 +18,5 @@ cd "$BUILD_DIR"
 DEF_TARGET_LIST="x86_64-softmmu,aarch64-softmmu"
 TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
 build_qemu
-make check $MAKEFLAGS
+check_qemu
 install_qemu
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 07/18] docker: gracefully skip check_qemu

[Alex Bennée]

Not all our images are able to run the tests. Rather than use features
we can just check for the existence and run-ability of gtester. If the
image has been setup for binfmt_misc it will be able to run anyway.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
v3
  - use command -v in preference to bash-ist type
---
 tests/docker/common.rc | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index 4ff5974016..4011561587 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -48,7 +48,13 @@ check_qemu()
     else
         INVOCATION="$@"
     fi
-    make $MAKEFLAGS $INVOCATION
+
+    if command -v gtester > /dev/null 2>&1 && \
+           gtester --version > /dev/null 2>&1; then
+        make $MAKEFLAGS $INVOCATION
+    else
+        echo "No working gtester, skipping make $INVOCATION"
+    fi
 }
 
 test_fail()
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 08/18] docker: Makefile.include don't include partial images

[Alex Bennée]

Rename DOCKER_INTERMEDIATE_IMAGES to DOCKER_PARTIAL_IMAGES and add the
incomplete cross compiler images that can build tests but can't build
QEMU itself. We also add debian, debian-bootstrap and the tricode
images to the list.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index b2a7e761cc..09fb7db7fa 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -6,7 +6,7 @@ DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 DOCKER_DEPRECATED_IMAGES := debian
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_INTERMEDIATE_IMAGES := debian8 debian9 debian8-mxe debian-ports debian-sid
+DOCKER_PARTIAL_IMAGES := debian debian8 debian9 debian8-mxe debian-ports debian-sid debian-bootstrap
 DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
@@ -121,6 +121,11 @@ docker-image-travis: NOUSER=1
 # Specialist build images, sometimes very limited tools
 docker-image-tricore-cross: docker-image-debian9
 
+# These images may be good enough for building tests but not for test builds
+DOCKER_PARTIAL_IMAGES += debian-alpha-cross debian-hppa-cross debian-m68k-cross debian-sh4-cross
+DOCKER_PARTIAL_IMAGES += debian-sparc64-cross debian-mips64-cross debian-riscv64-cross
+DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-cross
+
 # Rules for building linux-user powered images
 #
 # These are slower than using native cross compiler setups but can
@@ -137,7 +142,7 @@ docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
 
 # Expand all the pre-requistes for each docker image and test combination
-$(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
+$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
 		$(eval .PHONY: docker-$t@$i) \
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 09/18] docker: disable debian-powerpc-user-cross

[Alex Bennée]

While we can build we can't run the unit tests due to a missing system
call (249 - swapcontext). This could be a code generation issue as
other architectures don't seem to have this system call.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Alexander Graf <agraf@suse.de>
Cc: qemu-ppc@nongnu.org
---
 tests/docker/Makefile.include | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 09fb7db7fa..fe63aacf69 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -140,6 +140,9 @@ docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
 docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
 docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
+# Unfortunalty while we can build with this image we can't run unit
+# tests due to missing system call: 249 which causes an abort
+DOCKER_PARTIAL_IMAGES += debian-powerpc-user-cross
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 10/18] docker: add test-unit runner

[Alex Bennée]

This test doesn't even build QEMU, it just builds and runs all the
unit tests. Intended to make checking unit tests on all docker images
easier.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
v2
  - make script -e to error out on failure
  - add comment as to why the configure step is run
---
 tests/docker/test-unit | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100755 tests/docker/test-unit

diff --git a/tests/docker/test-unit b/tests/docker/test-unit
new file mode 100755
index 0000000000..8905d01150
--- /dev/null
+++ b/tests/docker/test-unit
@@ -0,0 +1,21 @@
+#!/bin/bash -e
+#
+# Build and run the unit tests
+#
+# Copyright (c) 2018 Linaro Ltd.
+#
+# Authors:
+#  Alex Bennée <alex.bennee@linaro.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+cd "$BUILD_DIR"
+
+# although we are not building QEMU itself we still need a configured
+# build for the unit tests to be built and run
+configure_qemu
+check_qemu check-unit
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 11/18] docker: add expansion for docker-test-FOO to Makefile.include

[Alex Bennée]

This allows us to run a particular test on all docker images. For
example:

  make docker-test-unit

Will run the unit tests on every supported image. At the same time
rename docker-test to docker-all-tests to be clearer.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
v2
  - docker-test -> docker-all-tests
---
 tests/docker/Makefile.include | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index fe63aacf69..e32c35be0d 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -151,7 +151,8 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
 	) \
 	$(foreach t,$(DOCKER_TESTS), \
-		$(eval docker-test: docker-$t@$i) \
+		$(eval docker-all-tests: docker-$t@$i) \
+		$(eval docker-$t: docker-$t@$i) \
 	) \
 )
 
@@ -161,7 +162,8 @@ docker:
 	@echo 'Available targets:'
 	@echo
 	@echo '    docker:              Print this help.'
-	@echo '    docker-test:         Run all image/test combinations.'
+	@echo '    docker-all-tests:    Run all image/test combinations.'
+	@echo '    docker-TEST:         Run TEST on all image combinations.'
 	@echo '    docker-clean:        Kill and remove residual docker testing containers.'
 	@echo '    docker-TEST@IMAGE:   Run "TEST" in container "IMAGE".'
 	@echo '                         Note: "TEST" is one of the listed test name,'
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 12/18] docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set

[Alex Bennée]

The addition of QEMU_TARGET was intended to ensure we fall back to
checking for the existence of an image if the build system was not
currently configured to build it. However this breaks the direct use
of the rule for building custom binfmt_misc images. We already check
for EXECUTABLE so let us just use that as a proxy for deciding if we
are just going to check the image exits.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index e32c35be0d..ec23620153 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -58,13 +58,11 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 	$(if $(EXECUTABLE),,\
 		$(error EXECUTABLE not set, debootstrap of debian-$* would fail))
-	$(if $(wildcard $(EXECUTABLE)),,\
-		$(error Please build $(EXECUTABLE) first))
 	$(if $(DEB_ARCH),,\
 		$(error DEB_ARCH not set, debootstrap of debian-$* would fail))
 	$(if $(DEB_TYPE),,\
 		$(error DEB_TYPE not set, debootstrap of debian-$* would fail))
-	$(if $(filter $(QEMU_TARGET),$(TARGET_DIRS)),				\
+	$(if $(wildcard $(EXECUTABLE)),						\
 		$(call quiet-command,						\
 			DEB_ARCH=$(DEB_ARCH)					\
 			DEB_TYPE=$(DEB_TYPE)					\
@@ -136,7 +134,6 @@ DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-c
 # broken so we need a qemu-linux-user for this target
 docker-binfmt-image-debian-powerpc-user: DEB_ARCH = powerpc
 docker-binfmt-image-debian-powerpc-user: DEB_TYPE = jessie
-docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
 docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
 docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 13/18] docker: add --hint to docker.py check

[Alex Bennée]

When a check fails we currently just report why we failed. This is not
totally helpful to people who want to boot-strap a new image. Add a
--hint option which we can pass down to give a bit more information.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 3 ++-
 tests/docker/docker.py        | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index ec23620153..c9e412f9d0 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -73,7 +73,8 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 			$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
 			"BUILD","binfmt debian-$* (debootstrapped)"),		\
 		$(call quiet-command,						\
-			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<,	\
+			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<	\
+			--hint "you will need to build $(EXECUTABLE)",		\
 			"CHECK", "debian-$* exists"))
 
 endif
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 2f81c6b13b..523f4b95a2 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -475,6 +475,7 @@ class CheckCommand(SubCommand):
                             default="checksum", help="check type")
         parser.add_argument("--olderthan", default=60, type=int,
                             help="number of minutes")
+        parser.add_argument("--hint", default="", help="hint to user")
 
     def run(self, args, argv):
         tag = args.tag
@@ -487,7 +488,7 @@ class CheckCommand(SubCommand):
 
         info = dkr.inspect_tag(tag)
         if info is None:
-            print("Image does not exist")
+            print("Image does not exist %s" % (args.hint))
             return 1
 
         if args.checktype == "checksum":
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 14/18] docker: Update debootstrap script after Debian migration from Alioth to Salsa

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

This silents the following warning:

  Cloning into './debootstrap.git'...
  warning: redirecting to https://salsa.debian.org/installer-team/debootstrap.git/

See https://lists.debian.org/debian-devel-announce/2018/01/msg00004.html

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-bootstrap.pre | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index 56e1aa7a21..ea324d6e4a 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -62,7 +62,7 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
         NEED_DEBOOTSTRAP=true
     fi
     if $NEED_DEBOOTSTRAP; then
-        DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
+        DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
         git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
         export DEBOOTSTRAP_DIR=./debootstrap.git
         DEBOOTSTRAP=./debootstrap.git/debootstrap
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 15/18] docker: add commentary to debian-bootstrap.docker

[Alex Bennée]

This is just a note that later versions of debootstrap don't
technically need this hack.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-bootstrap.docker | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
index 14212b9cf4..e13c26a7ed 100644
--- a/tests/docker/dockerfiles/debian-bootstrap.docker
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -9,6 +9,7 @@ FROM scratch
 ADD . /
 
 # Patch all mounts as docker already has stuff set up
+# (this is not needed for later debootstraps but is harmless atm)
 RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
 
 # Run stage 2
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 16/18] docker: ignore distro versioning of debootstrap

[Alex Bennée]

We do a minimum version check for the debootstrap but if the distro
has added their own minor version tick it would fail and fall-back to
the SCM version. This is sub-optimal as the latest/greatest version
may be broken at any one particular time. We fix that with a little
sed magic on the version string before passing to our ugly shell
versioning check.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-bootstrap.pre | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index ea324d6e4a..3b0ef95374 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -56,10 +56,13 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
     if [ -z $DEBOOTSTRAP ]; then
         echo "No debootstrap installed, attempting to install from SCM"
         NEED_DEBOOTSTRAP=true
-    elif ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; "${DEBOOTSTRAP}" --version \
-            | cut -d ' ' -f 2) | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -c &>/dev/null; then
-        echo "debootstrap too old, attempting to install from SCM"
-        NEED_DEBOOTSTRAP=true
+    else
+        INSTALLED_VERSION=$(${DEBOOTSTRAP} --version | sed 's/debootstrap \([0-9\.]*\)[^0-9\.]*.*/\1/')
+        if ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; echo "${INSTALLED_VERSION}") \
+                | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -C ; then
+            echo "debootstrap too old, attempting to install from SCM"
+            NEED_DEBOOTSTRAP=true
+        fi
     fi
     if $NEED_DEBOOTSTRAP; then
         DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 17/18] docker: perform basic binfmt_misc validation in docker.py

[Alex Bennée]

Setting up binfmt_misc is outside of the scope of the docker.py script
but we can at least validate it with any given executable so we have a
more useful error message than the sed line of deboostrap failing
cryptically.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Richard Henderson <richard.henderson@linaro.org>
---
 tests/docker/docker.py | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 523f4b95a2..a3f5b0c1b0 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -112,6 +112,31 @@ def _copy_binary_with_libs(src, dest_dir):
             so_path = os.path.dirname(l)
             _copy_with_mkdir(l , dest_dir, so_path)
 
+
+def _check_binfmt_misc(executable):
+    """Check binfmt_misc has entry for executable in the right place.
+
+    The details of setting up binfmt_misc are outside the scope of
+    this script but we should at least fail early with a useful
+    message if it won't work."""
+
+    binary = os.path.basename(executable)
+    binfmt_entry = "/proc/sys/fs/binfmt_misc/%s" % (binary)
+
+    if not os.path.exists(binfmt_entry):
+        print ("No binfmt_misc entry for %s" % (binary))
+        return False
+
+    with open(binfmt_entry) as x: entry = x.read()
+
+    qpath = "/usr/bin/%s" % (binary)
+    if not re.search("interpreter %s\n" % (qpath), entry):
+        print ("binfmt_misc for %s does not point to %s" % (binary, qpath))
+        return False
+
+    return True
+
+
 def _read_qemu_dockerfile(img_name):
     # special case for Debian linux-user images
     if img_name.startswith("debian") and img_name.endswith("user"):
@@ -315,6 +340,11 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Validate binfmt_misc will work
+            if args.include_executable:
+                if not _check_binfmt_misc(args.include_executable):
+                    return 1
+
             # Is there a .pre file to run in the build context?
             docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
             if os.path.exists(docker_pre):
-- 
2.17.1

[Qemu-devel] [PATCH v3 for 3.0 18/18] tests/tcg: remove runcom test

[Alex Bennée]

The combination of being rather esoteric and needing to support mmap @
0 means this only ever worked under translation. It has now regressed
even further and is no longer useful. Kill it.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/i386/Makefile.target |   5 -
 tests/tcg/i386/README          |   3 -
 tests/tcg/i386/pi_10.com       | Bin 54 -> 0 bytes
 tests/tcg/i386/runcom.c        | 192 ---------------------------------
 4 files changed, 200 deletions(-)
 delete mode 100644 tests/tcg/i386/pi_10.com
 delete mode 100644 tests/tcg/i386/runcom.c

diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target
index c1997a1624..b4033ba3d1 100644
--- a/tests/tcg/i386/Makefile.target
+++ b/tests/tcg/i386/Makefile.target
@@ -29,11 +29,6 @@ test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S test-i386.h test-i386
 	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ \
 	   $(<D)/test-i386.c $(<D)/test-i386-code16.S $(<D)/test-i386-vm86.S -lm
 
-# Specialist test runners
-run-runcom: TIMEOUT=30
-run-runcom: runcom pi_10.com
-	$(call run-test,$<,$(QEMU) ./runcom $(I386_SRC)/pi_10.com,"$< on $(TARGET_NAME)")
-
 ifeq ($(SPEED), slow)
 
 test-i386-fprem.ref: test-i386-fprem
diff --git a/tests/tcg/i386/README b/tests/tcg/i386/README
index 7a0a47bf27..09e88f30dc 100644
--- a/tests/tcg/i386/README
+++ b/tests/tcg/i386/README
@@ -25,9 +25,6 @@ and host CPUs.
 test-i386-fprem
 ---------------
 
-runcom
-------
-
 test-mmap
 ---------
 
diff --git a/tests/tcg/i386/pi_10.com b/tests/tcg/i386/pi_10.com
deleted file mode 100644
index 8993ba1a51bbc2aaddc3aa99ceea37ab3bf9ea9e..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 54
zcmV-60LlM|6SaSi!9cr&CiAO|!E5OPMfdfRtCIKOQkA$004Db(1I1kv)#B#&Gz2h`
MtF!{mAxm}F!x~T=Hvj+t

diff --git a/tests/tcg/i386/runcom.c b/tests/tcg/i386/runcom.c
deleted file mode 100644
index d60342bfc6..0000000000
--- a/tests/tcg/i386/runcom.c
+++ /dev/null
@@ -1,192 +0,0 @@
-/*
- * Simple example of use of vm86: launch a basic .com DOS executable
- */
-#include <stdlib.h>
-#include <stdio.h>
-#include <string.h>
-#include <inttypes.h>
-#include <unistd.h>
-#include <fcntl.h>
-#include <sys/mman.h>
-#include <signal.h>
-
-#include <linux/unistd.h>
-#include <asm/vm86.h>
-
-extern int vm86 (unsigned long int subfunction,
-		 struct vm86plus_struct *info);
-
-#define VIF_MASK                0x00080000
-
-//#define SIGTEST
-
-#define COM_BASE_ADDR    0x10100
-
-static void usage(void)
-{
-    printf("runcom version 0.1 (c) 2003 Fabrice Bellard\n"
-           "usage: runcom file.com\n"
-           "VM86 Run simple .com DOS executables (linux vm86 test mode)\n");
-    exit(1);
-}
-
-static inline void set_bit(uint8_t *a, unsigned int bit)
-{
-    a[bit / 8] |= (1 << (bit % 8));
-}
-
-static inline uint8_t *seg_to_linear(unsigned int seg, unsigned int reg)
-{
-    return (uint8_t *)((seg << 4) + (reg & 0xffff));
-}
-
-static inline void pushw(struct vm86_regs *r, int val)
-{
-    r->esp = (r->esp & ~0xffff) | ((r->esp - 2) & 0xffff);
-    *(uint16_t *)seg_to_linear(r->ss, r->esp) = val;
-}
-
-void dump_regs(struct vm86_regs *r)
-{
-    fprintf(stderr,
-            "EAX=%08lx EBX=%08lx ECX=%08lx EDX=%08lx\n"
-            "ESI=%08lx EDI=%08lx EBP=%08lx ESP=%08lx\n"
-            "EIP=%08lx EFL=%08lx\n"
-            "CS=%04x DS=%04x ES=%04x SS=%04x FS=%04x GS=%04x\n",
-            r->eax, r->ebx, r->ecx, r->edx, r->esi, r->edi, r->ebp, r->esp,
-            r->eip, r->eflags,
-            r->cs, r->ds, r->es, r->ss, r->fs, r->gs);
-}
-
-#ifdef SIGTEST
-void alarm_handler(int sig)
-{
-    fprintf(stderr, "alarm signal=%d\n", sig);
-    alarm(1);
-}
-#endif
-
-int main(int argc, char **argv)
-{
-    uint8_t *vm86_mem;
-    const char *filename;
-    int fd, ret, seg;
-    struct vm86plus_struct ctx;
-    struct vm86_regs *r;
-
-    if (argc != 2)
-        usage();
-    filename = argv[1];
-
-    vm86_mem = mmap((void *)0x00000000, 0x110000,
-                    PROT_WRITE | PROT_READ | PROT_EXEC,
-                    MAP_FIXED | MAP_ANON | MAP_PRIVATE, -1, 0);
-    if (vm86_mem == MAP_FAILED) {
-        perror("mmap");
-        exit(1);
-    }
-#ifdef SIGTEST
-    {
-        struct sigaction act;
-
-        act.sa_handler = alarm_handler;
-        sigemptyset(&act.sa_mask);
-        act.sa_flags = 0;
-        sigaction(SIGALRM, &act, NULL);
-        alarm(1);
-    }
-#endif
-
-    /* load the MSDOS .com executable */
-    fd = open(filename, O_RDONLY);
-    if (fd < 0) {
-        perror(filename);
-        exit(1);
-    }
-    ret = read(fd, vm86_mem + COM_BASE_ADDR, 65536 - 256);
-    if (ret < 0) {
-        perror("read");
-        exit(1);
-    }
-    close(fd);
-
-    memset(&ctx, 0, sizeof(ctx));
-    /* init basic registers */
-    r = &ctx.regs;
-    r->eip = 0x100;
-    r->esp = 0xfffe;
-    seg = (COM_BASE_ADDR - 0x100) >> 4;
-    r->cs = seg;
-    r->ss = seg;
-    r->ds = seg;
-    r->es = seg;
-    r->fs = seg;
-    r->gs = seg;
-    r->eflags = VIF_MASK;
-
-    /* put return code */
-    set_bit((uint8_t *)&ctx.int_revectored, 0x21);
-    *seg_to_linear(r->cs, 0) = 0xb4; /* mov ah, $0 */
-    *seg_to_linear(r->cs, 1) = 0x00;
-    *seg_to_linear(r->cs, 2) = 0xcd; /* int $0x21 */
-    *seg_to_linear(r->cs, 3) = 0x21;
-    pushw(&ctx.regs, 0x0000);
-
-    /* the value of these registers seem to be assumed by pi_10.com */
-    r->esi = 0x100;
-    r->ecx = 0xff;
-    r->ebp = 0x0900;
-    r->edi = 0xfffe;
-
-    for(;;) {
-        ret = vm86(VM86_ENTER, &ctx);
-        switch(VM86_TYPE(ret)) {
-        case VM86_INTx:
-            {
-                int int_num, ah;
-
-                int_num = VM86_ARG(ret);
-                if (int_num != 0x21)
-                    goto unknown_int;
-                ah = (r->eax >> 8) & 0xff;
-                switch(ah) {
-                case 0x00: /* exit */
-                    exit(0);
-                case 0x02: /* write char */
-                    {
-                        uint8_t c = r->edx;
-                        write(1, &c, 1);
-                    }
-                    break;
-                case 0x09: /* write string */
-                    {
-                        uint8_t c;
-                        for(;;) {
-                            c = *seg_to_linear(r->ds, r->edx);
-                            if (c == '$')
-                                break;
-                            write(1, &c, 1);
-                        }
-                        r->eax = (r->eax & ~0xff) | '$';
-                    }
-                    break;
-                default:
-                unknown_int:
-                    fprintf(stderr, "unsupported int 0x%02x\n", int_num);
-                    dump_regs(&ctx.regs);
-                    //                    exit(1);
-                }
-            }
-            break;
-        case VM86_SIGNAL:
-            /* a signal came, we just ignore that */
-            break;
-        case VM86_STI:
-            break;
-        default:
-            fprintf(stderr, "unhandled vm86 return code (0x%x)\n", ret);
-            dump_regs(&ctx.regs);
-            exit(1);
-        }
-    }
-}
-- 
2.17.1

Re: [Qemu-devel] [PATCH v3 for 3.0 00/18] docker fixes (and one tcg test tweak)

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Hi,
>
> I've missed the boat for today's rc1 but I'd like to get this merged
> before rc2. The new docker.py change is technically new functionality
> but I'm counting it as a usability bug fix as it replaces a random
> back trace failure with a preemptive failure and message mentioning
> binfmt_misc configuration. This would have saved Richard a lot of head
> scratching as he tried to setup a powerpc-user setup to test his
> setcontext fix (he had a custom binfmt_misc pointing to his src tree).
>
> Finally we also drop the runcom test. It was cute that it got
> resurrected but it is ultimately a pointless test for something I'm
> sure no one actually uses.
>
> There will be a follow-up RFC series after this that cleans-up some of
> the rough edges when your host is not an x86_64 box but that series
> won't be targeting the 3.0 release.
>
> : The following patches need review
> : patch docker/disable debian powerpc user cross.patch
> : patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
> : patch docker/Update debootstrap script after Debian migrat.patch
> : patch docker/ignore distro versioning of debootstrap.patch
> : patch docker/perform basic binfmt_misc validation in docke.patch
> : patch tests/tcg remove runcom test.patch

Ping?

--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 for 3.0 09/18] docker: disable debian-powerpc-user-cross

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> While we can build we can't run the unit tests due to a missing system
> call (249 - swapcontext). This could be a code generation issue as
> other architectures don't seem to have this system call.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Cc: David Gibson <david@gibson.dropbear.id.au>
> Cc: Alexander Graf <agraf@suse.de>
> Cc: qemu-ppc@nongnu.org

Dropping this patch now rth's setcontext fix is in.

> ---
>  tests/docker/Makefile.include | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 09fb7db7fa..fe63aacf69 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -140,6 +140,9 @@ docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
>  docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
>  docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
>  DOCKER_USER_IMAGES += debian-powerpc-user
> +# Unfortunalty while we can build with this image we can't run unit
> +# tests due to missing system call: 249 which causes an abort
> +DOCKER_PARTIAL_IMAGES += debian-powerpc-user-cross
>
>  # Expand all the pre-requistes for each docker image and test combination
>  $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 for 3.0 13/18] docker: add --hint to docker.py check

[Fam Zheng]

On Wed, Jul 18, 2018 at 4:03 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> When a check fails we currently just report why we failed. This is not
> totally helpful to people who want to boot-strap a new image. Add a
> --hint option which we can pass down to give a bit more information.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include | 3 ++-
>  tests/docker/docker.py        | 3 ++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index ec23620153..c9e412f9d0 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -73,7 +73,8 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
>                         $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
>                         "BUILD","binfmt debian-$* (debootstrapped)"),           \
>                 $(call quiet-command,                                           \
> -                       $(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<,       \
> +                       $(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<        \
> +                       --hint "you will need to build $(EXECUTABLE)",          \
>                         "CHECK", "debian-$* exists"))
>
>  endif
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 2f81c6b13b..523f4b95a2 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -475,6 +475,7 @@ class CheckCommand(SubCommand):
>                              default="checksum", help="check type")
>          parser.add_argument("--olderthan", default=60, type=int,
>                              help="number of minutes")
> +        parser.add_argument("--hint", default="", help="hint to user")
>
>      def run(self, args, argv):
>          tag = args.tag
> @@ -487,7 +488,7 @@ class CheckCommand(SubCommand):
>
>          info = dkr.inspect_tag(tag)
>          if info is None:
> -            print("Image does not exist")
> +            print("Image does not exist %s" % (args.hint))

No, please. This is just hacky and makes no sense to me. The Makefile
can do this

  $(DOCKER_SCRIPT) check ... || { echo "you will need to build
$(EXECUTABLE)"; exit 1 }

Fam

>              return 1
>
>          if args.checktype == "checksum":
> --
> 2.17.1
>

Re: [Qemu-devel] [PATCH v3 for 3.0 17/18] docker: perform basic binfmt_misc validation in docker.py

[Fam Zheng]

On Wed, Jul 18, 2018 at 4:02 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Setting up binfmt_misc is outside of the scope of the docker.py script
> but we can at least validate it with any given executable so we have a
> more useful error message than the sed line of deboostrap failing
> cryptically.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reported-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>  tests/docker/docker.py | 30 ++++++++++++++++++++++++++++++
>  1 file changed, 30 insertions(+)
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 523f4b95a2..a3f5b0c1b0 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -112,6 +112,31 @@ def _copy_binary_with_libs(src, dest_dir):
>              so_path = os.path.dirname(l)
>              _copy_with_mkdir(l , dest_dir, so_path)
>
> +
> +def _check_binfmt_misc(executable):
> +    """Check binfmt_misc has entry for executable in the right place.
> +
> +    The details of setting up binfmt_misc are outside the scope of
> +    this script but we should at least fail early with a useful
> +    message if it won't work."""
> +
> +    binary = os.path.basename(executable)
> +    binfmt_entry = "/proc/sys/fs/binfmt_misc/%s" % (binary)
> +
> +    if not os.path.exists(binfmt_entry):
> +        print ("No binfmt_misc entry for %s" % (binary))
> +        return False
> +
> +    with open(binfmt_entry) as x: entry = x.read()
> +
> +    qpath = "/usr/bin/%s" % (binary)

Is it intended to hardcode this to /usr/bin? I thought we were more
flexible than that..

Fam

> +    if not re.search("interpreter %s\n" % (qpath), entry):
> +        print ("binfmt_misc for %s does not point to %s" % (binary, qpath))
> +        return False
> +
> +    return True
> +
> +
>  def _read_qemu_dockerfile(img_name):
>      # special case for Debian linux-user images
>      if img_name.startswith("debian") and img_name.endswith("user"):
> @@ -315,6 +340,11 @@ class BuildCommand(SubCommand):
>              # Create a docker context directory for the build
>              docker_dir = tempfile.mkdtemp(prefix="docker_build")
>
> +            # Validate binfmt_misc will work
> +            if args.include_executable:
> +                if not _check_binfmt_misc(args.include_executable):
> +                    return 1
> +
>              # Is there a .pre file to run in the build context?
>              docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
>              if os.path.exists(docker_pre):
> --
> 2.17.1
>

Re: [Qemu-devel] [PATCH v3 for 3.0 00/18] docker fixes (and one tcg test tweak)

[Fam Zheng]

On Mon, Jul 23, 2018 at 6:03 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Alex Bennée <alex.bennee@linaro.org> writes:
>
> > Hi,
> >
> > I've missed the boat for today's rc1 but I'd like to get this merged
> > before rc2. The new docker.py change is technically new functionality
> > but I'm counting it as a usability bug fix as it replaces a random
> > back trace failure with a preemptive failure and message mentioning
> > binfmt_misc configuration. This would have saved Richard a lot of head
> > scratching as he tried to setup a powerpc-user setup to test his
> > setcontext fix (he had a custom binfmt_misc pointing to his src tree).
> >
> > Finally we also drop the runcom test. It was cute that it got
> > resurrected but it is ultimately a pointless test for something I'm
> > sure no one actually uses.
> >
> > There will be a follow-up RFC series after this that cleans-up some of
> > the rough edges when your host is not an x86_64 box but that series
> > won't be targeting the 3.0 release.
> >
> > : The following patches need review
> > : patch docker/disable debian powerpc user cross.patch
> > : patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
> > : patch docker/Update debootstrap script after Debian migrat.patch
> > : patch docker/ignore distro versioning of debootstrap.patch
> > : patch docker/perform basic binfmt_misc validation in docke.patch
> > : patch tests/tcg remove runcom test.patch
>
> Ping?

I had questions about patch 13 and 17. Otherwise looks good.
​ Maybe you can drop them for now (including patch 9) and send the PULL for
the rest.​
I don't know if we can catch up -rc2 but I guess the testing fixes are
fairly safe to sneak in. :)

(My development machine is affected by
an
 office power outage, and I was
​ also​
 busy with upgrading patchew.org. Sorry for the
​ ​
late reply
!)

Fam

Re: [Qemu-devel] [PATCH v3 for 3.0 13/18] docker: add --hint to docker.py check

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Wed, Jul 18, 2018 at 4:03 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> When a check fails we currently just report why we failed. This is not
>> totally helpful to people who want to boot-strap a new image. Add a
>> --hint option which we can pass down to give a bit more information.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/Makefile.include | 3 ++-
>>  tests/docker/docker.py        | 3 ++-
>>  2 files changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index ec23620153..c9e412f9d0 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -73,7 +73,8 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
>>                         $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
>>                         "BUILD","binfmt debian-$* (debootstrapped)"),           \
>>                 $(call quiet-command,                                           \
>> -                       $(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<,       \
>> +                       $(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<        \
>> +                       --hint "you will need to build $(EXECUTABLE)",          \
>>                         "CHECK", "debian-$* exists"))
>>
>>  endif
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index 2f81c6b13b..523f4b95a2 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -475,6 +475,7 @@ class CheckCommand(SubCommand):
>>                              default="checksum", help="check type")
>>          parser.add_argument("--olderthan", default=60, type=int,
>>                              help="number of minutes")
>> +        parser.add_argument("--hint", default="", help="hint to user")
>>
>>      def run(self, args, argv):
>>          tag = args.tag
>> @@ -487,7 +488,7 @@ class CheckCommand(SubCommand):
>>
>>          info = dkr.inspect_tag(tag)
>>          if info is None:
>> -            print("Image does not exist")
>> +            print("Image does not exist %s" % (args.hint))
>
> No, please. This is just hacky and makes no sense to me. The Makefile
> can do this
>
>   $(DOCKER_SCRIPT) check ... || { echo "you will need to build
> $(EXECUTABLE)"; exit 1 }

OK I'll do it in the Makefile.

>
> Fam
>
>>              return 1
>>
>>          if args.checktype == "checksum":
>> --
>> 2.17.1
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 for 3.0 17/18] docker: perform basic binfmt_misc validation in docker.py

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Wed, Jul 18, 2018 at 4:02 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Setting up binfmt_misc is outside of the scope of the docker.py script
>> but we can at least validate it with any given executable so we have a
>> more useful error message than the sed line of deboostrap failing
>> cryptically.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Reported-by: Richard Henderson <richard.henderson@linaro.org>
>> ---
>>  tests/docker/docker.py | 30 ++++++++++++++++++++++++++++++
>>  1 file changed, 30 insertions(+)
>>
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index 523f4b95a2..a3f5b0c1b0 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -112,6 +112,31 @@ def _copy_binary_with_libs(src, dest_dir):
>>              so_path = os.path.dirname(l)
>>              _copy_with_mkdir(l , dest_dir, so_path)
>>
>> +
>> +def _check_binfmt_misc(executable):
>> +    """Check binfmt_misc has entry for executable in the right place.
>> +
>> +    The details of setting up binfmt_misc are outside the scope of
>> +    this script but we should at least fail early with a useful
>> +    message if it won't work."""
>> +
>> +    binary = os.path.basename(executable)
>> +    binfmt_entry = "/proc/sys/fs/binfmt_misc/%s" % (binary)
>> +
>> +    if not os.path.exists(binfmt_entry):
>> +        print ("No binfmt_misc entry for %s" % (binary))
>> +        return False
>> +
>> +    with open(binfmt_entry) as x: entry = x.read()
>> +
>> +    qpath = "/usr/bin/%s" % (binary)
>
> Is it intended to hardcode this to /usr/bin? I thought we were more
> flexible than that..

That's where we currently copy it. However we are certainly capable of
being more flexible. For example as long at there is an entry visible to
the host file-system with the "F" flag we can actually create and run
docker images without copying QEMU inside the container.

However if we don't we certainly need to copy it to the same place in
the container than it is used outside the container.

>
> Fam
>
>> +    if not re.search("interpreter %s\n" % (qpath), entry):
>> +        print ("binfmt_misc for %s does not point to %s" % (binary, qpath))
>> +        return False
>> +
>> +    return True
>> +
>> +
>>  def _read_qemu_dockerfile(img_name):
>>      # special case for Debian linux-user images
>>      if img_name.startswith("debian") and img_name.endswith("user"):
>> @@ -315,6 +340,11 @@ class BuildCommand(SubCommand):
>>              # Create a docker context directory for the build
>>              docker_dir = tempfile.mkdtemp(prefix="docker_build")
>>
>> +            # Validate binfmt_misc will work
>> +            if args.include_executable:
>> +                if not _check_binfmt_misc(args.include_executable):
>> +                    return 1
>> +
>>              # Is there a .pre file to run in the build context?
>>              docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
>>              if os.path.exists(docker_pre):
>> --
>> 2.17.1
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 for 3.0 00/18] docker fixes (and one tcg test tweak)

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Mon, Jul 23, 2018 at 6:03 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>
>> > Hi,
>> >
>> > I've missed the boat for today's rc1 but I'd like to get this merged
>> > before rc2. The new docker.py change is technically new functionality
>> > but I'm counting it as a usability bug fix as it replaces a random
>> > back trace failure with a preemptive failure and message mentioning
>> > binfmt_misc configuration. This would have saved Richard a lot of head
>> > scratching as he tried to setup a powerpc-user setup to test his
>> > setcontext fix (he had a custom binfmt_misc pointing to his src tree).
>> >
>> > Finally we also drop the runcom test. It was cute that it got
>> > resurrected but it is ultimately a pointless test for something I'm
>> > sure no one actually uses.
>> >
>> > There will be a follow-up RFC series after this that cleans-up some of
>> > the rough edges when your host is not an x86_64 box but that series
>> > won't be targeting the 3.0 release.
>> >
>> > : The following patches need review
>> > : patch docker/disable debian powerpc user cross.patch
>> > : patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
>> > : patch docker/Update debootstrap script after Debian migrat.patch
>> > : patch docker/ignore distro versioning of debootstrap.patch
>> > : patch docker/perform basic binfmt_misc validation in docke.patch
>> > : patch tests/tcg remove runcom test.patch
>>
>> Ping?
>
> I had questions about patch 13 and 17. Otherwise looks good.
> ​ Maybe you can drop them for now (including patch 9) and send the PULL for
> the rest.​

I can drop/replace 13 - but 17 was there for a better reporting of a
failure case Peter found. We can certainly expand it and be smarter in
future iterations.

> I don't know if we can catch up -rc2 but I guess the testing fixes are
> fairly safe to sneak in. :)
>
> (My development machine is affected by
> an
>  office power outage, and I was
> ​ also​
>  busy with upgrading patchew.org. Sorry for the
> ​ ​
> late reply
> !)
>
> Fam


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 for 3.0 00/18] docker fixes (and one tcg test tweak)

[Fam Zheng]

On Tue, Jul 24, 2018 at 4:56 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Fam Zheng <famz@redhat.com> writes:
>
> > On Mon, Jul 23, 2018 at 6:03 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >>
> >>
> >> Alex Bennée <alex.bennee@linaro.org> writes:
> >>
> >> > Hi,
> >> >
> >> > I've missed the boat for today's rc1 but I'd like to get this merged
> >> > before rc2. The new docker.py change is technically new functionality
> >> > but I'm counting it as a usability bug fix as it replaces a random
> >> > back trace failure with a preemptive failure and message mentioning
> >> > binfmt_misc configuration. This would have saved Richard a lot of head
> >> > scratching as he tried to setup a powerpc-user setup to test his
> >> > setcontext fix (he had a custom binfmt_misc pointing to his src tree).
> >> >
> >> > Finally we also drop the runcom test. It was cute that it got
> >> > resurrected but it is ultimately a pointless test for something I'm
> >> > sure no one actually uses.
> >> >
> >> > There will be a follow-up RFC series after this that cleans-up some of
> >> > the rough edges when your host is not an x86_64 box but that series
> >> > won't be targeting the 3.0 release.
> >> >
> >> > : The following patches need review
> >> > : patch docker/disable debian powerpc user cross.patch
> >> > : patch docker/drop QEMU_TARGET check fallback in EXECUTABLE.patch
> >> > : patch docker/Update debootstrap script after Debian migrat.patch
> >> > : patch docker/ignore distro versioning of debootstrap.patch
> >> > : patch docker/perform basic binfmt_misc validation in docke.patch
> >> > : patch tests/tcg remove runcom test.patch
> >>
> >> Ping?
> >
> > I had questions about patch 13 and 17. Otherwise looks good.
> > Maybe you can drop them for now (including patch 9) and send the PULL for
> > the rest.
>
> I can drop/replace 13 - but 17 was there for a better reporting of a
> failure case Peter found. We can certainly expand it and be smarter in
> future iterations.


Sounds good to me.

Fam

>
>
> > I don't know if we can catch up -rc2 but I guess the testing fixes are
> > fairly safe to sneak in. :)
> >
> > (My development machine is affected by
> > an
> >  office power outage, and I was
> > also
> >  busy with upgrading patchew.org. Sorry for the
> >
> > late reply
> > !)
> >
> > Fam
>
>
> --
> Alex Bennée

Re: [Qemu-devel] [PATCH v3 for 3.0 18/18] tests/tcg: remove runcom test

[Peter Maydell]

On 17 July 2018 at 20:55, Alex Bennée <alex.bennee@linaro.org> wrote:
> The combination of being rather esoteric and needing to support mmap @
> 0 means this only ever worked under translation. It has now regressed
> even further and is no longer useful. Kill it.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

thanks
-- PMM