[Qemu-devel] [PATCH v3 00/20] Travis, Code Coverage and Cross Build updates

[Qemu-devel] [PATCH v3 00/20] Travis, Code Coverage and Cross Build updates

[Alex Bennée]

Hi,

Friday's code dump suffered a little from a rushed posting before the
weekend so I spent this morning cleaning up a few of the rough edges.
This version is much better for it:

  - Dropped .shippable changes

In the end it looks like building the ppc image won't work in
QEMU as it's pre-ci environment hasn't enough in it to build the
qemu-ppc linux user we need.

  - Better boostrap rule naming

The rule is now docker-binfmt-image-debian-% which means you can now
build any arbitrarily named debian-% image. If we are going support
similar binfmt bootstrapping for other distributions I suggest we
follow a similar scheme. Does anyone know if anything as funky as
debootstrap exists for anything else?

With a minor tweak to the debootstrap helper script you can now
boostrap Ubuntu images as well.

  - Dropped docker.py env hack (not needed)
  - Removed stray whitespace changes

The rest works as I outlined in the v2 message:

  I've picked up some more of Philippe's fixes and in the process had
  another look at the problem of cross compiling powerpc. In the end the
  consensus seems to be a linux-user build was the best solution as
  there are still supported native powerpc toolchains in Jessie (until
  2020). So I dusted off my old linux-user patches and tries to plumb
  them in a little better. Assuming you have ppc-linux-user in your
  target list and it's a --static you can just run:

    docker-test-build@debian-powerpc-user-cross TARGET_LIST=ppc-softmmu

  And it does the right thing. Once the debian-powerpc-user base image
  is created you can change your configuration and things will still
  work (until the debian-bootstrap recipe is changed and needs to be
  re-built).

It would be great if I could get reviews for the final few patches:

  patch 0011/docker filter out linux user builds for mingw.patch
  patch 0016/docker drop QEMU build dep from bootstrap.patch
  patch 0017/docker debian bootstrap.pre allow customising of .patch
  patch 0018/docker add special handling for FROM debian user .patch
  patch 0019/docker add special rule for deboostrapped images.patch
  patch 0020/docker add linux user powered cross builder for Q.patch

I'll send a pull request tomorrow for soft-freeze.

Alex Bennée (14):
  build-system: remove per-test GCOV reporting
  .gitignore: add .gcov files
  docker: add gcovr to travis image
  travis: add gcovr summary for GCOV build
  build-system: add clean-coverage target
  build-system: add coverage-report target
  linux-user: introduce preexit_cleanup
  linux-user: add gcov support to preexit_cleanup
  docker: filter out linux-user builds for mingw
  docker: drop QEMU build-dep from bootstrap
  docker: debian-bootstrap.pre allow customising of variant/url
  docker: add special handling for FROM:debian-%-user targets
  docker: add special rule for deboostrapped images
  docker: add linux-user powered cross builder for QEMU

Philippe Mathieu-Daudé (6):
  travis: do not waste time cloning unused submodules
  travis: test out-of-tree builds
  docker: ubuntu: Update the package list before installing new ones
  docker: ubuntu: Use SDL2
  docker: Clean the MXE base image
  docker: Do not run tests in 'intermediate' images

 .gitignore                                    |  1 +
 .travis.yml                                   | 14 +++-
 MAINTAINERS                                   |  1 +
 Makefile                                      | 24 +++++++
 docs/devel/testing.rst                        | 21 ++++--
 linux-user/Makefile.objs                      |  2 +-
 linux-user/exit.c                             | 35 ++++++++++
 linux-user/qemu.h                             |  8 +++
 linux-user/syscall.c                          | 10 +--
 scripts/travis/coverage-summary.sh            | 27 +++++++
 tests/Makefile.include                        | 10 ---
 tests/docker/Makefile.include                 | 70 +++++++++++++++++--
 tests/docker/docker.py                        |  4 ++
 .../dockerfiles/debian-bootstrap.docker       |  2 -
 tests/docker/dockerfiles/debian-bootstrap.pre | 11 ++-
 .../debian-powerpc-user-cross.docker          | 13 ++++
 tests/docker/dockerfiles/debian8-mxe.docker   |  2 +-
 tests/docker/dockerfiles/travis.docker        |  2 +-
 tests/docker/dockerfiles/ubuntu.docker        |  8 +--
 19 files changed, 222 insertions(+), 43 deletions(-)
 create mode 100644 linux-user/exit.c
 create mode 100755 scripts/travis/coverage-summary.sh
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-user-cross.docker

-- 
2.17.1

[Qemu-devel] [PATCH v3 01/20] travis: do not waste time cloning unused submodules

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Builds only require:
- dtc
- keycodemapdb
- capstone

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
[AJB: drop wget cache]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .travis.yml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index bd66c18fed..134d5331fe 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -66,8 +66,7 @@ git:
 before_install:
   - if [ "$TRAVIS_OS_NAME" == "osx" ]; then brew update ; fi
   - if [ "$TRAVIS_OS_NAME" == "osx" ]; then brew install libffi gettext glib pixman ; fi
-  - wget -O - http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz | tar -xvJ
-  - git submodule update --init --recursive
+  - git submodule update --init --recursive capstone dtc ui/keycodemapdb
 before_script:
   - ./configure ${CONFIG} || { cat config.log && exit 1; }
 script:
-- 
2.17.1

[Qemu-devel] [PATCH v3 02/20] travis: test out-of-tree builds

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Force one config to build 'out-of-tree' (object files and executables
are created in a tree outside the project source code).

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .travis.yml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 134d5331fe..32188d51f1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -50,6 +50,8 @@ notifications:
     on_failure: always
 env:
   global:
+    - SRC_DIR="."
+    - BUILD_DIR="."
     - TEST_CMD="make check"
     - MAKEFLAGS="-j3"
   matrix:
@@ -68,11 +70,15 @@ before_install:
   - if [ "$TRAVIS_OS_NAME" == "osx" ]; then brew install libffi gettext glib pixman ; fi
   - git submodule update --init --recursive capstone dtc ui/keycodemapdb
 before_script:
-  - ./configure ${CONFIG} || { cat config.log && exit 1; }
+  - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
+  - ${SRC_DIR}/configure ${CONFIG} || { cat config.log && exit 1; }
 script:
   - make ${MAKEFLAGS} && ${TEST_CMD}
 matrix:
   include:
+    # Test out-of-tree builds
+    - env: CONFIG="--enable-debug --enable-debug-tcg"
+           BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
     # Test with Clang for compile portability (Travis uses clang-5.0)
     - env: CONFIG="--disable-system"
       compiler: clang
-- 
2.17.1

[Qemu-devel] [PATCH v3 03/20] build-system: remove per-test GCOV reporting

[Alex Bennée]

I'm not entirely sure who's using this information and certainly in a
CI environment it just washes over as additional noise. Later patches
will provide new reporting options so a user who wants to analyse
individual tests will be able to use that to get the information.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/devel/testing.rst | 11 +++++------
 tests/Makefile.include | 10 ----------
 2 files changed, 5 insertions(+), 16 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index f33e5a8423..66ef219f69 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -158,12 +158,11 @@ rarely used. See "QEMU iotests" section below for more information.
 GCC gcov support
 ----------------
 
-``gcov`` is a GCC tool to analyze the testing coverage by instrumenting the
-tested code. To use it, configure QEMU with ``--enable-gcov`` option and build.
-Then run ``make check`` as usual. There will be additional ``gcov`` output as
-the testing goes on, showing the test coverage percentage numbers per analyzed
-source file. More detailed reports can be obtained by running ``gcov`` command
-on the output files under ``$build_dir/tests/``, please read the ``gcov``
+``gcov`` is a GCC tool to analyze the testing coverage by
+instrumenting the tested code. To use it, configure QEMU with
+``--enable-gcov`` option and build. Then run ``make check`` as usual.
+Reports can be obtained by running ``gcov`` command on the output
+files under ``$build_dir/tests/``, please read the ``gcov``
 documentation for more information.
 
 QEMU iotests
diff --git a/tests/Makefile.include b/tests/Makefile.include
index e8bb2d8f66..756474814a 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -891,26 +891,16 @@ GCOV_OPTIONS = -n $(if $(V),-f,)
 
 .PHONY: $(patsubst %, check-qtest-%, $(QTEST_TARGETS))
 $(patsubst %, check-qtest-%, $(QTEST_TARGETS)): check-qtest-%: subdir-%-softmmu $(check-qtest-y)
-	$(if $(CONFIG_GCOV),@rm -f *.gcda */*.gcda */*/*.gcda */*/*/*.gcda,)
 	$(call quiet-command,QTEST_QEMU_BINARY=$*-softmmu/qemu-system-$* \
 		QTEST_QEMU_IMG=qemu-img$(EXESUF) \
 		MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \
 		gtester $(GTESTER_OPTIONS) -m=$(SPEED) $(check-qtest-$*-y) $(check-qtest-generic-y),"GTESTER","$@")
-	$(if $(CONFIG_GCOV),@for f in $(gcov-files-$*-y) $(gcov-files-generic-y); do \
-	  echo Gcov report for $$f:;\
-	  $(GCOV) $(GCOV_OPTIONS) $$f -o `dirname $$f`; \
-	done,)
 
 .PHONY: $(patsubst %, check-%, $(check-unit-y) $(check-speed-y))
 $(patsubst %, check-%, $(check-unit-y) $(check-speed-y)): check-%: %
-	$(if $(CONFIG_GCOV),@rm -f *.gcda */*.gcda */*/*.gcda */*/*/*.gcda,)
 	$(call quiet-command, \
 		MALLOC_PERTURB_=$${MALLOC_PERTURB_:-$$(( $${RANDOM:-0} % 255 + 1))} \
 		gtester $(GTESTER_OPTIONS) -m=$(SPEED) $*,"GTESTER","$*")
-	$(if $(CONFIG_GCOV),@for f in $(gcov-files-$(subst tests/,,$*)-y) $(gcov-files-generic-y); do \
-	  echo Gcov report for $$f:;\
-	  $(GCOV) $(GCOV_OPTIONS) $$f -o `dirname $$f`; \
-	done,)
 
 # gtester tests with XML output
 
-- 
2.17.1

[Qemu-devel] [PATCH v3 04/20] .gitignore: add .gcov files

[Alex Bennée]

These are temporary files generated on gcov runs and shouldn't be
included in the source tree.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 9da3b3e626..5668d02782 100644
--- a/.gitignore
+++ b/.gitignore
@@ -155,6 +155,7 @@
 .sdk
 *.gcda
 *.gcno
+*.gcov
 /pc-bios/bios-pq/status
 /pc-bios/vgabios-pq/status
 /pc-bios/optionrom/linuxboot.asm
-- 
2.17.1

[Qemu-devel] [PATCH v3 05/20] docker: add gcovr to travis image

[Alex Bennée]

Useful for debugging if nothing else as the gcovr on the Travis images
are a little old.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/docker/dockerfiles/travis.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
index c5ad39b533..03ebfb0ef2 100644
--- a/tests/docker/dockerfiles/travis.docker
+++ b/tests/docker/dockerfiles/travis.docker
@@ -5,7 +5,7 @@ ENV LC_ALL en_US.UTF-8
 RUN cat /etc/apt/sources.list | sed "s/# deb-src/deb-src/" >> /etc/apt/sources.list
 RUN apt-get update
 RUN apt-get -y build-dep qemu
-RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools
+RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools gcovr
 # Travis tools require PhantomJS / Neo4j / Maven accessible
 # in their PATH (QEMU build won't access them).
 ENV PATH /usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-- 
2.17.1

[Qemu-devel] [PATCH v3 06/20] travis: add gcovr summary for GCOV build

[Alex Bennée]

This gives a more useful summary, sorted by descending % coverage,
after the tests have run. The final numbers will give an idea if our
coverage is getting better or worse.

To keep the width sane we need to post process the file that the old
gcovr tool generates. This is done with a mix of sed, awk and column
in the scripts/coverage-summary.sh script.

As quite a lot of lines don't get covered at all we filter out all the
0% lines. If the file doesn't appear it is not being exercised.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
v2
  - pushed finagling into scripts/coverage-summary.sh
    - dropped r-b as a big change
v3
  - move from scripts to scripts/travis and update MAINTAINERS
---
 .travis.yml                        |  3 +++
 MAINTAINERS                        |  1 +
 scripts/travis/coverage-summary.sh | 27 +++++++++++++++++++++++++++
 3 files changed, 31 insertions(+)
 create mode 100755 scripts/travis/coverage-summary.sh

diff --git a/.travis.yml b/.travis.yml
index 32188d51f1..95be6ec59f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -38,6 +38,7 @@ addons:
       - libvte-2.90-dev
       - sparse
       - uuid-dev
+      - gcovr
 
 # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
 # to prevent IRC notifications from forks. This was created using:
@@ -86,6 +87,8 @@ matrix:
       compiler: clang
     # gprof/gcov are GCC features
     - env: CONFIG="--enable-gprof --enable-gcov --disable-pie --target-list=aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
+      after_success:
+        - ${SRC_DIR}/scripts/travis/coverage-summary.sh
       compiler: gcc
     # We manually include builds which we disable "make check" for
     - env: CONFIG="--enable-debug --enable-tcg-interpreter"
diff --git a/MAINTAINERS b/MAINTAINERS
index 42a1892d6a..4917b8e48d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2165,6 +2165,7 @@ R: Philippe Mathieu-Daudé <f4bug@amsat.org>
 L: qemu-devel@nongnu.org
 S: Maintained
 F: .travis.yml
+F: scripts/travis/
 F: .shippable.yml
 F: tests/docker/
 F: tests/vm/
diff --git a/scripts/travis/coverage-summary.sh b/scripts/travis/coverage-summary.sh
new file mode 100755
index 0000000000..d7086cf9ca
--- /dev/null
+++ b/scripts/travis/coverage-summary.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+#
+# Author: Alex Bennée <alex.bennee@linaro.org>
+#
+# Summerise the state of code coverage with gcovr and tweak the output
+# to be more sane on Travis hosts. As we expect to be executed on a
+# throw away CI instance we do spam temp files all over the shop. You
+# most likely don't want to execute this script but just call gcovr
+# directly. See also "make coverage-report"
+#
+# This code is licensed under the GPL version 2 or later.  See
+# the COPYING file in the top-level directory.
+
+# first generate the coverage report
+gcovr -p -o raw-report.txt
+
+# strip the full-path and line markers
+sed s@$PWD\/@@ raw-report.txt | sed s/[0-9]\*[,-]//g > simplified.txt
+
+# reflow lines that got split
+awk '/.[ch]$/ { printf("%s", $0); next } 1' simplified.txt > rejoined.txt
+
+# columnify
+column -t rejoined.txt > final.txt
+
+# and dump, stripping out 0% coverage
+grep -v "0%" final.txt
-- 
2.17.1

[Qemu-devel] [PATCH v3 07/20] build-system: add clean-coverage target

[Alex Bennée]

This can be used to remove any stale coverage data before any
particular test run. This is useful for analysing individual tests.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---

v2
  - s/to any/to delete any/
---
 Makefile               | 11 +++++++++++
 docs/devel/testing.rst | 11 ++++++++---
 2 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 7ed9cc4a21..2b3413a5ba 100644
--- a/Makefile
+++ b/Makefile
@@ -723,6 +723,14 @@ module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
 	$(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
 	"GEN","$@")
 
+ifdef CONFIG_GCOV
+.PHONY: clean-coverage
+clean-coverage:
+	$(call quiet-command, \
+		find . \( -name '*.gcda' -o -name '*.gcov' \) -type f -exec rm {} +, \
+		"CLEAN", "coverage files")
+endif
+
 clean:
 # avoid old build problems by removing potentially incorrect old files
 	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
@@ -1073,6 +1081,9 @@ endif
 		echo '')
 	@echo  'Cleaning targets:'
 	@echo  '  clean           - Remove most generated files but keep the config'
+ifdef CONFIG_GCOV
+	@echo  '  clean-coverage  - Remove coverage files'
+endif
 	@echo  '  distclean       - Remove all generated files'
 	@echo  '  dist            - Build a distributable tarball'
 	@echo  ''
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 66ef219f69..7f04ca104e 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -161,9 +161,14 @@ GCC gcov support
 ``gcov`` is a GCC tool to analyze the testing coverage by
 instrumenting the tested code. To use it, configure QEMU with
 ``--enable-gcov`` option and build. Then run ``make check`` as usual.
-Reports can be obtained by running ``gcov`` command on the output
-files under ``$build_dir/tests/``, please read the ``gcov``
-documentation for more information.
+
+If you want to gather coverage information on a single test the ``make
+clean-coverage`` target can be used to delete any existing coverage
+information before running a single test.
+
+Reports can be obtained by running ``gcov`` command
+on the output files under ``$build_dir/tests/``, please read the
+``gcov`` documentation for more information.
 
 QEMU iotests
 ============
-- 
2.17.1

[Qemu-devel] [PATCH v3 08/20] build-system: add coverage-report target

[Alex Bennée]

This will build a coverage report under the current directory in
reports/coverage. At the users option a report can be generated by
directly invoking something like:

  make foo/bar/coverage-report.html

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
v2
  - s/generate/create/
---
 Makefile               | 13 +++++++++++++
 docs/devel/testing.rst | 11 ++++++++---
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 2b3413a5ba..68af7b5d7c 100644
--- a/Makefile
+++ b/Makefile
@@ -986,6 +986,16 @@ docs/interop/qemu-qmp-ref.dvi docs/interop/qemu-qmp-ref.html \
     docs/interop/qemu-qmp-ref.txt docs/interop/qemu-qmp-ref.7: \
 	docs/interop/qemu-qmp-ref.texi docs/interop/qemu-qmp-qapi.texi
 
+# Reports/Analysis
+
+%/coverage-report.html:
+	@mkdir -p $*
+	$(call quiet-command,\
+		gcovr -p --html --html-details -o $@, \
+		"GEN", "coverage-report.html")
+
+.PHONY: coverage-report
+coverage-report: $(CURDIR)/reports/coverage/coverage-report.html
 
 ifdef CONFIG_WIN32
 
@@ -1095,6 +1105,9 @@ endif
 	@echo  'Documentation targets:'
 	@echo  '  html info pdf txt'
 	@echo  '                  - Build documentation in specified format'
+ifdef CONFIG_GCOV
+	@echo  '  coverage-report - Create code coverage report'
+endif
 	@echo  ''
 ifdef CONFIG_WIN32
 	@echo  'Windows targets:'
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 7f04ca104e..5e19cd50da 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -166,9 +166,14 @@ If you want to gather coverage information on a single test the ``make
 clean-coverage`` target can be used to delete any existing coverage
 information before running a single test.
 
-Reports can be obtained by running ``gcov`` command
-on the output files under ``$build_dir/tests/``, please read the
-``gcov`` documentation for more information.
+You can generate a HTML coverage report by executing ``make
+coverage-report`` which will create
+./reports/coverage/coverage-report.html. If you want to create it
+elsewhere simply execute ``make /foo/bar/baz/coverage-report.html``.
+
+Further analysis can be conducted by running the ``gcov`` command
+directly on the various .gcda output files. Please read the ``gcov``
+documentation for more information.
 
 QEMU iotests
 ============
-- 
2.17.1

[Qemu-devel] [PATCH v3 09/20] linux-user: introduce preexit_cleanup

[Alex Bennée]

To avoid repeating ourselves move our preexit clean-up code into a
helper function. I figured the continuing effort to split of the
syscalls made it worthwhile creating a new file for it now.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Laurent Vivier <laurent@vivier.eu>
---
 linux-user/Makefile.objs |  2 +-
 linux-user/exit.c        | 28 ++++++++++++++++++++++++++++
 linux-user/qemu.h        |  8 ++++++++
 linux-user/syscall.c     | 10 ++--------
 4 files changed, 39 insertions(+), 9 deletions(-)
 create mode 100644 linux-user/exit.c

diff --git a/linux-user/Makefile.objs b/linux-user/Makefile.objs
index 59a5c17354..b5dfb71f25 100644
--- a/linux-user/Makefile.objs
+++ b/linux-user/Makefile.objs
@@ -1,7 +1,7 @@
 obj-y = main.o syscall.o strace.o mmap.o signal.o \
 	elfload.o linuxload.o uaccess.o uname.o \
 	safe-syscall.o $(TARGET_ABI_DIR)/signal.o \
-        $(TARGET_ABI_DIR)/cpu_loop.o
+        $(TARGET_ABI_DIR)/cpu_loop.o exit.o
 
 obj-$(TARGET_HAS_BFLT) += flatload.o
 obj-$(TARGET_I386) += vm86.o
diff --git a/linux-user/exit.c b/linux-user/exit.c
new file mode 100644
index 0000000000..aed8713fae
--- /dev/null
+++ b/linux-user/exit.c
@@ -0,0 +1,28 @@
+/*
+ *  exit support for qemu
+ *
+ *  Copyright (c) 2018 Alex Bennée <alex.bennee@linaro.org>
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+#include "qemu/osdep.h"
+#include "qemu.h"
+
+void preexit_cleanup(CPUArchState *env, int code)
+{
+#ifdef TARGET_GPROF
+        _mcleanup();
+#endif
+        gdb_exit(env, code);
+}
diff --git a/linux-user/qemu.h b/linux-user/qemu.h
index 793cd4df04..bb85c81aa4 100644
--- a/linux-user/qemu.h
+++ b/linux-user/qemu.h
@@ -623,6 +623,14 @@ static inline int is_error(abi_long ret)
     return (abi_ulong)ret >= (abi_ulong)(-4096);
 }
 
+/**
+ * preexit_cleanup: housekeeping before the guest exits
+ *
+ * env: the CPU state
+ * code: the exit code
+ */
+void preexit_cleanup(CPUArchState *env, int code);
+
 /* Include target-specific struct and function definitions;
  * they may need access to the target-independent structures
  * above, so include them last.
diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index 2117fb13b4..7c66442357 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -8018,10 +8018,7 @@ abi_long do_syscall(void *cpu_env, int num, abi_long arg1,
         }
 
         cpu_list_unlock();
-#ifdef TARGET_GPROF
-        _mcleanup();
-#endif
-        gdb_exit(cpu_env, arg1);
+        preexit_cleanup(cpu_env, arg1);
         _exit(arg1);
         ret = 0; /* avoid warning */
         break;
@@ -10127,10 +10124,7 @@ abi_long do_syscall(void *cpu_env, int num, abi_long arg1,
 #ifdef __NR_exit_group
         /* new thread calls */
     case TARGET_NR_exit_group:
-#ifdef TARGET_GPROF
-        _mcleanup();
-#endif
-        gdb_exit(cpu_env, arg1);
+        preexit_cleanup(cpu_env, arg1);
         ret = get_errno(exit_group(arg1));
         break;
 #endif
-- 
2.17.1

[Qemu-devel] [PATCH v3 10/20] linux-user: add gcov support to preexit_cleanup

[Alex Bennée]

As we don't always take the normal exit path when running a guest we
can skip the normal exit destructors where gcov normally dumps it's
info. The GCC manual suggests long running programs use __gcov_dump()
to flush out the coverage state periodically so we use that here.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 linux-user/exit.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/linux-user/exit.c b/linux-user/exit.c
index aed8713fae..14e94e28fa 100644
--- a/linux-user/exit.c
+++ b/linux-user/exit.c
@@ -19,10 +19,17 @@
 #include "qemu/osdep.h"
 #include "qemu.h"
 
+#ifdef CONFIG_GCOV
+extern void __gcov_dump(void);
+#endif
+
 void preexit_cleanup(CPUArchState *env, int code)
 {
 #ifdef TARGET_GPROF
         _mcleanup();
+#endif
+#ifdef CONFIG_GCOV
+        __gcov_dump();
 #endif
         gdb_exit(env, code);
 }
-- 
2.17.1

[Qemu-devel] [PATCH v3 11/20] docker: filter out linux-user builds for mingw

[Alex Bennée]

The recent change from TARGET_DIRS to TARGET_LIST (208ecb3e1) had the
effect of defaulting all docker builds to the current configured set
of targets. This is actually reasonable behaviour but does run into
problems if you have linux-user builds configured and you want to test
the windows cross builds. This commit fixes that by adding a
DOCKER_FILTER_TARGETS variable which is special-cased for mingw builds
so we don't pass the whole set down.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 91d9665517..1813ec0781 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -20,6 +20,9 @@ DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py
 TESTS ?= %
 IMAGES ?= %
 
+# This is used to filter targets from some docker builds
+DOCKER_FILTER_TARGETS ?=
+
 CUR_TIME := $(shell date +%Y-%m-%d-%H.%M.%S.$$$$)
 DOCKER_SRC_COPY := $(BUILD_DIR)/docker-src.$(CUR_TIME)
 
@@ -108,6 +111,12 @@ $(foreach i,$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES), \
 	) \
 )
 
+# Special cases
+#  mingw/windows builds cannot build linux-user
+docker-%-win32-cross: DOCKER_FILTER_TARGETS = %-linux-user
+docker-%-win64-cross: DOCKER_FILTER_TARGETS = %-linux-user
+docker-test-mingw@%: DOCKER_FILTER_TARGETS = %-linux-user
+
 docker:
 	@echo 'Build QEMU and run tests inside Docker containers'
 	@echo
@@ -174,7 +183,7 @@ docker-run: docker-qemu-src
 			$(if $V,,--rm) 					\
 			$(if $(DEBUG),-ti,)				\
 			$(if $(NETWORK),$(if $(subst $(NETWORK),,1),--net=$(NETWORK)),--net=none) \
-			-e TARGET_LIST=$(subst $(SPACE),$(COMMA),$(TARGET_LIST))	\
+			-e TARGET_LIST=$(subst $(SPACE),$(COMMA),$(filter-out $(DOCKER_FILTER_TARGETS),$(TARGET_LIST)))	\
 			-e EXTRA_CONFIGURE_OPTS="$(EXTRA_CONFIGURE_OPTS)" \
 			-e V=$V -e J=$J -e DEBUG=$(DEBUG)		\
 			-e SHOW_ENV=$(SHOW_ENV) 			\
@@ -195,7 +204,8 @@ docker-run: docker-qemu-src
 docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
 docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
 docker-run-%:
-	@$(MAKE) docker-run TEST=$(CMD) IMAGE=qemu:$(IMAGE)
+	@$(MAKE) docker-run TEST=$(CMD) IMAGE=qemu:$(IMAGE) DOCKER_FILTER_TARGETS=$(DOCKER_FILTER_TARGETS)
+
 
 docker-clean:
 	$(call quiet-command, $(DOCKER_SCRIPT) clean)
-- 
2.17.1

[Qemu-devel] [PATCH v3 12/20] docker: ubuntu: Update the package list before installing new ones

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Since docker caches the different layers, updating the package
list does not invalidate the previous "apt-get update" layer,
and it is likely "apt-get install" hits an outdated repository.

See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#apt-get

This fixes:

  $ make docker-image-ubuntu V=1
  ./tests/docker/docker.py build qemu:ubuntu tests/docker/dockerfiles/ubuntu.docker   --add-current-user
  Sending build context to Docker daemon  3.072kB
  [...]
  E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/libgles2-mesa_17.0.7-0ubuntu0.16.04.2_amd64.deb  404  Not Found
  E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/libgles2-mesa-dev_17.0.7-0ubuntu0.16.04.2_amd64.deb  404  Not Found
  E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
  The command '/bin/sh -c apt-get -y install $PACKAGES' returned a non-zero code: 100
  tests/docker/Makefile.include:40: recipe for target 'docker-image-ubuntu' failed
  make: *** [docker-image-ubuntu] Error 1

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/ubuntu.docker | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index dabbf2a8a4..c03520ce3f 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -1,7 +1,6 @@
 FROM ubuntu:16.04
 RUN echo "deb http://archive.ubuntu.com/ubuntu/ trusty universe multiverse" >> \
     /etc/apt/sources.list
-RUN apt-get update
 ENV PACKAGES flex bison \
     libusb-1.0-0-dev libiscsi-dev librados-dev libncurses5-dev libncursesw5-dev \
     libseccomp-dev libgnutls-dev libssh2-1-dev  libspice-server-dev \
@@ -13,6 +12,7 @@ ENV PACKAGES flex bison \
     libjemalloc-dev libcacard-dev libusbredirhost-dev libnfs-dev libcap-dev libattr1-dev \
     texinfo \
     gettext git make ccache python-yaml gcc clang sparse
-RUN apt-get -y install $PACKAGES
+RUN apt-get update && \
+    apt-get -y install $PACKAGES
 RUN dpkg -l $PACKAGES | sort > /packages.txt
 ENV FEATURES clang pyyaml
-- 
2.17.1

[Qemu-devel] [PATCH v3 13/20] docker: ubuntu: Use SDL2

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Do not test the deprecated API versions (see cabd35840749d).
Debian MXE MinGW cross images are already using SDL2.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/ubuntu.docker | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index c03520ce3f..7d724e7f53 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -5,7 +5,7 @@ ENV PACKAGES flex bison \
     libusb-1.0-0-dev libiscsi-dev librados-dev libncurses5-dev libncursesw5-dev \
     libseccomp-dev libgnutls-dev libssh2-1-dev  libspice-server-dev \
     libspice-protocol-dev libnss3-dev libfdt-dev \
-    libgtk-3-dev libvte-2.91-dev libsdl1.2-dev libpng12-dev libpixman-1-dev \
+    libgtk-3-dev libvte-2.91-dev libsdl2-dev libpng12-dev libpixman-1-dev \
     libvdeplug-dev liblzo2-dev libsnappy-dev libbz2-dev libxen-dev librdmacm-dev libibverbs-dev \
     libsasl2-dev libjpeg-turbo8-dev xfslibs-dev libcap-ng-dev libbrlapi-dev libcurl4-gnutls-dev \
     libbluetooth-dev librbd-dev libaio-dev glusterfs-common libnuma-dev libepoxy-dev libdrm-dev libgbm-dev \
@@ -15,4 +15,4 @@ ENV PACKAGES flex bison \
 RUN apt-get update && \
     apt-get -y install $PACKAGES
 RUN dpkg -l $PACKAGES | sort > /packages.txt
-ENV FEATURES clang pyyaml
+ENV FEATURES clang pyyaml sdl2
-- 
2.17.1

[Qemu-devel] [PATCH v3 14/20] docker: Clean the MXE base image

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Using the duplicated same package is confusing.

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian8-mxe.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian8-mxe.docker b/tests/docker/dockerfiles/debian8-mxe.docker
index 9b8e577b03..2df4cc8c5c 100644
--- a/tests/docker/dockerfiles/debian8-mxe.docker
+++ b/tests/docker/dockerfiles/debian8-mxe.docker
@@ -14,6 +14,6 @@ RUN apt-get update
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
     apt-get install -y --no-install-recommends \
         libpython2.7-stdlib \
-        $(apt-get -s install -y --no-install-recommends gw32.shared-mingw-w64 gw32.shared-mingw-w64 | egrep "^Inst mxe-x86-64-unknown-" | cut -d\  -f2)
+        $(apt-get -s install -y --no-install-recommends gw32.shared-mingw-w64 | egrep "^Inst mxe-x86-64-unknown-" | cut -d\  -f2)
 
 ENV PATH $PATH:/usr/lib/mxe/usr/bin/ 
-- 
2.17.1

[Qemu-devel] [PATCH v3 15/20] docker: Do not run tests in 'intermediate' images

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

We can still build the DOCKER_INTERMEDIATE_IMAGES images,
but they won't appear in 'make test*@$IMAGE'.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 1813ec0781..8641f5da2c 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -5,6 +5,8 @@
 DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 DOCKER_DEPRECATED_IMAGES := debian
+# we don't run tests on intermediate images (used as base by another image)
+DOCKER_INTERMEDIATE_IMAGES := debian8 debian9 debian8-mxe debian-ports debian-sid
 DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
@@ -101,7 +103,7 @@ docker-image-travis: NOUSER=1
 docker-image-tricore-cross: docker-image-debian9
 
 # Expand all the pre-requistes for each docker image and test combination
-$(foreach i,$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES), \
+$(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
 		$(eval .PHONY: docker-$t@$i) \
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
-- 
2.17.1

[Qemu-devel] [PATCH v3 16/20] docker: drop QEMU build-dep from bootstrap

[Alex Bennée]

This is best done with any child images that actually need it.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-bootstrap.docker | 2 --
 1 file changed, 2 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
index 3a9125e497..14212b9cf4 100644
--- a/tests/docker/dockerfiles/debian-bootstrap.docker
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -17,5 +17,3 @@ RUN /debootstrap/debootstrap --second-stage
 # At this point we can install additional packages if we want
 # Duplicate deb line as deb-src
 RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
-RUN apt-get update
-RUN apt-get -y build-dep qemu
-- 
2.17.1

[Qemu-devel] [PATCH v3 17/20] docker: debian-bootstrap.pre allow customising of variant/url

[Alex Bennée]

We default to the buildd variant as most of our images are for
building. However lets give the user the ability to specify "minbase"
if they want to create a simple base image for experimentation.

Allowing the tweaking of DEB_URL means we can also bootstrap other
Debian based OS's. For example:

  make docker-binfmt-image-debian-ubuntu-bionic-arm64 \
       DEB_ARCH=arm64 DEB_TYPE=bionic \
       DEB_VARIANT=minbase DEB_URL=http://ports.ubuntu.com/ \
       EXECUTABLE=./aarch64-linux-user/qemu-aarch64

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-bootstrap.pre | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index 7c76dce663..56e1aa7a21 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -32,6 +32,15 @@ if [ -z "${DEB_TYPE}" ]; then
 
 fi
 
+# The following allow finer grain control over the defaults
+if [ -z "${DEB_VARIANT}" ]; then
+    DEB_VARIANT=buildd
+fi
+
+if [ -z "${DEB_URL}" ]; then
+    DEB_URL="http://httpredir.debian.org/debian"
+fi
+
 # We check in order for
 #
 #  - DEBOOTSTRAP_DIR pointing at a development checkout
@@ -107,5 +116,5 @@ fi
 
 echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP} ${DEB_ARCH}/${DEB_TYPE}"
 
-${FAKEROOT} ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian || exit 1
+${FAKEROOT} ${DEBOOTSTRAP} --variant=$DEB_VARIANT --foreign --arch=$DEB_ARCH $DEB_TYPE . $DEB_URL || exit 1
 exit 0
-- 
2.17.1

[Qemu-devel] [PATCH v3 18/20] docker: add special handling for FROM:debian-%-user targets

[Alex Bennée]

These will have been build with debootstrap so we need to check
against the debian-bootstrap dockerfile. This does mean sticking to
debian-FOO-user as the naming conventions for boot-strapped images.
The actual cross image is built on top.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v3
  - don't include stray whitespace fix
---
 tests/docker/docker.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index b279836154..69e7130db7 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -113,6 +113,10 @@ def _copy_binary_with_libs(src, dest_dir):
             _copy_with_mkdir(l , dest_dir, so_path)
 
 def _read_qemu_dockerfile(img_name):
+    # special case for Debian linux-user images
+    if img_name.startswith("debian") and img_name.endswith("user"):
+        img_name = "debian-bootstrap"
+
     df = os.path.join(os.path.dirname(__file__), "dockerfiles",
                       img_name + ".docker")
     return open(df, "r").read()
-- 
2.17.1

[Qemu-devel] [PATCH v3 19/20] docker: add special rule for deboostrapped images

[Alex Bennée]

We might as well have a custom rule for this. For one thing the
dependencies are different. As the primary dependency for
docker-image-% could never be docker-image-debian-bootstrap we can
drop that test in the main rule as well.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v3:
  - changed rule to docker-binfmt-image-debian-%:
  - leave executable in the main rule
  - rm stray whitespace fix
---
 tests/docker/Makefile.include | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 8641f5da2c..fdc87b8a57 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -49,9 +49,6 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		"CHECK", "$*")
 else
 docker-image-%: $(DOCKER_FILES_DIR)/%.docker
-	@if test "$@" = docker-image-debian-bootstrap -a -z "$(EXECUTABLE)"; then \
-		echo WARNING: EXECUTABLE is not set, debootstrap may fail. 2>&1 ; \
-	fi
 	$(call quiet-command,\
 		$(DOCKER_SCRIPT) build qemu:$* $< \
 		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
@@ -59,6 +56,26 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
 		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
 		"BUILD","$*")
+
+# Special rule for debootstraped binfmt linux-user images
+docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
+	@if test -z "$(EXECUTABLE)"; then \
+		echo WARNING: EXECUTABLE is not set, debootstrap may fail. 2>&1 ; \
+	fi
+	@if test -z "$(DEB_ARCH)"; then 					\
+		echo WARNING: DEB_ARCH is not set, debootstrap may fail. 2>&1 ; \
+        fi
+	@if test -z "$(DEB_TYPE)"; then 					\
+		echo WARNING: DEB_TYPE is not set, debootstrap may fail. 2>&1 ; \
+        fi
+	$(call quiet-command,							\
+		$(DOCKER_SCRIPT) build qemu:debian-$* $< \
+		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
+		$(if $(NOUSER),,--add-current-user) \
+		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
+		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
+		"BUILD","binfmt debian-$* (debootstrapped)")
+
 endif
 
 # Enforce dependencies for composite images
-- 
2.17.1

[Qemu-devel] [PATCH v3 20/20] docker: add linux-user powered cross builder for QEMU

[Alex Bennée]

We can't use cross compilers in the current Debian stable and Debian
sid is sketchy as hell. So for powerpc fall back to dog-fooding our
own linux-user to do the build.

As we can only build the base image with a suitably configured
source tree we fall back to checking for it's existence when we can't
build it from scratch. However this does mean you don't have to keep
a static powerpc-linux-user in your active configuration just to
update the cross build image.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v3
  - fix for rename of docker-binfmt-image-debian-powerpc-user
---
 tests/docker/Makefile.include                 | 29 +++++++++++++++++++
 .../debian-powerpc-user-cross.docker          | 13 +++++++++
 2 files changed, 42 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-user-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index fdc87b8a57..85801702dd 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -119,6 +119,31 @@ docker-image-travis: NOUSER=1
 # Specialist build images, sometimes very limited tools
 docker-image-tricore-cross: docker-image-debian9
 
+# Rules for building linux-user powered images
+#
+# These are slower than using native cross compiler setups but can
+# work around issues with poorly working multi-arch systems and broken
+# packages.
+
+ifeq ($(filter ppc-linux-user,$(TARGET_LIST))$(CONFIG_STATIC),ppc-linux-usery)
+# Jessie is the last supported release for powerpc, but multi-arch is
+# broken so we need a qemu-linux-user for this target
+docker-binfmt-image-debian-powerpc-user: DEB_ARCH = powerpc
+docker-binfmt-image-debian-powerpc-user: DEB_TYPE = jessie
+docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
+DOCKER_USER_IMAGES += debian-powerpc-user
+else
+docker-binfmt-image-debian-powerpc-user:
+	$(call quiet-command, \
+		$(DOCKER_SCRIPT) check --quiet qemu:debian-powerpc-user \
+			$(DOCKER_FILES_DIR)/debian-bootstrap.docker, \
+		"CHECK", "debian-powerpc-user exists")
+endif
+
+# We build the QEMU compiler environment on top of the base image
+docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
+
+
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
@@ -154,6 +179,10 @@ docker:
 	@echo
 	@echo 'Available container images:'
 	@echo '    $(DOCKER_IMAGES)'
+ifneq ($(DOCKER_USER_IMAGES),)
+	@echo 'Available linux-user images:'
+	@echo '    $(DOCKER_USER_IMAGES)'
+endif
 	@echo
 	@echo 'Available tests:'
 	@echo '    $(DOCKER_TESTS)'
diff --git a/tests/docker/dockerfiles/debian-powerpc-user-cross.docker b/tests/docker/dockerfiles/debian-powerpc-user-cross.docker
new file mode 100644
index 0000000000..eadb142792
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-powerpc-user-cross.docker
@@ -0,0 +1,13 @@
+#
+# Docker powerpc cross-compiler target for QEMU
+#
+# We can't use current Debian stable cross-compilers to build powerpc
+# as it has been dropped as a release architecture. Using Debian Sid
+# is just far too sketchy a build environment. This leaves us the
+# final option of using linux-user. This image is based of the
+# debootstrapped qemu:debian-powerpc-user but doesn't need any extra
+# magic once it is setup.
+#
+FROM qemu:debian-powerpc-user
+
+RUN apt-get update && apt-get build-dep -yy qemu
-- 
2.17.1

Re: [Qemu-devel] [PATCH v3 07/20] build-system: add clean-coverage target

[Philippe Mathieu-Daudé]

Hi Alex,

On 07/02/2018 11:30 AM, Alex Bennée wrote:
> This can be used to remove any stale coverage data before any
> particular test run. This is useful for analysing individual tests.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> 
> v2
>   - s/to any/to delete any/
> ---
>  Makefile               | 11 +++++++++++
>  docs/devel/testing.rst | 11 ++++++++---
>  2 files changed, 19 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 7ed9cc4a21..2b3413a5ba 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -723,6 +723,14 @@ module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak
>  	$(addprefix $(SRC_PATH)/,$(patsubst %.mo,%.c,$(block-obj-m))), \
>  	"GEN","$@")
>  
> +ifdef CONFIG_GCOV
> +.PHONY: clean-coverage
> +clean-coverage:
> +	$(call quiet-command, \
> +		find . \( -name '*.gcda' -o -name '*.gcov' \) -type f -exec rm {} +, \
> +		"CLEAN", "coverage files")

Would "find $(BUILD_DIR) ..." be safer?

Regardless:
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> +endif
> +
>  clean:
>  # avoid old build problems by removing potentially incorrect old files
>  	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
> @@ -1073,6 +1081,9 @@ endif
>  		echo '')
>  	@echo  'Cleaning targets:'
>  	@echo  '  clean           - Remove most generated files but keep the config'
> +ifdef CONFIG_GCOV
> +	@echo  '  clean-coverage  - Remove coverage files'
> +endif
>  	@echo  '  distclean       - Remove all generated files'
>  	@echo  '  dist            - Build a distributable tarball'
>  	@echo  ''
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 66ef219f69..7f04ca104e 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -161,9 +161,14 @@ GCC gcov support
>  ``gcov`` is a GCC tool to analyze the testing coverage by
>  instrumenting the tested code. To use it, configure QEMU with
>  ``--enable-gcov`` option and build. Then run ``make check`` as usual.
> -Reports can be obtained by running ``gcov`` command on the output
> -files under ``$build_dir/tests/``, please read the ``gcov``
> -documentation for more information.
> +
> +If you want to gather coverage information on a single test the ``make
> +clean-coverage`` target can be used to delete any existing coverage
> +information before running a single test.
> +
> +Reports can be obtained by running ``gcov`` command
> +on the output files under ``$build_dir/tests/``, please read the
> +``gcov`` documentation for more information.
>  
>  QEMU iotests
>  ============
> 

Re: [Qemu-devel] [PATCH v3 09/20] linux-user: introduce preexit_cleanup

[Philippe Mathieu-Daudé]

Hi Alex,

On 07/02/2018 11:30 AM, Alex Bennée wrote:
> To avoid repeating ourselves move our preexit clean-up code into a
> helper function. I figured the continuing effort to split of the
> syscalls made it worthwhile creating a new file for it now.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> Reviewed-by: Laurent Vivier <laurent@vivier.eu>
> ---
>  linux-user/Makefile.objs |  2 +-
>  linux-user/exit.c        | 28 ++++++++++++++++++++++++++++
>  linux-user/qemu.h        |  8 ++++++++
>  linux-user/syscall.c     | 10 ++--------
>  4 files changed, 39 insertions(+), 9 deletions(-)
>  create mode 100644 linux-user/exit.c
> 
> diff --git a/linux-user/Makefile.objs b/linux-user/Makefile.objs
> index 59a5c17354..b5dfb71f25 100644
> --- a/linux-user/Makefile.objs
> +++ b/linux-user/Makefile.objs
> @@ -1,7 +1,7 @@
>  obj-y = main.o syscall.o strace.o mmap.o signal.o \
>  	elfload.o linuxload.o uaccess.o uname.o \
>  	safe-syscall.o $(TARGET_ABI_DIR)/signal.o \
> -        $(TARGET_ABI_DIR)/cpu_loop.o
> +        $(TARGET_ABI_DIR)/cpu_loop.o exit.o
>  
>  obj-$(TARGET_HAS_BFLT) += flatload.o
>  obj-$(TARGET_I386) += vm86.o
> diff --git a/linux-user/exit.c b/linux-user/exit.c
> new file mode 100644
> index 0000000000..aed8713fae
> --- /dev/null
> +++ b/linux-user/exit.c
> @@ -0,0 +1,28 @@
> +/*
> + *  exit support for qemu
> + *
> + *  Copyright (c) 2018 Alex Bennée <alex.bennee@linaro.org>
> + *
> + *  This program is free software; you can redistribute it and/or modify
> + *  it under the terms of the GNU General Public License as published by
> + *  the Free Software Foundation; either version 2 of the License, or
> + *  (at your option) any later version.
> + *
> + *  This program is distributed in the hope that it will be useful,
> + *  but WITHOUT ANY WARRANTY; without even the implied warranty of
> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + *  GNU General Public License for more details.
> + *
> + *  You should have received a copy of the GNU General Public License
> + *  along with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +#include "qemu/osdep.h"
> +#include "qemu.h"
> +
> +void preexit_cleanup(CPUArchState *env, int code)
> +{
> +#ifdef TARGET_GPROF
> +        _mcleanup();
> +#endif
> +        gdb_exit(env, code);
> +}

Using:

'configure' '--enable-gprof' '--enable-gcov' '--disable-pie' --static
[...]
C compiler        gcc-8
CFLAGS            -fprofile-arcs -ftest-coverage -g -g
LDFLAGS           -Wl,--warn-common -fprofile-arcs -ftest-coverage -m64
-static -g
gprof enabled     yes
static build      yes
PIE               no
gcov              gcov
gcov enabled      yes
docker            yes

I get:

$ make subdir-ppc-linux-user
[...]
  CC      ppc-linux-user/linux-user/exit.o
linux-user/exit.c: In function ‘preexit_cleanup’:
linux-user/exit.c:29:9: error: implicit declaration of function
‘_mcleanup’ [-Werror=implicit-function-declaration]
         _mcleanup();
         ^~~~~~~~~
linux-user/exit.c:29:9: error: nested extern declaration of ‘_mcleanup’
[-Werror=nested-externs]
cc1: all warnings being treated as errors
make[1]: *** [rules.mak:69: linux-user/exit.o] Error 1
make: *** [Makefile:481: subdir-ppc-linux-user] Error 2

I suppose gprof and static builds are exclusive.

> diff --git a/linux-user/qemu.h b/linux-user/qemu.h
> index 793cd4df04..bb85c81aa4 100644
> --- a/linux-user/qemu.h
> +++ b/linux-user/qemu.h
> @@ -623,6 +623,14 @@ static inline int is_error(abi_long ret)
>      return (abi_ulong)ret >= (abi_ulong)(-4096);
>  }
>  
> +/**
> + * preexit_cleanup: housekeeping before the guest exits
> + *
> + * env: the CPU state
> + * code: the exit code
> + */
> +void preexit_cleanup(CPUArchState *env, int code);
> +
>  /* Include target-specific struct and function definitions;
>   * they may need access to the target-independent structures
>   * above, so include them last.
> diff --git a/linux-user/syscall.c b/linux-user/syscall.c
> index 2117fb13b4..7c66442357 100644
> --- a/linux-user/syscall.c
> +++ b/linux-user/syscall.c
> @@ -8018,10 +8018,7 @@ abi_long do_syscall(void *cpu_env, int num, abi_long arg1,
>          }
>  
>          cpu_list_unlock();
> -#ifdef TARGET_GPROF
> -        _mcleanup();
> -#endif
> -        gdb_exit(cpu_env, arg1);
> +        preexit_cleanup(cpu_env, arg1);
>          _exit(arg1);
>          ret = 0; /* avoid warning */
>          break;
> @@ -10127,10 +10124,7 @@ abi_long do_syscall(void *cpu_env, int num, abi_long arg1,
>  #ifdef __NR_exit_group
>          /* new thread calls */
>      case TARGET_NR_exit_group:
> -#ifdef TARGET_GPROF
> -        _mcleanup();
> -#endif
> -        gdb_exit(cpu_env, arg1);
> +        preexit_cleanup(cpu_env, arg1);
>          ret = get_errno(exit_group(arg1));
>          break;
>  #endif
> 

Re: [Qemu-devel] [PATCH v3 09/20] linux-user: introduce preexit_cleanup

[Philippe Mathieu-Daudé]

On 07/02/2018 01:22 PM, Philippe Mathieu-Daudé wrote:
> Using:
> 
> 'configure' '--enable-gprof' '--enable-gcov' '--disable-pie' --static
> [...]
> C compiler        gcc-8
> CFLAGS            -fprofile-arcs -ftest-coverage -g -g
> LDFLAGS           -Wl,--warn-common -fprofile-arcs -ftest-coverage -m64
> -static -g
> gprof enabled     yes
> static build      yes
> PIE               no
> gcov              gcov
> gcov enabled      yes
> docker            yes
> 
> I get:
> 
> $ make subdir-ppc-linux-user
> [...]
>   CC      ppc-linux-user/linux-user/exit.o
> linux-user/exit.c: In function ‘preexit_cleanup’:
> linux-user/exit.c:29:9: error: implicit declaration of function
> ‘_mcleanup’ [-Werror=implicit-function-declaration]
>          _mcleanup();
>          ^~~~~~~~~
> linux-user/exit.c:29:9: error: nested extern declaration of ‘_mcleanup’
> [-Werror=nested-externs]
> cc1: all warnings being treated as errors
> make[1]: *** [rules.mak:69: linux-user/exit.o] Error 1
> make: *** [Makefile:481: subdir-ppc-linux-user] Error 2
> 
> I suppose gprof and static builds are exclusive.

I just noticed we don't have --static builds covered by our continuous
integration.
Yet too many matrix cases to cover?
We could ignore it for 'system' builds, but we should enable it for
--linux-user which is used by various distributions for with the binfmt
packages.

Regards,

Phil.

Re: [Qemu-devel] [PATCH v3 09/20] linux-user: introduce preexit_cleanup

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 07/02/2018 01:22 PM, Philippe Mathieu-Daudé wrote:
>> Using:
>>
>> 'configure' '--enable-gprof' '--enable-gcov' '--disable-pie' --static
>> [...]
>> C compiler        gcc-8
>> CFLAGS            -fprofile-arcs -ftest-coverage -g -g
>> LDFLAGS           -Wl,--warn-common -fprofile-arcs -ftest-coverage -m64
>> -static -g
>> gprof enabled     yes
>> static build      yes
>> PIE               no
>> gcov              gcov
>> gcov enabled      yes
>> docker            yes
>>
>> I get:
>>
>> $ make subdir-ppc-linux-user
>> [...]
>>   CC      ppc-linux-user/linux-user/exit.o
>> linux-user/exit.c: In function ‘preexit_cleanup’:
>> linux-user/exit.c:29:9: error: implicit declaration of function
>> ‘_mcleanup’ [-Werror=implicit-function-declaration]
>>          _mcleanup();
>>          ^~~~~~~~~
>> linux-user/exit.c:29:9: error: nested extern declaration of ‘_mcleanup’
>> [-Werror=nested-externs]
>> cc1: all warnings being treated as errors
>> make[1]: *** [rules.mak:69: linux-user/exit.o] Error 1
>> make: *** [Makefile:481: subdir-ppc-linux-user] Error 2
>>
>> I suppose gprof and static builds are exclusive.

Yeah I guess this must have always been there because I didn't change
the gprof code, just moved it.

>
> I just noticed we don't have --static builds covered by our continuous
> integration.
> Yet too many matrix cases to cover?
> We could ignore it for 'system' builds, but we should enable it for
> --linux-user which is used by various distributions for with the binfmt
> packages.

Yeah certainly a --disable-system --static build makes sense.

>
> Regards,
>
> Phil.


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 00/20] Travis, Code Coverage and Cross Build updates

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180702143021.18864-1-alex.bennee@linaro.org
Subject: [Qemu-devel] [PATCH v3 00/20] Travis, Code Coverage and Cross Build updates

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
4873e71047 docker: add linux-user powered cross builder for QEMU
34f577cda4 docker: add special rule for deboostrapped images
bbbe49493e docker: add special handling for FROM:debian-%-user targets
3adc2ab166 docker: debian-bootstrap.pre allow customising of variant/url
61cd7bb1f4 docker: drop QEMU build-dep from bootstrap
0024785428 docker: Do not run tests in 'intermediate' images
0f142f6713 docker: Clean the MXE base image
8779053bc8 docker: ubuntu: Use SDL2
305dc1ad2f docker: ubuntu: Update the package list before installing new ones
58bd56936a docker: filter out linux-user builds for mingw
589cc92212 linux-user: add gcov support to preexit_cleanup
57dcf92bea linux-user: introduce preexit_cleanup
f67baf9f57 build-system: add coverage-report target
9d58118bc7 build-system: add clean-coverage target
d750d825dc travis: add gcovr summary for GCOV build
b1fa9ff6d6 docker: add gcovr to travis image
8d87f85bb0 .gitignore: add .gcov files
ce2d745b8d build-system: remove per-test GCOV reporting
87bc2150db travis: test out-of-tree builds
1b44d66f6e travis: do not waste time cloning unused submodules

=== OUTPUT BEGIN ===
Checking PATCH 1/20: travis: do not waste time cloning unused submodules...
Checking PATCH 2/20: travis: test out-of-tree builds...
Checking PATCH 3/20: build-system: remove per-test GCOV reporting...
Checking PATCH 4/20: .gitignore: add .gcov files...
Checking PATCH 5/20: docker: add gcovr to travis image...
Checking PATCH 6/20: travis: add gcovr summary for GCOV build...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#58: 
new file mode 100755

total: 0 errors, 1 warnings, 49 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
Checking PATCH 7/20: build-system: add clean-coverage target...
Checking PATCH 8/20: build-system: add coverage-report target...
Checking PATCH 9/20: linux-user: introduce preexit_cleanup...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#33: 
new file mode 100644

total: 0 errors, 1 warnings, 72 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
Checking PATCH 10/20: linux-user: add gcov support to preexit_cleanup...
ERROR: externs should be avoided in .c files
#27: FILE: linux-user/exit.c:23:
+extern void __gcov_dump(void);

total: 1 errors, 0 warnings, 17 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 11/20: docker: filter out linux-user builds for mingw...
Checking PATCH 12/20: docker: ubuntu: Update the package list before installing new ones...
Checking PATCH 13/20: docker: ubuntu: Use SDL2...
Checking PATCH 14/20: docker: Clean the MXE base image...
Checking PATCH 15/20: docker: Do not run tests in 'intermediate' images...
Checking PATCH 16/20: docker: drop QEMU build-dep from bootstrap...
Checking PATCH 17/20: docker: debian-bootstrap.pre allow customising of variant/url...
Checking PATCH 18/20: docker: add special handling for FROM:debian-%-user targets...
Checking PATCH 19/20: docker: add special rule for deboostrapped images...
Checking PATCH 20/20: docker: add linux-user powered cross builder for QEMU...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#71: 
new file mode 100644

total: 0 errors, 1 warnings, 54 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH v3 17/20] docker: debian-bootstrap.pre allow customising of variant/url

[Philippe Mathieu-Daudé]

On 07/02/2018 11:30 AM, Alex Bennée wrote:
> We default to the buildd variant as most of our images are for
> building. However lets give the user the ability to specify "minbase"
> if they want to create a simple base image for experimentation.
> 
> Allowing the tweaking of DEB_URL means we can also bootstrap other
> Debian based OS's. For example:
> 
>   make docker-binfmt-image-debian-ubuntu-bionic-arm64 \
>        DEB_ARCH=arm64 DEB_TYPE=bionic \
>        DEB_VARIANT=minbase DEB_URL=http://ports.ubuntu.com/ \
>        EXECUTABLE=./aarch64-linux-user/qemu-aarch64
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  tests/docker/dockerfiles/debian-bootstrap.pre | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
> index 7c76dce663..56e1aa7a21 100755
> --- a/tests/docker/dockerfiles/debian-bootstrap.pre
> +++ b/tests/docker/dockerfiles/debian-bootstrap.pre
> @@ -32,6 +32,15 @@ if [ -z "${DEB_TYPE}" ]; then
>  
>  fi
>  
> +# The following allow finer grain control over the defaults
> +if [ -z "${DEB_VARIANT}" ]; then
> +    DEB_VARIANT=buildd
> +fi
> +
> +if [ -z "${DEB_URL}" ]; then
> +    DEB_URL="http://httpredir.debian.org/debian"
> +fi
> +
>  # We check in order for
>  #
>  #  - DEBOOTSTRAP_DIR pointing at a development checkout
> @@ -107,5 +116,5 @@ fi
>  
>  echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP} ${DEB_ARCH}/${DEB_TYPE}"
>  
> -${FAKEROOT} ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian || exit 1
> +${FAKEROOT} ${DEBOOTSTRAP} --variant=$DEB_VARIANT --foreign --arch=$DEB_ARCH $DEB_TYPE . $DEB_URL || exit 1
>  exit 0
> 

Re: [Qemu-devel] [PATCH v3 16/20] docker: drop QEMU build-dep from bootstrap

[Philippe Mathieu-Daudé]

On 07/02/2018 11:30 AM, Alex Bennée wrote:
> This is best done with any child images that actually need it.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  tests/docker/dockerfiles/debian-bootstrap.docker | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
> index 3a9125e497..14212b9cf4 100644
> --- a/tests/docker/dockerfiles/debian-bootstrap.docker
> +++ b/tests/docker/dockerfiles/debian-bootstrap.docker
> @@ -17,5 +17,3 @@ RUN /debootstrap/debootstrap --second-stage
>  # At this point we can install additional packages if we want
>  # Duplicate deb line as deb-src
>  RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
> -RUN apt-get update
> -RUN apt-get -y build-dep qemu
> 

Re: [Qemu-devel] [PATCH v3 18/20] docker: add special handling for FROM:debian-%-user targets

[Philippe Mathieu-Daudé]

On 07/02/2018 11:30 AM, Alex Bennée wrote:
> These will have been build with debootstrap so we need to check
> against the debian-bootstrap dockerfile. This does mean sticking to
> debian-FOO-user as the naming conventions for boot-strapped images.
> The actual cross image is built on top.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> 
> ---
> v3
>   - don't include stray whitespace fix
> ---
>  tests/docker/docker.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index b279836154..69e7130db7 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -113,6 +113,10 @@ def _copy_binary_with_libs(src, dest_dir):
>              _copy_with_mkdir(l , dest_dir, so_path)
>  
>  def _read_qemu_dockerfile(img_name):
> +    # special case for Debian linux-user images
> +    if img_name.startswith("debian") and img_name.endswith("user"):
> +        img_name = "debian-bootstrap"
> +
>      df = os.path.join(os.path.dirname(__file__), "dockerfiles",
>                        img_name + ".docker")
>      return open(df, "r").read()
> 

Re: [Qemu-devel] [PATCH v3 19/20] docker: add special rule for deboostrapped images

[Philippe Mathieu-Daudé]

On 07/02/2018 11:30 AM, Alex Bennée wrote:
> We might as well have a custom rule for this. For one thing the
> dependencies are different. As the primary dependency for
> docker-image-% could never be docker-image-debian-bootstrap we can
> drop that test in the main rule as well.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v3:
>   - changed rule to docker-binfmt-image-debian-%:
>   - leave executable in the main rule
>   - rm stray whitespace fix
> ---
>  tests/docker/Makefile.include | 23 ++++++++++++++++++++---
>  1 file changed, 20 insertions(+), 3 deletions(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 8641f5da2c..fdc87b8a57 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -49,9 +49,6 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		"CHECK", "$*")
>  else
>  docker-image-%: $(DOCKER_FILES_DIR)/%.docker
> -	@if test "$@" = docker-image-debian-bootstrap -a -z "$(EXECUTABLE)"; then \
> -		echo WARNING: EXECUTABLE is not set, debootstrap may fail. 2>&1 ; \
> -	fi
>  	$(call quiet-command,\
>  		$(DOCKER_SCRIPT) build qemu:$* $< \
>  		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
> @@ -59,6 +56,26 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
>  		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
>  		"BUILD","$*")
> +
> +# Special rule for debootstraped binfmt linux-user images
> +docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
> +	@if test -z "$(EXECUTABLE)"; then \
> +		echo WARNING: EXECUTABLE is not set, debootstrap may fail. 2>&1 ; \
> +	fi
> +	@if test -z "$(DEB_ARCH)"; then 					\
> +		echo WARNING: DEB_ARCH is not set, debootstrap may fail. 2>&1 ; \
> +        fi
> +	@if test -z "$(DEB_TYPE)"; then 					\
> +		echo WARNING: DEB_TYPE is not set, debootstrap may fail. 2>&1 ; \
> +        fi
> +	$(call quiet-command,							\

Here I had to add:

+               DEB_ARCH=$(DEB_ARCH) \
+               DEB_TYPE=$(DEB_TYPE) \

> +		$(DOCKER_SCRIPT) build qemu:debian-$* $< \
> +		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
> +		$(if $(NOUSER),,--add-current-user) \
> +		$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
> +		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
> +		"BUILD","binfmt debian-$* (debootstrapped)")
> +
>  endif
>  
>  # Enforce dependencies for composite images
> 

Re: [Qemu-devel] [PATCH v3 20/20] docker: add linux-user powered cross builder for QEMU

[Philippe Mathieu-Daudé]

On 07/02/2018 11:30 AM, Alex Bennée wrote:
> We can't use cross compilers in the current Debian stable and Debian
> sid is sketchy as hell. So for powerpc fall back to dog-fooding our
> own linux-user to do the build.
> 
> As we can only build the base image with a suitably configured
> source tree we fall back to checking for it's existence when we can't
> build it from scratch. However this does mean you don't have to keep
> a static powerpc-linux-user in your active configuration just to
> update the cross build image.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v3
>   - fix for rename of docker-binfmt-image-debian-powerpc-user
> ---
>  tests/docker/Makefile.include                 | 29 +++++++++++++++++++
>  .../debian-powerpc-user-cross.docker          | 13 +++++++++
>  2 files changed, 42 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debian-powerpc-user-cross.docker
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index fdc87b8a57..85801702dd 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -119,6 +119,31 @@ docker-image-travis: NOUSER=1
>  # Specialist build images, sometimes very limited tools
>  docker-image-tricore-cross: docker-image-debian9
>  
> +# Rules for building linux-user powered images
> +#
> +# These are slower than using native cross compiler setups but can
> +# work around issues with poorly working multi-arch systems and broken
> +# packages.
> +
> +ifeq ($(filter ppc-linux-user,$(TARGET_LIST))$(CONFIG_STATIC),ppc-linux-usery)
> +# Jessie is the last supported release for powerpc, but multi-arch is
> +# broken so we need a qemu-linux-user for this target
> +docker-binfmt-image-debian-powerpc-user: DEB_ARCH = powerpc
> +docker-binfmt-image-debian-powerpc-user: DEB_TYPE = jessie
> +docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
> +DOCKER_USER_IMAGES += debian-powerpc-user
> +else
> +docker-binfmt-image-debian-powerpc-user:
> +	$(call quiet-command, \
> +		$(DOCKER_SCRIPT) check --quiet qemu:debian-powerpc-user \
> +			$(DOCKER_FILES_DIR)/debian-bootstrap.docker, \
> +		"CHECK", "debian-powerpc-user exists")
> +endif
> +
> +# We build the QEMU compiler environment on top of the base image
> +docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
> +
> +
>  # Expand all the pre-requistes for each docker image and test combination
>  $(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
>  	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
> @@ -154,6 +179,10 @@ docker:
>  	@echo
>  	@echo 'Available container images:'
>  	@echo '    $(DOCKER_IMAGES)'
> +ifneq ($(DOCKER_USER_IMAGES),)
> +	@echo 'Available linux-user images:'
> +	@echo '    $(DOCKER_USER_IMAGES)'
> +endif
>  	@echo
>  	@echo 'Available tests:'
>  	@echo '    $(DOCKER_TESTS)'
> diff --git a/tests/docker/dockerfiles/debian-powerpc-user-cross.docker b/tests/docker/dockerfiles/debian-powerpc-user-cross.docker
> new file mode 100644
> index 0000000000..eadb142792
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-powerpc-user-cross.docker
> @@ -0,0 +1,13 @@
> +#
> +# Docker powerpc cross-compiler target for QEMU
> +#
> +# We can't use current Debian stable cross-compilers to build powerpc
> +# as it has been dropped as a release architecture. Using Debian Sid
> +# is just far too sketchy a build environment. This leaves us the
> +# final option of using linux-user. This image is based of the
> +# debootstrapped qemu:debian-powerpc-user but doesn't need any extra
> +# magic once it is setup.
> +#
> +FROM qemu:debian-powerpc-user
> +
> +RUN apt-get update && apt-get build-dep -yy qemu

RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get build-dep -yy qemu

to avoid the various:

...
debconf: (TERM is not set, so the dialog frontend is not usable.)
...

Also I think we should use the "Don't rebuild man-db" trick:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=561539#15

Re: [Qemu-devel] [PATCH v3 20/20] docker: add linux-user powered cross builder for QEMU

[Philippe Mathieu-Daudé]

On 07/02/2018 05:35 PM, Philippe Mathieu-Daudé wrote:
> On 07/02/2018 11:30 AM, Alex Bennée wrote:
>> We can't use cross compilers in the current Debian stable and Debian
>> sid is sketchy as hell. So for powerpc fall back to dog-fooding our
>> own linux-user to do the build.
>>
>> As we can only build the base image with a suitably configured
>> source tree we fall back to checking for it's existence when we can't
>> build it from scratch. However this does mean you don't have to keep
>> a static powerpc-linux-user in your active configuration just to
>> update the cross build image.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>
>> ---
>> v3
>>   - fix for rename of docker-binfmt-image-debian-powerpc-user
>> ---
>>  tests/docker/Makefile.include                 | 29 +++++++++++++++++++
>>  .../debian-powerpc-user-cross.docker          | 13 +++++++++
>>  2 files changed, 42 insertions(+)
>>  create mode 100644 tests/docker/dockerfiles/debian-powerpc-user-cross.docker
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index fdc87b8a57..85801702dd 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -119,6 +119,31 @@ docker-image-travis: NOUSER=1
>>  # Specialist build images, sometimes very limited tools
>>  docker-image-tricore-cross: docker-image-debian9
>>  
>> +# Rules for building linux-user powered images
>> +#
>> +# These are slower than using native cross compiler setups but can
>> +# work around issues with poorly working multi-arch systems and broken
>> +# packages.
>> +
>> +ifeq ($(filter ppc-linux-user,$(TARGET_LIST))$(CONFIG_STATIC),ppc-linux-usery)
>> +# Jessie is the last supported release for powerpc, but multi-arch is
>> +# broken so we need a qemu-linux-user for this target
>> +docker-binfmt-image-debian-powerpc-user: DEB_ARCH = powerpc
>> +docker-binfmt-image-debian-powerpc-user: DEB_TYPE = jessie
>> +docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
>> +DOCKER_USER_IMAGES += debian-powerpc-user
>> +else
>> +docker-binfmt-image-debian-powerpc-user:
>> +	$(call quiet-command, \
>> +		$(DOCKER_SCRIPT) check --quiet qemu:debian-powerpc-user \
>> +			$(DOCKER_FILES_DIR)/debian-bootstrap.docker, \
>> +		"CHECK", "debian-powerpc-user exists")
>> +endif
>> +
>> +# We build the QEMU compiler environment on top of the base image
>> +docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
>> +
>> +
>>  # Expand all the pre-requistes for each docker image and test combination
>>  $(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
>>  	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
>> @@ -154,6 +179,10 @@ docker:
>>  	@echo
>>  	@echo 'Available container images:'
>>  	@echo '    $(DOCKER_IMAGES)'
>> +ifneq ($(DOCKER_USER_IMAGES),)
>> +	@echo 'Available linux-user images:'
>> +	@echo '    $(DOCKER_USER_IMAGES)'
>> +endif
>>  	@echo
>>  	@echo 'Available tests:'
>>  	@echo '    $(DOCKER_TESTS)'
>> diff --git a/tests/docker/dockerfiles/debian-powerpc-user-cross.docker b/tests/docker/dockerfiles/debian-powerpc-user-cross.docker
>> new file mode 100644
>> index 0000000000..eadb142792
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/debian-powerpc-user-cross.docker
>> @@ -0,0 +1,13 @@
>> +#
>> +# Docker powerpc cross-compiler target for QEMU
>> +#
>> +# We can't use current Debian stable cross-compilers to build powerpc
>> +# as it has been dropped as a release architecture. Using Debian Sid
>> +# is just far too sketchy a build environment. This leaves us the
>> +# final option of using linux-user. This image is based of the
>> +# debootstrapped qemu:debian-powerpc-user but doesn't need any extra
>> +# magic once it is setup.
>> +#
>> +FROM qemu:debian-powerpc-user
>> +
>> +RUN apt-get update && apt-get build-dep -yy qemu
> 
> RUN apt-get update && \
>     DEBIAN_FRONTEND=noninteractive apt-get build-dep -yy qemu
> 
> to avoid the various:
> 
> ...
> debconf: (TERM is not set, so the dialog frontend is not usable.)
> ...
> 
> Also I think we should use the "Don't rebuild man-db" trick:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=561539#15

Using the following patch speeds up indeed:

-- >8 --
@@ -10,4 +10,7 @@
 #
 FROM qemu:debian-powerpc-user

-RUN apt-get update && apt-get build-dep -yy qemu
+RUN echo man-db man-db/auto-update boolean false | debconf-set-selections
+
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get build-dep -yy qemu
---