All of lore.kernel.org
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes)
@ 2019-02-07 18:37 Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 01/16] .cirrus.yml: basic compile and test for FreeBSD Alex Bennée
                   ` (16 more replies)
  0 siblings, 17 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée


Hi,

There are a few more additions from the last iteration including some
more fixes to try and get Travis green again. It's nice to see our ARM
features are now catching up we need a newer toolchain to build
things. I did try upgrading to debian 10 for the cross build but there
are some upstream problems with the multiarch build deps that get in
the way of that. People with Debian packaging experience can jump in
on:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921458
  (dependancy on gcc-s390x-linux-gnu fails on non-x86 hosts)

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921648
  (apt-get build-dep -a arm64 qemu fails on multiarch setup)

as it would be good to get these problems fixed as we get to
softfreeze (and given how much we leverage Debian's cross toolchains).

If all the testing is green by tomorrow I'll put together a pull
request. Unless people are desperate to grab the arm64 compiler image
now as a build fix?

The following patches need review
 patch 0004/.travis.yml fold disable tcg into alternate corou.patch
 patch 0007/docker add debian buster arm64 cross.patch
 patch 0008/tests make docker.py update use configured binfmt.patch
 patch 0009/tests make docker.py check for persistent configs.patch
 patch 0010/tests docker.py be even smarter with persistent b.patch
 patch 0011/tests PEP8 cleanup of docker.py mostly white spac.patch
 patch 0016/scripts qemu.py allow arches use KVM for their 32.patch


Alex Bennée (11):
  .travis.yml: separate tools and docs into another entry
  .travis.yml: fold --disable-tcg into alternate coroutine builds
  docker: add debian-buster-arm64-cross
  tests: make docker.py update use configured binfmt path
  tests: make docker.py check for persistent configs
  tests: docker.py be even smarter with persistent binfmt_misc
  tests: PEP8 cleanup of docker.py, mostly white space
  tests/vm: call make check directly for netbsd/freebsd/ubuntu.i386
  tests/vm: add --build-target option
  tests/vm: expose BUILD_TARGET, TARGET_LIST and EXTRA_CONFIGURE_OPTS
  scripts/qemu.py: allow arches use KVM for their 32bit cousins

Daniel P. Berrangé (1):
  .travis.yml: stop requesting libffi & gettext from homebrew

Ed Maste (1):
  .cirrus.yml: basic compile and test for FreeBSD

Gerd Hoffmann (1):
  tests/vm: move images to $HOME/.cache/qemu-vm/images

Philippe Mathieu-Daudé (2):
  MAINTAINERS: Add an entry for scripts/archive-source.sh
  archive-source.sh: Clone the submodules locally

 .cirrus.yml                                   |  16 +++
 .travis.yml                                   |  31 +++--
 MAINTAINERS                                   |   9 ++
 scripts/archive-source.sh                     |   7 ++
 scripts/qemu.py                               |  12 +-
 tests/docker/Makefile.include                 |   1 +
 tests/docker/docker.py                        | 110 +++++++++++++-----
 .../debian-buster-arm64-cross.docker          |  15 +++
 tests/docker/dockerfiles/debian10.docker      |  31 +++++
 tests/tcg/aarch64/Makefile.include            |   2 +-
 tests/vm/Makefile.include                     |  22 +++-
 tests/vm/basevm.py                            |   3 +
 tests/vm/freebsd                              |   3 +-
 tests/vm/netbsd                               |   3 +-
 tests/vm/ubuntu.i386                          |   3 +-
 15 files changed, 213 insertions(+), 55 deletions(-)
 create mode 100644 .cirrus.yml
 create mode 100644 tests/docker/dockerfiles/debian-buster-arm64-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian10.docker

-- 
2.20.1

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 01/16] .cirrus.yml: basic compile and test for FreeBSD
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 02/16] .travis.yml: stop requesting libffi & gettext from homebrew Alex Bennée
                   ` (15 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Ed Maste, Alex Bennée

From: Ed Maste <emaste@freebsd.org>

Signed-off-by: Ed Maste <emaste@freebsd.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .cirrus.yml | 16 ++++++++++++++++
 MAINTAINERS |  8 ++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 .cirrus.yml

diff --git a/.cirrus.yml b/.cirrus.yml
new file mode 100644
index 0000000000..303fe720d6
--- /dev/null
+++ b/.cirrus.yml
@@ -0,0 +1,16 @@
+freebsd_12_task:
+  freebsd_instance:
+    image: freebsd-12-0-release-amd64
+    cpu: 8
+    memory: 8G
+  env:
+    CIRRUS_CLONE_DEPTH: 1
+  install_script: pkg install -y
+    bison curl cyrus-sasl git glib gmake gnutls
+    nettle perl5 pixman pkgconf png usbredir
+  script:
+    - mkdir build
+    - cd build
+    - ../configure || { cat config.log; exit 1; }
+    - gmake -j8
+    - gmake -j8 V=1 check
diff --git a/MAINTAINERS b/MAINTAINERS
index 9a76845581..eecfafad56 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2475,6 +2475,14 @@ W: https://travis-ci.org/qemu/qemu
 W: https://app.shippable.com/github/qemu/qemu
 W: http://patchew.org/QEMU/
 
+FreeBSD Hosted Continuous Integration
+M: Ed Maste <emaste@freebsd.org>
+M: Li-Wen Hsu <lwhsu@freebsd.org>
+L: qemu-devel@nongnu.org
+S: Maintained
+F: .cirrus.yml
+W: https://cirrus-ci.com/github/qemu/qemu
+
 Guest Test Compilation Support
 M: Alex Bennée <alex.bennee@linaro.org>
 R: Philippe Mathieu-Daudé <f4bug@amsat.org>
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 02/16] .travis.yml: stop requesting libffi & gettext from homebrew
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 01/16] .cirrus.yml: basic compile and test for FreeBSD Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 03/16] .travis.yml: separate tools and docs into another entry Alex Bennée
                   ` (14 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Daniel P. Berrangé, Alex Bennée

From: Daniel P. Berrangé <berrange@redhat.com>

The default package set installed on macOS builders from Travis already
includes libffi and gettext as shown by log messages:

  Skipping install of libffi formula. It is already up-to-date.
  Using libffi
  Skipping install of gettext formula. It is already up-to-date.
  Using gettext

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .travis.yml | 2 --
 1 file changed, 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 87d9fa971c..beea941408 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -40,8 +40,6 @@ addons:
       - gcovr
   homebrew:
     packages:
-      - libffi
-      - gettext
       - glib
       - pixman
 
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 03/16] .travis.yml: separate tools and docs into another entry
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 01/16] .cirrus.yml: basic compile and test for FreeBSD Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 02/16] .travis.yml: stop requesting libffi & gettext from homebrew Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 04/16] .travis.yml: fold --disable-tcg into alternate coroutine builds Alex Bennée
                   ` (13 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée, Philippe Mathieu-Daudé

Re-building the tools and documents by default is a little wasteful as
they are not really affected by the main build options. Split tools
and documents into their own task with a minimal softmmu and
linux-user target list just to check they don't interact badly.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index beea941408..652be702ac 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -59,6 +59,7 @@ env:
   global:
     - SRC_DIR="."
     - BUILD_DIR="."
+    - BASE_CONFIG="--disable-docs --disable-tools"
     - TEST_CMD="make check -j3 V=1"
 
 
@@ -69,7 +70,7 @@ git:
 
 before_script:
   - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
-  - ${SRC_DIR}/configure ${CONFIG} || { cat config.log && exit 1; }
+  - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
 script:
   - make -j3 && ${TEST_CMD}
 
@@ -104,6 +105,17 @@ matrix:
         - CONFIG="--with-coroutine=sigaltstack --disable-linux-user"
 
 
+    # Check we can build docs and tools
+    - env:
+        - BASE_CONFIG="--enable-tools --enable-docs"
+        - CONFIG="--target-list=x86_64-softmmu,aarch64-linux-user"
+      addons:
+        apt:
+          packages:
+            - python-sphinx
+            - texinfo
+            - perl
+
     # Test out-of-tree builds
     - env:
         - CONFIG="--enable-debug --enable-debug-tcg"
@@ -241,5 +253,5 @@ matrix:
 
 
     - env:
-        - CONFIG="--disable-system --disable-docs"
+        - CONFIG="--disable-system"
         - TEST_CMD="make -j3 check-tcg V=1"
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 04/16] .travis.yml: fold --disable-tcg into alternate coroutine builds
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (2 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 03/16] .travis.yml: separate tools and docs into another entry Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 05/16] MAINTAINERS: Add an entry for scripts/archive-source.sh Alex Bennée
                   ` (12 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée

The alternate coroutine builds are really only of interest to people
running KVM (although I think you could use them for TCG if you really
tried). As they tend to run long lets kill two birds with one stone
and fold the --disable-tcg build into them.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .travis.yml | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 652be702ac..baa06b976a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -97,12 +97,16 @@ matrix:
         - CONFIG="--enable-modules --disable-linux-user"
 
 
+    # Alternate coroutines implementations are only really of interest to KVM users
+    # However we can't test against KVM on Travis so we can only run unit tests
     - env:
-        - CONFIG="--with-coroutine=ucontext --disable-linux-user"
+        - CONFIG="--with-coroutine=ucontext --disable-tcg"
+        - TEST_CMD="make check-unit -j3 V=1"
 
 
     - env:
-        - CONFIG="--with-coroutine=sigaltstack --disable-linux-user"
+        - CONFIG="--with-coroutine=sigaltstack --disable-tcg"
+        - TEST_CMD="make check-unit -j3 V=1"
 
 
     # Check we can build docs and tools
@@ -162,11 +166,6 @@ matrix:
         - TEST_CMD=""
 
 
-    - env:
-        - CONFIG="--disable-tcg"
-        - TEST_CMD=""
-
-
     # MacOSX builds
     - env:
         - CONFIG="--target-list=aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 05/16] MAINTAINERS: Add an entry for scripts/archive-source.sh
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (3 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 04/16] .travis.yml: fold --disable-tcg into alternate coroutine builds Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 06/16] archive-source.sh: Clone the submodules locally Alex Bennée
                   ` (11 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Philippe Mathieu-Daudé, Alex Bennée

From: Philippe Mathieu-Daudé <philmd@redhat.com>

The scripts/archive-source.sh is used by the VM tests, it makes
sense to add it in the "Build and test automation" section.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index eecfafad56..a2da141a92 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2471,6 +2471,7 @@ F: scripts/travis/
 F: .shippable.yml
 F: tests/docker/
 F: tests/vm/
+F: scripts/archive-source.sh
 W: https://travis-ci.org/qemu/qemu
 W: https://app.shippable.com/github/qemu/qemu
 W: http://patchew.org/QEMU/
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 06/16] archive-source.sh: Clone the submodules locally
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (4 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 05/16] MAINTAINERS: Add an entry for scripts/archive-source.sh Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 07/16] docker: add debian-buster-arm64-cross Alex Bennée
                   ` (10 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Philippe Mathieu-Daudé, Alex Bennée

From: Philippe Mathieu-Daudé <philmd@redhat.com>

We cloned the QEMU repository from the local storage. Since the
submodules are also available there, clone them too. This is
quicker and reduce network use.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
[AJB: incorporated review suggestions from danpb]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 scripts/archive-source.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 6eed2a29bd..d3a88f71e9 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -38,6 +38,13 @@ else
 fi
 git clone --shared . "$vroot_dir"
 test $? -ne 0 && error "failed to clone into '$vroot_dir'"
+for sm in $submodules; do
+    if test -d "$sm/.git"
+    then
+       git clone --shared "$sm" "$vroot_dir/$sm"
+       test $? -ne 0 && error "failed to clone submodule $sm"
+    fi
+done
 
 cd "$vroot_dir"
 test $? -ne 0 && error "failed to change into '$vroot_dir'"
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 07/16] docker: add debian-buster-arm64-cross
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (5 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 06/16] archive-source.sh: Clone the submodules locally Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 08/16] tests: make docker.py update use configured binfmt path Alex Bennée
                   ` (9 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée

We can't build QEMU with this but we can use this image to build newer
arm64 testcases which need more up to date tools.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include                 |  1 +
 .../debian-buster-arm64-cross.docker          | 15 +++++++++
 tests/docker/dockerfiles/debian10.docker      | 31 +++++++++++++++++++
 tests/tcg/aarch64/Makefile.include            |  2 +-
 4 files changed, 48 insertions(+), 1 deletion(-)
 create mode 100644 tests/docker/dockerfiles/debian-buster-arm64-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian10.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 7032c68895..83d43c50e4 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -90,6 +90,7 @@ docker-image-debian-amd64: docker-image-debian9
 docker-image-debian-armel-cross: docker-image-debian9
 docker-image-debian-armhf-cross: docker-image-debian9
 docker-image-debian-arm64-cross: docker-image-debian9
+docker-image-debian-buster-arm64-cross: docker-image-debian10
 docker-image-debian-mips-cross: docker-image-debian9
 docker-image-debian-mipsel-cross: docker-image-debian9
 docker-image-debian-mips64el-cross: docker-image-debian9
diff --git a/tests/docker/dockerfiles/debian-buster-arm64-cross.docker b/tests/docker/dockerfiles/debian-buster-arm64-cross.docker
new file mode 100644
index 0000000000..33ada13806
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-buster-arm64-cross.docker
@@ -0,0 +1,15 @@
+#
+# Docker arm64 cross-compiler target
+#
+# This docker target builds on the Debian's Buster base image. There
+# are no QEMU pre-requistes so this image can only be used to build
+# test cases.
+#
+FROM qemu:debian10
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture arm64
+RUN apt update
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt-get install -y --no-install-recommends \
+        crossbuild-essential-arm64
diff --git a/tests/docker/dockerfiles/debian10.docker b/tests/docker/dockerfiles/debian10.docker
new file mode 100644
index 0000000000..aeeb151b52
--- /dev/null
+++ b/tests/docker/dockerfiles/debian10.docker
@@ -0,0 +1,31 @@
+#
+# Docker multiarch cross-compiler target
+#
+# This docker target is builds on Debian cross compiler targets to build distro
+# with a selection of cross compilers for building test binaries.
+#
+# On its own you can't build much but the docker-foo-cross targets
+# build on top of the base debian image.
+#
+FROM debian:buster-slim
+
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
+
+# Install common build utilities
+RUN apt update
+RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+    apt install -y --no-install-recommends \
+        bison \
+        build-essential \
+        ca-certificates \
+        clang \
+        flex \
+        gettext \
+        git \
+        pkg-config \
+        psmisc \
+        python \
+        texinfo \
+        $(apt-get -s build-dep qemu | egrep ^Inst | fgrep '[all]' | cut -d\  -f2)
diff --git a/tests/tcg/aarch64/Makefile.include b/tests/tcg/aarch64/Makefile.include
index de32c91235..5d4e4c6f99 100644
--- a/tests/tcg/aarch64/Makefile.include
+++ b/tests/tcg/aarch64/Makefile.include
@@ -3,6 +3,6 @@
 # We don't have any bigendian build tools so we only use this for AArch64
 
 ifeq ($(TARGET_NAME),aarch64)
-DOCKER_IMAGE=debian-arm64-cross
+DOCKER_IMAGE=debian-buster-arm64-cross
 DOCKER_CROSS_COMPILER=aarch64-linux-gnu-gcc
 endif
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 08/16] tests: make docker.py update use configured binfmt path
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (6 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 07/16] docker: add debian-buster-arm64-cross Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 09/16] tests: make docker.py check for persistent configs Alex Bennée
                   ` (8 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée

When copying a QEMU binary into a linux-user docker image we should
check what the current configured binfmt_misc path is rather than
just assuming "/usr/bin/qemu-bin". Obviously if the user changes the
configuration afterwards they will break their images again.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 02d8a83847..30f463af9f 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -123,17 +123,17 @@ def _check_binfmt_misc(executable):
 
     if not os.path.exists(binfmt_entry):
         print ("No binfmt_misc entry for %s" % (binary))
-        return False
+        return None
 
     with open(binfmt_entry) as x: entry = x.read()
 
-    qpath = "/usr/bin/%s" % (binary)
-    if not re.search("interpreter %s\n" % (qpath), entry):
-        print ("binfmt_misc for %s does not point to %s" % (binary, qpath))
-        return False
-
-    return True
+    m = re.search("interpreter (\S+)\n", entry)
+    interp = m.group(1)
+    if interp and interp != executable:
+        print("binfmt_misc for %s does not point to %s, using %s" %
+              (binary, executable, interp))
 
+    return interp
 
 def _read_qemu_dockerfile(img_name):
     # special case for Debian linux-user images
@@ -394,9 +394,14 @@ class UpdateCommand(SubCommand):
         tmp = tempfile.NamedTemporaryFile(suffix="dckr.tar.gz")
         tmp_tar = TarFile(fileobj=tmp, mode='w')
 
-        # Add the executable to the tarball
-        bn = os.path.basename(args.executable)
-        ff = "/usr/bin/%s" % bn
+        # Add the executable to the tarball, using the current
+        # configured binfmt_misc path.
+        ff = _check_binfmt_misc(args.executable)
+        if not ff:
+            bn = os.path.basename(args.executable)
+            ff = "/usr/bin/%s" % bn
+            print ("No binfmt_misc configured: copied to %s" % (ff))
+
         tmp_tar.add(args.executable, arcname=ff)
 
         # Add any associated libraries
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 09/16] tests: make docker.py check for persistent configs
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (7 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 08/16] tests: make docker.py update use configured binfmt path Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 10/16] tests: docker.py be even smarter with persistent binfmt_misc Alex Bennée
                   ` (7 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée

binfmt_misc configured with the "F" flag opens the interpreter at
config time. This means it can use an already open file-descriptor to
run QEMU so there is no point trying to copy the binary into a
container.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 30f463af9f..768728785f 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -127,6 +127,11 @@ def _check_binfmt_misc(executable):
 
     with open(binfmt_entry) as x: entry = x.read()
 
+    if re.search("flags:.*F.*\n", entry):
+        print("binfmt_misc for %s uses persistent(F) mapping to host binary\n" %
+              (binary))
+        return None
+
     m = re.search("interpreter (\S+)\n", entry)
     interp = m.group(1)
     if interp and interp != executable:
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 10/16] tests: docker.py be even smarter with persistent binfmt_misc
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (8 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 09/16] tests: make docker.py check for persistent configs Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 11/16] tests: PEP8 cleanup of docker.py, mostly white space Alex Bennée
                   ` (6 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée

If we have a persistent mapping we don't need the QEMU binary copied
into the container as the kernel has already opened the file and will
pass the fd in. However the support libraries will still need to be
there.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 56 ++++++++++++++++++++++++++++--------------
 1 file changed, 38 insertions(+), 18 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 768728785f..a74338cb61 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -96,13 +96,22 @@ def _get_so_libs(executable):
 
     return libs
 
-def _copy_binary_with_libs(src, dest_dir):
-    """Copy a binary executable and all its dependent libraries.
+def _copy_binary_with_libs(src, bin_dest, dest_dir):
+    """Maybe copy a binary and all its dependent libraries.
+
+    If bin_dest isn't set we only copy the support libraries because
+    we don't need qemu in the docker path to run (due to persistent
+    mapping). Indeed users may get confused if we aren't running what
+    is in the image.
 
     This does rely on the host file-system being fairly multi-arch
-    aware so the file don't clash with the guests layout."""
+    aware so the file don't clash with the guests layout.
+    """
 
-    _copy_with_mkdir(src, dest_dir, "/usr/bin")
+    if bin_dest:
+        _copy_with_mkdir(src, dest_dir, os.path.dirname(bin_dest))
+    else:
+        print("only copying support libraries for %s" % (src))
 
     libs = _get_so_libs(src)
     if libs:
@@ -116,21 +125,26 @@ def _check_binfmt_misc(executable):
 
     The details of setting up binfmt_misc are outside the scope of
     this script but we should at least fail early with a useful
-    message if it won't work."""
+    message if it won't work.
+
+    Returns the configured binfmt path and a valid flag. For
+    persistent configurations we will still want to copy and dependent
+    libraries.
+    """
 
     binary = os.path.basename(executable)
     binfmt_entry = "/proc/sys/fs/binfmt_misc/%s" % (binary)
 
     if not os.path.exists(binfmt_entry):
         print ("No binfmt_misc entry for %s" % (binary))
-        return None
+        return None, False
 
     with open(binfmt_entry) as x: entry = x.read()
 
     if re.search("flags:.*F.*\n", entry):
         print("binfmt_misc for %s uses persistent(F) mapping to host binary\n" %
               (binary))
-        return None
+        return None, True
 
     m = re.search("interpreter (\S+)\n", entry)
     interp = m.group(1)
@@ -138,7 +152,8 @@ def _check_binfmt_misc(executable):
         print("binfmt_misc for %s does not point to %s, using %s" %
               (binary, executable, interp))
 
-    return interp
+    return interp, True
+
 
 def _read_qemu_dockerfile(img_name):
     # special case for Debian linux-user images
@@ -345,7 +360,8 @@ class BuildCommand(SubCommand):
 
             # Validate binfmt_misc will work
             if args.include_executable:
-                if not _check_binfmt_misc(args.include_executable):
+                qpath, enabled = _check_binfmt_misc(args.include_executable)
+                if not enabled:
                     return 1
 
             # Is there a .pre file to run in the build context?
@@ -368,7 +384,9 @@ class BuildCommand(SubCommand):
                 # FIXME: there is no checksum of this executable and the linked
                 # libraries, once the image built any change of this executable
                 # or any library won't trigger another build.
-                _copy_binary_with_libs(args.include_executable, docker_dir)
+                _copy_binary_with_libs(args.include_executable,
+                                       qpath, docker_dir)
+
             for filename in args.extra_files or []:
                 _copy_with_mkdir(filename, docker_dir)
                 cksum += [(filename, _file_checksum(filename))]
@@ -400,14 +418,16 @@ class UpdateCommand(SubCommand):
         tmp_tar = TarFile(fileobj=tmp, mode='w')
 
         # Add the executable to the tarball, using the current
-        # configured binfmt_misc path.
-        ff = _check_binfmt_misc(args.executable)
-        if not ff:
-            bn = os.path.basename(args.executable)
-            ff = "/usr/bin/%s" % bn
-            print ("No binfmt_misc configured: copied to %s" % (ff))
-
-        tmp_tar.add(args.executable, arcname=ff)
+        # configured binfmt_misc path. If we don't get a path then we
+        # only need the support libraries copied
+        ff, enabled = _check_binfmt_misc(args.executable)
+
+        if not enabled:
+            print("binfmt_misc not enabled, update disabled")
+            return 1
+
+        if ff:
+            tmp_tar.add(args.executable, arcname=ff)
 
         # Add any associated libraries
         libs = _get_so_libs(args.executable)
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH  v2 11/16] tests: PEP8 cleanup of docker.py, mostly white space
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (9 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 10/16] tests: docker.py be even smarter with persistent binfmt_misc Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 12/16] tests/vm: move images to $HOME/.cache/qemu-vm/images Alex Bennée
                   ` (5 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée

My editor keeps putting squiggly lines under a bunch of the python
lines to remind me how non-PEP8 compliant it is. Clean that up so it's
easier to spot new errors.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 48 ++++++++++++++++++++++++++++++------------
 1 file changed, 35 insertions(+), 13 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index a74338cb61..53a8c9c801 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -30,7 +30,7 @@ except ImportError:
     from io import StringIO
 from shutil import copy, rmtree
 from pwd import getpwuid
-from datetime import datetime,timedelta
+from datetime import datetime, timedelta
 
 
 FILTERED_ENV_NAMES = ['ftp_proxy', 'http_proxy', 'https_proxy']
@@ -43,9 +43,11 @@ def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
     return hashlib.sha1(text).hexdigest()
 
+
 def _file_checksum(filename):
     return _text_checksum(open(filename, 'rb').read())
 
+
 def _guess_docker_command():
     """ Guess a working docker command or raise exception if not found"""
     commands = [["docker"], ["sudo", "-n", "docker"]]
@@ -59,9 +61,10 @@ def _guess_docker_command():
         except OSError:
             pass
     commands_txt = "\n".join(["  " + " ".join(x) for x in commands])
-    raise Exception("Cannot find working docker command. Tried:\n%s" % \
+    raise Exception("Cannot find working docker command. Tried:\n%s" %
                     commands_txt)
 
+
 def _copy_with_mkdir(src, root_dir, sub_path='.'):
     """Copy src into root_dir, creating sub_path as needed."""
     dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
@@ -96,6 +99,7 @@ def _get_so_libs(executable):
 
     return libs
 
+
 def _copy_binary_with_libs(src, bin_dest, dest_dir):
     """Maybe copy a binary and all its dependent libraries.
 
@@ -117,7 +121,7 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
     if libs:
         for l in libs:
             so_path = os.path.dirname(l)
-            _copy_with_mkdir(l , dest_dir, so_path)
+            _copy_with_mkdir(l, dest_dir, so_path)
 
 
 def _check_binfmt_misc(executable):
@@ -142,7 +146,7 @@ def _check_binfmt_misc(executable):
     with open(binfmt_entry) as x: entry = x.read()
 
     if re.search("flags:.*F.*\n", entry):
-        print("binfmt_misc for %s uses persistent(F) mapping to host binary\n" %
+        print("binfmt_misc for %s uses persistent(F) mapping to host binary" %
               (binary))
         return None, True
 
@@ -164,6 +168,7 @@ def _read_qemu_dockerfile(img_name):
                       img_name + ".docker")
     return open(df, "r").read()
 
+
 def _dockerfile_preprocess(df):
     out = ""
     for l in df.splitlines():
@@ -181,6 +186,7 @@ def _dockerfile_preprocess(df):
         out += l + "\n"
     return out
 
+
 class Docker(object):
     """ Running Docker commands """
     def __init__(self):
@@ -248,7 +254,7 @@ class Docker(object):
 
     def build_image(self, tag, docker_dir, dockerfile,
                     quiet=True, user=False, argv=None, extra_files_cksum=[]):
-        if argv == None:
+        if argv is None:
             argv = []
 
         tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker")
@@ -269,7 +275,7 @@ class Docker(object):
 
         tmp_df.flush()
 
-        self._do_check(["build", "-t", tag, "-f", tmp_df.name] + argv + \
+        self._do_check(["build", "-t", tag, "-f", tmp_df.name] + argv +
                        [docker_dir],
                        quiet=quiet)
 
@@ -299,9 +305,11 @@ class Docker(object):
     def command(self, cmd, argv, quiet):
         return self._do([cmd] + argv, quiet=quiet)
 
+
 class SubCommand(object):
     """A SubCommand template base class"""
-    name = None # Subcommand name
+    name = None  # Subcommand name
+
     def shared_args(self, parser):
         parser.add_argument("--quiet", action="store_true",
                             help="Run quietly unless an error occurred")
@@ -309,6 +317,7 @@ class SubCommand(object):
     def args(self, parser):
         """Setup argument parser"""
         pass
+
     def run(self, args, argv):
         """Run command.
         args: parsed argument by argument parser.
@@ -316,18 +325,23 @@ class SubCommand(object):
         """
         pass
 
+
 class RunCommand(SubCommand):
     """Invoke docker run and take care of cleaning up"""
     name = "run"
+
     def args(self, parser):
         parser.add_argument("--keep", action="store_true",
                             help="Don't remove image when command completes")
+
     def run(self, args, argv):
         return Docker().run(argv, args.keep, quiet=args.quiet)
 
+
 class BuildCommand(SubCommand):
-    """ Build docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
+    """ Build docker image out of a dockerfile. Arg: <tag> <dockerfile>"""
     name = "build"
+
     def args(self, parser):
         parser.add_argument("--include-executable", "-e",
                             help="""Specify a binary that will be copied to the
@@ -392,8 +406,8 @@ class BuildCommand(SubCommand):
                 cksum += [(filename, _file_checksum(filename))]
 
             argv += ["--build-arg=" + k.lower() + "=" + v
-                        for k, v in os.environ.iteritems()
-                        if k.lower() in FILTERED_ENV_NAMES]
+                     for k, v in os.environ.iteritems()
+                     if k.lower() in FILTERED_ENV_NAMES]
             dkr.build_image(tag, docker_dir, dockerfile,
                             quiet=args.quiet, user=args.user, argv=argv,
                             extra_files_cksum=cksum)
@@ -402,9 +416,11 @@ class BuildCommand(SubCommand):
 
         return 0
 
+
 class UpdateCommand(SubCommand):
-    """ Update a docker image with new executables. Arguments: <tag> <executable>"""
+    """ Update a docker image with new executables. Args: <tag> <executable>"""
     name = "update"
+
     def args(self, parser):
         parser.add_argument("tag",
                             help="Image Tag")
@@ -457,16 +473,20 @@ class UpdateCommand(SubCommand):
 
         return 0
 
+
 class CleanCommand(SubCommand):
     """Clean up docker instances"""
     name = "clean"
+
     def run(self, args, argv):
         Docker().clean()
         return 0
 
+
 class ImagesCommand(SubCommand):
     """Run "docker images" command"""
     name = "images"
+
     def run(self, args, argv):
         return Docker().command("images", argv, args.quiet)
 
@@ -539,7 +559,7 @@ class CheckCommand(SubCommand):
 
         try:
             dkr = Docker()
-        except:
+        except subprocess.CalledProcessError:
             print("Docker not set up")
             return 1
 
@@ -578,7 +598,8 @@ class CheckCommand(SubCommand):
 
 def main():
     parser = argparse.ArgumentParser(description="A Docker helper",
-            usage="%s <subcommand> ..." % os.path.basename(sys.argv[0]))
+                                     usage="%s <subcommand> ..." %
+                                     os.path.basename(sys.argv[0]))
     subparsers = parser.add_subparsers(title="subcommands", help=None)
     for cls in SubCommand.__subclasses__():
         cmd = cls()
@@ -589,5 +610,6 @@ def main():
     args, argv = parser.parse_known_args()
     return args.cmdobj.run(args, argv)
 
+
 if __name__ == "__main__":
     sys.exit(main())
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 12/16] tests/vm: move images to $HOME/.cache/qemu-vm/images
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (10 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 11/16] tests: PEP8 cleanup of docker.py, mostly white space Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 13/16] tests/vm: call make check directly for netbsd/freebsd/ubuntu.i386 Alex Bennée
                   ` (4 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel
  Cc: qemu-arm, Gerd Hoffmann, Philippe Mathieu-Daudé, Alex Bennée

From: Gerd Hoffmann <kraxel@redhat.com>

It's easier to move around the images then, by replacing the
subdirectory with a symlink.  Allows to share the images between
multiple qemu checkouts for example.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/vm/Makefile.include | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index a98fb3027f..a58383d263 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -3,7 +3,8 @@
 .PHONY: vm-build-all vm-clean-all
 
 IMAGES := ubuntu.i386 freebsd netbsd openbsd centos
-IMAGE_FILES := $(patsubst %, tests/vm/%.img, $(IMAGES))
+IMAGES_DIR := $(HOME)/.cache/qemu-vm/images
+IMAGE_FILES := $(patsubst %, $(IMAGES_DIR)/%.img, $(IMAGES))
 
 .PRECIOUS: $(IMAGE_FILES)
 
@@ -24,9 +25,10 @@ vm-build-all: $(addprefix vm-build-, $(IMAGES))
 vm-clean-all:
 	rm -f $(IMAGE_FILES)
 
-tests/vm/%.img: $(SRC_PATH)/tests/vm/% \
-		$(SRC_PATH)/tests/vm/basevm.py \
-		$(SRC_PATH)/tests/vm/Makefile.include
+$(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \
+			$(SRC_PATH)/tests/vm/basevm.py \
+			$(SRC_PATH)/tests/vm/Makefile.include
+	@mkdir -p $(IMAGES_DIR)
 	$(call quiet-command, \
 		$< \
 		$(if $(V)$(DEBUG), --debug) \
@@ -37,7 +39,7 @@ tests/vm/%.img: $(SRC_PATH)/tests/vm/% \
 
 
 # Build in VM $(IMAGE)
-vm-build-%: tests/vm/%.img
+vm-build-%: $(IMAGES_DIR)/%.img
 	$(call quiet-command, \
 		$(SRC_PATH)/tests/vm/$* \
 		$(if $(V)$(DEBUG), --debug) \
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 13/16] tests/vm: call make check directly for netbsd/freebsd/ubuntu.i386
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (11 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 12/16] tests/vm: move images to $HOME/.cache/qemu-vm/images Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 14/16] tests/vm: add --build-target option Alex Bennée
                   ` (3 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée, Philippe Mathieu-Daudé

The "make check" target calls check-qtest which has the appropriate
system binaries as dependencies so we shouldn't need to do two steps
of make invocation. Doing it in two steps was a hangover from when our
make check couldn't run tests in parallel.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/vm/freebsd     | 1 -
 tests/vm/netbsd      | 1 -
 tests/vm/ubuntu.i386 | 1 -
 3 files changed, 3 deletions(-)

diff --git a/tests/vm/freebsd b/tests/vm/freebsd
index 19a3729172..a85c866c30 100755
--- a/tests/vm/freebsd
+++ b/tests/vm/freebsd
@@ -25,7 +25,6 @@ class FreeBSDVM(basevm.BaseVM):
         cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
         tar -xf /dev/vtbd1;
         ./configure {configure_opts};
-        gmake --output-sync -j{jobs} {verbose};
         gmake --output-sync -j{jobs} check {verbose};
     """
 
diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index fac6a7ce51..edea113bb5 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -25,7 +25,6 @@ class NetBSDVM(basevm.BaseVM):
         cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
         tar -xf /dev/rld1a;
         ./configure --python=python2.7 {configure_opts};
-        gmake --output-sync -j{jobs} {verbose};
         gmake --output-sync -j{jobs} check {verbose};
     """
 
diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
index 1b7e1ab8f0..252e514859 100755
--- a/tests/vm/ubuntu.i386
+++ b/tests/vm/ubuntu.i386
@@ -26,7 +26,6 @@ class UbuntuX86VM(basevm.BaseVM):
         sudo chmod a+r /dev/vdb;
         tar -xf /dev/vdb;
         ./configure {configure_opts};
-        make --output-sync -j{jobs};
         make --output-sync check -j{jobs} {verbose};
     """
 
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 14/16] tests/vm: add --build-target option
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (12 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 13/16] tests/vm: call make check directly for netbsd/freebsd/ubuntu.i386 Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 15/16] tests/vm: expose BUILD_TARGET, TARGET_LIST and EXTRA_CONFIGURE_OPTS Alex Bennée
                   ` (2 subsequent siblings)
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée, Philippe Mathieu-Daudé

This allows us to invoke the build with a custom target (for the VMs
that use the {target} format string specifier). Currently OpenBSD is
still hardwired due to problems running check.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/vm/basevm.py   | 3 +++
 tests/vm/freebsd     | 2 +-
 tests/vm/netbsd      | 2 +-
 tests/vm/ubuntu.i386 | 2 +-
 4 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 5caf77d6b8..bdca6cb2fc 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -228,6 +228,8 @@ def parse_args(vmcls):
                       help="build image")
     parser.add_option("--build-qemu",
                       help="build QEMU from source in guest")
+    parser.add_option("--build-target",
+                      help="QEMU build target", default="check")
     parser.add_option("--interactive", "-I", action="store_true",
                       help="Interactively run command")
     parser.add_option("--snapshot", "-s", action="store_true",
@@ -255,6 +257,7 @@ def main(vmcls):
             cmd = [vm.BUILD_SCRIPT.format(
                    configure_opts = " ".join(argv),
                    jobs=args.jobs,
+                   target=args.build_target,
                    verbose = "V=1" if args.verbose else "")]
         else:
             cmd = argv
diff --git a/tests/vm/freebsd b/tests/vm/freebsd
index a85c866c30..1d64c31b7c 100755
--- a/tests/vm/freebsd
+++ b/tests/vm/freebsd
@@ -25,7 +25,7 @@ class FreeBSDVM(basevm.BaseVM):
         cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
         tar -xf /dev/vtbd1;
         ./configure {configure_opts};
-        gmake --output-sync -j{jobs} check {verbose};
+        gmake --output-sync -j{jobs} {target} {verbose};
     """
 
     def build_image(self, img):
diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index edea113bb5..a2c3d13af4 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -25,7 +25,7 @@ class NetBSDVM(basevm.BaseVM):
         cd $(mktemp -d /var/tmp/qemu-test.XXXXXX);
         tar -xf /dev/rld1a;
         ./configure --python=python2.7 {configure_opts};
-        gmake --output-sync -j{jobs} check {verbose};
+        gmake --output-sync -j{jobs} {target} {verbose};
     """
 
     def build_image(self, img):
diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
index 252e514859..a22d137e76 100755
--- a/tests/vm/ubuntu.i386
+++ b/tests/vm/ubuntu.i386
@@ -26,7 +26,7 @@ class UbuntuX86VM(basevm.BaseVM):
         sudo chmod a+r /dev/vdb;
         tar -xf /dev/vdb;
         ./configure {configure_opts};
-        make --output-sync check -j{jobs} {verbose};
+        make --output-sync {target} -j{jobs} {verbose};
     """
 
     def _gen_cloud_init_iso(self):
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH  v2 15/16] tests/vm: expose BUILD_TARGET, TARGET_LIST and EXTRA_CONFIGURE_OPTS
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (13 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 14/16] tests/vm: add --build-target option Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 16/16] scripts/qemu.py: allow arches use KVM for their 32bit cousins Alex Bennée
  2019-02-07 22:01 ` [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Philippe Mathieu-Daudé
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée, Philippe Mathieu-Daudé

Now the underlying basevm support passes these along we can expose
some additional variables to our Makefile to allow more customised
tweaking of the build. For example:

  make vm-build-freebsd TARGET_LIST=aarch64-softmmu \
    EXTRA_CONFIGURE_OPTS="--disable-tools --disable-docs" \
    BUILD_TARGET=check-softfloat

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/vm/Makefile.include | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index a58383d263..992d823f6b 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -19,6 +19,11 @@ vm-test:
 	@echo ""
 	@echo "  vm-build-all                    - Build QEMU in all VMs"
 	@echo "  vm-clean-all                    - Clean up VM images"
+	@echo
+	@echo "Special variables:"
+	@echo "    BUILD_TARGET=foo		 - override the build target"
+	@echo "    TARGET_LIST=a,b,c    	 - Override target list in builds."
+	@echo '    EXTRA_CONFIGURE_OPTS="..."'
 
 vm-build-all: $(addprefix vm-build-, $(IMAGES))
 
@@ -47,6 +52,9 @@ vm-build-%: $(IMAGES_DIR)/%.img
 		$(if $(J),--jobs $(J)) \
 		$(if $(V),--verbose) \
 		--image "$<" \
-		--build-qemu $(SRC_PATH), \
+		$(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \
+		--build-qemu $(SRC_PATH) -- \
+		$(if $(TARGET_LIST),--target-list=$(TARGET_LIST)) \
+		$(if $(EXTRA_CONFIGURE_OPTS),$(EXTRA_CONFIGURE_OPTS)), \
 		"  VM-BUILD $*")
 
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [Qemu-devel] [PATCH v2 16/16] scripts/qemu.py: allow arches use KVM for their 32bit cousins
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (14 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 15/16] tests/vm: expose BUILD_TARGET, TARGET_LIST and EXTRA_CONFIGURE_OPTS Alex Bennée
@ 2019-02-07 18:37 ` Alex Bennée
  2019-02-07 22:01 ` [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Philippe Mathieu-Daudé
  16 siblings, 0 replies; 18+ messages in thread
From: Alex Bennée @ 2019-02-07 18:37 UTC (permalink / raw)
  To: qemu-devel; +Cc: qemu-arm, Alex Bennée

A lot of architectures can run their 32 bit cousins on KVM so the
kvm_available function needs to be a little less restricting when
deciding if KVM is available.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 scripts/qemu.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 0a5e02eb56..32b00af5cc 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -25,10 +25,18 @@ import tempfile
 
 LOG = logging.getLogger(__name__)
 
+# Mapping host architecture to any additional architectures it can
+# support which often includes its 32 bit cousin.
+ADDITIONAL_ARCHES = {
+    "x86_64" : "i386",
+    "aarch64" : "armhf"
+}
 
 def kvm_available(target_arch=None):
-    if target_arch and target_arch != os.uname()[4]:
-        return False
+    host_arch = os.uname()[4]
+    if target_arch and target_arch != host_arch:
+        if target_arch != ADDITIONAL_ARCHES.get(host_arch):
+            return False
     return os.access("/dev/kvm", os.R_OK | os.W_OK)
 
 
-- 
2.20.1

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* Re: [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes)
  2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
                   ` (15 preceding siblings ...)
  2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 16/16] scripts/qemu.py: allow arches use KVM for their 32bit cousins Alex Bennée
@ 2019-02-07 22:01 ` Philippe Mathieu-Daudé
  16 siblings, 0 replies; 18+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-02-07 22:01 UTC (permalink / raw)
  To: Alex Bennée, qemu-devel, Michael Tokarev; +Cc: qemu-arm

Cc'ing Michael.

On 2/7/19 7:37 PM, Alex Bennée wrote:
> 
> Hi,
> 
> There are a few more additions from the last iteration including some
> more fixes to try and get Travis green again. It's nice to see our ARM
> features are now catching up we need a newer toolchain to build
> things. I did try upgrading to debian 10 for the cross build but there
> are some upstream problems with the multiarch build deps that get in
> the way of that. People with Debian packaging experience can jump in
> on:
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921458
>   (dependancy on gcc-s390x-linux-gnu fails on non-x86 hosts)
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921648
>   (apt-get build-dep -a arm64 qemu fails on multiarch setup)
> 
> as it would be good to get these problems fixed as we get to
> softfreeze (and given how much we leverage Debian's cross toolchains).
> 
> If all the testing is green by tomorrow I'll put together a pull
> request. Unless people are desperate to grab the arm64 compiler image
> now as a build fix?
> 
> The following patches need review
>  patch 0004/.travis.yml fold disable tcg into alternate corou.patch
>  patch 0007/docker add debian buster arm64 cross.patch
>  patch 0008/tests make docker.py update use configured binfmt.patch
>  patch 0009/tests make docker.py check for persistent configs.patch
>  patch 0010/tests docker.py be even smarter with persistent b.patch
>  patch 0011/tests PEP8 cleanup of docker.py mostly white spac.patch
>  patch 0016/scripts qemu.py allow arches use KVM for their 32.patch
> 
> 
> Alex Bennée (11):
>   .travis.yml: separate tools and docs into another entry
>   .travis.yml: fold --disable-tcg into alternate coroutine builds
>   docker: add debian-buster-arm64-cross
>   tests: make docker.py update use configured binfmt path
>   tests: make docker.py check for persistent configs
>   tests: docker.py be even smarter with persistent binfmt_misc
>   tests: PEP8 cleanup of docker.py, mostly white space
>   tests/vm: call make check directly for netbsd/freebsd/ubuntu.i386
>   tests/vm: add --build-target option
>   tests/vm: expose BUILD_TARGET, TARGET_LIST and EXTRA_CONFIGURE_OPTS
>   scripts/qemu.py: allow arches use KVM for their 32bit cousins
> 
> Daniel P. Berrangé (1):
>   .travis.yml: stop requesting libffi & gettext from homebrew
> 
> Ed Maste (1):
>   .cirrus.yml: basic compile and test for FreeBSD
> 
> Gerd Hoffmann (1):
>   tests/vm: move images to $HOME/.cache/qemu-vm/images
> 
> Philippe Mathieu-Daudé (2):
>   MAINTAINERS: Add an entry for scripts/archive-source.sh
>   archive-source.sh: Clone the submodules locally
> 
>  .cirrus.yml                                   |  16 +++
>  .travis.yml                                   |  31 +++--
>  MAINTAINERS                                   |   9 ++
>  scripts/archive-source.sh                     |   7 ++
>  scripts/qemu.py                               |  12 +-
>  tests/docker/Makefile.include                 |   1 +
>  tests/docker/docker.py                        | 110 +++++++++++++-----
>  .../debian-buster-arm64-cross.docker          |  15 +++
>  tests/docker/dockerfiles/debian10.docker      |  31 +++++
>  tests/tcg/aarch64/Makefile.include            |   2 +-
>  tests/vm/Makefile.include                     |  22 +++-
>  tests/vm/basevm.py                            |   3 +
>  tests/vm/freebsd                              |   3 +-
>  tests/vm/netbsd                               |   3 +-
>  tests/vm/ubuntu.i386                          |   3 +-
>  15 files changed, 213 insertions(+), 55 deletions(-)
>  create mode 100644 .cirrus.yml
>  create mode 100644 tests/docker/dockerfiles/debian-buster-arm64-cross.docker
>  create mode 100644 tests/docker/dockerfiles/debian10.docker
> 

^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2019-02-07 22:01 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-07 18:37 [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 01/16] .cirrus.yml: basic compile and test for FreeBSD Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 02/16] .travis.yml: stop requesting libffi & gettext from homebrew Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 03/16] .travis.yml: separate tools and docs into another entry Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 04/16] .travis.yml: fold --disable-tcg into alternate coroutine builds Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 05/16] MAINTAINERS: Add an entry for scripts/archive-source.sh Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 06/16] archive-source.sh: Clone the submodules locally Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 07/16] docker: add debian-buster-arm64-cross Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 08/16] tests: make docker.py update use configured binfmt path Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 09/16] tests: make docker.py check for persistent configs Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 10/16] tests: docker.py be even smarter with persistent binfmt_misc Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 11/16] tests: PEP8 cleanup of docker.py, mostly white space Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 12/16] tests/vm: move images to $HOME/.cache/qemu-vm/images Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 13/16] tests/vm: call make check directly for netbsd/freebsd/ubuntu.i386 Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 14/16] tests/vm: add --build-target option Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 15/16] tests/vm: expose BUILD_TARGET, TARGET_LIST and EXTRA_CONFIGURE_OPTS Alex Bennée
2019-02-07 18:37 ` [Qemu-devel] [PATCH v2 16/16] scripts/qemu.py: allow arches use KVM for their 32bit cousins Alex Bennée
2019-02-07 22:01 ` [Qemu-devel] [PATCH v2 00/16] current testing/next queue (with build fixes) Philippe Mathieu-Daudé

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.