* [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts
@ 2018-07-18 10:04 Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 01/10] docker: rename docker-amd64 to docker-host Alex Bennée
` (10 more replies)
0 siblings, 11 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:04 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
Hi,
Our existing support for docker is fairly x86 centric. While docker
itself has support for multiple architectures not all architectures
are equal. For example Debian only packages the widest range of
cross-compilers in it's x86 images (although for example armhf is
available on aarch64 based images).
As the binfmt support is getting more solid we always have that fall
back option and I've converted a number of the guests that way.
Unfortunately I've overloaded the meaning of DOCKER_PARTIAL_IMAGES but
I was loathed to throw in yet another magic variable without some
discussion first. I'm not sure if my current path will end up with a
maze of twisty if/else statements all subtly alike.
Thoughts?
Alex Bennée (10):
docker: rename docker-amd64 to docker-host
docker: change docker-image to docker-all-images target
docker: add a placeholder for handling non-x86 hosts
docker: don't include docker-arm64-cross on aarch64 hosts
docker: fall-back to binfmt_misc debian-mips64el-user-cross on non-x86
docker: fall-back to binfmt_misc debian-ppc64el-user-cross on non-x86
docker: fall-back to binfmt_misc debian-s390x-user-cross on non-x86
docker: disable additional non-x86 images
tests: tcg skip docker images we can't build
tests/tcg: debian-mips64el-user-cross fallback
Makefile | 4 +-
tests/docker/Makefile.include | 58 +++++++++++++++++--
...debian-amd64.docker => debian-host.docker} | 4 +-
.../debian-mips64el-user-cross.docker | 16 +++++
.../debian-ppc64el-user-cross.docker | 16 +++++
.../debian-s390x-user-cross.docker | 16 +++++
tests/tcg/Makefile.include | 3 +
tests/tcg/Makefile.probe | 2 +-
tests/tcg/mips/Makefile.include | 5 ++
9 files changed, 116 insertions(+), 8 deletions(-)
rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host.docker} (91%)
create mode 100644 tests/docker/dockerfiles/debian-mips64el-user-cross.docker
create mode 100644 tests/docker/dockerfiles/debian-ppc64el-user-cross.docker
create mode 100644 tests/docker/dockerfiles/debian-s390x-user-cross.docker
--
2.17.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 01/10] docker: rename docker-amd64 to docker-host
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
@ 2018-07-18 10:04 ` Alex Bennée
2018-07-27 7:56 ` Fam Zheng
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 02/10] docker: change docker-image to docker-all-images target Alex Bennée
` (9 subsequent siblings)
10 siblings, 1 reply; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:04 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
When building on non-x86 systems the base system will be correct so if
we avoid too many x86'isms in the install we can still use the image.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.../dockerfiles/{debian-amd64.docker => debian-host.docker} | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host.docker} (91%)
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-host.docker
similarity index 91%
rename from tests/docker/dockerfiles/debian-amd64.docker
rename to tests/docker/dockerfiles/debian-host.docker
index eb13f06ed1..3605cc4658 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-host.docker
@@ -30,9 +30,9 @@ RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --with-glx --disabl
# netmap
RUN DEBIAN_FRONTEND=noninteractive eatmydata \
apt-get install -y --no-install-recommends \
- linux-headers-amd64
+ linux-headers-$(dpkg --print-architecture)
RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
-RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
+RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*) && make install
ENV QEMU_CONFIGURE_OPTS --enable-netmap
# gcrypt
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 02/10] docker: change docker-image to docker-all-images target
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 01/10] docker: rename docker-amd64 to docker-host Alex Bennée
@ 2018-07-18 10:04 ` Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 03/10] docker: add a placeholder for handling non-x86 hosts Alex Bennée
` (8 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:04 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
We also evaluate this list later in the build rule after all decisions
about what will and won't be built have been made.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c9e412f9d0..22adf6ac73 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -8,7 +8,6 @@ DOCKER_DEPRECATED_IMAGES := debian
# we don't run tests on intermediate images (used as base by another image)
DOCKER_PARTIAL_IMAGES := debian debian8 debian9 debian8-mxe debian-ports debian-sid debian-bootstrap
DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
-DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
# Use a global constant ccache directory to speed up repetitive builds
DOCKER_CCACHE_DIR := $$HOME/.cache/qemu-docker-ccache
@@ -34,8 +33,6 @@ $(DOCKER_SRC_COPY):
docker-qemu-src: $(DOCKER_SRC_COPY)
-docker-image: ${DOCKER_TARGETS}
-
# General rule for building docker images. If we are a sub-make
# invoked with SKIP_DOCKER_BUILD we still check the image is upto date
# though
@@ -154,6 +151,10 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
) \
)
+# Expand a rule to build all useable images
+DOCKER_TARGETS := $(patsubst %,docker-image-%,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES)))
+docker-all-images: $(DOCKER_TARGETS)
+
docker:
@echo 'Build QEMU and run tests inside Docker containers'
@echo
@@ -167,7 +168,7 @@ docker:
@echo ' Note: "TEST" is one of the listed test name,'
@echo ' or a script name under $$QEMU_SRC/tests/docker/;'
@echo ' "IMAGE" is one of the listed container name."'
- @echo ' docker-image: Build all images.'
+ @echo ' docker-all-images: Build all images.'
@echo ' docker-image-IMAGE: Build image "IMAGE".'
@echo ' docker-run: For manually running a "TEST" with "IMAGE"'
@echo
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 03/10] docker: add a placeholder for handling non-x86 hosts
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 01/10] docker: rename docker-amd64 to docker-host Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 02/10] docker: change docker-image to docker-all-images target Alex Bennée
@ 2018-07-18 10:04 ` Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 04/10] docker: don't include docker-arm64-cross on aarch64 hosts Alex Bennée
` (7 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:04 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
We want to keep all our variations in one place. This is that place.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 22adf6ac73..7d13ddd497 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -139,6 +139,18 @@ DOCKER_USER_IMAGES += debian-powerpc-user
# tests due to missing system call: 249 which causes an abort
DOCKER_PARTIAL_IMAGES += debian-powerpc-user-cross
+# Modifications for non-x86 hosts
+#
+# While docker itself is now multiarch aware and will generally do the
+# right thing (assuming the source also supports multiple
+# architectures) there are still variations we need to take into
+# account. These are all done here.
+
+ifneq ($(ARCH),x86_64)
+
+
+endif
+
# Expand all the pre-requistes for each docker image and test combination
$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 04/10] docker: don't include docker-arm64-cross on aarch64 hosts
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
` (2 preceding siblings ...)
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 03/10] docker: add a placeholder for handling non-x86 hosts Alex Bennée
@ 2018-07-18 10:04 ` Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 05/10] docker: fall-back to binfmt_misc debian-mips64el-user-cross on non-x86 Alex Bennée
` (6 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:04 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
Attempting to install the native compiler as a cross compiler rarely
goes well.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 7d13ddd497..2cb180ff6c 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -148,6 +148,10 @@ DOCKER_PARTIAL_IMAGES += debian-powerpc-user-cross
ifneq ($(ARCH),x86_64)
+ifeq ($(ARCH),aarch64)
+# no point having a cross compiler when it is the native one
+DOCKER_PARTIAL_IMAGES += debian-arm64-cross
+endif
endif
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 05/10] docker: fall-back to binfmt_misc debian-mips64el-user-cross on non-x86
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
` (3 preceding siblings ...)
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 04/10] docker: don't include docker-arm64-cross on aarch64 hosts Alex Bennée
@ 2018-07-18 10:05 ` Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 06/10] docker: fall-back to binfmt_misc debian-ppc64el-user-cross " Alex Bennée
` (5 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:05 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
The Debian archives don't package their large array of cross compilers
for all host architectures. For MIPS let's fall back to a linux-user
based image when we can't use the native ones.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 11 +++++++++++
.../debian-mips64el-user-cross.docker | 16 ++++++++++++++++
2 files changed, 27 insertions(+)
create mode 100644 tests/docker/dockerfiles/debian-mips64el-user-cross.docker
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 2cb180ff6c..5cfb1a57bb 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -153,8 +153,19 @@ ifeq ($(ARCH),aarch64)
DOCKER_PARTIAL_IMAGES += debian-arm64-cross
endif
+# No MIPS cross-compilers in non-x86 Debian images, so use linux-user
+docker-binfmt-image-debian-mips64el-user: DEB_ARCH = mips64el
+docker-binfmt-image-debian-mips64el-user: DEB_TYPE = stretch
+docker-binfmt-image-debian-mips64el-user: EXECUTABLE = ${BUILD_DIR}/mips64el-linux-user/qemu-mips64el
+docker-image-debian-mips64el-user-cross: docker-binfmt-image-debian-mips64el-user
+DOCKER_USER_IMAGES += debian-mips64el-user-cross
+DOCKER_PARTIAL_IMAGES += debian-mips-cross debian-mipsel-cross debian-mips64el-cross
+
+else
+DOCKER_PARTIAL_IMAGES += debian-mips64el-user-cross
endif
+
# Expand all the pre-requistes for each docker image and test combination
$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
diff --git a/tests/docker/dockerfiles/debian-mips64el-user-cross.docker b/tests/docker/dockerfiles/debian-mips64el-user-cross.docker
new file mode 100644
index 0000000000..98284bc635
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-mips64el-user-cross.docker
@@ -0,0 +1,16 @@
+#
+# Docker mips64el user cross-compiler target
+#
+# This is a binfmt image for when the MIPS cross compilers are not
+# available in the host architecture archives. This image is based of
+# the debootstrapped qemu:debian-mips64el-user but doesn't need any
+# extra magic once it is setup.
+#
+
+FROM qemu:debian-mips64el-user
+
+MAINTAINER Alex Bennee <alex.bennee@linaro.org>
+
+RUN echo man-db man-db/auto-update boolean false | debconf-set-selections
+RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get build-dep -yy qemu
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 06/10] docker: fall-back to binfmt_misc debian-ppc64el-user-cross on non-x86
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
` (4 preceding siblings ...)
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 05/10] docker: fall-back to binfmt_misc debian-mips64el-user-cross on non-x86 Alex Bennée
@ 2018-07-18 10:05 ` Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 07/10] docker: fall-back to binfmt_misc debian-s390x-user-cross " Alex Bennée
` (4 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:05 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
The Debian archives don't package their large array of cross compilers
for all host architectures. For ppc64el let's fall back to a linux-user
based image when we can't use the native ones.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 10 +++++++++-
.../dockerfiles/debian-ppc64el-user-cross.docker | 16 ++++++++++++++++
2 files changed, 25 insertions(+), 1 deletion(-)
create mode 100644 tests/docker/dockerfiles/debian-ppc64el-user-cross.docker
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 5cfb1a57bb..879bacde93 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -161,8 +161,16 @@ docker-image-debian-mips64el-user-cross: docker-binfmt-image-debian-mips64el-use
DOCKER_USER_IMAGES += debian-mips64el-user-cross
DOCKER_PARTIAL_IMAGES += debian-mips-cross debian-mipsel-cross debian-mips64el-cross
+# No ppc64el cross-compilers in non-x86 Debian images, so use linux-user
+docker-binfmt-image-debian-ppc64el-user: DEB_ARCH = ppc64el
+docker-binfmt-image-debian-ppc64el-user: DEB_TYPE = stretch
+docker-binfmt-image-debian-ppc64el-user: EXECUTABLE = ${BUILD_DIR}/ppc64le-linux-user/qemu-ppc64le
+docker-image-debian-ppc64el-user-cross: docker-binfmt-image-debian-ppc64el-user
+DOCKER_USER_IMAGES += debian-ppc64el-user-cross
+DOCKER_PARTIAL_IMAGES += debian-ppc64el-cross
+
else
-DOCKER_PARTIAL_IMAGES += debian-mips64el-user-cross
+DOCKER_PARTIAL_IMAGES += debian-mips64el-user-cross debian-ppc64el-user-cross
endif
diff --git a/tests/docker/dockerfiles/debian-ppc64el-user-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-user-cross.docker
new file mode 100644
index 0000000000..aeb207e9c2
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-ppc64el-user-cross.docker
@@ -0,0 +1,16 @@
+#
+# Docker ppc64el user cross-compiler target
+#
+# This is a binfmt image for when the ppc64el cross compilers are not
+# available in the host architecture archives. This image is based of
+# the debootstrapped qemu:debian-ppc64el-user but doesn't need any
+# extra magic once it is setup.
+#
+
+FROM qemu:debian-ppc64el-user
+
+MAINTAINER Alex Bennee <alex.bennee@linaro.org>
+
+RUN echo man-db man-db/auto-update boolean false | debconf-set-selections
+RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get build-dep -yy qemu
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 07/10] docker: fall-back to binfmt_misc debian-s390x-user-cross on non-x86
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
` (5 preceding siblings ...)
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 06/10] docker: fall-back to binfmt_misc debian-ppc64el-user-cross " Alex Bennée
@ 2018-07-18 10:05 ` Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 08/10] docker: disable additional non-x86 images Alex Bennée
` (3 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:05 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
The Debian archives don't package their large array of cross compilers
for all host architectures. For s390x let's fall back to a linux-user
based image when we can't use the native ones.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 9 ++++++++-
.../dockerfiles/debian-s390x-user-cross.docker | 16 ++++++++++++++++
2 files changed, 24 insertions(+), 1 deletion(-)
create mode 100644 tests/docker/dockerfiles/debian-s390x-user-cross.docker
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 879bacde93..e1eba21f06 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -169,8 +169,15 @@ docker-image-debian-ppc64el-user-cross: docker-binfmt-image-debian-ppc64el-user
DOCKER_USER_IMAGES += debian-ppc64el-user-cross
DOCKER_PARTIAL_IMAGES += debian-ppc64el-cross
+# No s390x cross-compilers in non-x86 Debian images, so use linux-user
+docker-binfmt-image-debian-s390x-user: DEB_ARCH = s390x
+docker-binfmt-image-debian-s390x-user: DEB_TYPE = stretch
+docker-binfmt-image-debian-s390x-user: EXECUTABLE = ${BUILD_DIR}/s390x-linux-user/qemu-s390x
+docker-image-debian-s390x-user-cross: docker-binfmt-image-debian-s390x-user
+DOCKER_USER_IMAGES += debian-s390x-user-cross
+DOCKER_PARTIAL_IMAGES += debian-s390x-cross
else
-DOCKER_PARTIAL_IMAGES += debian-mips64el-user-cross debian-ppc64el-user-cross
+DOCKER_PARTIAL_IMAGES += debian-mips64el-user-cross debian-ppc64el-user-cross debian-s390x-user-cross
endif
diff --git a/tests/docker/dockerfiles/debian-s390x-user-cross.docker b/tests/docker/dockerfiles/debian-s390x-user-cross.docker
new file mode 100644
index 0000000000..396763f10e
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-s390x-user-cross.docker
@@ -0,0 +1,16 @@
+#
+# Docker s390x user cross-compiler target
+#
+# This is a binfmt image for when the s390x cross compilers are not
+# available in the host architecture archives. This image is based of
+# the debootstrapped qemu:debian-s390x-user but doesn't need any
+# extra magic once it is setup.
+#
+
+FROM qemu:debian-s390x-user
+
+MAINTAINER Alex Bennee <alex.bennee@linaro.org>
+
+RUN echo man-db man-db/auto-update boolean false | debconf-set-selections
+RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get build-dep -yy qemu
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 08/10] docker: disable additional non-x86 images
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
` (6 preceding siblings ...)
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 07/10] docker: fall-back to binfmt_misc debian-s390x-user-cross " Alex Bennée
@ 2018-07-18 10:05 ` Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 09/10] tests: tcg skip docker images we can't build Alex Bennée
` (2 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:05 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
- travis - doesn't support non-x8
- centos7 - only AltArch, next release should have support
- debian-8 based - docker hub issue?
- ubuntu - image a bit old, maybe update to 18.04
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index e1eba21f06..2226abd092 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -148,9 +148,16 @@ DOCKER_PARTIAL_IMAGES += debian-powerpc-user-cross
ifneq ($(ARCH),x86_64)
+# Don't include non-multiarch images
+DOCKER_PARTIAL_IMAGES += travis centos7
+
ifeq ($(ARCH),aarch64)
# no point having a cross compiler when it is the native one
DOCKER_PARTIAL_IMAGES += debian-arm64-cross
+# hub.docker.com's older Debian images are not multiarch
+DOCKER_PARTIAL_IMAGES += debian-win32-cross debian-win64-cross
+# older Ubuntu's need fixing up as well
+DOCKER_PARTIAL_IMAGES += ubuntu
endif
# No MIPS cross-compilers in non-x86 Debian images, so use linux-user
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 09/10] tests: tcg skip docker images we can't build
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
` (7 preceding siblings ...)
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 08/10] docker: disable additional non-x86 images Alex Bennée
@ 2018-07-18 10:05 ` Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 10/10] tests/tcg: debian-mips64el-user-cross fallback Alex Bennée
2018-07-27 8:26 ` [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Fam Zheng
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:05 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée
We have a mechansim (DOCKER_PARTIAL_IMAGES) for marking stuff that is
incomplete or won't build. We can use this to avoid attempting to
build them at the top of a check-tcg build. We need to move the
inclusion of docker/Makefile.include to ensure everything is resolved
by this point.
We also have to check this in tests/tcg/Makefile.include to cover the
case where users are building directly out of the target tree.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
Makefile | 4 +++-
tests/tcg/Makefile.include | 3 +++
tests/tcg/Makefile.probe | 2 +-
3 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index 2da686be33..6bb8035557 100644
--- a/Makefile
+++ b/Makefile
@@ -434,6 +434,9 @@ dummy := $(call unnest-vars,, \
audio-obj-m \
trace-obj-y)
+# we need to know what Docker images can be built before we can
+# decide on all the tests that can be built.
+include $(SRC_PATH)/tests/docker/Makefile.include
include $(SRC_PATH)/tests/Makefile.include
all: $(DOCS) $(TOOLS) $(HELPERS-y) recurse-all modules
@@ -1069,7 +1072,6 @@ endif
# Dependencies in Makefile.objs files come from our recursive subdir rules
-include $(wildcard *.d tests/*.d)
-include $(SRC_PATH)/tests/docker/Makefile.include
include $(SRC_PATH)/tests/vm/Makefile.include
.PHONY: help
diff --git a/tests/tcg/Makefile.include b/tests/tcg/Makefile.include
index 57470b2a2c..e5f0298375 100644
--- a/tests/tcg/Makefile.include
+++ b/tests/tcg/Makefile.include
@@ -41,6 +41,8 @@ ifneq ($(DOCKER_IMAGE),)
# We also need the Docker make rules to depend on
include $(SRC_PATH)/tests/docker/Makefile.include
+ifneq ($(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGE)),)
+
DOCKER_COMPILE_CMD="$(DOCKER_SCRIPT) cc --user $(shell id -u) \
--cc $(DOCKER_CROSS_COMPILER) \
-i qemu:$(DOCKER_IMAGE) \
@@ -58,6 +60,7 @@ docker-build-guest-tests: $(DOCKER_PREREQ)
GUEST_BUILD=docker-build-guest-tests
+endif
endif
endif
diff --git a/tests/tcg/Makefile.probe b/tests/tcg/Makefile.probe
index 15c0412657..22c77d6b88 100644
--- a/tests/tcg/Makefile.probe
+++ b/tests/tcg/Makefile.probe
@@ -20,7 +20,7 @@ DOCKER_PREREQ:=
-include $(SRC_PATH)/tests/tcg/$(TARGET_NAME)/Makefile.include
ifndef CROSS_CC_GUEST
-ifneq ($(DOCKER_IMAGE),)
+ifneq ($(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGE)),)
DOCKER_PREREQ:=docker-image-$(DOCKER_IMAGE)
endif
endif
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH RFC 10/10] tests/tcg: debian-mips64el-user-cross fallback
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
` (8 preceding siblings ...)
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 09/10] tests: tcg skip docker images we can't build Alex Bennée
@ 2018-07-18 10:05 ` Alex Bennée
2018-07-27 8:26 ` [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Fam Zheng
10 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-18 10:05 UTC (permalink / raw)
To: cota, famz, berrange, f4bug, richard.henderson, balrogg, aurelien, agraf
Cc: qemu-devel, Alex Bennée, Aleksandar Markovic
When we can't use the debian-mips64el-cross to build our tests we can
use the user-cross fallback.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/tcg/mips/Makefile.include | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/tests/tcg/mips/Makefile.include b/tests/tcg/mips/Makefile.include
index 4a14fc078d..cc82e586fd 100644
--- a/tests/tcg/mips/Makefile.include
+++ b/tests/tcg/mips/Makefile.include
@@ -6,8 +6,13 @@
#
ifeq ($(TARGET_NAME),mips64el)
+ifneq ($(ARCH),x86_64)
+DOCKER_IMAGE=debian-mips64el-user-cross
+DOCKER_CROSS_COMPILER=gcc
+else
DOCKER_IMAGE=debian-mips64el-cross
DOCKER_CROSS_COMPILER=mips64el-linux-gnuabi64-gcc
+endif
else ifeq ($(TARGET_NAME),mips64)
DOCKER_IMAGE=debian-mips64-cross
DOCKER_CROSS_COMPILER=mips64-linux-gnuabi64-gcc
--
2.17.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH RFC 01/10] docker: rename docker-amd64 to docker-host
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 01/10] docker: rename docker-amd64 to docker-host Alex Bennée
@ 2018-07-27 7:56 ` Fam Zheng
2018-07-30 8:22 ` Alex Bennée
0 siblings, 1 reply; 15+ messages in thread
From: Fam Zheng @ 2018-07-27 7:56 UTC (permalink / raw)
To: Alex Bennée
Cc: cota, berrange, f4bug, richard.henderson, balrogg, aurelien,
agraf, qemu-devel
On Wed, 07/18 11:04, Alex Bennée wrote:
> When building on non-x86 systems the base system will be correct so if
> we avoid too many x86'isms in the install we can still use the image.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .../dockerfiles/{debian-amd64.docker => debian-host.docker} | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
> rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host.docker} (91%)
>
> diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-host.docker
> similarity index 91%
> rename from tests/docker/dockerfiles/debian-amd64.docker
> rename to tests/docker/dockerfiles/debian-host.docker
> index eb13f06ed1..3605cc4658 100644
> --- a/tests/docker/dockerfiles/debian-amd64.docker
> +++ b/tests/docker/dockerfiles/debian-host.docker
> @@ -30,9 +30,9 @@ RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --with-glx --disabl
> # netmap
> RUN DEBIAN_FRONTEND=noninteractive eatmydata \
> apt-get install -y --no-install-recommends \
> - linux-headers-amd64
> + linux-headers-$(dpkg --print-architecture)
> RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
> -RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
> +RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*) && make install
Curious: why isn't this "ls -d linux-headers-*-$(dpkg --print-architecture)", like above?
> ENV QEMU_CONFIGURE_OPTS --enable-netmap
>
> # gcrypt
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
` (9 preceding siblings ...)
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 10/10] tests/tcg: debian-mips64el-user-cross fallback Alex Bennée
@ 2018-07-27 8:26 ` Fam Zheng
2018-07-30 8:21 ` Alex Bennée
10 siblings, 1 reply; 15+ messages in thread
From: Fam Zheng @ 2018-07-27 8:26 UTC (permalink / raw)
To: Alex Bennée
Cc: cota, berrange, f4bug, richard.henderson, balrogg, aurelien,
agraf, qemu-devel
On Wed, 07/18 11:04, Alex Bennée wrote:
> Hi,
>
> Our existing support for docker is fairly x86 centric. While docker
> itself has support for multiple architectures not all architectures
> are equal. For example Debian only packages the widest range of
> cross-compilers in it's x86 images (although for example armhf is
> available on aarch64 based images).
>
> As the binfmt support is getting more solid we always have that fall
> back option and I've converted a number of the guests that way.
> Unfortunately I've overloaded the meaning of DOCKER_PARTIAL_IMAGES but
> I was loathed to throw in yet another magic variable without some
> discussion first. I'm not sure if my current path will end up with a
> maze of twisty if/else statements all subtly alike.
Just my two cents. So far it looks reasonable. Though I guess we could give
DOCKER_PARTIAL_IMAGES a better name or at least better documentation.
Would a per-arch variable list look cleaner? Like
DOCKER_IMAGES_GENERIC = debian
DOCKER_IMAGES_X86_64 = ubuntu fedora centos7
DOCKER_PARTIAL_IMAGES_X86_64 = ...
DOCKER_IMAGES_AARCH64 = ...
DOCKER_PARTIAL_IMAGES_AARCH64 = ...
Then the actual host arch can be used to expand rules:
docker-all-images: $(DOCKER_IMAGES_GENERIC)
docker-all-images: $(DOCKER_IMAGES_$(shell uname -m | tr '[a-z]' '[A-Z]))
Fam
>
> Thoughts?
>
> Alex Bennée (10):
> docker: rename docker-amd64 to docker-host
> docker: change docker-image to docker-all-images target
> docker: add a placeholder for handling non-x86 hosts
> docker: don't include docker-arm64-cross on aarch64 hosts
> docker: fall-back to binfmt_misc debian-mips64el-user-cross on non-x86
> docker: fall-back to binfmt_misc debian-ppc64el-user-cross on non-x86
> docker: fall-back to binfmt_misc debian-s390x-user-cross on non-x86
> docker: disable additional non-x86 images
> tests: tcg skip docker images we can't build
> tests/tcg: debian-mips64el-user-cross fallback
>
> Makefile | 4 +-
> tests/docker/Makefile.include | 58 +++++++++++++++++--
> ...debian-amd64.docker => debian-host.docker} | 4 +-
> .../debian-mips64el-user-cross.docker | 16 +++++
> .../debian-ppc64el-user-cross.docker | 16 +++++
> .../debian-s390x-user-cross.docker | 16 +++++
> tests/tcg/Makefile.include | 3 +
> tests/tcg/Makefile.probe | 2 +-
> tests/tcg/mips/Makefile.include | 5 ++
> 9 files changed, 116 insertions(+), 8 deletions(-)
> rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host.docker} (91%)
> create mode 100644 tests/docker/dockerfiles/debian-mips64el-user-cross.docker
> create mode 100644 tests/docker/dockerfiles/debian-ppc64el-user-cross.docker
> create mode 100644 tests/docker/dockerfiles/debian-s390x-user-cross.docker
>
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts
2018-07-27 8:26 ` [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Fam Zheng
@ 2018-07-30 8:21 ` Alex Bennée
0 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-30 8:21 UTC (permalink / raw)
To: Fam Zheng
Cc: cota, berrange, f4bug, richard.henderson, balrogg, aurelien,
agraf, qemu-devel
Fam Zheng <famz@redhat.com> writes:
> On Wed, 07/18 11:04, Alex Bennée wrote:
>> Hi,
>>
>> Our existing support for docker is fairly x86 centric. While docker
>> itself has support for multiple architectures not all architectures
>> are equal. For example Debian only packages the widest range of
>> cross-compilers in it's x86 images (although for example armhf is
>> available on aarch64 based images).
>>
>> As the binfmt support is getting more solid we always have that fall
>> back option and I've converted a number of the guests that way.
>> Unfortunately I've overloaded the meaning of DOCKER_PARTIAL_IMAGES but
>> I was loathed to throw in yet another magic variable without some
>> discussion first. I'm not sure if my current path will end up with a
>> maze of twisty if/else statements all subtly alike.
>
> Just my two cents. So far it looks reasonable. Though I guess we could give
> DOCKER_PARTIAL_IMAGES a better name or at least better documentation.
>
> Would a per-arch variable list look cleaner? Like
>
> DOCKER_IMAGES_GENERIC = debian
> DOCKER_IMAGES_X86_64 = ubuntu fedora centos7
> DOCKER_PARTIAL_IMAGES_X86_64 = ...
> DOCKER_IMAGES_AARCH64 = ...
> DOCKER_PARTIAL_IMAGES_AARCH64 = ...
>
> Then the actual host arch can be used to expand rules:
>
> docker-all-images: $(DOCKER_IMAGES_GENERIC)
> docker-all-images: $(DOCKER_IMAGES_$(shell uname -m | tr '[a-z]'
> '[A-Z]))
That might work better... I'll give that a go on v2
>
> Fam
>
>>
>> Thoughts?
>>
>> Alex Bennée (10):
>> docker: rename docker-amd64 to docker-host
>> docker: change docker-image to docker-all-images target
>> docker: add a placeholder for handling non-x86 hosts
>> docker: don't include docker-arm64-cross on aarch64 hosts
>> docker: fall-back to binfmt_misc debian-mips64el-user-cross on non-x86
>> docker: fall-back to binfmt_misc debian-ppc64el-user-cross on non-x86
>> docker: fall-back to binfmt_misc debian-s390x-user-cross on non-x86
>> docker: disable additional non-x86 images
>> tests: tcg skip docker images we can't build
>> tests/tcg: debian-mips64el-user-cross fallback
>>
>> Makefile | 4 +-
>> tests/docker/Makefile.include | 58 +++++++++++++++++--
>> ...debian-amd64.docker => debian-host.docker} | 4 +-
>> .../debian-mips64el-user-cross.docker | 16 +++++
>> .../debian-ppc64el-user-cross.docker | 16 +++++
>> .../debian-s390x-user-cross.docker | 16 +++++
>> tests/tcg/Makefile.include | 3 +
>> tests/tcg/Makefile.probe | 2 +-
>> tests/tcg/mips/Makefile.include | 5 ++
>> 9 files changed, 116 insertions(+), 8 deletions(-)
>> rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host.docker} (91%)
>> create mode 100644 tests/docker/dockerfiles/debian-mips64el-user-cross.docker
>> create mode 100644 tests/docker/dockerfiles/debian-ppc64el-user-cross.docker
>> create mode 100644 tests/docker/dockerfiles/debian-s390x-user-cross.docker
>>
>> --
>> 2.17.1
>>
--
Alex Bennée
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH RFC 01/10] docker: rename docker-amd64 to docker-host
2018-07-27 7:56 ` Fam Zheng
@ 2018-07-30 8:22 ` Alex Bennée
0 siblings, 0 replies; 15+ messages in thread
From: Alex Bennée @ 2018-07-30 8:22 UTC (permalink / raw)
To: Fam Zheng
Cc: cota, berrange, f4bug, richard.henderson, balrogg, aurelien,
agraf, qemu-devel
Fam Zheng <famz@redhat.com> writes:
> On Wed, 07/18 11:04, Alex Bennée wrote:
>> When building on non-x86 systems the base system will be correct so if
>> we avoid too many x86'isms in the install we can still use the image.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>> .../dockerfiles/{debian-amd64.docker => debian-host.docker} | 4 ++--
>> 1 file changed, 2 insertions(+), 2 deletions(-)
>> rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host.docker} (91%)
>>
>> diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-host.docker
>> similarity index 91%
>> rename from tests/docker/dockerfiles/debian-amd64.docker
>> rename to tests/docker/dockerfiles/debian-host.docker
>> index eb13f06ed1..3605cc4658 100644
>> --- a/tests/docker/dockerfiles/debian-amd64.docker
>> +++ b/tests/docker/dockerfiles/debian-host.docker
>> @@ -30,9 +30,9 @@ RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --with-glx --disabl
>> # netmap
>> RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>> apt-get install -y --no-install-recommends \
>> - linux-headers-amd64
>> + linux-headers-$(dpkg --print-architecture)
>> RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
>> -RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
>> +RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*) && make install
>
> Curious: why isn't this "ls -d linux-headers-*-$(dpkg
> --print-architecture)", like above?
Well there are only one set of headers, but you are right it would be
cleaner to be explicit.
>
>> ENV QEMU_CONFIGURE_OPTS --enable-netmap
>>
>> # gcrypt
>> --
>> 2.17.1
>>
--
Alex Bennée
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2018-07-30 8:22 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-18 10:04 [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 01/10] docker: rename docker-amd64 to docker-host Alex Bennée
2018-07-27 7:56 ` Fam Zheng
2018-07-30 8:22 ` Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 02/10] docker: change docker-image to docker-all-images target Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 03/10] docker: add a placeholder for handling non-x86 hosts Alex Bennée
2018-07-18 10:04 ` [Qemu-devel] [PATCH RFC 04/10] docker: don't include docker-arm64-cross on aarch64 hosts Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 05/10] docker: fall-back to binfmt_misc debian-mips64el-user-cross on non-x86 Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 06/10] docker: fall-back to binfmt_misc debian-ppc64el-user-cross " Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 07/10] docker: fall-back to binfmt_misc debian-s390x-user-cross " Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 08/10] docker: disable additional non-x86 images Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 09/10] tests: tcg skip docker images we can't build Alex Bennée
2018-07-18 10:05 ` [Qemu-devel] [PATCH RFC 10/10] tests/tcg: debian-mips64el-user-cross fallback Alex Bennée
2018-07-27 8:26 ` [Qemu-devel] [PATCH RFC 00/10] docker on non-x86 hosts Fam Zheng
2018-07-30 8:21 ` Alex Bennée
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.