[Qemu-devel] [RFC/POC PATCH 0/4] Building TCG tests with emdebian cross compilers

[Qemu-devel] [RFC/POC PATCH 0/4] Building TCG tests with emdebian cross compilers

[Alex Bennée]

Hi Pranith,

Here is a proof-of-concept series for you to consider rolling into the TCG tests
cleanup. It uses the existing docker make machinery to build a Debian
image which has arm, arm64 and ppc64el cross compilers in it. Now if
you run:

  make arm-tcg-tests

It will do the requisite build of the docker image and then use that
to build the TCG tests in the appropriate build directory.

These apply on top of your existing series. There is also a quick hack
to disable the running of the tests by default. I think we need two
stages, maybe a build-FOO-tcg-tests and run-FOO-tcg-tests.

What do you think?

Alex Bennée (4):
  tests/docker: add basic user mapping support
  new tests/docker/dockerfiles/debian-multiarch-cross.docker
  tests/tcg: don't run tests by default
  tests/tcg/Makefile: use docker target for arm-tcg-tests

 tests/docker/docker.py                             | 19 +++++++++++
 tests/docker/dockerfiles/debian-bootstrap.docker   |  3 ++
 .../dockerfiles/debian-multiarch-cross.docker      | 39 ++++++++++++++++++++++
 tests/tcg/Makefile.include                         | 20 +++++++++--
 tests/tcg/arm/Makefile                             |  2 +-
 tests/tcg/misc/Makefile                            |  4 +--
 6 files changed, 81 insertions(+), 6 deletions(-)
 create mode 100644 tests/docker/dockerfiles/debian-multiarch-cross.docker

-- 
2.11.0

[Qemu-devel] [RFC/POC PATCH 1/4] tests/docker: add basic user mapping support

[Alex Bennée]

Currently all docker builds are done by exporting a tarball to the
docker container and running the build as the containers root user.
Other use cases are possible however and it is possible to map a part
of users file-system to the container. This is useful for example for
doing cross-builds of arbitrary source trees. For this to work
smoothly the container needs to have a user created that maps cleanly
to the host system.

This adds a -u option to the docker script so that:

  DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \
    -u --include-executable=arm-linux-user/qemu-arm \
    debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker

Will build a container that can then be run like:

  docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \
    --user=alex:alex -w /src/ debian:armhf \
    sh -c "make clean && ./configure -s && make"

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py                           | 19 +++++++++++++++++++
 tests/docker/dockerfiles/debian-bootstrap.docker |  3 +++
 2 files changed, 22 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 37d83199e7..59baac6bae 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -12,6 +12,7 @@
 # the top-level directory.
 
 import os
+import stat
 import sys
 import subprocess
 import json
@@ -25,6 +26,7 @@ import signal
 from tarfile import TarFile, TarInfo
 from StringIO import StringIO
 from shutil import copy, rmtree
+from pwd import getpwuid
 
 
 DEVNULL = open(os.devnull, 'wb')
@@ -225,6 +227,8 @@ class BuildCommand(SubCommand):
                             help="""Specify a binary that will be copied to the
                             container together with all its dependent
                             libraries""")
+        parser.add_argument("--user", "-u", action="store_true",
+                            help="Add the current user to images passwd")
         parser.add_argument("tag",
                             help="Image Tag")
         parser.add_argument("dockerfile",
@@ -260,6 +264,21 @@ class BuildCommand(SubCommand):
                 _copy_binary_with_libs(args.include_executable,
                                        docker_dir)
 
+            if args.user:
+                uid = os.getuid()
+                uname = getpwuid(uid).pw_name
+                scriptlet = docker_dir+"/setup_user.sh"
+
+                # write scriptlet
+                setup = open(scriptlet, "w")
+                setup.write("#!/bin/sh\n")
+                setup.write("useradd -u %d -U %s" % (uid, uname))
+                setup.close()
+
+                st = os.stat(scriptlet)
+                os.chmod(scriptlet,
+                         st.st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
+
             dkr.build_image(tag, docker_dir, dockerfile,
                             quiet=args.quiet, argv=argv)
 
diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
index 3a9125e497..127782eedf 100644
--- a/tests/docker/dockerfiles/debian-bootstrap.docker
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -14,6 +14,9 @@ RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstra
 # Run stage 2
 RUN /debootstrap/debootstrap --second-stage
 
+# Do we want to tweak the user?
+RUN if test -e /setup_user.sh; then /setup_user.sh; fi
+
 # At this point we can install additional packages if we want
 # Duplicate deb line as deb-src
 RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
-- 
2.11.0

[Qemu-devel] [RFC/POC PATCH 2/4] new tests/docker/dockerfiles/debian-multiarch-cross.docker

[Alex Bennée]

This provides a basic Debian install with access to the emdebian cross
compilers for armhf, arm64 and ppc64el. On its own it provides the
basic cross build essentials for building any of those architectures.
More complex cross-compilations would need to install library
dependencies on top of this image.

An example use:

  docker run --rm -it \
    -v /home/alex/lsrc/qemu/qemu.git/:/home/alex/lsrc/qemu/qemu.git/ \
    --user=alex:alex \
    -w /home/alex/lsrc/qemu/qemu.git/arm-linux-user/tests \
    -e CROSS_CC="arm-linux-gnueabihf-gcc" \
    debian:cross make -f ../../tests/tcg/arm/Makefile

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .../dockerfiles/debian-multiarch-cross.docker      | 39 ++++++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-multiarch-cross.docker

diff --git a/tests/docker/dockerfiles/debian-multiarch-cross.docker b/tests/docker/dockerfiles/debian-multiarch-cross.docker
new file mode 100644
index 0000000000..dba66374c0
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-multiarch-cross.docker
@@ -0,0 +1,39 @@
+#
+# Docker multiarch cross-compiler target
+#
+# This docker target is builds on Debian and Emdebian's cross compiler targets
+# to build distro with a selection of cross compilers for building test binaries.
+#
+# This does not include the build dependancies for QEMU as there are
+# so many there are unresolvable clashes.
+#
+FROM debian:stable-slim
+
+# Do we want to tweak the user?
+ADD . /
+RUN if test -e /setup_user.sh; then /setup_user.sh; fi
+
+# Setup some basic tools we need
+RUN apt update
+RUN apt install -yy curl
+
+# Setup Emdebian
+RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
+RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
+
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
+
+# Add the foriegn architectures we want
+RUN dpkg --add-architecture armhf
+RUN dpkg --add-architecture arm64
+RUN dpkg --add-architecture ppc64el
+
+# Final update
+RUN apt update
+
+# Install the crossbuild essential packages
+RUN apt install -yy crossbuild-essential-arm64 crossbuild-essential-armhf crossbuild-essential-ppc64el
+
+# To install greater dependencies you do a
+# RUN apt-get build-dep -yy -a armhf qemu
-- 
2.11.0

[Qemu-devel] [RFC/POC PATCH 3/4] tests/tcg: don't run tests by default

[Alex Bennée]

If we are going to be doing the build in a different place we might
not have binfmt setup and certainly won't have a qemu built.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/arm/Makefile  | 2 +-
 tests/tcg/misc/Makefile | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/tcg/arm/Makefile b/tests/tcg/arm/Makefile
index a7c8953bf4..c374f719c8 100644
--- a/tests/tcg/arm/Makefile
+++ b/tests/tcg/arm/Makefile
@@ -24,7 +24,7 @@ LDFLAGS=-nostdlib
 
 TESTS=hello-arm arm-iwmmxt
 
-all: $(patsubst %,run-%,$(TESTS))
+all: $(TESTS)
 test: all
 
 # rules to run tests
diff --git a/tests/tcg/misc/Makefile b/tests/tcg/misc/Makefile
index 0aa6f2ea21..6070107898 100644
--- a/tests/tcg/misc/Makefile
+++ b/tests/tcg/misc/Makefile
@@ -29,7 +29,7 @@ TESTS=linux-test \
       sha1 \
       test-mmap
 
-all: $(patsubst %,run-%,$(TESTS))
+all: $(TESTS)
 test: all
 
 # rules to run tests
@@ -64,7 +64,7 @@ testthread: testthread.c
 
 # generic Linux and CPU test
 linux-test: linux-test.c
-	$(CROSS_CC) $(CFLAGS) $(LDFLAGS) -o $@ $< -lm
+	$(CROSS_CC) ${QEMU_INCLUDES} $(CFLAGS) $(LDFLAGS) -o $@ $< -lm
 
 test-mmap: test-mmap.c
 	$(CROSS_CC) $(CFLAGS) -Wall -O2 $(LDFLAGS) -o $@ $< -static
-- 
2.11.0

[Qemu-devel] [RFC/POC PATCH 4/4] tests/tcg/Makefile: use docker target for arm-tcg-tests

[Alex Bennée]

This means we can type:

  make arm-tcg-tests

And through the power of docker and Debian's emdebian cross tools we
build the misc and arm TCG targets.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/Makefile.include | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/tests/tcg/Makefile.include b/tests/tcg/Makefile.include
index 3eff12527c..6d8d666203 100644
--- a/tests/tcg/Makefile.include
+++ b/tests/tcg/Makefile.include
@@ -8,6 +8,7 @@ BUILD_DIR?=$(CURDIR)
 include $(BUILD_DIR)/config-host.mak	# brings in SRC_PATH
 
 UNAME_M := $(shell uname -m)
+BUILD_UID = $(shell id -u)
 
 tests-tcg: prepare $(UNAME_M)
 
@@ -22,9 +23,22 @@ i386:
 	cd $(BUILD_DIR)/i386-linux-user/tests/ && \
 	make -f $(SRC_PATH)/tests/tcg/i386/Makefile
 
-arm:
-	cd $(BUILD_DIR)/arm-linux-user/tests/ && \
-	make -f $(SRC_PATH)/tests/tcg/arm/Makefile
+arm-tcg-tests: docker-image-debian-multiarch-cross
+	mkdir -p $(BUILD_DIR)/arm-linux-user/tests
+	docker run --rm -it -v $(SRC_PATH):$(SRC_PATH) \
+		-v $(BUILD_DIR):$(BUILD_DIR) \
+		--user $(BUILD_ID):$(BUILD_ID) \
+		-w $(BUILD_DIR)/arm-linux-user/tests \
+		-e CROSS_CC="arm-linux-gnueabihf-gcc" \
+		qemu:debian-multiarch-cross \
+		make -f ../../tests/tcg/misc/Makefile
+	docker run --rm -it -v $(SRC_PATH):$(SRC_PATH) \
+		-v $(BUILD_DIR):$(BUILD_DIR) \
+		--user $(BUILD_ID):$(BUILD_ID) \
+		-w $(BUILD_DIR)/arm-linux-user/tests \
+		-e CROSS_CC="arm-linux-gnueabihf-gcc" \
+		qemu:debian-multiarch-cross \
+		make -f ../../tests/tcg/arm/Makefile
 
 aarch64:
 	cd $(BUILD_DIR)/aarch64-linux-user/tests/ && \
-- 
2.11.0

Re: [Qemu-devel] [RFC/POC PATCH 0/4] Building TCG tests with emdebian cross compilers

[Marc-André Lureau]

Hi

----- Original Message -----
> Hi Pranith,
> 
> Here is a proof-of-concept series for you to consider rolling into the TCG
> tests
> cleanup. It uses the existing docker make machinery to build a Debian
> image which has arm, arm64 and ppc64el cross compilers in it. Now if
> you run:
> 
>   make arm-tcg-tests
> 
> It will do the requisite build of the docker image and then use that
> to build the TCG tests in the appropriate build directory.
> 
> These apply on top of your existing series. There is also a quick hack
> to disable the running of the tests by default. I think we need two
> stages, maybe a build-FOO-tcg-tests and run-FOO-tcg-tests.
> 
> What do you think?

I like the idea, as long as you can also run the tcg tests without docker. How many of the qemu archs debian cross tools support? last time I looked there was annoying limitations, but I can't remember the details.

crosstool-ng has perhaps more potential, especially if devs would share their config/samples (we could then have a cross-distro packaging with flatpack?)

> Alex Bennée (4):
>   tests/docker: add basic user mapping support
>   new tests/docker/dockerfiles/debian-multiarch-cross.docker
>   tests/tcg: don't run tests by default
>   tests/tcg/Makefile: use docker target for arm-tcg-tests
> 
>  tests/docker/docker.py                             | 19 +++++++++++
>  tests/docker/dockerfiles/debian-bootstrap.docker   |  3 ++
>  .../dockerfiles/debian-multiarch-cross.docker      | 39
>  ++++++++++++++++++++++
>  tests/tcg/Makefile.include                         | 20 +++++++++--
>  tests/tcg/arm/Makefile                             |  2 +-
>  tests/tcg/misc/Makefile                            |  4 +--
>  6 files changed, 81 insertions(+), 6 deletions(-)
>  create mode 100644 tests/docker/dockerfiles/debian-multiarch-cross.docker
> 
> --
> 2.11.0
> 
> 

Re: [Qemu-devel] [RFC/POC PATCH 0/4] Building TCG tests with emdebian cross compilers

[Alex Bennée]

Marc-André Lureau <mlureau@redhat.com> writes:

> Hi
>
> ----- Original Message -----
>> Hi Pranith,
>>
>> Here is a proof-of-concept series for you to consider rolling into the TCG
>> tests
>> cleanup. It uses the existing docker make machinery to build a Debian
>> image which has arm, arm64 and ppc64el cross compilers in it. Now if
>> you run:
>>
>>   make arm-tcg-tests
>>
>> It will do the requisite build of the docker image and then use that
>> to build the TCG tests in the appropriate build directory.
>>
>> These apply on top of your existing series. There is also a quick hack
>> to disable the running of the tests by default. I think we need two
>> stages, maybe a build-FOO-tcg-tests and run-FOO-tcg-tests.
>>
>> What do you think?
>
> I like the idea, as long as you can also run the tcg tests without
> docker.

Yes, the idea is the build deposits the test binaries in host systems
$QEMU_SRC/$arch-linux-user/tests (preferably statically linked) where
the test machinery can then pick them up and run them.

> How many of the qemu archs debian cross tools support? last
> time I looked there was annoying limitations, but I can't remember the
> details.

I think currently the emdebian compilers are arm, ppc and mips.

> crosstool-ng has perhaps more potential, especially if devs would
> share their config/samples (we could then have a cross-distro
> packaging with flatpack?)

Or simply a plain container which can hold pre-builts for the rarer
compilers?

There are a couple of problems I'm trying to solve.

The first is having a standard way to cross-build stuff so we can least
have a reference mechanism that is available to all devs. While we all
probably have random compilers sitting on our systems it would be nice
to have one method we could give new devs to say "this will build you
some foreign test binaries".

The second problem is as soon as you move away from simple libc
dependencies you start having to deal with development header
pre-requisites and that can get messy on your main system even with
multi-lib setups. A docker container can be created with all the
$FOO-arch dependencies for QEMU quite easily without breaking your main
machine. You simply need to run a "apt-get build-dep -a $ARCH qemu"
before the configure/build step.

It would be nice to come up with a solution to the proliferation of
containers though. The debootstrap container build script is generic but
currently we don't expose all the architecture variants in the make
system. We leave the user to build the containers they need and manually
call them for the build.

>
>> Alex Bennée (4):
>>   tests/docker: add basic user mapping support
>>   new tests/docker/dockerfiles/debian-multiarch-cross.docker
>>   tests/tcg: don't run tests by default
>>   tests/tcg/Makefile: use docker target for arm-tcg-tests
>>
>>  tests/docker/docker.py                             | 19 +++++++++++
>>  tests/docker/dockerfiles/debian-bootstrap.docker   |  3 ++
>>  .../dockerfiles/debian-multiarch-cross.docker      | 39
>>  ++++++++++++++++++++++
>>  tests/tcg/Makefile.include                         | 20 +++++++++--
>>  tests/tcg/arm/Makefile                             |  2 +-
>>  tests/tcg/misc/Makefile                            |  4 +--
>>  6 files changed, 81 insertions(+), 6 deletions(-)
>>  create mode 100644 tests/docker/dockerfiles/debian-multiarch-cross.docker
>>
>> --
>> 2.11.0
>>
>>


--
Alex Bennée