[Qemu-devel] [PATCH v6 0/8] docker: Support building qemu-user powered docker test images

[Qemu-devel] [PATCH v6 0/8] docker: Support building qemu-user powered docker test images

[Fam Zheng]

v6: A few tweaks to make this more mergable. Including:
    - Add and catch magic pre script exit code "3" to allow skipping image.
    - Tweak debian-bootstrap.pre to return '3' when appropriate (It should
      still return 1 for debootstrap command failure).
    - Fix "realpath" to "readlink -e" to be compatible with centos6.
    - Add the last patch to skip test if the image is skipped by .pre.

Alex, if this looks good to you, I can send a pull req today.

Alex Bennée (5):
  tests/docker/docker.py: docker_dir outside build
  tests/docker/docker.py: support --include-executable
  tests/docker/docker.py: check and run .pre script
  tests/docker/dockerfiles: new debian-bootstrap.docker
  tests/docker/docker.py: add update operation

Fam Zheng (3):
  docker: More sensible run script
  docker: Fix exit code if $CMD failed
  docker: Don't start a container that doesn't exist

 tests/docker/Makefile.include                    |   9 +-
 tests/docker/docker.py                           | 150 +++++++++++++++++++++--
 tests/docker/dockerfiles/debian-bootstrap.docker |  21 ++++
 tests/docker/dockerfiles/debian-bootstrap.pre    |  87 +++++++++++++
 tests/docker/run                                 |  16 ++-
 5 files changed, 267 insertions(+), 16 deletions(-)
 create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
 create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre

-- 
2.7.4

[Qemu-devel] [PATCH v6 1/8] tests/docker/docker.py: docker_dir outside build

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

Instead of letting the build_image create the temporary working dir we
move the creation to the build command. This is preparation for the
later patches where additional files can be added to the build context
before the build step is run.

We also ensure we remove the build context after we are done (mkdtemp
doesn't do this automatically for you).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Message-id: 1468335639-24582-2-git-send-email-alex.bennee@linaro.org
---
 tests/docker/docker.py | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 0151362..ae40bb3 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,7 +20,7 @@ import atexit
 import uuid
 import argparse
 import tempfile
-from shutil import copy
+from shutil import copy, rmtree
 
 def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
@@ -87,20 +87,20 @@ class Docker(object):
         labels = json.loads(resp)[0]["Config"].get("Labels", {})
         return labels.get("com.qemu.dockerfile-checksum", "")
 
-    def build_image(self, tag, dockerfile, df_path, quiet=True, argv=None):
+    def build_image(self, tag, docker_dir, dockerfile, quiet=True, argv=None):
         if argv == None:
             argv = []
-        tmp_dir = tempfile.mkdtemp(prefix="docker_build")
 
-        tmp_df = tempfile.NamedTemporaryFile(dir=tmp_dir, suffix=".docker")
+        tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker")
         tmp_df.write(dockerfile)
 
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
                      _text_checksum(dockerfile))
         tmp_df.flush()
+
         self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
-                 [tmp_dir],
+                 [docker_dir],
                  quiet=quiet)
 
     def image_matches_dockerfile(self, tag, dockerfile):
@@ -164,10 +164,15 @@ class BuildCommand(SubCommand):
         if dkr.image_matches_dockerfile(tag, dockerfile):
             if not args.quiet:
                 print "Image is up to date."
-            return 0
+        else:
+            # Create a docker context directory for the build
+            docker_dir = tempfile.mkdtemp(prefix="docker_build")
+
+            dkr.build_image(tag, docker_dir, dockerfile,
+                            quiet=args.quiet, argv=argv)
+
+            rmtree(docker_dir)
 
-        dkr.build_image(tag, dockerfile, args.dockerfile,
-                        quiet=args.quiet, argv=argv)
         return 0
 
 class CleanCommand(SubCommand):
-- 
2.7.4

[Qemu-devel] [PATCH v6 2/8] tests/docker/docker.py: support --include-executable

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

When passed the path to a binary we copy it and any linked libraries (if
it is dynamically linked) into the docker build context. These can then
be included by a dockerfile with the line:

  # Copy all of context into container
  ADD . /

This is mainly intended for setting up foreign architecture docker
images which use qemu-$arch to do cross-architecture linux-user
execution. It also relies on the host and guest file-system following
reasonable multi-arch layouts so the copied libraries don't clash with
the guest ones.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468335639-24582-3-git-send-email-alex.bennee@linaro.org
---
 tests/docker/docker.py | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index ae40bb3..96d906e 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,6 +20,7 @@ import atexit
 import uuid
 import argparse
 import tempfile
+import re
 from shutil import copy, rmtree
 
 def _text_checksum(text):
@@ -38,6 +39,54 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
+def _copy_with_mkdir(src, root_dir, sub_path):
+    """Copy src into root_dir, creating sub_path as needed."""
+    dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
+    try:
+        os.makedirs(dest_dir)
+    except OSError:
+        # we can safely ignore already created directories
+        pass
+
+    dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
+    copy(src, dest_file)
+
+
+def _get_so_libs(executable):
+    """Return a list of libraries associated with an executable.
+
+    The paths may be symbolic links which would need to be resolved to
+    ensure theright data is copied."""
+
+    libs = []
+    ldd_re = re.compile(r"(/.*/)(\S*)")
+    try:
+        ldd_output = subprocess.check_output(["ldd", executable])
+        for line in ldd_output.split("\n"):
+            search = ldd_re.search(line)
+            if search and len(search.groups()) == 2:
+                so_path = search.groups()[0]
+                so_lib = search.groups()[1]
+                libs.append("%s/%s" % (so_path, so_lib))
+    except subprocess.CalledProcessError:
+        print "%s had no associated libraries (static build?)" % (executable)
+
+    return libs
+
+def _copy_binary_with_libs(src, dest_dir):
+    """Copy a binary executable and all its dependant libraries.
+
+    This does rely on the host file-system being fairly multi-arch
+    aware so the file don't clash with the guests layout."""
+
+    _copy_with_mkdir(src, dest_dir, "/usr/bin")
+
+    libs = _get_so_libs(src)
+    if libs:
+        for l in libs:
+            so_path = os.path.dirname(l)
+            _copy_with_mkdir(l , dest_dir, so_path)
+
 class Docker(object):
     """ Running Docker commands """
     def __init__(self):
@@ -151,6 +200,10 @@ class BuildCommand(SubCommand):
     """ Build docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
     name = "build"
     def args(self, parser):
+        parser.add_argument("--include-executable", "-e",
+                            help="""Specify a binary that will be copied to the
+                            container together with all its dependent
+                            libraries""")
         parser.add_argument("tag",
                             help="Image Tag")
         parser.add_argument("dockerfile",
@@ -168,6 +221,11 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Do we include a extra binary?
+            if args.include_executable:
+                _copy_binary_with_libs(args.include_executable,
+                                       docker_dir)
+
             dkr.build_image(tag, docker_dir, dockerfile,
                             quiet=args.quiet, argv=argv)
 
-- 
2.7.4

[Qemu-devel] [PATCH v6 3/8] tests/docker/docker.py: check and run .pre script

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

The docker script will now search for an associated $dockerfile.pre
script which gets run in the same build context as the dockerfile will
be. This is to support pre-seeding the build context before running the
docker build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468335639-24582-4-git-send-email-alex.bennee@linaro.org
---
 tests/docker/docker.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 96d906e..b61d934 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -221,6 +221,18 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Is there a .pre file to run in the build context?
+            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
+            if os.path.exists(docker_pre):
+                rc = subprocess.call(os.path.realpath(docker_pre),
+                                     cwd=docker_dir)
+                if rc == 3:
+                    print "Skip"
+                    return 0
+                else:
+                    print "%s exited with code %d" % (docker_pre, rc)
+                    return 1
+
             # Do we include a extra binary?
             if args.include_executable:
                 _copy_binary_with_libs(args.include_executable,
-- 
2.7.4

[Qemu-devel] [PATCH v6 4/8] tests/docker/dockerfiles: new debian-bootstrap.docker

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

Together with the debian-bootstrap.pre script can now build an arbitrary
architecture of Debian using debootstrap. This allows debootstrap to set
up its first stage before the container is built.

To build a container you need a command line like:

  DEB_ARCH=armhf DEB_TYPE=testing \
    ./tests/docker/docker.py build \
    --include-executable=arm-linux-user/qemu-arm debian:armhf \
    ./tests/docker/dockerfiles/debian-bootstrap.docker

Although a number of non-debian systems package the debootstrap script
it is fairly portable in itself. Assuming we have some sort of fakeroot
implementation we can just clone the upstream repository and use the
script from there.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468335639-24582-5-git-send-email-alex.bennee@linaro.org
---
 tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++
 tests/docker/dockerfiles/debian-bootstrap.pre    | 87 ++++++++++++++++++++++++
 2 files changed, 108 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
 create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre

diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
new file mode 100644
index 0000000..3a9125e
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -0,0 +1,21 @@
+# Create Debian Bootstrap Image
+#
+# This is intended to be pre-poluated by:
+#  - a first stage debootstrap (see debian-bootstrap.pre)
+#  - a native qemu-$arch that binfmt_misc will run
+FROM scratch
+
+# Add everything from the context into the container
+ADD . /
+
+# Patch all mounts as docker already has stuff set up
+RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
+
+# Run stage 2
+RUN /debootstrap/debootstrap --second-stage
+
+# At this point we can install additional packages if we want
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
+RUN apt-get update
+RUN apt-get -y build-dep qemu
diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
new file mode 100755
index 0000000..55850be
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -0,0 +1,87 @@
+#!/bin/sh
+#
+# Simple wrapper for debootstrap, run in the docker build context
+#
+FAKEROOT=`which fakeroot 2> /dev/null`
+
+exit_and_skip()
+{
+    exit 3
+}
+
+#
+# fakeroot is needed to run the bootstrap stage
+#
+if [ -z $FAKEROOT ]; then
+    echo "Please install fakeroot to enable bootstraping"
+    exit_and_skip
+fi
+
+# We check in order for
+#
+#  - DEBOOTSTRAP_DIR pointing at a development checkout
+#  - PATH for the debootstrap script (installed)
+#
+# If neither option works then we checkout debootstrap from its
+# upstream SCM and run it from there.
+#
+
+if [ -z $DEBOOTSTRAP_DIR ]; then
+    DEBOOTSTRAP=`which debootstrap 2> /dev/null`
+    if [ -z $DEBOOTSTRAP ]; then
+        echo "No debootstrap installed, attempting to install from SCM"
+        DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
+        git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
+        export DEBOOTSTRAP_DIR=./debootstrap.git
+        DEBOOTSTRAP=./debootstrap.git/debootstrap
+    fi
+else
+    DEBOOTSTRAP=${DEBOOTSTRAP_DIR}/debootstrap
+    if [ ! -f $DEBOOTSTRAP ]; then
+        echo "Couldn't find script at ${DEBOOTSTRAP}"
+        exit_and_skip
+    fi
+fi
+
+#
+# Finally check to see if any qemu's are installed
+#
+BINFMT_DIR=/proc/sys/fs/binfmt_misc
+if [ ! -e $BINFMT_DIR ]; then
+   echo "binfmt_misc needs enabling for a QEMU bootstrap to work"
+   exit_and_skip
+else
+    # DEB_ARCH and QEMU arch names are not totally aligned
+    case "${DEB_ARCH}" in
+        amd64)
+            QEMU=qemu-i386
+            ;;
+        armel|armhf)
+            QEMU=qemu-arm
+            ;;
+        arm64)
+            QEMU=qemu-aarch64
+            ;;
+        powerpc)
+            QEMU=qemu-ppc
+            ;;
+        ppc64el)
+            QEMU=qemu-ppc64le
+            ;;
+        s390)
+            QEMU=qemu-s390x
+            ;;
+        *)
+            QEMU=qemu-${DEB_ARCH}
+        ;;
+    esac
+    if [ ! -e "${BINFMT_DIR}/$QEMU" ]; then
+        echo "No binfmt_misc rule to run $QEMU, can't bootstrap"
+        exit_and_skip
+    fi
+fi
+
+echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP}"
+
+${FAKEROOT} ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian || exit 1
+exit 0
-- 
2.7.4

[Qemu-devel] [PATCH v6 5/8] tests/docker/docker.py: add update operation

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

This adds a new operation to the docker script to allow updating of
binaries in an existing container. This is because it would be
inefficient to re-build the whole container just for an update to the
QEMU binary.

To update the executable run:

    ./tests/docker/docker.py update \
        debian:armhf ./arm-linux-user/qemu-arm

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468335639-24582-6-git-send-email-alex.bennee@linaro.org
---
 tests/docker/docker.py | 59 +++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index b61d934..fd162c0 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -21,6 +21,8 @@ import uuid
 import argparse
 import tempfile
 import re
+from tarfile import TarFile, TarInfo
+from StringIO import StringIO
 from shutil import copy, rmtree
 
 def _text_checksum(text):
@@ -94,9 +96,11 @@ class Docker(object):
         self._instances = []
         atexit.register(self._kill_instances)
 
-    def _do(self, cmd, quiet=True, **kwargs):
+    def _do(self, cmd, quiet=True, infile=None, **kwargs):
         if quiet:
             kwargs["stdout"] = subprocess.PIPE
+        if infile:
+            kwargs["stdin"] = infile
         return subprocess.call(self._command + cmd, **kwargs)
 
     def _do_kill_instances(self, only_known, only_active=True):
@@ -152,6 +156,11 @@ class Docker(object):
                  [docker_dir],
                  quiet=quiet)
 
+    def update_image(self, tag, tarball, quiet=True):
+        "Update a tagged image using "
+
+        self._do(["build", "-t", tag, "-"], quiet=quiet, infile=tarball)
+
     def image_matches_dockerfile(self, tag, dockerfile):
         try:
             checksum = self.get_image_dockerfile_checksum(tag)
@@ -245,6 +254,54 @@ class BuildCommand(SubCommand):
 
         return 0
 
+class UpdateCommand(SubCommand):
+    """ Update a docker image with new executables. Arguments: <tag> <executable>"""
+    name = "update"
+    def args(self, parser):
+        parser.add_argument("tag",
+                            help="Image Tag")
+        parser.add_argument("executable",
+                            help="Executable to copy")
+
+    def run(self, args, argv):
+        # Create a temporary tarball with our whole build context and
+        # dockerfile for the update
+        tmp = tempfile.NamedTemporaryFile(suffix="dckr.tar.gz")
+        tmp_tar = TarFile(fileobj=tmp, mode='w')
+
+        # Add the executable to the tarball
+        bn = os.path.basename(args.executable)
+        ff = "/usr/bin/%s" % bn
+        tmp_tar.add(args.executable, arcname=ff)
+
+        # Add any associated libraries
+        libs = _get_so_libs(args.executable)
+        if libs:
+            for l in libs:
+                tmp_tar.add(os.path.realpath(l), arcname=l)
+
+        # Create a Docker buildfile
+        df = StringIO()
+        df.write("FROM %s\n" % args.tag)
+        df.write("ADD . /\n")
+        df.seek(0)
+
+        df_tar = TarInfo(name="Dockerfile")
+        df_tar.size = len(df.buf)
+        tmp_tar.addfile(df_tar, fileobj=df)
+
+        tmp_tar.close()
+
+        # reset the file pointers
+        tmp.flush()
+        tmp.seek(0)
+
+        # Run the build with our tarball context
+        dkr = Docker()
+        dkr.update_image(args.tag, tmp, quiet=args.quiet)
+
+        return 0
+
 class CleanCommand(SubCommand):
     """Clean up docker instances"""
     name = "clean"
-- 
2.7.4

[Qemu-devel] [PATCH v6 6/8] docker: More sensible run script

[Fam Zheng]

It is very easy to figure out current directory and bash option from the
execution, so do less in the Makefile invocation command line, and
figure both options in the script.

This makes the next patch easier.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1468335639-24582-7-git-send-email-alex.bennee@linaro.org
---
 tests/docker/Makefile.include |  4 +---
 tests/docker/run              | 12 +++++++++---
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index f88c0a7..c5546ee 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -114,10 +114,8 @@ docker-run-%: docker-qemu-src
 				-e CCACHE_DIR=/var/tmp/ccache \
 				-v $$(realpath $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \
 				-v $(DOCKER_CCACHE_DIR):/var/tmp/ccache:z \
-				-w /var/tmp/qemu \
 				qemu:$(IMAGE) \
-				$(if $V,/bin/bash -x ,) \
-				./run \
+				/var/tmp/qemu/run \
 				$(CMD); \
 			, "  RUN $(CMD) in $(IMAGE)")))
 
diff --git a/tests/docker/run b/tests/docker/run
index ec3d119..ddfac05 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -11,6 +11,12 @@
 # or (at your option) any later version. See the COPYING file in
 # the top-level directory.
 
+if test -n "$V"; then
+    set -x
+fi
+
+BASE="$(dirname $(readlink -e $0))"
+
 # Prepare the environment
 . /etc/profile || true
 export PATH=/usr/lib/ccache:$PATH
@@ -24,10 +30,10 @@ export TEST_DIR=/tmp/qemu-test
 mkdir -p $TEST_DIR/{src,build,install}
 
 # Extract the source tarballs
-tar -C $TEST_DIR/src -xzf qemu.tgz
+tar -C $TEST_DIR/src -xzf $BASE/qemu.tgz
 for p in dtc pixman; do
-    if test -f $p.tgz; then
-        tar -C $TEST_DIR/src/$p -xzf $p.tgz
+    if test -f $BASE/$p.tgz; then
+        tar -C $TEST_DIR/src/$p -xzf $BASE/$p.tgz
         export FEATURES="$FEATURES $p"
     fi
 done
-- 
2.7.4

[Qemu-devel] [PATCH v6 7/8] docker: Fix exit code if $CMD failed

[Fam Zheng]

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1468335639-24582-8-git-send-email-alex.bennee@linaro.org
---
 tests/docker/run | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/docker/run b/tests/docker/run
index ddfac05..d85d49a 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -11,6 +11,8 @@
 # or (at your option) any later version. See the COPYING file in
 # the top-level directory.
 
+set -e
+
 if test -n "$V"; then
     set -x
 fi
@@ -61,4 +63,6 @@ elif test -n "$DEBUG"; then
     echo
     # Force error after shell exits
     $SHELL && exit 1
+else
+    exit 1
 fi
-- 
2.7.4

[Qemu-devel] [PATCH v6 8/8] docker: Don't start a container that doesn't exist

[Fam Zheng]

Image building targets are dependencies of test running targets, so when
a docker image doesn't exist, it means it's skipped (due to dependency
checks in pre script). Therefore, skip the test too.

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/Makefile.include | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c5546ee..2bb8a51 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -105,7 +105,9 @@ docker-run-%: docker-qemu-src
 	fi
 	$(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
 		$(call quiet-command,\
-			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
+			if sudo docker images --format={{.Repository}}:{{.Tag}} | \
+					grep -qx qemu:$(IMAGE); then \
+				$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
 				-t \
 				$(if $(DEBUG),-i,--net=none) \
 				-e TARGET_LIST=$(TARGET_LIST) \
@@ -117,6 +119,7 @@ docker-run-%: docker-qemu-src
 				qemu:$(IMAGE) \
 				/var/tmp/qemu/run \
 				$(CMD); \
+			fi \
 			, "  RUN $(CMD) in $(IMAGE)")))
 
 docker-clean:
-- 
2.7.4

Re: [Qemu-devel] [PATCH v6 4/8] tests/docker/dockerfiles: new debian-bootstrap.docker

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> From: Alex Bennée <alex.bennee@linaro.org>
>
> Together with the debian-bootstrap.pre script can now build an arbitrary
> architecture of Debian using debootstrap. This allows debootstrap to set
> up its first stage before the container is built.
>
> To build a container you need a command line like:
>
>   DEB_ARCH=armhf DEB_TYPE=testing \
>     ./tests/docker/docker.py build \
>     --include-executable=arm-linux-user/qemu-arm debian:armhf \
>     ./tests/docker/dockerfiles/debian-bootstrap.docker
>
> Although a number of non-debian systems package the debootstrap script
> it is fairly portable in itself. Assuming we have some sort of fakeroot
> implementation we can just clone the upstream repository and use the
> script from there.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-id: 1468335639-24582-5-git-send-email-alex.bennee@linaro.org

Acked-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++
>  tests/docker/dockerfiles/debian-bootstrap.pre    | 87 ++++++++++++++++++++++++
>  2 files changed, 108 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre
>
> diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
> new file mode 100644
> index 0000000..3a9125e
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-bootstrap.docker
> @@ -0,0 +1,21 @@
> +# Create Debian Bootstrap Image
> +#
> +# This is intended to be pre-poluated by:
> +#  - a first stage debootstrap (see debian-bootstrap.pre)
> +#  - a native qemu-$arch that binfmt_misc will run
> +FROM scratch
> +
> +# Add everything from the context into the container
> +ADD . /
> +
> +# Patch all mounts as docker already has stuff set up
> +RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
> +
> +# Run stage 2
> +RUN /debootstrap/debootstrap --second-stage
> +
> +# At this point we can install additional packages if we want
> +# Duplicate deb line as deb-src
> +RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
> +RUN apt-get update
> +RUN apt-get -y build-dep qemu
> diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
> new file mode 100755
> index 0000000..55850be
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-bootstrap.pre
> @@ -0,0 +1,87 @@
> +#!/bin/sh
> +#
> +# Simple wrapper for debootstrap, run in the docker build context
> +#
> +FAKEROOT=`which fakeroot 2> /dev/null`
> +
> +exit_and_skip()
> +{
> +    exit 3
> +}
> +
> +#
> +# fakeroot is needed to run the bootstrap stage
> +#
> +if [ -z $FAKEROOT ]; then
> +    echo "Please install fakeroot to enable bootstraping"
> +    exit_and_skip
> +fi
> +
> +# We check in order for
> +#
> +#  - DEBOOTSTRAP_DIR pointing at a development checkout
> +#  - PATH for the debootstrap script (installed)
> +#
> +# If neither option works then we checkout debootstrap from its
> +# upstream SCM and run it from there.
> +#
> +
> +if [ -z $DEBOOTSTRAP_DIR ]; then
> +    DEBOOTSTRAP=`which debootstrap 2> /dev/null`
> +    if [ -z $DEBOOTSTRAP ]; then
> +        echo "No debootstrap installed, attempting to install from SCM"
> +        DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
> +        git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
> +        export DEBOOTSTRAP_DIR=./debootstrap.git
> +        DEBOOTSTRAP=./debootstrap.git/debootstrap
> +    fi
> +else
> +    DEBOOTSTRAP=${DEBOOTSTRAP_DIR}/debootstrap
> +    if [ ! -f $DEBOOTSTRAP ]; then
> +        echo "Couldn't find script at ${DEBOOTSTRAP}"
> +        exit_and_skip
> +    fi
> +fi
> +
> +#
> +# Finally check to see if any qemu's are installed
> +#
> +BINFMT_DIR=/proc/sys/fs/binfmt_misc
> +if [ ! -e $BINFMT_DIR ]; then
> +   echo "binfmt_misc needs enabling for a QEMU bootstrap to work"
> +   exit_and_skip
> +else
> +    # DEB_ARCH and QEMU arch names are not totally aligned
> +    case "${DEB_ARCH}" in
> +        amd64)
> +            QEMU=qemu-i386
> +            ;;
> +        armel|armhf)
> +            QEMU=qemu-arm
> +            ;;
> +        arm64)
> +            QEMU=qemu-aarch64
> +            ;;
> +        powerpc)
> +            QEMU=qemu-ppc
> +            ;;
> +        ppc64el)
> +            QEMU=qemu-ppc64le
> +            ;;
> +        s390)
> +            QEMU=qemu-s390x
> +            ;;
> +        *)
> +            QEMU=qemu-${DEB_ARCH}
> +        ;;
> +    esac
> +    if [ ! -e "${BINFMT_DIR}/$QEMU" ]; then
> +        echo "No binfmt_misc rule to run $QEMU, can't bootstrap"
> +        exit_and_skip
> +    fi
> +fi
> +
> +echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP}"
> +
> +${FAKEROOT} ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian || exit 1
> +exit 0


--
Alex Bennée

Re: [Qemu-devel] [PATCH v6 3/8] tests/docker/docker.py: check and run .pre script

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> From: Alex Bennée <alex.bennee@linaro.org>
>
> The docker script will now search for an associated $dockerfile.pre
> script which gets run in the same build context as the dockerfile will
> be. This is to support pre-seeding the build context before running the
> docker build.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-id: 1468335639-24582-4-git-send-email-alex.bennee@linaro.org
> ---
>  tests/docker/docker.py | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 96d906e..b61d934 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -221,6 +221,18 @@ class BuildCommand(SubCommand):
>              # Create a docker context directory for the build
>              docker_dir = tempfile.mkdtemp(prefix="docker_build")
>
> +            # Is there a .pre file to run in the build context?
> +            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
> +            if os.path.exists(docker_pre):
> +                rc = subprocess.call(os.path.realpath(docker_pre),
> +                                     cwd=docker_dir)
> +                if rc == 3:
> +                    print "Skip"
> +                    return 0
> +                else:
> +                    print "%s exited with code %d" % (docker_pre, rc)
> +                    return 1
> +

We need to let an exit of 0 through ;-)

>              # Do we include a extra binary?
>              if args.include_executable:
>                  _copy_binary_with_libs(args.include_executable,


--
Alex Bennée

Re: [Qemu-devel] [PATCH v6 0/8] docker: Support building qemu-user powered docker test images

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> v6: A few tweaks to make this more mergable. Including:
>     - Add and catch magic pre script exit code "3" to allow skipping image.
>     - Tweak debian-bootstrap.pre to return '3' when appropriate (It should
>       still return 1 for debootstrap command failure).
>     - Fix "realpath" to "readlink -e" to be compatible with centos6.
>     - Add the last patch to skip test if the image is skipped by .pre.
>
> Alex, if this looks good to you, I can send a pull req today.

If you could apply the following changes to the appropriate patches I
think we are good to go:

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 2bb8a51..d85f50f 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -46,7 +46,8 @@ docker-image: ${DOCKER_TARGETS}
 docker-image-%: $(DOCKER_FILES_DIR)/%.docker
        $(call quiet-command,\
                $(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
-		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache),\
+		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
+		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
                "  BUILD $*")

 # Expand all the pre-requistes for each docker image and test combination
@@ -95,6 +96,7 @@ docker:
        @echo '    DEBUG=1              Stop and drop to shell in the created container'
        @echo '                         before running the command.'
        @echo '    NOCACHE=1            Ignore cache when build images.'
+	@echo '    EXECUTABLE=<path>    Include executable in image.'

 docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
 docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index fd162c0..40bda9d 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -238,7 +238,7 @@ class BuildCommand(SubCommand):
                 if rc == 3:
                     print "Skip"
                     return 0
-                else:
+                elif rc != 0:
                     print "%s exited with code %d" % (docker_pre, rc)
                     return 1

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index 55850be..5d9c8d5 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -81,7 +81,7 @@ else
     fi
 fi

-echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP}"
+echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP} ${DEB_ARCH}/${DEB_TYPE}"

 ${FAKEROOT} ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian || exit 1
 exit 0


>
> Alex Bennée (5):
>   tests/docker/docker.py: docker_dir outside build
>   tests/docker/docker.py: support --include-executable
>   tests/docker/docker.py: check and run .pre script
>   tests/docker/dockerfiles: new debian-bootstrap.docker
>   tests/docker/docker.py: add update operation
>
> Fam Zheng (3):
>   docker: More sensible run script
>   docker: Fix exit code if $CMD failed
>   docker: Don't start a container that doesn't exist
>
>  tests/docker/Makefile.include                    |   9 +-
>  tests/docker/docker.py                           | 150 +++++++++++++++++++++--
>  tests/docker/dockerfiles/debian-bootstrap.docker |  21 ++++
>  tests/docker/dockerfiles/debian-bootstrap.pre    |  87 +++++++++++++
>  tests/docker/run                                 |  16 ++-
>  5 files changed, 267 insertions(+), 16 deletions(-)
>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre


--
Alex Bennée

Re: [Qemu-devel] [PATCH v6 8/8] docker: Don't start a container that doesn't exist

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> Image building targets are dependencies of test running targets, so when
> a docker image doesn't exist, it means it's skipped (due to dependency
> checks in pre script). Therefore, skip the test too.
>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  tests/docker/Makefile.include | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index c5546ee..2bb8a51 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -105,7 +105,9 @@ docker-run-%: docker-qemu-src
>  	fi
>  	$(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
>  		$(call quiet-command,\
> -			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
> +			if sudo docker images --format={{.Repository}}:{{.Tag}} | \
> +					grep -qx qemu:$(IMAGE); then \
> +				$(SRC_PATH)/tests/docker/docker.py run
> $(if $V,,--rm) \

The sudo belongs in the docker.py script

>  				-t \
>  				$(if $(DEBUG),-i,--net=none) \
>  				-e TARGET_LIST=$(TARGET_LIST) \
> @@ -117,6 +119,7 @@ docker-run-%: docker-qemu-src
>  				qemu:$(IMAGE) \
>  				/var/tmp/qemu/run \
>  				$(CMD); \
> +			fi \
>  			, "  RUN $(CMD) in $(IMAGE)")))
>
>  docker-clean:


--
Alex Bennée

Re: [Qemu-devel] [PATCH v6 0/8] docker: Support building qemu-user powered docker test images

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> v6: A few tweaks to make this more mergable. Including:
>     - Add and catch magic pre script exit code "3" to allow skipping image.
>     - Tweak debian-bootstrap.pre to return '3' when appropriate (It should
>       still return 1 for debootstrap command failure).
>     - Fix "realpath" to "readlink -e" to be compatible with centos6.
>     - Add the last patch to skip test if the image is skipped by .pre.
>
> Alex, if this looks good to you, I can send a pull req today.

I've rolled the fixes into v7:

  https://github.com/stsquad/qemu/tree/review/docker-user-v7

>
> Alex Bennée (5):
>   tests/docker/docker.py: docker_dir outside build
>   tests/docker/docker.py: support --include-executable
>   tests/docker/docker.py: check and run .pre script
>   tests/docker/dockerfiles: new debian-bootstrap.docker
>   tests/docker/docker.py: add update operation
>
> Fam Zheng (3):
>   docker: More sensible run script
>   docker: Fix exit code if $CMD failed
>   docker: Don't start a container that doesn't exist
>
>  tests/docker/Makefile.include                    |   9 +-
>  tests/docker/docker.py                           | 150 +++++++++++++++++++++--
>  tests/docker/dockerfiles/debian-bootstrap.docker |  21 ++++
>  tests/docker/dockerfiles/debian-bootstrap.pre    |  87 +++++++++++++
>  tests/docker/run                                 |  16 ++-
>  5 files changed, 267 insertions(+), 16 deletions(-)
>  create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
>  create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre


--
Alex Bennée

Re: [Qemu-devel] [PATCH v6 2/8] tests/docker/docker.py: support --include-executable

[Daniel P. Berrange]

On Tue, Jul 19, 2016 at 04:16:42PM +0800, Fam Zheng wrote:
> From: Alex Bennée <alex.bennee@linaro.org>
> 
> When passed the path to a binary we copy it and any linked libraries (if
> it is dynamically linked) into the docker build context. These can then
> be included by a dockerfile with the line:
> 
>   # Copy all of context into container
>   ADD . /
> 
> This is mainly intended for setting up foreign architecture docker
> images which use qemu-$arch to do cross-architecture linux-user
> execution. It also relies on the host and guest file-system following
> reasonable multi-arch layouts so the copied libraries don't clash with
> the guest ones.

So that's going to fail on anything other than Debian derivatives,
since they'll be using /usr/lib or /usr/lib64 for all arches.

IMHO it'd be better to simply reject use of qemu-$arch if it is
not statically linked, rather than trying to deal with fact that
libraries between host FS and foreign guest arch FS may clash.

All distros except Fedora have long provided static qemu-$arch
builds and I've recently improved Fedora to also provide static
qemu$arch builds in F24 & rawhide.

So I don't see much compelling reason to support dynamically
linked qemu-$arch binaries - it'll just cause pain & suffering
with clashing libs on many distros.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH v6 2/8] tests/docker/docker.py: support --include-executable

[Alex Bennée]

Daniel P. Berrange <berrange@redhat.com> writes:

> On Tue, Jul 19, 2016 at 04:16:42PM +0800, Fam Zheng wrote:
>> From: Alex Bennée <alex.bennee@linaro.org>
>>
>> When passed the path to a binary we copy it and any linked libraries (if
>> it is dynamically linked) into the docker build context. These can then
>> be included by a dockerfile with the line:
>>
>>   # Copy all of context into container
>>   ADD . /
>>
>> This is mainly intended for setting up foreign architecture docker
>> images which use qemu-$arch to do cross-architecture linux-user
>> execution. It also relies on the host and guest file-system following
>> reasonable multi-arch layouts so the copied libraries don't clash with
>> the guest ones.
>
> So that's going to fail on anything other than Debian derivatives,
> since they'll be using /usr/lib or /usr/lib64 for all arches.

Well currently we only have a bootstrap for Debian anyway. However it
doesn't preclude you from using -static builds if you want.

Is Debian really the only multi-arch supporting distro out there? For
example I tested this in Arch as well. I'm fairly sure Gentoo is doing
it right too.

> IMHO it'd be better to simply reject use of qemu-$arch if it is
> not statically linked, rather than trying to deal with fact that
> libraries between host FS and foreign guest arch FS may clash.
>
> All distros except Fedora have long provided static qemu-$arch
> builds and I've recently improved Fedora to also provide static
> qemu$arch builds in F24 & rawhide.

That's great but we also want to test our built qemus. For example the
hot-path stuff has all been tested inside docker containers because it
makes the setting up of test cases a lot simpler and reproducible.

>
> So I don't see much compelling reason to support dynamically
> linked qemu-$arch binaries - it'll just cause pain & suffering
> with clashing libs on many distros.

I do, because I built with dynamic libs all the time for system
emulation. This way it is simple to build test other arches when I'm
making system emulation changes without having to constantly switch from
one build configuration to another.

I'll look into if there is any way we can do better with warnings on
non-multiarch systems?

--
Alex Bennée

Re: [Qemu-devel] [PATCH v6 3/8] tests/docker/docker.py: check and run .pre script

[Fam Zheng]

On Tue, 07/19 10:29, Alex Bennée wrote:
> > diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> > index 96d906e..b61d934 100755
> > --- a/tests/docker/docker.py
> > +++ b/tests/docker/docker.py
> > @@ -221,6 +221,18 @@ class BuildCommand(SubCommand):
> >              # Create a docker context directory for the build
> >              docker_dir = tempfile.mkdtemp(prefix="docker_build")
> >
> > +            # Is there a .pre file to run in the build context?
> > +            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
> > +            if os.path.exists(docker_pre):
> > +                rc = subprocess.call(os.path.realpath(docker_pre),
> > +                                     cwd=docker_dir)
> > +                if rc == 3:
> > +                    print "Skip"
> > +                    return 0
> > +                else:
> > +                    print "%s exited with code %d" % (docker_pre, rc)
> > +                    return 1
> > +
> 
> We need to let an exit of 0 through ;-)

Yes, I forgot that!

Fam