[Qemu-devel] [PULL 00/10] Docker patches

[Qemu-devel] [PULL 00/10] Docker patches

[Fam Zheng]

Hi Peter,

The following changes since commit d25321f210f745e428bc5c82f3dfad7a4a95a6ae:

  Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into staging (2016-07-19 13:00:35 +0100)

are available in the git repository at:

  git@github.com:famz/qemu tags/docker-pull-request

for you to fetch changes up to 1b596a4f6b5f28533dca315bf3ba178207bb7f23:

  docker: pass EXECUTABLE to build script (2016-07-19 22:10:09 +0800)

----------------------------------------------------------------

- Alex's qemu-user based debootstrap testing.
- Some small improvements.

----------------------------------------------------------------

Alex Bennée (6):
  tests/docker/docker.py: docker_dir outside build
  tests/docker/docker.py: support --include-executable
  tests/docker/docker.py: check and run .pre script
  tests/docker/dockerfiles: new debian-bootstrap.docker
  tests/docker/docker.py: add update operation
  docker: pass EXECUTABLE to build script

Fam Zheng (4):
  docker: More sensible run script
  docker: Fix exit code if $CMD failed
  docker: Add "images" subcommand to docker.py
  docker: Don't start a container that doesn't exist

 tests/docker/Makefile.include                    |  14 +-
 tests/docker/docker.py                           | 159 +++++++++++++++++++++--
 tests/docker/dockerfiles/debian-bootstrap.docker |  21 +++
 tests/docker/dockerfiles/debian-bootstrap.pre    |  87 +++++++++++++
 tests/docker/run                                 |  16 ++-
 5 files changed, 280 insertions(+), 17 deletions(-)
 create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
 create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre

-- 
2.7.4

[Qemu-devel] [PULL 01/10] tests/docker/docker.py: docker_dir outside build

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

Instead of letting the build_image create the temporary working dir we
move the creation to the build command. This is preparation for the
later patches where additional files can be added to the build context
before the build step is run.

We also ensure we remove the build context after we are done (mkdtemp
doesn't do this automatically for you).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-2-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/docker.py | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 0151362..ae40bb3 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,7 +20,7 @@ import atexit
 import uuid
 import argparse
 import tempfile
-from shutil import copy
+from shutil import copy, rmtree
 
 def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
@@ -87,20 +87,20 @@ class Docker(object):
         labels = json.loads(resp)[0]["Config"].get("Labels", {})
         return labels.get("com.qemu.dockerfile-checksum", "")
 
-    def build_image(self, tag, dockerfile, df_path, quiet=True, argv=None):
+    def build_image(self, tag, docker_dir, dockerfile, quiet=True, argv=None):
         if argv == None:
             argv = []
-        tmp_dir = tempfile.mkdtemp(prefix="docker_build")
 
-        tmp_df = tempfile.NamedTemporaryFile(dir=tmp_dir, suffix=".docker")
+        tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker")
         tmp_df.write(dockerfile)
 
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
                      _text_checksum(dockerfile))
         tmp_df.flush()
+
         self._do(["build", "-t", tag, "-f", tmp_df.name] + argv + \
-                 [tmp_dir],
+                 [docker_dir],
                  quiet=quiet)
 
     def image_matches_dockerfile(self, tag, dockerfile):
@@ -164,10 +164,15 @@ class BuildCommand(SubCommand):
         if dkr.image_matches_dockerfile(tag, dockerfile):
             if not args.quiet:
                 print "Image is up to date."
-            return 0
+        else:
+            # Create a docker context directory for the build
+            docker_dir = tempfile.mkdtemp(prefix="docker_build")
+
+            dkr.build_image(tag, docker_dir, dockerfile,
+                            quiet=args.quiet, argv=argv)
+
+            rmtree(docker_dir)
 
-        dkr.build_image(tag, dockerfile, args.dockerfile,
-                        quiet=args.quiet, argv=argv)
         return 0
 
 class CleanCommand(SubCommand):
-- 
2.7.4

[Qemu-devel] [PULL 02/10] tests/docker/docker.py: support --include-executable

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

When passed the path to a binary we copy it and any linked libraries (if
it is dynamically linked) into the docker build context. These can then
be included by a dockerfile with the line:

  # Copy all of context into container
  ADD . /

This is mainly intended for setting up foreign architecture docker
images which use qemu-$arch to do cross-architecture linux-user
execution. It also relies on the host and guest file-system following
reasonable multi-arch layouts so the copied libraries don't clash with
the guest ones.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-3-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/docker.py | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index ae40bb3..96d906e 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,6 +20,7 @@ import atexit
 import uuid
 import argparse
 import tempfile
+import re
 from shutil import copy, rmtree
 
 def _text_checksum(text):
@@ -38,6 +39,54 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
+def _copy_with_mkdir(src, root_dir, sub_path):
+    """Copy src into root_dir, creating sub_path as needed."""
+    dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
+    try:
+        os.makedirs(dest_dir)
+    except OSError:
+        # we can safely ignore already created directories
+        pass
+
+    dest_file = "%s/%s" % (dest_dir, os.path.basename(src))
+    copy(src, dest_file)
+
+
+def _get_so_libs(executable):
+    """Return a list of libraries associated with an executable.
+
+    The paths may be symbolic links which would need to be resolved to
+    ensure theright data is copied."""
+
+    libs = []
+    ldd_re = re.compile(r"(/.*/)(\S*)")
+    try:
+        ldd_output = subprocess.check_output(["ldd", executable])
+        for line in ldd_output.split("\n"):
+            search = ldd_re.search(line)
+            if search and len(search.groups()) == 2:
+                so_path = search.groups()[0]
+                so_lib = search.groups()[1]
+                libs.append("%s/%s" % (so_path, so_lib))
+    except subprocess.CalledProcessError:
+        print "%s had no associated libraries (static build?)" % (executable)
+
+    return libs
+
+def _copy_binary_with_libs(src, dest_dir):
+    """Copy a binary executable and all its dependant libraries.
+
+    This does rely on the host file-system being fairly multi-arch
+    aware so the file don't clash with the guests layout."""
+
+    _copy_with_mkdir(src, dest_dir, "/usr/bin")
+
+    libs = _get_so_libs(src)
+    if libs:
+        for l in libs:
+            so_path = os.path.dirname(l)
+            _copy_with_mkdir(l , dest_dir, so_path)
+
 class Docker(object):
     """ Running Docker commands """
     def __init__(self):
@@ -151,6 +200,10 @@ class BuildCommand(SubCommand):
     """ Build docker image out of a dockerfile. Arguments: <tag> <dockerfile>"""
     name = "build"
     def args(self, parser):
+        parser.add_argument("--include-executable", "-e",
+                            help="""Specify a binary that will be copied to the
+                            container together with all its dependent
+                            libraries""")
         parser.add_argument("tag",
                             help="Image Tag")
         parser.add_argument("dockerfile",
@@ -168,6 +221,11 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Do we include a extra binary?
+            if args.include_executable:
+                _copy_binary_with_libs(args.include_executable,
+                                       docker_dir)
+
             dkr.build_image(tag, docker_dir, dockerfile,
                             quiet=args.quiet, argv=argv)
 
-- 
2.7.4

[Qemu-devel] [PULL 03/10] tests/docker/docker.py: check and run .pre script

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

The docker script will now search for an associated $dockerfile.pre
script which gets run in the same build context as the dockerfile will
be. This is to support pre-seeding the build context before running the
docker build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-4-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/docker.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 96d906e..76750c4 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -221,6 +221,18 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Is there a .pre file to run in the build context?
+            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
+            if os.path.exists(docker_pre):
+                rc = subprocess.call(os.path.realpath(docker_pre),
+                                     cwd=docker_dir)
+                if rc == 3:
+                    print "Skip"
+                    return 0
+                elif rc != 0:
+                    print "%s exited with code %d" % (docker_pre, rc)
+                    return 1
+
             # Do we include a extra binary?
             if args.include_executable:
                 _copy_binary_with_libs(args.include_executable,
-- 
2.7.4

[Qemu-devel] [PULL 04/10] tests/docker/dockerfiles: new debian-bootstrap.docker

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

Together with the debian-bootstrap.pre script can now build an arbitrary
architecture of Debian using debootstrap. This allows debootstrap to set
up its first stage before the container is built.

To build a container you need a command line like:

  DEB_ARCH=armhf DEB_TYPE=testing \
    ./tests/docker/docker.py build \
    --include-executable=arm-linux-user/qemu-arm debian:armhf \
    ./tests/docker/dockerfiles/debian-bootstrap.docker

Although a number of non-debian systems package the debootstrap script
it is fairly portable in itself. Assuming we have some sort of fakeroot
implementation we can just clone the upstream repository and use the
script from there.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-5-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/dockerfiles/debian-bootstrap.docker | 21 ++++++
 tests/docker/dockerfiles/debian-bootstrap.pre    | 87 ++++++++++++++++++++++++
 2 files changed, 108 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-bootstrap.docker
 create mode 100755 tests/docker/dockerfiles/debian-bootstrap.pre

diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
new file mode 100644
index 0000000..3a9125e
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -0,0 +1,21 @@
+# Create Debian Bootstrap Image
+#
+# This is intended to be pre-poluated by:
+#  - a first stage debootstrap (see debian-bootstrap.pre)
+#  - a native qemu-$arch that binfmt_misc will run
+FROM scratch
+
+# Add everything from the context into the container
+ADD . /
+
+# Patch all mounts as docker already has stuff set up
+RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
+
+# Run stage 2
+RUN /debootstrap/debootstrap --second-stage
+
+# At this point we can install additional packages if we want
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
+RUN apt-get update
+RUN apt-get -y build-dep qemu
diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
new file mode 100755
index 0000000..5d9c8d5
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -0,0 +1,87 @@
+#!/bin/sh
+#
+# Simple wrapper for debootstrap, run in the docker build context
+#
+FAKEROOT=`which fakeroot 2> /dev/null`
+
+exit_and_skip()
+{
+    exit 3
+}
+
+#
+# fakeroot is needed to run the bootstrap stage
+#
+if [ -z $FAKEROOT ]; then
+    echo "Please install fakeroot to enable bootstraping"
+    exit_and_skip
+fi
+
+# We check in order for
+#
+#  - DEBOOTSTRAP_DIR pointing at a development checkout
+#  - PATH for the debootstrap script (installed)
+#
+# If neither option works then we checkout debootstrap from its
+# upstream SCM and run it from there.
+#
+
+if [ -z $DEBOOTSTRAP_DIR ]; then
+    DEBOOTSTRAP=`which debootstrap 2> /dev/null`
+    if [ -z $DEBOOTSTRAP ]; then
+        echo "No debootstrap installed, attempting to install from SCM"
+        DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
+        git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
+        export DEBOOTSTRAP_DIR=./debootstrap.git
+        DEBOOTSTRAP=./debootstrap.git/debootstrap
+    fi
+else
+    DEBOOTSTRAP=${DEBOOTSTRAP_DIR}/debootstrap
+    if [ ! -f $DEBOOTSTRAP ]; then
+        echo "Couldn't find script at ${DEBOOTSTRAP}"
+        exit_and_skip
+    fi
+fi
+
+#
+# Finally check to see if any qemu's are installed
+#
+BINFMT_DIR=/proc/sys/fs/binfmt_misc
+if [ ! -e $BINFMT_DIR ]; then
+   echo "binfmt_misc needs enabling for a QEMU bootstrap to work"
+   exit_and_skip
+else
+    # DEB_ARCH and QEMU arch names are not totally aligned
+    case "${DEB_ARCH}" in
+        amd64)
+            QEMU=qemu-i386
+            ;;
+        armel|armhf)
+            QEMU=qemu-arm
+            ;;
+        arm64)
+            QEMU=qemu-aarch64
+            ;;
+        powerpc)
+            QEMU=qemu-ppc
+            ;;
+        ppc64el)
+            QEMU=qemu-ppc64le
+            ;;
+        s390)
+            QEMU=qemu-s390x
+            ;;
+        *)
+            QEMU=qemu-${DEB_ARCH}
+        ;;
+    esac
+    if [ ! -e "${BINFMT_DIR}/$QEMU" ]; then
+        echo "No binfmt_misc rule to run $QEMU, can't bootstrap"
+        exit_and_skip
+    fi
+fi
+
+echo "Building a rootfs using ${FAKEROOT} and ${DEBOOTSTRAP} ${DEB_ARCH}/${DEB_TYPE}"
+
+${FAKEROOT} ${DEBOOTSTRAP} --variant=buildd --foreign --arch=$DEB_ARCH $DEB_TYPE . http://httpredir.debian.org/debian || exit 1
+exit 0
-- 
2.7.4

[Qemu-devel] [PULL 05/10] tests/docker/docker.py: add update operation

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

This adds a new operation to the docker script to allow updating of
binaries in an existing container. This is because it would be
inefficient to re-build the whole container just for an update to the
QEMU binary.

To update the executable run:

    ./tests/docker/docker.py update \
        debian:armhf ./arm-linux-user/qemu-arm

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-6-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/docker.py | 59 +++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 76750c4..40bda9d 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -21,6 +21,8 @@ import uuid
 import argparse
 import tempfile
 import re
+from tarfile import TarFile, TarInfo
+from StringIO import StringIO
 from shutil import copy, rmtree
 
 def _text_checksum(text):
@@ -94,9 +96,11 @@ class Docker(object):
         self._instances = []
         atexit.register(self._kill_instances)
 
-    def _do(self, cmd, quiet=True, **kwargs):
+    def _do(self, cmd, quiet=True, infile=None, **kwargs):
         if quiet:
             kwargs["stdout"] = subprocess.PIPE
+        if infile:
+            kwargs["stdin"] = infile
         return subprocess.call(self._command + cmd, **kwargs)
 
     def _do_kill_instances(self, only_known, only_active=True):
@@ -152,6 +156,11 @@ class Docker(object):
                  [docker_dir],
                  quiet=quiet)
 
+    def update_image(self, tag, tarball, quiet=True):
+        "Update a tagged image using "
+
+        self._do(["build", "-t", tag, "-"], quiet=quiet, infile=tarball)
+
     def image_matches_dockerfile(self, tag, dockerfile):
         try:
             checksum = self.get_image_dockerfile_checksum(tag)
@@ -245,6 +254,54 @@ class BuildCommand(SubCommand):
 
         return 0
 
+class UpdateCommand(SubCommand):
+    """ Update a docker image with new executables. Arguments: <tag> <executable>"""
+    name = "update"
+    def args(self, parser):
+        parser.add_argument("tag",
+                            help="Image Tag")
+        parser.add_argument("executable",
+                            help="Executable to copy")
+
+    def run(self, args, argv):
+        # Create a temporary tarball with our whole build context and
+        # dockerfile for the update
+        tmp = tempfile.NamedTemporaryFile(suffix="dckr.tar.gz")
+        tmp_tar = TarFile(fileobj=tmp, mode='w')
+
+        # Add the executable to the tarball
+        bn = os.path.basename(args.executable)
+        ff = "/usr/bin/%s" % bn
+        tmp_tar.add(args.executable, arcname=ff)
+
+        # Add any associated libraries
+        libs = _get_so_libs(args.executable)
+        if libs:
+            for l in libs:
+                tmp_tar.add(os.path.realpath(l), arcname=l)
+
+        # Create a Docker buildfile
+        df = StringIO()
+        df.write("FROM %s\n" % args.tag)
+        df.write("ADD . /\n")
+        df.seek(0)
+
+        df_tar = TarInfo(name="Dockerfile")
+        df_tar.size = len(df.buf)
+        tmp_tar.addfile(df_tar, fileobj=df)
+
+        tmp_tar.close()
+
+        # reset the file pointers
+        tmp.flush()
+        tmp.seek(0)
+
+        # Run the build with our tarball context
+        dkr = Docker()
+        dkr.update_image(args.tag, tmp, quiet=args.quiet)
+
+        return 0
+
 class CleanCommand(SubCommand):
     """Clean up docker instances"""
     name = "clean"
-- 
2.7.4

[Qemu-devel] [PULL 06/10] docker: More sensible run script

[Fam Zheng]

It is very easy to figure out current directory and bash option from the
execution, so do less in the Makefile invocation command line, and
figure both options in the script.

This makes the next patch easier.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-7-git-send-email-famz@redhat.com
---
 tests/docker/Makefile.include |  4 +---
 tests/docker/run              | 12 +++++++++---
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index f88c0a7..c5546ee 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -114,10 +114,8 @@ docker-run-%: docker-qemu-src
 				-e CCACHE_DIR=/var/tmp/ccache \
 				-v $$(realpath $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \
 				-v $(DOCKER_CCACHE_DIR):/var/tmp/ccache:z \
-				-w /var/tmp/qemu \
 				qemu:$(IMAGE) \
-				$(if $V,/bin/bash -x ,) \
-				./run \
+				/var/tmp/qemu/run \
 				$(CMD); \
 			, "  RUN $(CMD) in $(IMAGE)")))
 
diff --git a/tests/docker/run b/tests/docker/run
index ec3d119..ddfac05 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -11,6 +11,12 @@
 # or (at your option) any later version. See the COPYING file in
 # the top-level directory.
 
+if test -n "$V"; then
+    set -x
+fi
+
+BASE="$(dirname $(readlink -e $0))"
+
 # Prepare the environment
 . /etc/profile || true
 export PATH=/usr/lib/ccache:$PATH
@@ -24,10 +30,10 @@ export TEST_DIR=/tmp/qemu-test
 mkdir -p $TEST_DIR/{src,build,install}
 
 # Extract the source tarballs
-tar -C $TEST_DIR/src -xzf qemu.tgz
+tar -C $TEST_DIR/src -xzf $BASE/qemu.tgz
 for p in dtc pixman; do
-    if test -f $p.tgz; then
-        tar -C $TEST_DIR/src/$p -xzf $p.tgz
+    if test -f $BASE/$p.tgz; then
+        tar -C $TEST_DIR/src/$p -xzf $BASE/$p.tgz
         export FEATURES="$FEATURES $p"
     fi
 done
-- 
2.7.4

[Qemu-devel] [PULL 07/10] docker: Fix exit code if $CMD failed

[Fam Zheng]

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-8-git-send-email-famz@redhat.com
---
 tests/docker/run | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/docker/run b/tests/docker/run
index ddfac05..d85d49a 100755
--- a/tests/docker/run
+++ b/tests/docker/run
@@ -11,6 +11,8 @@
 # or (at your option) any later version. See the COPYING file in
 # the top-level directory.
 
+set -e
+
 if test -n "$V"; then
     set -x
 fi
@@ -61,4 +63,6 @@ elif test -n "$DEBUG"; then
     echo
     # Force error after shell exits
     $SHELL && exit 1
+else
+    exit 1
 fi
-- 
2.7.4

[Qemu-devel] [PULL 08/10] docker: Add "images" subcommand to docker.py

[Fam Zheng]

This is a wrapper for the 'docker images' command.

Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-9-git-send-email-famz@redhat.com
---
 tests/docker/docker.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 40bda9d..222a105 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -179,6 +179,9 @@ class Docker(object):
             self._instances.remove(label)
         return ret
 
+    def command(self, cmd, argv, quiet):
+        return self._do([cmd] + argv, quiet=quiet)
+
 class SubCommand(object):
     """A SubCommand template base class"""
     name = None # Subcommand name
@@ -309,6 +312,12 @@ class CleanCommand(SubCommand):
         Docker().clean()
         return 0
 
+class ImagesCommand(SubCommand):
+    """Run "docker images" command"""
+    name = "images"
+    def run(self, args, argv):
+        return Docker().command("images", argv, args.quiet)
+
 def main():
     parser = argparse.ArgumentParser(description="A Docker helper",
             usage="%s <subcommand> ..." % os.path.basename(sys.argv[0]))
-- 
2.7.4

[Qemu-devel] [PULL 09/10] docker: Don't start a container that doesn't exist

[Fam Zheng]

Image building targets are dependencies of test running targets, so when
a docker image doesn't exist, it means it's skipped (due to dependency
checks in pre script). Therefore, skip the test too.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-10-git-send-email-famz@redhat.com
---
 tests/docker/Makefile.include | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c5546ee..e7f0023 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -105,7 +105,10 @@ docker-run-%: docker-qemu-src
 	fi
 	$(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
 		$(call quiet-command,\
-			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
+			if $(SRC_PATH)/tests/docker/docker.py images \
+				--format={{.Repository}}:{{.Tag}} | \
+					grep -qx qemu:$(IMAGE); then \
+				$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
 				-t \
 				$(if $(DEBUG),-i,--net=none) \
 				-e TARGET_LIST=$(TARGET_LIST) \
@@ -117,6 +120,7 @@ docker-run-%: docker-qemu-src
 				qemu:$(IMAGE) \
 				/var/tmp/qemu/run \
 				$(CMD); \
+			fi \
 			, "  RUN $(CMD) in $(IMAGE)")))
 
 docker-clean:
-- 
2.7.4

[Qemu-devel] [PULL 10/10] docker: pass EXECUTABLE to build script

[Fam Zheng]

From: Alex Bennée <alex.bennee@linaro.org>

To build a docker image with which needs qemu linux-user emulation we
need to pass --include-executable to the build script. Using the same
mechanism as for other container controls we enable the option is
EXECUTABLE is set on the make command line e.g:

    make docker-image-debian-bootstrap V=1 J=9 DEB_ARCH=armhf \
        DEB_TYPE=stable EXECUTABLE=./arm-linux-user/qemu-arm

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-11-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/docker/Makefile.include | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index e7f0023..78af468 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -46,7 +46,8 @@ docker-image: ${DOCKER_TARGETS}
 docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 	$(call quiet-command,\
 		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
-		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache),\
+		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
+		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
 		"  BUILD $*")
 
 # Expand all the pre-requistes for each docker image and test combination
@@ -95,6 +96,7 @@ docker:
 	@echo '    DEBUG=1              Stop and drop to shell in the created container'
 	@echo '                         before running the command.'
 	@echo '    NOCACHE=1            Ignore cache when build images.'
+	@echo '    EXECUTABLE=<path>    Include executable in image.'
 
 docker-run-%: CMD = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\1/')
 docker-run-%: IMAGE = $(shell echo '$@' | sed -e 's/docker-run-\([^@]*\)@\(.*\)/\2/')
-- 
2.7.4

Re: [Qemu-devel] [PULL 00/10] Docker patches

[Peter Maydell]

On 19 July 2016 at 15:31, Fam Zheng <famz@redhat.com> wrote:
> Hi Peter,
>
> The following changes since commit d25321f210f745e428bc5c82f3dfad7a4a95a6ae:
>
>   Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into staging (2016-07-19 13:00:35 +0100)
>
> are available in the git repository at:
>
>   git@github.com:famz/qemu tags/docker-pull-request
>
> for you to fetch changes up to 1b596a4f6b5f28533dca315bf3ba178207bb7f23:
>
>   docker: pass EXECUTABLE to build script (2016-07-19 22:10:09 +0800)
>
> ----------------------------------------------------------------
>
> - Alex's qemu-user based debootstrap testing.
> - Some small improvements.
>
> ----------------------------------------------------------------

Hi; I'm afraid you've forgotten to add your signed-off-by line
as submaintainer to the patches which were written by Alex.
Could you fix this and resend, please? (You can just resend the
cover letter email.)

thanks
-- PMM

Re: [Qemu-devel] [PULL 00/10] Docker patches

[Fam Zheng]

On Wed, 07/20 11:31, Peter Maydell wrote:
> On 19 July 2016 at 15:31, Fam Zheng <famz@redhat.com> wrote:
> > Hi Peter,
> >
> > The following changes since commit d25321f210f745e428bc5c82f3dfad7a4a95a6ae:
> >
> >   Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into staging (2016-07-19 13:00:35 +0100)
> >
> > are available in the git repository at:
> >
> >   git@github.com:famz/qemu tags/docker-pull-request
> >
> > for you to fetch changes up to 1b596a4f6b5f28533dca315bf3ba178207bb7f23:
> >
> >   docker: pass EXECUTABLE to build script (2016-07-19 22:10:09 +0800)
> >
> > ----------------------------------------------------------------
> >
> > - Alex's qemu-user based debootstrap testing.
> > - Some small improvements.
> >
> > ----------------------------------------------------------------
> 
> Hi; I'm afraid you've forgotten to add your signed-off-by line
> as submaintainer to the patches which were written by Alex.
> Could you fix this and resend, please? (You can just resend the
> cover letter email.)

Amusing. I explicitly checked they are in the emails, which I think are added
by "git format-patch -s". But that is apparently not enough because the tag
doesn't get it.

Resending in a while.

Fam

Re: [Qemu-devel] [PULL 09/10] docker: Don't start a container that doesn't exist

[Paolo Bonzini]

On 19/07/2016 16:31, Fam Zheng wrote:
> Image building targets are dependencies of test running targets, so when
> a docker image doesn't exist, it means it's skipped (due to dependency
> checks in pre script). Therefore, skip the test too.
> 
> Signed-off-by: Fam Zheng <famz@redhat.com>
> Message-id: 1468934445-32183-10-git-send-email-famz@redhat.com
> ---
>  tests/docker/Makefile.include | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index c5546ee..e7f0023 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -105,7 +105,10 @@ docker-run-%: docker-qemu-src
>  	fi
>  	$(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
>  		$(call quiet-command,\
> -			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
> +			if $(SRC_PATH)/tests/docker/docker.py images \
> +				--format={{.Repository}}:{{.Tag}} | \
> +					grep -qx qemu:$(IMAGE); then \
> +				$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
>  				-t \
>  				$(if $(DEBUG),-i,--net=none) \
>  				-e TARGET_LIST=$(TARGET_LIST) \
> @@ -117,6 +120,7 @@ docker-run-%: docker-qemu-src
>  				qemu:$(IMAGE) \
>  				/var/tmp/qemu/run \
>  				$(CMD); \
> +			fi \
>  			, "  RUN $(CMD) in $(IMAGE)")))
>  
>  docker-clean:
> 

This breaks the version of Docker in RHEL7.2, which doesn't have
--format.  Is it possible to revert it?

Paolo

Re: [Qemu-devel] [PULL 09/10] docker: Don't start a container that doesn't exist

[Fam Zheng]

On Tue, 08/02 20:59, Paolo Bonzini wrote:
> 
> 
> On 19/07/2016 16:31, Fam Zheng wrote:
> > Image building targets are dependencies of test running targets, so when
> > a docker image doesn't exist, it means it's skipped (due to dependency
> > checks in pre script). Therefore, skip the test too.
> > 
> > Signed-off-by: Fam Zheng <famz@redhat.com>
> > Message-id: 1468934445-32183-10-git-send-email-famz@redhat.com
> > ---
> >  tests/docker/Makefile.include | 6 +++++-
> >  1 file changed, 5 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> > index c5546ee..e7f0023 100644
> > --- a/tests/docker/Makefile.include
> > +++ b/tests/docker/Makefile.include
> > @@ -105,7 +105,10 @@ docker-run-%: docker-qemu-src
> >  	fi
> >  	$(if $(filter $(TESTS),$(CMD)),$(if $(filter $(IMAGES),$(IMAGE)), \
> >  		$(call quiet-command,\
> > -			$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
> > +			if $(SRC_PATH)/tests/docker/docker.py images \
> > +				--format={{.Repository}}:{{.Tag}} | \
> > +					grep -qx qemu:$(IMAGE); then \
> > +				$(SRC_PATH)/tests/docker/docker.py run $(if $V,,--rm) \
> >  				-t \
> >  				$(if $(DEBUG),-i,--net=none) \
> >  				-e TARGET_LIST=$(TARGET_LIST) \
> > @@ -117,6 +120,7 @@ docker-run-%: docker-qemu-src
> >  				qemu:$(IMAGE) \
> >  				/var/tmp/qemu/run \
> >  				$(CMD); \
> > +			fi \
> >  			, "  RUN $(CMD) in $(IMAGE)")))
> >  
> >  docker-clean:
> > 
> 
> This breaks the version of Docker in RHEL7.2, which doesn't have
> --format.  Is it possible to revert it?

I'll work on a fix today.

Fam