[Qemu-devel] [PATCH 0/3] docker tests fixes

[Qemu-devel] [PATCH 0/3] docker tests fixes

[Sascha Silbe]

A couple of fixes for issues encountered while trying out the new
docker test support. The debian-bootstrap image still doesn't build
for me, but that's a problem for another day.

Thanks for the docker test support, BTW. The centos6 test came in
rather handy today for testing the glib < 2.30 compatibility code.

Sascha Silbe (3):
  docker.py: don't hang on large docker output
  docker: avoid dependency on 'realpath' package
  docker: debian-bootstrap.pre: print helpful message if
    DEB_ARCH/DEB_TYPE unset

 tests/docker/Makefile.include                 |  2 +-
 tests/docker/docker.py                        |  9 ++++++---
 tests/docker/dockerfiles/debian-bootstrap.pre | 13 +++++++++++++
 3 files changed, 20 insertions(+), 4 deletions(-)

-- 
1.9.1

[Qemu-devel] [PATCH 1/3] docker.py: don't hang on large docker output

[Sascha Silbe]

Unlike Popen.communicate(), subprocess.call() doesn't read from the
stdout file descriptor. If the child process produces more output than
fits into the pipe buffer, it will block indefinitely.

If we don't intend to consume the output, just send it straight to
/dev/null to avoid this issue.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
---
This fixes a hang for me when building the Ubuntu docker image (empty
docker image cache).

 tests/docker/docker.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 222a105..efb2bf4 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -25,6 +25,10 @@ from tarfile import TarFile, TarInfo
 from StringIO import StringIO
 from shutil import copy, rmtree
 
+
+DEVNULL = open(os.devnull, 'wb')
+
+
 def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
     return hashlib.sha1(text).hexdigest()
@@ -34,8 +38,7 @@ def _guess_docker_command():
     commands = [["docker"], ["sudo", "-n", "docker"]]
     for cmd in commands:
         if subprocess.call(cmd + ["images"],
-                           stdout=subprocess.PIPE,
-                           stderr=subprocess.PIPE) == 0:
+                           stdout=DEVNULL, stderr=DEVNULL) == 0:
             return cmd
     commands_txt = "\n".join(["  " + " ".join(x) for x in commands])
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
@@ -98,7 +101,7 @@ class Docker(object):
 
     def _do(self, cmd, quiet=True, infile=None, **kwargs):
         if quiet:
-            kwargs["stdout"] = subprocess.PIPE
+            kwargs["stdout"] = DEVNULL
         if infile:
             kwargs["stdin"] = infile
         return subprocess.call(self._command + cmd, **kwargs)
-- 
1.9.1

Re: [Qemu-devel] [PATCH 1/3] docker.py: don't hang on large docker output

[Janosch Frank]

On 08/18/2016 10:12 PM, Sascha Silbe wrote:
> Unlike Popen.communicate(), subprocess.call() doesn't read from the
> stdout file descriptor. If the child process produces more output than
> fits into the pipe buffer, it will block indefinitely.
> 
> If we don't intend to consume the output, just send it straight to
> /dev/null to avoid this issue.
> 
> Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>

Reviewed-by: Janosch Frank <frankja@linux.vnet.ibm.com>

[Qemu-devel] [PATCH 2/3] docker: avoid dependency on 'realpath' package

[Sascha Silbe]

The 'realpath' executable is shipped in a separate package that isn't
installed by default on some distros.

We already use 'readlink -e' (provided by GNU coreutils) in some other
part of the code, so let's settle for that instead.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
---
Too bad there isn't a POSIX equivalent of this.

 tests/docker/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 4f4707d..1b20db0 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -116,7 +116,7 @@ docker-run-%: docker-qemu-src
 				-e EXTRA_CONFIGURE_OPTS=$(EXTRA_CONFIGURE_OPTS) \
 				-e V=$V -e J=$J -e DEBUG=$(DEBUG)\
 				-e CCACHE_DIR=/var/tmp/ccache \
-				-v $$(realpath $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \
+				-v $$(readlink -e $(DOCKER_SRC_COPY)):/var/tmp/qemu:z$(COMMA)ro \
 				-v $(DOCKER_CCACHE_DIR):/var/tmp/ccache:z \
 				qemu:$(IMAGE) \
 				/var/tmp/qemu/run \
-- 
1.9.1

[Qemu-devel] [PATCH 3/3] docker: debian-bootstrap.pre: print helpful message if DEB_ARCH/DEB_TYPE unset

[Sascha Silbe]

The debian-bootstrap image doesn't choose a default architecture and
distribution version, instead the user has to set both DEB_ARCH and
DEB_TYPE in the environment. Print a reasonably helpful message if
either of them isn't set instead of complaining about "qemu-" being
missing or erroring out because we cannot cd to the mirror URL.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
---

I haven't figured out a good way to warn about qemu-user-* being
missing because EXECUTABLE isn't set. debian-bootstrap.pre runs before
docker.py copies the executable so I cannot check in
debian-bootstrap.pre whether the binfmt interpreter exists. The
EXECUTABLE environment variable needs to be set only when run via
make, so checking it in debian-bootstrap.pre is no good either. And an
additional docker-image-debian-bootstrap rule in the Makefile that
checks if EXECUTABLE is set would override the regular rule, not
enhance it.

 tests/docker/dockerfiles/debian-bootstrap.pre | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index 5d9c8d5..2ae363f 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -15,6 +15,19 @@ exit_and_skip()
 if [ -z $FAKEROOT ]; then
     echo "Please install fakeroot to enable bootstraping"
     exit_and_skip
+
+fi
+
+if [ -z "${DEB_ARCH}" ]; then
+    echo "Please set DEB_ARCH to choose an architecture (e.g. armhf)"
+    exit_and_skip
+
+fi
+
+if [ -z "${DEB_TYPE}" ]; then
+    echo "Please set DEB_TYPE to a Debian archive name (e.g. testing)"
+    exit_and_skip
+
 fi
 
 # We check in order for
-- 
1.9.1

Re: [Qemu-devel] [PATCH 0/3] docker tests fixes

[Alex Bennée]

Sascha Silbe <silbe@linux.vnet.ibm.com> writes:

> A couple of fixes for issues encountered while trying out the new
> docker test support. The debian-bootstrap image still doesn't build
> for me, but that's a problem for another day.

Can you offer any details? What host OS and what errors do you get?

>
> Thanks for the docker test support, BTW. The centos6 test came in
> rather handy today for testing the glib < 2.30 compatibility code.
>
> Sascha Silbe (3):
>   docker.py: don't hang on large docker output
>   docker: avoid dependency on 'realpath' package
>   docker: debian-bootstrap.pre: print helpful message if
>     DEB_ARCH/DEB_TYPE unset
>
>  tests/docker/Makefile.include                 |  2 +-
>  tests/docker/docker.py                        |  9 ++++++---
>  tests/docker/dockerfiles/debian-bootstrap.pre | 13 +++++++++++++
>  3 files changed, 20 insertions(+), 4 deletions(-)


--
Alex Bennée

Re: [Qemu-devel] [PATCH 0/3] docker tests fixes

[Sascha Silbe]

Dear Alex,

Alex Bennée <alex.bennee@linaro.org> writes:

> Sascha Silbe <silbe@linux.vnet.ibm.com> writes:
>
>> A couple of fixes for issues encountered while trying out the new
>> docker test support. The debian-bootstrap image still doesn't build
>> for me, but that's a problem for another day.
>
> Can you offer any details? What host OS and what errors do you get?

I'll assume you're talking about the remaining issue, rather than the
ones fixed by this series.

Haven't spent much time yet trying to figure it out. It looks like
sources.list inside the image is missing "deb-src" lines.

Running docker.py manually without --quiet (would be nice if DEBUG=1 did
that, BTW) I get the following output:

=== Begin ===
silbe@oc4731375738:~/build/qemu-devel$ date; DEB_ARCH=armel DEB_TYPE=stable EXECUTABLE=/usr/bin/qemu-arm-static DEBUG=1 /home/silbe/recoverable/qemu/tests/docker/docker.py build qemu:debian-bootstrap /home/silbe/recoverable/qemu/tests/docker/dockerfiles/debian-bootstrap.docker --include-executable=/usr/bin/qemu-arm-static
Tue Aug 23 15:39:00 CEST 2016
Building a rootfs using /usr/bin/fakeroot and /usr/sbin/debootstrap armel/stable
W: Cannot check Release signature; keyring file not available /usr/share/keyrings/debian-archive-keyring.gpg
[...]
I: Extracting zlib1g...
/usr/bin/qemu-arm-static had no associated libraries (static build?)
Sending build context to Docker daemon 201.2 MB
Sending build context to Docker daemon 
Step 0 : FROM scratch
 ---> 
Step 1 : ADD . /
 ---> 366e15a5b4d8
Removing intermediate container 865d34d36d2b
Step 2 : RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
 ---> Running in cf539dd3a3fd
 ---> 313acea8974a
Removing intermediate container cf539dd3a3fd
Step 3 : RUN /debootstrap/debootstrap --second-stage
 ---> Running in 7851b75bda73
I: Keyring file not available at /usr/share/keyrings/debian-archive-keyring.gpg; switching to https mirror https://mirrors.kernel.org/debian
I: Installing core packages...
[...]
I: Configuring libc-bin...
I: Base system installed successfully.
 ---> 68b8522ff2e7
Removing intermediate container 7851b75bda73
Step 4 : RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
 ---> Running in 4cc5be448b57
 ---> 4fa669cb8511
Removing intermediate container 4cc5be448b57
Step 5 : RUN apt-get update
 ---> Running in 354a69607da0
Reading package lists...
 ---> 3025b4f79141
Removing intermediate container 354a69607da0
Step 6 : RUN apt-get -y build-dep qemu
 ---> Running in 81bfa9e9fce8
Reading package lists...
Building dependency tree...
E: You must put some 'source' URIs in your sources.list
INFO[0184] The command [/bin/sh -c apt-get -y build-dep qemu] returned a non-zero code: 100 
=== End ===


Host is an x86_64 running Ubuntu 14.04.

Sascha
-- 
Softwareentwicklung Sascha Silbe, Niederhofenstraße 5/1, 71229 Leonberg
https://se-silbe.de/
USt-IdNr. DE281696641