[Qemu-devel] [PATCH 0/4] tests/docker: add podman support

[Qemu-devel] [PATCH 0/4] tests/docker: add podman support

[Marc-André Lureau]

Hi,

podman allows to run containers in a similar fashion as docker, but
without daemon or root privileges. Thank you podman!

I haven't done extensive testing. Basic make docker-test rules
work. There seems to be a few issues with permissions at run time
(podman ps fails), but that seems not directly related to this series.

There was also a small issue running make check, due to not having
network address at all by default. See "qemu-sockets: do not require
configured ipv4/ipv6 address" patch for the proposed solution.

Marc-André Lureau (4):
  docker.py: add podman support
  tests/docker: add podman support
  docker: update fedora to f30
  qemu-sockets: do not require configured ipv4/ipv6 address

 util/qemu-sockets.c                    |  8 ++---
 Makefile                               |  2 +-
 tests/docker/Makefile.include          | 17 ++++++++--
 tests/docker/docker.py                 | 43 +++++++++++++++++++++++---
 tests/docker/dockerfiles/fedora.docker |  2 +-
 5 files changed, 56 insertions(+), 16 deletions(-)

-- 
2.22.0.rc1.1.g079e7d2849.dirty

[Qemu-devel] [PATCH 1/4] docker.py: add podman support

[Marc-André Lureau]

Add a --engine option to select either docker, podman or auto.

Among other advantages, podman allows to run rootless & daemonless
containers, fortunately sharing compatible CLI with docker.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/docker/docker.py | 43 +++++++++++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 5 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 53a8c9c801..1f59a78b10 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -20,6 +20,7 @@ import hashlib
 import atexit
 import uuid
 import argparse
+import enum
 import tempfile
 import re
 import signal
@@ -38,6 +39,26 @@ FILTERED_ENV_NAMES = ['ftp_proxy', 'http_proxy', 'https_proxy']
 
 DEVNULL = open(os.devnull, 'wb')
 
+class EngineEnum(enum.IntEnum):
+    AUTO = 1
+    DOCKER = 2
+    PODMAN = 3
+
+    def __str__(self):
+        return self.name.lower()
+
+    def __repr__(self):
+        return str(self)
+
+    @staticmethod
+    def argparse(s):
+        try:
+            return EngineEnum[s.upper()]
+        except KeyError:
+            return s
+
+
+USE_ENGINE = EngineEnum.AUTO
 
 def _text_checksum(text):
     """Calculate a digest string unique to the text content"""
@@ -48,9 +69,14 @@ def _file_checksum(filename):
     return _text_checksum(open(filename, 'rb').read())
 
 
-def _guess_docker_command():
-    """ Guess a working docker command or raise exception if not found"""
-    commands = [["docker"], ["sudo", "-n", "docker"]]
+def _guess_engine_command():
+    """ Guess a working engine command or raise exception if not found"""
+    commands = []
+
+    if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.PODMAN]:
+        commands += [["podman"]]
+    if USE_ENGINE in [EngineEnum.AUTO, EngineEnum.DOCKER]:
+        commands += [["docker"], ["sudo", "-n", "docker"]]
     for cmd in commands:
         try:
             # docker version will return the client details in stdout
@@ -61,7 +87,7 @@ def _guess_docker_command():
         except OSError:
             pass
     commands_txt = "\n".join(["  " + " ".join(x) for x in commands])
-    raise Exception("Cannot find working docker command. Tried:\n%s" %
+    raise Exception("Cannot find working engine command. Tried:\n%s" %
                     commands_txt)
 
 
@@ -190,7 +216,7 @@ def _dockerfile_preprocess(df):
 class Docker(object):
     """ Running Docker commands """
     def __init__(self):
-        self._command = _guess_docker_command()
+        self._command = _guess_engine_command()
         self._instances = []
         atexit.register(self._kill_instances)
         signal.signal(signal.SIGTERM, self._kill_instances)
@@ -502,6 +528,8 @@ class ProbeCommand(SubCommand):
                 print("yes")
             elif docker._command[0] == "sudo":
                 print("sudo")
+            elif docker._command[0] == "podman":
+                print("podman")
         except Exception:
             print("no")
 
@@ -597,9 +625,13 @@ class CheckCommand(SubCommand):
 
 
 def main():
+    global USE_ENGINE
+
     parser = argparse.ArgumentParser(description="A Docker helper",
                                      usage="%s <subcommand> ..." %
                                      os.path.basename(sys.argv[0]))
+    parser.add_argument("--engine", type=EngineEnum.argparse, choices=list(EngineEnum),
+                        help="specify which container engine to use")
     subparsers = parser.add_subparsers(title="subcommands", help=None)
     for cls in SubCommand.__subclasses__():
         cmd = cls()
@@ -608,6 +640,7 @@ def main():
         cmd.args(subp)
         subp.set_defaults(cmdobj=cmd)
     args, argv = parser.parse_known_args()
+    USE_ENGINE = args.engine
     return args.cmdobj.run(args, argv)
 
 
-- 
2.22.0.rc1.1.g079e7d2849.dirty

[Qemu-devel] [PATCH 2/4] tests/docker: add podman support

[Marc-André Lureau]

Allow to specify the container engine to run with ENGINE variable.

By default, ENGINE=auto and will select either podman or docker.

With current podman, we have to use a uidmap trick in order to be able
to rw-share the ccache directory with the container user.

With a user 1000, the default mapping is:
1000 (host) -> 0 (container).

So write access to /var/tmp/ccache ends will end with permission
denied error.

With "--uidmap 1000:0:1 --uidmap 0:1:1000", the mapping is:
1000 (host) -> 0 (container, 1st namespace) -> 1000 (container, 2nd namespace).

(the rest is mumbo jumbo to avoid holes in the range of UIDs)

A future podman version may have an option such as --userns-keep-uid.
Thanks to Debarshi Ray for the help!

Cc: Debarshi Ray <rishi@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 Makefile                      |  2 +-
 tests/docker/Makefile.include | 17 ++++++++++++++---
 2 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index e02b88bcb1..e3a150ac4c 100644
--- a/Makefile
+++ b/Makefile
@@ -1118,7 +1118,7 @@ endif
 	@echo  ''
 	@echo  'Test targets:'
 	@echo  '  check           - Run all tests (check-help for details)'
-	@echo  '  docker          - Help about targets running tests inside Docker containers'
+	@echo  '  docker          - Help about targets running tests inside containers'
 	@echo  '  vm-test         - Help about targets running tests inside VM'
 	@echo  ''
 	@echo  'Documentation targets:'
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c0e1bf57a3..2bf679767e 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -17,7 +17,9 @@ DOCKER_TESTS := $(notdir $(shell \
 
 DOCKER_TOOLS := travis
 
-DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py
+ENGINE := auto
+
+DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py --engine $(ENGINE)
 
 TESTS ?= %
 IMAGES ?= %
@@ -145,7 +147,7 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
 )
 
 docker:
-	@echo 'Build QEMU and run tests inside Docker containers'
+	@echo 'Build QEMU and run tests inside Docker or Podman containers'
 	@echo
 	@echo 'Available targets:'
 	@echo
@@ -192,6 +194,14 @@ endif
 	@echo '    EXECUTABLE=<path>    Include executable in image.'
 	@echo '    EXTRA_FILES="<path> [... <path>]"'
 	@echo '                         Include extra files in image.'
+	@echo '    ENGINE=auto/docker/podman'
+	@echo '                         Specify which container engine to run.'
+
+UID=$(shell id -u)
+UID1=$(shell expr $(UID) + 1)
+ifeq ($(shell $(DOCKER_SCRIPT) probe),podman)
+PODMAN=1
+endif
 
 # This rule if for directly running against an arbitrary docker target.
 # It is called by the expanded docker targets (e.g. make
@@ -211,7 +221,8 @@ docker-run: docker-qemu-src
 			"  COPYING $(EXECUTABLE) to $(IMAGE)"))
 	$(call quiet-command,						\
 		$(DOCKER_SCRIPT) run 					\
-			$(if $(NOUSER),,-u $(shell id -u)) 		\
+			$(if $(NOUSER),,-u $(UID)			\
+				$(if $(PODMAN),--uidmap $(UID):0:1 --uidmap 0:1:$(UID) --uidmap $(UID1):$(UID1):64536)) 		\
 			--security-opt seccomp=unconfined		\
 			$(if $V,,--rm) 					\
 			$(if $(DEBUG),-ti,)				\
-- 
2.22.0.rc1.1.g079e7d2849.dirty

[Qemu-devel] [PATCH 3/4] docker: update fedora to f30

[Marc-André Lureau]

Released last month.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/docker/dockerfiles/fedora.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 69d4a7f5d7..1496b68ba1 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -1,4 +1,4 @@
-FROM fedora:29
+FROM fedora:30
 ENV PACKAGES \
     bc \
     bison \
-- 
2.22.0.rc1.1.g079e7d2849.dirty

[Qemu-devel] [PATCH 4/4] qemu-sockets: do not require configured ipv4/ipv6 address

[Marc-André Lureau]

podman containers without network don't have ipv4/ipv6 addresses other
than loopback address. However, some of our tests require
getaddrinfo("127.0.0.1") to succeed.

Alternatively, we may want to treat 127.0.0.1 as a special case, to
keep the AI_ADDRCONFIG convenience.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 util/qemu-sockets.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c
index 8850a280a8..f9c1392a05 100644
--- a/util/qemu-sockets.c
+++ b/util/qemu-sockets.c
@@ -31,10 +31,6 @@
 #include "qapi/qobject-output-visitor.h"
 #include "qemu/cutils.h"
 
-#ifndef AI_ADDRCONFIG
-# define AI_ADDRCONFIG 0
-#endif
-
 #ifndef AI_V4MAPPED
 # define AI_V4MAPPED 0
 #endif
@@ -385,7 +381,7 @@ static struct addrinfo *inet_parse_connect_saddr(InetSocketAddress *saddr,
 
     memset(&ai, 0, sizeof(ai));
 
-    ai.ai_flags = AI_CANONNAME | AI_ADDRCONFIG;
+    ai.ai_flags = AI_CANONNAME;
     if (atomic_read(&useV4Mapped)) {
         ai.ai_flags |= AI_V4MAPPED;
     }
@@ -472,7 +468,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
 
     /* lookup peer addr */
     memset(&ai,0, sizeof(ai));
-    ai.ai_flags = AI_CANONNAME | AI_V4MAPPED | AI_ADDRCONFIG;
+    ai.ai_flags = AI_CANONNAME | AI_V4MAPPED;
     ai.ai_family = inet_ai_family_from_address(sraddr, &err);
     ai.ai_socktype = SOCK_DGRAM;
 
-- 
2.22.0.rc1.1.g079e7d2849.dirty

Re: [Qemu-devel] [PATCH 0/4] tests/docker: add podman support

[Gerd Hoffmann]

On Fri, May 24, 2019 at 01:40:07AM +0200, Marc-André Lureau wrote:
> Hi,
> 
> podman allows to run containers in a similar fashion as docker, but
> without daemon or root privileges. Thank you podman!

Well, I saw a rather dramatic increase on disk usage when running podman
rootless.  Looked like podman did a full container image copy for each
docker file step instead of properly stacking incremental changes.
Didn't investigate why.

Therefore my "docker" looks like this:

   kraxel@sirius ~# cat bin/docker
   #!/bin/sh
   proxy="https_proxy,http_proxy,ftp_proxy,no_proxy"
   exec /usr/bin/sudo --preserve-env="${proxy}" /usr/bin/podman "$@"

So, yes, podman can run our docker tests just fine, but the rootless
mode has some hickups still.

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 3/4] docker: update fedora to f30

[Philippe Mathieu-Daudé]

On 5/24/19 1:40 AM, Marc-André Lureau wrote:
> Released last month.
> 
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  tests/docker/dockerfiles/fedora.docker | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
> index 69d4a7f5d7..1496b68ba1 100644
> --- a/tests/docker/dockerfiles/fedora.docker
> +++ b/tests/docker/dockerfiles/fedora.docker
> @@ -1,4 +1,4 @@
> -FROM fedora:29
> +FROM fedora:30

Hmm this patch is pending for review:
https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg00819.html

>  ENV PACKAGES \
>      bc \
>      bison \
> 

Re: [Qemu-devel] [PATCH 3/4] docker: update fedora to f30

[Marc-André Lureau]

Hi

On Fri, May 24, 2019 at 9:41 AM Philippe Mathieu-Daudé
<philmd@redhat.com> wrote:
>
> On 5/24/19 1:40 AM, Marc-André Lureau wrote:
> > Released last month.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  tests/docker/dockerfiles/fedora.docker | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
> > index 69d4a7f5d7..1496b68ba1 100644
> > --- a/tests/docker/dockerfiles/fedora.docker
> > +++ b/tests/docker/dockerfiles/fedora.docker
> > @@ -1,4 +1,4 @@
> > -FROM fedora:29
> > +FROM fedora:30
>
> Hmm this patch is pending for review:
> https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg00819.html

Oh I missed that. Maybe we should use "latest" to avoid bumping the
version every 6 months.

fwiw we have different versions:

tests/docker/dockerfiles/fedora-cris-cross.docker:FROM fedora:latest
tests/docker/dockerfiles/fedora-i386-cross.docker:FROM fedora:29
tests/docker/dockerfiles/fedora.docker:FROM fedora:29

In 62559b916 "tests: update Fedora i386 cross image to Fedora 29", Daniel said:

    Using the "latest" tag is not a good idea because this changes what
    release it points to every 6 months. Together with caching of docker
    builds this can cause confusion where CI has cached & built with Fedora
    N, while a developer tries to reproduce a CI problem with Fedora N + 1,
    or vica-verca.

But at the same time, Daniel bumped f28 to f29 in commit 19a9978db1.

It's confusing, do we need some stability or follow the latest?

>
> >  ENV PACKAGES \
> >      bc \
> >      bison \
> >

Re: [Qemu-devel] [PATCH 3/4] docker: update fedora to f30

[Philippe Mathieu-Daudé]

On 5/24/19 1:17 PM, Marc-André Lureau wrote:
> Hi
> 
> On Fri, May 24, 2019 at 9:41 AM Philippe Mathieu-Daudé
> <philmd@redhat.com> wrote:
>>
>> On 5/24/19 1:40 AM, Marc-André Lureau wrote:
>>> Released last month.
>>>
>>> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
>>> ---
>>>  tests/docker/dockerfiles/fedora.docker | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
>>> index 69d4a7f5d7..1496b68ba1 100644
>>> --- a/tests/docker/dockerfiles/fedora.docker
>>> +++ b/tests/docker/dockerfiles/fedora.docker
>>> @@ -1,4 +1,4 @@
>>> -FROM fedora:29
>>> +FROM fedora:30
>>
>> Hmm this patch is pending for review:
>> https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg00819.html
> 
> Oh I missed that. Maybe we should use "latest" to avoid bumping the
> version every 6 months.
> 
> fwiw we have different versions:
> 
> tests/docker/dockerfiles/fedora-cris-cross.docker:FROM fedora:latest
> tests/docker/dockerfiles/fedora-i386-cross.docker:FROM fedora:29
> tests/docker/dockerfiles/fedora.docker:FROM fedora:29
> 
> In 62559b916 "tests: update Fedora i386 cross image to Fedora 29", Daniel said:
> 
>     Using the "latest" tag is not a good idea because this changes what
>     release it points to every 6 months. Together with caching of docker
>     builds this can cause confusion where CI has cached & built with Fedora
>     N, while a developer tries to reproduce a CI problem with Fedora N + 1,
>     or vica-verca.
> 
> But at the same time, Daniel bumped f28 to f29 in commit 19a9978db1.
> 
> It's confusing, do we need some stability or follow the latest?

Tracking a stable release helps to handle new compiler warnings when
bisecting.

See also:

commit 5b9b49d7bd3e0da13e8f6d58578443a11817f56e
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Fri Jan 12 12:11:43 2018 +0100

    docker: change Fedora base image to fedora:27

    Using "fedora:latest" makes behavior different depending on when you
    actually pulled the image from the docker repository.  In my case,
    the supposedly "latest" image was a Fedora 25 download from 8 months
    ago, and the new "test-debug" test was failing.

    Use "27" to improve reproducibility and make it clear when the image
    is obsolete.

Why we don't add a new file when a new version get released?
See: https://lists.gnu.org/archive/html/qemu-devel/2018-01/msg03868.html

Re: [Qemu-devel] [PATCH 3/4] docker: update fedora to f30

[Alex Bennée]

Marc-André Lureau <marcandre.lureau@redhat.com> writes:

> Hi
>
> On Fri, May 24, 2019 at 9:41 AM Philippe Mathieu-Daudé
> <philmd@redhat.com> wrote:
>>
>> On 5/24/19 1:40 AM, Marc-André Lureau wrote:
>> > Released last month.
>> >
>> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
>> > ---
>> >  tests/docker/dockerfiles/fedora.docker | 2 +-
>> >  1 file changed, 1 insertion(+), 1 deletion(-)
>> >
>> > diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
>> > index 69d4a7f5d7..1496b68ba1 100644
>> > --- a/tests/docker/dockerfiles/fedora.docker
>> > +++ b/tests/docker/dockerfiles/fedora.docker
>> > @@ -1,4 +1,4 @@
>> > -FROM fedora:29
>> > +FROM fedora:30
>>
>> Hmm this patch is pending for review:
>> https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg00819.html
>
> Oh I missed that. Maybe we should use "latest" to avoid bumping the
> version every 6 months.
>
> fwiw we have different versions:
>
> tests/docker/dockerfiles/fedora-cris-cross.docker:FROM fedora:latest

I'll fix that

> tests/docker/dockerfiles/fedora-i386-cross.docker:FROM fedora:29

The cross compilers images are different as they generally don't want to
include the "native" libraries. For Debian based ones we do have a base
image that everything builds up from but we haven't bothered for fedora
as we only have a few so far.

> tests/docker/dockerfiles/fedora.docker:FROM fedora:29
>
> In 62559b916 "tests: update Fedora i386 cross image to Fedora 29", Daniel said:
>
>     Using the "latest" tag is not a good idea because this changes what
>     release it points to every 6 months. Together with caching of docker
>     builds this can cause confusion where CI has cached & built with Fedora
>     N, while a developer tries to reproduce a CI problem with Fedora N + 1,
>     or vica-verca.
>
> But at the same time, Daniel bumped f28 to f29 in commit 19a9978db1.
>
> It's confusing, do we need some stability or follow the latest?
>
>>
>> >  ENV PACKAGES \
>> >      bc \
>> >      bison \
>> >


--
Alex Bennée

Re: [Qemu-devel] [PATCH 4/4] qemu-sockets: do not require configured ipv4/ipv6 address

[Daniel P. Berrangé]

On Fri, May 24, 2019 at 01:40:11AM +0200, Marc-André Lureau wrote:
> podman containers without network don't have ipv4/ipv6 addresses other
> than loopback address. However, some of our tests require
> getaddrinfo("127.0.0.1") to succeed.
> 
> Alternatively, we may want to treat 127.0.0.1 as a special case, to
> keep the AI_ADDRCONFIG convenience.

Keeping AI_ADDRCONFIG is mandatory as this is required for correctly
operating IPv4/IPv6 dual stack support.

In tests/socket-helpers.h I have a couple of APIs designed to help
in this case.

  int socket_can_bind_connect(const char *hostname);
  int socket_check_protocol_support(bool *has_ipv4, bool *has_ipv6);

The latter function just calls the former with "127.0.0.1" and
"::1" and reports results =.

The intention is that any test which relies on using getaddrinfo()
should call one of these functions, and then skip any tests which
have a dependancy on this working.

We certainly don't do these checks it all our tests right now
though, so it is not surprising if some fail.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH 3/4] docker: update fedora to f30

[Daniel P. Berrangé]

On Fri, May 24, 2019 at 01:17:17PM +0200, Marc-André Lureau wrote:
> Hi
> 
> On Fri, May 24, 2019 at 9:41 AM Philippe Mathieu-Daudé
> <philmd@redhat.com> wrote:
> >
> > On 5/24/19 1:40 AM, Marc-André Lureau wrote:
> > > Released last month.
> > >
> > > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > ---
> > >  tests/docker/dockerfiles/fedora.docker | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
> > > index 69d4a7f5d7..1496b68ba1 100644
> > > --- a/tests/docker/dockerfiles/fedora.docker
> > > +++ b/tests/docker/dockerfiles/fedora.docker
> > > @@ -1,4 +1,4 @@
> > > -FROM fedora:29
> > > +FROM fedora:30
> >
> > Hmm this patch is pending for review:
> > https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg00819.html
> 
> Oh I missed that. Maybe we should use "latest" to avoid bumping the
> version every 6 months.
> 
> fwiw we have different versions:
> 
> tests/docker/dockerfiles/fedora-cris-cross.docker:FROM fedora:latest
> tests/docker/dockerfiles/fedora-i386-cross.docker:FROM fedora:29
> tests/docker/dockerfiles/fedora.docker:FROM fedora:29
> 
> In 62559b916 "tests: update Fedora i386 cross image to Fedora 29", Daniel said:
> 
>     Using the "latest" tag is not a good idea because this changes what
>     release it points to every 6 months. Together with caching of docker
>     builds this can cause confusion where CI has cached & built with Fedora
>     N, while a developer tries to reproduce a CI problem with Fedora N + 1,
>     or vica-verca.
> 
> But at the same time, Daniel bumped f28 to f29 in commit 19a9978db1.
> 
> It's confusing, do we need some stability or follow the latest?

The problem is introduced by local caching. "latest" may point to "29"
today, but the CI system had cached content meaining its use of "latest"
still resolved to "28".

Using "29" meant both CI & developers saw the same image, even when
caching is used.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH 3/4] docker: update fedora to f30

[Philippe Mathieu-Daudé]

On 6/5/19 5:10 PM, Daniel P. Berrangé wrote:
> On Fri, May 24, 2019 at 01:17:17PM +0200, Marc-André Lureau wrote:
>> Hi
>>
>> On Fri, May 24, 2019 at 9:41 AM Philippe Mathieu-Daudé
>> <philmd@redhat.com> wrote:
>>>
>>> On 5/24/19 1:40 AM, Marc-André Lureau wrote:
>>>> Released last month.
>>>>
>>>> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
>>>> ---
>>>>  tests/docker/dockerfiles/fedora.docker | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
>>>> index 69d4a7f5d7..1496b68ba1 100644
>>>> --- a/tests/docker/dockerfiles/fedora.docker
>>>> +++ b/tests/docker/dockerfiles/fedora.docker
>>>> @@ -1,4 +1,4 @@
>>>> -FROM fedora:29
>>>> +FROM fedora:30
>>>
>>> Hmm this patch is pending for review:
>>> https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg00819.html
>>
>> Oh I missed that. Maybe we should use "latest" to avoid bumping the
>> version every 6 months.
>>
>> fwiw we have different versions:
>>
>> tests/docker/dockerfiles/fedora-cris-cross.docker:FROM fedora:latest
>> tests/docker/dockerfiles/fedora-i386-cross.docker:FROM fedora:29
>> tests/docker/dockerfiles/fedora.docker:FROM fedora:29
>>
>> In 62559b916 "tests: update Fedora i386 cross image to Fedora 29", Daniel said:
>>
>>     Using the "latest" tag is not a good idea because this changes what
>>     release it points to every 6 months. Together with caching of docker
>>     builds this can cause confusion where CI has cached & built with Fedora
>>     N, while a developer tries to reproduce a CI problem with Fedora N + 1,
>>     or vica-verca.
>>
>> But at the same time, Daniel bumped f28 to f29 in commit 19a9978db1.
>>
>> It's confusing, do we need some stability or follow the latest?
> 
> The problem is introduced by local caching. "latest" may point to "29"
> today, but the CI system had cached content meaining its use of "latest"
> still resolved to "28".
> 
> Using "29" meant both CI & developers saw the same image, even when
> caching is used.

Should we invert the default NOCACHE value?

See commits c1958e9d & 6fe3ae3f:

commit c1958e9d54c8de85ceda7c96b302b75a2f67b4e1
Author: Fam Zheng <famz@redhat.com>
Date:   Fri Nov 3 21:12:29 2017 +0800

    docker: Improved image checksum

    When a base image locally defined by QEMU, such as in the debian images,
    is updated, the dockerfile checksum mechanism in docker.py still skips
    updating the derived image, because it only looks at the literal content
    of the dockerfile, without considering changes to the base image.

    For example we have a recent fix e58c1f9b35e81 that fixed
    debian-win64-cross by updating its base image, debian8-mxe, but due to
    above "feature" of docker.py the image in question is automatically NOT
    rebuilt unless you add NOCACHE=1. It is noticed on Shippable:

    https://app.shippable.com/github/qemu/qemu/runs/541/2/console

    because after the fix is merged, the error still occurs, and the log
    shows the container image is, as explained above, not updated.

    This is because at the time docker.py was written, there wasn't any
    dependencies between QEMU's docker images.

    Now improve this to preprocess any "FROM qemu:*" directives in the
    dockerfiles while doing checksum, and inline the base image's dockerfile
    content, recursively. This ensures any changes on the depended _QEMU_
    images are taken into account.

    This means for external images that we expect to retrieve from docker
    registries, we still do it as before. It is not perfect, because
    registry images can get updated too. Technically we could substitute the
    image name with its hex ID as obtained with $(docker images $IMAGE
    --format="{{.Id}}"), but --format is not supported by RHEL 7, so leave
    it for now.

commit 6fe3ae3f194a675a3b73b6beab3ed5dd35db3be3
Author: Alex Bennée <alex.bennee@linaro.org>
Date:   Tue Jul 25 14:34:23 2017 +0100

    docker: docker.py make --no-cache skip checksum test

    If you invoke with NOCACHE=1 we pass --no-cache in the argv to
    docker.py but may still not force a rebuild if the dockerfile checksum
    hasn't changed. By testing for its presence we can force builds
    without having to manually remove the docker image.