[PATCH v1 0/3] current testing/next queue

[PATCH v1 0/3] current testing/next queue

[Alex Bennée]

Hi,

There isn't much in my queue at the moment. The move of the various
compilers to buster fixes one of the shippable problems. I have a
longer series cooking to support multiarch docker builds which I'll
try and get posted by the end of this week.

Alex Bennée (2):
  tests/docker: move most cross compilers to buster base
  tests/docker: better handle symlinked libs

Thomas Huth (1):
  gitlab-ci: Refresh the list of iotests

 .gitlab-ci.yml                                   | 12 ++++++------
 tests/docker/Makefile.include                    | 16 ++++++++--------
 tests/docker/docker.py                           |  3 ++-
 tests/docker/dockerfiles/debian-amd64.docker     |  2 +-
 .../docker/dockerfiles/debian-armel-cross.docker |  2 +-
 .../docker/dockerfiles/debian-armhf-cross.docker |  2 +-
 .../dockerfiles/debian-mips64el-cross.docker     |  2 +-
 .../dockerfiles/debian-mipsel-cross.docker       |  2 +-
 .../dockerfiles/debian-ppc64el-cross.docker      |  2 +-
 .../docker/dockerfiles/debian-s390x-cross.docker |  2 +-
 10 files changed, 23 insertions(+), 22 deletions(-)

-- 
2.20.1

[PATCH v1 1/3] tests/docker: move most cross compilers to buster base

[Alex Bennée]

This includes fixing up the dependencies (Which were already wrong for
one of the mips variants).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/Makefile.include                    | 16 ++++++++--------
 tests/docker/dockerfiles/debian-amd64.docker     |  2 +-
 .../docker/dockerfiles/debian-armel-cross.docker |  2 +-
 .../docker/dockerfiles/debian-armhf-cross.docker |  2 +-
 .../dockerfiles/debian-mips64el-cross.docker     |  2 +-
 .../dockerfiles/debian-mipsel-cross.docker       |  2 +-
 .../dockerfiles/debian-ppc64el-cross.docker      |  2 +-
 .../docker/dockerfiles/debian-s390x-cross.docker |  2 +-
 8 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 19dbe26169..43a8678688 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -91,19 +91,12 @@ endif
 # Enforce dependencies for composite images
 docker-image-debian9-mxe: docker-image-debian9
 ifeq ($(HOST_ARCH),x86_64)
-docker-image-debian-amd64: docker-image-debian9
+docker-image-debian-amd64: docker-image-debian10
 DOCKER_PARTIAL_IMAGES += debian-amd64-cross
 else
 docker-image-debian-amd64-cross: docker-image-debian10
 DOCKER_PARTIAL_IMAGES += debian-amd64
 endif
-docker-image-debian-armel-cross: docker-image-debian9
-docker-image-debian-armhf-cross: docker-image-debian9
-docker-image-debian-mips-cross: docker-image-debian9
-docker-image-debian-mipsel-cross: docker-image-debian9
-docker-image-debian-mips64el-cross: docker-image-debian9
-docker-image-debian-ppc64el-cross: docker-image-debian9
-docker-image-debian-s390x-cross: docker-image-debian9
 docker-image-debian-win32-cross: docker-image-debian9-mxe
 docker-image-debian-win64-cross: docker-image-debian9-mxe
 
@@ -118,12 +111,19 @@ endif
 
 docker-image-debian-alpha-cross: docker-image-debian10
 docker-image-debian-arm64-cross: docker-image-debian10
+docker-image-debian-armel-cross: docker-image-debian10
+docker-image-debian-armhf-cross: docker-image-debian10
 docker-image-debian-hppa-cross: docker-image-debian10
 docker-image-debian-m68k-cross: docker-image-debian10
+docker-image-debian-mips-cross: docker-image-debian10
 docker-image-debian-mips64-cross: docker-image-debian10
+docker-image-debian-mips64el-cross: docker-image-debian10
+docker-image-debian-mipsel-cross: docker-image-debian10
 docker-image-debian-powerpc-cross: docker-image-debian10
 docker-image-debian-ppc64-cross: docker-image-debian10
+docker-image-debian-ppc64el-cross: docker-image-debian10
 docker-image-debian-riscv64-cross: docker-image-debian10
+docker-image-debian-s390x-cross: docker-image-debian10
 docker-image-debian-sh4-cross: docker-image-debian10
 docker-image-debian-sparc64-cross: docker-image-debian10
 
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index 431e947ebd..3b860af106 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -4,7 +4,7 @@
 # This docker target builds on the debian Stretch base image. Further
 # libraries which are not widely available are installed by hand.
 #
-FROM qemu:debian9
+FROM qemu:debian10
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
 RUN apt update && \
diff --git a/tests/docker/dockerfiles/debian-armel-cross.docker b/tests/docker/dockerfiles/debian-armel-cross.docker
index 15378f8ea2..e3794a61c9 100644
--- a/tests/docker/dockerfiles/debian-armel-cross.docker
+++ b/tests/docker/dockerfiles/debian-armel-cross.docker
@@ -3,7 +3,7 @@
 #
 # This docker target builds on the debian Stretch base image.
 #
-FROM qemu:debian9
+FROM qemu:debian10
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
 # Add the foreign architecture we want and install dependencies
diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 4a20af6fe1..e163b8b956 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -3,7 +3,7 @@
 #
 # This docker target builds on the debian Stretch base image.
 #
-FROM qemu:debian9
+FROM qemu:debian10
 
 # Add the foreign architecture we want and install dependencies
 RUN dpkg --add-architecture armhf
diff --git a/tests/docker/dockerfiles/debian-mips64el-cross.docker b/tests/docker/dockerfiles/debian-mips64el-cross.docker
index 2fca112405..453b53ef72 100644
--- a/tests/docker/dockerfiles/debian-mips64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-mips64el-cross.docker
@@ -4,7 +4,7 @@
 # This docker target builds on the debian Stretch base image.
 #
 
-FROM qemu:debian9
+FROM qemu:debian10
 
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
index 4abf7832ac..3b6e975c68 100644
--- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -3,7 +3,7 @@
 #
 # This docker target builds on the debian Stretch base image.
 #
-FROM qemu:debian9
+FROM qemu:debian10
 
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
diff --git a/tests/docker/dockerfiles/debian-ppc64el-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
index 9973df9ff7..cd386f01d9 100644
--- a/tests/docker/dockerfiles/debian-ppc64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
@@ -3,7 +3,7 @@
 #
 # This docker target builds on the debian Stretch base image.
 #
-FROM qemu:debian9
+FROM qemu:debian10
 
 # Add the foreign architecture we want and install dependencies
 RUN dpkg --add-architecture ppc64el && \
diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index eb73c98855..43fe59836f 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -3,7 +3,7 @@
 #
 # This docker target builds on the debian Stretch base image.
 #
-FROM qemu:debian9
+FROM qemu:debian10
 
 # Add the s390x architecture
 RUN dpkg --add-architecture s390x
-- 
2.20.1

[PATCH v1 2/3] tests/docker: better handle symlinked libs

[Alex Bennée]

When we are copying we want to ensure we grab the first
resolution (the found in path section). However even that binary might
be a symlink so lets make sure we chase the symlinks to copy the right
binary to where it can be found.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 31d8adf836..96d4326d53 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -109,7 +109,7 @@ def _get_so_libs(executable):
     ensure theright data is copied."""
 
     libs = []
-    ldd_re = re.compile(r"(/.*/)(\S*)")
+    ldd_re = re.compile(r"=> ?(/.*/)(\S*)")
     try:
         ldd_output = subprocess.check_output(["ldd", executable]).decode('utf-8')
         for line in ldd_output.split("\n"):
@@ -145,6 +145,7 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
     if libs:
         for l in libs:
             so_path = os.path.dirname(l)
+            real_l = os.path.realpath(l)
             _copy_with_mkdir(l, dest_dir, so_path)
 
 
-- 
2.20.1

[PATCH v1 3/3] gitlab-ci: Refresh the list of iotests

[Alex Bennée]

From: Thomas Huth <thuth@redhat.com>

iotest 147 and 205 have recently been marked as "NBD-only", so they
are currently simply skipped and thus can be removed.

iotest 129 occasionally fails in the gitlab-CI, and according to Max,
there are some known issues with this test (see for example this URL:
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html ),
so for the time being, let's disable it until the problems are fixed.

The iotests 040, 127, 203 and 256 are scheduled to become part of "make
check-block", so we also do not have to test them seperately here anymore.

On the other side, new iotests have been added to the QEMU repository
in the past months, so we can now add some new test > 256 instead.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200121131936.8214-1-thuth@redhat.com>
---
 .gitlab-ci.yml | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 228783993e..c15e394f09 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -54,12 +54,12 @@ build-tcg-disabled:
  - make check-qapi-schema
  - cd tests/qemu-iotests/
  - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
-            052 063 077 086 101 104 106 113 147 148 150 151 152 157 159 160
-            163 170 171 183 184 192 194 197 205 208 215 221 222 226 227 236
- - ./check -qcow2 028 040 051 056 057 058 065 067 068 082 085 091 095 096 102
-            122 124 127 129 132 139 142 144 145 147 151 152 155 157 165 194
-            196 197 200 202 203 205 208 209 215 216 218 222 227 234 246 247
-            248 250 254 255 256
+            052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
+            170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
+ - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
+            124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
+            208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
+            260 261 262 263 264 270 272 273 277 279
 
 build-user:
  script:
-- 
2.20.1

Re: [PATCH v1 2/3] tests/docker: better handle symlinked libs

[Wainer dos Santos Moschetta]

On 1/22/20 8:22 AM, Alex Bennée wrote:
> When we are copying we want to ensure we grab the first
> resolution (the found in path section). However even that binary might
> be a symlink so lets make sure we chase the symlinks to copy the right
> binary to where it can be found.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   tests/docker/docker.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 31d8adf836..96d4326d53 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -109,7 +109,7 @@ def _get_so_libs(executable):
>       ensure theright data is copied."""
>   
>       libs = []
> -    ldd_re = re.compile(r"(/.*/)(\S*)")
> +    ldd_re = re.compile(r"=> ?(/.*/)(\S*)")
>       try:
>           ldd_output = subprocess.check_output(["ldd", executable]).decode('utf-8')
>           for line in ldd_output.split("\n"):
> @@ -145,6 +145,7 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
>       if libs:
>           for l in libs:
>               so_path = os.path.dirname(l)
> +            real_l = os.path.realpath(l)

real_l is not used.

- Wainer

>               _copy_with_mkdir(l, dest_dir, so_path)
>   
>   

Re: [PATCH v1 0/3] current testing/next queue

[Wainer dos Santos Moschetta]

Hi Alex,

On 1/22/20 8:22 AM, Alex Bennée wrote:
> Hi,
>
> There isn't much in my queue at the moment. The move of the various
> compilers to buster fixes one of the shippable problems. I have a
> longer series cooking to support multiarch docker builds which I'll
> try and get posted by the end of this week.

Do you mind to pick this up too? ->

[PATCH 0/1] travis.yml: Missing genisoimage package

https://www.mail-archive.com/qemu-devel@nongnu.org/msg669898.html

- Wainer


>
> Alex Bennée (2):
>    tests/docker: move most cross compilers to buster base
>    tests/docker: better handle symlinked libs
>
> Thomas Huth (1):
>    gitlab-ci: Refresh the list of iotests
>
>   .gitlab-ci.yml                                   | 12 ++++++------
>   tests/docker/Makefile.include                    | 16 ++++++++--------
>   tests/docker/docker.py                           |  3 ++-
>   tests/docker/dockerfiles/debian-amd64.docker     |  2 +-
>   .../docker/dockerfiles/debian-armel-cross.docker |  2 +-
>   .../docker/dockerfiles/debian-armhf-cross.docker |  2 +-
>   .../dockerfiles/debian-mips64el-cross.docker     |  2 +-
>   .../dockerfiles/debian-mipsel-cross.docker       |  2 +-
>   .../dockerfiles/debian-ppc64el-cross.docker      |  2 +-
>   .../docker/dockerfiles/debian-s390x-cross.docker |  2 +-
>   10 files changed, 23 insertions(+), 22 deletions(-)
>

Re: [PATCH v1 0/3] current testing/next queue

[Alex Bennée]

Wainer dos Santos Moschetta <wainersm@redhat.com> writes:

> Hi Alex,
>
> On 1/22/20 8:22 AM, Alex Bennée wrote:
>> Hi,
>>
>> There isn't much in my queue at the moment. The move of the various
>> compilers to buster fixes one of the shippable problems. I have a
>> longer series cooking to support multiarch docker builds which I'll
>> try and get posted by the end of this week.
>
> Do you mind to pick this up too? ->
>
> [PATCH 0/1] travis.yml: Missing genisoimage package
>
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg669898.html

Sure - queued.

>
> - Wainer
>
>
>>
>> Alex Bennée (2):
>>    tests/docker: move most cross compilers to buster base
>>    tests/docker: better handle symlinked libs
>>
>> Thomas Huth (1):
>>    gitlab-ci: Refresh the list of iotests
>>
>>   .gitlab-ci.yml                                   | 12 ++++++------
>>   tests/docker/Makefile.include                    | 16 ++++++++--------
>>   tests/docker/docker.py                           |  3 ++-
>>   tests/docker/dockerfiles/debian-amd64.docker     |  2 +-
>>   .../docker/dockerfiles/debian-armel-cross.docker |  2 +-
>>   .../docker/dockerfiles/debian-armhf-cross.docker |  2 +-
>>   .../dockerfiles/debian-mips64el-cross.docker     |  2 +-
>>   .../dockerfiles/debian-mipsel-cross.docker       |  2 +-
>>   .../dockerfiles/debian-ppc64el-cross.docker      |  2 +-
>>   .../docker/dockerfiles/debian-s390x-cross.docker |  2 +-
>>   10 files changed, 23 insertions(+), 22 deletions(-)
>>


-- 
Alex Bennée

Re: [PATCH v1 2/3] tests/docker: better handle symlinked libs

[Philippe Mathieu-Daudé]

On 1/22/20 2:24 PM, Wainer dos Santos Moschetta wrote:
> 
> On 1/22/20 8:22 AM, Alex Bennée wrote:
>> When we are copying we want to ensure we grab the first
>> resolution (the found in path section). However even that binary might
>> be a symlink so lets make sure we chase the symlinks to copy the right
>> binary to where it can be found.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>   tests/docker/docker.py | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index 31d8adf836..96d4326d53 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -109,7 +109,7 @@ def _get_so_libs(executable):
>>       ensure theright data is copied."""
>>       libs = []
>> -    ldd_re = re.compile(r"(/.*/)(\S*)")
>> +    ldd_re = re.compile(r"=> ?(/.*/)(\S*)")

Why the 'optional space' after "=>"?

>>       try:
>>           ldd_output = subprocess.check_output(["ldd", 
>> executable]).decode('utf-8')
>>           for line in ldd_output.split("\n"):
>> @@ -145,6 +145,7 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
>>       if libs:
>>           for l in libs:
>>               so_path = os.path.dirname(l)
>> +            real_l = os.path.realpath(l)
> 
> real_l is not used.

I suppose the idea is to use it as:

                 _copy_with_mkdir(real_l, dest_dir, so_path)

> 
> - Wainer
> 
>>               _copy_with_mkdir(l, dest_dir, so_path)
> 

Re: [PATCH v1 2/3] tests/docker: better handle symlinked libs

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> On 1/22/20 2:24 PM, Wainer dos Santos Moschetta wrote:
>> On 1/22/20 8:22 AM, Alex Bennée wrote:
>>> When we are copying we want to ensure we grab the first
>>> resolution (the found in path section). However even that binary might
>>> be a symlink so lets make sure we chase the symlinks to copy the right
>>> binary to where it can be found.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>>   tests/docker/docker.py | 3 ++-
>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>>> index 31d8adf836..96d4326d53 100755
>>> --- a/tests/docker/docker.py
>>> +++ b/tests/docker/docker.py
>>> @@ -109,7 +109,7 @@ def _get_so_libs(executable):
>>>       ensure theright data is copied."""
>>>       libs = []
>>> -    ldd_re = re.compile(r"(/.*/)(\S*)")
>>> +    ldd_re = re.compile(r"=> ?(/.*/)(\S*)")
>
> Why the 'optional space' after "=>"?

Actually what I want is an optional "=> "

>
>>>       try:
>>>           ldd_output = subprocess.check_output(["ldd",
>>> executable]).decode('utf-8')
>>>           for line in ldd_output.split("\n"):
>>> @@ -145,6 +145,7 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
>>>       if libs:
>>>           for l in libs:
>>>               so_path = os.path.dirname(l)
>>> +            real_l = os.path.realpath(l)
>> real_l is not used.
>
> I suppose the idea is to use it as:
>
>                 _copy_with_mkdir(real_l, dest_dir, so_path)

Yes - looks like I need to retest.

>
>> - Wainer
>> 
>>>               _copy_with_mkdir(l, dest_dir, so_path)
>> 


-- 
Alex Bennée

Re: [PATCH v1 2/3] tests/docker: better handle symlinked libs

[Philippe Mathieu-Daudé]

On 1/22/20 4:58 PM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> 
>> On 1/22/20 2:24 PM, Wainer dos Santos Moschetta wrote:
>>> On 1/22/20 8:22 AM, Alex Bennée wrote:
>>>> When we are copying we want to ensure we grab the first
>>>> resolution (the found in path section). However even that binary might
>>>> be a symlink so lets make sure we chase the symlinks to copy the right
>>>> binary to where it can be found.
>>>>
>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>> ---
>>>>    tests/docker/docker.py | 3 ++-
>>>>    1 file changed, 2 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>>>> index 31d8adf836..96d4326d53 100755
>>>> --- a/tests/docker/docker.py
>>>> +++ b/tests/docker/docker.py
>>>> @@ -109,7 +109,7 @@ def _get_so_libs(executable):
>>>>        ensure theright data is copied."""
>>>>        libs = []
>>>> -    ldd_re = re.compile(r"(/.*/)(\S*)")
>>>> +    ldd_re = re.compile(r"=> ?(/.*/)(\S*)")
>>
>> Why the 'optional space' after "=>"?
> 
> Actually what I want is an optional "=> "

Maybe r"(?:\S+ => )?(\S*) \(:?0x[0-9a-f]+\)"?

optional non-capturing upto "=> ",
capture until space,
non-capture address in parenthesis.

> 
>>
>>>>        try:
>>>>            ldd_output = subprocess.check_output(["ldd",
>>>> executable]).decode('utf-8')
>>>>            for line in ldd_output.split("\n"):
>>>> @@ -145,6 +145,7 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
>>>>        if libs:
>>>>            for l in libs:
>>>>                so_path = os.path.dirname(l)
>>>> +            real_l = os.path.realpath(l)
>>> real_l is not used.
>>
>> I suppose the idea is to use it as:
>>
>>                  _copy_with_mkdir(real_l, dest_dir, so_path)
> 
> Yes - looks like I need to retest.
> 
>>
>>> - Wainer
>>>
>>>>                _copy_with_mkdir(l, dest_dir, so_path)
>>>
> 
>