[PATCH v2 0/7] Run cross-compilation build tests in the gitlab-CI

[PATCH v2 0/7] Run cross-compilation build tests in the gitlab-CI

[Thomas Huth]

Now that we can use all our QEMU build containers in the gitlab-CI,
we can also run the cross-compilation jobs there. Of course, some
problems have to be fixed first, so this is taken care of in the first
four patches.

The following two patches make sure that we can also enable WHPX builds with
our debian-win64-cross container, so that we can compile-test this accelerator
code now, too.

The last patch then finally enables the cross-compilation jobs in the CI.

v2:
 - Dropped patches that are not necessary anymore
 - Added the first two patches to fix problems with the new meson build
   system

Thomas Huth (7):
  configure: Add system = 'linux' for meson when cross-compiling
  tests/docker: Install python3-setuptools in the debian9-mxe containers
  tests/Makefile: test-image-locking needs CONFIG_POSIX
  tests/Makefile: test-replication needs CONFIG_POSIX
  dockerfiles/debian-win64-cross: Download WHPX MinGW headers
  configure: Allow automatic WHPX detection
  gitlab-ci: Add cross-compiling build tests

 .gitlab-ci.d/crossbuilds.yml                  | 113 ++++++++++++++++++
 .gitlab-ci.yml                                |   1 +
 MAINTAINERS                                   |   1 +
 configure                                     |   4 +
 tests/Makefile.include                        |   6 +-
 .../dockerfiles/debian-win64-cross.docker     |   9 +-
 tests/docker/dockerfiles/debian9-mxe.docker   |   2 +-
 7 files changed, 133 insertions(+), 3 deletions(-)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml

-- 
2.18.2

[PATCH v2 1/7] configure: Add system = 'linux' for meson when cross-compiling

[Thomas Huth]

Meson needs the "system = xyz" line when cross-compiling. We are already
adding a "system = 'windows'" for the MinGW cross-compilation case here,
so let's add a "system = 'linux'" now for Linux hosts, too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 configure | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/configure b/configure
index d9ca87fbbb..01204ba0b5 100755
--- a/configure
+++ b/configure
@@ -8208,6 +8208,9 @@ if test -n "$cross_prefix"; then
             ?:*) pre_prefix=/ ;;
         esac
     fi
+    if test "$linux" = "yes" ; then
+        echo "system = 'linux'" >> $cross
+    fi
     case "$ARCH" in
         i386|x86_64)
             echo "cpu_family = 'x86'" >> $cross
-- 
2.18.2

Re: [PATCH v2 1/7] configure: Add system = 'linux' for meson when cross-compiling

[Daniel P. Berrangé]

On Sun, Aug 23, 2020 at 01:17:51PM +0200, Thomas Huth wrote:
> Meson needs the "system = xyz" line when cross-compiling. We are already
> adding a "system = 'windows'" for the MinGW cross-compilation case here,
> so let's add a "system = 'linux'" now for Linux hosts, too.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  configure | 3 +++
>  1 file changed, 3 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2 2/7] tests/docker: Install python3-setuptools in the debian9-mxe containers

[Thomas Huth]

The python setuptools are a requirement for meson, so we need to install
this additional package now.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/docker/dockerfiles/debian9-mxe.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian9-mxe.docker b/tests/docker/dockerfiles/debian9-mxe.docker
index ae2c222a6f..7865c821f4 100644
--- a/tests/docker/dockerfiles/debian9-mxe.docker
+++ b/tests/docker/dockerfiles/debian9-mxe.docker
@@ -8,7 +8,7 @@ FROM qemu/debian9
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
-    apt install -y --no-install-recommends gnupg dirmngr
+    apt install -y --no-install-recommends gnupg dirmngr python3-setuptools
 
 # Add the foreign architecture we want and install dependencies
 RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C6BF758A33A3A276 && \
-- 
2.18.2

Re: [PATCH v2 2/7] tests/docker: Install python3-setuptools in the debian9-mxe containers

[Daniel P. Berrangé]

On Sun, Aug 23, 2020 at 01:17:52PM +0200, Thomas Huth wrote:
> The python setuptools are a requirement for meson, so we need to install
> this additional package now.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/docker/dockerfiles/debian9-mxe.docker | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2 3/7] tests/Makefile: test-image-locking needs CONFIG_POSIX

[Thomas Huth]

test-image-locking.c uses the qemu_lock_fd_test() function which is
only available on POSIX-like systems.

Reviewed-by: John Snow <jsnow@redhat.com>
Message-Id: <20200804170055.2851-4-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/Makefile.include | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 9ac8f5b86a..497f1f21ff 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -87,7 +87,9 @@ check-unit-$(CONFIG_BLOCK) += tests/test-blockjob$(EXESUF)
 check-unit-$(CONFIG_BLOCK) += tests/test-blockjob-txn$(EXESUF)
 check-unit-$(CONFIG_BLOCK) += tests/test-block-backend$(EXESUF)
 check-unit-$(CONFIG_BLOCK) += tests/test-block-iothread$(EXESUF)
+ifeq ($(CONFIG_POSIX),y)
 check-unit-$(CONFIG_BLOCK) += tests/test-image-locking$(EXESUF)
+endif
 check-unit-y += tests/test-x86-cpuid$(EXESUF)
 # all code tested by test-x86-cpuid is inside topology.h
 ifeq ($(CONFIG_SOFTMMU),y)
-- 
2.18.2

Re: [PATCH v2 3/7] tests/Makefile: test-image-locking needs CONFIG_POSIX

[Daniel P. Berrangé]

On Sun, Aug 23, 2020 at 01:17:53PM +0200, Thomas Huth wrote:
> test-image-locking.c uses the qemu_lock_fd_test() function which is
> only available on POSIX-like systems.
> 
> Reviewed-by: John Snow <jsnow@redhat.com>
> Message-Id: <20200804170055.2851-4-thuth@redhat.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/Makefile.include | 2 ++
>  1 file changed, 2 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2 4/7] tests/Makefile: test-replication needs CONFIG_POSIX

[Thomas Huth]

test-replication uses sigaction() and friends which are only available
on POSIX-like systems.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200804170055.2851-5-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/Makefile.include | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 497f1f21ff..a9746c2ecb 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -156,11 +156,13 @@ check-unit-$(CONFIG_BLOCK)  += tests/test-crypto-afsplit$(EXESUF)
 check-unit-$(call land,$(CONFIG_BLOCK),$(CONFIG_QEMU_PRIVATE_XTS)) += tests/test-crypto-xts$(EXESUF)
 check-unit-$(CONFIG_BLOCK)  += tests/test-crypto-block$(EXESUF)
 check-unit-y += tests/test-logging$(EXESUF)
-check-unit-$(call land,$(CONFIG_BLOCK),$(CONFIG_REPLICATION)) += tests/test-replication$(EXESUF)
 check-unit-$(CONFIG_SOFTMMU) += tests/test-bufferiszero$(EXESUF)
 check-unit-y += tests/test-uuid$(EXESUF)
 check-unit-y += tests/ptimer-test$(EXESUF)
 check-unit-y += tests/test-qapi-util$(EXESUF)
+ifeq ($(CONFIG_BLOCK)$(CONFIG_REPLICATION)$(CONFIG_POSIX),yyy)
+check-unit-y += tests/test-replication$(EXESUF)
+endif
 
 generated-files-y += tests/test-qapi-types.h
 generated-files-y += tests/include/test-qapi-types-sub-module.h
-- 
2.18.2

Re: [PATCH v2 4/7] tests/Makefile: test-replication needs CONFIG_POSIX

[Daniel P. Berrangé]

On Sun, Aug 23, 2020 at 01:17:54PM +0200, Thomas Huth wrote:
> test-replication uses sigaction() and friends which are only available
> on POSIX-like systems.
> 
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Message-Id: <20200804170055.2851-5-thuth@redhat.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/Makefile.include | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2 5/7] dockerfiles/debian-win64-cross: Download WHPX MinGW headers

[Thomas Huth]

To compile-test the WHPX accelerator, we need to download these system
headers first (they are unfortunately not part of any released and
packaged MinGW toolchain yet).

Idea taken from another patch by Stefan Weil.

Message-Id: <20200804170055.2851-12-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/docker/dockerfiles/debian-win64-cross.docker | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
index 2fc9cfcbc6..4cc4a3f365 100644
--- a/tests/docker/dockerfiles/debian-win64-cross.docker
+++ b/tests/docker/dockerfiles/debian-win64-cross.docker
@@ -32,7 +32,14 @@ RUN apt-get update && \
         mxe-$TARGET-w64-mingw32.shared-sdl2 \
         mxe-$TARGET-w64-mingw32.shared-sdl2-mixer \
         mxe-$TARGET-w64-mingw32.shared-sdl2-gfx \
-        mxe-$TARGET-w64-mingw32.shared-zlib
+        mxe-$TARGET-w64-mingw32.shared-zlib \
+        curl && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvEmulation.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvemulation.h?format=raw" && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvPlatform.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatform.h?format=raw" && \
+    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/winhvplatformdefs.h \
+        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatformdefs.h?format=raw"
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=x86_64-w64-mingw32.shared-
-- 
2.18.2

Re: [PATCH v2 5/7] dockerfiles/debian-win64-cross: Download WHPX MinGW headers

[Daniel P. Berrangé]

On Sun, Aug 23, 2020 at 01:17:55PM +0200, Thomas Huth wrote:
> To compile-test the WHPX accelerator, we need to download these system
> headers first (they are unfortunately not part of any released and
> packaged MinGW toolchain yet).
> 
> Idea taken from another patch by Stefan Weil.
> 
> Message-Id: <20200804170055.2851-12-thuth@redhat.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/docker/dockerfiles/debian-win64-cross.docker | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)


> 
> diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
> index 2fc9cfcbc6..4cc4a3f365 100644
> --- a/tests/docker/dockerfiles/debian-win64-cross.docker
> +++ b/tests/docker/dockerfiles/debian-win64-cross.docker
> @@ -32,7 +32,14 @@ RUN apt-get update && \
>          mxe-$TARGET-w64-mingw32.shared-sdl2 \
>          mxe-$TARGET-w64-mingw32.shared-sdl2-mixer \
>          mxe-$TARGET-w64-mingw32.shared-sdl2-gfx \
> -        mxe-$TARGET-w64-mingw32.shared-zlib
> +        mxe-$TARGET-w64-mingw32.shared-zlib \
> +        curl && \
> +    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvEmulation.h \
> +        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvemulation.h?format=raw" && \
> +    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvPlatform.h \
> +        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatform.h?format=raw" && \
> +    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/winhvplatformdefs.h \
> +        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatformdefs.h?format=raw"

My only concern here is whether sourceforge.net  URLs are reliable enough
for CI, or whether we should instead be mirroring the w64 repo on
git.qemu.org ?

I guess this will be a short term issue until a new mingw release is made
and arrives i nthe distros, so we can try with the curl approach for
now and change if it causes problems

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 5/7] dockerfiles/debian-win64-cross: Download WHPX MinGW headers

[Thomas Huth]

On 01/09/2020 18.25, Daniel P. Berrangé wrote:
> On Sun, Aug 23, 2020 at 01:17:55PM +0200, Thomas Huth wrote:
>> To compile-test the WHPX accelerator, we need to download these system
>> headers first (they are unfortunately not part of any released and
>> packaged MinGW toolchain yet).
>>
>> Idea taken from another patch by Stefan Weil.
>>
>> Message-Id: <20200804170055.2851-12-thuth@redhat.com>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  tests/docker/dockerfiles/debian-win64-cross.docker | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> 
>>
>> diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
>> index 2fc9cfcbc6..4cc4a3f365 100644
>> --- a/tests/docker/dockerfiles/debian-win64-cross.docker
>> +++ b/tests/docker/dockerfiles/debian-win64-cross.docker
>> @@ -32,7 +32,14 @@ RUN apt-get update && \
>>          mxe-$TARGET-w64-mingw32.shared-sdl2 \
>>          mxe-$TARGET-w64-mingw32.shared-sdl2-mixer \
>>          mxe-$TARGET-w64-mingw32.shared-sdl2-gfx \
>> -        mxe-$TARGET-w64-mingw32.shared-zlib
>> +        mxe-$TARGET-w64-mingw32.shared-zlib \
>> +        curl && \
>> +    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvEmulation.h \
>> +        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvemulation.h?format=raw" && \
>> +    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/WinHvPlatform.h \
>> +        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatform.h?format=raw" && \
>> +    curl -s -S -o /usr/lib/mxe/usr/x86_64-w64-mingw32.shared/include/winhvplatformdefs.h \
>> +        "https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winhvplatformdefs.h?format=raw"
> 
> My only concern here is whether sourceforge.net  URLs are reliable enough
> for CI, or whether we should instead be mirroring the w64 repo on
> git.qemu.org ?
> 
> I guess this will be a short term issue until a new mingw release is made
> and arrives i nthe distros, so we can try with the curl approach for
> now and change if it causes problems

That's my assumption, too. And since the containers are only rebuild
when something changed (as far as I understood), the download should
only occur quite rarely, so there is hopefully really not a big
dependency here.

> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Thanks!

 Thomas

[PATCH v2 6/7] configure: Allow automatic WHPX detection

[Thomas Huth]

The whpx variable is currently initialized to "no" which causes the WHPX
check to skip the detection unless the user specified --enable-whpx.
Since the detection code should be able to figure it out correctly, let's
initialized the variable to "" on MinGW-builds for proper auto-detection
instead.

Message-Id: <20200804170055.2851-11-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Stefan Weil <sw@weilnetz.de>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 configure | 1 +
 1 file changed, 1 insertion(+)

diff --git a/configure b/configure
index 01204ba0b5..2ca3c1d8e9 100755
--- a/configure
+++ b/configure
@@ -849,6 +849,7 @@ case $targetos in
 MINGW32*)
   mingw32="yes"
   hax="yes"
+  whpx=""
   vhost_user="no"
   audio_possible_drivers="dsound sdl"
   if check_include dsound.h; then
-- 
2.18.2

Re: [PATCH v2 6/7] configure: Allow automatic WHPX detection

[Daniel P. Berrangé]

On Sun, Aug 23, 2020 at 01:17:56PM +0200, Thomas Huth wrote:
> The whpx variable is currently initialized to "no" which causes the WHPX
> check to skip the detection unless the user specified --enable-whpx.
> Since the detection code should be able to figure it out correctly, let's
> initialized the variable to "" on MinGW-builds for proper auto-detection
> instead.
> 
> Message-Id: <20200804170055.2851-11-thuth@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Reviewed-by: Stefan Weil <sw@weilnetz.de>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  configure | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2 7/7] gitlab-ci: Add cross-compiling build tests

[Thomas Huth]

Now that we can use all our QEMU test containers in the gitlab-CI, we can
easily add some jobs that test cross-compilation for various architectures.
There is just only small ugliness: Since the shared runners on gitlab.com
are single-threaded, we have to split each compilation job into two parts
(--disable-user and --disable-system), and exclude some additional targets,
to avoid that the jobs are running too long and hitting the timeout of 1 h.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.d/crossbuilds.yml | 113 +++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml               |   1 +
 MAINTAINERS                  |   1 +
 3 files changed, 115 insertions(+)
 create mode 100644 .gitlab-ci.d/crossbuilds.yml

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
new file mode 100644
index 0000000000..4ec7226b5c
--- /dev/null
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -0,0 +1,113 @@
+
+.cross_system_build_job_template: &cross_system_build_job_definition
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-user
+        --target-list-exclude="aarch64-softmmu i386-softmmu microblaze-softmmu
+          mips-softmmu mipsel-softmmu mips64-softmmu ppc64-softmmu sh4-softmmu
+          xtensa-softmmu"
+    - make -j$(expr $(nproc) + 1) all check-build
+
+.cross_user_build_job_template: &cross_user_build_job_definition
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system
+    - make -j$(expr $(nproc) + 1) all check-build
+
+cross-armel-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-armel-cross
+
+cross-armel-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-armel-cross
+
+cross-armhf-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-armhf-cross
+
+cross-armhf-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-armhf-cross
+
+cross-arm64-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-arm64-cross
+
+cross-arm64-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-arm64-cross
+
+cross-mips-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-mips-cross
+
+cross-mips-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-mips-cross
+
+cross-mipsel-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-mipsel-cross
+
+cross-mipsel-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-mipsel-cross
+
+cross-mips64el-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-mips64el-cross
+
+cross-mips64el-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-mips64el-cross
+
+cross-ppc64el-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-ppc64el-cross
+
+cross-ppc64el-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-ppc64el-cross
+
+cross-s390x-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-s390x-cross
+
+cross-s390x-user:
+  <<: *cross_user_build_job_definition
+  variables:
+    IMAGE: debian-s390x-cross
+
+cross-win32-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-win32-cross
+
+cross-win64-system:
+  <<: *cross_system_build_job_definition
+  variables:
+    IMAGE: debian-win64-cross
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b7967b9a13..8a4b67ecca 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -18,6 +18,7 @@ include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
+  - local: '/.gitlab-ci.d/crossbuilds.yml'
 
 .native_build_job_template: &native_build_job_definition
   stage: build
diff --git a/MAINTAINERS b/MAINTAINERS
index 0886eb3d2b..2731f7a594 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3067,6 +3067,7 @@ M: Alex Bennée <alex.bennee@linaro.org>
 R: Wainer dos Santos Moschetta <wainersm@redhat.com>
 S: Maintained
 F: .gitlab-ci.yml
+F: .gitlab-ci.d/crossbuilds.yml
 
 Guest Test Compilation Support
 M: Alex Bennée <alex.bennee@linaro.org>
-- 
2.18.2

Re: [PATCH v2 7/7] gitlab-ci: Add cross-compiling build tests

[Daniel P. Berrangé]

On Sun, Aug 23, 2020 at 01:17:57PM +0200, Thomas Huth wrote:
> Now that we can use all our QEMU test containers in the gitlab-CI, we can
> easily add some jobs that test cross-compilation for various architectures.
> There is just only small ugliness: Since the shared runners on gitlab.com
> are single-threaded, we have to split each compilation job into two parts
> (--disable-user and --disable-system), and exclude some additional targets,
> to avoid that the jobs are running too long and hitting the timeout of 1 h.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.d/crossbuilds.yml | 113 +++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml               |   1 +
>  MAINTAINERS                  |   1 +
>  3 files changed, 115 insertions(+)
>  create mode 100644 .gitlab-ci.d/crossbuilds.yml

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

> 
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> new file mode 100644
> index 0000000000..4ec7226b5c
> --- /dev/null
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -0,0 +1,113 @@
> +
> +.cross_system_build_job_template: &cross_system_build_job_definition
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  script:
> +    - mkdir build
> +    - cd build
> +    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
> +      ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-user
> +        --target-list-exclude="aarch64-softmmu i386-softmmu microblaze-softmmu
> +          mips-softmmu mipsel-softmmu mips64-softmmu ppc64-softmmu sh4-softmmu
> +          xtensa-softmmu"

What does this leave enabled ?  Would it be shorter to just say
--target-list="...explicit targets we want..." ?  It would be clearer
to review at least.

> +    - make -j$(expr $(nproc) + 1) all check-build

None the less, it is functionally fine so

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 7/7] gitlab-ci: Add cross-compiling build tests

[Thomas Huth]

On 01/09/2020 18.27, Daniel P. Berrangé wrote:
> On Sun, Aug 23, 2020 at 01:17:57PM +0200, Thomas Huth wrote:
>> Now that we can use all our QEMU test containers in the gitlab-CI, we can
>> easily add some jobs that test cross-compilation for various architectures.
>> There is just only small ugliness: Since the shared runners on gitlab.com
>> are single-threaded, we have to split each compilation job into two parts
>> (--disable-user and --disable-system), and exclude some additional targets,
>> to avoid that the jobs are running too long and hitting the timeout of 1 h.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  .gitlab-ci.d/crossbuilds.yml | 113 +++++++++++++++++++++++++++++++++++
>>  .gitlab-ci.yml               |   1 +
>>  MAINTAINERS                  |   1 +
>>  3 files changed, 115 insertions(+)
>>  create mode 100644 .gitlab-ci.d/crossbuilds.yml
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> 
>>
>> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
>> new file mode 100644
>> index 0000000000..4ec7226b5c
>> --- /dev/null
>> +++ b/.gitlab-ci.d/crossbuilds.yml
>> @@ -0,0 +1,113 @@
>> +
>> +.cross_system_build_job_template: &cross_system_build_job_definition
>> +  stage: build
>> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>> +  script:
>> +    - mkdir build
>> +    - cd build
>> +    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
>> +      ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-user
>> +        --target-list-exclude="aarch64-softmmu i386-softmmu microblaze-softmmu
>> +          mips-softmmu mipsel-softmmu mips64-softmmu ppc64-softmmu sh4-softmmu
>> +          xtensa-softmmu"
> 
> What does this leave enabled ?  Would it be shorter to just say
> --target-list="...explicit targets we want..." ?  It would be clearer
> to review at least.

I basically excluded all targets that have a second compile test
coverage, e.g. i386-softmmu code is mostly also covered by
x86_64-softmmu, sh4-softmmu is covered by sh4eb-softmmu etc.

The --target-list-exclude also comes in handy when new targets are added
to QEMU - you then don't have to remember to add the new targets here,
they are picked up automatically.

>> +    - make -j$(expr $(nproc) + 1) all check-build
> 
> None the less, it is functionally fine so
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Thanks!

 Thomas

Re: [PATCH v2 0/7] Run cross-compilation build tests in the gitlab-CI

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1751 bytes --]

Build system parts

Acked-by: Paolo Bonzini <pbonzini@redhat.com>

Il dom 23 ago 2020, 13:18 Thomas Huth <thuth@redhat.com> ha scritto:

> Now that we can use all our QEMU build containers in the gitlab-CI,
> we can also run the cross-compilation jobs there. Of course, some
> problems have to be fixed first, so this is taken care of in the first
> four patches.
>
> The following two patches make sure that we can also enable WHPX builds
> with
> our debian-win64-cross container, so that we can compile-test this
> accelerator
> code now, too.
>
> The last patch then finally enables the cross-compilation jobs in the CI.
>
> v2:
>  - Dropped patches that are not necessary anymore
>  - Added the first two patches to fix problems with the new meson build
>    system
>
> Thomas Huth (7):
>   configure: Add system = 'linux' for meson when cross-compiling
>   tests/docker: Install python3-setuptools in the debian9-mxe containers
>   tests/Makefile: test-image-locking needs CONFIG_POSIX
>   tests/Makefile: test-replication needs CONFIG_POSIX
>   dockerfiles/debian-win64-cross: Download WHPX MinGW headers
>   configure: Allow automatic WHPX detection
>   gitlab-ci: Add cross-compiling build tests
>
>  .gitlab-ci.d/crossbuilds.yml                  | 113 ++++++++++++++++++
>  .gitlab-ci.yml                                |   1 +
>  MAINTAINERS                                   |   1 +
>  configure                                     |   4 +
>  tests/Makefile.include                        |   6 +-
>  .../dockerfiles/debian-win64-cross.docker     |   9 +-
>  tests/docker/dockerfiles/debian9-mxe.docker   |   2 +-
>  7 files changed, 133 insertions(+), 3 deletions(-)
>  create mode 100644 .gitlab-ci.d/crossbuilds.yml
>
> --
> 2.18.2
>
>

[-- Attachment #2: Type: text/html, Size: 2337 bytes --]

Re: [PATCH v2 0/7] Run cross-compilation build tests in the gitlab-CI

[Daniel P. Berrangé]

On Sun, Aug 23, 2020 at 01:17:50PM +0200, Thomas Huth wrote:
> Now that we can use all our QEMU build containers in the gitlab-CI,
> we can also run the cross-compilation jobs there. Of course, some
> problems have to be fixed first, so this is taken care of in the first
> four patches.
> 
> The following two patches make sure that we can also enable WHPX builds with
> our debian-win64-cross container, so that we can compile-test this accelerator
> code now, too.
> 
> The last patch then finally enables the cross-compilation jobs in the CI.
> 
> v2:
>  - Dropped patches that are not necessary anymore
>  - Added the first two patches to fix problems with the new meson build
>    system
> 
> Thomas Huth (7):
>   configure: Add system = 'linux' for meson when cross-compiling
>   tests/docker: Install python3-setuptools in the debian9-mxe containers
>   tests/Makefile: test-image-locking needs CONFIG_POSIX
>   tests/Makefile: test-replication needs CONFIG_POSIX
>   dockerfiles/debian-win64-cross: Download WHPX MinGW headers
>   configure: Allow automatic WHPX detection
>   gitlab-ci: Add cross-compiling build tests
> 
>  .gitlab-ci.d/crossbuilds.yml                  | 113 ++++++++++++++++++
>  .gitlab-ci.yml                                |   1 +
>  MAINTAINERS                                   |   1 +
>  configure                                     |   4 +
>  tests/Makefile.include                        |   6 +-
>  .../dockerfiles/debian-win64-cross.docker     |   9 +-
>  tests/docker/dockerfiles/debian9-mxe.docker   |   2 +-
>  7 files changed, 133 insertions(+), 3 deletions(-)
>  create mode 100644 .gitlab-ci.d/crossbuilds.yml

You seem to have not sent the mail for 

  [PATCH 8/7] delete obsolete shippable config

;-P


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 0/7] Run cross-compilation build tests in the gitlab-CI

[Thomas Huth]

On 01/09/2020 18.29, Daniel P. Berrangé wrote:
> On Sun, Aug 23, 2020 at 01:17:50PM +0200, Thomas Huth wrote:
>> Now that we can use all our QEMU build containers in the gitlab-CI,
>> we can also run the cross-compilation jobs there. Of course, some
>> problems have to be fixed first, so this is taken care of in the first
>> four patches.
>>
>> The following two patches make sure that we can also enable WHPX builds with
>> our debian-win64-cross container, so that we can compile-test this accelerator
>> code now, too.
>>
>> The last patch then finally enables the cross-compilation jobs in the CI.
>>
>> v2:
>>  - Dropped patches that are not necessary anymore
>>  - Added the first two patches to fix problems with the new meson build
>>    system
>>
>> Thomas Huth (7):
>>   configure: Add system = 'linux' for meson when cross-compiling
>>   tests/docker: Install python3-setuptools in the debian9-mxe containers
>>   tests/Makefile: test-image-locking needs CONFIG_POSIX
>>   tests/Makefile: test-replication needs CONFIG_POSIX
>>   dockerfiles/debian-win64-cross: Download WHPX MinGW headers
>>   configure: Allow automatic WHPX detection
>>   gitlab-ci: Add cross-compiling build tests
>>
>>  .gitlab-ci.d/crossbuilds.yml                  | 113 ++++++++++++++++++
>>  .gitlab-ci.yml                                |   1 +
>>  MAINTAINERS                                   |   1 +
>>  configure                                     |   4 +
>>  tests/Makefile.include                        |   6 +-
>>  .../dockerfiles/debian-win64-cross.docker     |   9 +-
>>  tests/docker/dockerfiles/debian9-mxe.docker   |   2 +-
>>  7 files changed, 133 insertions(+), 3 deletions(-)
>>  create mode 100644 .gitlab-ci.d/crossbuilds.yml
> 
> You seem to have not sent the mail for 
> 
>   [PATCH 8/7] delete obsolete shippable config
> 
> ;-P

Yeah, I wanted to chat with Alex about that first once the gitlab stuff
has been mreged...

 Thomas