[PATCH 0/3] Gitlab-CI improvements

[PATCH 0/3] Gitlab-CI improvements

[Thomas Huth]

Here are three patches for some small issues that I noticed in our
gitlab-CI files recently...

Thomas Huth (3):
  gitlab-ci: Merge "build-disabled" with "build-without-default-features"
  gitlab-ci: Remove superfluous "dnf install" statement
  gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

 .gitlab-ci.d/buildtest.yml      | 99 +++++----------------------------
 .gitlab-ci.d/custom-runners.yml | 12 ++--
 docs/devel/ci.rst               | 13 +++++
 3 files changed, 32 insertions(+), 92 deletions(-)

-- 
2.27.0

[PATCH 1/3] gitlab-ci: Merge "build-disabled" with "build-without-default-features"

[Thomas Huth]

Both jobs are testing more or less the same thing (building QEMU with
features disabled), so we are wasting precious CI cycles here by doing
this twice. Merge the jobs by using --without-default-features by default
and just adding some additional --disable-... switches which are not
covered by the generic switch (yet). And while we're at it, also test
compilation with "--disable-fdt" (which forces us to change the list
of targets in this job, though, since some targets do not work without
fdt).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.d/buildtest.yml | 97 +++++---------------------------------
 1 file changed, 13 insertions(+), 84 deletions(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 903ee65f32..f390f98044 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -202,85 +202,6 @@ acceptance-system-opensuse:
     MAKE_CHECK_ARGS: check-acceptance
 
 
-build-disabled:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-fedora-container
-  variables:
-    IMAGE: fedora
-    CONFIGURE_ARGS:
-      --disable-attr
-      --disable-auth-pam
-      --disable-avx2
-      --disable-bochs
-      --disable-brlapi
-      --disable-bzip2
-      --disable-cap-ng
-      --disable-capstone
-      --disable-cloop
-      --disable-coroutine-pool
-      --disable-curl
-      --disable-curses
-      --disable-dmg
-      --disable-docs
-      --disable-gcrypt
-      --disable-glusterfs
-      --disable-gnutls
-      --disable-gtk
-      --disable-guest-agent
-      --disable-iconv
-      --disable-keyring
-      --disable-kvm
-      --disable-libiscsi
-      --disable-libpmem
-      --disable-libssh
-      --disable-libudev
-      --disable-libusb
-      --disable-libxml2
-      --disable-linux-aio
-      --disable-live-block-migration
-      --disable-lzo
-      --disable-malloc-trim
-      --disable-mpath
-      --disable-nettle
-      --disable-numa
-      --disable-opengl
-      --disable-parallels
-      --disable-pie
-      --disable-qcow1
-      --disable-qed
-      --disable-qom-cast-debug
-      --disable-rbd
-      --disable-rdma
-      --disable-replication
-      --disable-sdl
-      --disable-seccomp
-      --disable-slirp
-      --disable-smartcard
-      --disable-snappy
-      --disable-sparse
-      --disable-spice
-      --disable-strip
-      --disable-tpm
-      --disable-usb-redir
-      --disable-vdi
-      --disable-vhost-crypto
-      --disable-vhost-net
-      --disable-vhost-scsi
-      --disable-vhost-kernel
-      --disable-vhost-user
-      --disable-vhost-vdpa
-      --disable-vhost-vsock
-      --disable-virglrenderer
-      --disable-vnc
-      --disable-vte
-      --disable-vvfat
-      --disable-xen
-      --disable-zstd
-    TARGETS: arm-softmmu i386-softmmu ppc64-softmmu mips64-softmmu
-      s390x-softmmu i386-linux-user
-    MAKE_CHECK_ARGS: check-qtest SPEED=slow
-
 # This jobs explicitly disable TCG (--disable-tcg), KVM is detected by
 # the configure script. The container doesn't contain Xen headers so
 # Xen accelerator is not detected / selected. As result it build the
@@ -649,12 +570,20 @@ build-without-default-devices:
 build-without-default-features:
   extends: .native_build_job_template
   needs:
-    job: amd64-debian-container
+    job: amd64-fedora-container
   variables:
-    IMAGE: debian-amd64
-    CONFIGURE_ARGS: --without-default-features --disable-user
-        --target-list-exclude=arm-softmmu,i386-softmmu,mipsel-softmmu,mips64-softmmu,ppc-softmmu
-    MAKE_CHECK_ARGS: check-unit
+    IMAGE: fedora
+    CONFIGURE_ARGS:
+      --without-default-features
+      --disable-capstone
+      --disable-fdt
+      --disable-pie
+      --disable-qom-cast-debug
+      --disable-slirp
+      --disable-strip
+    TARGETS: avr-softmmu i386-softmmu mips64-softmmu s390x-softmmu sh4-softmmu
+      sparc64-softmmu hexagon-linux-user i386-linux-user s390x-linux-user
+    MAKE_CHECK_ARGS: check-unit check-qtest SPEED=slow
 
 build-libvhost-user:
   stage: build
-- 
2.27.0

Re: [PATCH 1/3] gitlab-ci: Merge "build-disabled" with "build-without-default-features"

[Willian Rampazzo]

On Fri, Jul 30, 2021 at 11:38 AM Thomas Huth <thuth@redhat.com> wrote:
>
> Both jobs are testing more or less the same thing (building QEMU with
> features disabled), so we are wasting precious CI cycles here by doing
> this twice. Merge the jobs by using --without-default-features by default
> and just adding some additional --disable-... switches which are not
> covered by the generic switch (yet). And while we're at it, also test
> compilation with "--disable-fdt" (which forces us to change the list
> of targets in this job, though, since some targets do not work without
> fdt).
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.d/buildtest.yml | 97 +++++---------------------------------
>  1 file changed, 13 insertions(+), 84 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

[PATCH 2/3] gitlab-ci: Remove superfluous "dnf install" statement

[Thomas Huth]

The container already features meson and ninja, so there is no need
to try to install it with dnf again.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.d/buildtest.yml | 2 --
 1 file changed, 2 deletions(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index f390f98044..38f08452f1 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -590,8 +590,6 @@ build-libvhost-user:
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   needs:
     job: amd64-fedora-container
-  before_script:
-    - dnf install -y meson ninja-build
   script:
     - mkdir subprojects/libvhost-user/build
     - cd subprojects/libvhost-user/build
-- 
2.27.0

Re: [PATCH 2/3] gitlab-ci: Remove superfluous "dnf install" statement

[Willian Rampazzo]

On Fri, Jul 30, 2021 at 11:38 AM Thomas Huth <thuth@redhat.com> wrote:
>
> The container already features meson and ninja, so there is no need
> to try to install it with dnf again.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.d/buildtest.yml | 2 --
>  1 file changed, 2 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

[PATCH 3/3] gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

[Thomas Huth]

The patch that recently introduced the S390X_RUNNER_AVAILABLE variable
in custom-runners.yml missed that the bottom half of the file is rather
about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to
AARCH64_RUNNER_AVAILABLE in those jobs.

Finally mention both variables in our CI documentation, too.

Fixes: c5dd0f0342 ("Improve rules for the staging branch")
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.d/custom-runners.yml | 12 ++++++------
 docs/devel/ci.rst               | 13 +++++++++++++
 2 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index 564b94565d..0d3e4a7b4b 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -137,7 +137,7 @@ ubuntu-20.04-aarch64-all-linux-static:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
  # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
@@ -157,7 +157,7 @@ ubuntu-20.04-aarch64-all:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -174,7 +174,7 @@ ubuntu-20.04-aarch64-alldbg:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -193,7 +193,7 @@ ubuntu-20.04-aarch64-clang:
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
    when: manual
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
    when: manual
  script:
  - mkdir build
@@ -211,7 +211,7 @@ ubuntu-20.04-aarch64-tci:
  - aarch64
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
  script:
  - mkdir build
  - cd build
@@ -228,7 +228,7 @@ ubuntu-20.04-aarch64-notcg:
  rules:
  - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
    when: manual
- - if: "$S390X_RUNNER_AVAILABLE"
+ - if: "$AARCH64_RUNNER_AVAILABLE"
    when: manual
  script:
  - mkdir build
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 205572510c..558327457c 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -48,6 +48,19 @@ these artifacts are not already cached, downloading them make the jobs
 reach the timeout limit). Set this variable to have the tests using the
 Avocado framework run automatically.
 
+AARCH64_RUNNER_AVAILABLE
+~~~~~~~~~~~~~~~~~~~~~~~~
+If you've got access to an aarch64 host that can be used as a gitlab-CI
+runner, you can set this variable to enable the tests that require this
+kind of host. The runner should be tagged with "aarch64".
+
+S390X_RUNNER_AVAILABLE
+~~~~~~~~~~~~~~~~~~~~~~
+If you've got access to an IBM Z host that can be used as a gitlab-CI
+runner, you can set this variable to enable the tests that require this
+kind of host. The runner should be tagged with "s390x".
+
+
 Jobs on Custom Runners
 ======================
 
-- 
2.27.0

Re: [PATCH 3/3] gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

[Philippe Mathieu-Daudé]

On 7/30/21 4:38 PM, Thomas Huth wrote:
> The patch that recently introduced the S390X_RUNNER_AVAILABLE variable
> in custom-runners.yml missed that the bottom half of the file is rather
> about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to
> AARCH64_RUNNER_AVAILABLE in those jobs.

Oops.

> Finally mention both variables in our CI documentation, too.

Thanks.

Maybe "Fix ${arch}_RUNNER_AVAILABLE" as subject.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> Fixes: c5dd0f0342 ("Improve rules for the staging branch")
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 12 ++++++------
>  docs/devel/ci.rst               | 13 +++++++++++++
>  2 files changed, 19 insertions(+), 6 deletions(-)
> 
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> index 564b94565d..0d3e4a7b4b 100644
> --- a/.gitlab-ci.d/custom-runners.yml
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -137,7 +137,7 @@ ubuntu-20.04-aarch64-all-linux-static:
>   - aarch64
>   rules:
>   - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
> - - if: "$S390X_RUNNER_AVAILABLE"
> + - if: "$AARCH64_RUNNER_AVAILABLE"
>   script:
>   # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763
>   # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages
> @@ -157,7 +157,7 @@ ubuntu-20.04-aarch64-all:
>   - aarch64
>   rules:
>   - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
> - - if: "$S390X_RUNNER_AVAILABLE"
> + - if: "$AARCH64_RUNNER_AVAILABLE"
>   script:
>   - mkdir build
>   - cd build
> @@ -174,7 +174,7 @@ ubuntu-20.04-aarch64-alldbg:
>   - aarch64
>   rules:
>   - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
> - - if: "$S390X_RUNNER_AVAILABLE"
> + - if: "$AARCH64_RUNNER_AVAILABLE"
>   script:
>   - mkdir build
>   - cd build
> @@ -193,7 +193,7 @@ ubuntu-20.04-aarch64-clang:
>   rules:
>   - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
>     when: manual
> - - if: "$S390X_RUNNER_AVAILABLE"
> + - if: "$AARCH64_RUNNER_AVAILABLE"
>     when: manual
>   script:
>   - mkdir build
> @@ -211,7 +211,7 @@ ubuntu-20.04-aarch64-tci:
>   - aarch64
>   rules:
>   - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
> - - if: "$S390X_RUNNER_AVAILABLE"
> + - if: "$AARCH64_RUNNER_AVAILABLE"
>   script:
>   - mkdir build
>   - cd build
> @@ -228,7 +228,7 @@ ubuntu-20.04-aarch64-notcg:
>   rules:
>   - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH =~ /^staging/'
>     when: manual
> - - if: "$S390X_RUNNER_AVAILABLE"
> + - if: "$AARCH64_RUNNER_AVAILABLE"
>     when: manual
>   script:
>   - mkdir build
> diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
> index 205572510c..558327457c 100644
> --- a/docs/devel/ci.rst
> +++ b/docs/devel/ci.rst
> @@ -48,6 +48,19 @@ these artifacts are not already cached, downloading them make the jobs
>  reach the timeout limit). Set this variable to have the tests using the
>  Avocado framework run automatically.
>  
> +AARCH64_RUNNER_AVAILABLE
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +If you've got access to an aarch64 host that can be used as a gitlab-CI
> +runner, you can set this variable to enable the tests that require this
> +kind of host. The runner should be tagged with "aarch64".
> +
> +S390X_RUNNER_AVAILABLE
> +~~~~~~~~~~~~~~~~~~~~~~
> +If you've got access to an IBM Z host that can be used as a gitlab-CI
> +runner, you can set this variable to enable the tests that require this
> +kind of host. The runner should be tagged with "s390x".
> +
> +
>  Jobs on Custom Runners
>  ======================
>  
> 

Re: [PATCH 3/3] gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them

[Willian Rampazzo]

On Fri, Jul 30, 2021 at 11:38 AM Thomas Huth <thuth@redhat.com> wrote:
>
> The patch that recently introduced the S390X_RUNNER_AVAILABLE variable
> in custom-runners.yml missed that the bottom half of the file is rather
> about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to
> AARCH64_RUNNER_AVAILABLE in those jobs.
>
> Finally mention both variables in our CI documentation, too.
>
> Fixes: c5dd0f0342 ("Improve rules for the staging branch")
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.d/custom-runners.yml | 12 ++++++------
>  docs/devel/ci.rst               | 13 +++++++++++++
>  2 files changed, 19 insertions(+), 6 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH 0/3] Gitlab-CI improvements

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> Here are three patches for some small issues that I noticed in our
> gitlab-CI files recently...

Queued to for-6.1/misc-fixes-for-rc2, thanks.

>
> Thomas Huth (3):
>   gitlab-ci: Merge "build-disabled" with "build-without-default-features"
>   gitlab-ci: Remove superfluous "dnf install" statement
>   gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them
>
>  .gitlab-ci.d/buildtest.yml      | 99 +++++----------------------------
>  .gitlab-ci.d/custom-runners.yml | 12 ++--
>  docs/devel/ci.rst               | 13 +++++
>  3 files changed, 32 insertions(+), 92 deletions(-)


-- 
Alex Bennée