[PATCH v1 0/7] testing/next pre-PR (docker, gitlab, tcg)

[PATCH v1 0/7] testing/next pre-PR (docker, gitlab, tcg)

[Alex Bennée]

Hi,

Another day, another testing/next series. Mostly this contains a few
hot fixes for docker breakages and other stuff that I picked off the
list. There is a late breaking patch to try and make the aarch64 CI is
little less twitchy. We are still seeing hanging tests on the s390x
box but so far have been unable to track down why it's hanging and
more importantly why we are not seeing the runner kill errant tasks.

As there are hot fixes I'd like to turn around a PR from this soon so
any review of the following is welcome:

 - .gitlab: use less aggressive nproc on our aarch64/32 runners
 - tests/docker: fix the IMAGE for build invocation

Alex Bennée (2):
  tests/docker: fix the IMAGE for build invocation
  .gitlab: use less aggressive nproc on our aarch64/32 runners

Daniel P. Berrangé (1):
  gitlab: compare CIRRUS_nn vars against 'null' not ""

Paolo Bonzini (1):
  tests/tcg: disable xtensa-linux-user again

Richard Henderson (2):
  test/tcg/arm: Use -mfloat-abi=soft for test-armv6m-undef
  tests/tcg/i386: Use explicit suffix on fist insns

Thomas Huth (1):
  gitlab-ci: Fix the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs

 configure                                     |  5 +++-
 tests/tcg/i386/test-i386-fp-exceptions.c      | 24 +++++++++----------
 tests/tcg/i386/test-i386.c                    |  2 +-
 .gitlab-ci.d/base.yml                         |  2 +-
 .gitlab-ci.d/buildtest.yml                    | 22 ++++++++---------
 .../custom-runners/ubuntu-20.04-aarch32.yml   |  4 ++--
 .../custom-runners/ubuntu-20.04-aarch64.yml   | 24 +++++++++----------
 tests/docker/Makefile.include                 |  2 +-
 tests/tcg/arm/Makefile.softmmu-target         |  2 +-
 9 files changed, 44 insertions(+), 43 deletions(-)

-- 
2.30.2

[PATCH v1 1/7] test/tcg/arm: Use -mfloat-abi=soft for test-armv6m-undef

[Alex Bennée]

From: Richard Henderson <richard.henderson@linaro.org>

GCC11 from crossbuild-essential-armhf from ubuntu 22.04 errors:
cc1: error: ‘-mfloat-abi=hard’: selected architecture lacks an FPU

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220604032713.174976-1-richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/arm/Makefile.softmmu-target | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/tcg/arm/Makefile.softmmu-target b/tests/tcg/arm/Makefile.softmmu-target
index 3fe237ba39..7df88ddea8 100644
--- a/tests/tcg/arm/Makefile.softmmu-target
+++ b/tests/tcg/arm/Makefile.softmmu-target
@@ -20,7 +20,7 @@ LDFLAGS+=-nostdlib -N -static
 
 # Specific Test Rules
 
-test-armv6m-undef: EXTRA_CFLAGS+=-mcpu=cortex-m0
+test-armv6m-undef: EXTRA_CFLAGS+=-mcpu=cortex-m0 -mfloat-abi=soft
 
 run-test-armv6m-undef: QEMU_OPTS+=-semihosting -M microbit -kernel
 run-plugin-test-armv6m-undef-%: QEMU_OPTS+=-semihosting -M microbit -kernel
-- 
2.30.2

[PATCH v1 2/7] tests/tcg/i386: Use explicit suffix on fist insns

[Alex Bennée]

From: Richard Henderson <richard.henderson@linaro.org>

Fixes a number of assembler warnings of the form:

test-i386.c: Assembler messages:
test-i386.c:869: Warning: no instruction mnemonic suffix given
  and no register operands; using default for `fist'

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220527171143.168276-1-richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/i386/test-i386-fp-exceptions.c | 24 ++++++++++++------------
 tests/tcg/i386/test-i386.c               |  2 +-
 2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/tests/tcg/i386/test-i386-fp-exceptions.c b/tests/tcg/i386/test-i386-fp-exceptions.c
index dfb7117c17..d445f13c33 100644
--- a/tests/tcg/i386/test-i386-fp-exceptions.c
+++ b/tests/tcg/i386/test-i386-fp-exceptions.c
@@ -423,35 +423,35 @@ int main(void)
     }
 
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fistp %0" : "=m" (res_16) : "t" (1.5L) : "st");
+    __asm__ volatile ("fistps %0" : "=m" (res_16) : "t" (1.5L) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != PE) {
         printf("FAIL: fistp inexact\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fistp %0" : "=m" (res_16) : "t" (32767.5L) : "st");
+    __asm__ volatile ("fistps %0" : "=m" (res_16) : "t" (32767.5L) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
         printf("FAIL: fistp 32767.5\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fistp %0" : "=m" (res_16) : "t" (-32768.51L) : "st");
+    __asm__ volatile ("fistps %0" : "=m" (res_16) : "t" (-32768.51L) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
         printf("FAIL: fistp -32768.51\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fistp %0" : "=m" (res_16) : "t" (ld_nan) : "st");
+    __asm__ volatile ("fistps %0" : "=m" (res_16) : "t" (ld_nan) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
         printf("FAIL: fistp nan\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fistp %0" : "=m" (res_16) : "t" (ld_invalid_1.ld) :
+    __asm__ volatile ("fistps %0" : "=m" (res_16) : "t" (ld_invalid_1.ld) :
                       "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
@@ -538,49 +538,49 @@ int main(void)
     }
 
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fisttp %0" : "=m" (res_16) : "t" (1.5L) : "st");
+    __asm__ volatile ("fisttps %0" : "=m" (res_16) : "t" (1.5L) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != PE) {
         printf("FAIL: fisttp inexact\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fisttp %0" : "=m" (res_16) : "t" (32768.0L) : "st");
+    __asm__ volatile ("fisttps %0" : "=m" (res_16) : "t" (32768.0L) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
         printf("FAIL: fisttp 32768\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fisttp %0" : "=m" (res_16) : "t" (32768.5L) : "st");
+    __asm__ volatile ("fisttps %0" : "=m" (res_16) : "t" (32768.5L) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
         printf("FAIL: fisttp 32768.5\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fisttp %0" : "=m" (res_16) : "t" (-32769.0L) : "st");
+    __asm__ volatile ("fisttps %0" : "=m" (res_16) : "t" (-32769.0L) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
         printf("FAIL: fisttp -32769\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fisttp %0" : "=m" (res_16) : "t" (-32769.5L) : "st");
+    __asm__ volatile ("fisttps %0" : "=m" (res_16) : "t" (-32769.5L) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
         printf("FAIL: fisttp -32769.5\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fisttp %0" : "=m" (res_16) : "t" (ld_nan) : "st");
+    __asm__ volatile ("fisttps %0" : "=m" (res_16) : "t" (ld_nan) : "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
         printf("FAIL: fisttp nan\n");
         ret = 1;
     }
     __asm__ volatile ("fnclex");
-    __asm__ volatile ("fisttp %0" : "=m" (res_16) : "t" (ld_invalid_1.ld) :
+    __asm__ volatile ("fisttps %0" : "=m" (res_16) : "t" (ld_invalid_1.ld) :
                       "st");
     __asm__ volatile ("fnstsw" : "=a" (sw));
     if ((sw & EXC) != IE) {
diff --git a/tests/tcg/i386/test-i386.c b/tests/tcg/i386/test-i386.c
index 18d5609665..ac8d5a3c1f 100644
--- a/tests/tcg/i386/test-i386.c
+++ b/tests/tcg/i386/test-i386.c
@@ -866,7 +866,7 @@ void test_fcvt(double a)
         uint16_t val16;
         val16 = (fpuc & ~0x0c00) | (i << 10);
         asm volatile ("fldcw %0" : : "m" (val16));
-        asm volatile ("fist %0" : "=m" (wa) : "t" (a));
+        asm volatile ("fists %0" : "=m" (wa) : "t" (a));
         asm volatile ("fistl %0" : "=m" (ia) : "t" (a));
         asm volatile ("fistpll %0" : "=m" (lla) : "t" (a) : "st");
         asm volatile ("frndint ; fstl %0" : "=m" (ra) : "t" (a));
-- 
2.30.2

[PATCH v1 3/7] gitlab-ci: Fix the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs

[Alex Bennée]

From: Thomas Huth <thuth@redhat.com>

The job definitions recently got a second "variables:" section by
accident and thus are failing now if one tries to run them. Merge
the two sections into one again to fix the issue.

And while we're at it, bump the timeout here (70 minutes are currently
not enough for the aarch64 job). The jobs are marked as manual anyway,
so if the user starts them, they want to see their result for sure and
then it's annoying if the job timeouts too early.

Fixes: e312d1fdbb ("gitlab: convert build/container jobs to .base_job_template")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220603124809.70794-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/buildtest.yml | 22 ++++++++++------------
 1 file changed, 10 insertions(+), 12 deletions(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 544385f5be..cb7cad44b5 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -357,16 +357,15 @@ build-cfi-aarch64:
       --enable-safe-stack --enable-slirp=git
     TARGETS: aarch64-softmmu
     MAKE_CHECK_ARGS: check-build
-  timeout: 70m
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
     # skipped until the situation has been solved.
     QEMU_JOB_SKIPPED: 1
+  timeout: 90m
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
 
 check-cfi-aarch64:
   extends: .native_test_job_template
@@ -398,16 +397,15 @@ build-cfi-ppc64-s390x:
       --enable-safe-stack --enable-slirp=git
     TARGETS: ppc64-softmmu s390x-softmmu
     MAKE_CHECK_ARGS: check-build
-  timeout: 70m
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
     # skipped until the situation has been solved.
     QEMU_JOB_SKIPPED: 1
+  timeout: 80m
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
 
 check-cfi-ppc64-s390x:
   extends: .native_test_job_template
-- 
2.30.2

[PATCH v1 4/7] tests/docker: fix the IMAGE for build invocation

[Alex Bennée]

We inadvertently broke the ability to run local builds when the code
was re-factored. The result was the run stanza failing to find the
docker image with it's qemu/ prefix.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Fixes: d39eaa2266 ("tests/docker: simplify docker-TEST@IMAGE targets")
---
 tests/docker/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index e68f91b853..ef4518d9eb 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -171,7 +171,7 @@ DOCKER_TESTS := $(if $(TESTS), $(filter $(TESTS), $(__TESTS)), $(__TESTS))
 $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS), \
 		$(eval .PHONY: docker-$t@$i) \
-		$(eval docker-$t@$i: docker-image-$i; @$(MAKE) docker-run TEST=$t IMAGE=$i) \
+		$(eval docker-$t@$i: docker-image-$i; @$(MAKE) docker-run TEST=$t IMAGE=qemu/$i) \
 	) \
 	$(foreach t,$(DOCKER_TESTS), \
 		$(eval docker-all-tests: docker-$t@$i) \
-- 
2.30.2

[PATCH v1 5/7] tests/tcg: disable xtensa-linux-user again

[Alex Bennée]

From: Paolo Bonzini <pbonzini@redhat.com>

The move from tests/tcg/configure.sh started enabling the container image
for xtensa-linux-user, which fails because the compiler does not have
the full set of headers.  The cause is the "xtensa*-softmmu)" case
in tests/tcg/configure.sh which became just "xtensa*)" in the new
probe_target_compiler shell function.  Look out for xtensa*-linux-user
and do not configure it.

Reported-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220608135727.1341946-1-pbonzini@redhat.com>
Fixes: cd362defbb ("tests/tcg: merge configure.sh back into main configure script")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 configure | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index e69537c756..4b12a8094c 100755
--- a/configure
+++ b/configure
@@ -2024,7 +2024,6 @@ probe_target_compiler() {
         container_cross_prefix=x86_64-linux-gnu-
         ;;
       xtensa*)
-        # FIXME: xtensa-linux-user?
         container_hosts=x86_64
         container_image=debian-xtensa-cross
 
@@ -2481,6 +2480,10 @@ for target in $target_list; do
   echo "# Automatically generated by configure - do not modify" > $config_target_mak
   echo "TARGET_NAME=$arch" >> $config_target_mak
   case $target in
+    xtensa*-linux-user)
+      # the toolchain is not complete with headers, only build softmmu tests
+      continue
+      ;;
     *-softmmu)
       test -f $source_path/tests/tcg/$arch/Makefile.softmmu-target || continue
       qemu="qemu-system-$arch"
-- 
2.30.2

[PATCH v1 6/7] gitlab: compare CIRRUS_nn vars against 'null' not ""

[Alex Bennée]

From: Daniel P. Berrangé <berrange@redhat.com>

The GitLab variable comparisons don't have shell like semantics where
an unset variable compares equal to empty string. We need to explicitly
test against 'null' to detect an unset variable.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220608160651.248781-1-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/base.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index f334f3ded7..69b36c148a 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -13,7 +13,7 @@
     #############################################################
 
     # Cirrus jobs can't run unless the creds / target repo are set
-    - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
+    - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
       when: never
 
     # Publishing jobs should only run on the default branch in upstream
-- 
2.30.2

[PATCH v1 7/7] .gitlab: use less aggressive nproc on our aarch64/32 runners

[Alex Bennée]

Running on all 80 cores of our aarch64 runner does occasionally
trigger a race condition which fails the build. However the CI system
is not the time and place to play with much heisenbugs so turn down
the nproc to "only" use 40 cores in the build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .../custom-runners/ubuntu-20.04-aarch32.yml   |  4 ++--
 .../custom-runners/ubuntu-20.04-aarch64.yml   | 24 +++++++++----------
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
index 47856ac53c..1998460d06 100644
--- a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
+++ b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
@@ -19,5 +19,5 @@ ubuntu-20.04-aarch32-all:
  - mkdir build
  - cd build
  - ../configure --cross-prefix=arm-linux-gnueabihf-
- - make --output-sync -j`nproc`
- - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc --ignore=40`
+ - make --output-sync -j`nproc --ignore=40` check V=1
diff --git a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml
index 951e490db1..65718a188a 100644
--- a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml
+++ b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml
@@ -17,9 +17,9 @@ ubuntu-20.04-aarch64-all-linux-static:
  - mkdir build
  - cd build
  - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh
- - make --output-sync -j`nproc`
- - make --output-sync -j`nproc` check V=1
- - make --output-sync -j`nproc` check-tcg V=1
+ - make --output-sync -j`nproc --ignore=40`
+ - make --output-sync -j`nproc --ignore=40` check V=1
+ - make --output-sync -j`nproc --ignore=40` check-tcg V=1
 
 ubuntu-20.04-aarch64-all:
  needs: []
@@ -38,8 +38,8 @@ ubuntu-20.04-aarch64-all:
  - mkdir build
  - cd build
  - ../configure --disable-libssh
- - make --output-sync -j`nproc`
- - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc --ignore=40`
+ - make --output-sync -j`nproc --ignore=40` check V=1
 
 ubuntu-20.04-aarch64-alldbg:
  needs: []
@@ -55,8 +55,8 @@ ubuntu-20.04-aarch64-alldbg:
  - cd build
  - ../configure --enable-debug --disable-libssh
  - make clean
- - make --output-sync -j`nproc`
- - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc --ignore=40`
+ - make --output-sync -j`nproc --ignore=40` check V=1
 
 ubuntu-20.04-aarch64-clang:
  needs: []
@@ -75,8 +75,8 @@ ubuntu-20.04-aarch64-clang:
  - mkdir build
  - cd build
  - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers
- - make --output-sync -j`nproc`
- - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc --ignore=40`
+ - make --output-sync -j`nproc --ignore=40` check V=1
 
 ubuntu-20.04-aarch64-tci:
  needs: []
@@ -95,7 +95,7 @@ ubuntu-20.04-aarch64-tci:
  - mkdir build
  - cd build
  - ../configure --disable-libssh --enable-tcg-interpreter
- - make --output-sync -j`nproc`
+ - make --output-sync -j`nproc --ignore=40`
 
 ubuntu-20.04-aarch64-notcg:
  needs: []
@@ -114,5 +114,5 @@ ubuntu-20.04-aarch64-notcg:
  - mkdir build
  - cd build
  - ../configure --disable-libssh --disable-tcg
- - make --output-sync -j`nproc`
- - make --output-sync -j`nproc` check V=1
+ - make --output-sync -j`nproc --ignore=40`
+ - make --output-sync -j`nproc --ignore=40` check V=1
-- 
2.30.2

Re: [PATCH v1 3/7] gitlab-ci: Fix the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs

[Richard Henderson]

On 6/13/22 10:12, Alex Bennée wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> The job definitions recently got a second "variables:" section by
> accident and thus are failing now if one tries to run them. Merge
> the two sections into one again to fix the issue.
> 
> And while we're at it, bump the timeout here (70 minutes are currently
> not enough for the aarch64 job). The jobs are marked as manual anyway,
> so if the user starts them, they want to see their result for sure and
> then it's annoying if the job timeouts too early.
> 
> Fixes: e312d1fdbb ("gitlab: convert build/container jobs to .base_job_template")
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> Acked-by: Richard Henderson <richard.henderson@linaro.org>
> Message-Id: <20220603124809.70794-1-thuth@redhat.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.d/buildtest.yml | 22 ++++++++++------------
>   1 file changed, 10 insertions(+), 12 deletions(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 544385f5be..cb7cad44b5 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -357,16 +357,15 @@ build-cfi-aarch64:
>         --enable-safe-stack --enable-slirp=git
>       TARGETS: aarch64-softmmu
>       MAKE_CHECK_ARGS: check-build
> -  timeout: 70m
> -  artifacts:
> -    expire_in: 2 days
> -    paths:
> -      - build
> -  variables:
>       # FIXME: This job is often failing, likely due to out-of-memory problems in
>       # the constrained containers of the shared runners. Thus this is marked as
>       # skipped until the situation has been solved.
>       QEMU_JOB_SKIPPED: 1
> +  timeout: 90m
> +  artifacts:
> +    expire_in: 2 days
> +    paths:
> +      - build

FWIW, 90 minutes was close, but insufficient:

https://gitlab.com/qemu-project/qemu/-/jobs/2584472225

But certainly, let us fix the job definition:
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

Re: [PATCH v1 4/7] tests/docker: fix the IMAGE for build invocation

[Richard Henderson]

On 6/13/22 10:12, Alex Bennée wrote:
> We inadvertently broke the ability to run local builds when the code
> was re-factored. The result was the run stanza failing to find the
> docker image with it's qemu/ prefix.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Fixes: d39eaa2266 ("tests/docker: simplify docker-TEST@IMAGE targets")
> ---
>   tests/docker/Makefile.include | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index e68f91b853..ef4518d9eb 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -171,7 +171,7 @@ DOCKER_TESTS := $(if $(TESTS), $(filter $(TESTS), $(__TESTS)), $(__TESTS))
>   $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES)), \
>   	$(foreach t,$(DOCKER_TESTS), \
>   		$(eval .PHONY: docker-$t@$i) \
> -		$(eval docker-$t@$i: docker-image-$i; @$(MAKE) docker-run TEST=$t IMAGE=$i) \
> +		$(eval docker-$t@$i: docker-image-$i; @$(MAKE) docker-run TEST=$t IMAGE=qemu/$i) \
>   	) \
>   	$(foreach t,$(DOCKER_TESTS), \
>   		$(eval docker-all-tests: docker-$t@$i) \

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [PATCH v1 5/7] tests/tcg: disable xtensa-linux-user again

[Richard Henderson]

On 6/13/22 10:12, Alex Bennée wrote:
> From: Paolo Bonzini <pbonzini@redhat.com>
> 
> The move from tests/tcg/configure.sh started enabling the container image
> for xtensa-linux-user, which fails because the compiler does not have
> the full set of headers.  The cause is the "xtensa*-softmmu)" case
> in tests/tcg/configure.sh which became just "xtensa*)" in the new
> probe_target_compiler shell function.  Look out for xtensa*-linux-user
> and do not configure it.
> 
> Reported-by: Alex Bennée <alex.bennee@linaro.org>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> Message-Id: <20220608135727.1341946-1-pbonzini@redhat.com>
> Fixes: cd362defbb ("tests/tcg: merge configure.sh back into main configure script")
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

Re: [PATCH v1 7/7] .gitlab: use less aggressive nproc on our aarch64/32 runners

[Richard Henderson]

On 6/13/22 10:12, Alex Bennée wrote:
> Running on all 80 cores of our aarch64 runner does occasionally
> trigger a race condition which fails the build. However the CI system
> is not the time and place to play with much heisenbugs so turn down
> the nproc to "only" use 40 cores in the build.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .../custom-runners/ubuntu-20.04-aarch32.yml   |  4 ++--
>   .../custom-runners/ubuntu-20.04-aarch64.yml   | 24 +++++++++----------
>   2 files changed, 14 insertions(+), 14 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Richard Henderson <richard.henderson@linaro.org>


r~

Re: [PATCH v1 5/7] tests/tcg: disable xtensa-linux-user again

[Philippe Mathieu-Daudé via]

On 13/6/22 19:12, Alex Bennée wrote:
> From: Paolo Bonzini <pbonzini@redhat.com>
> 
> The move from tests/tcg/configure.sh started enabling the container image
> for xtensa-linux-user, which fails because the compiler does not have
> the full set of headers.  The cause is the "xtensa*-softmmu)" case
> in tests/tcg/configure.sh which became just "xtensa*)" in the new
> probe_target_compiler shell function.  Look out for xtensa*-linux-user
> and do not configure it.
> 
> Reported-by: Alex Bennée <alex.bennee@linaro.org>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> Message-Id: <20220608135727.1341946-1-pbonzini@redhat.com>
> Fixes: cd362defbb ("tests/tcg: merge configure.sh back into main configure script")
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   configure | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Re: [PATCH v1 7/7] .gitlab: use less aggressive nproc on our aarch64/32 runners

[Philippe Mathieu-Daudé via]

On 13/6/22 19:12, Alex Bennée wrote:
> Running on all 80 cores of our aarch64 runner does occasionally
> trigger a race condition which fails the build. However the CI system
> is not the time and place to play with much heisenbugs so turn down
> the nproc to "only" use 40 cores in the build.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .../custom-runners/ubuntu-20.04-aarch32.yml   |  4 ++--
>   .../custom-runners/ubuntu-20.04-aarch64.yml   | 24 +++++++++----------
>   2 files changed, 14 insertions(+), 14 deletions(-)
> 
> diff --git a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
> index 47856ac53c..1998460d06 100644
> --- a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
> +++ b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
> @@ -19,5 +19,5 @@ ubuntu-20.04-aarch32-all:
>    - mkdir build
>    - cd build
>    - ../configure --cross-prefix=arm-linux-gnueabihf-
> - - make --output-sync -j`nproc`
> - - make --output-sync -j`nproc` check V=1
> + - make --output-sync -j`nproc --ignore=40`
> + - make --output-sync -j`nproc --ignore=40` check V=1

Alternatively:

-j$(expr $(nproc) / 2)

So we don't have to update the configs if the runner gets its hardware 
updated.

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Re: [PATCH v1 3/7] gitlab-ci: Fix the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs

[Thomas Huth]

On 13/06/2022 23.46, Richard Henderson wrote:
> On 6/13/22 10:12, Alex Bennée wrote:
>> From: Thomas Huth <thuth@redhat.com>
>>
>> The job definitions recently got a second "variables:" section by
>> accident and thus are failing now if one tries to run them. Merge
>> the two sections into one again to fix the issue.
>>
>> And while we're at it, bump the timeout here (70 minutes are currently
>> not enough for the aarch64 job). The jobs are marked as manual anyway,
>> so if the user starts them, they want to see their result for sure and
>> then it's annoying if the job timeouts too early.
>>
>> Fixes: e312d1fdbb ("gitlab: convert build/container jobs to 
>> .base_job_template")
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> Acked-by: Richard Henderson <richard.henderson@linaro.org>
>> Message-Id: <20220603124809.70794-1-thuth@redhat.com>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>   .gitlab-ci.d/buildtest.yml | 22 ++++++++++------------
>>   1 file changed, 10 insertions(+), 12 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
>> index 544385f5be..cb7cad44b5 100644
>> --- a/.gitlab-ci.d/buildtest.yml
>> +++ b/.gitlab-ci.d/buildtest.yml
>> @@ -357,16 +357,15 @@ build-cfi-aarch64:
>>         --enable-safe-stack --enable-slirp=git
>>       TARGETS: aarch64-softmmu
>>       MAKE_CHECK_ARGS: check-build
>> -  timeout: 70m
>> -  artifacts:
>> -    expire_in: 2 days
>> -    paths:
>> -      - build
>> -  variables:
>>       # FIXME: This job is often failing, likely due to out-of-memory 
>> problems in
>>       # the constrained containers of the shared runners. Thus this is 
>> marked as
>>       # skipped until the situation has been solved.
>>       QEMU_JOB_SKIPPED: 1
>> +  timeout: 90m
>> +  artifacts:
>> +    expire_in: 2 days
>> +    paths:
>> +      - build
> 
> FWIW, 90 minutes was close, but insufficient:
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/2584472225

Hmm, it was working at least once for me while I was working on the patch. 
But as I already wrote here:

  https://lists.gnu.org/archive/html/qemu-devel/2022-06/msg00463.html

I think nobody really used this build-cfi-aarch64 in month ... so we should 
maybe have a try with the 90 min timeout first (maybe the CI servers were 
just a little bit overloaded when you tried), but if the test continues to 
hit the 90 minutes timeout, I'd say we rather delete it instead of bumping 
the timeout even further. 90 minutes are really very close to the pain level 
already - at least for me.

> But certainly, let us fix the job definition:
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

Thanks!

  Thomas

Re: [PATCH v1 3/7] gitlab-ci: Fix the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs

[Daniel P. Berrangé]

On Tue, Jun 14, 2022 at 06:30:47AM +0200, Thomas Huth wrote:
> On 13/06/2022 23.46, Richard Henderson wrote:
> > On 6/13/22 10:12, Alex Bennée wrote:
> > > From: Thomas Huth <thuth@redhat.com>
> > > 
> > > The job definitions recently got a second "variables:" section by
> > > accident and thus are failing now if one tries to run them. Merge
> > > the two sections into one again to fix the issue.
> > > 
> > > And while we're at it, bump the timeout here (70 minutes are currently
> > > not enough for the aarch64 job). The jobs are marked as manual anyway,
> > > so if the user starts them, they want to see their result for sure and
> > > then it's annoying if the job timeouts too early.
> > > 
> > > Fixes: e312d1fdbb ("gitlab: convert build/container jobs to
> > > .base_job_template")
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > Acked-by: Richard Henderson <richard.henderson@linaro.org>
> > > Message-Id: <20220603124809.70794-1-thuth@redhat.com>
> > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > ---
> > >   .gitlab-ci.d/buildtest.yml | 22 ++++++++++------------
> > >   1 file changed, 10 insertions(+), 12 deletions(-)
> > > 
> > > diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> > > index 544385f5be..cb7cad44b5 100644
> > > --- a/.gitlab-ci.d/buildtest.yml
> > > +++ b/.gitlab-ci.d/buildtest.yml
> > > @@ -357,16 +357,15 @@ build-cfi-aarch64:
> > >         --enable-safe-stack --enable-slirp=git
> > >       TARGETS: aarch64-softmmu
> > >       MAKE_CHECK_ARGS: check-build
> > > -  timeout: 70m
> > > -  artifacts:
> > > -    expire_in: 2 days
> > > -    paths:
> > > -      - build
> > > -  variables:
> > >       # FIXME: This job is often failing, likely due to
> > > out-of-memory problems in
> > >       # the constrained containers of the shared runners. Thus this
> > > is marked as
> > >       # skipped until the situation has been solved.
> > >       QEMU_JOB_SKIPPED: 1
> > > +  timeout: 90m
> > > +  artifacts:
> > > +    expire_in: 2 days
> > > +    paths:
> > > +      - build
> > 
> > FWIW, 90 minutes was close, but insufficient:
> > 
> > https://gitlab.com/qemu-project/qemu/-/jobs/2584472225
> 
> Hmm, it was working at least once for me while I was working on the patch.
> But as I already wrote here:
> 
>  https://lists.gnu.org/archive/html/qemu-devel/2022-06/msg00463.html
> 
> I think nobody really used this build-cfi-aarch64 in month ... so we should
> maybe have a try with the 90 min timeout first (maybe the CI servers were
> just a little bit overloaded when you tried), but if the test continues to
> hit the 90 minutes timeout, I'd say we rather delete it instead of bumping
> the timeout even further. 90 minutes are really very close to the pain level
> already - at least for me.

The CFI jobs seem to massively slow down and timeout waaaaaaay
more often than any other job. I've seen the CFI jobs run
successfully in 45 minutes, and yet they frequently take so long
that they can't even complete in double that. CFI is certainly
slower at compile but not in a non-deterministic manner that
would randomly double compilation time. I would be willing to
blame CI overload if all our other jobs were showing similar
magnitude of slow down, but AFAIK, they are not showing this.
I worry that there are genuine problems with the CFI builds
that result in non-deterministic runtime problems in functional
testing. IOW not merely running slowly, but genuine hang


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|