[PATCH 0/4] Test more in less time in the Gitlab-CI

[PATCH 0/4] Test more in less time in the Gitlab-CI

[Thomas Huth]

This patch series adds two new "build-system" pipelines to the Gitlab-CI,
one based on Debian and one on CentOS. We then use these build pipelines
to test the targets that were missing so far (e.g. the two new targets
rx-softmmu and avr-softmmu), and move some of the targets from the other
build-system pipelines here, too, so that the total testing time gets
shorter (at least 5 minutes from what I've seen so far).

Thomas Huth (4):
  tests/docker: Add python3-venv and netcat to the debian-amd64
    container
  iotests: Select a default machine for the rx and avr targets
  tests/acceptance: Disable the rx sash and arm cubieboard replay test
    on Gitlab
  gitlab-ci.yml: Add build-system-debian and build-system-centos jobs

 .gitlab-ci.yml                               | 88 ++++++++++++++++----
 tests/acceptance/machine_rx_gdbsim.py        |  4 +
 tests/acceptance/replay_kernel.py            |  1 +
 tests/docker/dockerfiles/debian-amd64.docker |  4 +-
 tests/qemu-iotests/check                     | 14 ++--
 5 files changed, 90 insertions(+), 21 deletions(-)

-- 
2.18.1

[PATCH 1/4] tests/docker: Add python3-venv and netcat to the debian-amd64 container

[Thomas Huth]

Without python3-venv, I get the following message when trying to
run the acceptance tests within the debian container:

 The virtual environment was not created successfully because ensurepip is not
 available.  On Debian/Ubuntu systems, you need to install the python3-venv
 package using the following command.
    apt-get install python3-venv
 You may need to use sudo with that command.  After installing the python3-venv
 package, recreate your virtual environment.

Let's do it as the message suggests.

And while we're at it, also add netcat here since it is required for
some of the acceptance tests.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/docker/dockerfiles/debian-amd64.docker | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index 8fdfd6a6b0..d2500dcff1 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -20,7 +20,9 @@ RUN apt update && \
         librdmacm-dev \
         libsasl2-dev \
         libsnappy-dev \
-        libvte-dev
+        libvte-dev \
+        netcat-openbsd \
+        python3-venv
 
 # virgl
 RUN apt update && \
-- 
2.18.1

[PATCH 2/4] iotests: Select a default machine for the rx and avr targets

[Thomas Huth]

If you are building only with either the new rx-softmmu or avr-softmmu
target, "make check-block" fails a couple of tests since there is no
default machine defined in these new targets. We have to select a machine
in the "check" script for these, just like we already do for the arm- and
tricore-softmmu targets.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/qemu-iotests/check | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index e0d8049012..0657f7286c 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -595,15 +595,19 @@ then
 fi
 export QEMU_PROG="$(type -p "$QEMU_PROG")"
 
+export QEMU_OPTIONS="-nodefaults -display none -accel qtest"
 case "$QEMU_PROG" in
     *qemu-system-arm|*qemu-system-aarch64)
-        export QEMU_OPTIONS="-nodefaults -display none -machine virt -accel qtest"
+        export QEMU_OPTIONS="$QEMU_OPTIONS -machine virt"
         ;;
-    *qemu-system-tricore)
-        export QEMU_OPTIONS="-nodefaults -display none -machine tricore_testboard -accel qtest"
+    *qemu-system-avr)
+        export QEMU_OPTIONS="$QEMU_OPTIONS -machine mega2560"
+        ;;
+    *qemu-system-rx)
+        export QEMU_OPTIONS="$QEMU_OPTIONS -machine gdbsim-r5f562n8"
         ;;
-    *)
-        export QEMU_OPTIONS="-nodefaults -display none -accel qtest"
+    *qemu-system-tricore)
+        export QEMU_OPTIONS="-$QEMU_OPTIONS -machine tricore_testboard"
         ;;
 esac
 
-- 
2.18.1

[PATCH 3/4] tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab

[Thomas Huth]

These tests always time out on Gitlab, not sure what's happening here.
Let's disable them until somebody has enough spare time to debug the
issues.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/acceptance/machine_rx_gdbsim.py | 4 ++++
 tests/acceptance/replay_kernel.py     | 1 +
 2 files changed, 5 insertions(+)

diff --git a/tests/acceptance/machine_rx_gdbsim.py b/tests/acceptance/machine_rx_gdbsim.py
index bff63e421d..0c72506028 100644
--- a/tests/acceptance/machine_rx_gdbsim.py
+++ b/tests/acceptance/machine_rx_gdbsim.py
@@ -8,6 +8,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import os
+
+from avocado import skipIf
 from avocado_qemu import Test
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
@@ -42,6 +45,7 @@ class RxGdbSimMachine(Test):
         # FIXME limit baudrate on chardev, else we type too fast
         #exec_command_and_wait_for_pattern(self, 'version', gcc_version)
 
+    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
     def test_linux_sash(self):
         """
         Boots a Linux kernel and checks that the console is operational.
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 62d2db8c64..b79fc8daf8 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -126,6 +126,7 @@ class ReplayKernel(LinuxKernelTest):
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
 
+    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
     def test_arm_cubieboard_initrd(self):
         """
         :avocado: tags=arch:arm
-- 
2.18.1

[PATCH 4/4] gitlab-ci.yml: Add build-system-debian and build-system-centos jobs

[Thomas Huth]

We were missing the two new targets avr-softmmu and rx-softmmu in the
gitlab-CI so far, and did not add some of the "other endianess" targets
like sh4eb-softmmu yet.
Since the current build-system-* jobs run already for a very long time,
let's do not add these missing targets there, but introduce two new
additional build jobs, one running with Debian and one running with
CentOS, and add the new targets there. Also move some targets from
the old build-system-* jobs to these new targets, to distribute the
load and reduce the runtime of the CI.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 88 +++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 73 insertions(+), 15 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 362e5ee755..e96bcd50f8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -53,68 +53,126 @@ include:
     - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
     - du -chs $HOME/avocado/data/cache
 
-build-system-ubuntu-main:
+build-system-ubuntu:
   <<: *native_build_job_definition
   variables:
     IMAGE: ubuntu2004
-    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu
-      moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu
-      riscv64-softmmu sparc-softmmu
+    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
+      moxie-softmmu microblazeel-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
       - build
 
-check-system-ubuntu-main:
+check-system-ubuntu:
   <<: *native_test_job_definition
   needs:
-    - job: build-system-ubuntu-main
+    - job: build-system-ubuntu
       artifacts: true
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
-acceptance-system-ubuntu-main:
+acceptance-system-ubuntu:
   <<: *native_test_job_definition
   needs:
-    - job: build-system-ubuntu-main
+    - job: build-system-ubuntu
       artifacts: true
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
   <<: *post_acceptance
 
-build-system-fedora-alt:
+build-system-debian:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: debian-amd64
+    TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
+      riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    paths:
+      - build
+
+check-system-debian:
+  <<: *native_test_job_definition
+  needs:
+    - job: build-system-debian
+      artifacts: true
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-debian:
+  <<: *native_test_job_definition
+  needs:
+    - job: build-system-debian
+      artifacts: true
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: check-acceptance
+  <<: *post_acceptance
+
+build-system-fedora:
   <<: *native_build_job_definition
   variables:
     IMAGE: fedora
     TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu
-      riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu
-      xtensa-softmmu nios2-softmmu or1k-softmmu
+      xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
       - build
 
-check-system-fedora-alt:
+check-system-fedora:
   <<: *native_test_job_definition
   needs:
-    - job: build-system-fedora-alt
+    - job: build-system-fedora
       artifacts: true
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-system-fedora-alt:
+acceptance-system-fedora:
   <<: *native_test_job_definition
   needs:
-    - job: build-system-fedora-alt
+    - job: build-system-fedora
       artifacts: true
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
   <<: *post_acceptance
 
+build-system-centos:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: centos8
+    TARGETS: ppc64-softmmu lm32-softmmu or1k-softmmu s390x-softmmu
+      x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    paths:
+      - build
+
+check-system-centos:
+  <<: *native_test_job_definition
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos8
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-centos:
+  <<: *native_test_job_definition
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos8
+    MAKE_CHECK_ARGS: check-acceptance
+  <<: *post_acceptance
+
 build-disabled:
   <<: *native_build_job_definition
   variables:
-- 
2.18.1

Re: [PATCH 3/4] tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab

[Wainer dos Santos Moschetta]

On 7/23/20 9:27 AM, Thomas Huth wrote:
> These tests always time out on Gitlab, not sure what's happening here.
> Let's disable them until somebody has enough spare time to debug the
> issues.

It's fair to me.

>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   tests/acceptance/machine_rx_gdbsim.py | 4 ++++
>   tests/acceptance/replay_kernel.py     | 1 +
>   2 files changed, 5 insertions(+)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/tests/acceptance/machine_rx_gdbsim.py b/tests/acceptance/machine_rx_gdbsim.py
> index bff63e421d..0c72506028 100644
> --- a/tests/acceptance/machine_rx_gdbsim.py
> +++ b/tests/acceptance/machine_rx_gdbsim.py
> @@ -8,6 +8,9 @@
>   # This work is licensed under the terms of the GNU GPL, version 2 or
>   # later.  See the COPYING file in the top-level directory.
>   
> +import os
> +
> +from avocado import skipIf
>   from avocado_qemu import Test
>   from avocado_qemu import exec_command_and_wait_for_pattern
>   from avocado_qemu import wait_for_console_pattern
> @@ -42,6 +45,7 @@ class RxGdbSimMachine(Test):
>           # FIXME limit baudrate on chardev, else we type too fast
>           #exec_command_and_wait_for_pattern(self, 'version', gcc_version)
>   
> +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
>       def test_linux_sash(self):
>           """
>           Boots a Linux kernel and checks that the console is operational.
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> index 62d2db8c64..b79fc8daf8 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -126,6 +126,7 @@ class ReplayKernel(LinuxKernelTest):
>   
>           self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
>   
> +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
>       def test_arm_cubieboard_initrd(self):
>           """
>           :avocado: tags=arch:arm

Re: [PATCH 3/4] tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab

[Philippe Mathieu-Daudé]

On 7/23/20 2:27 PM, Thomas Huth wrote:
> These tests always time out on Gitlab, not sure what's happening here.
> Let's disable them until somebody has enough spare time to debug the
> issues.

Is the Avocado cache working? Is it failing with an empty cache?
If so, maybe we need to run avocado cache fetching in a previous
step, before the testing job.

Can you share a url of failed job so I can have a look?

> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/acceptance/machine_rx_gdbsim.py | 4 ++++
>  tests/acceptance/replay_kernel.py     | 1 +
>  2 files changed, 5 insertions(+)
> 
> diff --git a/tests/acceptance/machine_rx_gdbsim.py b/tests/acceptance/machine_rx_gdbsim.py
> index bff63e421d..0c72506028 100644
> --- a/tests/acceptance/machine_rx_gdbsim.py
> +++ b/tests/acceptance/machine_rx_gdbsim.py
> @@ -8,6 +8,9 @@
>  # This work is licensed under the terms of the GNU GPL, version 2 or
>  # later.  See the COPYING file in the top-level directory.
>  
> +import os
> +
> +from avocado import skipIf
>  from avocado_qemu import Test
>  from avocado_qemu import exec_command_and_wait_for_pattern
>  from avocado_qemu import wait_for_console_pattern
> @@ -42,6 +45,7 @@ class RxGdbSimMachine(Test):
>          # FIXME limit baudrate on chardev, else we type too fast
>          #exec_command_and_wait_for_pattern(self, 'version', gcc_version)
>  
> +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
>      def test_linux_sash(self):
>          """
>          Boots a Linux kernel and checks that the console is operational.
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> index 62d2db8c64..b79fc8daf8 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -126,6 +126,7 @@ class ReplayKernel(LinuxKernelTest):
>  
>          self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
>  
> +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
>      def test_arm_cubieboard_initrd(self):
>          """
>          :avocado: tags=arch:arm
> 

Re: [PATCH 3/4] tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab

[Thomas Huth]

On 23/07/2020 14.53, Philippe Mathieu-Daudé wrote:
> On 7/23/20 2:27 PM, Thomas Huth wrote:
>> These tests always time out on Gitlab, not sure what's happening here.
>> Let's disable them until somebody has enough spare time to debug the
>> issues.
> 
> Is the Avocado cache working? Is it failing with an empty cache?
> If so, maybe we need to run avocado cache fetching in a previous
> step, before the testing job.
> 
> Can you share a url of failed job so I can have a look?

CentOS:
 https://gitlab.com/huth/qemu/-/jobs/651179303

Debian:
 https://gitlab.com/huth/qemu/-/jobs/651179296

... might be a cache issue, indeed.

 Thomas

Re: [PATCH 3/4] tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab

[Cleber Rosa]

----- Thomas Huth <thuth@redhat.com> wrote:
> On 23/07/2020 14.53, Philippe Mathieu-Daudé wrote:
> > On 7/23/20 2:27 PM, Thomas Huth wrote:
> >> These tests always time out on Gitlab, not sure what's happening here.
> >> Let's disable them until somebody has enough spare time to debug the
> >> issues.
> > 
> > Is the Avocado cache working? Is it failing with an empty cache?
> > If so, maybe we need to run avocado cache fetching in a previous
> > step, before the testing job.
> > 
> > Can you share a url of failed job so I can have a look?
> 
> CentOS:
>  https://gitlab.com/huth/qemu/-/jobs/651179303
> 
I don't see the sash files being downloaded/checked ahead of test execution, so that may be an issue indeed. But for the cubieboard, that is being done, so does not look like a cache issue.

Either way, it's better to disable these and re-enable them after a proper investigation (I'll follow up with that).

Reviewed-by: Cleber Rosa <crosa@redhat.com>


> Debian:
>  https://gitlab.com/huth/qemu/-/jobs/651179296
> 
> ... might be a cache issue, indeed.
> 
>  Thomas