[PATCH 0/4] travis-ci: Add a KVM-only s390x job

[PATCH 0/4] travis-ci: Add a KVM-only s390x job

[Philippe Mathieu-Daudé]

Add a Travis job to build a KVM-only QEMU on s390x.

This series also contains few fixes for Travis/s390x.

Philippe Mathieu-Daudé (4):
  tests/test-util-filemonitor: Fix Travis-CI $ARCH env variable name
  tests/test-util-sockets: Skip test on non-x86 Travis containers
  tests/migration: Disable autoconverge test on Travis-CI s390x
  .travis.yml: Add a KVM-only s390x job

 tests/qtest/migration-test.c  | 10 +++++++++
 tests/test-util-filemonitor.c |  2 +-
 tests/test-util-sockets.c     |  7 ++++++
 .travis.yml                   | 42 +++++++++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+), 1 deletion(-)

-- 
2.21.1

[PATCH 1/4] tests/test-util-filemonitor: Fix Travis-CI $ARCH env variable name

[Philippe Mathieu-Daudé]

While we can find reference of a 'TRAVIS_ARCH' variable in
the environment and source [1], per the Travis-CI multi-arch
documentation [2] the variable is named TRAVIS_CPU_ARCH.

[1] https://github.com/travis-ci/travis-build/blob/v10.0.0/lib/travis/build/bash/travis_setup_env.bash#L39
[2] https://docs.travis-ci.com/user/multi-cpu-architectures/#identifying-cpu-architecture-of-build-jobs

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/test-util-filemonitor.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/test-util-filemonitor.c b/tests/test-util-filemonitor.c
index 45009c69f4..e703a7f8fc 100644
--- a/tests/test-util-filemonitor.c
+++ b/tests/test-util-filemonitor.c
@@ -415,7 +415,7 @@ test_file_monitor_events(void)
      * This test does not work on Travis LXD containers since some
      * syscalls are blocked in that environment.
      */
-    travis_arch = getenv("TRAVIS_ARCH");
+    travis_arch = getenv("TRAVIS_CPU_ARCH");
     if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
         g_test_skip("Test does not work on non-x86 Travis containers.");
         return;
-- 
2.21.1

[PATCH 2/4] tests/test-util-sockets: Skip test on non-x86 Travis containers

[Philippe Mathieu-Daudé]

Similarly to commit 4f370b1098, test-util-sockets fails in
restricted non-x86 Travis containers since they apparently
blacklisted some required system calls there.
Let's simply skip the test if we detect such an environment.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/test-util-sockets.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/test-util-sockets.c b/tests/test-util-sockets.c
index 5fd947c7bf..046ebec8ba 100644
--- a/tests/test-util-sockets.c
+++ b/tests/test-util-sockets.c
@@ -231,11 +231,18 @@ static void test_socket_fd_pass_num_nocli(void)
 int main(int argc, char **argv)
 {
     bool has_ipv4, has_ipv6;
+    char *travis_arch;
 
     socket_init();
 
     g_test_init(&argc, &argv, NULL);
 
+    travis_arch = getenv("TRAVIS_CPU_ARCH");
+    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
+        g_printerr("Test does not work on non-x86 Travis containers.");
+        goto end;
+    }
+
     /* We're creating actual IPv4/6 sockets, so we should
      * check if the host running tests actually supports
      * each protocol to avoid breaking tests on machines
-- 
2.21.1

[PATCH 3/4] tests/migration: Disable autoconverge test on Travis-CI s390x

[Philippe Mathieu-Daudé]

This test fails on Travis-CI s390x when configured with --disable-tcg:

  $ make check-qtest
    TEST    check-qtest-s390x: tests/qtest/boot-serial-test
  qemu-system-s390x: -accel tcg: invalid accelerator tcg
  qemu-system-s390x: falling back to KVM
    TEST    check-qtest-s390x: tests/qtest/pxe-test
    TEST    check-qtest-s390x: tests/qtest/test-netfilter
    TEST    check-qtest-s390x: tests/qtest/test-filter-mirror
    TEST    check-qtest-s390x: tests/qtest/test-filter-redirector
    TEST    check-qtest-s390x: tests/qtest/drive_del-test
    TEST    check-qtest-s390x: tests/qtest/device-plug-test
    TEST    check-qtest-s390x: tests/qtest/virtio-ccw-test
    TEST    check-qtest-s390x: tests/qtest/cpu-plug-test
    TEST    check-qtest-s390x: tests/qtest/migration-test
  **
  ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
  ERROR - Bail out! ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
  make: *** [tests/Makefile.include:633: check-qtest-s390x] Error 1

Per David Gilbert, it might be a threshold requiring tuning:
"it could just be the writing is slow on s390 and the migration
thread fast; in which case the autocomplete wouldn't be needed.
Perhaps we just need to reduce the bandwidth limit."

Disable the autoconverge test for now.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/qtest/migration-test.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index 3d6cc83b88..878399666e 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -1181,6 +1181,16 @@ static void test_migrate_auto_converge(void)
     MigrateStart *args = migrate_start_new();
     QTestState *from, *to;
     int64_t remaining, percentage;
+    char *travis_arch;
+
+    /*
+     * This test does not work reliably on Travis s390x.
+     */
+    travis_arch = getenv("TRAVIS_CPU_ARCH");
+    if (travis_arch && g_str_equal(travis_arch, "s390x")) {
+        g_test_skip("Test does not work reliably on s390x Travis containers.");
+        return;
+    }
 
     /*
      * We want the test to be stable and as fast as possible.
-- 
2.21.1

[PATCH 4/4] .travis.yml: Add a KVM-only s390x job

[Philippe Mathieu-Daudé]

Add a job to build QEMU on s390x with TCG disabled, so
this configuration won't bitrot over time.

This job is quick, running check-unit: Ran for 4 min 48 sec
https://travis-ci.org/github/philmd/qemu/jobs/663659486

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index b92798ac3b..0109f02ed4 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -524,6 +524,48 @@ jobs:
               $(exit $BUILD_RC);
           fi
 
+    - name: "[s390x] GCC check (KVM)"
+      arch: s390x
+      dist: bionic
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+          # Tests dependencies
+          - genisoimage
+      env:
+        - TEST_CMD="make check-unit"
+        - CONFIG="--disable-containers --disable-tcg --enable-kvm --disable-tools"
+      script:
+        - ( cd ${SRC_DIR} ; git submodule update --init roms/SLOF )
+        - BUILD_RC=0 && make -j${JOBS} || BUILD_RC=$?
+        - |
+          if [ "$BUILD_RC" -eq 0 ] ; then
+              mv pc-bios/s390-ccw/*.img pc-bios/ ;
+              ${TEST_CMD} ;
+          else
+              $(exit $BUILD_RC);
+          fi
+
     # Release builds
     # The make-release script expect a QEMU version, so our tag must start with a 'v'.
     # This is the case when release candidate tags are created.
-- 
2.21.1

Re: [PATCH 3/4] tests/migration: Disable autoconverge test on Travis-CI s390x

[Dr. David Alan Gilbert]

* Philippe Mathieu-Daudé (philmd@redhat.com) wrote:
> This test fails on Travis-CI s390x when configured with --disable-tcg:
> 
>   $ make check-qtest
>     TEST    check-qtest-s390x: tests/qtest/boot-serial-test
>   qemu-system-s390x: -accel tcg: invalid accelerator tcg
>   qemu-system-s390x: falling back to KVM
>     TEST    check-qtest-s390x: tests/qtest/pxe-test
>     TEST    check-qtest-s390x: tests/qtest/test-netfilter
>     TEST    check-qtest-s390x: tests/qtest/test-filter-mirror
>     TEST    check-qtest-s390x: tests/qtest/test-filter-redirector
>     TEST    check-qtest-s390x: tests/qtest/drive_del-test
>     TEST    check-qtest-s390x: tests/qtest/device-plug-test
>     TEST    check-qtest-s390x: tests/qtest/virtio-ccw-test
>     TEST    check-qtest-s390x: tests/qtest/cpu-plug-test
>     TEST    check-qtest-s390x: tests/qtest/migration-test
>   **
>   ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
>   ERROR - Bail out! ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
>   make: *** [tests/Makefile.include:633: check-qtest-s390x] Error 1

Before we take the hammer to it, could you try reducing it's initial
bandwidth, the code that says:

    /*
     * Set the initial parameters so that the migration could not converge
     * without throttling.
     */
    migrate_set_parameter_int(from, "downtime-limit", 1);
    migrate_set_parameter_int(from, "max-bandwidth", 100000000); /* ~100Mb/s */

knock a couple of 0's off that and see if it's happy.

Dave

> Per David Gilbert, it might be a threshold requiring tuning:
> "it could just be the writing is slow on s390 and the migration
> thread fast; in which case the autocomplete wouldn't be needed.
> Perhaps we just need to reduce the bandwidth limit."
> 
> Disable the autoconverge test for now.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/qtest/migration-test.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
> index 3d6cc83b88..878399666e 100644
> --- a/tests/qtest/migration-test.c
> +++ b/tests/qtest/migration-test.c
> @@ -1181,6 +1181,16 @@ static void test_migrate_auto_converge(void)
>      MigrateStart *args = migrate_start_new();
>      QTestState *from, *to;
>      int64_t remaining, percentage;
> +    char *travis_arch;
> +
> +    /*
> +     * This test does not work reliably on Travis s390x.
> +     */
> +    travis_arch = getenv("TRAVIS_CPU_ARCH");
> +    if (travis_arch && g_str_equal(travis_arch, "s390x")) {
> +        g_test_skip("Test does not work reliably on s390x Travis containers.");
> +        return;
> +    }
>  
>      /*
>       * We want the test to be stable and as fast as possible.
> -- 
> 2.21.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [PATCH 3/4] tests/migration: Disable autoconverge test on Travis-CI s390x

[Philippe Mathieu-Daudé]

On 3/17/20 9:26 PM, Dr. David Alan Gilbert wrote:
> * Philippe Mathieu-Daudé (philmd@redhat.com) wrote:
>> This test fails on Travis-CI s390x when configured with --disable-tcg:
>>
>>    $ make check-qtest
>>      TEST    check-qtest-s390x: tests/qtest/boot-serial-test
>>    qemu-system-s390x: -accel tcg: invalid accelerator tcg
>>    qemu-system-s390x: falling back to KVM
>>      TEST    check-qtest-s390x: tests/qtest/pxe-test
>>      TEST    check-qtest-s390x: tests/qtest/test-netfilter
>>      TEST    check-qtest-s390x: tests/qtest/test-filter-mirror
>>      TEST    check-qtest-s390x: tests/qtest/test-filter-redirector
>>      TEST    check-qtest-s390x: tests/qtest/drive_del-test
>>      TEST    check-qtest-s390x: tests/qtest/device-plug-test
>>      TEST    check-qtest-s390x: tests/qtest/virtio-ccw-test
>>      TEST    check-qtest-s390x: tests/qtest/cpu-plug-test
>>      TEST    check-qtest-s390x: tests/qtest/migration-test
>>    **
>>    ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
>>    ERROR - Bail out! ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
>>    make: *** [tests/Makefile.include:633: check-qtest-s390x] Error 1
> 
> Before we take the hammer to it, could you try reducing it's initial
> bandwidth, the code that says:
> 
>      /*
>       * Set the initial parameters so that the migration could not converge
>       * without throttling.
>       */
>      migrate_set_parameter_int(from, "downtime-limit", 1);
>      migrate_set_parameter_int(from, "max-bandwidth", 100000000); /* ~100Mb/s */
> 
> knock a couple of 0's off that and see if it's happy.

You are right! One 0 is not enough, but a couple makes the test succeed.

Can that change have other (bad) effects on other archs?
IOW can I send a generic patch updating this value, or should I add a 
check on the architecture?

Thanks!

> 
> Dave
> 
>> Per David Gilbert, it might be a threshold requiring tuning:
>> "it could just be the writing is slow on s390 and the migration
>> thread fast; in which case the autocomplete wouldn't be needed.
>> Perhaps we just need to reduce the bandwidth limit."
>>
>> Disable the autoconverge test for now.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   tests/qtest/migration-test.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
>> index 3d6cc83b88..878399666e 100644
>> --- a/tests/qtest/migration-test.c
>> +++ b/tests/qtest/migration-test.c
>> @@ -1181,6 +1181,16 @@ static void test_migrate_auto_converge(void)
>>       MigrateStart *args = migrate_start_new();
>>       QTestState *from, *to;
>>       int64_t remaining, percentage;
>> +    char *travis_arch;
>> +
>> +    /*
>> +     * This test does not work reliably on Travis s390x.
>> +     */
>> +    travis_arch = getenv("TRAVIS_CPU_ARCH");
>> +    if (travis_arch && g_str_equal(travis_arch, "s390x")) {
>> +        g_test_skip("Test does not work reliably on s390x Travis containers.");
>> +        return;
>> +    }
>>   
>>       /*
>>        * We want the test to be stable and as fast as possible.
>> -- 
>> 2.21.1
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

Re: [PATCH 1/4] tests/test-util-filemonitor: Fix Travis-CI $ARCH env variable name

[Daniel P. Berrangé]

On Tue, Mar 17, 2020 at 09:05:38PM +0100, Philippe Mathieu-Daudé wrote:
> While we can find reference of a 'TRAVIS_ARCH' variable in
> the environment and source [1], per the Travis-CI multi-arch
> documentation [2] the variable is named TRAVIS_CPU_ARCH.
> 
> [1] https://github.com/travis-ci/travis-build/blob/v10.0.0/lib/travis/build/bash/travis_setup_env.bash#L39
> [2] https://docs.travis-ci.com/user/multi-cpu-architectures/#identifying-cpu-architecture-of-build-jobs
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/test-util-filemonitor.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/4] tests/test-util-sockets: Skip test on non-x86 Travis containers

[Daniel P. Berrangé]

On Tue, Mar 17, 2020 at 09:05:39PM +0100, Philippe Mathieu-Daudé wrote:
> Similarly to commit 4f370b1098, test-util-sockets fails in
> restricted non-x86 Travis containers since they apparently
> blacklisted some required system calls there.
> Let's simply skip the test if we detect such an environment.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/test-util-sockets.c | 7 +++++++
>  1 file changed, 7 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 3/4] tests/migration: Disable autoconverge test on Travis-CI s390x

[Dr. David Alan Gilbert]

* Philippe Mathieu-Daudé (philmd@redhat.com) wrote:
> On 3/17/20 9:26 PM, Dr. David Alan Gilbert wrote:
> > * Philippe Mathieu-Daudé (philmd@redhat.com) wrote:
> > > This test fails on Travis-CI s390x when configured with --disable-tcg:
> > > 
> > >    $ make check-qtest
> > >      TEST    check-qtest-s390x: tests/qtest/boot-serial-test
> > >    qemu-system-s390x: -accel tcg: invalid accelerator tcg
> > >    qemu-system-s390x: falling back to KVM
> > >      TEST    check-qtest-s390x: tests/qtest/pxe-test
> > >      TEST    check-qtest-s390x: tests/qtest/test-netfilter
> > >      TEST    check-qtest-s390x: tests/qtest/test-filter-mirror
> > >      TEST    check-qtest-s390x: tests/qtest/test-filter-redirector
> > >      TEST    check-qtest-s390x: tests/qtest/drive_del-test
> > >      TEST    check-qtest-s390x: tests/qtest/device-plug-test
> > >      TEST    check-qtest-s390x: tests/qtest/virtio-ccw-test
> > >      TEST    check-qtest-s390x: tests/qtest/cpu-plug-test
> > >      TEST    check-qtest-s390x: tests/qtest/migration-test
> > >    **
> > >    ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
> > >    ERROR - Bail out! ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
> > >    make: *** [tests/Makefile.include:633: check-qtest-s390x] Error 1
> > 
> > Before we take the hammer to it, could you try reducing it's initial
> > bandwidth, the code that says:
> > 
> >      /*
> >       * Set the initial parameters so that the migration could not converge
> >       * without throttling.
> >       */
> >      migrate_set_parameter_int(from, "downtime-limit", 1);
> >      migrate_set_parameter_int(from, "max-bandwidth", 100000000); /* ~100Mb/s */
> > 
> > knock a couple of 0's off that and see if it's happy.
> 
> You are right! One 0 is not enough, but a couple makes the test succeed.
> 
> Can that change have other (bad) effects on other archs?
> IOW can I send a generic patch updating this value, or should I add a check
> on the architecture?

I think generic is fine;  as soon as the code notices that it's
throttling it increases the bandwidth to let it complete, so it should
be OK to have this low.

Dave

> Thanks!
> 
> > 
> > Dave
> > 
> > > Per David Gilbert, it might be a threshold requiring tuning:
> > > "it could just be the writing is slow on s390 and the migration
> > > thread fast; in which case the autocomplete wouldn't be needed.
> > > Perhaps we just need to reduce the bandwidth limit."
> > > 
> > > Disable the autoconverge test for now.
> > > 
> > > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > > ---
> > >   tests/qtest/migration-test.c | 10 ++++++++++
> > >   1 file changed, 10 insertions(+)
> > > 
> > > diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
> > > index 3d6cc83b88..878399666e 100644
> > > --- a/tests/qtest/migration-test.c
> > > +++ b/tests/qtest/migration-test.c
> > > @@ -1181,6 +1181,16 @@ static void test_migrate_auto_converge(void)
> > >       MigrateStart *args = migrate_start_new();
> > >       QTestState *from, *to;
> > >       int64_t remaining, percentage;
> > > +    char *travis_arch;
> > > +
> > > +    /*
> > > +     * This test does not work reliably on Travis s390x.
> > > +     */
> > > +    travis_arch = getenv("TRAVIS_CPU_ARCH");
> > > +    if (travis_arch && g_str_equal(travis_arch, "s390x")) {
> > > +        g_test_skip("Test does not work reliably on s390x Travis containers.");
> > > +        return;
> > > +    }
> > >       /*
> > >        * We want the test to be stable and as fast as possible.
> > > -- 
> > > 2.21.1
> > > 
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK