[kvm-unit-tests PATCH] gitlab-ci.yml: Add new s390x targets to run tests with TCG and KVM accel

[kvm-unit-tests PATCH] gitlab-ci.yml: Add new s390x targets to run tests with TCG and KVM accel

[Marcelo Bandeira Condotta]

From: Marcelo Bandeira Condotta <mcondotta@redhat.com>

A new s390x z15 VM provided by IBM Community Cloud will be used to run
the s390x KVM Unit tests natively with both TCG and KVM accel options.

Signed-off-by: Marcelo Bandeira Condotta <mbandeir@redhat.com>
---
 .gitlab-ci.yml | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d97e27e..bc7a115 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -155,3 +155,31 @@ cirrus-ci-macos-i386:
 
 cirrus-ci-macos-x86-64:
  <<: *cirrus_build_job_definition
+
+test-s390x-tcg:
+  stage: test
+  before_script: []
+  tags:
+    - s390x-z15-vm
+  script:
+    - ./configure --arch=s390x
+    - make -j2
+    - ACCEL=tcg ./run_tests.sh
+     selftest-setup intercept emulator sieve skey diag10 diag308 vector diag288
+     stsi sclp-1g sclp-3g
+     | tee results.txt
+    - if grep -q FAIL results.txt ; then exit 1 ; fi
+
+test-s390x-kvm:
+  stage: test
+  before_script: []
+  tags:
+    - s390x-z15-vm
+  script:
+    - ./configure --arch=s390x
+    - make -j2
+    - ACCEL=kvm ./run_tests.sh
+     selftest-setup intercept emulator sieve skey diag10 diag308 vector diag288
+     stsi sclp-1g sclp-3g
+     | tee results.txt
+    - if grep -q FAIL results.txt ; then exit 1 ; fi
-- 
2.26.2

Re: [kvm-unit-tests PATCH] gitlab-ci.yml: Add new s390x targets to run tests with TCG and KVM accel

[Paolo Bonzini]

On 08/02/21 16:02, Marcelo Bandeira Condotta wrote:
> From: Marcelo Bandeira Condotta <mcondotta@redhat.com>
> 
> A new s390x z15 VM provided by IBM Community Cloud will be used to run
> the s390x KVM Unit tests natively with both TCG and KVM accel options.
> 
> Signed-off-by: Marcelo Bandeira Condotta <mbandeir@redhat.com>
> ---
>   .gitlab-ci.yml | 28 ++++++++++++++++++++++++++++
>   1 file changed, 28 insertions(+)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index d97e27e..bc7a115 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -155,3 +155,31 @@ cirrus-ci-macos-i386:
>   
>   cirrus-ci-macos-x86-64:
>    <<: *cirrus_build_job_definition
> +
> +test-s390x-tcg:
> +  stage: test
> +  before_script: []
> +  tags:
> +    - s390x-z15-vm
> +  script:
> +    - ./configure --arch=s390x
> +    - make -j2
> +    - ACCEL=tcg ./run_tests.sh
> +     selftest-setup intercept emulator sieve skey diag10 diag308 vector diag288
> +     stsi sclp-1g sclp-3g
> +     | tee results.txt
> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
> +
> +test-s390x-kvm:
> +  stage: test
> +  before_script: []
> +  tags:
> +    - s390x-z15-vm
> +  script:
> +    - ./configure --arch=s390x
> +    - make -j2
> +    - ACCEL=kvm ./run_tests.sh
> +     selftest-setup intercept emulator sieve skey diag10 diag308 vector diag288
> +     stsi sclp-1g sclp-3g
> +     | tee results.txt
> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
> 

So it will have a custom runner?  That's nice!

Do you have an example run already?

Paolo

Re: [kvm-unit-tests PATCH] gitlab-ci.yml: Add new s390x targets to run tests with TCG and KVM accel

[Thomas Huth]

On 08/02/2021 16.07, Paolo Bonzini wrote:
> On 08/02/21 16:02, Marcelo Bandeira Condotta wrote:
>> From: Marcelo Bandeira Condotta <mcondotta@redhat.com>
>>
>> A new s390x z15 VM provided by IBM Community Cloud will be used to run
>> the s390x KVM Unit tests natively with both TCG and KVM accel options.
>>
>> Signed-off-by: Marcelo Bandeira Condotta <mbandeir@redhat.com>
>> ---
>>   .gitlab-ci.yml | 28 ++++++++++++++++++++++++++++
>>   1 file changed, 28 insertions(+)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index d97e27e..bc7a115 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -155,3 +155,31 @@ cirrus-ci-macos-i386:
>>   cirrus-ci-macos-x86-64:
>>    <<: *cirrus_build_job_definition
>> +
>> +test-s390x-tcg:
>> +  stage: test
>> +  before_script: []
>> +  tags:
>> +    - s390x-z15-vm
>> +  script:
>> +    - ./configure --arch=s390x
>> +    - make -j2
>> +    - ACCEL=tcg ./run_tests.sh
>> +     selftest-setup intercept emulator sieve skey diag10 diag308 vector 
>> diag288
>> +     stsi sclp-1g sclp-3g
>> +     | tee results.txt
>> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
>> +
>> +test-s390x-kvm:
>> +  stage: test
>> +  before_script: []
>> +  tags:
>> +    - s390x-z15-vm
>> +  script:
>> +    - ./configure --arch=s390x
>> +    - make -j2
>> +    - ACCEL=kvm ./run_tests.sh
>> +     selftest-setup intercept emulator sieve skey diag10 diag308 vector 
>> diag288
>> +     stsi sclp-1g sclp-3g
>> +     | tee results.txt
>> +    - if grep -q FAIL results.txt ; then exit 1 ; fi

Acked-by: Thomas Huth <thuth@redhat.com>

> 
> So it will have a custom runner?  That's nice!
> 
> Do you have an example run already?

I've been in touch with Marcelo during the past days already, and I've 
already registered the runner that he set up on the s390x machine, so it 
should theoretically work now once this patch has been merged.

  Thomas

Re: [kvm-unit-tests PATCH] gitlab-ci.yml: Add new s390x targets to run tests with TCG and KVM accel

[Paolo Bonzini]

On 08/02/21 16:13, Thomas Huth wrote:
> On 08/02/2021 16.07, Paolo Bonzini wrote:
>> On 08/02/21 16:02, Marcelo Bandeira Condotta wrote:
>>> From: Marcelo Bandeira Condotta <mcondotta@redhat.com>
>>>
>>> A new s390x z15 VM provided by IBM Community Cloud will be used to run
>>> the s390x KVM Unit tests natively with both TCG and KVM accel options.
>>>
>>> Signed-off-by: Marcelo Bandeira Condotta <mbandeir@redhat.com>
>>> ---
>>>   .gitlab-ci.yml | 28 ++++++++++++++++++++++++++++
>>>   1 file changed, 28 insertions(+)
>>>
>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>> index d97e27e..bc7a115 100644
>>> --- a/.gitlab-ci.yml
>>> +++ b/.gitlab-ci.yml
>>> @@ -155,3 +155,31 @@ cirrus-ci-macos-i386:
>>>   cirrus-ci-macos-x86-64:
>>>    <<: *cirrus_build_job_definition
>>> +
>>> +test-s390x-tcg:
>>> +  stage: test
>>> +  before_script: []
>>> +  tags:
>>> +    - s390x-z15-vm
>>> +  script:
>>> +    - ./configure --arch=s390x
>>> +    - make -j2
>>> +    - ACCEL=tcg ./run_tests.sh
>>> +     selftest-setup intercept emulator sieve skey diag10 diag308 
>>> vector diag288
>>> +     stsi sclp-1g sclp-3g
>>> +     | tee results.txt
>>> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
>>> +
>>> +test-s390x-kvm:
>>> +  stage: test
>>> +  before_script: []
>>> +  tags:
>>> +    - s390x-z15-vm
>>> +  script:
>>> +    - ./configure --arch=s390x
>>> +    - make -j2
>>> +    - ACCEL=kvm ./run_tests.sh
>>> +     selftest-setup intercept emulator sieve skey diag10 diag308 
>>> vector diag288
>>> +     stsi sclp-1g sclp-3g
>>> +     | tee results.txt
>>> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
> 
> Acked-by: Thomas Huth <thuth@redhat.com>
> 
>>
>> So it will have a custom runner?  That's nice!
>>
>> Do you have an example run already?
> 
> I've been in touch with Marcelo during the past days already, and I've 
> already registered the runner that he set up on the s390x machine, so it 
> should theoretically work now once this patch has been merged.

What's the reason to add test-s390x-tcg?  It would really cover only a 
different TCG backend.

Paolo

Re: [kvm-unit-tests PATCH] gitlab-ci.yml: Add new s390x targets to run tests with TCG and KVM accel

[Paolo Bonzini]

On 08/02/21 16:02, Marcelo Bandeira Condotta wrote:
> From: Marcelo Bandeira Condotta <mcondotta@redhat.com>
> 
> A new s390x z15 VM provided by IBM Community Cloud will be used to run
> the s390x KVM Unit tests natively with both TCG and KVM accel options.
> 
> Signed-off-by: Marcelo Bandeira Condotta <mbandeir@redhat.com>
> ---
>   .gitlab-ci.yml | 28 ++++++++++++++++++++++++++++
>   1 file changed, 28 insertions(+)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index d97e27e..bc7a115 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -155,3 +155,31 @@ cirrus-ci-macos-i386:
>   
>   cirrus-ci-macos-x86-64:
>    <<: *cirrus_build_job_definition
> +
> +test-s390x-tcg:
> +  stage: test
> +  before_script: []
> +  tags:
> +    - s390x-z15-vm
> +  script:
> +    - ./configure --arch=s390x
> +    - make -j2
> +    - ACCEL=tcg ./run_tests.sh
> +     selftest-setup intercept emulator sieve skey diag10 diag308 vector diag288
> +     stsi sclp-1g sclp-3g
> +     | tee results.txt
> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
> +
> +test-s390x-kvm:
> +  stage: test
> +  before_script: []
> +  tags:
> +    - s390x-z15-vm
> +  script:
> +    - ./configure --arch=s390x
> +    - make -j2
> +    - ACCEL=kvm ./run_tests.sh
> +     selftest-setup intercept emulator sieve skey diag10 diag308 vector diag288
> +     stsi sclp-1g sclp-3g
> +     | tee results.txt
> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
> 

There is still a problem, in that every contributor's CI runs will mark 
the s390 as job as pending.  I think you should use variables to enable 
these tests only on forks that want them.

While at it, I agree with Thomas on testing {clang,GCC} on KVM rather 
than GCC on {KVM,TCG}.

Thanks!

Paolo

Re: [kvm-unit-tests PATCH] gitlab-ci.yml: Add new s390x targets to run tests with TCG and KVM accel

[Thomas Huth]

On 08/02/2021 16.32, Paolo Bonzini wrote:
> On 08/02/21 16:13, Thomas Huth wrote:
>> On 08/02/2021 16.07, Paolo Bonzini wrote:
>>> On 08/02/21 16:02, Marcelo Bandeira Condotta wrote:
>>>> From: Marcelo Bandeira Condotta <mcondotta@redhat.com>
>>>>
>>>> A new s390x z15 VM provided by IBM Community Cloud will be used to run
>>>> the s390x KVM Unit tests natively with both TCG and KVM accel options.
>>>>
>>>> Signed-off-by: Marcelo Bandeira Condotta <mbandeir@redhat.com>
>>>> ---
>>>>   .gitlab-ci.yml | 28 ++++++++++++++++++++++++++++
>>>>   1 file changed, 28 insertions(+)
>>>>
>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>> index d97e27e..bc7a115 100644
>>>> --- a/.gitlab-ci.yml
>>>> +++ b/.gitlab-ci.yml
>>>> @@ -155,3 +155,31 @@ cirrus-ci-macos-i386:
>>>>   cirrus-ci-macos-x86-64:
>>>>    <<: *cirrus_build_job_definition
>>>> +
>>>> +test-s390x-tcg:
>>>> +  stage: test
>>>> +  before_script: []
>>>> +  tags:
>>>> +    - s390x-z15-vm
>>>> +  script:
>>>> +    - ./configure --arch=s390x
>>>> +    - make -j2
>>>> +    - ACCEL=tcg ./run_tests.sh
>>>> +     selftest-setup intercept emulator sieve skey diag10 diag308 vector 
>>>> diag288
>>>> +     stsi sclp-1g sclp-3g
>>>> +     | tee results.txt
>>>> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
>>>> +
>>>> +test-s390x-kvm:
>>>> +  stage: test
>>>> +  before_script: []
>>>> +  tags:
>>>> +    - s390x-z15-vm
>>>> +  script:
>>>> +    - ./configure --arch=s390x
>>>> +    - make -j2
>>>> +    - ACCEL=kvm ./run_tests.sh
>>>> +     selftest-setup intercept emulator sieve skey diag10 diag308 vector 
>>>> diag288
>>>> +     stsi sclp-1g sclp-3g
>>>> +     | tee results.txt
>>>> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
>>
>> Acked-by: Thomas Huth <thuth@redhat.com>
>>
>>>
>>> So it will have a custom runner?  That's nice!
>>>
>>> Do you have an example run already?
>>
>> I've been in touch with Marcelo during the past days already, and I've 
>> already registered the runner that he set up on the s390x machine, so it 
>> should theoretically work now once this patch has been merged.
> 
> What's the reason to add test-s390x-tcg?  It would really cover only a 
> different TCG backend.

I've removed the tcg target from the patch, changed it so that it now runs 
all s390x tests (the ones that do not work are skipped automatically), and 
restricted the job to projects where a variable called 
S390X_Z15_RUNNER_AVAILABLE has been set (so that the job does not get stuck 
on systems where the runner is not available), and finally pushed it to the 
master branch, so we should have automatic test coverage on s390x now. Thank 
you very much, Marcelo!

  Thomas

Re: [kvm-unit-tests PATCH] gitlab-ci.yml: Add new s390x targets to run tests with TCG and KVM accel

[Thomas Huth]

On 08/02/2021 18.54, Thomas Huth wrote:
> On 08/02/2021 16.32, Paolo Bonzini wrote:
>> On 08/02/21 16:13, Thomas Huth wrote:
>>> On 08/02/2021 16.07, Paolo Bonzini wrote:
>>>> On 08/02/21 16:02, Marcelo Bandeira Condotta wrote:
>>>>> From: Marcelo Bandeira Condotta <mcondotta@redhat.com>
>>>>>
>>>>> A new s390x z15 VM provided by IBM Community Cloud will be used to run
>>>>> the s390x KVM Unit tests natively with both TCG and KVM accel options.
>>>>>
>>>>> Signed-off-by: Marcelo Bandeira Condotta <mbandeir@redhat.com>
>>>>> ---
>>>>>   .gitlab-ci.yml | 28 ++++++++++++++++++++++++++++
>>>>>   1 file changed, 28 insertions(+)
>>>>>
>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>>> index d97e27e..bc7a115 100644
>>>>> --- a/.gitlab-ci.yml
>>>>> +++ b/.gitlab-ci.yml
>>>>> @@ -155,3 +155,31 @@ cirrus-ci-macos-i386:
>>>>>   cirrus-ci-macos-x86-64:
>>>>>    <<: *cirrus_build_job_definition
>>>>> +
>>>>> +test-s390x-tcg:
>>>>> +  stage: test
>>>>> +  before_script: []
>>>>> +  tags:
>>>>> +    - s390x-z15-vm
>>>>> +  script:
>>>>> +    - ./configure --arch=s390x
>>>>> +    - make -j2
>>>>> +    - ACCEL=tcg ./run_tests.sh
>>>>> +     selftest-setup intercept emulator sieve skey diag10 diag308 
>>>>> vector diag288
>>>>> +     stsi sclp-1g sclp-3g
>>>>> +     | tee results.txt
>>>>> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
>>>>> +
>>>>> +test-s390x-kvm:
>>>>> +  stage: test
>>>>> +  before_script: []
>>>>> +  tags:
>>>>> +    - s390x-z15-vm
>>>>> +  script:
>>>>> +    - ./configure --arch=s390x
>>>>> +    - make -j2
>>>>> +    - ACCEL=kvm ./run_tests.sh
>>>>> +     selftest-setup intercept emulator sieve skey diag10 diag308 
>>>>> vector diag288
>>>>> +     stsi sclp-1g sclp-3g
>>>>> +     | tee results.txt
>>>>> +    - if grep -q FAIL results.txt ; then exit 1 ; fi
>>>
>>> Acked-by: Thomas Huth <thuth@redhat.com>
>>>
>>>>
>>>> So it will have a custom runner?  That's nice!
>>>>
>>>> Do you have an example run already?
>>>
>>> I've been in touch with Marcelo during the past days already, and I've 
>>> already registered the runner that he set up on the s390x machine, so it 
>>> should theoretically work now once this patch has been merged.
>>
>> What's the reason to add test-s390x-tcg?  It would really cover only a 
>> different TCG backend.
> 
> I've removed the tcg target from the patch, changed it so that it now runs 
> all s390x tests [...]
... and of course one of the tests ("smp") is flaky and succeeded on the
staging branch, but failed on the master branch. Sigh.
I've pushed yet another patch on top to only run the solid tests now:

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 13c798f..b7c0571 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -163,7 +163,10 @@ s390x-kvm:
   script:
    - ./configure --arch=s390x
    - make -j$(nproc)
-  - ACCEL=kvm ./run_tests.sh | tee results.txt
+  - ACCEL=kvm ./run_tests.sh
+      selftest-setup intercept emulator sieve sthyi skey diag10 diag308 pfmf
+      cmm vector gs iep cpumodel diag288 stsi sclp-1g sclp-3g css skrf sie
+      | tee results.txt
    - grep -q PASS results.txt && ! grep -q FAIL results.txt
   only:
    variables:


  Thomas