Re: [Qemu-devel] [PATCH v2 6/7] Acceptance Tests: add variants definition for architectures

[Cleber Rosa <crosa@redhat.com>]

On 10/10/18 6:59 AM, Philippe Mathieu-Daudé wrote:
> On 10/10/2018 01:26, Cleber Rosa wrote:
>> One of the Avocado features relevant to virtualization testing is the
>> ability to reuse tests in different scenarios, known as variants.
>> This adds a JSON based variants file, that can be used to run most
>> tests in a number of different architectures.  It can be run with:
>>
>>    $ avocado run \
>>      --json-variants-load=tests/acceptance/variants/arch.json \
>>      --filter-by-tags='-x86_64' -- tests/acceptance/
>>
>> Currently this covers 5 architectures, resulting in the execution
>> of 25 different combinations.
> 
> IMHO we should add this file with all supported archs, and CANCEL a job
> if there is no preferred machine in Qemu::MACHINE_TYPES.
> 

While your proposal works, I see two issues:

 1) It defines a policy, which will affect all tests, existing and new -
even for tests that don't launch a VM and don't care about the arch.

 2) CANCEL is mostly silent, and the amount of noise in test results
would be quite large.

So, the real problem is that there are other issues when making basic
use of QEMUMachine with other archs.  The idea is to incrementally test
other archs, make the modifications necessary to scripts/qemu.py, and
end up with all architectures covered.

What do you think?

- Cleber.

>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
>>  1 file changed, 92 insertions(+)
>>  create mode 100644 tests/acceptance/variants/arch.json