Re: [kvm-unit-tests PATCH] travis.yml: Enable running of tests with TCG

[Thomas Huth <thuth@redhat.com>]

On 06/09/2019 11.43, David Hildenbrand wrote:
> On 30.08.19 20:45, Thomas Huth wrote:
>> Currently the tests at the end of the .travis.yml script are ignored,
>> since we can not use KVM in the Travis containers. But we can actually
>> run of some of the kvm-unit-tests with TCG instead, to make sure that
>> the binaries are not completely broken.
>> Thus introduce a new TESTS variable that lists the tests which we can
>> run with TCG. Unfortunately, the ppc64 and s390x QEMUs in Ubuntu also
>> need some extra love: The ppc64 version only works with the additional
>> "cap-htm=off" setting. And the s390x package lacks the firmware and
>> refuses to work unless we provide a fake firmware file here. Any file
>> works since the firmware is skipped when "-kernel" is used, so we can
>> simply use one of the pre-existing files in the source tree.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  .travis.yml | 19 ++++++++++++++++++-
>>  1 file changed, 18 insertions(+), 1 deletion(-)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index a4a165d..6c14953 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -20,24 +20,40 @@ env:
>>    matrix:
>>      - CONFIG=""
>>        BUILD_DIR="."
>> +      TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
>> +             vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
>>      - CONFIG=""
>>        BUILD_DIR="x86-builddir"
>> +      TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
>> +             tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
>>      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
>>        BUILD_DIR="."
>> +      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
>>      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
>>        BUILD_DIR="arm-buildir"
>> +      TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
>>      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
>>        BUILD_DIR="."
>> +      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
>>      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
>>        BUILD_DIR="arm64-buildir"
>> +      TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
>>      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
>>        BUILD_DIR="."
>> +      TESTS="spapr_hcall emulator rtas-set-time-of-day"
>> +      ACCEL="tcg,cap-htm=off"
>>      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
>>        BUILD_DIR="ppc64le-buildir"
>> +      TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
>> +      ACCEL="tcg,cap-htm=off"
>>      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
>>        BUILD_DIR="."
>> +      TESTS="diag10 diag308"
>> +      ACCEL="tcg,firmware=s390x/run"
>>      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
>>        BUILD_DIR="s390x-builddir"
>> +      TESTS="sieve"
>> +      ACCEL="tcg,firmware=s390x/run"
> 
> What about the other s390x tests? (is the QEMU binary too old to make
> them pass?)

Yes. All the others are failing.

> The issue with TCG is that you can easily get false negatives in case
> the QEMU binary changes (e.g., new Ubuntu release).

That should not be a big deal, since the Ubuntu version is locked to a
certain release in the travis.yml file. So we should not get a new
version of QEMU by accident here, only if we explicitly update the
"dist:" line in the yml file.

> "to make sure that the binaries are not completely broken" - while I
> understand the intuition behind that, I wonder if this is relevant in
> practice
Well, the test coverage here is certainly far from being perfect, but
it's still better than nothing. Imagine one of the patches changes
common code in the lib/ folder and it only works on one architecture
that the author tested. Chances are quite good that we detect that
problem with this patch now.

 Thomas