Re: [kvm-unit-tests PATCH] travis.yml: Enable running of tests with TCG

[Thomas Huth <thuth@redhat.com>]

On 06/09/2019 12.02, David Hildenbrand wrote:
> On 06.09.19 11:54, Thomas Huth wrote:
>> On 06/09/2019 11.43, David Hildenbrand wrote:
>>> On 30.08.19 20:45, Thomas Huth wrote:
>>>> Currently the tests at the end of the .travis.yml script are ignored,
>>>> since we can not use KVM in the Travis containers. But we can actually
>>>> run of some of the kvm-unit-tests with TCG instead, to make sure that
>>>> the binaries are not completely broken.
>>>> Thus introduce a new TESTS variable that lists the tests which we can
>>>> run with TCG. Unfortunately, the ppc64 and s390x QEMUs in Ubuntu also
>>>> need some extra love: The ppc64 version only works with the additional
>>>> "cap-htm=off" setting. And the s390x package lacks the firmware and
>>>> refuses to work unless we provide a fake firmware file here. Any file
>>>> works since the firmware is skipped when "-kernel" is used, so we can
>>>> simply use one of the pre-existing files in the source tree.
>>>>
>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>> ---
>>>>  .travis.yml | 19 ++++++++++++++++++-
>>>>  1 file changed, 18 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/.travis.yml b/.travis.yml
>>>> index a4a165d..6c14953 100644
>>>> --- a/.travis.yml
>>>> +++ b/.travis.yml
>>>> @@ -20,24 +20,40 @@ env:
>>>>    matrix:
>>>>      - CONFIG=""
>>>>        BUILD_DIR="."
>>>> +      TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
>>>> +             vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
>>>>      - CONFIG=""
>>>>        BUILD_DIR="x86-builddir"
>>>> +      TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
>>>> +             tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
>>>>      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
>>>>        BUILD_DIR="."
>>>> +      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
>>>>      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
>>>>        BUILD_DIR="arm-buildir"
>>>> +      TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
>>>>      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
>>>>        BUILD_DIR="."
>>>> +      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
>>>>      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
>>>>        BUILD_DIR="arm64-buildir"
>>>> +      TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
>>>>      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
>>>>        BUILD_DIR="."
>>>> +      TESTS="spapr_hcall emulator rtas-set-time-of-day"
>>>> +      ACCEL="tcg,cap-htm=off"
>>>>      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
>>>>        BUILD_DIR="ppc64le-buildir"
>>>> +      TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
>>>> +      ACCEL="tcg,cap-htm=off"
>>>>      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
>>>>        BUILD_DIR="."
>>>> +      TESTS="diag10 diag308"
>>>> +      ACCEL="tcg,firmware=s390x/run"
>>>>      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
>>>>        BUILD_DIR="s390x-builddir"
>>>> +      TESTS="sieve"
>>>> +      ACCEL="tcg,firmware=s390x/run"
>>>
>>> What about the other s390x tests? (is the QEMU binary too old to make
>>> them pass?)
>>
>> Yes. All the others are failing.
> 
> So the QEMU versions is that old that not even the selftest works? :/ Is
> there a more recent distribution we can use?

I already tested with "dist: bionic", and that's all I've got to get
working there. According to
https://docs.travis-ci.com/user/reference/overview/ there are no newer
distros available yet.

>>> The issue with TCG is that you can easily get false negatives in case
>>> the QEMU binary changes (e.g., new Ubuntu release).
>>
>> That should not be a big deal, since the Ubuntu version is locked to a
>> certain release in the travis.yml file. So we should not get a new
>> version of QEMU by accident here, only if we explicitly update the
>> "dist:" line in the yml file.
> 
> Or if the dist updates QEMU. Not sure how often that happens in QEMU.
> (but as not even the selfest passes, it cannot really get any worse :D )

I don't think that they will update the QEMU version during the
life-cycle of a certain release - they likely only apply some bug fixes,
which won't change much of the behavior here. Remember that
qemu-system-s390x is almost completely broken in these Ubuntu versions
since they do not ship the firmware image in the package. I only got
something running by luck with that "firmware=..." trick.

 Thomas