Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test - Philippe Mathieu-Daudé

From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
To: "Alex Bennée" <alex.bennee@linaro.org>,
	"Peter Maydell" <peter.maydell@linaro.org>
Cc: QEMU Developers <qemu-devel@nongnu.org>
Subject: Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
Date: Fri, 8 May 2020 15:07:04 +0200	[thread overview]
Message-ID: <15b8c809-d8f8-4ca0-9815-d35a87fc5f83@redhat.com> (raw)
In-Reply-To: <871rnv5vva.fsf@linaro.org>

On 5/7/20 10:32 PM, Alex Bennée wrote:
> 
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
>> On Thu, 7 May 2020 at 19:38, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>>
>>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>>
>>>> The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run.
>>>> Add a possibility to users to skip this particular test, by setting
>>>> the AVOCADO_SKIP_SLOW_TESTS environment variable:
>>>>
>>>>    $ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance
>>>>    ...
>>>>      (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min
>>>>    ...
>>>
>>> 7m = 420s - I'm seeing:
>>>
>>>    hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg"
>>>    Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
>>>      Time (mean ± σ):     162.179 s ±  3.138 s    [User: 204.726 s, System: 9.663 s]
>>>      Range (min … max):   158.651 s … 170.036 s    10 runs
>>>
>>> Is this on very slow hardware?
>>
>> Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz 16-core
>> clang sanitizer build, which probably slows it down a bit.
>>
>> But even 200 seconds is an order of magnitude slower than any
>> of the other tests that check-acceptance runs. I think we
>> should be aiming for tests here to be ~30 seconds at most,
>> or the whole thing will take forever by the time we have
>> decent coverage of most machines.
> 
> I think we could say the same for a whole bunch of the tests that are
> based on full distro downloads. What is this one running under the hood
> anyway?

Download time is now counted separately, this was one big feature 
request after the last KVM forum.

The second request was to improve test timing management, like having 
different timers tracking if the test is behaving as expected. I.e. you 
might want to boot a VM and run 'ping -c 3'. The whole test shouldn't 
take more than 7min, but we want to be sure the 'ping' command doesn't 
take more than 3.x seconds. I haven't heard about a such feature ready yet.

I really hope you are using a common cache location and now download the 
artifacts each time...

> 
> 
>>
>> thanks
>> -- PMM
> 
> 