* [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
@ 2020-05-07 16:22 Philippe Mathieu-Daudé
2020-05-07 18:38 ` Alex Bennée
2020-05-09 13:18 ` Lukas Straub
0 siblings, 2 replies; 9+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-05-07 16:22 UTC (permalink / raw)
To: qemu-devel; +Cc: Peter Maydell, Philippe Mathieu-Daudé
The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run.
Add a possibility to users to skip this particular test, by setting
the AVOCADO_SKIP_SLOW_TESTS environment variable:
$ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance
...
(05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min
...
Reported-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
tests/acceptance/boot_linux.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
index 075a386300..a8df50d769 100644
--- a/tests/acceptance/boot_linux.py
+++ b/tests/acceptance/boot_linux.py
@@ -15,6 +15,7 @@
from qemu.accel import kvm_available
from qemu.accel import tcg_available
+from avocado import skipIf
from avocado.utils import cloudinit
from avocado.utils import network
from avocado.utils import vmimage
@@ -159,6 +160,7 @@ def add_common_args(self):
self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
+ @skipIf(os.getenv('AVOCADO_SKIP_SLOW_TESTS'), 'Test takes >7min')
def test_virt_tcg(self):
"""
:avocado: tags=accel:tcg
--
2.21.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
2020-05-07 16:22 [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test Philippe Mathieu-Daudé
@ 2020-05-07 18:38 ` Alex Bennée
2020-05-07 18:45 ` Peter Maydell
2020-05-09 13:18 ` Lukas Straub
1 sibling, 1 reply; 9+ messages in thread
From: Alex Bennée @ 2020-05-07 18:38 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: Peter Maydell, qemu-devel
Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run.
> Add a possibility to users to skip this particular test, by setting
> the AVOCADO_SKIP_SLOW_TESTS environment variable:
>
> $ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance
> ...
> (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min
> ...
7m = 420s - I'm seeing:
hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg"
Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
Time (mean ± σ): 162.179 s ± 3.138 s [User: 204.726 s, System: 9.663 s]
Range (min … max): 158.651 s … 170.036 s 10 runs
Is this on very slow hardware?
>
> Reported-by: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> tests/acceptance/boot_linux.py | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
> index 075a386300..a8df50d769 100644
> --- a/tests/acceptance/boot_linux.py
> +++ b/tests/acceptance/boot_linux.py
> @@ -15,6 +15,7 @@
> from qemu.accel import kvm_available
> from qemu.accel import tcg_available
>
> +from avocado import skipIf
> from avocado.utils import cloudinit
> from avocado.utils import network
> from avocado.utils import vmimage
> @@ -159,6 +160,7 @@ def add_common_args(self):
> self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
>
> + @skipIf(os.getenv('AVOCADO_SKIP_SLOW_TESTS'), 'Test takes >7min')
> def test_virt_tcg(self):
> """
> :avocado: tags=accel:tcg
--
Alex Bennée
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
2020-05-07 18:38 ` Alex Bennée
@ 2020-05-07 18:45 ` Peter Maydell
2020-05-07 20:32 ` Alex Bennée
0 siblings, 1 reply; 9+ messages in thread
From: Peter Maydell @ 2020-05-07 18:45 UTC (permalink / raw)
To: Alex Bennée; +Cc: Philippe Mathieu-Daudé, QEMU Developers
On Thu, 7 May 2020 at 19:38, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>
> > The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run.
> > Add a possibility to users to skip this particular test, by setting
> > the AVOCADO_SKIP_SLOW_TESTS environment variable:
> >
> > $ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance
> > ...
> > (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min
> > ...
>
> 7m = 420s - I'm seeing:
>
> hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg"
> Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
> Time (mean ± σ): 162.179 s ± 3.138 s [User: 204.726 s, System: 9.663 s]
> Range (min … max): 158.651 s … 170.036 s 10 runs
>
> Is this on very slow hardware?
Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz 16-core
clang sanitizer build, which probably slows it down a bit.
But even 200 seconds is an order of magnitude slower than any
of the other tests that check-acceptance runs. I think we
should be aiming for tests here to be ~30 seconds at most,
or the whole thing will take forever by the time we have
decent coverage of most machines.
thanks
-- PMM
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
2020-05-07 18:45 ` Peter Maydell
@ 2020-05-07 20:32 ` Alex Bennée
2020-05-08 13:07 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 9+ messages in thread
From: Alex Bennée @ 2020-05-07 20:32 UTC (permalink / raw)
To: Peter Maydell; +Cc: Philippe Mathieu-Daudé, QEMU Developers
Peter Maydell <peter.maydell@linaro.org> writes:
> On Thu, 7 May 2020 at 19:38, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>
>> > The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run.
>> > Add a possibility to users to skip this particular test, by setting
>> > the AVOCADO_SKIP_SLOW_TESTS environment variable:
>> >
>> > $ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance
>> > ...
>> > (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min
>> > ...
>>
>> 7m = 420s - I'm seeing:
>>
>> hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg"
>> Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
>> Time (mean ± σ): 162.179 s ± 3.138 s [User: 204.726 s, System: 9.663 s]
>> Range (min … max): 158.651 s … 170.036 s 10 runs
>>
>> Is this on very slow hardware?
>
> Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz 16-core
> clang sanitizer build, which probably slows it down a bit.
>
> But even 200 seconds is an order of magnitude slower than any
> of the other tests that check-acceptance runs. I think we
> should be aiming for tests here to be ~30 seconds at most,
> or the whole thing will take forever by the time we have
> decent coverage of most machines.
I think we could say the same for a whole bunch of the tests that are
based on full distro downloads. What is this one running under the hood
anyway?
>
> thanks
> -- PMM
--
Alex Bennée
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
2020-05-07 20:32 ` Alex Bennée
@ 2020-05-08 13:07 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 9+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-05-08 13:07 UTC (permalink / raw)
To: Alex Bennée, Peter Maydell; +Cc: QEMU Developers
On 5/7/20 10:32 PM, Alex Bennée wrote:
>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
>> On Thu, 7 May 2020 at 19:38, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>>
>>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>>
>>>> The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run.
>>>> Add a possibility to users to skip this particular test, by setting
>>>> the AVOCADO_SKIP_SLOW_TESTS environment variable:
>>>>
>>>> $ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance
>>>> ...
>>>> (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min
>>>> ...
>>>
>>> 7m = 420s - I'm seeing:
>>>
>>> hyperfine "./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg"
>>> Benchmark #1: ./tests/venv/bin/avocado run ./tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg
>>> Time (mean ± σ): 162.179 s ± 3.138 s [User: 204.726 s, System: 9.663 s]
>>> Range (min … max): 158.651 s … 170.036 s 10 runs
>>>
>>> Is this on very slow hardware?
>>
>> Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz 16-core
>> clang sanitizer build, which probably slows it down a bit.
>>
>> But even 200 seconds is an order of magnitude slower than any
>> of the other tests that check-acceptance runs. I think we
>> should be aiming for tests here to be ~30 seconds at most,
>> or the whole thing will take forever by the time we have
>> decent coverage of most machines.
>
> I think we could say the same for a whole bunch of the tests that are
> based on full distro downloads. What is this one running under the hood
> anyway?
Download time is now counted separately, this was one big feature
request after the last KVM forum.
The second request was to improve test timing management, like having
different timers tracking if the test is behaving as expected. I.e. you
might want to boot a VM and run 'ping -c 3'. The whole test shouldn't
take more than 7min, but we want to be sure the 'ping' command doesn't
take more than 3.x seconds. I haven't heard about a such feature ready yet.
I really hope you are using a common cache location and now download the
artifacts each time...
>
>
>>
>> thanks
>> -- PMM
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
2020-05-07 16:22 [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test Philippe Mathieu-Daudé
2020-05-07 18:38 ` Alex Bennée
@ 2020-05-09 13:18 ` Lukas Straub
2020-05-11 9:11 ` Peter Maydell
1 sibling, 1 reply; 9+ messages in thread
From: Lukas Straub @ 2020-05-09 13:18 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: Peter Maydell, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 1695 bytes --]
On Thu, 7 May 2020 18:22:35 +0200
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> The BootLinuxAarch64.test_virt_tcg is reported to take >7min to run.
> Add a possibility to users to skip this particular test, by setting
> the AVOCADO_SKIP_SLOW_TESTS environment variable:
>
> $ AVOCADO_SKIP_SLOW_TESTS=please make check-acceptance
> ...
> (05/88) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: SKIP: Test takes >7min
> ...
>
> Reported-by: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> tests/acceptance/boot_linux.py | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
> index 075a386300..a8df50d769 100644
> --- a/tests/acceptance/boot_linux.py
> +++ b/tests/acceptance/boot_linux.py
> @@ -15,6 +15,7 @@
> from qemu.accel import kvm_available
> from qemu.accel import tcg_available
>
> +from avocado import skipIf
> from avocado.utils import cloudinit
> from avocado.utils import network
> from avocado.utils import vmimage
> @@ -159,6 +160,7 @@ def add_common_args(self):
> self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
>
> + @skipIf(os.getenv('AVOCADO_SKIP_SLOW_TESTS'), 'Test takes >7min')
> def test_virt_tcg(self):
> """
> :avocado: tags=accel:tcg
Hi,
Why not simply add slow tag to the test. Like:
:avocado: tags=slow
The slow tests can then be skipped with
$ make check-acceptance AVOCADO_TAGS='-t -slow'
Regards,
Lukas Straub
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
2020-05-09 13:18 ` Lukas Straub
@ 2020-05-11 9:11 ` Peter Maydell
2020-05-11 16:51 ` Lukas Straub
0 siblings, 1 reply; 9+ messages in thread
From: Peter Maydell @ 2020-05-11 9:11 UTC (permalink / raw)
To: Lukas Straub; +Cc: Philippe Mathieu-Daudé, QEMU Developers
On Sat, 9 May 2020 at 14:18, Lukas Straub <lukasstraub2@web.de> wrote:
> Hi,
> Why not simply add slow tag to the test. Like:
> :avocado: tags=slow
>
> The slow tests can then be skipped with
> $ make check-acceptance AVOCADO_TAGS='-t -slow'
Is it possible to have the default be "do the fast stuff"
and only do the slow stuff if the user asks? That's the
way round that we do the iotests, I think.
thanks
-- PMM
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
2020-05-11 9:11 ` Peter Maydell
@ 2020-05-11 16:51 ` Lukas Straub
2020-05-26 9:14 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 9+ messages in thread
From: Lukas Straub @ 2020-05-11 16:51 UTC (permalink / raw)
To: Peter Maydell; +Cc: Philippe Mathieu-Daudé, QEMU Developers
[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]
On Mon, 11 May 2020 10:11:20 +0100
Peter Maydell <peter.maydell@linaro.org> wrote:
> On Sat, 9 May 2020 at 14:18, Lukas Straub <lukasstraub2@web.de> wrote:
> > Hi,
> > Why not simply add slow tag to the test. Like:
> > :avocado: tags=slow
> >
> > The slow tests can then be skipped with
> > $ make check-acceptance AVOCADO_TAGS='-t -slow'
>
> Is it possible to have the default be "do the fast stuff"
> and only do the slow stuff if the user asks? That's the
> way round that we do the iotests, I think.
I'd set AVOCADO_TAGS to '-t -slow' by default, but now that I look at the Makefile it's not that simple:
AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
We'd have to change the tests to skip if their target was not built. Than we can use tags for uses like this (and more like net, disk, ... tags). That would make it easier to filter the tests one wants to run: https://avocado-framework.readthedocs.io/en/78.0/guides/user/chapters/tags.html
Regards,
Lukas Straub
> thanks
> -- PMM
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test
2020-05-11 16:51 ` Lukas Straub
@ 2020-05-26 9:14 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 9+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-05-26 9:14 UTC (permalink / raw)
To: Lukas Straub, Peter Maydell
Cc: Alex Bennée, QEMU Developers, Cleber Rosa
+Cleber
On 5/11/20 6:51 PM, Lukas Straub wrote:
> On Mon, 11 May 2020 10:11:20 +0100
> Peter Maydell <peter.maydell@linaro.org> wrote:
>
>> On Sat, 9 May 2020 at 14:18, Lukas Straub <lukasstraub2@web.de> wrote:
>>> Hi,
>>> Why not simply add slow tag to the test. Like:
>>> :avocado: tags=slow
>>>
>>> The slow tests can then be skipped with
>>> $ make check-acceptance AVOCADO_TAGS='-t -slow'
>>
>> Is it possible to have the default be "do the fast stuff"
>> and only do the slow stuff if the user asks? That's the
>> way round that we do the iotests, I think.
>
> I'd set AVOCADO_TAGS to '-t -slow' by default, but now that I look at the Makefile it's not that simple:
>
> AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
>
> We'd have to change the tests to skip if their target was not built. Than we can use tags for uses like this (and more like net, disk, ... tags). That would make it easier to filter the tests one wants to run: https://avocado-framework.readthedocs.io/en/78.0/guides/user/chapters/tags.html
Cleber what do you think of this approach?
>
> Regards,
> Lukas Straub
>
>> thanks
>> -- PMM
>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2020-05-26 9:15 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-07 16:22 [PATCH] tests/acceptance/boot_linux: Skip slow Aarch64 'virt' machine TCG test Philippe Mathieu-Daudé
2020-05-07 18:38 ` Alex Bennée
2020-05-07 18:45 ` Peter Maydell
2020-05-07 20:32 ` Alex Bennée
2020-05-08 13:07 ` Philippe Mathieu-Daudé
2020-05-09 13:18 ` Lukas Straub
2020-05-11 9:11 ` Peter Maydell
2020-05-11 16:51 ` Lukas Straub
2020-05-26 9:14 ` Philippe Mathieu-Daudé
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).