Re: [PATCH-for-5.0 1/7] tests/acceptance/machine_sparc_leon3: Disable HelenOS test

From: Willian Rampazzo <wrampazz@redhat.com>
To: "Philippe Mathieu-Daudé" <philmd@redhat.com>
Cc: "Eduardo Habkost" <ehabkost@redhat.com>,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
	"Richard Henderson" <richard.henderson@linaro.org>,
	"Mark Cave-Ayland" <mark.cave-ayland@ilande.co.uk>,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Fabien Chouteau" <chouteau@adacore.com>,
	qemu-devel <qemu-devel@nongnu.org>,
	"KONRAD Frederic" <frederic.konrad@adacore.com>,
	"Jiri Gaisler" <jiri@gaisler.se>,
	"Cleber Rosa" <crosa@redhat.com>,
	"Alex Bennée" <alex.bennee@linaro.org>,
	"Artyom Tarasenko" <atar4qemu@gmail.com>,
	"Richard Henderson" <rth@twiddle.net>
Subject: Re: [PATCH-for-5.0 1/7] tests/acceptance/machine_sparc_leon3: Disable HelenOS test
Date: Wed, 1 Apr 2020 14:43:20 -0300	[thread overview]
Message-ID: <CAKJDGDaE_OjqigaxXSJRkv7wfouLV-tsDQaXnWJXpn4F8DkPwQ@mail.gmail.com> (raw)
In-Reply-To: <e5384733-7812-1e05-fdf1-92c08c457c4f@redhat.com>

On Tue, Mar 31, 2020 at 5:07 PM Philippe Mathieu-Daudé
<philmd@redhat.com> wrote:
<snip>
>
> First job failed by timeout, 2nd succeeded:
> https://travis-ci.org/github/philmd/qemu/jobs/669265466
>
> However "Ran for 46 min 48 sec"
>
>  From the log:
>
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9
> Fetching asset from
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9
> Fetching asset from
> tests/acceptance/ppc_prep_40p.py:IbmPrep40pMachine.test_openbios_and_netbsd
> ...
>   (13/82)
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio:
>   SKIP: untrusted code
>   (24/82)
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic:
>   SKIP: storage limited
> ...
>   (25/82)
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9:
>   SKIP: storage limited
> ...
>   (63/82)
> tests/acceptance/ppc_prep_40p.py:IbmPrep40pMachine.test_openbios_and_netbsd:
>   SKIP: Running on Travis-CI
>
> Is it possible that we are now fetching assets for tests we are not
> running? In particular the one marked @skip because the downloading time
> was too long on Travis?

Yes, your assumption is correct, this execution of Avocado downloaded
assets for tests that were skipped. Let me try to explain how the
asset feature works today on Avocado.

Avocado has two basic ways to work with assets:

1. Parse limited use cases of `fetch_asset` call in the test file and
execute them. This operation can happen in two different scenarios.
First, when using the command line `avocado assets fetch <test_file>`.
In this case, it is a standalone execution of each fetch call and the
test is not executed at all. Second, by running the test. The, enabled
by default, plugin FetchAssetJob will do the same operation of parsing
the test file and executing occurrences of `fetch_asset` call before
the tests start to run. Again, the fetch time is not computed in the
job time.

2. Execute the `fetch_asset` call from each test during the test
execution. In this case, the FetchAssetJob plugin should be disabled.
The fetch time is added to the job time as the asset download occurs
during the test execution.

The acceptance tests which make use of `fetch_asset` are all using the
first method with FetchAssetJob plugin enabled. As Avocado is parsing
the test file before it starts to run the tests, it is not aware of
possible skips that may occur during a test execution due to possible
dynamic dependency.

This is not the desired behavior, as you mentioned, Avocado is
downloading an asset that will not be used because its test will be
skipped. To minimize the damage on the download side, the Travis job
is holding the avocado cache. It means the download should happen just
once. This does not minimize the damage to space usage.

One possible workaround here is to temporarily disable the
FetchAssetJob plugin, now that the needed assets are on Travis Avocado
cache. The downside is that when an asset is not available in the
cache, it will be downloaded during the test execution and the
download time will be added to the job time. I don't know if it is
possible to manually remove an asset from Travis Avocado cache. If so,
this can be done for the tests that should be skipped because of space
usage.

We have been trying to make the Asset feature as flexible as possible
to accommodate the use cases we have been identifying. Thanks for
reporting this!

>
> RESULTS    : PASS 65 | ERROR 0 | FAIL 0 | SKIP 14 | WARN 0 | INTERRUPT 0
> | CANCEL 3
> JOB TIME   : 1480.72 s
>
> Does this "JOB TIME" sums the 'Fetching asset' part?

Answered in the comments above.

>
> Thanks,
>
> Phil.
>

Willian