Re: [PATCH 1/2] Acceptance Tests: restore downloading of VM images

[Cleber Rosa <crosa@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 1997 bytes --]

On Fri, Feb 26, 2021 at 01:01:28AM +0100, Philippe Mathieu-Daudé wrote:
> On 2/26/21 12:21 AM, Cleber Rosa wrote:
> > The "get-vm-images" target defined in tests/Makefile.include is a
> > prerequisite for "check-acceptance", so that those files get
> > downloaded before the Avocado job even starts.
> > 
> > It looks like on c401c058a1c a TARGETS variable was introduced with a
> > different content than it was previously coming from the main
> > Makefile.  From that point on, the "get-vm-images" succeed without
> > doing anything because there was no matching architecture to download.
> 
> Any idea about how to detect such side effects (tests silently
> disabled) automatically?
> 

It wasn't really that any tests were disabled... they all continued to
run.  In this case it was a broken make rule that caused the download
of the images, ahead of time, to not be performed.

But your question is still valid and something that could happen.  The
best answer I have is that all job results could and should also be
persisted in a structured way that is succeptible to being queried.
Then on top of that, you can build queries to show stability metrics,
regressions, etc.

To that regards, I can speak about three possibilities:

1) Avocado has support for Fedora's resultsdb[1][2]

2) Because the Acceptance tests are already communicating the test
results to GitLab (via junit), using the GitLab API that lets you
query the detailed test results

3) In addition to that, Marcelo (cc'd here) has written an Avocado plugin
that will export test resutls suitable to be used on a datawarehouse
tool developed by the Continuous Kernel Integration project[3]. This
is not generally available at the moment, but should be available
soon.

Regards,
- Cleber.

[1] - https://taskotron.fedoraproject.org/resultsdb/results
[2] - https://avocado-framework.readthedocs.io/en/85.0/plugins/optional/results.html#resultsdb-plugin
[3] - https://cki-project.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]