[PATCH 0/2] Acceptance Tests: restore the use of target related information

[PATCH 0/2] Acceptance Tests: restore the use of target related information

[Cleber Rosa]

This addresses two issues, restoring them to their previous behavior:

The first one, reported by Thomas Huth, a test timeout (and thus a job
failed) on CI.  The test was interrupted because of a download
timeout.  The download timeout happened due to the fact that "make
get-vm-images" is currently broken and the images matching the QEMU
targets are not being downloaded before the Avocado job.

The second one is the use of the same target related information, but
to filter out tests that don't have a matching "arch" tag.  So, if one
is not building, say "aarch64-softmmu", tests tagged with "arch:
aarch64" won't even make into the Avocado job.

For review purposes, the following job that shows the download of
ppc64le, s390x and x86_64 (but not aarch64) Fedora 31 images on a
acceptance-system-centos job:

   https://gitlab.com/cleber.gnu/qemu/-/jobs/1057043012#L63

And the following shows the download of the aarch64 Fedora 31 image
(but not any other) on a acceptance-system-ubuntu job:

   https://gitlab.com/cleber.gnu/qemu/-/jobs/1057043005#L61

Cleber Rosa (2):
  Acceptance Tests: restore downloading of VM images
  Acceptance Tests: restore filtering of tests by target arch

 tests/Makefile.include | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

-- 
2.25.4

[PATCH 1/2] Acceptance Tests: restore downloading of VM images

[Cleber Rosa]

The "get-vm-images" target defined in tests/Makefile.include is a
prerequisite for "check-acceptance", so that those files get
downloaded before the Avocado job even starts.

It looks like on c401c058a1c a TARGETS variable was introduced with a
different content than it was previously coming from the main
Makefile.  From that point on, the "get-vm-images" succeed without
doing anything because there was no matching architecture to download.

This restores the download of images (that match targets to be built)
before the job starts, eliminating downloads and their associated
failures during the tests.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index d34254fb29..dbd53a9de6 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -109,7 +109,8 @@ $(TESTS_RESULTS_DIR):
 
 check-venv: $(TESTS_VENV_DIR)
 
-FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS))
+FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS)))
+FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS))
 FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x
 FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES))
 
-- 
2.25.4

[PATCH 2/2] Acceptance Tests: restore filtering of tests by target arch

[Cleber Rosa]

Previously, tests were being filtered by the matching target
architectures to be built.  The benefit, compared to the current
situation, is a more concise test job that won't show tests canceled
because a matching QEMU binary was not found (those tests won't even
be attempted).

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index dbd53a9de6..799e47169c 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -92,7 +92,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
+AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
 
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
 	$(call quiet-command, \
-- 
2.25.4

Re: [PATCH 1/2] Acceptance Tests: restore downloading of VM images

[Philippe Mathieu-Daudé]

On 2/26/21 12:21 AM, Cleber Rosa wrote:
> The "get-vm-images" target defined in tests/Makefile.include is a
> prerequisite for "check-acceptance", so that those files get
> downloaded before the Avocado job even starts.
> 
> It looks like on c401c058a1c a TARGETS variable was introduced with a
> different content than it was previously coming from the main
> Makefile.  From that point on, the "get-vm-images" succeed without
> doing anything because there was no matching architecture to download.

Any idea about how to detect such side effects (tests silently
disabled) automatically?

Re: [PATCH 1/2] Acceptance Tests: restore downloading of VM images

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1997 bytes --]

On Fri, Feb 26, 2021 at 01:01:28AM +0100, Philippe Mathieu-Daudé wrote:
> On 2/26/21 12:21 AM, Cleber Rosa wrote:
> > The "get-vm-images" target defined in tests/Makefile.include is a
> > prerequisite for "check-acceptance", so that those files get
> > downloaded before the Avocado job even starts.
> > 
> > It looks like on c401c058a1c a TARGETS variable was introduced with a
> > different content than it was previously coming from the main
> > Makefile.  From that point on, the "get-vm-images" succeed without
> > doing anything because there was no matching architecture to download.
> 
> Any idea about how to detect such side effects (tests silently
> disabled) automatically?
> 

It wasn't really that any tests were disabled... they all continued to
run.  In this case it was a broken make rule that caused the download
of the images, ahead of time, to not be performed.

But your question is still valid and something that could happen.  The
best answer I have is that all job results could and should also be
persisted in a structured way that is succeptible to being queried.
Then on top of that, you can build queries to show stability metrics,
regressions, etc.

To that regards, I can speak about three possibilities:

1) Avocado has support for Fedora's resultsdb[1][2]

2) Because the Acceptance tests are already communicating the test
results to GitLab (via junit), using the GitLab API that lets you
query the detailed test results

3) In addition to that, Marcelo (cc'd here) has written an Avocado plugin
that will export test resutls suitable to be used on a datawarehouse
tool developed by the Continuous Kernel Integration project[3]. This
is not generally available at the moment, but should be available
soon.

Regards,
- Cleber.

[1] - https://taskotron.fedoraproject.org/resultsdb/results
[2] - https://avocado-framework.readthedocs.io/en/85.0/plugins/optional/results.html#resultsdb-plugin
[3] - https://cki-project.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/2] Acceptance Tests: restore downloading of VM images

[Wainer dos Santos Moschetta]

On 2/25/21 8:21 PM, Cleber Rosa wrote:
> The "get-vm-images" target defined in tests/Makefile.include is a
> prerequisite for "check-acceptance", so that those files get
> downloaded before the Avocado job even starts.
>
> It looks like on c401c058a1c a TARGETS variable was introduced with a
> different content than it was previously coming from the main
> Makefile.  From that point on, the "get-vm-images" succeed without
> doing anything because there was no matching architecture to download.
>
> This restores the download of images (that match targets to be built)
> before the job starts, eliminating downloads and their associated
> failures during the tests.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/Makefile.include | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d34254fb29..dbd53a9de6 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -109,7 +109,8 @@ $(TESTS_RESULTS_DIR):
>   
>   check-venv: $(TESTS_VENV_DIR)
>   
> -FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS))
> +FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS)))
> +FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS))
>   FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x
>   FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES))
>   

Re: [PATCH 2/2] Acceptance Tests: restore filtering of tests by target arch

[Wainer dos Santos Moschetta]

On 2/25/21 8:21 PM, Cleber Rosa wrote:
> Previously, tests were being filtered by the matching target
> architectures to be built.  The benefit, compared to the current
> situation, is a more concise test job that won't show tests canceled
> because a matching QEMU binary was not found (those tests won't even
> be attempted).
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/Makefile.include | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index dbd53a9de6..799e47169c 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -92,7 +92,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>   # Any number of command separated loggers are accepted.  For more
>   # information please refer to "avocado --help".
>   AVOCADO_SHOW=app
> -AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
> +AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
>   
>   $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>   	$(call quiet-command, \

Re: [PATCH 1/2] Acceptance Tests: restore downloading of VM images

[Philippe Mathieu-Daudé]

On 2/27/21 12:04 AM, Cleber Rosa wrote:
> On Fri, Feb 26, 2021 at 01:01:28AM +0100, Philippe Mathieu-Daudé wrote:
>> On 2/26/21 12:21 AM, Cleber Rosa wrote:
>>> The "get-vm-images" target defined in tests/Makefile.include is a
>>> prerequisite for "check-acceptance", so that those files get
>>> downloaded before the Avocado job even starts.
>>>
>>> It looks like on c401c058a1c a TARGETS variable was introduced with a
>>> different content than it was previously coming from the main
>>> Makefile.  From that point on, the "get-vm-images" succeed without
>>> doing anything because there was no matching architecture to download.
>>
>> Any idea about how to detect such side effects (tests silently
>> disabled) automatically?
>>
> 
> It wasn't really that any tests were disabled... they all continued to
> run.  In this case it was a broken make rule that caused the download
> of the images, ahead of time, to not be performed.
> 
> But your question is still valid and something that could happen.  The
> best answer I have is that all job results could and should also be
> persisted in a structured way that is succeptible to being queried.
> Then on top of that, you can build queries to show stability metrics,
> regressions, etc.
> 
> To that regards, I can speak about three possibilities:
> 
> 1) Avocado has support for Fedora's resultsdb[1][2]
> 
> 2) Because the Acceptance tests are already communicating the test
> results to GitLab (via junit), using the GitLab API that lets you
> query the detailed test results
> 
> 3) In addition to that, Marcelo (cc'd here) has written an Avocado plugin
> that will export test resutls suitable to be used on a datawarehouse
> tool developed by the Continuous Kernel Integration project[3]. This
> is not generally available at the moment, but should be available
> soon.
> 
> Regards,
> - Cleber.
> 
> [1] - https://taskotron.fedoraproject.org/resultsdb/results
> [2] - https://avocado-framework.readthedocs.io/en/85.0/plugins/optional/results.html#resultsdb-plugin
> [3] - https://cki-project.org

Wow this is thrilling!

Maybe we could use fosshost to run a resultsdb VM.

Re: [PATCH 1/2] Acceptance Tests: restore downloading of VM images

[Willian Rampazzo]

On Thu, Feb 25, 2021 at 8:21 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> The "get-vm-images" target defined in tests/Makefile.include is a
> prerequisite for "check-acceptance", so that those files get
> downloaded before the Avocado job even starts.
>
> It looks like on c401c058a1c a TARGETS variable was introduced with a
> different content than it was previously coming from the main
> Makefile.  From that point on, the "get-vm-images" succeed without
> doing anything because there was no matching architecture to download.
>
> This restores the download of images (that match targets to be built)
> before the job starts, eliminating downloads and their associated
> failures during the tests.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/Makefile.include | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH 2/2] Acceptance Tests: restore filtering of tests by target arch

[Willian Rampazzo]

On Thu, Feb 25, 2021 at 8:21 PM Cleber Rosa <crosa@redhat.com> wrote:
>
> Previously, tests were being filtered by the matching target
> architectures to be built.  The benefit, compared to the current
> situation, is a more concise test job that won't show tests canceled
> because a matching QEMU binary was not found (those tests won't even
> be attempted).
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/Makefile.include | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>