* [PATCH v3 0/3] tests/Makefile: improvements on make check-acceptance
@ 2021-09-23 16:11 Willian Rampazzo
2021-09-23 16:11 ` [PATCH v3 1/3] tests/Makefile: allow control over tags during check-acceptance Willian Rampazzo
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Willian Rampazzo @ 2021-09-23 16:11 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
Improve the documentation about how to run a single test within a file
or all the tests from a file using the standalone avocado command.
Add a way to run tests based on tags using the `make check-acceptance`
command.
Add a way to run specific tests using the `make check-acceptance` command.
GitLab pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/376122402
Changes from V2:
- Rename the environment variable to AVOCADO_TESTS
- Adjust the documentation to mention `make check-venv`
- Adjust the documentation to mention `avocado list` command
- Add the AVOCADO_TAGS implementation patch to the series.
Changes from V1:
- Rename TESTFILES to AVOCADO_TEST_FILES on patch 2
- Add Suggested-by tag on patch 2
Willian Rampazzo (3):
tests/Makefile: allow control over tags during check-acceptance
docs/devel/testing: add instruction to run a single acceptance test
tests/Makefile: add AVOCADO_TESTS option to make check-acceptance
docs/devel/testing.rst | 69 ++++++++++++++++++++++++++++++++++++++++++
tests/Makefile.include | 17 ++++++++---
2 files changed, 82 insertions(+), 4 deletions(-)
--
2.31.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v3 1/3] tests/Makefile: allow control over tags during check-acceptance
2021-09-23 16:11 [PATCH v3 0/3] tests/Makefile: improvements on make check-acceptance Willian Rampazzo
@ 2021-09-23 16:11 ` Willian Rampazzo
2021-09-25 11:06 ` Philippe Mathieu-Daudé
2021-09-23 16:11 ` [PATCH v3 2/3] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: Willian Rampazzo @ 2021-09-23 16:11 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
Although it is possible to run a specific test using the avocado
command-line, a user may want to use a specific tag while running the
``make check-acceptance`` during the development or debugging.
This allows using the AVOCADO_TAGS environment variable where the user
takes total control of which tests should run based on the tags defined.
This also makes the check-acceptance command flexible to restrict tests
based on tags while running on CI.
e.g.:
AVOCADO_TAGS="foo bar baz" make check-acceptance
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
---
docs/devel/testing.rst | 14 ++++++++++++++
tests/Makefile.include | 12 +++++++++---
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4a0abbf23d..d1841e35d5 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -732,6 +732,20 @@ available. On Debian and Ubuntu based systems, depending on the
specific version, they may be on packages named ``python3-venv`` and
``python3-pip``.
+It is also possible to run tests based on tags using the
+``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment
+variable:
+
+.. code::
+
+ make check-acceptance AVOCADO_TAGS=quick
+
+Note that tags separated with commas have an AND behavior, while tags
+separated by spaces have an OR behavior. For more information on Avocado
+tags, see:
+
+ https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
+
The scripts installed inside the virtual environment may be used
without an "activation". For instance, the Avocado test runner
may be invoked by running:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6e16c05f10..f6484e5b31 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -92,7 +92,12 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
# Any number of command separated loggers are accepted. For more
# information please refer to "avocado --help".
AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
+ifndef AVOCADO_TAGS
+ AVOCADO_CMDLINE_TAGS=$(patsubst %-softmmu,-t arch:%, \
+ $(filter %-softmmu,$(TARGETS)))
+else
+ AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS))
+endif
$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
$(call quiet-command, \
@@ -128,8 +133,9 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
$(call quiet-command, \
$(TESTS_VENV_DIR)/bin/python -m avocado \
--show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
- --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
- $(AVOCADO_TAGS) \
+ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
+ --filter-by-tags-include-empty-key) \
+ $(AVOCADO_CMDLINE_TAGS) \
$(if $(GITLAB_CI),,--failfast) tests/acceptance, \
"AVOCADO", "tests/acceptance")
--
2.31.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v3 2/3] docs/devel/testing: add instruction to run a single acceptance test
2021-09-23 16:11 [PATCH v3 0/3] tests/Makefile: improvements on make check-acceptance Willian Rampazzo
2021-09-23 16:11 ` [PATCH v3 1/3] tests/Makefile: allow control over tags during check-acceptance Willian Rampazzo
@ 2021-09-23 16:11 ` Willian Rampazzo
2021-09-25 11:04 ` Philippe Mathieu-Daudé
2021-09-23 16:11 ` [PATCH v3 3/3] tests/Makefile: add AVOCADO_TESTS option to make check-acceptance Willian Rampazzo
2021-09-27 16:42 ` [PATCH v3 0/3] tests/Makefile: improvements on " Philippe Mathieu-Daudé
3 siblings, 1 reply; 8+ messages in thread
From: Willian Rampazzo @ 2021-09-23 16:11 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
Add instructions to the Acceptance tests section about running a
single test file or a test within the test file.
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
---
docs/devel/testing.rst | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index d1841e35d5..c9f6b97f87 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -754,6 +754,34 @@ may be invoked by running:
tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
+Note that if ``make check-acceptance`` was not executed before, it is
+possible to create the Python virtual environment with the dependencies
+needed running:
+
+ .. code::
+
+ make check-venv
+
+It is also possible to run tests from a single file or a single test within
+a test file. To run tests from a single file within the build tree, use:
+
+ .. code::
+
+ tests/venv/bin/avocado run tests/acceptance/$TESTFILE
+
+To run a single test within a test file, use:
+
+ .. code::
+
+ tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
+
+Valid test names are visible in the output from any previous execution
+of Avocado or ``make check-acceptance``, and can also be queried using:
+
+ .. code::
+
+ tests/venv/bin/avocado list tests/acceptance
+
Manual Installation
-------------------
--
2.31.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v3 3/3] tests/Makefile: add AVOCADO_TESTS option to make check-acceptance
2021-09-23 16:11 [PATCH v3 0/3] tests/Makefile: improvements on make check-acceptance Willian Rampazzo
2021-09-23 16:11 ` [PATCH v3 1/3] tests/Makefile: allow control over tags during check-acceptance Willian Rampazzo
2021-09-23 16:11 ` [PATCH v3 2/3] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo
@ 2021-09-23 16:11 ` Willian Rampazzo
2021-09-25 11:05 ` Philippe Mathieu-Daudé
2021-09-27 16:42 ` [PATCH v3 0/3] tests/Makefile: improvements on " Philippe Mathieu-Daudé
3 siblings, 1 reply; 8+ messages in thread
From: Willian Rampazzo @ 2021-09-23 16:11 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
Add the possibility of running all the tests from a single file, or
multiple files, running a single test within a file or multiple tests
within multiple files using `make check-acceptance` and the
AVOCADO_TESTS environment variable.
Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
---
docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
tests/Makefile.include | 5 ++++-
2 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index c9f6b97f87..64c9744795 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -746,6 +746,33 @@ tags, see:
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
+To run a single test file, a couple of them, or a test within a file
+using the ``make check-acceptance`` command, set the ``AVOCADO_TESTS``
+environment variable with the test files or test names. To run all
+tests from a single file, use:
+
+ .. code::
+
+ make check-acceptance AVOCADO_TESTS=$FILEPATH
+
+The same is valid to run tests from multiple test files:
+
+ .. code::
+
+ make check-acceptance AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
+
+To run a single test within a file, use:
+
+ .. code::
+
+ make check-acceptance AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
+
+The same is valid to run single tests from multiple test files:
+
+ .. code::
+
+ make check-acceptance AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
+
The scripts installed inside the virtual environment may be used
without an "activation". For instance, the Avocado test runner
may be invoked by running:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index f6484e5b31..e69c4fae53 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+ifndef AVOCADO_TESTS
+ AVOCADO_TESTS=tests/acceptance
+endif
# Controls the output generated by Avocado when running tests.
# Any number of command separated loggers are accepted. For more
# information please refer to "avocado --help".
@@ -136,7 +139,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
$(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
--filter-by-tags-include-empty-key) \
$(AVOCADO_CMDLINE_TAGS) \
- $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
+ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
"AVOCADO", "tests/acceptance")
# Consolidated targets
--
2.31.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH v3 2/3] docs/devel/testing: add instruction to run a single acceptance test
2021-09-23 16:11 ` [PATCH v3 2/3] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo
@ 2021-09-25 11:04 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 8+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-25 11:04 UTC (permalink / raw)
To: Willian Rampazzo, qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
On 9/23/21 18:11, Willian Rampazzo wrote:
> Add instructions to the Acceptance tests section about running a
> single test file or a test within the test file.
>
> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> ---
> docs/devel/testing.rst | 28 ++++++++++++++++++++++++++++
> 1 file changed, 28 insertions(+)
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v3 3/3] tests/Makefile: add AVOCADO_TESTS option to make check-acceptance
2021-09-23 16:11 ` [PATCH v3 3/3] tests/Makefile: add AVOCADO_TESTS option to make check-acceptance Willian Rampazzo
@ 2021-09-25 11:05 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 8+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-25 11:05 UTC (permalink / raw)
To: Willian Rampazzo, qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
On 9/23/21 18:11, Willian Rampazzo wrote:
> Add the possibility of running all the tests from a single file, or
> multiple files, running a single test within a file or multiple tests
> within multiple files using `make check-acceptance` and the
> AVOCADO_TESTS environment variable.
>
> Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> ---
> docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
> tests/Makefile.include | 5 ++++-
> 2 files changed, 31 insertions(+), 1 deletion(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v3 1/3] tests/Makefile: allow control over tags during check-acceptance
2021-09-23 16:11 ` [PATCH v3 1/3] tests/Makefile: allow control over tags during check-acceptance Willian Rampazzo
@ 2021-09-25 11:06 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 8+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-25 11:06 UTC (permalink / raw)
To: Willian Rampazzo, qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
On 9/23/21 18:11, Willian Rampazzo wrote:
> Although it is possible to run a specific test using the avocado
> command-line, a user may want to use a specific tag while running the
> ``make check-acceptance`` during the development or debugging.
>
> This allows using the AVOCADO_TAGS environment variable where the user
> takes total control of which tests should run based on the tags defined.
>
> This also makes the check-acceptance command flexible to restrict tests
> based on tags while running on CI.
>
> e.g.:
>
> AVOCADO_TAGS="foo bar baz" make check-acceptance
>
> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> ---
> docs/devel/testing.rst | 14 ++++++++++++++
> tests/Makefile.include | 12 +++++++++---
> 2 files changed, 23 insertions(+), 3 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v3 0/3] tests/Makefile: improvements on make check-acceptance
2021-09-23 16:11 [PATCH v3 0/3] tests/Makefile: improvements on make check-acceptance Willian Rampazzo
` (2 preceding siblings ...)
2021-09-23 16:11 ` [PATCH v3 3/3] tests/Makefile: add AVOCADO_TESTS option to make check-acceptance Willian Rampazzo
@ 2021-09-27 16:42 ` Philippe Mathieu-Daudé
3 siblings, 0 replies; 8+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-27 16:42 UTC (permalink / raw)
To: Willian Rampazzo, qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
On 9/23/21 18:11, Willian Rampazzo wrote:
> Improve the documentation about how to run a single test within a file
> or all the tests from a file using the standalone avocado command.
>
> Add a way to run tests based on tags using the `make check-acceptance`
> command.
>
> Add a way to run specific tests using the `make check-acceptance` command.
>
> GitLab pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/376122402
>
> Changes from V2:
> - Rename the environment variable to AVOCADO_TESTS
> - Adjust the documentation to mention `make check-venv`
> - Adjust the documentation to mention `avocado list` command
> - Add the AVOCADO_TAGS implementation patch to the series.
>
> Changes from V1:
> - Rename TESTFILES to AVOCADO_TEST_FILES on patch 2
> - Add Suggested-by tag on patch 2
>
> Willian Rampazzo (3):
> tests/Makefile: allow control over tags during check-acceptance
> docs/devel/testing: add instruction to run a single acceptance test
> tests/Makefile: add AVOCADO_TESTS option to make check-acceptance
Thanks, applied to my integration-testing tree.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-09-27 16:45 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-23 16:11 [PATCH v3 0/3] tests/Makefile: improvements on make check-acceptance Willian Rampazzo
2021-09-23 16:11 ` [PATCH v3 1/3] tests/Makefile: allow control over tags during check-acceptance Willian Rampazzo
2021-09-25 11:06 ` Philippe Mathieu-Daudé
2021-09-23 16:11 ` [PATCH v3 2/3] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo
2021-09-25 11:04 ` Philippe Mathieu-Daudé
2021-09-23 16:11 ` [PATCH v3 3/3] tests/Makefile: add AVOCADO_TESTS option to make check-acceptance Willian Rampazzo
2021-09-25 11:05 ` Philippe Mathieu-Daudé
2021-09-27 16:42 ` [PATCH v3 0/3] tests/Makefile: improvements on " Philippe Mathieu-Daudé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.