* [PATCH 0/2] docs/devel/testing: improvements on make check-acceptance
@ 2021-09-22 19:03 Willian Rampazzo
2021-09-22 19:03 ` [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo
2021-09-22 19:03 ` [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance Willian Rampazzo
0 siblings, 2 replies; 11+ messages in thread
From: Willian Rampazzo @ 2021-09-22 19:03 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
Improve the documentation about how to run a single test within a file
or all the tests from a file using the standalone avocado command.
Add a way to run specific tests using the `make check-acceptance` command.
GitLab pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/375486996
Willian Rampazzo (2):
docs/devel/testing: add instruction to run a single acceptance test
tests/Makefile: add TESTFILES option to make check-acceptance
docs/devel/testing.rst | 41 +++++++++++++++++++++++++++++++++++++++++
tests/Makefile.include | 5 ++++-
2 files changed, 45 insertions(+), 1 deletion(-)
--
2.31.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test
2021-09-22 19:03 [PATCH 0/2] docs/devel/testing: improvements on make check-acceptance Willian Rampazzo
@ 2021-09-22 19:03 ` Willian Rampazzo
2021-09-23 8:27 ` Daniel P. Berrangé
2021-09-22 19:03 ` [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance Willian Rampazzo
1 sibling, 1 reply; 11+ messages in thread
From: Willian Rampazzo @ 2021-09-22 19:03 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
Add instructions to the Acceptance tests section about running a
single test file or a test within the test file.
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
---
docs/devel/testing.rst | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4a0abbf23d..b03df34f7b 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -740,6 +740,20 @@ may be invoked by running:
tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
+It is also possible to run tests from a single file or a single test
+within a test file. To run tests from a single file within the build
+tree, use:
+
+ .. code::
+
+ tests/venv/bin/avocado run tests/acceptance/$TESTFILE
+
+To run a single test within a test file, use:
+
+ .. code::
+
+ tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
+
Manual Installation
-------------------
--
2.31.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance
2021-09-22 19:03 [PATCH 0/2] docs/devel/testing: improvements on make check-acceptance Willian Rampazzo
2021-09-22 19:03 ` [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo
@ 2021-09-22 19:03 ` Willian Rampazzo
2021-09-22 19:08 ` Philippe Mathieu-Daudé
1 sibling, 1 reply; 11+ messages in thread
From: Willian Rampazzo @ 2021-09-22 19:03 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
Add the possibility of running all the tests from a single file, or
multiple files, running a single test within a file or multiple tests
within multiple files using `make check-acceptance` and the TESTFILES
environment variable.
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
---
docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
tests/Makefile.include | 5 ++++-
2 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index b03df34f7b..f887463f2e 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -732,6 +732,33 @@ available. On Debian and Ubuntu based systems, depending on the
specific version, they may be on packages named ``python3-venv`` and
``python3-pip``.
+To run a single test file, a couple of them, or a test within a file
+using the ``make check-acceptance`` command, set the ``TESTFILES``
+environment variable with the test files or test names. To run all
+tests from a single file, use:
+
+ .. code::
+
+ make check-acceptance TESTFILES=$FILEPATH
+
+The same is valid to run tests from multiple test files:
+
+ .. code::
+
+ make check-acceptance TESTFILES='$FILEPATH1 $FILEPATH2'
+
+To run a single test within a file, use:
+
+ .. code::
+
+ make check-acceptance TESTFILES=$FILEPATH:$TESTCLASS.$TESTNAME
+
+The same is valid to run single tests from multiple test files:
+
+ .. code::
+
+ make check-acceptance TESTFILES='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
+
The scripts installed inside the virtual environment may be used
without an "activation". For instance, the Avocado test runner
may be invoked by running:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6e16c05f10..82d7ef7a20 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+ifndef TESTFILES
+ TESTFILES=tests/acceptance
+endif
# Controls the output generated by Avocado when running tests.
# Any number of command separated loggers are accepted. For more
# information please refer to "avocado --help".
@@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
--show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
--filter-by-tags-include-empty --filter-by-tags-include-empty-key \
$(AVOCADO_TAGS) \
- $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
+ $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
"AVOCADO", "tests/acceptance")
# Consolidated targets
--
2.31.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance
2021-09-22 19:03 ` [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance Willian Rampazzo
@ 2021-09-22 19:08 ` Philippe Mathieu-Daudé
2021-09-22 19:46 ` Willian Rampazzo
0 siblings, 1 reply; 11+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-22 19:08 UTC (permalink / raw)
To: Willian Rampazzo, qemu-devel
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée
On 9/22/21 21:03, Willian Rampazzo wrote:
> Add the possibility of running all the tests from a single file, or
> multiple files, running a single test within a file or multiple tests
> within multiple files using `make check-acceptance` and the TESTFILES
> environment variable.
>
> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> ---
> docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
> tests/Makefile.include | 5 ++++-
> 2 files changed, 31 insertions(+), 1 deletion(-)
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 6e16c05f10..82d7ef7a20 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> +ifndef TESTFILES
> + TESTFILES=tests/acceptance
> +endif
> # Controls the output generated by Avocado when running tests.
> # Any number of command separated loggers are accepted. For more
> # information please refer to "avocado --help".
> @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> $(AVOCADO_TAGS) \
> - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
> + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
Since this is Avocado specific, maybe call the variable
AVOCADO_TESTFILES (similar to AVOCADO_TAGS)?
> "AVOCADO", "tests/acceptance")
>
> # Consolidated targets
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance
2021-09-22 19:08 ` Philippe Mathieu-Daudé
@ 2021-09-22 19:46 ` Willian Rampazzo
2021-09-23 9:34 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 11+ messages in thread
From: Willian Rampazzo @ 2021-09-22 19:46 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa,
Alex Bennée
On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé
<philmd@redhat.com> wrote:
>
> On 9/22/21 21:03, Willian Rampazzo wrote:
> > Add the possibility of running all the tests from a single file, or
> > multiple files, running a single test within a file or multiple tests
> > within multiple files using `make check-acceptance` and the TESTFILES
> > environment variable.
> >
> > Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> > ---
> > docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
> > tests/Makefile.include | 5 ++++-
> > 2 files changed, 31 insertions(+), 1 deletion(-)
>
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 6e16c05f10..82d7ef7a20 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
> > TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> > TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> > TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> > +ifndef TESTFILES
> > + TESTFILES=tests/acceptance
> > +endif
> > # Controls the output generated by Avocado when running tests.
> > # Any number of command separated loggers are accepted. For more
> > # information please refer to "avocado --help".
> > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> > $(AVOCADO_TAGS) \
> > - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
> > + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
>
> Since this is Avocado specific, maybe call the variable
> AVOCADO_TESTFILES (similar to AVOCADO_TAGS)?
I don't see a problem with changing that to AVOCADO_TESTFILES. I was
trying to make things shorter and easy to remember. If the too-long
variable name is not a problem, I can change that.
>
> > "AVOCADO", "tests/acceptance")
> >
> > # Consolidated targets
> >
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test
2021-09-22 19:03 ` [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo
@ 2021-09-23 8:27 ` Daniel P. Berrangé
2021-09-23 11:53 ` Willian Rampazzo
0 siblings, 1 reply; 11+ messages in thread
From: Daniel P. Berrangé @ 2021-09-23 8:27 UTC (permalink / raw)
To: Willian Rampazzo
Cc: Peter Maydell, Thomas Huth, Philippe Mathieu-Daudé,
qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa,
Alex Bennée
On Wed, Sep 22, 2021 at 04:03:23PM -0300, Willian Rampazzo wrote:
> Add instructions to the Acceptance tests section about running a
> single test file or a test within the test file.
>
> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> ---
> docs/devel/testing.rst | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 4a0abbf23d..b03df34f7b 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -740,6 +740,20 @@ may be invoked by running:
>
> tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
>
> +It is also possible to run tests from a single file or a single test
> +within a test file. To run tests from a single file within the build
> +tree, use:
> +
> + .. code::
> +
> + tests/venv/bin/avocado run tests/acceptance/$TESTFILE
Before running this users may well need to run
make check-venv
we can't assume they have previously done "make check-acceptance"
as they're possibly just reproducing a failure from gitlab
CI locally, not running the whole suite.
> +
> +To run a single test within a test file, use:
> +
> + .. code::
> +
> + tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
> +
Valid test names are visible in the output from any previous execution
of avocado, and can also be queried using
tests/venv/bin/avocado list tests/acceptance
> Manual Installation
> -------------------
>
> --
> 2.31.1
>
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance
2021-09-22 19:46 ` Willian Rampazzo
@ 2021-09-23 9:34 ` Philippe Mathieu-Daudé
2021-09-23 9:42 ` Daniel P. Berrangé
0 siblings, 1 reply; 11+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-23 9:34 UTC (permalink / raw)
To: Willian Rampazzo
Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé,
qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa,
Alex Bennée
On 9/22/21 21:46, Willian Rampazzo wrote:
> On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé
> <philmd@redhat.com> wrote:
>>
>> On 9/22/21 21:03, Willian Rampazzo wrote:
>>> Add the possibility of running all the tests from a single file, or
>>> multiple files, running a single test within a file or multiple tests
>>> within multiple files using `make check-acceptance` and the TESTFILES
>>> environment variable.
>>>
>>> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
>>> ---
>>> docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
>>> tests/Makefile.include | 5 ++++-
>>> 2 files changed, 31 insertions(+), 1 deletion(-)
>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 6e16c05f10..82d7ef7a20 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
>>> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>> TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
>>> TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>> +ifndef TESTFILES
>>> + TESTFILES=tests/acceptance
>>> +endif
>>> # Controls the output generated by Avocado when running tests.
>>> # Any number of command separated loggers are accepted. For more
>>> # information please refer to "avocado --help".
>>> @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>> --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
>>> $(AVOCADO_TAGS) \
>>> - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
>>> + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
>>
>> Since this is Avocado specific, maybe call the variable
>> AVOCADO_TESTFILES (similar to AVOCADO_TAGS)?
>
> I don't see a problem with changing that to AVOCADO_TESTFILES. I was
> trying to make things shorter and easy to remember. If the too-long
> variable name is not a problem, I can change that.
This is the generic tests/Makefile, so $TESTFILES might be confusing,
which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW,
AVOCADO_TAGS).
Thomas, do you have a preference?
Otherwise:
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance
2021-09-23 9:34 ` Philippe Mathieu-Daudé
@ 2021-09-23 9:42 ` Daniel P. Berrangé
2021-09-23 9:53 ` Philippe Mathieu-Daudé
2021-09-23 11:54 ` Willian Rampazzo
0 siblings, 2 replies; 11+ messages in thread
From: Daniel P. Berrangé @ 2021-09-23 9:42 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Peter Maydell, Thomas Huth, qemu-devel,
Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
Alex Bennée
On Thu, Sep 23, 2021 at 11:34:18AM +0200, Philippe Mathieu-Daudé wrote:
> On 9/22/21 21:46, Willian Rampazzo wrote:
> > On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé
> > <philmd@redhat.com> wrote:
> > >
> > > On 9/22/21 21:03, Willian Rampazzo wrote:
> > > > Add the possibility of running all the tests from a single file, or
> > > > multiple files, running a single test within a file or multiple tests
> > > > within multiple files using `make check-acceptance` and the TESTFILES
> > > > environment variable.
> > > >
> > > > Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> > > > ---
> > > > docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
> > > > tests/Makefile.include | 5 ++++-
> > > > 2 files changed, 31 insertions(+), 1 deletion(-)
> > >
> > > > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > > > index 6e16c05f10..82d7ef7a20 100644
> > > > --- a/tests/Makefile.include
> > > > +++ b/tests/Makefile.include
> > > > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
> > > > TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> > > > TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> > > > TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> > > > +ifndef TESTFILES
> > > > + TESTFILES=tests/acceptance
> > > > +endif
> > > > # Controls the output generated by Avocado when running tests.
> > > > # Any number of command separated loggers are accepted. For more
> > > > # information please refer to "avocado --help".
> > > > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> > > > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> > > > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> > > > $(AVOCADO_TAGS) \
> > > > - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
> > > > + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
> > >
> > > Since this is Avocado specific, maybe call the variable
> > > AVOCADO_TESTFILES (similar to AVOCADO_TAGS)?
> >
> > I don't see a problem with changing that to AVOCADO_TESTFILES. I was
> > trying to make things shorter and easy to remember. If the too-long
> > variable name is not a problem, I can change that.
>
> This is the generic tests/Makefile, so $TESTFILES might be confusing,
> which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW,
> AVOCADO_TAGS).
IIUC, this is not actually just test files - it is test files plus the
test names. So better just $(AVOCADO_TESTS)
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance
2021-09-23 9:42 ` Daniel P. Berrangé
@ 2021-09-23 9:53 ` Philippe Mathieu-Daudé
2021-09-23 11:54 ` Willian Rampazzo
1 sibling, 0 replies; 11+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-23 9:53 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Peter Maydell, Thomas Huth, qemu-devel,
Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
Alex Bennée
On 9/23/21 11:42, Daniel P. Berrangé wrote:
> On Thu, Sep 23, 2021 at 11:34:18AM +0200, Philippe Mathieu-Daudé wrote:
>> On 9/22/21 21:46, Willian Rampazzo wrote:
>>> On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé
>>> <philmd@redhat.com> wrote:
>>>>
>>>> On 9/22/21 21:03, Willian Rampazzo wrote:
>>>>> Add the possibility of running all the tests from a single file, or
>>>>> multiple files, running a single test within a file or multiple tests
>>>>> within multiple files using `make check-acceptance` and the TESTFILES
>>>>> environment variable.
>>>>>
>>>>> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
>>>>> ---
>>>>> docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
>>>>> tests/Makefile.include | 5 ++++-
>>>>> 2 files changed, 31 insertions(+), 1 deletion(-)
>>>>
>>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>>> index 6e16c05f10..82d7ef7a20 100644
>>>>> --- a/tests/Makefile.include
>>>>> +++ b/tests/Makefile.include
>>>>> @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
>>>>> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>>> TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
>>>>> TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>>>> +ifndef TESTFILES
>>>>> + TESTFILES=tests/acceptance
>>>>> +endif
>>>>> # Controls the output generated by Avocado when running tests.
>>>>> # Any number of command separated loggers are accepted. For more
>>>>> # information please refer to "avocado --help".
>>>>> @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>>>> --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>>>>> --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
>>>>> $(AVOCADO_TAGS) \
>>>>> - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
>>>>> + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
>>>>
>>>> Since this is Avocado specific, maybe call the variable
>>>> AVOCADO_TESTFILES (similar to AVOCADO_TAGS)?
>>>
>>> I don't see a problem with changing that to AVOCADO_TESTFILES. I was
>>> trying to make things shorter and easy to remember. If the too-long
>>> variable name is not a problem, I can change that.
>>
>> This is the generic tests/Makefile, so $TESTFILES might be confusing,
>> which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW,
>> AVOCADO_TAGS).
>
> IIUC, this is not actually just test files - it is test files plus the
> test names. So better just $(AVOCADO_TESTS)
Good point, I agree.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test
2021-09-23 8:27 ` Daniel P. Berrangé
@ 2021-09-23 11:53 ` Willian Rampazzo
0 siblings, 0 replies; 11+ messages in thread
From: Willian Rampazzo @ 2021-09-23 11:53 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Peter Maydell, Thomas Huth, Philippe Mathieu-Daudé,
qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa,
Alex Bennée
On Thu, Sep 23, 2021 at 5:27 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Wed, Sep 22, 2021 at 04:03:23PM -0300, Willian Rampazzo wrote:
> > Add instructions to the Acceptance tests section about running a
> > single test file or a test within the test file.
> >
> > Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> > ---
> > docs/devel/testing.rst | 14 ++++++++++++++
> > 1 file changed, 14 insertions(+)
> >
> > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> > index 4a0abbf23d..b03df34f7b 100644
> > --- a/docs/devel/testing.rst
> > +++ b/docs/devel/testing.rst
> > @@ -740,6 +740,20 @@ may be invoked by running:
> >
> > tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
> >
> > +It is also possible to run tests from a single file or a single test
> > +within a test file. To run tests from a single file within the build
> > +tree, use:
> > +
> > + .. code::
> > +
> > + tests/venv/bin/avocado run tests/acceptance/$TESTFILE
>
> Before running this users may well need to run
>
> make check-venv
>
> we can't assume they have previously done "make check-acceptance"
> as they're possibly just reproducing a failure from gitlab
> CI locally, not running the whole suite.
Indeed!
>
> > +
> > +To run a single test within a test file, use:
> > +
> > + .. code::
> > +
> > + tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
> > +
>
> Valid test names are visible in the output from any previous execution
> of avocado, and can also be queried using
>
> tests/venv/bin/avocado list tests/acceptance
>
Ack.
> > Manual Installation
> > -------------------
> >
> > --
> > 2.31.1
> >
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance
2021-09-23 9:42 ` Daniel P. Berrangé
2021-09-23 9:53 ` Philippe Mathieu-Daudé
@ 2021-09-23 11:54 ` Willian Rampazzo
1 sibling, 0 replies; 11+ messages in thread
From: Willian Rampazzo @ 2021-09-23 11:54 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Peter Maydell, Thomas Huth, Philippe Mathieu-Daudé,
qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa,
Alex Bennée
On Thu, Sep 23, 2021 at 6:42 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Sep 23, 2021 at 11:34:18AM +0200, Philippe Mathieu-Daudé wrote:
> > On 9/22/21 21:46, Willian Rampazzo wrote:
> > > On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé
> > > <philmd@redhat.com> wrote:
> > > >
> > > > On 9/22/21 21:03, Willian Rampazzo wrote:
> > > > > Add the possibility of running all the tests from a single file, or
> > > > > multiple files, running a single test within a file or multiple tests
> > > > > within multiple files using `make check-acceptance` and the TESTFILES
> > > > > environment variable.
> > > > >
> > > > > Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> > > > > ---
> > > > > docs/devel/testing.rst | 27 +++++++++++++++++++++++++++
> > > > > tests/Makefile.include | 5 ++++-
> > > > > 2 files changed, 31 insertions(+), 1 deletion(-)
> > > >
> > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > > > > index 6e16c05f10..82d7ef7a20 100644
> > > > > --- a/tests/Makefile.include
> > > > > +++ b/tests/Makefile.include
> > > > > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES)
> > > > > TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> > > > > TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> > > > > TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> > > > > +ifndef TESTFILES
> > > > > + TESTFILES=tests/acceptance
> > > > > +endif
> > > > > # Controls the output generated by Avocado when running tests.
> > > > > # Any number of command separated loggers are accepted. For more
> > > > > # information please refer to "avocado --help".
> > > > > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> > > > > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> > > > > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> > > > > $(AVOCADO_TAGS) \
> > > > > - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
> > > > > + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \
> > > >
> > > > Since this is Avocado specific, maybe call the variable
> > > > AVOCADO_TESTFILES (similar to AVOCADO_TAGS)?
> > >
> > > I don't see a problem with changing that to AVOCADO_TESTFILES. I was
> > > trying to make things shorter and easy to remember. If the too-long
> > > variable name is not a problem, I can change that.
> >
> > This is the generic tests/Makefile, so $TESTFILES might be confusing,
> > which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW,
> > AVOCADO_TAGS).
>
> IIUC, this is not actually just test files - it is test files plus the
> test names. So better just $(AVOCADO_TESTS)
>
Ack. I'll send another version soon.
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-09-23 11:59 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-22 19:03 [PATCH 0/2] docs/devel/testing: improvements on make check-acceptance Willian Rampazzo
2021-09-22 19:03 ` [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo
2021-09-23 8:27 ` Daniel P. Berrangé
2021-09-23 11:53 ` Willian Rampazzo
2021-09-22 19:03 ` [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance Willian Rampazzo
2021-09-22 19:08 ` Philippe Mathieu-Daudé
2021-09-22 19:46 ` Willian Rampazzo
2021-09-23 9:34 ` Philippe Mathieu-Daudé
2021-09-23 9:42 ` Daniel P. Berrangé
2021-09-23 9:53 ` Philippe Mathieu-Daudé
2021-09-23 11:54 ` Willian Rampazzo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).