* [PATCH 0/2] docs/devel/testing: improvements on make check-acceptance @ 2021-09-22 19:03 Willian Rampazzo 2021-09-22 19:03 ` [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo 2021-09-22 19:03 ` [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance Willian Rampazzo 0 siblings, 2 replies; 11+ messages in thread From: Willian Rampazzo @ 2021-09-22 19:03 UTC (permalink / raw) To: qemu-devel Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé, Philippe Mathieu-Daudé, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée Improve the documentation about how to run a single test within a file or all the tests from a file using the standalone avocado command. Add a way to run specific tests using the `make check-acceptance` command. GitLab pipeline: https://gitlab.com/willianrampazzo/qemu/-/pipelines/375486996 Willian Rampazzo (2): docs/devel/testing: add instruction to run a single acceptance test tests/Makefile: add TESTFILES option to make check-acceptance docs/devel/testing.rst | 41 +++++++++++++++++++++++++++++++++++++++++ tests/Makefile.include | 5 ++++- 2 files changed, 45 insertions(+), 1 deletion(-) -- 2.31.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test 2021-09-22 19:03 [PATCH 0/2] docs/devel/testing: improvements on make check-acceptance Willian Rampazzo @ 2021-09-22 19:03 ` Willian Rampazzo 2021-09-23 8:27 ` Daniel P. Berrangé 2021-09-22 19:03 ` [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance Willian Rampazzo 1 sibling, 1 reply; 11+ messages in thread From: Willian Rampazzo @ 2021-09-22 19:03 UTC (permalink / raw) To: qemu-devel Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé, Philippe Mathieu-Daudé, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée Add instructions to the Acceptance tests section about running a single test file or a test within the test file. Signed-off-by: Willian Rampazzo <willianr@redhat.com> --- docs/devel/testing.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 4a0abbf23d..b03df34f7b 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -740,6 +740,20 @@ may be invoked by running: tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ +It is also possible to run tests from a single file or a single test +within a test file. To run tests from a single file within the build +tree, use: + + .. code:: + + tests/venv/bin/avocado run tests/acceptance/$TESTFILE + +To run a single test within a test file, use: + + .. code:: + + tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME + Manual Installation ------------------- -- 2.31.1 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test 2021-09-22 19:03 ` [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo @ 2021-09-23 8:27 ` Daniel P. Berrangé 2021-09-23 11:53 ` Willian Rampazzo 0 siblings, 1 reply; 11+ messages in thread From: Daniel P. Berrangé @ 2021-09-23 8:27 UTC (permalink / raw) To: Willian Rampazzo Cc: Peter Maydell, Thomas Huth, Philippe Mathieu-Daudé, qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée On Wed, Sep 22, 2021 at 04:03:23PM -0300, Willian Rampazzo wrote: > Add instructions to the Acceptance tests section about running a > single test file or a test within the test file. > > Signed-off-by: Willian Rampazzo <willianr@redhat.com> > --- > docs/devel/testing.rst | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > index 4a0abbf23d..b03df34f7b 100644 > --- a/docs/devel/testing.rst > +++ b/docs/devel/testing.rst > @@ -740,6 +740,20 @@ may be invoked by running: > > tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ > > +It is also possible to run tests from a single file or a single test > +within a test file. To run tests from a single file within the build > +tree, use: > + > + .. code:: > + > + tests/venv/bin/avocado run tests/acceptance/$TESTFILE Before running this users may well need to run make check-venv we can't assume they have previously done "make check-acceptance" as they're possibly just reproducing a failure from gitlab CI locally, not running the whole suite. > + > +To run a single test within a test file, use: > + > + .. code:: > + > + tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME > + Valid test names are visible in the output from any previous execution of avocado, and can also be queried using tests/venv/bin/avocado list tests/acceptance > Manual Installation > ------------------- > > -- > 2.31.1 > Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test 2021-09-23 8:27 ` Daniel P. Berrangé @ 2021-09-23 11:53 ` Willian Rampazzo 0 siblings, 0 replies; 11+ messages in thread From: Willian Rampazzo @ 2021-09-23 11:53 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Peter Maydell, Thomas Huth, Philippe Mathieu-Daudé, qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée On Thu, Sep 23, 2021 at 5:27 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Wed, Sep 22, 2021 at 04:03:23PM -0300, Willian Rampazzo wrote: > > Add instructions to the Acceptance tests section about running a > > single test file or a test within the test file. > > > > Signed-off-by: Willian Rampazzo <willianr@redhat.com> > > --- > > docs/devel/testing.rst | 14 ++++++++++++++ > > 1 file changed, 14 insertions(+) > > > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > > index 4a0abbf23d..b03df34f7b 100644 > > --- a/docs/devel/testing.rst > > +++ b/docs/devel/testing.rst > > @@ -740,6 +740,20 @@ may be invoked by running: > > > > tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ > > > > +It is also possible to run tests from a single file or a single test > > +within a test file. To run tests from a single file within the build > > +tree, use: > > + > > + .. code:: > > + > > + tests/venv/bin/avocado run tests/acceptance/$TESTFILE > > Before running this users may well need to run > > make check-venv > > we can't assume they have previously done "make check-acceptance" > as they're possibly just reproducing a failure from gitlab > CI locally, not running the whole suite. Indeed! > > > + > > +To run a single test within a test file, use: > > + > > + .. code:: > > + > > + tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME > > + > > Valid test names are visible in the output from any previous execution > of avocado, and can also be queried using > > tests/venv/bin/avocado list tests/acceptance > Ack. > > Manual Installation > > ------------------- > > > > -- > > 2.31.1 > > > > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance 2021-09-22 19:03 [PATCH 0/2] docs/devel/testing: improvements on make check-acceptance Willian Rampazzo 2021-09-22 19:03 ` [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo @ 2021-09-22 19:03 ` Willian Rampazzo 2021-09-22 19:08 ` Philippe Mathieu-Daudé 1 sibling, 1 reply; 11+ messages in thread From: Willian Rampazzo @ 2021-09-22 19:03 UTC (permalink / raw) To: qemu-devel Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé, Philippe Mathieu-Daudé, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée Add the possibility of running all the tests from a single file, or multiple files, running a single test within a file or multiple tests within multiple files using `make check-acceptance` and the TESTFILES environment variable. Signed-off-by: Willian Rampazzo <willianr@redhat.com> --- docs/devel/testing.rst | 27 +++++++++++++++++++++++++++ tests/Makefile.include | 5 ++++- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index b03df34f7b..f887463f2e 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -732,6 +732,33 @@ available. On Debian and Ubuntu based systems, depending on the specific version, they may be on packages named ``python3-venv`` and ``python3-pip``. +To run a single test file, a couple of them, or a test within a file +using the ``make check-acceptance`` command, set the ``TESTFILES`` +environment variable with the test files or test names. To run all +tests from a single file, use: + + .. code:: + + make check-acceptance TESTFILES=$FILEPATH + +The same is valid to run tests from multiple test files: + + .. code:: + + make check-acceptance TESTFILES='$FILEPATH1 $FILEPATH2' + +To run a single test within a file, use: + + .. code:: + + make check-acceptance TESTFILES=$FILEPATH:$TESTCLASS.$TESTNAME + +The same is valid to run single tests from multiple test files: + + .. code:: + + make check-acceptance TESTFILES='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2' + The scripts installed inside the virtual environment may be used without an "activation". For instance, the Avocado test runner may be invoked by running: diff --git a/tests/Makefile.include b/tests/Makefile.include index 6e16c05f10..82d7ef7a20 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results +ifndef TESTFILES + TESTFILES=tests/acceptance +endif # Controls the output generated by Avocado when running tests. # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help". @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ $(AVOCADO_TAGS) \ - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \ "AVOCADO", "tests/acceptance") # Consolidated targets -- 2.31.1 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance 2021-09-22 19:03 ` [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance Willian Rampazzo @ 2021-09-22 19:08 ` Philippe Mathieu-Daudé 2021-09-22 19:46 ` Willian Rampazzo 0 siblings, 1 reply; 11+ messages in thread From: Philippe Mathieu-Daudé @ 2021-09-22 19:08 UTC (permalink / raw) To: Willian Rampazzo, qemu-devel Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée On 9/22/21 21:03, Willian Rampazzo wrote: > Add the possibility of running all the tests from a single file, or > multiple files, running a single test within a file or multiple tests > within multiple files using `make check-acceptance` and the TESTFILES > environment variable. > > Signed-off-by: Willian Rampazzo <willianr@redhat.com> > --- > docs/devel/testing.rst | 27 +++++++++++++++++++++++++++ > tests/Makefile.include | 5 ++++- > 2 files changed, 31 insertions(+), 1 deletion(-) > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 6e16c05f10..82d7ef7a20 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) > TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv > TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt > TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results > +ifndef TESTFILES > + TESTFILES=tests/acceptance > +endif > # Controls the output generated by Avocado when running tests. > # Any number of command separated loggers are accepted. For more > # information please refer to "avocado --help". > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ > $(AVOCADO_TAGS) \ > - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ > + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \ Since this is Avocado specific, maybe call the variable AVOCADO_TESTFILES (similar to AVOCADO_TAGS)? > "AVOCADO", "tests/acceptance") > > # Consolidated targets > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance 2021-09-22 19:08 ` Philippe Mathieu-Daudé @ 2021-09-22 19:46 ` Willian Rampazzo 2021-09-23 9:34 ` Philippe Mathieu-Daudé 0 siblings, 1 reply; 11+ messages in thread From: Willian Rampazzo @ 2021-09-22 19:46 UTC (permalink / raw) To: Philippe Mathieu-Daudé Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé, qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé <philmd@redhat.com> wrote: > > On 9/22/21 21:03, Willian Rampazzo wrote: > > Add the possibility of running all the tests from a single file, or > > multiple files, running a single test within a file or multiple tests > > within multiple files using `make check-acceptance` and the TESTFILES > > environment variable. > > > > Signed-off-by: Willian Rampazzo <willianr@redhat.com> > > --- > > docs/devel/testing.rst | 27 +++++++++++++++++++++++++++ > > tests/Makefile.include | 5 ++++- > > 2 files changed, 31 insertions(+), 1 deletion(-) > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > index 6e16c05f10..82d7ef7a20 100644 > > --- a/tests/Makefile.include > > +++ b/tests/Makefile.include > > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) > > TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv > > TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt > > TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results > > +ifndef TESTFILES > > + TESTFILES=tests/acceptance > > +endif > > # Controls the output generated by Avocado when running tests. > > # Any number of command separated loggers are accepted. For more > > # information please refer to "avocado --help". > > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images > > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ > > $(AVOCADO_TAGS) \ > > - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ > > + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \ > > Since this is Avocado specific, maybe call the variable > AVOCADO_TESTFILES (similar to AVOCADO_TAGS)? I don't see a problem with changing that to AVOCADO_TESTFILES. I was trying to make things shorter and easy to remember. If the too-long variable name is not a problem, I can change that. > > > "AVOCADO", "tests/acceptance") > > > > # Consolidated targets > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance 2021-09-22 19:46 ` Willian Rampazzo @ 2021-09-23 9:34 ` Philippe Mathieu-Daudé 2021-09-23 9:42 ` Daniel P. Berrangé 0 siblings, 1 reply; 11+ messages in thread From: Philippe Mathieu-Daudé @ 2021-09-23 9:34 UTC (permalink / raw) To: Willian Rampazzo Cc: Peter Maydell, Thomas Huth, Daniel P . Berrangé, qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée On 9/22/21 21:46, Willian Rampazzo wrote: > On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé > <philmd@redhat.com> wrote: >> >> On 9/22/21 21:03, Willian Rampazzo wrote: >>> Add the possibility of running all the tests from a single file, or >>> multiple files, running a single test within a file or multiple tests >>> within multiple files using `make check-acceptance` and the TESTFILES >>> environment variable. >>> >>> Signed-off-by: Willian Rampazzo <willianr@redhat.com> >>> --- >>> docs/devel/testing.rst | 27 +++++++++++++++++++++++++++ >>> tests/Makefile.include | 5 ++++- >>> 2 files changed, 31 insertions(+), 1 deletion(-) >> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include >>> index 6e16c05f10..82d7ef7a20 100644 >>> --- a/tests/Makefile.include >>> +++ b/tests/Makefile.include >>> @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) >>> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv >>> TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt >>> TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results >>> +ifndef TESTFILES >>> + TESTFILES=tests/acceptance >>> +endif >>> # Controls the output generated by Avocado when running tests. >>> # Any number of command separated loggers are accepted. For more >>> # information please refer to "avocado --help". >>> @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images >>> --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ >>> --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ >>> $(AVOCADO_TAGS) \ >>> - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ >>> + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \ >> >> Since this is Avocado specific, maybe call the variable >> AVOCADO_TESTFILES (similar to AVOCADO_TAGS)? > > I don't see a problem with changing that to AVOCADO_TESTFILES. I was > trying to make things shorter and easy to remember. If the too-long > variable name is not a problem, I can change that. This is the generic tests/Makefile, so $TESTFILES might be confusing, which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW, AVOCADO_TAGS). Thomas, do you have a preference? Otherwise: Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance 2021-09-23 9:34 ` Philippe Mathieu-Daudé @ 2021-09-23 9:42 ` Daniel P. Berrangé 2021-09-23 9:53 ` Philippe Mathieu-Daudé 2021-09-23 11:54 ` Willian Rampazzo 0 siblings, 2 replies; 11+ messages in thread From: Daniel P. Berrangé @ 2021-09-23 9:42 UTC (permalink / raw) To: Philippe Mathieu-Daudé Cc: Peter Maydell, Thomas Huth, qemu-devel, Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa, Alex Bennée On Thu, Sep 23, 2021 at 11:34:18AM +0200, Philippe Mathieu-Daudé wrote: > On 9/22/21 21:46, Willian Rampazzo wrote: > > On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé > > <philmd@redhat.com> wrote: > > > > > > On 9/22/21 21:03, Willian Rampazzo wrote: > > > > Add the possibility of running all the tests from a single file, or > > > > multiple files, running a single test within a file or multiple tests > > > > within multiple files using `make check-acceptance` and the TESTFILES > > > > environment variable. > > > > > > > > Signed-off-by: Willian Rampazzo <willianr@redhat.com> > > > > --- > > > > docs/devel/testing.rst | 27 +++++++++++++++++++++++++++ > > > > tests/Makefile.include | 5 ++++- > > > > 2 files changed, 31 insertions(+), 1 deletion(-) > > > > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > > > index 6e16c05f10..82d7ef7a20 100644 > > > > --- a/tests/Makefile.include > > > > +++ b/tests/Makefile.include > > > > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) > > > > TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv > > > > TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt > > > > TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results > > > > +ifndef TESTFILES > > > > + TESTFILES=tests/acceptance > > > > +endif > > > > # Controls the output generated by Avocado when running tests. > > > > # Any number of command separated loggers are accepted. For more > > > > # information please refer to "avocado --help". > > > > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images > > > > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > > > > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ > > > > $(AVOCADO_TAGS) \ > > > > - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ > > > > + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \ > > > > > > Since this is Avocado specific, maybe call the variable > > > AVOCADO_TESTFILES (similar to AVOCADO_TAGS)? > > > > I don't see a problem with changing that to AVOCADO_TESTFILES. I was > > trying to make things shorter and easy to remember. If the too-long > > variable name is not a problem, I can change that. > > This is the generic tests/Makefile, so $TESTFILES might be confusing, > which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW, > AVOCADO_TAGS). IIUC, this is not actually just test files - it is test files plus the test names. So better just $(AVOCADO_TESTS) Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance 2021-09-23 9:42 ` Daniel P. Berrangé @ 2021-09-23 9:53 ` Philippe Mathieu-Daudé 2021-09-23 11:54 ` Willian Rampazzo 1 sibling, 0 replies; 11+ messages in thread From: Philippe Mathieu-Daudé @ 2021-09-23 9:53 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Peter Maydell, Thomas Huth, qemu-devel, Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa, Alex Bennée On 9/23/21 11:42, Daniel P. Berrangé wrote: > On Thu, Sep 23, 2021 at 11:34:18AM +0200, Philippe Mathieu-Daudé wrote: >> On 9/22/21 21:46, Willian Rampazzo wrote: >>> On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé >>> <philmd@redhat.com> wrote: >>>> >>>> On 9/22/21 21:03, Willian Rampazzo wrote: >>>>> Add the possibility of running all the tests from a single file, or >>>>> multiple files, running a single test within a file or multiple tests >>>>> within multiple files using `make check-acceptance` and the TESTFILES >>>>> environment variable. >>>>> >>>>> Signed-off-by: Willian Rampazzo <willianr@redhat.com> >>>>> --- >>>>> docs/devel/testing.rst | 27 +++++++++++++++++++++++++++ >>>>> tests/Makefile.include | 5 ++++- >>>>> 2 files changed, 31 insertions(+), 1 deletion(-) >>>> >>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include >>>>> index 6e16c05f10..82d7ef7a20 100644 >>>>> --- a/tests/Makefile.include >>>>> +++ b/tests/Makefile.include >>>>> @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) >>>>> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv >>>>> TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt >>>>> TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results >>>>> +ifndef TESTFILES >>>>> + TESTFILES=tests/acceptance >>>>> +endif >>>>> # Controls the output generated by Avocado when running tests. >>>>> # Any number of command separated loggers are accepted. For more >>>>> # information please refer to "avocado --help". >>>>> @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images >>>>> --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ >>>>> --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ >>>>> $(AVOCADO_TAGS) \ >>>>> - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ >>>>> + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \ >>>> >>>> Since this is Avocado specific, maybe call the variable >>>> AVOCADO_TESTFILES (similar to AVOCADO_TAGS)? >>> >>> I don't see a problem with changing that to AVOCADO_TESTFILES. I was >>> trying to make things shorter and easy to remember. If the too-long >>> variable name is not a problem, I can change that. >> >> This is the generic tests/Makefile, so $TESTFILES might be confusing, >> which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW, >> AVOCADO_TAGS). > > IIUC, this is not actually just test files - it is test files plus the > test names. So better just $(AVOCADO_TESTS) Good point, I agree. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance 2021-09-23 9:42 ` Daniel P. Berrangé 2021-09-23 9:53 ` Philippe Mathieu-Daudé @ 2021-09-23 11:54 ` Willian Rampazzo 1 sibling, 0 replies; 11+ messages in thread From: Willian Rampazzo @ 2021-09-23 11:54 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Peter Maydell, Thomas Huth, Philippe Mathieu-Daudé, qemu-devel, Wainer dos Santos Moschetta, Cleber Rosa, Alex Bennée On Thu, Sep 23, 2021 at 6:42 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Thu, Sep 23, 2021 at 11:34:18AM +0200, Philippe Mathieu-Daudé wrote: > > On 9/22/21 21:46, Willian Rampazzo wrote: > > > On Wed, Sep 22, 2021 at 4:08 PM Philippe Mathieu-Daudé > > > <philmd@redhat.com> wrote: > > > > > > > > On 9/22/21 21:03, Willian Rampazzo wrote: > > > > > Add the possibility of running all the tests from a single file, or > > > > > multiple files, running a single test within a file or multiple tests > > > > > within multiple files using `make check-acceptance` and the TESTFILES > > > > > environment variable. > > > > > > > > > > Signed-off-by: Willian Rampazzo <willianr@redhat.com> > > > > > --- > > > > > docs/devel/testing.rst | 27 +++++++++++++++++++++++++++ > > > > > tests/Makefile.include | 5 ++++- > > > > > 2 files changed, 31 insertions(+), 1 deletion(-) > > > > > > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > > > > index 6e16c05f10..82d7ef7a20 100644 > > > > > --- a/tests/Makefile.include > > > > > +++ b/tests/Makefile.include > > > > > @@ -88,6 +88,9 @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) > > > > > TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv > > > > > TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt > > > > > TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results > > > > > +ifndef TESTFILES > > > > > + TESTFILES=tests/acceptance > > > > > +endif > > > > > # Controls the output generated by Avocado when running tests. > > > > > # Any number of command separated loggers are accepted. For more > > > > > # information please refer to "avocado --help". > > > > > @@ -130,7 +133,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images > > > > > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > > > > > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ > > > > > $(AVOCADO_TAGS) \ > > > > > - $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ > > > > > + $(if $(GITLAB_CI),,--failfast) $(TESTFILES), \ > > > > > > > > Since this is Avocado specific, maybe call the variable > > > > AVOCADO_TESTFILES (similar to AVOCADO_TAGS)? > > > > > > I don't see a problem with changing that to AVOCADO_TESTFILES. I was > > > trying to make things shorter and easy to remember. If the too-long > > > variable name is not a problem, I can change that. > > > > This is the generic tests/Makefile, so $TESTFILES might be confusing, > > which is why I prefer the explicit AVOCADO_ prefix (AVOCADO_SHOW, > > AVOCADO_TAGS). > > IIUC, this is not actually just test files - it is test files plus the > test names. So better just $(AVOCADO_TESTS) > Ack. I'll send another version soon. > > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-09-23 11:59 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-22 19:03 [PATCH 0/2] docs/devel/testing: improvements on make check-acceptance Willian Rampazzo 2021-09-22 19:03 ` [PATCH 1/2] docs/devel/testing: add instruction to run a single acceptance test Willian Rampazzo 2021-09-23 8:27 ` Daniel P. Berrangé 2021-09-23 11:53 ` Willian Rampazzo 2021-09-22 19:03 ` [PATCH 2/2] tests/Makefile: add TESTFILES option to make check-acceptance Willian Rampazzo 2021-09-22 19:08 ` Philippe Mathieu-Daudé 2021-09-22 19:46 ` Willian Rampazzo 2021-09-23 9:34 ` Philippe Mathieu-Daudé 2021-09-23 9:42 ` Daniel P. Berrangé 2021-09-23 9:53 ` Philippe Mathieu-Daudé 2021-09-23 11:54 ` Willian Rampazzo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.