From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55972) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gEkD3-0001Sh-A8 for qemu-devel@nongnu.org; Mon, 22 Oct 2018 20:06:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gEkD0-0000IE-4n for qemu-devel@nongnu.org; Mon, 22 Oct 2018 20:06:09 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:43102) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gEkCz-0000I0-TF for qemu-devel@nongnu.org; Mon, 22 Oct 2018 20:06:06 -0400 Received: by mail-wr1-f65.google.com with SMTP id t10-v6so5226652wrn.10 for ; Mon, 22 Oct 2018 17:06:05 -0700 (PDT) References: <20181018153134.8493-1-crosa@redhat.com> <20181018153134.8493-3-crosa@redhat.com> From: =?UTF-8?Q?Philippe_Mathieu-Daud=c3=a9?= Message-ID: <090fadb9-26db-4152-46e8-eb22bf104b53@redhat.com> Date: Tue, 23 Oct 2018 02:06:02 +0200 MIME-Version: 1.0 In-Reply-To: <20181018153134.8493-3-crosa@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] [PATCH v7 2/3] Acceptance tests: add make rule for running them List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Cleber Rosa , qemu-devel@nongnu.org Cc: Caio Carrara , =?UTF-8?Q?Alex_Benn=c3=a9e?= , Wainer dos Santos Moschetta , Laszlo Ersek , Fam Zheng , Eduardo Habkost , Stefan Hajnoczi , =?UTF-8?Q?Philippe_Mathieu-Daud=c3=a9?= On 18/10/18 17:31, Cleber Rosa wrote: > The acceptance (aka functional, aka Avocado-based) tests are > Python files located in "tests/acceptance" that need to be run > with the Avocado libs and test runner. > > Let's provide a convenient way for QEMU developers to run them, > by making use of the tests-venv with the required setup. > > Also, while the Avocado test runner will take care of creating a > location to save test results to, it was understood that it's better > if the results are kept within the build tree. > > Signed-off-by: Cleber Rosa > Acked-by: Stefan Hajnoczi > Acked-by: Wainer dos Santos Moschetta > Reviewed-by: Caio Carrara Reviewed-by: Philippe Mathieu-Daudé > --- > docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++----- > tests/Makefile.include | 21 +++++++++++++++++++-- > tests/requirements.txt | 1 + > 3 files changed, 58 insertions(+), 7 deletions(-) > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > index fcfad87614..da20be4d27 100644 > --- a/docs/devel/testing.rst > +++ b/docs/devel/testing.rst > @@ -543,10 +543,39 @@ Tests based on ``avocado_qemu.Test`` can easily: > - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test > - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html > > -Installation > ------------- > +Running tests > +------------- > > -To install Avocado and its dependencies, run: > +You can run the acceptance tests simply by executing: > + > +.. code:: > + > + make check-acceptance > + > +This involves the automatic creation of Python virtual environment > +within the build tree (at ``tests/venv``) which will have all the > +right dependencies, and will save tests results also within the > +build tree (at ``tests/results``). > + > +Note: the build environment must be using a Python 3 stack, and have > +the ``venv`` and ``pip`` packages installed. If necessary, make sure > +``configure`` is called with ``--python=`` and that those modules are > +available. On Debian and Ubuntu based systems, depending on the > +specific version, they may be on packages named ``python3-venv`` and > +``python3-pip``. > + > +The scripts installed inside the virtual environment may be used > +without an "activation". For instance, the Avocado test runner > +may be invoked by running: > + > + .. code:: > + > + tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/ > + > +Manual Installation > +------------------- > + > +To manually install Avocado and its dependencies, run: > > .. code:: > > @@ -687,11 +716,15 @@ The exact QEMU binary to be used on QEMUMachine. > Uninstalling Avocado > -------------------- > > -If you've followed the installation instructions above, you can easily > -uninstall Avocado. Start by listing the packages you have installed:: > +If you've followed the manual installation instructions above, you can > +easily uninstall Avocado. Start by listing the packages you have > +installed:: > > pip list --user > > And remove any package you want with:: > > pip uninstall > + > +If you've used ``make check-acceptance``, the Python virtual environment where > +Avocado is installed will be cleaned up as part of ``make check-clean``. > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 004a86f274..3e4dac2817 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -11,6 +11,7 @@ check-help: > @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" > @echo " $(MAKE) check-block Run block tests" > @echo " $(MAKE) check-tcg Run TCG tests" > + @echo " $(MAKE) check-acceptance Run all acceptance (functional) tests" > @echo " $(MAKE) check-report.html Generates an HTML test report" > @echo " $(MAKE) check-venv Creates a Python venv for tests" > @echo " $(MAKE) check-clean Clean the tests" > @@ -902,10 +903,15 @@ check-decodetree: > > # Python venv for running tests > > -.PHONY: check-venv > +.PHONY: check-venv check-acceptance > > TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv > TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt > +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results > +# Controls the output generated by Avocado when running tests. > +# Any number of command separated loggers are accepted. For more > +# information please refer to "avocado --help". > +AVOCADO_SHOW=none > > $(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1) > ifeq ($(.SHELLSTATUS),0) > @@ -922,8 +928,19 @@ $(TESTS_VENV_DIR): > $(error "venv directory for tests requires Python 3") > endif > > +$(TESTS_RESULTS_DIR): > + $(call quiet-command, mkdir -p $@, \ > + MKDIR, $@) > + > check-venv: $(TESTS_VENV_DIR) > > +check-acceptance: check-venv $(TESTS_RESULTS_DIR) > + $(call quiet-command, \ > + $(TESTS_VENV_DIR)/bin/python -m avocado \ > + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > + --failfast=on $(SRC_PATH)/tests/acceptance, \ > + "AVOCADO", "tests/acceptance") > + > # Consolidated targets > > .PHONY: check-qapi-schema check-qtest check-unit check check-clean > @@ -937,7 +954,7 @@ check-clean: > rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y) > rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y)) > rm -f tests/test-qapi-gen-timestamp > - rm -rf $(TESTS_VENV_DIR) > + rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR) > > clean: check-clean > > diff --git a/tests/requirements.txt b/tests/requirements.txt > index d39f9d1576..64c6e27a94 100644 > --- a/tests/requirements.txt > +++ b/tests/requirements.txt > @@ -1,3 +1,4 @@ > # Add Python module requirements, one per line, to be installed > # in the tests/venv Python virtual environment. For more info, > # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 > +avocado-framework==65.0 >