[Qemu-devel] [PATCH v6 0/3] Bootstrap Python venv and acceptance/functional tests

[Qemu-devel] [PATCH v6 0/3] Bootstrap Python venv and acceptance/functional tests

[Cleber Rosa]

TL;DR
=====

Allow acceptance tests to be run with `make check-acceptance`.

Details
=======

This introduces a Python virtual environment that will be setup within
the QEMU build directory, that will contain the exact environment that
tests may require.

There's one current caveat: it requires Python 3, as it's based on the
venv module.  This was based on some discussions and perception about
standardizing on Python 3, but can easily be made to accommodate Python
2 as well.

Example of bootstrap and test execution on Travis-CI:

https://travis-ci.org/qemu/qemu/jobs/439331028#L2508

   ...
      VENV    /home/travis/build/qemu/qemu/tests/venv
      MKDIR   /home/travis/build/qemu/qemu/tests/results
      PIP     /home/travis/build/qemu/qemu/tests/venv-requirements.txt
      AVOCADO tests/acceptance
    JOB ID     : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d
    JOB LOG    : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log
     (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test:  PASS (3.57 s)
     (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.04 s)
     (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
     (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
     (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.04 s)
     (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.04 s)
    RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
    JOB TIME   : 3.90 s
   ...

Changes from v5:
================

 * Renamed tests/venv-requirements.txt to tests/requirements.txt
   (Caio).

 * Moved the Travis package dependencies (python3-pip and Python3.4-venv)
   to the specific job block  (Philippe / Alex).

Changes from v4:
================

 * Added Python 3 version check and error message when running under
   Python 2.  (Philippe)

 * Added notes about using Avocado from the virtual environment
   without previous activation. (Philippe)

Ideas discussed, but not implemented:

 * Instructions to activate/deactivate the venv: the idea is to be
   use the venv as transparently as possible.

 * Move/rename tests/venv-requirements: the ideal, as per the first
   commit is to provide one venv for all Python based tests to run.

 * Changed the Travis package requirements (python3-pip,
   python3.4-venv) from the global setting to the job specific one:
   waiting on further feedback from'Alex Bennée.

Changes from v3:
================

 * Fixed typo in commit message (s/requiment/requirement/).  (Eric)

Changes from v2:
================

 * Make the $(TESTS_VENV_DIR) target depend on the
   venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv
   runs.  With this, updates on the file are reflected on the
   venv. (Philippe)

 * Run pip with "python -m pip".  It may have been installed reusing
   the system wide packages, and then the script may not be available
   on the venv. (Philippe)

 * Dropped Python version on Travis, and using the version supplied
   by the distro (3.4). (Philippe)

 * Added "python3.4-venv" package requirement on Travis. (Philippe)

 * Added variable (AVOCADO_SHOW) with logging streams to be shown
   while running the acceptance tests.  By default it's set to none,
   the equivalent of the quiet mode used on previous versions.
   (Philippe)

 * On Travis, set the AVOCADO_SHOW variable to "app", so that the
   individual test results can be easily seen.  (Philippe)

Ideas discussed, but not implemented:

  * Run pip with "$(PYTHON) -m pip -q install ..." because it points
    to the system wide Python installation. (Philippe)

  * Drop the "--system-site-packages" flag.  Waiting on another round
    of tests to determine if they are really the cause of some package
    installation problems.

Changes from v1:
================

 * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
   the source path ($SRC_PATH instead of $BUILD_DIR)

 * Create the venv with "--system-site-packages", which allows the
   reuse of packages (and no additional downloads) in case there's a
   package installed system wide providing the same package and
   version.

 * Run Avocado with "python -m avocado".  It may have been installed
   reusing the system wide packages, and then the script may not
   be available on the venv.

 * Improved documentation describing the Python 3, venv and pip
   requirements.

 * Updated avocado-framework requirement to latest released version
   (65.0)

 * (New commit) Added support for running the acceptance tests on
   Travis.

Ideas discussed, but not implemented:

 * Install external packages such as python3-pip on Debian based
   systems, deemed too invasive on developer's systems.

 * Allow the use of Python 2, and consequently the "virtualenv"
   module.

Cleber Rosa (3):
  Bootstrap Python venv for tests
  Acceptance tests: add make rule for running them
  Travis support for the acceptance tests

 .travis.yml            |  8 ++++++++
 docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++-----
 tests/Makefile.include | 43 ++++++++++++++++++++++++++++++++++++++++++
 tests/requirements.txt |  4 ++++
 4 files changed, 93 insertions(+), 5 deletions(-)
 create mode 100644 tests/requirements.txt

-- 
2.17.1

[Qemu-devel] [PATCH v6 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

A number of QEMU tests are written in Python, and may benefit
from an untainted Python venv.

By using make rules, tests that depend on specific Python libs
can set that rule as a requirement, along with rules that require
the presence or installation of specific libraries.

The tests/venv-requirements.txt is supposed to contain the
Python requirements that should be added to the venv created
by check-venv.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/Makefile.include | 26 ++++++++++++++++++++++++++
 tests/requirements.txt |  3 +++
 2 files changed, 29 insertions(+)
 create mode 100644 tests/requirements.txt

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 5eadfd52f9..4b88a119e0 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -12,6 +12,7 @@ check-help:
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
+	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
 	@echo
 	@echo "Please note that HTML reports do not regenerate if the unit tests"
@@ -1017,6 +1018,30 @@ check-decodetree:
           ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
           TEST, decodetree.py)
 
+# Python venv for running tests
+
+.PHONY: check-venv
+
+TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
+TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
+
+PYTHON_VER := $(findstring Python 3,$(shell $(PYTHON) -V))
+ifeq ($(PYTHON_VER),Python 3)
+$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
+	$(call quiet-command, \
+            $(PYTHON) -m venv --system-site-packages $@, \
+            VENV, $@)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
+            PIP, $(TESTS_VENV_REQ))
+	$(call quiet-command, touch $@)
+else
+$(TESTS_VENV_DIR):
+       $(error "venv directory for tests requires Python 3")
+endif
+
+check-venv: $(TESTS_VENV_DIR)
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -1030,6 +1055,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
+	rm -rf $(TESTS_VENV_DIR)
 
 clean: check-clean
 
diff --git a/tests/requirements.txt b/tests/requirements.txt
new file mode 100644
index 0000000000..d39f9d1576
--- /dev/null
+++ b/tests/requirements.txt
@@ -0,0 +1,3 @@
+# Add Python module requirements, one per line, to be installed
+# in the tests/venv Python virtual environment. For more info,
+# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-- 
2.17.1

[Qemu-devel] [PATCH v6 2/3] Acceptance tests: add make rule for running them

[Cleber Rosa]

The acceptance (aka functional, aka Avocado-based) tests are
Python files located in "tests/acceptance" that need to be run
with the Avocado libs and test runner.

Let's provide a convenient way for QEMU developers to run them,
by making use of the tests-venv with the required setup.

Also, while the Avocado test runner will take care of creating a
location to save test results to, it was understood that it's better
if the results are kept within the build tree.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++-----
 tests/Makefile.include | 21 +++++++++++++++++++--
 tests/requirements.txt |  1 +
 3 files changed, 58 insertions(+), 7 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 727c4019b5..e04feaccb5 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -545,10 +545,39 @@ Tests based on ``avocado_qemu.Test`` can easily:
    - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
    - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
 
-Installation
-------------
+Running tests
+-------------
 
-To install Avocado and its dependencies, run:
+You can run the acceptance tests simply by executing:
+
+.. code::
+
+  make check-acceptance
+
+This involves the automatic creation of Python virtual environment
+within the build tree (at ``tests/venv``) which will have all the
+right dependencies, and will save tests results also within the
+build tree (at ``tests/results``).
+
+Note: the build environment must be using a Python 3 stack, and have
+the ``venv`` and ``pip`` packages installed.  If necessary, make sure
+``configure`` is called with ``--python=`` and that those modules are
+available.  On Debian and Ubuntu based systems, depending on the
+specific version, they may be on packages named ``python3-venv`` and
+``python3-pip``.
+
+The scripts installed inside the virtual environment may be used
+without an "activation".  For instance, the Avocado test runner
+may be invoked by running:
+
+ .. code::
+
+  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
+
+Manual Installation
+-------------------
+
+To manually install Avocado and its dependencies, run:
 
 .. code::
 
@@ -689,11 +718,15 @@ The exact QEMU binary to be used on QEMUMachine.
 Uninstalling Avocado
 --------------------
 
-If you've followed the installation instructions above, you can easily
-uninstall Avocado.  Start by listing the packages you have installed::
+If you've followed the manual installation instructions above, you can
+easily uninstall Avocado.  Start by listing the packages you have
+installed::
 
   pip list --user
 
 And remove any package you want with::
 
   pip uninstall <package_name>
+
+If you've used ``make check-acceptance``, the Python virtual environment where
+Avocado is installed will be cleaned up as part of ``make check-clean``.
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 4b88a119e0..406a9bdaf8 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -11,6 +11,7 @@ check-help:
 	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
+	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
 	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
@@ -1020,10 +1021,15 @@ check-decodetree:
 
 # Python venv for running tests
 
-.PHONY: check-venv
+.PHONY: check-venv check-acceptance
 
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
+TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+# Controls the output generated by Avocado when running tests.
+# Any number of command separated loggers are accepted.  For more
+# information please refer to "avocado --help".
+AVOCADO_SHOW=none
 
 PYTHON_VER := $(findstring Python 3,$(shell $(PYTHON) -V))
 ifeq ($(PYTHON_VER),Python 3)
@@ -1040,8 +1046,19 @@ $(TESTS_VENV_DIR):
        $(error "venv directory for tests requires Python 3")
 endif
 
+$(TESTS_RESULTS_DIR):
+	$(call quiet-command, mkdir -p $@, \
+            MKDIR, $@)
+
 check-venv: $(TESTS_VENV_DIR)
 
+check-acceptance: check-venv $(TESTS_RESULTS_DIR)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+            --failfast=on $(SRC_PATH)/tests/acceptance, \
+            "AVOCADO", "tests/acceptance")
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -1055,7 +1072,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
-	rm -rf $(TESTS_VENV_DIR)
+	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
 
 clean: check-clean
 
diff --git a/tests/requirements.txt b/tests/requirements.txt
index d39f9d1576..64c6e27a94 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,3 +1,4 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
+avocado-framework==65.0
-- 
2.17.1

[Qemu-devel] [PATCH v6 3/3] Travis support for the acceptance tests

[Cleber Rosa]

This enables the execution of the acceptance tests on Travis.

Because the Travis environment is based on Ubuntu Trusty, it requires
the python3-pip and python3.4-venv packages.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 .travis.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 95be6ec59f..aa49c7b114 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -117,6 +117,14 @@ matrix:
     - env: CONFIG="--target-list=x86_64-softmmu"
       python:
         - "3.6"
+    # Acceptance (Functional) tests
+    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
+           TEST_CMD="make AVOCADO_SHOW=app check-acceptance"
+      addons:
+        apt:
+          packages:
+            - python3-pip
+            - python3.4-venv
     # Using newer GCC with sanitizers
     - addons:
         apt:
-- 
2.17.1

Re: [Qemu-devel] [PATCH v6 0/3] Bootstrap Python venv and acceptance/functional tests

[Caio Carrara]

On 17-10-2018 13:05, Cleber Rosa wrote:
> TL;DR
> =====
> 
> Allow acceptance tests to be run with `make check-acceptance`.
> 
> Details
> =======
> 
> This introduces a Python virtual environment that will be setup within
> the QEMU build directory, that will contain the exact environment that
> tests may require.
> 
> There's one current caveat: it requires Python 3, as it's based on the
> venv module.  This was based on some discussions and perception about
> standardizing on Python 3, but can easily be made to accommodate Python
> 2 as well.
> (...)

Reviewed-by: Caio Carrara <ccarrara@redhat.com>

-- 
Caio Carrara
Software Engineer, Virt Team
Red Hat
ccarrara@redhat.com

Re: [Qemu-devel] [PATCH v6 1/3] Bootstrap Python venv for tests

[Wainer dos Santos Moschetta]

On 10/17/2018 01:05 PM, Cleber Rosa wrote:
> A number of QEMU tests are written in Python, and may benefit
> from an untainted Python venv.
>
> By using make rules, tests that depend on specific Python libs
> can set that rule as a requirement, along with rules that require
> the presence or installation of specific libraries.
>
> The tests/venv-requirements.txt is supposed to contain the
> Python requirements that should be added to the venv created
> by check-venv.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>   tests/Makefile.include | 26 ++++++++++++++++++++++++++
>   tests/requirements.txt |  3 +++
>   2 files changed, 29 insertions(+)
>   create mode 100644 tests/requirements.txt
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 5eadfd52f9..4b88a119e0 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -12,6 +12,7 @@ check-help:
>   	@echo " $(MAKE) check-block          Run block tests"
>   	@echo " $(MAKE) check-tcg            Run TCG tests"
>   	@echo " $(MAKE) check-report.html    Generates an HTML test report"
> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>   	@echo " $(MAKE) check-clean          Clean the tests"
>   	@echo
>   	@echo "Please note that HTML reports do not regenerate if the unit tests"
> @@ -1017,6 +1018,30 @@ check-decodetree:
>             ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>             TEST, decodetree.py)
>   
> +# Python venv for running tests
> +
> +.PHONY: check-venv
> +
> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> +TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> +
> +PYTHON_VER := $(findstring Python 3,$(shell $(PYTHON) -V))
> +ifeq ($(PYTHON_VER),Python 3)
> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> +	$(call quiet-command, \
> +            $(PYTHON) -m venv --system-site-packages $@, \
> +            VENV, $@)
> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
> +            PIP, $(TESTS_VENV_REQ))
> +	$(call quiet-command, touch $@)
> +else
> +$(TESTS_VENV_DIR):
> +       $(error "venv directory for tests requires Python 3")

Above line is indented with space instead of tab. Thus, it breaks the build:

# make -j4
Python 2.7.15
/root/qemu/tests/Makefile.include:1040: *** "venv directory for tests 
requires Python 3".  Stop.

Also make check-help prints python's version when the build was 
configured with py2 (or the system lacks py3):

# make check-help
Python 2.7.15
Regression testing targets:

  make check                Run all tests
  make check-qtest-TARGET   Run qtest tests for given target
<output omitted>

I tested this patch series on Fedora 28 x86_64. It works fine except for 
above issues, so once fixing them:

Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


> +endif
> +
> +check-venv: $(TESTS_VENV_DIR)
> +
>   # Consolidated targets
>   
>   .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -1030,6 +1055,7 @@ check-clean:
>   	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>   	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>   	rm -f tests/test-qapi-gen-timestamp
> +	rm -rf $(TESTS_VENV_DIR)
>   
>   clean: check-clean
>   
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> new file mode 100644
> index 0000000000..d39f9d1576
> --- /dev/null
> +++ b/tests/requirements.txt
> @@ -0,0 +1,3 @@
> +# Add Python module requirements, one per line, to be installed
> +# in the tests/venv Python virtual environment. For more info,
> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1