[Qemu-devel] [PATCH v4 0/3] Bootstrap Python venv and acceptance/functional tests

[Qemu-devel] [PATCH v4 0/3] Bootstrap Python venv and acceptance/functional tests

[Cleber Rosa]

TL;DR
=====

Allow acceptance tests to be run with `make check-acceptance`.

Details
=======

This introduces a Python virtual environment that will be setup within
the QEMU build directory, that will contain the exact environment that
tests may require.

There's one current caveat: it requires Python 3, as it's based on the
venv module.  This was based on some discussions and perception about
standardizing on Python 3, but can easily be made to accommodate Python
2 as well.

Example of bootstrap and test execution on Travis-CI:

https://travis-ci.org/qemu/qemu/jobs/439331028#L2508

   ...
      VENV    /home/travis/build/qemu/qemu/tests/venv
      MKDIR   /home/travis/build/qemu/qemu/tests/results
      PIP     /home/travis/build/qemu/qemu/tests/venv-requirements.txt
      AVOCADO tests/acceptance
    JOB ID     : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d
    JOB LOG    : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log
     (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test:  PASS (3.57 s)
     (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.04 s)
     (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
     (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
     (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.04 s)
     (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.04 s)
    RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
    JOB TIME   : 3.90 s
   ...

Changes from v3:
================

 * Fixed typo in commit message (s/requiment/requirement/).  (Eric)

Changes from v2:
================

 * Make the $(TESTS_VENV_DIR) target depend on the
   venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv
   runs.  With this, updates on the file are reflected on the
   venv. (Philippe)

 * Run pip with "python -m pip".  It may have been installed reusing
   the system wide packages, and then the script may not be available
   on the venv. (Philippe)

 * Dropped Python version on Travis, and using the version supplied
   by the distro (3.4). (Philippe)

 * Added "python3.4-venv" package requirement on Travis. (Philippe)

 * Added variable (AVOCADO_SHOW) with logging streams to be shown
   while running the acceptance tests.  By default it's set to none,
   the equivalent of the quiet mode used on previous versions.
   (Philippe)

 * On Travis, set the AVOCADO_SHOW variable to "app", so that the
   individual test results can be easily seen.  (Philippe)

Ideas discussed, but not implemented:

  * Run pip with "$(PYTHON) -m pip -q install ..." because it points
    to the system wide Python installation. (Philippe)

  * Drop the "--system-site-packages" flag.  Waiting on another round
    of tests to determine if they are really the cause of some package
    installation problems.

Changes from v1:
================

 * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
   the source path ($SRC_PATH instead of $BUILD_DIR)

 * Create the venv with "--system-site-packages", which allows the
   reuse of packages (and no additional downloads) in case there's a
   package installed system wide providing the same package and
   version.

 * Run Avocado with "python -m avocado".  It may have been installed
   reusing the system wide packages, and then the script may not
   be available on the venv.

 * Improved documentation describing the Python 3, venv and pip
   requirements.

 * Updated avocado-framework requirement to latest released version
   (65.0)

 * (New commit) Added support for running the acceptance tests on
   Travis.

Ideas discussed, but not implemented:

 * Install external packages such as python3-pip on Debian based
   systems, deemed too invasive on developer's systems.

 * Allow the use of Python 2, and consequently the "virtualenv"
   module.

Cleber Rosa (3):
  Bootstrap Python venv for tests
  Acceptance tests: add make rule for running them
  Travis support for the acceptance tests

 .travis.yml                 |  5 +++++
 docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
 tests/Makefile.include      | 37 +++++++++++++++++++++++++++++++++++++
 tests/venv-requirements.txt |  4 ++++
 4 files changed, 76 insertions(+), 5 deletions(-)
 create mode 100644 tests/venv-requirements.txt

-- 
2.17.1

[Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

A number of QEMU tests are written in Python, and may benefit
from an untainted Python venv.

By using make rules, tests that depend on specific Python libs
can set that rule as a requirement, along with rules that require
the presence or installation of specific libraries.

The tests/venv-requirements.txt is supposed to contain the
Python requirements that should be added to the venv created
by check-venv.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include      | 20 ++++++++++++++++++++
 tests/venv-requirements.txt |  3 +++
 2 files changed, 23 insertions(+)
 create mode 100644 tests/venv-requirements.txt

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 5eadfd52f9..b66180efa1 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -12,6 +12,7 @@ check-help:
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
+	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
 	@echo
 	@echo "Please note that HTML reports do not regenerate if the unit tests"
@@ -1017,6 +1018,24 @@ check-decodetree:
           ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
           TEST, decodetree.py)
 
+# Python venv for running tests
+
+.PHONY: check-venv
+
+TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
+TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
+
+$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
+	$(call quiet-command, \
+            $(PYTHON) -m venv --system-site-packages $@, \
+            VENV, $@)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
+            PIP, $(TESTS_VENV_REQ))
+	$(call quiet-command, touch $@)
+
+check-venv: $(TESTS_VENV_DIR)
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -1030,6 +1049,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
+	rm -rf $(TESTS_VENV_DIR)
 
 clean: check-clean
 
diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
new file mode 100644
index 0000000000..d39f9d1576
--- /dev/null
+++ b/tests/venv-requirements.txt
@@ -0,0 +1,3 @@
+# Add Python module requirements, one per line, to be installed
+# in the tests/venv Python virtual environment. For more info,
+# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-- 
2.17.1

[Qemu-devel] [PATCH v4 2/3] Acceptance tests: add make rule for running them

[Cleber Rosa]

The acceptance (aka functional, aka Avocado-based) tests are
Python files located in "tests/acceptance" that need to be run
with the Avocado libs and test runner.

Let's provide a convenient way for QEMU developers to run them,
by making use of the tests-venv with the required setup.

Also, while the Avocado test runner will take care of creating a
location to save test results to, it was understood that it's better
if the results are kept within the build tree.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
 tests/Makefile.include      | 21 +++++++++++++++++++--
 tests/venv-requirements.txt |  1 +
 3 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 727c4019b5..b992a2961d 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -545,10 +545,31 @@ Tests based on ``avocado_qemu.Test`` can easily:
    - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
    - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
 
-Installation
-------------
+Running tests
+-------------
 
-To install Avocado and its dependencies, run:
+You can run the acceptance tests simply by executing:
+
+.. code::
+
+  make check-acceptance
+
+This involves the automatic creation of Python virtual environment
+within the build tree (at ``tests/venv``) which will have all the
+right dependencies, and will save tests results also within the
+build tree (at ``tests/results``).
+
+Note: the build environment must be using a Python 3 stack, and have
+the ``venv`` and ``pip`` packages installed.  If necessary, make sure
+``configure`` is called with ``--python=`` and that those modules are
+available.  On Debian and Ubuntu based systems, depending on the
+specific version, they may be on packages named ``python3-venv`` and
+``python3-pip``.
+
+Manual Installation
+-------------------
+
+To manually install Avocado and its dependencies, run:
 
 .. code::
 
@@ -689,11 +710,15 @@ The exact QEMU binary to be used on QEMUMachine.
 Uninstalling Avocado
 --------------------
 
-If you've followed the installation instructions above, you can easily
-uninstall Avocado.  Start by listing the packages you have installed::
+If you've followed the manual installation instructions above, you can
+easily uninstall Avocado.  Start by listing the packages you have
+installed::
 
   pip list --user
 
 And remove any package you want with::
 
   pip uninstall <package_name>
+
+If you've used ``make check-acceptance``, the Python virtual environment where
+Avocado is installed will be cleaned up as part of ``make check-clean``.
diff --git a/tests/Makefile.include b/tests/Makefile.include
index b66180efa1..75547fc947 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -11,6 +11,7 @@ check-help:
 	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
+	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
 	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
@@ -1020,10 +1021,15 @@ check-decodetree:
 
 # Python venv for running tests
 
-.PHONY: check-venv
+.PHONY: check-venv check-acceptance
 
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
+TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+# Controls the output generated by Avocado when running tests.
+# Any number of command separated loggers are accepted.  For more
+# information please refer to "avocado --help".
+AVOCADO_SHOW=none
 
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
 	$(call quiet-command, \
@@ -1034,8 +1040,19 @@ $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
             PIP, $(TESTS_VENV_REQ))
 	$(call quiet-command, touch $@)
 
+$(TESTS_RESULTS_DIR):
+	$(call quiet-command, mkdir -p $@, \
+            MKDIR, $@)
+
 check-venv: $(TESTS_VENV_DIR)
 
+check-acceptance: check-venv $(TESTS_RESULTS_DIR)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+            --failfast=on $(SRC_PATH)/tests/acceptance, \
+            "AVOCADO", "tests/acceptance")
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -1049,7 +1066,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
-	rm -rf $(TESTS_VENV_DIR)
+	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
 
 clean: check-clean
 
diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
index d39f9d1576..64c6e27a94 100644
--- a/tests/venv-requirements.txt
+++ b/tests/venv-requirements.txt
@@ -1,3 +1,4 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
+avocado-framework==65.0
-- 
2.17.1

[Qemu-devel] [PATCH v4 3/3] Travis support for the acceptance tests

[Cleber Rosa]

This enables the execution of the acceptance tests on Travis.

Because the Travis environment is based on Ubuntu Trusty, it requires
the python3-pip and python3.4-venv packages.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .travis.yml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 95be6ec59f..f55f799c52 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -36,6 +36,8 @@ addons:
       - liburcu-dev
       - libusb-1.0-0-dev
       - libvte-2.90-dev
+      - python3-pip
+      - python3.4-venv
       - sparse
       - uuid-dev
       - gcovr
@@ -117,6 +119,9 @@ matrix:
     - env: CONFIG="--target-list=x86_64-softmmu"
       python:
         - "3.6"
+    # Acceptance (Functional) tests
+    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
+           TEST_CMD="make AVOCADO_SHOW=app check-acceptance"
     # Using newer GCC with sanitizers
     - addons:
         apt:
-- 
2.17.1

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

Hi Cleber,

On 12/10/2018 18:53, Cleber Rosa wrote:
> A number of QEMU tests are written in Python, and may benefit
> from an untainted Python venv.
> 
> By using make rules, tests that depend on specific Python libs
> can set that rule as a requirement, along with rules that require
> the presence or installation of specific libraries.
> 
> The tests/venv-requirements.txt is supposed to contain the
> Python requirements that should be added to the venv created
> by check-venv.

Maybe you (or Eduardo...) what you wrote in the cover:

 There's one current caveat: it requires Python 3, as it's based on the
 venv module.

To explain:

$ make check-acceptance
/usr/bin/python2: No module named venv
make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1

> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/Makefile.include      | 20 ++++++++++++++++++++
>  tests/venv-requirements.txt |  3 +++
>  2 files changed, 23 insertions(+)
>  create mode 100644 tests/venv-requirements.txt
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 5eadfd52f9..b66180efa1 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -12,6 +12,7 @@ check-help:
>  	@echo " $(MAKE) check-block          Run block tests"
>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>  	@echo " $(MAKE) check-clean          Clean the tests"
>  	@echo
>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
> @@ -1017,6 +1018,24 @@ check-decodetree:
>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>            TEST, decodetree.py)
>  
> +# Python venv for running tests
> +
> +.PHONY: check-venv
> +
> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
> +
> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> +	$(call quiet-command, \
> +            $(PYTHON) -m venv --system-site-packages $@, \
> +            VENV, $@)
> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
> +            PIP, $(TESTS_VENV_REQ))
> +	$(call quiet-command, touch $@)

Hmm maybe we should print something like:

  "You can now activate this virtual environment using:
    source $(TESTS_VENV_DIR)/tests/venv/bin/activate"

> +
> +check-venv: $(TESTS_VENV_DIR)
> +
>  # Consolidated targets
>  
>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -1030,6 +1049,7 @@ check-clean:
>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>  	rm -f tests/test-qapi-gen-timestamp
> +	rm -rf $(TESTS_VENV_DIR)
>  
>  clean: check-clean
>  
> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
> new file mode 100644
> index 0000000000..d39f9d1576
> --- /dev/null
> +++ b/tests/venv-requirements.txt
> @@ -0,0 +1,3 @@
> +# Add Python module requirements, one per line, to be installed
> +# in the tests/venv Python virtual environment. For more info,
> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> 

Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [Qemu-devel] [PATCH v4 2/3] Acceptance tests: add make rule for running them

[Philippe Mathieu-Daudé]

On 12/10/2018 18:53, Cleber Rosa wrote:
> The acceptance (aka functional, aka Avocado-based) tests are
> Python files located in "tests/acceptance" that need to be run
> with the Avocado libs and test runner.
> 
> Let's provide a convenient way for QEMU developers to run them,
> by making use of the tests-venv with the required setup.
> 
> Also, while the Avocado test runner will take care of creating a
> location to save test results to, it was understood that it's better
> if the results are kept within the build tree.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>  tests/Makefile.include      | 21 +++++++++++++++++++--
>  tests/venv-requirements.txt |  1 +
>  3 files changed, 50 insertions(+), 7 deletions(-)
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 727c4019b5..b992a2961d 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -545,10 +545,31 @@ Tests based on ``avocado_qemu.Test`` can easily:
>     - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
>     - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
>  
> -Installation
> -------------
> +Running tests
> +-------------
>  
> -To install Avocado and its dependencies, run:
> +You can run the acceptance tests simply by executing:
> +
> +.. code::
> +
> +  make check-acceptance
> +
> +This involves the automatic creation of Python virtual environment
> +within the build tree (at ``tests/venv``) which will have all the
> +right dependencies, and will save tests results also within the
> +build tree (at ``tests/results``).

Missing: how to activate the virtualenv.

> +
> +Note: the build environment must be using a Python 3 stack, and have
> +the ``venv`` and ``pip`` packages installed.  If necessary, make sure
> +``configure`` is called with ``--python=`` and that those modules are
> +available.  On Debian and Ubuntu based systems, depending on the
> +specific version, they may be on packages named ``python3-venv`` and
> +``python3-pip``.
> +
> +Manual Installation
> +-------------------
> +
> +To manually install Avocado and its dependencies, run:
>  
>  .. code::
>  
> @@ -689,11 +710,15 @@ The exact QEMU binary to be used on QEMUMachine.
>  Uninstalling Avocado
>  --------------------
>  
> -If you've followed the installation instructions above, you can easily
> -uninstall Avocado.  Start by listing the packages you have installed::
> +If you've followed the manual installation instructions above, you can
> +easily uninstall Avocado.  Start by listing the packages you have
> +installed::
>  
>    pip list --user
>  
>  And remove any package you want with::
>  
>    pip uninstall <package_name>
> +
> +If you've used ``make check-acceptance``, the Python virtual environment where
> +Avocado is installed will be cleaned up as part of ``make check-clean``.

Add a line about deactivating the venv?

> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index b66180efa1..75547fc947 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -11,6 +11,7 @@ check-help:
>  	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
>  	@echo " $(MAKE) check-block          Run block tests"
>  	@echo " $(MAKE) check-tcg            Run TCG tests"
> +	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>  	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>  	@echo " $(MAKE) check-clean          Clean the tests"
> @@ -1020,10 +1021,15 @@ check-decodetree:
>  
>  # Python venv for running tests
>  
> -.PHONY: check-venv
> +.PHONY: check-venv check-acceptance
>  
>  TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>  TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> +# Controls the output generated by Avocado when running tests.
> +# Any number of command separated loggers are accepted.  For more
> +# information please refer to "avocado --help".
> +AVOCADO_SHOW=none
>  
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>  	$(call quiet-command, \
> @@ -1034,8 +1040,19 @@ $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>              PIP, $(TESTS_VENV_REQ))
>  	$(call quiet-command, touch $@)
>  
> +$(TESTS_RESULTS_DIR):
> +	$(call quiet-command, mkdir -p $@, \
> +            MKDIR, $@)
> +
>  check-venv: $(TESTS_VENV_DIR)
>  
> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m avocado \
> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> +            --failfast=on $(SRC_PATH)/tests/acceptance, \
> +            "AVOCADO", "tests/acceptance")
> +
>  # Consolidated targets
>  
>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -1049,7 +1066,7 @@ check-clean:
>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>  	rm -f tests/test-qapi-gen-timestamp
> -	rm -rf $(TESTS_VENV_DIR)
> +	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
>  
>  clean: check-clean
>  
> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
> index d39f9d1576..64c6e27a94 100644
> --- a/tests/venv-requirements.txt
> +++ b/tests/venv-requirements.txt
> @@ -1,3 +1,4 @@
>  # Add Python module requirements, one per line, to be installed
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> +avocado-framework==65.0
> 

Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [Qemu-devel] [PATCH v4 0/3] Bootstrap Python venv and acceptance/functional tests

[Philippe Mathieu-Daudé]

On 12/10/2018 18:53, Cleber Rosa wrote:
> TL;DR
> =====
> 
> Allow acceptance tests to be run with `make check-acceptance`.
> 
> Details
> =======
> 
> This introduces a Python virtual environment that will be setup within
> the QEMU build directory, that will contain the exact environment that
> tests may require.
> 
> There's one current caveat: it requires Python 3, as it's based on the
> venv module.  This was based on some discussions and perception about
> standardizing on Python 3, but can easily be made to accommodate Python
> 2 as well.
> 
> Example of bootstrap and test execution on Travis-CI:
> 
> https://travis-ci.org/qemu/qemu/jobs/439331028#L2508

If you activate Travis on your github account, you can test that in your
namespace without having to open zombie pull requests there... A simple
push to your repository will trigger a full Travis build.

This is how I use it btw, canceling the jobs I'm not interested in, to
quickly run the others.
i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600

> 
>    ...
>       VENV    /home/travis/build/qemu/qemu/tests/venv
>       MKDIR   /home/travis/build/qemu/qemu/tests/results
>       PIP     /home/travis/build/qemu/qemu/tests/venv-requirements.txt
>       AVOCADO tests/acceptance
>     JOB ID     : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d
>     JOB LOG    : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log
>      (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test:  PASS (3.57 s)
>      (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.04 s)
>      (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>      (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>      (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.04 s)
>      (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.04 s)
>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>     JOB TIME   : 3.90 s
>    ...
> 
> Changes from v3:
> ================
> 
>  * Fixed typo in commit message (s/requiment/requirement/).  (Eric)
> 
> Changes from v2:
> ================
> 
>  * Make the $(TESTS_VENV_DIR) target depend on the
>    venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv
>    runs.  With this, updates on the file are reflected on the
>    venv. (Philippe)
> 
>  * Run pip with "python -m pip".  It may have been installed reusing
>    the system wide packages, and then the script may not be available
>    on the venv. (Philippe)
> 
>  * Dropped Python version on Travis, and using the version supplied
>    by the distro (3.4). (Philippe)
> 
>  * Added "python3.4-venv" package requirement on Travis. (Philippe)
> 
>  * Added variable (AVOCADO_SHOW) with logging streams to be shown
>    while running the acceptance tests.  By default it's set to none,
>    the equivalent of the quiet mode used on previous versions.
>    (Philippe)
> 
>  * On Travis, set the AVOCADO_SHOW variable to "app", so that the
>    individual test results can be easily seen.  (Philippe)
> 
> Ideas discussed, but not implemented:
> 
>   * Run pip with "$(PYTHON) -m pip -q install ..." because it points
>     to the system wide Python installation. (Philippe)
> 
>   * Drop the "--system-site-packages" flag.  Waiting on another round
>     of tests to determine if they are really the cause of some package
>     installation problems.
> 
> Changes from v1:
> ================
> 
>  * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
>    the source path ($SRC_PATH instead of $BUILD_DIR)
> 
>  * Create the venv with "--system-site-packages", which allows the
>    reuse of packages (and no additional downloads) in case there's a
>    package installed system wide providing the same package and
>    version.
> 
>  * Run Avocado with "python -m avocado".  It may have been installed
>    reusing the system wide packages, and then the script may not
>    be available on the venv.
> 
>  * Improved documentation describing the Python 3, venv and pip
>    requirements.
> 
>  * Updated avocado-framework requirement to latest released version
>    (65.0)
> 
>  * (New commit) Added support for running the acceptance tests on
>    Travis.
> 
> Ideas discussed, but not implemented:
> 
>  * Install external packages such as python3-pip on Debian based
>    systems, deemed too invasive on developer's systems.
> 
>  * Allow the use of Python 2, and consequently the "virtualenv"
>    module.
> 
> Cleber Rosa (3):
>   Bootstrap Python venv for tests
>   Acceptance tests: add make rule for running them
>   Travis support for the acceptance tests
> 
>  .travis.yml                 |  5 +++++
>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>  tests/Makefile.include      | 37 +++++++++++++++++++++++++++++++++++++
>  tests/venv-requirements.txt |  4 ++++
>  4 files changed, 76 insertions(+), 5 deletions(-)
>  create mode 100644 tests/venv-requirements.txt
> 

Re: [Qemu-devel] [PATCH v4 3/3] Travis support for the acceptance tests

[Philippe Mathieu-Daudé]

Hi Cleber,

On 12/10/2018 18:53, Cleber Rosa wrote:
> This enables the execution of the acceptance tests on Travis.
> 
> Because the Travis environment is based on Ubuntu Trusty, it requires
> the python3-pip and python3.4-venv packages.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .travis.yml | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 95be6ec59f..f55f799c52 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -36,6 +36,8 @@ addons:
>        - liburcu-dev
>        - libusb-1.0-0-dev
>        - libvte-2.90-dev
> +      - python3-pip
> +      - python3.4-venv

I'd prefer not put Python specific version in the generic addons list...

>        - sparse
>        - uuid-dev
>        - gcovr
> @@ -117,6 +119,9 @@ matrix:
>      - env: CONFIG="--target-list=x86_64-softmmu"
>        python:
>          - "3.6"
> +    # Acceptance (Functional) tests
> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
> +           TEST_CMD="make AVOCADO_SHOW=app check-acceptance"

... but rather in the single test that requires it:

         addons:
           apt:
             packages:
               - python3-pip
               - python3.4-venv

Alex what do you prefer?

>      # Using newer GCC with sanitizers
>      - addons:
>          apt:
> 

Both configs:
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Eduardo Habkost]

On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> On 12/10/2018 18:53, Cleber Rosa wrote:
> > A number of QEMU tests are written in Python, and may benefit
> > from an untainted Python venv.
> > 
> > By using make rules, tests that depend on specific Python libs
> > can set that rule as a requirement, along with rules that require
> > the presence or installation of specific libraries.
> > 
> > The tests/venv-requirements.txt is supposed to contain the
> > Python requirements that should be added to the venv created
> > by check-venv.
> 
> Maybe you (or Eduardo...) what you wrote in the cover:
> 
>  There's one current caveat: it requires Python 3, as it's based on the
>  venv module.
> 
> To explain:
> 
> $ make check-acceptance
> /usr/bin/python2: No module named venv
> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1

Oops, this doesn't look very friendly.

But note that this would become a non-issue if we start requiring
Python 3 for building QEMU.


> 
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/Makefile.include      | 20 ++++++++++++++++++++
> >  tests/venv-requirements.txt |  3 +++
> >  2 files changed, 23 insertions(+)
> >  create mode 100644 tests/venv-requirements.txt
> > 
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 5eadfd52f9..b66180efa1 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -12,6 +12,7 @@ check-help:
> >  	@echo " $(MAKE) check-block          Run block tests"
> >  	@echo " $(MAKE) check-tcg            Run TCG tests"
> >  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
> > +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
> >  	@echo " $(MAKE) check-clean          Clean the tests"
> >  	@echo
> >  	@echo "Please note that HTML reports do not regenerate if the unit tests"
> > @@ -1017,6 +1018,24 @@ check-decodetree:
> >            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
> >            TEST, decodetree.py)
> >  
> > +# Python venv for running tests
> > +
> > +.PHONY: check-venv
> > +
> > +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> > +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
> > +
> > +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> > +	$(call quiet-command, \
> > +            $(PYTHON) -m venv --system-site-packages $@, \
> > +            VENV, $@)
> > +	$(call quiet-command, \
> > +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
> > +            PIP, $(TESTS_VENV_REQ))
> > +	$(call quiet-command, touch $@)
> 
> Hmm maybe we should print something like:
> 
>   "You can now activate this virtual environment using:
>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"

I'm not sure this would be necessary: I expect usage of the venv
to be completely transparent.

If we require people to learn what venv is and manually activate
it, I'd say we have failed to provide usable tools for running
the tests.


> 
> > +
> > +check-venv: $(TESTS_VENV_DIR)
> > +
> >  # Consolidated targets
> >  
> >  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> > @@ -1030,6 +1049,7 @@ check-clean:
> >  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
> >  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
> >  	rm -f tests/test-qapi-gen-timestamp
> > +	rm -rf $(TESTS_VENV_DIR)
> >  
> >  clean: check-clean
> >  
> > diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
> > new file mode 100644
> > index 0000000000..d39f9d1576
> > --- /dev/null
> > +++ b/tests/venv-requirements.txt
> > @@ -0,0 +1,3 @@
> > +# Add Python module requirements, one per line, to be installed
> > +# in the tests/venv Python virtual environment. For more info,
> > +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> > 
> 
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

-- 
Eduardo

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Caio Carrara]

On 13-10-2018 00:37, Eduardo Habkost wrote:
> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
>> Hi Cleber,
>>
>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>> A number of QEMU tests are written in Python, and may benefit
>>> from an untainted Python venv.
>>>
>>> By using make rules, tests that depend on specific Python libs
>>> can set that rule as a requirement, along with rules that require
>>> the presence or installation of specific libraries.
>>>
>>> The tests/venv-requirements.txt is supposed to contain the
>>> Python requirements that should be added to the venv created
>>> by check-venv.
>>
>> Maybe you (or Eduardo...) what you wrote in the cover:
>>
>>  There's one current caveat: it requires Python 3, as it's based on the
>>  venv module.
>>
>> To explain:
>>
>> $ make check-acceptance
>> /usr/bin/python2: No module named venv
>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
> 
> Oops, this doesn't look very friendly.
> 
> But note that this would become a non-issue if we start requiring
> Python 3 for building QEMU.
> 
> 
>>
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>  tests/venv-requirements.txt |  3 +++
>>>  2 files changed, 23 insertions(+)
>>>  create mode 100644 tests/venv-requirements.txt
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 5eadfd52f9..b66180efa1 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -12,6 +12,7 @@ check-help:
>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>  	@echo
>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>            TEST, decodetree.py)
>>>  
>>> +# Python venv for running tests
>>> +
>>> +.PHONY: check-venv
>>> +
>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>> +
>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>> +	$(call quiet-command, \
>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>> +            VENV, $@)
>>> +	$(call quiet-command, \
>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>> +            PIP, $(TESTS_VENV_REQ))
>>> +	$(call quiet-command, touch $@)
>>
>> Hmm maybe we should print something like:
>>
>>   "You can now activate this virtual environment using:
>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
> 
> I'm not sure this would be necessary: I expect usage of the venv
> to be completely transparent.
> 
> If we require people to learn what venv is and manually activate
> it, I'd say we have failed to provide usable tools for running
> the tests.
> 

Actually this is not necessary since the avocado is being called from
the "venv python binary" as you can see in the check-acceptance target.

This way all the requirements installed in the test venv can be used
without activating the virtual environment.

> 
>>
>>> +
>>> +check-venv: $(TESTS_VENV_DIR)
>>> +
>>>  # Consolidated targets
>>>  
>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>  	rm -f tests/test-qapi-gen-timestamp
>>> +	rm -rf $(TESTS_VENV_DIR)
>>>  
>>>  clean: check-clean
>>>  
>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>> new file mode 100644
>>> index 0000000000..d39f9d1576
>>> --- /dev/null
>>> +++ b/tests/venv-requirements.txt
>>> @@ -0,0 +1,3 @@
>>> +# Add Python module requirements, one per line, to be installed
>>> +# in the tests/venv Python virtual environment. For more info,
>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>
>>
>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> 


-- 
Caio Carrara
Software Engineer, Virt Team
Red Hat
ccarrara@redhat.com

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Caio Carrara]

On 12-10-2018 13:53, Cleber Rosa wrote:
> A number of QEMU tests are written in Python, and may benefit
> from an untainted Python venv.
> 
> By using make rules, tests that depend on specific Python libs
> can set that rule as a requirement, along with rules that require
> the presence or installation of specific libraries.
> 
> The tests/venv-requirements.txt is supposed to contain the
> Python requirements that should be added to the venv created
> by check-venv.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/Makefile.include      | 20 ++++++++++++++++++++
>  tests/venv-requirements.txt |  3 +++

Any special reason to name `venv-requirements.txt` instead of only
`requirements.txt`? I think the second way is better (if there's no
cons) since it's the default from most of Python projects. Besides that
seems more semantic to have a `tests/requirements.txt` file that
registers the requirements for tests.

Still in this topic. As far as I could see, the files in `tests/`
directory are almost all C source code. The right place for the Python
requirements seems the `tests/acceptance/` directory specially because
the subject of all this patches. Keeping only one requirements file for
all kinds of tests (that uses Python) can be problematic in my opinion.
If it makes any sense, probably is also a good idea rename the target
from `check-venv` to something like `check-acceptance-venv`.

>  2 files changed, 23 insertions(+)
>  create mode 100644 tests/venv-requirements.txt
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 5eadfd52f9..b66180efa1 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -12,6 +12,7 @@ check-help:
>  	@echo " $(MAKE) check-block          Run block tests"
>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>  	@echo " $(MAKE) check-clean          Clean the tests"
>  	@echo
>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
> @@ -1017,6 +1018,24 @@ check-decodetree:
>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>            TEST, decodetree.py)
>  
> +# Python venv for running tests
> +
> +.PHONY: check-venv
> +
> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
> +
> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> +	$(call quiet-command, \
> +            $(PYTHON) -m venv --system-site-packages $@, \
> +            VENV, $@)
> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
> +            PIP, $(TESTS_VENV_REQ))
> +	$(call quiet-command, touch $@)
> +
> +check-venv: $(TESTS_VENV_DIR)
> +
>  # Consolidated targets
>  
>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -1030,6 +1049,7 @@ check-clean:
>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>  	rm -f tests/test-qapi-gen-timestamp
> +	rm -rf $(TESTS_VENV_DIR)
>  
>  clean: check-clean
>  
> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
> new file mode 100644
> index 0000000000..d39f9d1576
> --- /dev/null
> +++ b/tests/venv-requirements.txt
> @@ -0,0 +1,3 @@
> +# Add Python module requirements, one per line, to be installed
> +# in the tests/venv Python virtual environment. For more info,
> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> 


-- 
Caio Carrara
Software Engineer, Virt Team
Red Hat
ccarrara@redhat.com

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

On 15/10/2018 21:04, Caio Carrara wrote:
> On 12-10-2018 13:53, Cleber Rosa wrote:
>> A number of QEMU tests are written in Python, and may benefit
>> from an untainted Python venv.
>>
>> By using make rules, tests that depend on specific Python libs
>> can set that rule as a requirement, along with rules that require
>> the presence or installation of specific libraries.
>>
>> The tests/venv-requirements.txt is supposed to contain the
>> Python requirements that should be added to the venv created
>> by check-venv.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>  tests/venv-requirements.txt |  3 +++
> 
> Any special reason to name `venv-requirements.txt` instead of only
> `requirements.txt`? I think the second way is better (if there's no
> cons) since it's the default from most of Python projects. Besides that
> seems more semantic to have a `tests/requirements.txt` file that
> registers the requirements for tests.

Agreed.

> 
> Still in this topic. As far as I could see, the files in `tests/`
> directory are almost all C source code. The right place for the Python
> requirements seems the `tests/acceptance/` directory specially because
> the subject of all this patches. Keeping only one requirements file for
> all kinds of tests (that uses Python) can be problematic in my opinion.
> If it makes any sense, probably is also a good idea rename the target
> from `check-venv` to something like `check-acceptance-venv`.

Yes.

> 
>>  2 files changed, 23 insertions(+)
>>  create mode 100644 tests/venv-requirements.txt
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 5eadfd52f9..b66180efa1 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -12,6 +12,7 @@ check-help:
>>  	@echo " $(MAKE) check-block          Run block tests"
>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>  	@echo
>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>            TEST, decodetree.py)
>>  
>> +# Python venv for running tests
>> +
>> +.PHONY: check-venv
>> +
>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>> +
>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>> +	$(call quiet-command, \
>> +            $(PYTHON) -m venv --system-site-packages $@, \
>> +            VENV, $@)
>> +	$(call quiet-command, \
>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>> +            PIP, $(TESTS_VENV_REQ))
>> +	$(call quiet-command, touch $@)
>> +
>> +check-venv: $(TESTS_VENV_DIR)
>> +
>>  # Consolidated targets
>>  
>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>> @@ -1030,6 +1049,7 @@ check-clean:
>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>  	rm -f tests/test-qapi-gen-timestamp
>> +	rm -rf $(TESTS_VENV_DIR)
>>  
>>  clean: check-clean
>>  
>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>> new file mode 100644
>> index 0000000000..d39f9d1576
>> --- /dev/null
>> +++ b/tests/venv-requirements.txt
>> @@ -0,0 +1,3 @@
>> +# Add Python module requirements, one per line, to be installed
>> +# in the tests/venv Python virtual environment. For more info,
>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>
> 
> 

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

Hi Caio,

On 15/10/2018 20:41, Caio Carrara wrote:
> On 13-10-2018 00:37, Eduardo Habkost wrote:
>> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
>>> Hi Cleber,
>>>
>>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>>> A number of QEMU tests are written in Python, and may benefit
>>>> from an untainted Python venv.
>>>>
>>>> By using make rules, tests that depend on specific Python libs
>>>> can set that rule as a requirement, along with rules that require
>>>> the presence or installation of specific libraries.
>>>>
>>>> The tests/venv-requirements.txt is supposed to contain the
>>>> Python requirements that should be added to the venv created
>>>> by check-venv.
>>>
>>> Maybe you (or Eduardo...) what you wrote in the cover:
>>>
>>>  There's one current caveat: it requires Python 3, as it's based on the
>>>  venv module.
>>>
>>> To explain:
>>>
>>> $ make check-acceptance
>>> /usr/bin/python2: No module named venv
>>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
>>
>> Oops, this doesn't look very friendly.
>>
>> But note that this would become a non-issue if we start requiring
>> Python 3 for building QEMU.
>>
>>
>>>
>>>>
>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>>  tests/venv-requirements.txt |  3 +++
>>>>  2 files changed, 23 insertions(+)
>>>>  create mode 100644 tests/venv-requirements.txt
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index 5eadfd52f9..b66180efa1 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -12,6 +12,7 @@ check-help:
>>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>>  	@echo
>>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>>            TEST, decodetree.py)
>>>>  
>>>> +# Python venv for running tests
>>>> +
>>>> +.PHONY: check-venv
>>>> +
>>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>>> +
>>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>> +	$(call quiet-command, \
>>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>>> +            VENV, $@)
>>>> +	$(call quiet-command, \
>>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>>> +            PIP, $(TESTS_VENV_REQ))
>>>> +	$(call quiet-command, touch $@)
>>>
>>> Hmm maybe we should print something like:
>>>
>>>   "You can now activate this virtual environment using:
>>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
>>
>> I'm not sure this would be necessary: I expect usage of the venv
>> to be completely transparent.
>>
>> If we require people to learn what venv is and manually activate
>> it, I'd say we have failed to provide usable tools for running
>> the tests.
>>
> 
> Actually this is not necessary since the avocado is being called from
> the "venv python binary" as you can see in the check-acceptance target.
> 
> This way all the requirements installed in the test venv can be used
> without activating the virtual environment.

Well this is only true if you call 'make check-acceptance', not if you
want to filter tests, or run the single file you are working on...
Or am I missing something? The only option user-configurable (without
activating the venv) is the output of all tests via the AVOCADO_SHOW env
var.

This might be enough for a maintainer checking his subsystem, but I
don't find this practical for a acceptance test writer. And we want for
people to contribute adding tests, right?
Well, if we have maintainer running them, this is already a win :)

> 
>>
>>>
>>>> +
>>>> +check-venv: $(TESTS_VENV_DIR)
>>>> +
>>>>  # Consolidated targets
>>>>  
>>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>>  	rm -f tests/test-qapi-gen-timestamp
>>>> +	rm -rf $(TESTS_VENV_DIR)
>>>>  
>>>>  clean: check-clean
>>>>  
>>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>>> new file mode 100644
>>>> index 0000000000..d39f9d1576
>>>> --- /dev/null
>>>> +++ b/tests/venv-requirements.txt
>>>> @@ -0,0 +1,3 @@
>>>> +# Add Python module requirements, one per line, to be installed
>>>> +# in the tests/venv Python virtual environment. For more info,
>>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>
>>>
>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>
> 
> 

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Eduardo Habkost]

On Tue, Oct 16, 2018 at 12:28:07AM +0200, Philippe Mathieu-Daudé wrote:
> Hi Caio,
> 
> On 15/10/2018 20:41, Caio Carrara wrote:
> > On 13-10-2018 00:37, Eduardo Habkost wrote:
> >> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
> >>> Hi Cleber,
> >>>
> >>> On 12/10/2018 18:53, Cleber Rosa wrote:
> >>>> A number of QEMU tests are written in Python, and may benefit
> >>>> from an untainted Python venv.
> >>>>
> >>>> By using make rules, tests that depend on specific Python libs
> >>>> can set that rule as a requirement, along with rules that require
> >>>> the presence or installation of specific libraries.
> >>>>
> >>>> The tests/venv-requirements.txt is supposed to contain the
> >>>> Python requirements that should be added to the venv created
> >>>> by check-venv.
> >>>
> >>> Maybe you (or Eduardo...) what you wrote in the cover:
> >>>
> >>>  There's one current caveat: it requires Python 3, as it's based on the
> >>>  venv module.
> >>>
> >>> To explain:
> >>>
> >>> $ make check-acceptance
> >>> /usr/bin/python2: No module named venv
> >>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
> >>
> >> Oops, this doesn't look very friendly.
> >>
> >> But note that this would become a non-issue if we start requiring
> >> Python 3 for building QEMU.
> >>
> >>
> >>>
> >>>>
> >>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> >>>> ---
> >>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
> >>>>  tests/venv-requirements.txt |  3 +++
> >>>>  2 files changed, 23 insertions(+)
> >>>>  create mode 100644 tests/venv-requirements.txt
> >>>>
> >>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>>> index 5eadfd52f9..b66180efa1 100644
> >>>> --- a/tests/Makefile.include
> >>>> +++ b/tests/Makefile.include
> >>>> @@ -12,6 +12,7 @@ check-help:
> >>>>  	@echo " $(MAKE) check-block          Run block tests"
> >>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
> >>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
> >>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
> >>>>  	@echo " $(MAKE) check-clean          Clean the tests"
> >>>>  	@echo
> >>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
> >>>> @@ -1017,6 +1018,24 @@ check-decodetree:
> >>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
> >>>>            TEST, decodetree.py)
> >>>>  
> >>>> +# Python venv for running tests
> >>>> +
> >>>> +.PHONY: check-venv
> >>>> +
> >>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> >>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
> >>>> +
> >>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> >>>> +	$(call quiet-command, \
> >>>> +            $(PYTHON) -m venv --system-site-packages $@, \
> >>>> +            VENV, $@)
> >>>> +	$(call quiet-command, \
> >>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
> >>>> +            PIP, $(TESTS_VENV_REQ))
> >>>> +	$(call quiet-command, touch $@)
> >>>
> >>> Hmm maybe we should print something like:
> >>>
> >>>   "You can now activate this virtual environment using:
> >>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
> >>
> >> I'm not sure this would be necessary: I expect usage of the venv
> >> to be completely transparent.
> >>
> >> If we require people to learn what venv is and manually activate
> >> it, I'd say we have failed to provide usable tools for running
> >> the tests.
> >>
> > 
> > Actually this is not necessary since the avocado is being called from
> > the "venv python binary" as you can see in the check-acceptance target.
> > 
> > This way all the requirements installed in the test venv can be used
> > without activating the virtual environment.
> 
> Well this is only true if you call 'make check-acceptance', not if you
> want to filter tests, or run the single file you are working on...
> Or am I missing something? The only option user-configurable (without
> activating the venv) is the output of all tests via the AVOCADO_SHOW env
> var.
> 
> This might be enough for a maintainer checking his subsystem, but I
> don't find this practical for a acceptance test writer. And we want for
> people to contribute adding tests, right?
> Well, if we have maintainer running them, this is already a win :)
> 

Good point: these are important use cases too.

Now, we need to decide what's the best interface for performing
those tasks.

Existing unit tests and qtest-based tests use Makefile variables
to select test cases to run.  But I'm not sure this is the most
usable way to do it.

Telling people to manually activate the venv and run avocado
manually doesn't sound desirable to me: people would get a
completely different behavior from `check-acceptance`: they'll
get log files in a different location, and get confused if extra
avocado arguments are required to make some tests work.

Personally, I think most people would be more comfortable using a
simple `./tests/acceptance/run` wrapper script, that would
transparently invoke avocado inside the venv with the right
arguments.

Bonus points if we make it possible to execute single test cases
directly using `python tests/acceptance/mytestcase.py` or
`./tests/acceptance/mytestcase.py`.


> > 
> >>
> >>>
> >>>> +
> >>>> +check-venv: $(TESTS_VENV_DIR)
> >>>> +
> >>>>  # Consolidated targets
> >>>>  
> >>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> >>>> @@ -1030,6 +1049,7 @@ check-clean:
> >>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
> >>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
> >>>>  	rm -f tests/test-qapi-gen-timestamp
> >>>> +	rm -rf $(TESTS_VENV_DIR)
> >>>>  
> >>>>  clean: check-clean
> >>>>  
> >>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
> >>>> new file mode 100644
> >>>> index 0000000000..d39f9d1576
> >>>> --- /dev/null
> >>>> +++ b/tests/venv-requirements.txt
> >>>> @@ -0,0 +1,3 @@
> >>>> +# Add Python module requirements, one per line, to be installed
> >>>> +# in the tests/venv Python virtual environment. For more info,
> >>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> >>>>
> >>>
> >>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> >>
> > 
> > 

-- 
Eduardo

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/12/18 11:37 PM, Eduardo Habkost wrote:
> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
>> Hi Cleber,
>>
>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>> A number of QEMU tests are written in Python, and may benefit
>>> from an untainted Python venv.
>>>
>>> By using make rules, tests that depend on specific Python libs
>>> can set that rule as a requirement, along with rules that require
>>> the presence or installation of specific libraries.
>>>
>>> The tests/venv-requirements.txt is supposed to contain the
>>> Python requirements that should be added to the venv created
>>> by check-venv.
>>
>> Maybe you (or Eduardo...) what you wrote in the cover:
>>
>>  There's one current caveat: it requires Python 3, as it's based on the
>>  venv module.
>>
>> To explain:
>>
>> $ make check-acceptance
>> /usr/bin/python2: No module named venv
>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
> 
> Oops, this doesn't look very friendly.
> 
> But note that this would become a non-issue if we start requiring
> Python 3 for building QEMU.
> 
> 

IIUC, printing a "Can't create tests venv on Python 2", and aborting,
would be enough, right?

- Cleber.

>>
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>  tests/venv-requirements.txt |  3 +++
>>>  2 files changed, 23 insertions(+)
>>>  create mode 100644 tests/venv-requirements.txt
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 5eadfd52f9..b66180efa1 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -12,6 +12,7 @@ check-help:
>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>  	@echo
>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>            TEST, decodetree.py)
>>>  
>>> +# Python venv for running tests
>>> +
>>> +.PHONY: check-venv
>>> +
>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>> +
>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>> +	$(call quiet-command, \
>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>> +            VENV, $@)
>>> +	$(call quiet-command, \
>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>> +            PIP, $(TESTS_VENV_REQ))
>>> +	$(call quiet-command, touch $@)
>>
>> Hmm maybe we should print something like:
>>
>>   "You can now activate this virtual environment using:
>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
> 
> I'm not sure this would be necessary: I expect usage of the venv
> to be completely transparent.
> 
> If we require people to learn what venv is and manually activate
> it, I'd say we have failed to provide usable tools for running
> the tests.
> 
> 
>>
>>> +
>>> +check-venv: $(TESTS_VENV_DIR)
>>> +
>>>  # Consolidated targets
>>>  
>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>  	rm -f tests/test-qapi-gen-timestamp
>>> +	rm -rf $(TESTS_VENV_DIR)
>>>  
>>>  clean: check-clean
>>>  
>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>> new file mode 100644
>>> index 0000000000..d39f9d1576
>>> --- /dev/null
>>> +++ b/tests/venv-requirements.txt
>>> @@ -0,0 +1,3 @@
>>> +# Add Python module requirements, one per line, to be installed
>>> +# in the tests/venv Python virtual environment. For more info,
>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>
>>
>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/15/18 2:41 PM, Caio Carrara wrote:
> On 13-10-2018 00:37, Eduardo Habkost wrote:
>> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
>>> Hi Cleber,
>>>
>>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>>> A number of QEMU tests are written in Python, and may benefit
>>>> from an untainted Python venv.
>>>>
>>>> By using make rules, tests that depend on specific Python libs
>>>> can set that rule as a requirement, along with rules that require
>>>> the presence or installation of specific libraries.
>>>>
>>>> The tests/venv-requirements.txt is supposed to contain the
>>>> Python requirements that should be added to the venv created
>>>> by check-venv.
>>>
>>> Maybe you (or Eduardo...) what you wrote in the cover:
>>>
>>>  There's one current caveat: it requires Python 3, as it's based on the
>>>  venv module.
>>>
>>> To explain:
>>>
>>> $ make check-acceptance
>>> /usr/bin/python2: No module named venv
>>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
>>
>> Oops, this doesn't look very friendly.
>>
>> But note that this would become a non-issue if we start requiring
>> Python 3 for building QEMU.
>>
>>
>>>
>>>>
>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>>  tests/venv-requirements.txt |  3 +++
>>>>  2 files changed, 23 insertions(+)
>>>>  create mode 100644 tests/venv-requirements.txt
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index 5eadfd52f9..b66180efa1 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -12,6 +12,7 @@ check-help:
>>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>>  	@echo
>>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>>            TEST, decodetree.py)
>>>>  
>>>> +# Python venv for running tests
>>>> +
>>>> +.PHONY: check-venv
>>>> +
>>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>>> +
>>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>> +	$(call quiet-command, \
>>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>>> +            VENV, $@)
>>>> +	$(call quiet-command, \
>>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>>> +            PIP, $(TESTS_VENV_REQ))
>>>> +	$(call quiet-command, touch $@)
>>>
>>> Hmm maybe we should print something like:
>>>
>>>   "You can now activate this virtual environment using:
>>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
>>
>> I'm not sure this would be necessary: I expect usage of the venv
>> to be completely transparent.
>>
>> If we require people to learn what venv is and manually activate
>> it, I'd say we have failed to provide usable tools for running
>> the tests.
>>
> 
> Actually this is not necessary since the avocado is being called from
> the "venv python binary" as you can see in the check-acceptance target.
> 

Exactly.

> This way all the requirements installed in the test venv can be used
> without activating the virtual environment.
> 

Exactly.

- Cleber.

>>
>>>
>>>> +
>>>> +check-venv: $(TESTS_VENV_DIR)
>>>> +
>>>>  # Consolidated targets
>>>>  
>>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>>  	rm -f tests/test-qapi-gen-timestamp
>>>> +	rm -rf $(TESTS_VENV_DIR)
>>>>  
>>>>  clean: check-clean
>>>>  
>>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>>> new file mode 100644
>>>> index 0000000000..d39f9d1576
>>>> --- /dev/null
>>>> +++ b/tests/venv-requirements.txt
>>>> @@ -0,0 +1,3 @@
>>>> +# Add Python module requirements, one per line, to be installed
>>>> +# in the tests/venv Python virtual environment. For more info,
>>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>
>>>
>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>
> 
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/15/18 6:28 PM, Philippe Mathieu-Daudé wrote:
> Hi Caio,
> 
> On 15/10/2018 20:41, Caio Carrara wrote:
>> On 13-10-2018 00:37, Eduardo Habkost wrote:
>>> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
>>>> Hi Cleber,
>>>>
>>>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>>>> A number of QEMU tests are written in Python, and may benefit
>>>>> from an untainted Python venv.
>>>>>
>>>>> By using make rules, tests that depend on specific Python libs
>>>>> can set that rule as a requirement, along with rules that require
>>>>> the presence or installation of specific libraries.
>>>>>
>>>>> The tests/venv-requirements.txt is supposed to contain the
>>>>> Python requirements that should be added to the venv created
>>>>> by check-venv.
>>>>
>>>> Maybe you (or Eduardo...) what you wrote in the cover:
>>>>
>>>>  There's one current caveat: it requires Python 3, as it's based on the
>>>>  venv module.
>>>>
>>>> To explain:
>>>>
>>>> $ make check-acceptance
>>>> /usr/bin/python2: No module named venv
>>>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
>>>
>>> Oops, this doesn't look very friendly.
>>>
>>> But note that this would become a non-issue if we start requiring
>>> Python 3 for building QEMU.
>>>
>>>
>>>>
>>>>>
>>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>>> ---
>>>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>>>  tests/venv-requirements.txt |  3 +++
>>>>>  2 files changed, 23 insertions(+)
>>>>>  create mode 100644 tests/venv-requirements.txt
>>>>>
>>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>>> index 5eadfd52f9..b66180efa1 100644
>>>>> --- a/tests/Makefile.include
>>>>> +++ b/tests/Makefile.include
>>>>> @@ -12,6 +12,7 @@ check-help:
>>>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>>>  	@echo
>>>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>>>            TEST, decodetree.py)
>>>>>  
>>>>> +# Python venv for running tests
>>>>> +
>>>>> +.PHONY: check-venv
>>>>> +
>>>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>>>> +
>>>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>>> +	$(call quiet-command, \
>>>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>>>> +            VENV, $@)
>>>>> +	$(call quiet-command, \
>>>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>>>> +            PIP, $(TESTS_VENV_REQ))
>>>>> +	$(call quiet-command, touch $@)
>>>>
>>>> Hmm maybe we should print something like:
>>>>
>>>>   "You can now activate this virtual environment using:
>>>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
>>>
>>> I'm not sure this would be necessary: I expect usage of the venv
>>> to be completely transparent.
>>>
>>> If we require people to learn what venv is and manually activate
>>> it, I'd say we have failed to provide usable tools for running
>>> the tests.
>>>
>>
>> Actually this is not necessary since the avocado is being called from
>> the "venv python binary" as you can see in the check-acceptance target.
>>
>> This way all the requirements installed in the test venv can be used
>> without activating the virtual environment.
> 
> Well this is only true if you call 'make check-acceptance', not if you
> want to filter tests, or run the single file you are working on...
> Or am I missing something? The only option user-configurable (without
> activating the venv) is the output of all tests via the AVOCADO_SHOW env
> var.

Allowing someone to just run `make check-acceptance` is the primary and
most important goal here IMO.  It's supposed to be simple, which means,
not having too many knobs.

> 
> This might be enough for a maintainer checking his subsystem, but I
> don't find this practical for a acceptance test writer. And we want for
> people to contribute adding tests, right?
> Well, if we have maintainer running them, this is already a win :)
> 

Today, with the code AS-IS, you can still do:

 $ tests/venv/bin/avocado run [all your options] tests/acceptance

I think the subsystem and maintainers question will come, but it's a
different discussion.  Contradicting myself now and getting into that
discussion, :P, it could be as simple as `make TAG=foo check-acceptance`
that gets passed to Avocado.  Or, in line with other check targets,
`make check-acceptance-foo`.

- Cleber.

>>
>>>
>>>>
>>>>> +
>>>>> +check-venv: $(TESTS_VENV_DIR)
>>>>> +
>>>>>  # Consolidated targets
>>>>>  
>>>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>>>  	rm -f tests/test-qapi-gen-timestamp
>>>>> +	rm -rf $(TESTS_VENV_DIR)
>>>>>  
>>>>>  clean: check-clean
>>>>>  
>>>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>>>> new file mode 100644
>>>>> index 0000000000..d39f9d1576
>>>>> --- /dev/null
>>>>> +++ b/tests/venv-requirements.txt
>>>>> @@ -0,0 +1,3 @@
>>>>> +# Add Python module requirements, one per line, to be installed
>>>>> +# in the tests/venv Python virtual environment. For more info,
>>>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>>
>>>>
>>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>
>>
>>

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/15/18 6:40 PM, Eduardo Habkost wrote:
> On Tue, Oct 16, 2018 at 12:28:07AM +0200, Philippe Mathieu-Daudé wrote:
>> Hi Caio,
>>
>> On 15/10/2018 20:41, Caio Carrara wrote:
>>> On 13-10-2018 00:37, Eduardo Habkost wrote:
>>>> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
>>>>> Hi Cleber,
>>>>>
>>>>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>>>>> A number of QEMU tests are written in Python, and may benefit
>>>>>> from an untainted Python venv.
>>>>>>
>>>>>> By using make rules, tests that depend on specific Python libs
>>>>>> can set that rule as a requirement, along with rules that require
>>>>>> the presence or installation of specific libraries.
>>>>>>
>>>>>> The tests/venv-requirements.txt is supposed to contain the
>>>>>> Python requirements that should be added to the venv created
>>>>>> by check-venv.
>>>>>
>>>>> Maybe you (or Eduardo...) what you wrote in the cover:
>>>>>
>>>>>  There's one current caveat: it requires Python 3, as it's based on the
>>>>>  venv module.
>>>>>
>>>>> To explain:
>>>>>
>>>>> $ make check-acceptance
>>>>> /usr/bin/python2: No module named venv
>>>>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
>>>>
>>>> Oops, this doesn't look very friendly.
>>>>
>>>> But note that this would become a non-issue if we start requiring
>>>> Python 3 for building QEMU.
>>>>
>>>>
>>>>>
>>>>>>
>>>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>>>> ---
>>>>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>>>>  tests/venv-requirements.txt |  3 +++
>>>>>>  2 files changed, 23 insertions(+)
>>>>>>  create mode 100644 tests/venv-requirements.txt
>>>>>>
>>>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>>>> index 5eadfd52f9..b66180efa1 100644
>>>>>> --- a/tests/Makefile.include
>>>>>> +++ b/tests/Makefile.include
>>>>>> @@ -12,6 +12,7 @@ check-help:
>>>>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>>>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>>>>  	@echo
>>>>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>>>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>>>>            TEST, decodetree.py)
>>>>>>  
>>>>>> +# Python venv for running tests
>>>>>> +
>>>>>> +.PHONY: check-venv
>>>>>> +
>>>>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>>>>> +
>>>>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>>>> +	$(call quiet-command, \
>>>>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>>>>> +            VENV, $@)
>>>>>> +	$(call quiet-command, \
>>>>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>>>>> +            PIP, $(TESTS_VENV_REQ))
>>>>>> +	$(call quiet-command, touch $@)
>>>>>
>>>>> Hmm maybe we should print something like:
>>>>>
>>>>>   "You can now activate this virtual environment using:
>>>>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
>>>>
>>>> I'm not sure this would be necessary: I expect usage of the venv
>>>> to be completely transparent.
>>>>
>>>> If we require people to learn what venv is and manually activate
>>>> it, I'd say we have failed to provide usable tools for running
>>>> the tests.
>>>>
>>>
>>> Actually this is not necessary since the avocado is being called from
>>> the "venv python binary" as you can see in the check-acceptance target.
>>>
>>> This way all the requirements installed in the test venv can be used
>>> without activating the virtual environment.
>>
>> Well this is only true if you call 'make check-acceptance', not if you
>> want to filter tests, or run the single file you are working on...
>> Or am I missing something? The only option user-configurable (without
>> activating the venv) is the output of all tests via the AVOCADO_SHOW env
>> var.
>>
>> This might be enough for a maintainer checking his subsystem, but I
>> don't find this practical for a acceptance test writer. And we want for
>> people to contribute adding tests, right?
>> Well, if we have maintainer running them, this is already a win :)
>>
> 
> Good point: these are important use cases too.
> 
> Now, we need to decide what's the best interface for performing
> those tasks.
> 
> Existing unit tests and qtest-based tests use Makefile variables
> to select test cases to run.  But I'm not sure this is the most
> usable way to do it.
> 

I also fear about getting too deep into the Makefiles, adding content
for every new test, etc.  It's certainly not the way to go here.

> Telling people to manually activate the venv and run avocado
> manually doesn't sound desirable to me: people would get a
> completely different behavior from `check-acceptance`: they'll
> get log files in a different location, and get confused if extra
> avocado arguments are required to make some tests work.
> 

Agreed.  The whole point of this work, IMO, is to provide a seamless and
transparent way to execute the most common task.  At this point, we
should be telling people to run *all* tests we have, so selecting
specific tests is something that we have some time to deal with.

> Personally, I think most people would be more comfortable using a
> simple `./tests/acceptance/run` wrapper script, that would
> transparently invoke avocado inside the venv with the right
> arguments.
> 

I have something in mind which seems to relate to your idea of `run`.
Basically, when we get to the point of having more complex test suites,
we can have "avocado job scripts", using the "Job API", to create and
run jobs with a specific selection of tests and custom options (such as
specific varianters for some tests, etc).

For instance, we may want to have a "job_storage_migration.py", an
Avocado job script (not a test), that includes a pre-tests plugin
execution that sets up some storage, a test suitewith a few acceptance
tests, another test suite with some iotests run with different variants,
and a post-tests plugin that cleans up the environment.

Until then, I don't know what I would put into `run`.  A command that
calls `make check-acceptance`?  I'm confused by that.

> Bonus points if we make it possible to execute single test cases
> directly using `python tests/acceptance/mytestcase.py` or
> `./tests/acceptance/mytestcase.py`.
> 

This is possible with:

#!/usr/bin/env python

from avocado import main

[test]

if __name__ == "__main__":
    main()

But is it really worth it?  IMO, it's not.

- Cleber.

> 
>>>
>>>>
>>>>>
>>>>>> +
>>>>>> +check-venv: $(TESTS_VENV_DIR)
>>>>>> +
>>>>>>  # Consolidated targets
>>>>>>  
>>>>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>>>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>>>>  	rm -f tests/test-qapi-gen-timestamp
>>>>>> +	rm -rf $(TESTS_VENV_DIR)
>>>>>>  
>>>>>>  clean: check-clean
>>>>>>  
>>>>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>>>>> new file mode 100644
>>>>>> index 0000000000..d39f9d1576
>>>>>> --- /dev/null
>>>>>> +++ b/tests/venv-requirements.txt
>>>>>> @@ -0,0 +1,3 @@
>>>>>> +# Add Python module requirements, one per line, to be installed
>>>>>> +# in the tests/venv Python virtual environment. For more info,
>>>>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>>>
>>>>>
>>>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>>
>>>
>>>
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/15/18 3:04 PM, Caio Carrara wrote:
> On 12-10-2018 13:53, Cleber Rosa wrote:
>> A number of QEMU tests are written in Python, and may benefit
>> from an untainted Python venv.
>>
>> By using make rules, tests that depend on specific Python libs
>> can set that rule as a requirement, along with rules that require
>> the presence or installation of specific libraries.
>>
>> The tests/venv-requirements.txt is supposed to contain the
>> Python requirements that should be added to the venv created
>> by check-venv.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>  tests/venv-requirements.txt |  3 +++
> 
> Any special reason to name `venv-requirements.txt` instead of only
> `requirements.txt`? I think the second way is better (if there's no
> cons) since it's the default from most of Python projects. Besides that
> seems more semantic to have a `tests/requirements.txt` file that
> registers the requirements for tests.
> 

It was a decision based on the fact that other components may have other
requirements, basically a namespace thing and a "explicit is better than
implicit" thing.

The env-requirements.txt file is not for "tests" in general, but for the
tests that we want to be run out of the venv.  This can quickly become a
different thing (look at the tests/docker, tests/vm, tests/migration, etc).

> Still in this topic. As far as I could see, the files in `tests/`
> directory are almost all C source code. The right place for the Python
> requirements seems the `tests/acceptance/` directory specially because
> the subject of all this patches. Keeping only one requirements file for
> all kinds of tests (that uses Python) can be problematic in my opinion.
> If it makes any sense, probably is also a good idea rename the target
> from `check-venv` to something like `check-acceptance-venv`.
> 

I agree with some of your points here.  The way I see this working is
either:

1) Have a single venv created for running Python based tests, and have
its requirements listed in tests/venv-requirements.txt

2) Have one venv created for every type of Python based tests, and have
requirements.txt in the specific directories (such as
tests/acceptance/requirements.txt, tests/vm/requirements.txt), and
multiple `check-[type]-venv` targets.

I obviously chose 1, simply because I find it overkill to have different
venvs at this point.  I don't see the additional value option #2 would
have at this point.

- Cleber.

>>  2 files changed, 23 insertions(+)
>>  create mode 100644 tests/venv-requirements.txt
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 5eadfd52f9..b66180efa1 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -12,6 +12,7 @@ check-help:
>>  	@echo " $(MAKE) check-block          Run block tests"
>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>  	@echo
>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>            TEST, decodetree.py)
>>  
>> +# Python venv for running tests
>> +
>> +.PHONY: check-venv
>> +
>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>> +
>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>> +	$(call quiet-command, \
>> +            $(PYTHON) -m venv --system-site-packages $@, \
>> +            VENV, $@)
>> +	$(call quiet-command, \
>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>> +            PIP, $(TESTS_VENV_REQ))
>> +	$(call quiet-command, touch $@)
>> +
>> +check-venv: $(TESTS_VENV_DIR)
>> +
>>  # Consolidated targets
>>  
>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>> @@ -1030,6 +1049,7 @@ check-clean:
>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>  	rm -f tests/test-qapi-gen-timestamp
>> +	rm -rf $(TESTS_VENV_DIR)
>>  
>>  clean: check-clean
>>  
>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>> new file mode 100644
>> index 0000000000..d39f9d1576
>> --- /dev/null
>> +++ b/tests/venv-requirements.txt
>> @@ -0,0 +1,3 @@
>> +# Add Python module requirements, one per line, to be installed
>> +# in the tests/venv Python virtual environment. For more info,
>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>
> 
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

Le mar. 16 oct. 2018 16:08, Cleber Rosa <crosa@redhat.com> a écrit :

>
>
> On 10/15/18 6:40 PM, Eduardo Habkost wrote:
> > On Tue, Oct 16, 2018 at 12:28:07AM +0200, Philippe Mathieu-Daudé wrote:
> >> Hi Caio,
> >>
> >> On 15/10/2018 20:41, Caio Carrara wrote:
> >>> On 13-10-2018 00:37, Eduardo Habkost wrote:
> >>>> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé
> wrote:
> >>>>> Hi Cleber,
> >>>>>
> >>>>> On 12/10/2018 18:53, Cleber Rosa wrote:
> >>>>>> A number of QEMU tests are written in Python, and may benefit
> >>>>>> from an untainted Python venv.
> >>>>>>
> >>>>>> By using make rules, tests that depend on specific Python libs
> >>>>>> can set that rule as a requirement, along with rules that require
> >>>>>> the presence or installation of specific libraries.
> >>>>>>
> >>>>>> The tests/venv-requirements.txt is supposed to contain the
> >>>>>> Python requirements that should be added to the venv created
> >>>>>> by check-venv.
> >>>>>
> >>>>> Maybe you (or Eduardo...) what you wrote in the cover:
> >>>>>
> >>>>>  There's one current caveat: it requires Python 3, as it's based on
> the
> >>>>>  venv module.
> >>>>>
> >>>>> To explain:
> >>>>>
> >>>>> $ make check-acceptance
> >>>>> /usr/bin/python2: No module named venv
> >>>>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:]
> Error 1
> >>>>
> >>>> Oops, this doesn't look very friendly.
> >>>>
> >>>> But note that this would become a non-issue if we start requiring
> >>>> Python 3 for building QEMU.
> >>>>
> >>>>
> >>>>>
> >>>>>>
> >>>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> >>>>>> ---
> >>>>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
> >>>>>>  tests/venv-requirements.txt |  3 +++
> >>>>>>  2 files changed, 23 insertions(+)
> >>>>>>  create mode 100644 tests/venv-requirements.txt
> >>>>>>
> >>>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>>>>> index 5eadfd52f9..b66180efa1 100644
> >>>>>> --- a/tests/Makefile.include
> >>>>>> +++ b/tests/Makefile.include
> >>>>>> @@ -12,6 +12,7 @@ check-help:
> >>>>>>          @echo " $(MAKE) check-block          Run block tests"
> >>>>>>          @echo " $(MAKE) check-tcg            Run TCG tests"
> >>>>>>          @echo " $(MAKE) check-report.html    Generates an HTML
> test report"
> >>>>>> +        @echo " $(MAKE) check-venv           Creates a Python venv
> for tests"
> >>>>>>          @echo " $(MAKE) check-clean          Clean the tests"
> >>>>>>          @echo
> >>>>>>          @echo "Please note that HTML reports do not regenerate if
> the unit tests"
> >>>>>> @@ -1017,6 +1018,24 @@ check-decodetree:
> >>>>>>            ./check.sh "$(PYTHON)"
> "$(SRC_PATH)/scripts/decodetree.py", \
> >>>>>>            TEST, decodetree.py)
> >>>>>>
> >>>>>> +# Python venv for running tests
> >>>>>> +
> >>>>>> +.PHONY: check-venv
> >>>>>> +
> >>>>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> >>>>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
> >>>>>> +
> >>>>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> >>>>>> +        $(call quiet-command, \
> >>>>>> +            $(PYTHON) -m venv --system-site-packages $@, \
> >>>>>> +            VENV, $@)
> >>>>>> +        $(call quiet-command, \
> >>>>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r
> $(TESTS_VENV_REQ), \
> >>>>>> +            PIP, $(TESTS_VENV_REQ))
> >>>>>> +        $(call quiet-command, touch $@)
> >>>>>
> >>>>> Hmm maybe we should print something like:
> >>>>>
> >>>>>   "You can now activate this virtual environment using:
> >>>>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
> >>>>
> >>>> I'm not sure this would be necessary: I expect usage of the venv
> >>>> to be completely transparent.
> >>>>
> >>>> If we require people to learn what venv is and manually activate
> >>>> it, I'd say we have failed to provide usable tools for running
> >>>> the tests.
> >>>>
> >>>
> >>> Actually this is not necessary since the avocado is being called from
> >>> the "venv python binary" as you can see in the check-acceptance target.
> >>>
> >>> This way all the requirements installed in the test venv can be used
> >>> without activating the virtual environment.
> >>
> >> Well this is only true if you call 'make check-acceptance', not if you
> >> want to filter tests, or run the single file you are working on...
> >> Or am I missing something? The only option user-configurable (without
> >> activating the venv) is the output of all tests via the AVOCADO_SHOW env
> >> var.
> >>
> >> This might be enough for a maintainer checking his subsystem, but I
> >> don't find this practical for a acceptance test writer. And we want for
> >> people to contribute adding tests, right?
> >> Well, if we have maintainer running them, this is already a win :)
> >>
> >
> > Good point: these are important use cases too.
> >
> > Now, we need to decide what's the best interface for performing
> > those tasks.
> >
> > Existing unit tests and qtest-based tests use Makefile variables
> > to select test cases to run.  But I'm not sure this is the most
> > usable way to do it.
> >
>
> I also fear about getting too deep into the Makefiles, adding content
> for every new test, etc.  It's certainly not the way to go here.
>
> > Telling people to manually activate the venv and run avocado
> > manually doesn't sound desirable to me: people would get a
> > completely different behavior from `check-acceptance`: they'll
> > get log files in a different location, and get confused if extra
> > avocado arguments are required to make some tests work.
> >
>
> Agreed.  The whole point of this work, IMO, is to provide a seamless and
> transparent way to execute the most common task.  At this point, we
> should be telling people to run *all* tests we have, so selecting
> specific tests is something that we have some time to deal with.
>
> > Personally, I think most people would be more comfortable using a
> > simple `./tests/acceptance/run` wrapper script, that would
> > transparently invoke avocado inside the venv with the right
> > arguments.
> >
>
> I have something in mind which seems to relate to your idea of `run`.
> Basically, when we get to the point of having more complex test suites,
> we can have "avocado job scripts", using the "Job API", to create and
> run jobs with a specific selection of tests and custom options (such as
> specific varianters for some tests, etc).
>
> For instance, we may want to have a "job_storage_migration.py", an
> Avocado job script (not a test), that includes a pre-tests plugin
> execution that sets up some storage, a test suitewith a few acceptance
> tests, another test suite with some iotests run with different variants,
> and a post-tests plugin that cleans up the environment.
>
> Until then, I don't know what I would put into `run`.  A command that
> calls `make check-acceptance`?  I'm confused by that.
>
> > Bonus points if we make it possible to execute single test cases
> > directly using `python tests/acceptance/mytestcase.py` or
> > `./tests/acceptance/mytestcase.py`.
> >
>
> This is possible with:
>
> #!/usr/bin/env python
>
> from avocado import main
>
> [test]
>
> if __name__ == "__main__":
>     main()


> But is it really worth it?  IMO, it's not.
>

If that seamlessly uses the venv, I think it is.


> - Cleber.
>
> >
> >>>
> >>>>
> >>>>>
> >>>>>> +
> >>>>>> +check-venv: $(TESTS_VENV_DIR)
> >>>>>> +
> >>>>>>  # Consolidated targets
> >>>>>>
> >>>>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> >>>>>> @@ -1030,6 +1049,7 @@ check-clean:
> >>>>>>          rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
> >>>>>>          rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST),
> $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
> >>>>>>          rm -f tests/test-qapi-gen-timestamp
> >>>>>> +        rm -rf $(TESTS_VENV_DIR)
> >>>>>>
> >>>>>>  clean: check-clean
> >>>>>>
> >>>>>> diff --git a/tests/venv-requirements.txt
> b/tests/venv-requirements.txt
> >>>>>> new file mode 100644
> >>>>>> index 0000000000..d39f9d1576
> >>>>>> --- /dev/null
> >>>>>> +++ b/tests/venv-requirements.txt
> >>>>>> @@ -0,0 +1,3 @@
> >>>>>> +# Add Python module requirements, one per line, to be installed
> >>>>>> +# in the tests/venv Python virtual environment. For more info,
> >>>>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> >>>>>>
> >>>>>
> >>>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> >>>>
> >>>
> >>>
> >
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/15/18 6:22 PM, Philippe Mathieu-Daudé wrote:
> On 15/10/2018 21:04, Caio Carrara wrote:
>> On 12-10-2018 13:53, Cleber Rosa wrote:
>>> A number of QEMU tests are written in Python, and may benefit
>>> from an untainted Python venv.
>>>
>>> By using make rules, tests that depend on specific Python libs
>>> can set that rule as a requirement, along with rules that require
>>> the presence or installation of specific libraries.
>>>
>>> The tests/venv-requirements.txt is supposed to contain the
>>> Python requirements that should be added to the venv created
>>> by check-venv.
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>  tests/venv-requirements.txt |  3 +++
>>
>> Any special reason to name `venv-requirements.txt` instead of only
>> `requirements.txt`? I think the second way is better (if there's no
>> cons) since it's the default from most of Python projects. Besides that
>> seems more semantic to have a `tests/requirements.txt` file that
>> registers the requirements for tests.
> 
> Agreed.
> 

I found it to be more explicit.  Naming it "requirements.txt", is
usually done in first-level directories of Python projects.  So, I'd put
it in "tests/acceptance/requirements.txt" if it was supposed to be used
*only* by acceptance tests (see previous response).

But, if the majority prefers the more traditional name, I can certainly
go with it.

>>
>> Still in this topic. As far as I could see, the files in `tests/`
>> directory are almost all C source code. The right place for the Python
>> requirements seems the `tests/acceptance/` directory specially because
>> the subject of all this patches. Keeping only one requirements file for
>> all kinds of tests (that uses Python) can be problematic in my opinion.
>> If it makes any sense, probably is also a good idea rename the target
>> from `check-venv` to something like `check-acceptance-venv`.
> 
> Yes.
> 

I disagree because that was not my original motivation.  As explained in
the previous response, the motivation was to have a venv that could be
used for running all Python based tests.

- Cleber.

>>
>>>  2 files changed, 23 insertions(+)
>>>  create mode 100644 tests/venv-requirements.txt
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 5eadfd52f9..b66180efa1 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -12,6 +12,7 @@ check-help:
>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>  	@echo
>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>            TEST, decodetree.py)
>>>  
>>> +# Python venv for running tests
>>> +
>>> +.PHONY: check-venv
>>> +
>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>> +
>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>> +	$(call quiet-command, \
>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>> +            VENV, $@)
>>> +	$(call quiet-command, \
>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>> +            PIP, $(TESTS_VENV_REQ))
>>> +	$(call quiet-command, touch $@)
>>> +
>>> +check-venv: $(TESTS_VENV_DIR)
>>> +
>>>  # Consolidated targets
>>>  
>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>  	rm -f tests/test-qapi-gen-timestamp
>>> +	rm -rf $(TESTS_VENV_DIR)
>>>  
>>>  clean: check-clean
>>>  
>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>> new file mode 100644
>>> index 0000000000..d39f9d1576
>>> --- /dev/null
>>> +++ b/tests/venv-requirements.txt
>>> @@ -0,0 +1,3 @@
>>> +# Add Python module requirements, one per line, to be installed
>>> +# in the tests/venv Python virtual environment. For more info,
>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>
>>
>>

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [PATCH v4 2/3] Acceptance tests: add make rule for running them

[Cleber Rosa]

On 10/12/18 5:37 PM, Philippe Mathieu-Daudé wrote:
> On 12/10/2018 18:53, Cleber Rosa wrote:
>> The acceptance (aka functional, aka Avocado-based) tests are
>> Python files located in "tests/acceptance" that need to be run
>> with the Avocado libs and test runner.
>>
>> Let's provide a convenient way for QEMU developers to run them,
>> by making use of the tests-venv with the required setup.
>>
>> Also, while the Avocado test runner will take care of creating a
>> location to save test results to, it was understood that it's better
>> if the results are kept within the build tree.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>>  tests/Makefile.include      | 21 +++++++++++++++++++--
>>  tests/venv-requirements.txt |  1 +
>>  3 files changed, 50 insertions(+), 7 deletions(-)
>>
>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>> index 727c4019b5..b992a2961d 100644
>> --- a/docs/devel/testing.rst
>> +++ b/docs/devel/testing.rst
>> @@ -545,10 +545,31 @@ Tests based on ``avocado_qemu.Test`` can easily:
>>     - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
>>     - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
>>  
>> -Installation
>> -------------
>> +Running tests
>> +-------------
>>  
>> -To install Avocado and its dependencies, run:
>> +You can run the acceptance tests simply by executing:
>> +
>> +.. code::
>> +
>> +  make check-acceptance
>> +
>> +This involves the automatic creation of Python virtual environment
>> +within the build tree (at ``tests/venv``) which will have all the
>> +right dependencies, and will save tests results also within the
>> +build tree (at ``tests/results``).
> 
> Missing: how to activate the virtualenv.
> 

I think there's a clear difference in expectations here... the goal of
the venv is not be emphasized... IMO it's supposed to be as transparent
as possible.

>> +
>> +Note: the build environment must be using a Python 3 stack, and have
>> +the ``venv`` and ``pip`` packages installed.  If necessary, make sure
>> +``configure`` is called with ``--python=`` and that those modules are
>> +available.  On Debian and Ubuntu based systems, depending on the
>> +specific version, they may be on packages named ``python3-venv`` and
>> +``python3-pip``.
>> +
>> +Manual Installation
>> +-------------------
>> +
>> +To manually install Avocado and its dependencies, run:
>>  
>>  .. code::
>>  
>> @@ -689,11 +710,15 @@ The exact QEMU binary to be used on QEMUMachine.
>>  Uninstalling Avocado
>>  --------------------
>>  
>> -If you've followed the installation instructions above, you can easily
>> -uninstall Avocado.  Start by listing the packages you have installed::
>> +If you've followed the manual installation instructions above, you can
>> +easily uninstall Avocado.  Start by listing the packages you have
>> +installed::
>>  
>>    pip list --user
>>  
>>  And remove any package you want with::
>>  
>>    pip uninstall <package_name>
>> +
>> +If you've used ``make check-acceptance``, the Python virtual environment where
>> +Avocado is installed will be cleaned up as part of ``make check-clean``.
> 
> Add a line about deactivating the venv?
> 

Ditto here.  Anyway, I can change it if most people think the venv usage
should be explicit (I'm clearly against it, though).

- Cleber.

>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index b66180efa1..75547fc947 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -11,6 +11,7 @@ check-help:
>>  	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
>>  	@echo " $(MAKE) check-block          Run block tests"
>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>> +	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>  	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>  	@echo " $(MAKE) check-clean          Clean the tests"
>> @@ -1020,10 +1021,15 @@ check-decodetree:
>>  
>>  # Python venv for running tests
>>  
>> -.PHONY: check-venv
>> +.PHONY: check-venv check-acceptance
>>  
>>  TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>  TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>> +# Controls the output generated by Avocado when running tests.
>> +# Any number of command separated loggers are accepted.  For more
>> +# information please refer to "avocado --help".
>> +AVOCADO_SHOW=none
>>  
>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>  	$(call quiet-command, \
>> @@ -1034,8 +1040,19 @@ $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>              PIP, $(TESTS_VENV_REQ))
>>  	$(call quiet-command, touch $@)
>>  
>> +$(TESTS_RESULTS_DIR):
>> +	$(call quiet-command, mkdir -p $@, \
>> +            MKDIR, $@)
>> +
>>  check-venv: $(TESTS_VENV_DIR)
>>  
>> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
>> +	$(call quiet-command, \
>> +            $(TESTS_VENV_DIR)/bin/python -m avocado \
>> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> +            --failfast=on $(SRC_PATH)/tests/acceptance, \
>> +            "AVOCADO", "tests/acceptance")
>> +
>>  # Consolidated targets
>>  
>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>> @@ -1049,7 +1066,7 @@ check-clean:
>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>  	rm -f tests/test-qapi-gen-timestamp
>> -	rm -rf $(TESTS_VENV_DIR)
>> +	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
>>  
>>  clean: check-clean
>>  
>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>> index d39f9d1576..64c6e27a94 100644
>> --- a/tests/venv-requirements.txt
>> +++ b/tests/venv-requirements.txt
>> @@ -1,3 +1,4 @@
>>  # Add Python module requirements, one per line, to be installed
>>  # in the tests/venv Python virtual environment. For more info,
>>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>> +avocado-framework==65.0
>>
> 
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> 

Re: [Qemu-devel] [PATCH v4 0/3] Bootstrap Python venv and acceptance/functional tests

[Cleber Rosa]

On 10/12/18 5:44 PM, Philippe Mathieu-Daudé wrote:
> On 12/10/2018 18:53, Cleber Rosa wrote:
>> TL;DR
>> =====
>>
>> Allow acceptance tests to be run with `make check-acceptance`.
>>
>> Details
>> =======
>>
>> This introduces a Python virtual environment that will be setup within
>> the QEMU build directory, that will contain the exact environment that
>> tests may require.
>>
>> There's one current caveat: it requires Python 3, as it's based on the
>> venv module.  This was based on some discussions and perception about
>> standardizing on Python 3, but can easily be made to accommodate Python
>> 2 as well.
>>
>> Example of bootstrap and test execution on Travis-CI:
>>
>> https://travis-ci.org/qemu/qemu/jobs/439331028#L2508
> 
> If you activate Travis on your github account, you can test that in your
> namespace without having to open zombie pull requests there... A simple
> push to your repository will trigger a full Travis build.
> 
> This is how I use it btw, canceling the jobs I'm not interested in, to
> quickly run the others.
> i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600
> 

Yep, I've done that shortly after sending this. :)  Thanks for the tip,
though.

One thing I miss is the ability, without editing the .travis.yaml, to
"choose" just one/few jobs in a build.  Or maybe even the order they run
(Of course I want the acceptance tests job first).

Do you know anything about that?

- Cleber.

>>
>>    ...
>>       VENV    /home/travis/build/qemu/qemu/tests/venv
>>       MKDIR   /home/travis/build/qemu/qemu/tests/results
>>       PIP     /home/travis/build/qemu/qemu/tests/venv-requirements.txt
>>       AVOCADO tests/acceptance
>>     JOB ID     : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d
>>     JOB LOG    : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log
>>      (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test:  PASS (3.57 s)
>>      (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.04 s)
>>      (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>      (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>      (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.04 s)
>>      (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.04 s)
>>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>     JOB TIME   : 3.90 s
>>    ...
>>
>> Changes from v3:
>> ================
>>
>>  * Fixed typo in commit message (s/requiment/requirement/).  (Eric)
>>
>> Changes from v2:
>> ================
>>
>>  * Make the $(TESTS_VENV_DIR) target depend on the
>>    venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv
>>    runs.  With this, updates on the file are reflected on the
>>    venv. (Philippe)
>>
>>  * Run pip with "python -m pip".  It may have been installed reusing
>>    the system wide packages, and then the script may not be available
>>    on the venv. (Philippe)
>>
>>  * Dropped Python version on Travis, and using the version supplied
>>    by the distro (3.4). (Philippe)
>>
>>  * Added "python3.4-venv" package requirement on Travis. (Philippe)
>>
>>  * Added variable (AVOCADO_SHOW) with logging streams to be shown
>>    while running the acceptance tests.  By default it's set to none,
>>    the equivalent of the quiet mode used on previous versions.
>>    (Philippe)
>>
>>  * On Travis, set the AVOCADO_SHOW variable to "app", so that the
>>    individual test results can be easily seen.  (Philippe)
>>
>> Ideas discussed, but not implemented:
>>
>>   * Run pip with "$(PYTHON) -m pip -q install ..." because it points
>>     to the system wide Python installation. (Philippe)
>>
>>   * Drop the "--system-site-packages" flag.  Waiting on another round
>>     of tests to determine if they are really the cause of some package
>>     installation problems.
>>
>> Changes from v1:
>> ================
>>
>>  * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
>>    the source path ($SRC_PATH instead of $BUILD_DIR)
>>
>>  * Create the venv with "--system-site-packages", which allows the
>>    reuse of packages (and no additional downloads) in case there's a
>>    package installed system wide providing the same package and
>>    version.
>>
>>  * Run Avocado with "python -m avocado".  It may have been installed
>>    reusing the system wide packages, and then the script may not
>>    be available on the venv.
>>
>>  * Improved documentation describing the Python 3, venv and pip
>>    requirements.
>>
>>  * Updated avocado-framework requirement to latest released version
>>    (65.0)
>>
>>  * (New commit) Added support for running the acceptance tests on
>>    Travis.
>>
>> Ideas discussed, but not implemented:
>>
>>  * Install external packages such as python3-pip on Debian based
>>    systems, deemed too invasive on developer's systems.
>>
>>  * Allow the use of Python 2, and consequently the "virtualenv"
>>    module.
>>
>> Cleber Rosa (3):
>>   Bootstrap Python venv for tests
>>   Acceptance tests: add make rule for running them
>>   Travis support for the acceptance tests
>>
>>  .travis.yml                 |  5 +++++
>>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>>  tests/Makefile.include      | 37 +++++++++++++++++++++++++++++++++++++
>>  tests/venv-requirements.txt |  4 ++++
>>  4 files changed, 76 insertions(+), 5 deletions(-)
>>  create mode 100644 tests/venv-requirements.txt
>>
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/16/18 10:20 AM, Philippe Mathieu-Daudé wrote:

>>
>> This is possible with:
>>
>> #!/usr/bin/env python
>>
>> from avocado import main
>>
>> [test]
>>
>> if __name__ == "__main__":
>>     main()
> 
> 
>> But is it really worth it?  IMO, it's not.
>>
> 
> If that seamlessly uses the venv, I think it is.
> 
>

I would not seamlessly use the venv, unless the venv is activated.

To me, we're clearly trying to stretch a very simple thing (the venv)
into something that should automagically do a lot more, but can't easily
do so.

- Cleber.

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

On 16/10/2018 15:50, Cleber Rosa wrote:

> On 10/12/18 11:37 PM, Eduardo Habkost wrote:
>> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
>>> Hi Cleber,
>>>
>>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>>> A number of QEMU tests are written in Python, and may benefit
>>>> from an untainted Python venv.
>>>>
>>>> By using make rules, tests that depend on specific Python libs
>>>> can set that rule as a requirement, along with rules that require
>>>> the presence or installation of specific libraries.
>>>>
>>>> The tests/venv-requirements.txt is supposed to contain the
>>>> Python requirements that should be added to the venv created
>>>> by check-venv.
>>>
>>> Maybe you (or Eduardo...) what you wrote in the cover:
>>>
>>>  There's one current caveat: it requires Python 3, as it's based on the
>>>  venv module.
>>>
>>> To explain:
>>>
>>> $ make check-acceptance
>>> /usr/bin/python2: No module named venv
>>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
>>
>> Oops, this doesn't look very friendly.
>>
>> But note that this would become a non-issue if we start requiring
>> Python 3 for building QEMU.
>>
>>
> 
> IIUC, printing a "Can't create tests venv on Python 2", and aborting,
> would be enough, right?

Certainly.

> 
> - Cleber.
> 
>>>
>>>>
>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>>  tests/venv-requirements.txt |  3 +++
>>>>  2 files changed, 23 insertions(+)
>>>>  create mode 100644 tests/venv-requirements.txt
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index 5eadfd52f9..b66180efa1 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -12,6 +12,7 @@ check-help:
>>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>>  	@echo
>>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>>            TEST, decodetree.py)
>>>>  
>>>> +# Python venv for running tests
>>>> +
>>>> +.PHONY: check-venv
>>>> +
>>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>>> +
>>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>> +	$(call quiet-command, \
>>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>>> +            VENV, $@)
>>>> +	$(call quiet-command, \
>>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>>> +            PIP, $(TESTS_VENV_REQ))
>>>> +	$(call quiet-command, touch $@)
>>>
>>> Hmm maybe we should print something like:
>>>
>>>   "You can now activate this virtual environment using:
>>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
>>
>> I'm not sure this would be necessary: I expect usage of the venv
>> to be completely transparent.
>>
>> If we require people to learn what venv is and manually activate
>> it, I'd say we have failed to provide usable tools for running
>> the tests.
>>
>>
>>>
>>>> +
>>>> +check-venv: $(TESTS_VENV_DIR)
>>>> +
>>>>  # Consolidated targets
>>>>  
>>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>>  	rm -f tests/test-qapi-gen-timestamp
>>>> +	rm -rf $(TESTS_VENV_DIR)
>>>>  
>>>>  clean: check-clean
>>>>  
>>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>>> new file mode 100644
>>>> index 0000000000..d39f9d1576
>>>> --- /dev/null
>>>> +++ b/tests/venv-requirements.txt
>>>> @@ -0,0 +1,3 @@
>>>> +# Add Python module requirements, one per line, to be installed
>>>> +# in the tests/venv Python virtual environment. For more info,
>>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>
>>>
>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>
> 

Re: [Qemu-devel] [PATCH v4 1/3] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

On 16/10/2018 15:56, Cleber Rosa wrote:
> On 10/15/18 6:28 PM, Philippe Mathieu-Daudé wrote:
>> Hi Caio,
>>
>> On 15/10/2018 20:41, Caio Carrara wrote:
>>> On 13-10-2018 00:37, Eduardo Habkost wrote:
>>>> On Fri, Oct 12, 2018 at 11:30:39PM +0200, Philippe Mathieu-Daudé wrote:
>>>>> Hi Cleber,
>>>>>
>>>>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>>>>> A number of QEMU tests are written in Python, and may benefit
>>>>>> from an untainted Python venv.
>>>>>>
>>>>>> By using make rules, tests that depend on specific Python libs
>>>>>> can set that rule as a requirement, along with rules that require
>>>>>> the presence or installation of specific libraries.
>>>>>>
>>>>>> The tests/venv-requirements.txt is supposed to contain the
>>>>>> Python requirements that should be added to the venv created
>>>>>> by check-venv.
>>>>>
>>>>> Maybe you (or Eduardo...) what you wrote in the cover:
>>>>>
>>>>>  There's one current caveat: it requires Python 3, as it's based on the
>>>>>  venv module.
>>>>>
>>>>> To explain:
>>>>>
>>>>> $ make check-acceptance
>>>>> /usr/bin/python2: No module named venv
>>>>> make: *** [/home/phil/source/qemu/tests/Makefile.include:1033:] Error 1
>>>>
>>>> Oops, this doesn't look very friendly.
>>>>
>>>> But note that this would become a non-issue if we start requiring
>>>> Python 3 for building QEMU.
>>>>
>>>>
>>>>>
>>>>>>
>>>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>>>> ---
>>>>>>  tests/Makefile.include      | 20 ++++++++++++++++++++
>>>>>>  tests/venv-requirements.txt |  3 +++
>>>>>>  2 files changed, 23 insertions(+)
>>>>>>  create mode 100644 tests/venv-requirements.txt
>>>>>>
>>>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>>>> index 5eadfd52f9..b66180efa1 100644
>>>>>> --- a/tests/Makefile.include
>>>>>> +++ b/tests/Makefile.include
>>>>>> @@ -12,6 +12,7 @@ check-help:
>>>>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>>>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>>>>  	@echo
>>>>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>>>>> @@ -1017,6 +1018,24 @@ check-decodetree:
>>>>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>>>>            TEST, decodetree.py)
>>>>>>  
>>>>>> +# Python venv for running tests
>>>>>> +
>>>>>> +.PHONY: check-venv
>>>>>> +
>>>>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>>>>> +
>>>>>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>>>> +	$(call quiet-command, \
>>>>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>>>>> +            VENV, $@)
>>>>>> +	$(call quiet-command, \
>>>>>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>>>>>> +            PIP, $(TESTS_VENV_REQ))
>>>>>> +	$(call quiet-command, touch $@)
>>>>>
>>>>> Hmm maybe we should print something like:
>>>>>
>>>>>   "You can now activate this virtual environment using:
>>>>>     source $(TESTS_VENV_DIR)/tests/venv/bin/activate"
>>>>
>>>> I'm not sure this would be necessary: I expect usage of the venv
>>>> to be completely transparent.
>>>>
>>>> If we require people to learn what venv is and manually activate
>>>> it, I'd say we have failed to provide usable tools for running
>>>> the tests.
>>>>
>>>
>>> Actually this is not necessary since the avocado is being called from
>>> the "venv python binary" as you can see in the check-acceptance target.
>>>
>>> This way all the requirements installed in the test venv can be used
>>> without activating the virtual environment.
>>
>> Well this is only true if you call 'make check-acceptance', not if you
>> want to filter tests, or run the single file you are working on...
>> Or am I missing something? The only option user-configurable (without
>> activating the venv) is the output of all tests via the AVOCADO_SHOW env
>> var.
> 
> Allowing someone to just run `make check-acceptance` is the primary and
> most important goal here IMO.  It's supposed to be simple, which means,
> not having too many knobs.
> 
>>
>> This might be enough for a maintainer checking his subsystem, but I
>> don't find this practical for a acceptance test writer. And we want for
>> people to contribute adding tests, right?
>> Well, if we have maintainer running them, this is already a win :)
>>
> 
> Today, with the code AS-IS, you can still do:
> 
>  $ tests/venv/bin/avocado run [all your options] tests/acceptance

If this is documented, we don't need the other tricks and your series is
fine as it!

> 
> I think the subsystem and maintainers question will come, but it's a
> different discussion.  Contradicting myself now and getting into that
> discussion, :P, it could be as simple as `make TAG=foo check-acceptance`
> that gets passed to Avocado.  Or, in line with other check targets,
> `make check-acceptance-foo`.
> 
> - Cleber.
> 
>>>
>>>>
>>>>>
>>>>>> +
>>>>>> +check-venv: $(TESTS_VENV_DIR)
>>>>>> +
>>>>>>  # Consolidated targets
>>>>>>  
>>>>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>>>>> @@ -1030,6 +1049,7 @@ check-clean:
>>>>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>>>>  	rm -f tests/test-qapi-gen-timestamp
>>>>>> +	rm -rf $(TESTS_VENV_DIR)
>>>>>>  
>>>>>>  clean: check-clean
>>>>>>  
>>>>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>>>>> new file mode 100644
>>>>>> index 0000000000..d39f9d1576
>>>>>> --- /dev/null
>>>>>> +++ b/tests/venv-requirements.txt
>>>>>> @@ -0,0 +1,3 @@
>>>>>> +# Add Python module requirements, one per line, to be installed
>>>>>> +# in the tests/venv Python virtual environment. For more info,
>>>>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>>>
>>>>>
>>>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>>
>>>
>>>
> 

Re: [Qemu-devel] [PATCH v4 0/3] Bootstrap Python venv and acceptance/functional tests

[Philippe Mathieu-Daudé]

On 16/10/2018 16:27, Cleber Rosa wrote:
> On 10/12/18 5:44 PM, Philippe Mathieu-Daudé wrote:
>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>> TL;DR
>>> =====
>>>
>>> Allow acceptance tests to be run with `make check-acceptance`.
>>>
>>> Details
>>> =======
>>>
>>> This introduces a Python virtual environment that will be setup within
>>> the QEMU build directory, that will contain the exact environment that
>>> tests may require.
>>>
>>> There's one current caveat: it requires Python 3, as it's based on the
>>> venv module.  This was based on some discussions and perception about
>>> standardizing on Python 3, but can easily be made to accommodate Python
>>> 2 as well.
>>>
>>> Example of bootstrap and test execution on Travis-CI:
>>>
>>> https://travis-ci.org/qemu/qemu/jobs/439331028#L2508
>>
>> If you activate Travis on your github account, you can test that in your
>> namespace without having to open zombie pull requests there... A simple
>> push to your repository will trigger a full Travis build.
>>
>> This is how I use it btw, canceling the jobs I'm not interested in, to
>> quickly run the others.
>> i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600
>>
> 
> Yep, I've done that shortly after sending this. :)  Thanks for the tip,
> though.
> 
> One thing I miss is the ability, without editing the .travis.yaml, to
> "choose" just one/few jobs in a build.  Or maybe even the order they run
> (Of course I want the acceptance tests job first).
> 
> Do you know anything about that?

I just add an extra commit that drop every tests but the ones I'm
interested to run and keep this patch until I'm happy with the series.
I push as 'myseries_only' then I simply do "git branch myseries_only~
myseries && git push series" to run against everything, go to check my
mailbox and review patches until it finishes, then I redact the cover
and post.

> 
> - Cleber.
> 
>>>
>>>    ...
>>>       VENV    /home/travis/build/qemu/qemu/tests/venv
>>>       MKDIR   /home/travis/build/qemu/qemu/tests/results
>>>       PIP     /home/travis/build/qemu/qemu/tests/venv-requirements.txt
>>>       AVOCADO tests/acceptance
>>>     JOB ID     : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d
>>>     JOB LOG    : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log
>>>      (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test:  PASS (3.57 s)
>>>      (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version:  PASS (0.04 s)
>>>      (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>>      (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS (0.04 s)
>>>      (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  PASS (0.04 s)
>>>      (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS (0.04 s)
>>>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>>     JOB TIME   : 3.90 s
>>>    ...
>>>
>>> Changes from v3:
>>> ================
>>>
>>>  * Fixed typo in commit message (s/requiment/requirement/).  (Eric)
>>>
>>> Changes from v2:
>>> ================
>>>
>>>  * Make the $(TESTS_VENV_DIR) target depend on the
>>>    venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv
>>>    runs.  With this, updates on the file are reflected on the
>>>    venv. (Philippe)
>>>
>>>  * Run pip with "python -m pip".  It may have been installed reusing
>>>    the system wide packages, and then the script may not be available
>>>    on the venv. (Philippe)
>>>
>>>  * Dropped Python version on Travis, and using the version supplied
>>>    by the distro (3.4). (Philippe)
>>>
>>>  * Added "python3.4-venv" package requirement on Travis. (Philippe)
>>>
>>>  * Added variable (AVOCADO_SHOW) with logging streams to be shown
>>>    while running the acceptance tests.  By default it's set to none,
>>>    the equivalent of the quiet mode used on previous versions.
>>>    (Philippe)
>>>
>>>  * On Travis, set the AVOCADO_SHOW variable to "app", so that the
>>>    individual test results can be easily seen.  (Philippe)
>>>
>>> Ideas discussed, but not implemented:
>>>
>>>   * Run pip with "$(PYTHON) -m pip -q install ..." because it points
>>>     to the system wide Python installation. (Philippe)
>>>
>>>   * Drop the "--system-site-packages" flag.  Waiting on another round
>>>     of tests to determine if they are really the cause of some package
>>>     installation problems.
>>>
>>> Changes from v1:
>>> ================
>>>
>>>  * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
>>>    the source path ($SRC_PATH instead of $BUILD_DIR)
>>>
>>>  * Create the venv with "--system-site-packages", which allows the
>>>    reuse of packages (and no additional downloads) in case there's a
>>>    package installed system wide providing the same package and
>>>    version.
>>>
>>>  * Run Avocado with "python -m avocado".  It may have been installed
>>>    reusing the system wide packages, and then the script may not
>>>    be available on the venv.
>>>
>>>  * Improved documentation describing the Python 3, venv and pip
>>>    requirements.
>>>
>>>  * Updated avocado-framework requirement to latest released version
>>>    (65.0)
>>>
>>>  * (New commit) Added support for running the acceptance tests on
>>>    Travis.
>>>
>>> Ideas discussed, but not implemented:
>>>
>>>  * Install external packages such as python3-pip on Debian based
>>>    systems, deemed too invasive on developer's systems.
>>>
>>>  * Allow the use of Python 2, and consequently the "virtualenv"
>>>    module.
>>>
>>> Cleber Rosa (3):
>>>   Bootstrap Python venv for tests
>>>   Acceptance tests: add make rule for running them
>>>   Travis support for the acceptance tests
>>>
>>>  .travis.yml                 |  5 +++++
>>>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>>>  tests/Makefile.include      | 37 +++++++++++++++++++++++++++++++++++++
>>>  tests/venv-requirements.txt |  4 ++++
>>>  4 files changed, 76 insertions(+), 5 deletions(-)
>>>  create mode 100644 tests/venv-requirements.txt
>>>
>>
> 

Re: [Qemu-devel] [PATCH v4 3/3] Travis support for the acceptance tests

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> Hi Cleber,
>
> On 12/10/2018 18:53, Cleber Rosa wrote:
>> This enables the execution of the acceptance tests on Travis.
>>
>> Because the Travis environment is based on Ubuntu Trusty, it requires
>> the python3-pip and python3.4-venv packages.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  .travis.yml | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 95be6ec59f..f55f799c52 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -36,6 +36,8 @@ addons:
>>        - liburcu-dev
>>        - libusb-1.0-0-dev
>>        - libvte-2.90-dev
>> +      - python3-pip
>> +      - python3.4-venv
>
> I'd prefer not put Python specific version in the generic addons list...
>
>>        - sparse
>>        - uuid-dev
>>        - gcovr
>> @@ -117,6 +119,9 @@ matrix:
>>      - env: CONFIG="--target-list=x86_64-softmmu"
>>        python:
>>          - "3.6"
>> +    # Acceptance (Functional) tests
>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>> +           TEST_CMD="make AVOCADO_SHOW=app check-acceptance"
>
> ... but rather in the single test that requires it:
>
>          addons:
>            apt:
>              packages:
>                - python3-pip
>                - python3.4-venv
>
> Alex what do you prefer?

Out-of-band package managers can potentially add complexity to the build
environment so we should limit them to where they are needed.

We really need to transition to using docker for all out build
environments as the default Travis environment is getting steadily
crustier and out of date. We need to solve the caching/hub problem first
though.

With the move to the test:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>


>
>>      # Using newer GCC with sanitizers
>>      - addons:
>>          apt:
>>
>
> Both configs:
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>


--
Alex Bennée