[Qemu-devel] [PATCH v2 0/3] Bootstrap Python venv and acceptance/functional tests

[Qemu-devel] [PATCH v2 0/3] Bootstrap Python venv and acceptance/functional tests

[Cleber Rosa]

TL;DR
=====

Allow acceptance tests to be run with `make check-acceptance`.

Details
=======

This introduces a Python virtual environment that will be setup within
the QEMU build directory, that will contain the exact environment that
tests may require.

There's one current caveat: it requires Python 3, as it's based on the
venv module.  This was based on some discussions and perception about
standardizing on Python 3, but can easily be made to accommodate Python
2 as well.

Changes from v1:
================

 * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
   the source path ($SRC_PATH instead of $BUILD_DIR)

 * Create the venv with "--system-site-packages", which allows the
   reuse of packages (and no additional downloads) in case there's a
   package installed system wide providing the same package and
   version.

 * Run Avocado with "python -m avocado".  It may have been installed
   reusing the system wide packages, and then the script may not
   be available on the venv.

 * Improved documentation describing the Python 3, venv and pip
   requirements.

 * Updated avocado-framework requirement to latest released version
   (65.0)

 * (New commit) Added support for running the acceptance tests on
   Travis.

Ideas discussed, but not implemented:

 * Install external packages such as python3-pip on Debian based
   systems, deemed too invasive on developer's systems.

 * Allow the use of Python 2, and consequently the "virtualenv"
   module.

Cleber Rosa (3):
  Bootstrap Python venv for tests
  Acceptance tests: add make rule for running them
  Travis support for the acceptance tests

 .travis.yml                 |  6 ++++++
 docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
 tests/Makefile.include      | 32 ++++++++++++++++++++++++++++++++
 tests/venv-requirements.txt |  4 ++++
 4 files changed, 72 insertions(+), 5 deletions(-)
 create mode 100644 tests/venv-requirements.txt

-- 
2.17.1

[Qemu-devel] [PATCH v2 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

A number of QEMU tests are written in Python, and may benefit
from an untainted Python venv.

By using make rules, tests that depend on specific Python libs
can set that rule as a requiment, along with rules that require
the presence or installation of specific libraries.

The tests/venv-requirements.txt is supposed to contain the
Python requirements that should be added to the venv created
by check-venv.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include      | 19 +++++++++++++++++++
 tests/venv-requirements.txt |  3 +++
 2 files changed, 22 insertions(+)
 create mode 100644 tests/venv-requirements.txt

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 7a3059bf6c..68af79927d 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -12,6 +12,7 @@ check-help:
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
+	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
 	@echo
 	@echo "Please note that HTML reports do not regenerate if the unit tests"
@@ -1015,6 +1016,23 @@ check-decodetree:
           ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
           TEST, decodetree.py)
 
+# Python venv for running tests
+
+.PHONY: check-venv
+
+TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
+TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
+
+$(TESTS_VENV_DIR):
+	$(call quiet-command, \
+            $(PYTHON) -m venv --system-site-packages $@, \
+            VENV, $@)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
+            PIP, $(TESTS_VENV_REQ))
+
+check-venv: $(TESTS_VENV_DIR)
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -1028,6 +1046,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
+	rm -rf $(TESTS_VENV_DIR)
 
 clean: check-clean
 
diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
new file mode 100644
index 0000000000..d39f9d1576
--- /dev/null
+++ b/tests/venv-requirements.txt
@@ -0,0 +1,3 @@
+# Add Python module requirements, one per line, to be installed
+# in the tests/venv Python virtual environment. For more info,
+# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-- 
2.17.1

[Qemu-devel] [PATCH v2 2/3] Acceptance tests: add make rule for running them

[Cleber Rosa]

The acceptance (aka functional, aka Avocado-based) tests are
Python files located in "tests/acceptance" that need to be run
with the Avocado libs and test runner.

Let's provide a convenient way for QEMU developers to run them,
by making use of the tests-venv with the required setup.

Also, while the Avocado test runner will take care of creating a
location to save test results to, it was understood that it's better
if the results are kept within the build tree.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
 tests/Makefile.include      | 17 +++++++++++++++--
 tests/venv-requirements.txt |  1 +
 3 files changed, 46 insertions(+), 7 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 727c4019b5..b992a2961d 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -545,10 +545,31 @@ Tests based on ``avocado_qemu.Test`` can easily:
    - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
    - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
 
-Installation
-------------
+Running tests
+-------------
 
-To install Avocado and its dependencies, run:
+You can run the acceptance tests simply by executing:
+
+.. code::
+
+  make check-acceptance
+
+This involves the automatic creation of Python virtual environment
+within the build tree (at ``tests/venv``) which will have all the
+right dependencies, and will save tests results also within the
+build tree (at ``tests/results``).
+
+Note: the build environment must be using a Python 3 stack, and have
+the ``venv`` and ``pip`` packages installed.  If necessary, make sure
+``configure`` is called with ``--python=`` and that those modules are
+available.  On Debian and Ubuntu based systems, depending on the
+specific version, they may be on packages named ``python3-venv`` and
+``python3-pip``.
+
+Manual Installation
+-------------------
+
+To manually install Avocado and its dependencies, run:
 
 .. code::
 
@@ -689,11 +710,15 @@ The exact QEMU binary to be used on QEMUMachine.
 Uninstalling Avocado
 --------------------
 
-If you've followed the installation instructions above, you can easily
-uninstall Avocado.  Start by listing the packages you have installed::
+If you've followed the manual installation instructions above, you can
+easily uninstall Avocado.  Start by listing the packages you have
+installed::
 
   pip list --user
 
 And remove any package you want with::
 
   pip uninstall <package_name>
+
+If you've used ``make check-acceptance``, the Python virtual environment where
+Avocado is installed will be cleaned up as part of ``make check-clean``.
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 68af79927d..00fdf9913e 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -11,6 +11,7 @@ check-help:
 	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
+	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
 	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
@@ -1018,10 +1019,11 @@ check-decodetree:
 
 # Python venv for running tests
 
-.PHONY: check-venv
+.PHONY: check-venv check-acceptance
 
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
+TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 
 $(TESTS_VENV_DIR):
 	$(call quiet-command, \
@@ -1031,8 +1033,19 @@ $(TESTS_VENV_DIR):
             $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
             PIP, $(TESTS_VENV_REQ))
 
+$(TESTS_RESULTS_DIR):
+	$(call quiet-command, mkdir -p $@, \
+            MKDIR, $@)
+
 check-venv: $(TESTS_VENV_DIR)
 
+check-acceptance: check-venv $(TESTS_RESULTS_DIR)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=none run --job-results-dir=$(TESTS_RESULTS_DIR) --failfast=on \
+            $(SRC_PATH)/tests/acceptance, \
+            "AVOCADO", "tests/acceptance")
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -1046,7 +1059,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
-	rm -rf $(TESTS_VENV_DIR)
+	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
 
 clean: check-clean
 
diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
index d39f9d1576..64c6e27a94 100644
--- a/tests/venv-requirements.txt
+++ b/tests/venv-requirements.txt
@@ -1,3 +1,4 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
+avocado-framework==65.0
-- 
2.17.1

[Qemu-devel] [PATCH v2 3/3] Travis support for the acceptance tests

[Cleber Rosa]

This enables the execution of the acceptance tests on Travis.

Because the Travis environment is based on Ubuntu Trusty, it requires
the python3-pip.

Note: while another supposedely required component on newer versions
(such as on Bionic) split the Python 3 installation further on the
python3-venv package.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .travis.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 95be6ec59f..db1a31ea51 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -36,6 +36,7 @@ addons:
       - liburcu-dev
       - libusb-1.0-0-dev
       - libvte-2.90-dev
+      - python3-pip
       - sparse
       - uuid-dev
       - gcovr
@@ -117,6 +118,11 @@ matrix:
     - env: CONFIG="--target-list=x86_64-softmmu"
       python:
         - "3.6"
+    # Acceptance (Functional) tests
+    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
+           TEST_CMD="make check-acceptance"
+      python:
+        - "3.6"
     # Using newer GCC with sanitizers
     - addons:
         apt:
-- 
2.17.1

Re: [Qemu-devel] [PATCH v2 1/3] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

Hi Cleber,

On 09/10/2018 06:18, Cleber Rosa wrote:
> A number of QEMU tests are written in Python, and may benefit
> from an untainted Python venv.
> 
> By using make rules, tests that depend on specific Python libs
> can set that rule as a requiment, along with rules that require
> the presence or installation of specific libraries.
> 
> The tests/venv-requirements.txt is supposed to contain the
> Python requirements that should be added to the venv created
> by check-venv.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/Makefile.include      | 19 +++++++++++++++++++
>  tests/venv-requirements.txt |  3 +++
>  2 files changed, 22 insertions(+)
>  create mode 100644 tests/venv-requirements.txt
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 7a3059bf6c..68af79927d 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -12,6 +12,7 @@ check-help:
>  	@echo " $(MAKE) check-block          Run block tests"
>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>  	@echo " $(MAKE) check-clean          Clean the tests"
>  	@echo
>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
> @@ -1015,6 +1016,23 @@ check-decodetree:
>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>            TEST, decodetree.py)
>  
> +# Python venv for running tests
> +
> +.PHONY: check-venv
> +
> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
> +
> +$(TESTS_VENV_DIR):
> +	$(call quiet-command, \
> +            $(PYTHON) -m venv --system-site-packages $@, \

Why do we want the '--system-site-packages' flag?

> +            VENV, $@)
> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \

On Ubuntu I have to use (after installing python3-pip + python3.4-venv):

               $(PYTHON) -m pip -q install -r $(TESTS_VENV_REQ), \

> +            PIP, $(TESTS_VENV_REQ))
> +
> +check-venv: $(TESTS_VENV_DIR)

Same problem from v1: if check-venv failed and the directory exists, the
rule succeeds.

> +
>  # Consolidated targets
>  
>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -1028,6 +1046,7 @@ check-clean:
>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>  	rm -f tests/test-qapi-gen-timestamp
> +	rm -rf $(TESTS_VENV_DIR)
>  
>  clean: check-clean
>  
> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
> new file mode 100644
> index 0000000000..d39f9d1576
> --- /dev/null
> +++ b/tests/venv-requirements.txt
> @@ -0,0 +1,3 @@
> +# Add Python module requirements, one per line, to be installed
> +# in the tests/venv Python virtual environment. For more info,
> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> 

Re: [Qemu-devel] [PATCH v2 3/3] Travis support for the acceptance tests

[Philippe Mathieu-Daudé]

Hi Cleber,

On 09/10/2018 06:18, Cleber Rosa wrote:
> This enables the execution of the acceptance tests on Travis.

Did you test this? =)

> 
> Because the Travis environment is based on Ubuntu Trusty, it requires
> the python3-pip.
> 
> Note: while another supposedely required component on newer versions
> (such as on Bionic) split the Python 3 installation further on the
> python3-venv package.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .travis.yml | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 95be6ec59f..db1a31ea51 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -36,6 +36,7 @@ addons:
>        - liburcu-dev
>        - libusb-1.0-0-dev
>        - libvte-2.90-dev
> +      - python3-pip
>        - sparse
>        - uuid-dev
>        - gcovr
> @@ -117,6 +118,11 @@ matrix:
>      - env: CONFIG="--target-list=x86_64-softmmu"
>        python:
>          - "3.6"
> +    # Acceptance (Functional) tests
> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
> +           TEST_CMD="make check-acceptance"
> +      python:
> +        - "3.6"
>      # Using newer GCC with sanitizers
>      - addons:
>          apt:
> 

Using the following patch:

-- >8 --
diff --git a/.travis.yml b/.travis.yml
index 95be6ec59f..87e0c9a13f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -118,4 +118,15 @@ matrix:
       python:
         - "3.6"
+    # Acceptance (Functional) tests
+    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
+           TEST_CMD="make check-acceptance"
+           # sudo rm /usr/local/bin/pip*
+      python:
+        - "3.6"
+      addons:
+        apt:
+          packages:
+            - python3-pip
+            - python3.4-venv
     # Using newer GCC with sanitizers
     - addons:
---

I got some improvements until:

  VENV    /home/travis/build/philmd/qemu/tests/venv
  MKDIR   /home/travis/build/philmd/qemu/tests/results
  PIP     /home/travis/build/philmd/qemu/tests/venv-requirements.txt
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in
main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line
283, in run
    requirement_set.install(install_options, global_options,
root=options.root_path)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
    self.move_wheel_files(self.source_dir, root=root)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in
move_wheel_files
    pycompile=self.pycompile,
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in
move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber
    os.makedirs(destsubdir)
  File "/usr/lib/python3.4/os.py", line 237, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied:
'/usr/local/lib/python3.4/dist-packages/avocado'

See: https://travis-ci.org/philmd/qemu/jobs/439138706

Re: [Qemu-devel] [PATCH v2 3/3] Travis support for the acceptance tests

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> Hi Cleber,
>
> On 09/10/2018 06:18, Cleber Rosa wrote:
>> This enables the execution of the acceptance tests on Travis.
>
> Did you test this? =)
>
>>
>> Because the Travis environment is based on Ubuntu Trusty, it requires
>> the python3-pip.
>>
>> Note: while another supposedely required component on newer versions
>> (such as on Bionic) split the Python 3 installation further on the
>> python3-venv package.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  .travis.yml | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 95be6ec59f..db1a31ea51 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -36,6 +36,7 @@ addons:
>>        - liburcu-dev
>>        - libusb-1.0-0-dev
>>        - libvte-2.90-dev
>> +      - python3-pip
>>        - sparse
>>        - uuid-dev
>>        - gcovr
>> @@ -117,6 +118,11 @@ matrix:
>>      - env: CONFIG="--target-list=x86_64-softmmu"
>>        python:
>>          - "3.6"
>> +    # Acceptance (Functional) tests
>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>> +           TEST_CMD="make check-acceptance"
>> +      python:
>> +        - "3.6"
>>      # Using newer GCC with sanitizers
>>      - addons:
>>          apt:
>>
>
> Using the following patch:
>
> -- >8 --
> diff --git a/.travis.yml b/.travis.yml
> index 95be6ec59f..87e0c9a13f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -118,4 +118,15 @@ matrix:
>        python:
>          - "3.6"
> +    # Acceptance (Functional) tests
> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
> +           TEST_CMD="make check-acceptance"
> +           # sudo rm /usr/local/bin/pip*

That snuck in ^

> +      python:
> +        - "3.6"
> +      addons:
> +        apt:
> +          packages:
> +            - python3-pip
> +            - python3.4-venv
>      # Using newer GCC with sanitizers
>      - addons:
> ---
>
> I got some improvements until:
>
>   VENV    /home/travis/build/philmd/qemu/tests/venv
>   MKDIR   /home/travis/build/philmd/qemu/tests/results
>   PIP     /home/travis/build/philmd/qemu/tests/venv-requirements.txt
> Exception:
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in
> main
>     status = self.run(options, args)
>   File "/usr/lib/python3/dist-packages/pip/commands/install.py", line
> 283, in run
>     requirement_set.install(install_options, global_options,
> root=options.root_path)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
>     requirement.install(install_options, global_options, *args, **kwargs)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
>     self.move_wheel_files(self.source_dir, root=root)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in
> move_wheel_files
>     pycompile=self.pycompile,
>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in
> move_wheel_files
>     clobber(source, lib_dir, True)
>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber
>     os.makedirs(destsubdir)
>   File "/usr/lib/python3.4/os.py", line 237, in makedirs
>     mkdir(name, mode)
> PermissionError: [Errno 13] Permission denied:
> '/usr/local/lib/python3.4/dist-packages/avocado'
>
> See: https://travis-ci.org/philmd/qemu/jobs/439138706

Don't we need to do a pip install or is avocado already included?

--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 3/3] Travis support for the acceptance tests

[Philippe Mathieu-Daudé]

On 09/10/2018 16:15, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> 
>> Hi Cleber,
>>
>> On 09/10/2018 06:18, Cleber Rosa wrote:
>>> This enables the execution of the acceptance tests on Travis.
>>
>> Did you test this? =)
>>
>>>
>>> Because the Travis environment is based on Ubuntu Trusty, it requires
>>> the python3-pip.
>>>
>>> Note: while another supposedely required component on newer versions
>>> (such as on Bionic) split the Python 3 installation further on the
>>> python3-venv package.
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  .travis.yml | 6 ++++++
>>>  1 file changed, 6 insertions(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 95be6ec59f..db1a31ea51 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -36,6 +36,7 @@ addons:
>>>        - liburcu-dev
>>>        - libusb-1.0-0-dev
>>>        - libvte-2.90-dev
>>> +      - python3-pip
>>>        - sparse
>>>        - uuid-dev
>>>        - gcovr
>>> @@ -117,6 +118,11 @@ matrix:
>>>      - env: CONFIG="--target-list=x86_64-softmmu"
>>>        python:
>>>          - "3.6"
>>> +    # Acceptance (Functional) tests
>>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>>> +           TEST_CMD="make check-acceptance"
>>> +      python:
>>> +        - "3.6"
>>>      # Using newer GCC with sanitizers
>>>      - addons:
>>>          apt:
>>>
>>
>> Using the following patch:
>>
>> -- >8 --
>> diff --git a/.travis.yml b/.travis.yml
>> index 95be6ec59f..87e0c9a13f 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -118,4 +118,15 @@ matrix:
>>        python:
>>          - "3.6"
>> +    # Acceptance (Functional) tests
>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>> +           TEST_CMD="make check-acceptance"
>> +           # sudo rm /usr/local/bin/pip*
> 
> That snuck in ^

=)

Travis-ci image comes with this version which is Python2.

>> +      python:
>> +        - "3.6"
>> +      addons:
>> +        apt:
>> +          packages:
>> +            - python3-pip
>> +            - python3.4-venv
>>      # Using newer GCC with sanitizers
>>      - addons:
>> ---
>>
>> I got some improvements until:
>>
>>   VENV    /home/travis/build/philmd/qemu/tests/venv
>>   MKDIR   /home/travis/build/philmd/qemu/tests/results
>>   PIP     /home/travis/build/philmd/qemu/tests/venv-requirements.txt
>> Exception:
>> Traceback (most recent call last):
>>   File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in
>> main
>>     status = self.run(options, args)
>>   File "/usr/lib/python3/dist-packages/pip/commands/install.py", line
>> 283, in run
>>     requirement_set.install(install_options, global_options,
>> root=options.root_path)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
>>     requirement.install(install_options, global_options, *args, **kwargs)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
>>     self.move_wheel_files(self.source_dir, root=root)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in
>> move_wheel_files
>>     pycompile=self.pycompile,
>>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in
>> move_wheel_files
>>     clobber(source, lib_dir, True)
>>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber
>>     os.makedirs(destsubdir)
>>   File "/usr/lib/python3.4/os.py", line 237, in makedirs
>>     mkdir(name, mode)
>> PermissionError: [Errno 13] Permission denied:
>> '/usr/local/lib/python3.4/dist-packages/avocado'
>>
>> See: https://travis-ci.org/philmd/qemu/jobs/439138706
> 
> Don't we need to do a pip install or is avocado already included?

Yes I think we do. This is why I asked Cleber on patch #1 of this series
why he choose to use "venv ... --system-site-packages".

Re: [Qemu-devel] [PATCH v2 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/9/18 9:25 AM, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> On 09/10/2018 06:18, Cleber Rosa wrote:
>> A number of QEMU tests are written in Python, and may benefit
>> from an untainted Python venv.
>>
>> By using make rules, tests that depend on specific Python libs
>> can set that rule as a requiment, along with rules that require
>> the presence or installation of specific libraries.
>>
>> The tests/venv-requirements.txt is supposed to contain the
>> Python requirements that should be added to the venv created
>> by check-venv.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  tests/Makefile.include      | 19 +++++++++++++++++++
>>  tests/venv-requirements.txt |  3 +++
>>  2 files changed, 22 insertions(+)
>>  create mode 100644 tests/venv-requirements.txt
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 7a3059bf6c..68af79927d 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -12,6 +12,7 @@ check-help:
>>  	@echo " $(MAKE) check-block          Run block tests"
>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>  	@echo
>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>> @@ -1015,6 +1016,23 @@ check-decodetree:
>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>            TEST, decodetree.py)
>>  
>> +# Python venv for running tests
>> +
>> +.PHONY: check-venv
>> +
>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>> +
>> +$(TESTS_VENV_DIR):
>> +	$(call quiet-command, \
>> +            $(PYTHON) -m venv --system-site-packages $@, \
> 
> Why do we want the '--system-site-packages' flag?
> 

It allows the reuse of packages (with no additional downloads) in case
there's a package installed system wide providing the same package and
version.  If you have, say, an RPM or DEB (or even a pip) package
installed system wide, it'll be copied over to the venv.  This was based
on a suggestion by Eduardo.

>> +            VENV, $@)
>> +	$(call quiet-command, \
>> +            $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
> 
> On Ubuntu I have to use (after installing python3-pip + python3.4-venv):
> 
>                $(PYTHON) -m pip -q install -r $(TESTS_VENV_REQ), \
> 

I don't think running $(PYTHON) is the right thing to do here because it
points to non-venv Python interpreter.  IMO it should be:

                $(TESTS_VENV_DIR)/bin/python -m pip ...

>> +            PIP, $(TESTS_VENV_REQ))
>> +
>> +check-venv: $(TESTS_VENV_DIR)
> 
> Same problem from v1: if check-venv failed and the directory exists, the
> rule succeeds.
> 

I really don't know how to make all the steps of this target "atomic",
unless I do an ugly "venv ... && touch ... && pip ...".  The best I
managed to think was to flag the requirement as fulfilled by touching
the venv directory after pip finishes.

- Cleber.

>> +
>>  # Consolidated targets
>>  
>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>> @@ -1028,6 +1046,7 @@ check-clean:
>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>  	rm -f tests/test-qapi-gen-timestamp
>> +	rm -rf $(TESTS_VENV_DIR)
>>  
>>  clean: check-clean
>>  
>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>> new file mode 100644
>> index 0000000000..d39f9d1576
>> --- /dev/null
>> +++ b/tests/venv-requirements.txt
>> @@ -0,0 +1,3 @@
>> +# Add Python module requirements, one per line, to be installed
>> +# in the tests/venv Python virtual environment. For more info,
>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>

Re: [Qemu-devel] [PATCH v2 1/3] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

On 09/10/2018 18:00, Cleber Rosa wrote:
> 
> 
> On 10/9/18 9:25 AM, Philippe Mathieu-Daudé wrote:
>> Hi Cleber,
>>
>> On 09/10/2018 06:18, Cleber Rosa wrote:
>>> A number of QEMU tests are written in Python, and may benefit
>>> from an untainted Python venv.
>>>
>>> By using make rules, tests that depend on specific Python libs
>>> can set that rule as a requiment, along with rules that require
>>> the presence or installation of specific libraries.
>>>
>>> The tests/venv-requirements.txt is supposed to contain the
>>> Python requirements that should be added to the venv created
>>> by check-venv.
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  tests/Makefile.include      | 19 +++++++++++++++++++
>>>  tests/venv-requirements.txt |  3 +++
>>>  2 files changed, 22 insertions(+)
>>>  create mode 100644 tests/venv-requirements.txt
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 7a3059bf6c..68af79927d 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -12,6 +12,7 @@ check-help:
>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>  	@echo
>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>> @@ -1015,6 +1016,23 @@ check-decodetree:
>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>            TEST, decodetree.py)
>>>  
>>> +# Python venv for running tests
>>> +
>>> +.PHONY: check-venv
>>> +
>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>> +
>>> +$(TESTS_VENV_DIR):
>>> +	$(call quiet-command, \
>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>
>> Why do we want the '--system-site-packages' flag?
>>
> 
> It allows the reuse of packages (with no additional downloads) in case
> there's a package installed system wide providing the same package and
> version.  If you have, say, an RPM or DEB (or even a pip) package
> installed system wide, it'll be copied over to the venv.  This was based
> on a suggestion by Eduardo.

Some system-site-packages are broken (or too old?).

> 
>>> +            VENV, $@)
>>> +	$(call quiet-command, \
>>> +            $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
>>
>> On Ubuntu I have to use (after installing python3-pip + python3.4-venv):
>>
>>                $(PYTHON) -m pip -q install -r $(TESTS_VENV_REQ), \
>>
> 
> I don't think running $(PYTHON) is the right thing to do here because it
> points to non-venv Python interpreter.  IMO it should be:
> 
>                 $(TESTS_VENV_DIR)/bin/python -m pip ...

Yes it worked: https://travis-ci.org/philmd/qemu/jobs/439216885

> 
>>> +            PIP, $(TESTS_VENV_REQ))
>>> +
>>> +check-venv: $(TESTS_VENV_DIR)
>>
>> Same problem from v1: if check-venv failed and the directory exists, the
>> rule succeeds.
>>
> 
> I really don't know how to make all the steps of this target "atomic",
> unless I do an ugly "venv ... && touch ... && pip ...".  The best I
> managed to think was to flag the requirement as fulfilled by touching
> the venv directory after pip finishes.
> 
> - Cleber.
> 
>>> +
>>>  # Consolidated targets
>>>  
>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>> @@ -1028,6 +1046,7 @@ check-clean:
>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>  	rm -f tests/test-qapi-gen-timestamp
>>> +	rm -rf $(TESTS_VENV_DIR)
>>>  
>>>  clean: check-clean
>>>  
>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>> new file mode 100644
>>> index 0000000000..d39f9d1576
>>> --- /dev/null
>>> +++ b/tests/venv-requirements.txt
>>> @@ -0,0 +1,3 @@
>>> +# Add Python module requirements, one per line, to be installed
>>> +# in the tests/venv Python virtual environment. For more info,
>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>

Re: [Qemu-devel] [PATCH v2 2/3] Acceptance tests: add make rule for running them

[Philippe Mathieu-Daudé]

On 09/10/2018 06:18, Cleber Rosa wrote:
> The acceptance (aka functional, aka Avocado-based) tests are
> Python files located in "tests/acceptance" that need to be run
> with the Avocado libs and test runner.
> 
> Let's provide a convenient way for QEMU developers to run them,
> by making use of the tests-venv with the required setup.
> 
> Also, while the Avocado test runner will take care of creating a
> location to save test results to, it was understood that it's better
> if the results are kept within the build tree.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>  tests/Makefile.include      | 17 +++++++++++++++--
>  tests/venv-requirements.txt |  1 +
>  3 files changed, 46 insertions(+), 7 deletions(-)
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 727c4019b5..b992a2961d 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -545,10 +545,31 @@ Tests based on ``avocado_qemu.Test`` can easily:
>     - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
>     - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
>  
> -Installation
> -------------
> +Running tests
> +-------------
>  
> -To install Avocado and its dependencies, run:
> +You can run the acceptance tests simply by executing:
> +
> +.. code::
> +
> +  make check-acceptance
> +
> +This involves the automatic creation of Python virtual environment
> +within the build tree (at ``tests/venv``) which will have all the
> +right dependencies, and will save tests results also within the
> +build tree (at ``tests/results``).
> +
> +Note: the build environment must be using a Python 3 stack, and have
> +the ``venv`` and ``pip`` packages installed.  If necessary, make sure
> +``configure`` is called with ``--python=`` and that those modules are
> +available.  On Debian and Ubuntu based systems, depending on the
> +specific version, they may be on packages named ``python3-venv`` and
> +``python3-pip``.
> +
> +Manual Installation
> +-------------------
> +
> +To manually install Avocado and its dependencies, run:
>  
>  .. code::
>  
> @@ -689,11 +710,15 @@ The exact QEMU binary to be used on QEMUMachine.
>  Uninstalling Avocado
>  --------------------
>  
> -If you've followed the installation instructions above, you can easily
> -uninstall Avocado.  Start by listing the packages you have installed::
> +If you've followed the manual installation instructions above, you can
> +easily uninstall Avocado.  Start by listing the packages you have
> +installed::
>  
>    pip list --user
>  
>  And remove any package you want with::
>  
>    pip uninstall <package_name>
> +
> +If you've used ``make check-acceptance``, the Python virtual environment where
> +Avocado is installed will be cleaned up as part of ``make check-clean``.
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 68af79927d..00fdf9913e 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -11,6 +11,7 @@ check-help:
>  	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
>  	@echo " $(MAKE) check-block          Run block tests"
>  	@echo " $(MAKE) check-tcg            Run TCG tests"
> +	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>  	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>  	@echo " $(MAKE) check-clean          Clean the tests"
> @@ -1018,10 +1019,11 @@ check-decodetree:
>  
>  # Python venv for running tests
>  
> -.PHONY: check-venv
> +.PHONY: check-venv check-acceptance
>  
>  TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>  TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  
>  $(TESTS_VENV_DIR):
>  	$(call quiet-command, \
> @@ -1031,8 +1033,19 @@ $(TESTS_VENV_DIR):
>              $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
>              PIP, $(TESTS_VENV_REQ))
>  
> +$(TESTS_RESULTS_DIR):
> +	$(call quiet-command, mkdir -p $@, \
> +            MKDIR, $@)
> +
>  check-venv: $(TESTS_VENV_DIR)
>  
> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m avocado \
> +            --show=none run --job-results-dir=$(TESTS_RESULTS_DIR) --failfast=on \

Can we add an overwritable env var such AVOCADO_SHOW=none ?
I'd use it for Travis, to see which test failed on the console output.

> +            $(SRC_PATH)/tests/acceptance, \
> +            "AVOCADO", "tests/acceptance")
> +
>  # Consolidated targets
>  
>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -1046,7 +1059,7 @@ check-clean:
>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>  	rm -f tests/test-qapi-gen-timestamp
> -	rm -rf $(TESTS_VENV_DIR)
> +	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
>  
>  clean: check-clean
>  
> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
> index d39f9d1576..64c6e27a94 100644
> --- a/tests/venv-requirements.txt
> +++ b/tests/venv-requirements.txt
> @@ -1,3 +1,4 @@
>  # Add Python module requirements, one per line, to be installed
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> +avocado-framework==65.0
> 

Re: [Qemu-devel] [PATCH v2 3/3] Travis support for the acceptance tests

[Cleber Rosa]

On 10/9/18 9:46 AM, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> On 09/10/2018 06:18, Cleber Rosa wrote:
>> This enables the execution of the acceptance tests on Travis.
> 
> Did you test this? =)
> 

I did have some jobs on Travis that looked promising.  But yeah, later I
found some issues. :/

>>
>> Because the Travis environment is based on Ubuntu Trusty, it requires
>> the python3-pip.
>>
>> Note: while another supposedely required component on newer versions
>> (such as on Bionic) split the Python 3 installation further on the
>> python3-venv package.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  .travis.yml | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 95be6ec59f..db1a31ea51 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -36,6 +36,7 @@ addons:
>>        - liburcu-dev
>>        - libusb-1.0-0-dev
>>        - libvte-2.90-dev
>> +      - python3-pip
>>        - sparse
>>        - uuid-dev
>>        - gcovr
>> @@ -117,6 +118,11 @@ matrix:
>>      - env: CONFIG="--target-list=x86_64-softmmu"
>>        python:
>>          - "3.6"
>> +    # Acceptance (Functional) tests
>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>> +           TEST_CMD="make check-acceptance"
>> +      python:
>> +        - "3.6"
>>      # Using newer GCC with sanitizers
>>      - addons:
>>          apt:
>>
> 
> Using the following patch:
> 
> -- >8 --
> diff --git a/.travis.yml b/.travis.yml
> index 95be6ec59f..87e0c9a13f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -118,4 +118,15 @@ matrix:
>        python:
>          - "3.6"
> +    # Acceptance (Functional) tests
> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
> +           TEST_CMD="make check-acceptance"
> +           # sudo rm /usr/local/bin/pip*
> +      python:
> +        - "3.6"
> +      addons:
> +        apt:
> +          packages:
> +            - python3-pip
> +            - python3.4-venv

This patch revealed to me that, even though we're asking Travis for a
Python 3.6 stack, we're manually pointing to the primary Python (3.4)
installation.  This is in itself a source of problems.

Read on...

>      # Using newer GCC with sanitizers
>      - addons:
> ---
> 
> I got some improvements until:
> 
>   VENV    /home/travis/build/philmd/qemu/tests/venv
>   MKDIR   /home/travis/build/philmd/qemu/tests/results
>   PIP     /home/travis/build/philmd/qemu/tests/venv-requirements.txt
> Exception:
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in
> main
>     status = self.run(options, args)
>   File "/usr/lib/python3/dist-packages/pip/commands/install.py", line
> 283, in run
>     requirement_set.install(install_options, global_options,
> root=options.root_path)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
>     requirement.install(install_options, global_options, *args, **kwargs)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
>     self.move_wheel_files(self.source_dir, root=root)
>   File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in
> move_wheel_files
>     pycompile=self.pycompile,
>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in
> move_wheel_files
>     clobber(source, lib_dir, True)
>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber
>     os.makedirs(destsubdir)
>   File "/usr/lib/python3.4/os.py", line 237, in makedirs
>     mkdir(name, mode)
> PermissionError: [Errno 13] Permission denied:
> '/usr/local/lib/python3.4/dist-packages/avocado'
> 

This looks to me like it's caused by the use of the system wide Python
3.4 installation, as mentioned above.  That, and possibly the
"--system-wide-packages" flag to the venv.

So, in order to understand if, in those environments, we should let
"configure" pick up the right Python, I did the following experiment:

 $ python3 -m venv /tmp/py3-build
 $ . /tmp/py3-build/bin/activate
 $ which python
 /tmp/py3-build/bin/python
 $ /tmp/py3-build/bin/python --version
 Python 3.6.6

This shows that when a venv is activated, the "right" Python should be
available in the $PATH.  So, building QEMU without pointint to a
specific Python binary (when inside a venv) gives you:

 $ mkdir -p /tmp/qemu-build
 $ cd /tmp/qemu-build
 $ ~/src/qemu/configure --target-list="x86_64-softmmu"
 $ make print-PYTHON
 PYTHON=python -B
 $ which python
 /tmp/py3-build/bin/python

The whole question now is how/where Travis puts the requested Python
version (python: - "3.6").  Is it on a venv?  Is it on a specific
location? Is that reliable enough?

I'm going to look for those answers now.

- Cleber.

> See: https://travis-ci.org/philmd/qemu/jobs/439138706
> 

Re: [Qemu-devel] [PATCH v2 1/3] Bootstrap Python venv for tests

[Cleber Rosa]

On 10/9/18 12:13 PM, Philippe Mathieu-Daudé wrote:
> On 09/10/2018 18:00, Cleber Rosa wrote:
>>
>>
>> On 10/9/18 9:25 AM, Philippe Mathieu-Daudé wrote:
>>> Hi Cleber,
>>>
>>> On 09/10/2018 06:18, Cleber Rosa wrote:
>>>> A number of QEMU tests are written in Python, and may benefit
>>>> from an untainted Python venv.
>>>>
>>>> By using make rules, tests that depend on specific Python libs
>>>> can set that rule as a requiment, along with rules that require
>>>> the presence or installation of specific libraries.
>>>>
>>>> The tests/venv-requirements.txt is supposed to contain the
>>>> Python requirements that should be added to the venv created
>>>> by check-venv.
>>>>
>>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>  tests/Makefile.include      | 19 +++++++++++++++++++
>>>>  tests/venv-requirements.txt |  3 +++
>>>>  2 files changed, 22 insertions(+)
>>>>  create mode 100644 tests/venv-requirements.txt
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index 7a3059bf6c..68af79927d 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -12,6 +12,7 @@ check-help:
>>>>  	@echo " $(MAKE) check-block          Run block tests"
>>>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>>>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>>> +	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>>>  	@echo " $(MAKE) check-clean          Clean the tests"
>>>>  	@echo
>>>>  	@echo "Please note that HTML reports do not regenerate if the unit tests"
>>>> @@ -1015,6 +1016,23 @@ check-decodetree:
>>>>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>>>            TEST, decodetree.py)
>>>>  
>>>> +# Python venv for running tests
>>>> +
>>>> +.PHONY: check-venv
>>>> +
>>>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>>>> +
>>>> +$(TESTS_VENV_DIR):
>>>> +	$(call quiet-command, \
>>>> +            $(PYTHON) -m venv --system-site-packages $@, \
>>>
>>> Why do we want the '--system-site-packages' flag?
>>>
>>
>> It allows the reuse of packages (with no additional downloads) in case
>> there's a package installed system wide providing the same package and
>> version.  If you have, say, an RPM or DEB (or even a pip) package
>> installed system wide, it'll be copied over to the venv.  This was based
>> on a suggestion by Eduardo.
> 
> Some system-site-packages are broken (or too old?).
> 

It could be, yes.  I'm starting to believe that this may bring more
problems that it'd solve.

>>
>>>> +            VENV, $@)
>>>> +	$(call quiet-command, \
>>>> +            $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
>>>
>>> On Ubuntu I have to use (after installing python3-pip + python3.4-venv):
>>>
>>>                $(PYTHON) -m pip -q install -r $(TESTS_VENV_REQ), \
>>>
>>
>> I don't think running $(PYTHON) is the right thing to do here because it
>> points to non-venv Python interpreter.  IMO it should be:
>>
>>                 $(TESTS_VENV_DIR)/bin/python -m pip ...
> 
> Yes it worked: https://travis-ci.org/philmd/qemu/jobs/439216885
> 

Nice! I'll take a picture of that and hang on my wall. :)

Now, there's still one point: which Python version was used there? I'm
guessing it was Python 3.4, and not the Travis "provided" 3.6, right?

- Cleber.

>>
>>>> +            PIP, $(TESTS_VENV_REQ))
>>>> +
>>>> +check-venv: $(TESTS_VENV_DIR)
>>>
>>> Same problem from v1: if check-venv failed and the directory exists, the
>>> rule succeeds.
>>>
>>
>> I really don't know how to make all the steps of this target "atomic",
>> unless I do an ugly "venv ... && touch ... && pip ...".  The best I
>> managed to think was to flag the requirement as fulfilled by touching
>> the venv directory after pip finishes.
>>
>> - Cleber.
>>
>>>> +
>>>>  # Consolidated targets
>>>>  
>>>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>>>> @@ -1028,6 +1046,7 @@ check-clean:
>>>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>>>  	rm -f tests/test-qapi-gen-timestamp
>>>> +	rm -rf $(TESTS_VENV_DIR)
>>>>  
>>>>  clean: check-clean
>>>>  
>>>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>>>> new file mode 100644
>>>> index 0000000000..d39f9d1576
>>>> --- /dev/null
>>>> +++ b/tests/venv-requirements.txt
>>>> @@ -0,0 +1,3 @@
>>>> +# Add Python module requirements, one per line, to be installed
>>>> +# in the tests/venv Python virtual environment. For more info,
>>>> +# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>

Re: [Qemu-devel] [PATCH v2 2/3] Acceptance tests: add make rule for running them

[Cleber Rosa]

On 10/9/18 12:18 PM, Philippe Mathieu-Daudé wrote:
> On 09/10/2018 06:18, Cleber Rosa wrote:
>> The acceptance (aka functional, aka Avocado-based) tests are
>> Python files located in "tests/acceptance" that need to be run
>> with the Avocado libs and test runner.
>>
>> Let's provide a convenient way for QEMU developers to run them,
>> by making use of the tests-venv with the required setup.
>>
>> Also, while the Avocado test runner will take care of creating a
>> location to save test results to, it was understood that it's better
>> if the results are kept within the build tree.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>>  tests/Makefile.include      | 17 +++++++++++++++--
>>  tests/venv-requirements.txt |  1 +
>>  3 files changed, 46 insertions(+), 7 deletions(-)
>>
>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>> index 727c4019b5..b992a2961d 100644
>> --- a/docs/devel/testing.rst
>> +++ b/docs/devel/testing.rst
>> @@ -545,10 +545,31 @@ Tests based on ``avocado_qemu.Test`` can easily:
>>     - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
>>     - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
>>  
>> -Installation
>> -------------
>> +Running tests
>> +-------------
>>  
>> -To install Avocado and its dependencies, run:
>> +You can run the acceptance tests simply by executing:
>> +
>> +.. code::
>> +
>> +  make check-acceptance
>> +
>> +This involves the automatic creation of Python virtual environment
>> +within the build tree (at ``tests/venv``) which will have all the
>> +right dependencies, and will save tests results also within the
>> +build tree (at ``tests/results``).
>> +
>> +Note: the build environment must be using a Python 3 stack, and have
>> +the ``venv`` and ``pip`` packages installed.  If necessary, make sure
>> +``configure`` is called with ``--python=`` and that those modules are
>> +available.  On Debian and Ubuntu based systems, depending on the
>> +specific version, they may be on packages named ``python3-venv`` and
>> +``python3-pip``.
>> +
>> +Manual Installation
>> +-------------------
>> +
>> +To manually install Avocado and its dependencies, run:
>>  
>>  .. code::
>>  
>> @@ -689,11 +710,15 @@ The exact QEMU binary to be used on QEMUMachine.
>>  Uninstalling Avocado
>>  --------------------
>>  
>> -If you've followed the installation instructions above, you can easily
>> -uninstall Avocado.  Start by listing the packages you have installed::
>> +If you've followed the manual installation instructions above, you can
>> +easily uninstall Avocado.  Start by listing the packages you have
>> +installed::
>>  
>>    pip list --user
>>  
>>  And remove any package you want with::
>>  
>>    pip uninstall <package_name>
>> +
>> +If you've used ``make check-acceptance``, the Python virtual environment where
>> +Avocado is installed will be cleaned up as part of ``make check-clean``.
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 68af79927d..00fdf9913e 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -11,6 +11,7 @@ check-help:
>>  	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
>>  	@echo " $(MAKE) check-block          Run block tests"
>>  	@echo " $(MAKE) check-tcg            Run TCG tests"
>> +	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
>>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>>  	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>  	@echo " $(MAKE) check-clean          Clean the tests"
>> @@ -1018,10 +1019,11 @@ check-decodetree:
>>  
>>  # Python venv for running tests
>>  
>> -.PHONY: check-venv
>> +.PHONY: check-venv check-acceptance
>>  
>>  TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>  TESTS_VENV_REQ=$(SRC_PATH)/tests/venv-requirements.txt
>> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>  
>>  $(TESTS_VENV_DIR):
>>  	$(call quiet-command, \
>> @@ -1031,8 +1033,19 @@ $(TESTS_VENV_DIR):
>>              $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
>>              PIP, $(TESTS_VENV_REQ))
>>  
>> +$(TESTS_RESULTS_DIR):
>> +	$(call quiet-command, mkdir -p $@, \
>> +            MKDIR, $@)
>> +
>>  check-venv: $(TESTS_VENV_DIR)
>>  
>> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
>> +	$(call quiet-command, \
>> +            $(TESTS_VENV_DIR)/bin/python -m avocado \
>> +            --show=none run --job-results-dir=$(TESTS_RESULTS_DIR) --failfast=on \
> 
> Can we add an overwritable env var such AVOCADO_SHOW=none ?
> I'd use it for Travis, to see which test failed on the console output.
> 

Yes, I was expecting a question/suggestion along those lines, because I
felt the need for that too.

Do you think we should use a non quiet version by default?  Even people
running `make check-acceptance` on their own systems might get
frustrated as the number of tests, and test run time, increases.

- Cleber.

>> +            $(SRC_PATH)/tests/acceptance, \
>> +            "AVOCADO", "tests/acceptance")
>> +
>>  # Consolidated targets
>>  
>>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
>> @@ -1046,7 +1059,7 @@ check-clean:
>>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>>  	rm -f tests/test-qapi-gen-timestamp
>> -	rm -rf $(TESTS_VENV_DIR)
>> +	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
>>  
>>  clean: check-clean
>>  
>> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
>> index d39f9d1576..64c6e27a94 100644
>> --- a/tests/venv-requirements.txt
>> +++ b/tests/venv-requirements.txt
>> @@ -1,3 +1,4 @@
>>  # Add Python module requirements, one per line, to be installed
>>  # in the tests/venv Python virtual environment. For more info,
>>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>> +avocado-framework==65.0
>>

Re: [Qemu-devel] [PATCH v2 3/3] Travis support for the acceptance tests

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> On 10/9/18 9:46 AM, Philippe Mathieu-Daudé wrote:
>> Hi Cleber,
>>
>> On 09/10/2018 06:18, Cleber Rosa wrote:
>>> This enables the execution of the acceptance tests on Travis.
>>
>> Did you test this? =)
>>
>
> I did have some jobs on Travis that looked promising.  But yeah, later I
> found some issues. :/
>
>>>
>>> Because the Travis environment is based on Ubuntu Trusty, it requires
>>> the python3-pip.
>>>
>>> Note: while another supposedely required component on newer versions
>>> (such as on Bionic) split the Python 3 installation further on the
>>> python3-venv package.
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  .travis.yml | 6 ++++++
>>>  1 file changed, 6 insertions(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 95be6ec59f..db1a31ea51 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -36,6 +36,7 @@ addons:
>>>        - liburcu-dev
>>>        - libusb-1.0-0-dev
>>>        - libvte-2.90-dev
>>> +      - python3-pip
>>>        - sparse
>>>        - uuid-dev
>>>        - gcovr
>>> @@ -117,6 +118,11 @@ matrix:
>>>      - env: CONFIG="--target-list=x86_64-softmmu"
>>>        python:
>>>          - "3.6"
>>> +    # Acceptance (Functional) tests
>>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>>> +           TEST_CMD="make check-acceptance"
>>> +      python:
>>> +        - "3.6"
>>>      # Using newer GCC with sanitizers
>>>      - addons:
>>>          apt:
>>>
>>
>> Using the following patch:
>>
>> -- >8 --
>> diff --git a/.travis.yml b/.travis.yml
>> index 95be6ec59f..87e0c9a13f 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -118,4 +118,15 @@ matrix:
>>        python:
>>          - "3.6"
>> +    # Acceptance (Functional) tests
>> +    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
>> +           TEST_CMD="make check-acceptance"
>> +           # sudo rm /usr/local/bin/pip*
>> +      python:
>> +        - "3.6"
>> +      addons:
>> +        apt:
>> +          packages:
>> +            - python3-pip
>> +            - python3.4-venv
>
> This patch revealed to me that, even though we're asking Travis for a
> Python 3.6 stack, we're manually pointing to the primary Python (3.4)
> installation.  This is in itself a source of problems.
>
> Read on...
>
>>      # Using newer GCC with sanitizers
>>      - addons:
>> ---
>>
>> I got some improvements until:
>>
>>   VENV    /home/travis/build/philmd/qemu/tests/venv
>>   MKDIR   /home/travis/build/philmd/qemu/tests/results
>>   PIP     /home/travis/build/philmd/qemu/tests/venv-requirements.txt
>> Exception:
>> Traceback (most recent call last):
>>   File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in
>> main
>>     status = self.run(options, args)
>>   File "/usr/lib/python3/dist-packages/pip/commands/install.py", line
>> 283, in run
>>     requirement_set.install(install_options, global_options,
>> root=options.root_path)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
>>     requirement.install(install_options, global_options, *args, **kwargs)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
>>     self.move_wheel_files(self.source_dir, root=root)
>>   File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in
>> move_wheel_files
>>     pycompile=self.pycompile,
>>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 206, in
>> move_wheel_files
>>     clobber(source, lib_dir, True)
>>   File "/usr/lib/python3/dist-packages/pip/wheel.py", line 193, in clobber
>>     os.makedirs(destsubdir)
>>   File "/usr/lib/python3.4/os.py", line 237, in makedirs
>>     mkdir(name, mode)
>> PermissionError: [Errno 13] Permission denied:
>> '/usr/local/lib/python3.4/dist-packages/avocado'
>>
>
> This looks to me like it's caused by the use of the system wide Python
> 3.4 installation, as mentioned above.  That, and possibly the
> "--system-wide-packages" flag to the venv.
>
> So, in order to understand if, in those environments, we should let
> "configure" pick up the right Python, I did the following experiment:
>
>  $ python3 -m venv /tmp/py3-build
>  $ . /tmp/py3-build/bin/activate
>  $ which python
>  /tmp/py3-build/bin/python
>  $ /tmp/py3-build/bin/python --version
>  Python 3.6.6
>
> This shows that when a venv is activated, the "right" Python should be
> available in the $PATH.  So, building QEMU without pointint to a
> specific Python binary (when inside a venv) gives you:
>
>  $ mkdir -p /tmp/qemu-build
>  $ cd /tmp/qemu-build
>  $ ~/src/qemu/configure --target-list="x86_64-softmmu"
>  $ make print-PYTHON
>  PYTHON=python -B
>  $ which python
>  /tmp/py3-build/bin/python
>
> The whole question now is how/where Travis puts the requested Python
> version (python: - "3.6").  Is it on a venv?  Is it on a specific
> location? Is that reliable enough?
>
> I'm going to look for those answers now.

You can experiment with the Travis environment on your own system using:

  make docker-test-build@travis DEBUG=1

>
> - Cleber.
>
>> See: https://travis-ci.org/philmd/qemu/jobs/439138706
>>


--
Alex Bennée