All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v6 00/25] python: create installable package
@ 2021-05-12 23:12 John Snow
  2021-05-12 23:12 ` [PATCH v6 01/25] iotests/297: add --namespace-packages to mypy arguments John Snow
                   ` (25 more replies)
  0 siblings, 26 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Based-on: <20210512214642.2803189-1-jsnow@redhat.com>\r
CI: https://gitlab.com/jsnow/qemu/-/pipelines/302010131\r
GitLab: https://gitlab.com/jsnow/qemu/-/tree/python-package-mk3\r
MR: https://gitlab.com/jsnow/qemu/-/merge_requests/4\r
\r
I invite you to leave review comments on my mock merge request on\r
gitlab, submitted against my own mirror. I will, as always, also respond\r
to feedback on-list.\r
\r
ABOUT\r
=====\r
\r
This series factors the python/qemu directory as an installable\r
package. It does not yet actually change the mechanics of how any other\r
python source in the tree actually consumes it (yet), beyond the import\r
path -- some import statements change in a few places.\r
\r
RATIONALE\r
=========\r
\r
The primary motivation of this series is primarily to formalize our\r
dependencies on mypy, flake8, isort, and pylint alongside versions that\r
are known to work. It does this using the setup.cfg and setup.py\r
files. It also adds explicitly pinned versions (using Pipfile.lock) of\r
these dependencies that should behave in a repeatable and known way for\r
developers and CI environments both. Lastly, it enables those CI checks\r
such that we can enforce Python coding quality checks via the CI tests.\r
\r
An auxiliary motivation is that this package is formatted in such a way\r
that it COULD be uploaded to https://pypi.org/project/qemu and installed\r
independently of qemu.git with `pip install qemu`, but that button\r
remains *unpushed* and this series *will not* cause any such\r
releases. We have time to debate finer points like API guarantees and\r
versioning even after this series is merged.\r
\r
Other bits of interest\r
----------------------\r
\r
With the python tooling as a proper package, you can install this\r
package in editable or production mode to a virtual environment, your\r
local user environment, or your system packages. The primary benefit of\r
this is to gain access to QMP tooling regardless of CWD, without needing\r
to battle sys.path (and confounding other python analysis tools).\r
\r
For example: when developing, you may go to qemu/python/ and run `make\r
venv` followed by `pipenv shell` to activate a virtual environment that\r
contains the qemu python packages. These packages will always reflect\r
the current version of the source files in the tree. When you are\r
finished, you can simply exit the shell (^d) to remove these packages\r
from your python environment.\r
\r
When not developing, you could install a version of this package to your\r
environment outright to gain access to the QMP and QEMUMachine classes\r
for lightweight scripting and testing by using pip: "pip install\r
[--user] ."\r
\r
TESTING THIS SERIES\r
===================\r
\r
First of all, nothing should change. Without any intervention,\r
everything should behave exactly as it did before. The only new\r
information here comes from how to interact with and run the linters\r
that will be enforcing code quality standards in this subdirectory.\r
\r
There are various invocations available that will test subtly different\r
combinations using subtly different environments. I am assuming some\r
light knowledge of Python environments and installing Python packages\r
here. If you have questions, I would be delighted to answer them.\r
\r
To test the new tests, CD to ./python/ first, and then:\r
\r
0. Try "make" or "make help" to get a sense of this series.\r
\r
1. Try "make venv && pipenv shell" to get a venv with the package\r
   installed to it in editable mode. Ctrl+d exits this venv shell. While\r
   in this shell, any python script that uses "from qemu.[qmp|machine]\r
   import ..." should work correctly regardless of where the script is,\r
   or what your CWD is.\r
\r
   This will pull some packages from PyPI and install them into the\r
   virtual environment, leaving your normal environment untouched.\r
\r
   You will need Python 3.6 and pipenv installed on your system to do\r
   this step. For Fedora: "dnf install python36 pipenv" will do the\r
   trick. If you don't have this, skip down to #4 and onwards.\r
\r
2. Try "make check" while still in the shell to run the Python linters\r
using the venv built in the previous step. This will run avocado, which\r
will in turn execute mypy, flake8, isort and pylint with the correct\r
arguments.\r
\r
3. Having exited the shell from above, try "make venv-check". This will\r
create and update the venv if needed, then run 'make check' within the\r
context of that shell. It should pass as long as the above did. You\r
should be able to run "make distclean" prior to running "make\r
venv-check" and have the entire process work start to finish.\r
\r
4. Still outside of the venv, you may try running "make check". This\r
will not install anything, but unless you have the right Python\r
dependencies installed, these tests may fail for you. You might try\r
using "pip install --user .[devel]" to install the development packages\r
needed to run the tests successfully to your local user's python\r
environment. Once done, you will probably want to "pip uninstall qemu"\r
to remove the qemu packages to avoid them interfering with other things.\r
\r
5. "make distclean" will delete the venv and any temporary files that\r
may have been created by packaging, installing, testing, etc.\r
\r
6. You may also (if you wish) create your own environment using a Python\r
   other than 3.6, bypassing pipenv. This may be useful for environments\r
   in which you simply do not have Python 3.6 readily available, or\r
   explicitly wish to test against a different version.\r
\r
   Using the Python of your choice:\r
\r
   - `> make distclean` (Remove the .venv, if you created one.)\r
   - `> /usr/bin/python3.9 -m venv .venv`\r
   - `> source .venv/bin/activate` (or activate.[fish|csh])\r
   - `> make develop`  (Installs linter deps to this venv)\r
   - `> make check`    (Runs tests using current env)\r
   - `> deactivate`    (Leaves the venv)\r
\r
   This *should* pass, and if it doesn't, it's a bug. Due to\r
   dependencies changing like shifting sands, it's impossible to promise\r
   support for every last combination of dependency and python\r
   version. This is why the "make venv-check" target uses Python 3 and a\r
   very explicit set of packages instead. Still, this form *should*\r
   pass.\r
\r
7. If you are running Fedora or a distro that packages multiple Python\r
versions, you can try running "make check-tox" to run the linters under\r
all Python versions from Python 3.6 to 3.9. This will create a series of\r
virtual environments under the .tox folder, leaving your existing\r
environment unscathed. This should pass, provided you have the requisite\r
Python versions on hand. When you are done, "make distclean" should\r
remove any testing files.\r
\r
CHANGELOG\r
=========\r
\r
V6\r
--\r
\r
```\r
001/25:[----] [--] 'iotests/297: add --namespace-packages to mypy arguments'\r
002/25:[----] [-C] 'python: create qemu packages'\r
003/25:[----] [--] 'python: create utils sub-package'\r
004/25:[0005] [FC] 'python: add qemu package installer'\r
005/25:[0002] [FC] 'python: add VERSION file'\r
006/25:[----] [--] 'python: add directory structure README.rst files'\r
007/25:[down] 'python: add MANIFEST.in'\r
008/25:[0001] [FC] 'python: Add pipenv support'\r
009/25:[----] [--] 'python: add pylint import exceptions'\r
010/25:[----] [--] 'python: move pylintrc into setup.cfg'\r
011/25:[0083] [FC] 'python: add pylint to pipenv'\r
012/25:[----] [--] 'python: move flake8 config to setup.cfg'\r
013/25:[----] [--] 'python: add excluded dirs to flake8 config'\r
014/25:[0038] [FC] 'python: Add flake8 to pipenv'\r
015/25:[----] [--] 'python: move mypy.ini into setup.cfg'\r
016/25:[0050] [FC] 'python: add mypy to pipenv'\r
017/25:[----] [--] 'python: move .isort.cfg into setup.cfg'\r
018/25:[0006] [FC] 'python/qemu: add isort to pipenv'\r
019/25:[0004] [FC] 'python/qemu: add qemu package itself to pipenv'\r
020/25:[0014] [FC] 'python: add devel package requirements to setuptools'\r
021/25:[down] 'python: add avocado-framework and tests'\r
022/25:[0015] [FC] 'python: add Makefile for some common tasks'\r
023/25:[0018] [FC] 'python: add .gitignore'\r
024/25:[0004] [FC] 'gitlab: add python linters to CI'\r
025/25:[down] 'python: add tox support'\r
```\r
\r
- 04: Changed maintainer blurb in PACKAGE.rst\r
- 05: VERSION is now 0.6.1a0\r
- 07: New, needed for both tox and using Pipenv to install the qemu\r
      package(s) in editable mode.\r
- 08: Add README entries explaining the purpose of the pipenv files.\r
- 11: pylint dependency was bumped to 2.7.0.\r
- 14: pipenv changes.\r
- 16: pipenv changes.\r
- 18: isort pinned at >= 5.1.2 now; see commit message.\r
- 19: pipenv changes.\r
- 20: Context differences from isort, pylint, and pipenv changes.\r
- 22:\r
   - Add 'make develop'\r
   - Add more README.rst entries.\r
   - Change pytest invocation to avocado.\r
- 23:\r
   - Add comments to gitignore, split ignores out by function/category\r
   - Ignore vscode files. (I like vscode now!)\r
- 24: Use the 'needs' feature of Gitlab CI and move the test to 'test'\r
  stage\r
\r
STATUS\r
=======\r
\r
```\r
+ [01] iotests-297-add-namespace      # [RB] CR [SOB] JS\r
+ [02] python-create-qemu-packages    # [SOB] JS\r
+ [03] python-create-utils-sub        # [SOB] JS\r
+ [04] python-add-qemu-package        # [SOB] JS\r
+ [05] python-add-version-file        # [RB] CR [SOB] JS\r
+ [06] python-add-directory-structure # [SOB] JS\r
+ [07] python-add-manifest-in         # [SOB] JS\r
+ [08] python-add-pipenv-support      # [RB] CR [SOB] JS\r
+ [09] python-add-pylint-import       # [RB] CR [SOB] JS\r
+ [10] python-move-pylintrc-into      # [RB] CR [TB] CR [SOB] JS\r
+ [11] python-add-pylint-to-pipenv    # [SOB] JS\r
+ [12] python-move-flake8-config-to   # [RB] CR [SOB] JS\r
+ [13] python-add-excluded-dirs-to    # [SOB] JS\r
+ [14] python-add-flake8-to-pipenv    # [RB] CR [TB] CR [SOB] JS\r
+ [15] python-move-mypy-ini-into      # [SOB] JS\r
+ [16] python-add-mypy-to-pipenv      # [RB] CR [TB] CR [SOB] JS\r
+ [17] python-move-isort-cfg-into     # [RB] CR [SOB] JS\r
+ [18] python-qemu-add-isort-to       # [SOB] JS\r
+ [19] python-qemu-add-qemu-package   # [RB] CR [TB] CR [SOB] JS\r
+ [20] python-add-devel-package       # [SOB] JS\r
+ [21] python-add-avocado-framework   # [SOB] JS\r
+ [22] python-add-makefile-for-some   # [SOB] JS\r
+ [23] python-add-gitignore           # [SOB] JS\r
+ [24] gitlab-add-python-linters-to   # [SOB] JS\r
> [25] python-add-tox-support         # [SOB] JS\r
```\r
\r
John Snow (25):\r
  iotests/297: add --namespace-packages to mypy arguments\r
  python: create qemu packages\r
  python: create utils sub-package\r
  python: add qemu package installer\r
  python: add VERSION file\r
  python: add directory structure README.rst files\r
  python: add MANIFEST.in\r
  python: Add pipenv support\r
  python: add pylint import exceptions\r
  python: move pylintrc into setup.cfg\r
  python: add pylint to pipenv\r
  python: move flake8 config to setup.cfg\r
  python: add excluded dirs to flake8 config\r
  python: Add flake8 to pipenv\r
  python: move mypy.ini into setup.cfg\r
  python: add mypy to pipenv\r
  python: move .isort.cfg into setup.cfg\r
  python/qemu: add isort to pipenv\r
  python/qemu: add qemu package itself to pipenv\r
  python: add devel package requirements to setuptools\r
  python: add avocado-framework and tests\r
  python: add Makefile for some common tasks\r
  python: add .gitignore\r
  gitlab: add python linters to CI\r
  python: add tox support\r
\r
 python/PACKAGE.rst                          |  33 +++\r
 python/README.rst                           |  53 +++++\r
 python/qemu/README.rst                      |   8 +\r
 python/qemu/machine/README.rst              |   9 +\r
 python/qemu/qmp/README.rst                  |   9 +\r
 python/qemu/utils/README.rst                |   7 +\r
 .gitlab-ci.yml                              |  12 +\r
 python/.gitignore                           |  20 ++\r
 python/MANIFEST.in                          |   2 +\r
 python/Makefile                             |  47 ++++\r
 python/Pipfile                              |  13 ++\r
 python/Pipfile.lock                         | 231 ++++++++++++++++++++\r
 python/VERSION                              |   1 +\r
 python/avocado.cfg                          |  10 +\r
 python/mypy.ini                             |   4 -\r
 python/qemu/.flake8                         |   2 -\r
 python/qemu/.isort.cfg                      |   7 -\r
 python/qemu/__init__.py                     |  11 -\r
 python/qemu/machine/__init__.py             |  36 +++\r
 python/qemu/{ => machine}/console_socket.py |   0\r
 python/qemu/{ => machine}/machine.py        |  16 +-\r
 python/qemu/{ => machine}/qtest.py          |   3 +-\r
 python/qemu/pylintrc                        |  58 -----\r
 python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-\r
 python/qemu/utils/__init__.py               |  23 ++\r
 python/qemu/{ => utils}/accel.py            |   0\r
 python/setup.cfg                            |  79 +++++++\r
 python/setup.py                             |  23 ++\r
 python/tests/flake8.sh                      |   2 +\r
 python/tests/isort.sh                       |   2 +\r
 python/tests/mypy.sh                        |   2 +\r
 python/tests/pylint.sh                      |   2 +\r
 python/tox.ini                              |  13 ++\r
 tests/acceptance/avocado_qemu/__init__.py   |   4 +-\r
 tests/acceptance/virtio-gpu.py              |   2 +-\r
 tests/docker/dockerfiles/fedora.docker      |   2 +\r
 tests/qemu-iotests/297                      |   1 +\r
 tests/qemu-iotests/300                      |   4 +-\r
 tests/qemu-iotests/iotests.py               |   2 +-\r
 tests/vm/aarch64vm.py                       |   2 +-\r
 tests/vm/basevm.py                          |   2 +-\r
 41 files changed, 672 insertions(+), 97 deletions(-)\r
 create mode 100644 python/PACKAGE.rst\r
 create mode 100644 python/README.rst\r
 create mode 100644 python/qemu/README.rst\r
 create mode 100644 python/qemu/machine/README.rst\r
 create mode 100644 python/qemu/qmp/README.rst\r
 create mode 100644 python/qemu/utils/README.rst\r
 create mode 100644 python/.gitignore\r
 create mode 100644 python/MANIFEST.in\r
 create mode 100644 python/Makefile\r
 create mode 100644 python/Pipfile\r
 create mode 100644 python/Pipfile.lock\r
 create mode 100644 python/VERSION\r
 create mode 100644 python/avocado.cfg\r
 delete mode 100644 python/mypy.ini\r
 delete mode 100644 python/qemu/.flake8\r
 delete mode 100644 python/qemu/.isort.cfg\r
 delete mode 100644 python/qemu/__init__.py\r
 create mode 100644 python/qemu/machine/__init__.py\r
 rename python/qemu/{ => machine}/console_socket.py (100%)\r
 rename python/qemu/{ => machine}/machine.py (98%)\r
 rename python/qemu/{ => machine}/qtest.py (98%)\r
 delete mode 100644 python/qemu/pylintrc\r
 rename python/qemu/{qmp.py => qmp/__init__.py} (96%)\r
 create mode 100644 python/qemu/utils/__init__.py\r
 rename python/qemu/{ => utils}/accel.py (100%)\r
 create mode 100644 python/setup.cfg\r
 create mode 100755 python/setup.py\r
 create mode 100755 python/tests/flake8.sh\r
 create mode 100755 python/tests/isort.sh\r
 create mode 100755 python/tests/mypy.sh\r
 create mode 100755 python/tests/pylint.sh\r
 create mode 100644 python/tox.ini\r
\r
-- \r
2.30.2\r
\r



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 01/25] iotests/297: add --namespace-packages to mypy arguments
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 02/25] python: create qemu packages John Snow
                   ` (24 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

mypy is kind of weird about how it handles imports. For legacy reasons,
it won't load PEP 420 namespaces, because of logic implemented prior to
that becoming a standard.

So, if you plan on using any, you have to pass
--namespace-packages. Alright, fine.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 tests/qemu-iotests/297 | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index a37910b42d9..433b7323368 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -95,6 +95,7 @@ def run_linters():
                             '--warn-redundant-casts',
                             '--warn-unused-ignores',
                             '--no-implicit-reexport',
+                            '--namespace-packages',
                             filename),
                            env=env,
                            check=False,
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 02/25] python: create qemu packages
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
  2021-05-12 23:12 ` [PATCH v6 01/25] iotests/297: add --namespace-packages to mypy arguments John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-18 14:31   ` Cleber Rosa
  2021-05-21 17:15   ` Willian Rampazzo
  2021-05-12 23:12 ` [PATCH v6 03/25] python: create utils sub-package John Snow
                   ` (23 subsequent siblings)
  25 siblings, 2 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

move python/qemu/*.py to python/qemu/[machine, qmp]/*.py and update import
directives across the tree.

This is done to create a PEP420 namespace package, in which we may
create subpackages. To do this, the namespace directory ("qemu") should
not have any modules in it. Those files will go into new 'machine' and 'qmp'
subpackages instead.

Implement machine/__init__.py making the top-level classes and functions
from its various modules available directly inside the package. Change
qmp.py to qmp/__init__.py similarly, such that all of the useful QMP
library classes are available directly from "qemu.qmp" instead of
"qemu.qmp.qmp".

Signed-off-by: John Snow <jsnow@redhat.com>


---

Note for reviewers: in the next patch, I add a utils sub-package and
move qemu/machine/accel.py to qemu/utils/accel.py. If we like it that
way, we can squash it in here if we want, or just leave it as its own
follow-up patch, I am just leaving it as something that will be easy to
un-do or change for now.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/{qemu => }/.isort.cfg                |  0
 python/qemu/__init__.py                     | 11 ------
 python/qemu/{ => machine}/.flake8           |  0
 python/qemu/machine/__init__.py             | 41 +++++++++++++++++++++
 python/qemu/{ => machine}/accel.py          |  0
 python/qemu/{ => machine}/console_socket.py |  0
 python/qemu/{ => machine}/machine.py        | 16 +++++---
 python/qemu/{ => machine}/pylintrc          |  0
 python/qemu/{ => machine}/qtest.py          |  3 +-
 python/qemu/{qmp.py => qmp/__init__.py}     | 12 +++++-
 tests/acceptance/avocado_qemu/__init__.py   |  4 +-
 tests/acceptance/virtio-gpu.py              |  2 +-
 tests/qemu-iotests/300                      |  4 +-
 tests/qemu-iotests/iotests.py               |  2 +-
 tests/vm/aarch64vm.py                       |  2 +-
 tests/vm/basevm.py                          |  3 +-
 16 files changed, 73 insertions(+), 27 deletions(-)
 rename python/{qemu => }/.isort.cfg (100%)
 delete mode 100644 python/qemu/__init__.py
 rename python/qemu/{ => machine}/.flake8 (100%)
 create mode 100644 python/qemu/machine/__init__.py
 rename python/qemu/{ => machine}/accel.py (100%)
 rename python/qemu/{ => machine}/console_socket.py (100%)
 rename python/qemu/{ => machine}/machine.py (98%)
 rename python/qemu/{ => machine}/pylintrc (100%)
 rename python/qemu/{ => machine}/qtest.py (99%)
 rename python/qemu/{qmp.py => qmp/__init__.py} (96%)

diff --git a/python/qemu/.isort.cfg b/python/.isort.cfg
similarity index 100%
rename from python/qemu/.isort.cfg
rename to python/.isort.cfg
diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
deleted file mode 100644
index 4ca06c34a41..00000000000
--- a/python/qemu/__init__.py
+++ /dev/null
@@ -1,11 +0,0 @@
-# QEMU library
-#
-# Copyright (C) 2015-2016 Red Hat Inc.
-# Copyright (C) 2012 IBM Corp.
-#
-# Authors:
-#  Fam Zheng <famz@redhat.com>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.  See
-# the COPYING file in the top-level directory.
-#
diff --git a/python/qemu/.flake8 b/python/qemu/machine/.flake8
similarity index 100%
rename from python/qemu/.flake8
rename to python/qemu/machine/.flake8
diff --git a/python/qemu/machine/__init__.py b/python/qemu/machine/__init__.py
new file mode 100644
index 00000000000..0ac6c1e36e3
--- /dev/null
+++ b/python/qemu/machine/__init__.py
@@ -0,0 +1,41 @@
+"""
+QEMU development and testing library.
+
+This library provides a few high-level classes for driving QEMU from a
+test suite, not intended for production use.
+
+- QEMUMachine: Configure and Boot a QEMU VM
+ - QEMUQtestMachine: VM class, with a qtest socket.
+
+- QEMUQtestProtocol: Connect to, send/receive qtest messages.
+
+- list_accel: List available accelerators
+- kvm_available: Probe for KVM support
+- tcg_available: Probe for TCG support
+"""
+
+# Copyright (C) 2020-2021 John Snow for Red Hat Inc.
+# Copyright (C) 2015-2016 Red Hat Inc.
+# Copyright (C) 2012 IBM Corp.
+#
+# Authors:
+#  John Snow <jsnow@redhat.com>
+#  Fam Zheng <fam@euphon.net>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.  See
+# the COPYING file in the top-level directory.
+#
+
+from .accel import kvm_available, list_accel, tcg_available
+from .machine import QEMUMachine
+from .qtest import QEMUQtestMachine, QEMUQtestProtocol
+
+
+__all__ = (
+    'list_accel',
+    'kvm_available',
+    'tcg_available',
+    'QEMUMachine',
+    'QEMUQtestProtocol',
+    'QEMUQtestMachine',
+)
diff --git a/python/qemu/accel.py b/python/qemu/machine/accel.py
similarity index 100%
rename from python/qemu/accel.py
rename to python/qemu/machine/accel.py
diff --git a/python/qemu/console_socket.py b/python/qemu/machine/console_socket.py
similarity index 100%
rename from python/qemu/console_socket.py
rename to python/qemu/machine/console_socket.py
diff --git a/python/qemu/machine.py b/python/qemu/machine/machine.py
similarity index 98%
rename from python/qemu/machine.py
rename to python/qemu/machine/machine.py
index 0df5b2f386f..dea343afeba 100644
--- a/python/qemu/machine.py
+++ b/python/qemu/machine/machine.py
@@ -38,8 +38,14 @@
     Type,
 )
 
-from . import console_socket, qmp
-from .qmp import QMPMessage, QMPReturnValue, SocketAddrT
+from qemu.qmp import (
+    QEMUMonitorProtocol,
+    QMPMessage,
+    QMPReturnValue,
+    SocketAddrT,
+)
+
+from . import console_socket
 
 
 LOG = logging.getLogger(__name__)
@@ -139,7 +145,7 @@ def __init__(self,
         self._events: List[QMPMessage] = []
         self._iolog: Optional[str] = None
         self._qmp_set = True   # Enable QMP monitor by default.
-        self._qmp_connection: Optional[qmp.QEMUMonitorProtocol] = None
+        self._qmp_connection: Optional[QEMUMonitorProtocol] = None
         self._qemu_full_args: Tuple[str, ...] = ()
         self._temp_dir: Optional[str] = None
         self._launched = False
@@ -316,7 +322,7 @@ def _pre_launch(self) -> None:
             if self._remove_monitor_sockfile:
                 assert isinstance(self._monitor_address, str)
                 self._remove_files.append(self._monitor_address)
-            self._qmp_connection = qmp.QEMUMonitorProtocol(
+            self._qmp_connection = QEMUMonitorProtocol(
                 self._monitor_address,
                 server=True,
                 nickname=self._name
@@ -543,7 +549,7 @@ def set_qmp_monitor(self, enabled: bool = True) -> None:
         self._qmp_set = enabled
 
     @property
-    def _qmp(self) -> qmp.QEMUMonitorProtocol:
+    def _qmp(self) -> QEMUMonitorProtocol:
         if self._qmp_connection is None:
             raise QEMUMachineError("Attempt to access QMP with no connection")
         return self._qmp_connection
diff --git a/python/qemu/pylintrc b/python/qemu/machine/pylintrc
similarity index 100%
rename from python/qemu/pylintrc
rename to python/qemu/machine/pylintrc
diff --git a/python/qemu/qtest.py b/python/qemu/machine/qtest.py
similarity index 99%
rename from python/qemu/qtest.py
rename to python/qemu/machine/qtest.py
index 39a0cf62fe9..53926e434a7 100644
--- a/python/qemu/qtest.py
+++ b/python/qemu/machine/qtest.py
@@ -26,8 +26,9 @@
     TextIO,
 )
 
+from qemu.qmp import SocketAddrT
+
 from .machine import QEMUMachine
-from .qmp import SocketAddrT
 
 
 class QEMUQtestProtocol:
diff --git a/python/qemu/qmp.py b/python/qemu/qmp/__init__.py
similarity index 96%
rename from python/qemu/qmp.py
rename to python/qemu/qmp/__init__.py
index 2cd4d43036c..9606248a3d2 100644
--- a/python/qemu/qmp.py
+++ b/python/qemu/qmp/__init__.py
@@ -1,4 +1,14 @@
-""" QEMU Monitor Protocol Python class """
+"""
+QEMU Monitor Protocol (QMP) development library & tooling.
+
+This package provides a fairly low-level class for communicating to QMP
+protocol servers, as implemented by QEMU, the QEMU Guest Agent, and the
+QEMU Storage Daemon. This library is not intended for production use.
+
+`QEMUMonitorProtocol` is the primary class of interest, and all errors
+raised derive from `QMPError`.
+"""
+
 # Copyright (C) 2009, 2010 Red Hat Inc.
 #
 # Authors:
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 83b1741ec85..ff7bf81f1a9 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -40,9 +40,9 @@
 
 sys.path.append(os.path.join(SOURCE_DIR, 'python'))
 
-from qemu.accel import kvm_available
-from qemu.accel import tcg_available
 from qemu.machine import QEMUMachine
+from qemu.machine import kvm_available
+from qemu.machine import tcg_available
 
 def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
diff --git a/tests/acceptance/virtio-gpu.py b/tests/acceptance/virtio-gpu.py
index ab18cddbb73..0685e30bcae 100644
--- a/tests/acceptance/virtio-gpu.py
+++ b/tests/acceptance/virtio-gpu.py
@@ -10,7 +10,7 @@
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import is_readable_executable_file
 
-from qemu.accel import kvm_available
+from qemu.machine import kvm_available
 
 import os
 import socket
diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
index b475a92c478..fe94de84edd 100755
--- a/tests/qemu-iotests/300
+++ b/tests/qemu-iotests/300
@@ -28,7 +28,7 @@ import iotests
 
 # Import qemu after iotests.py has amended sys.path
 # pylint: disable=wrong-import-order
-import qemu
+from qemu.machine import machine
 
 BlockBitmapMapping = List[Dict[str, object]]
 
@@ -466,7 +466,7 @@ class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
         # the failed migration
         try:
             self.vm_b.shutdown()
-        except qemu.machine.AbnormalShutdown:
+        except machine.AbnormalShutdown:
             pass
 
     def test_aliased_bitmap_name_too_long(self) -> None:
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 12e876fa67d..c144cb13e00 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -39,7 +39,7 @@
 
 # pylint: disable=import-error, wrong-import-position
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu import qtest
+from qemu.machine import qtest
 from qemu.qmp import QMPMessage
 
 # Use this logger for logging messages directly from the iotests module
diff --git a/tests/vm/aarch64vm.py b/tests/vm/aarch64vm.py
index d70ab843b6b..39ff99b0859 100644
--- a/tests/vm/aarch64vm.py
+++ b/tests/vm/aarch64vm.py
@@ -14,7 +14,7 @@
 import sys
 import subprocess
 import basevm
-from qemu.accel import kvm_available
+from qemu.machine import kvm_available
 
 # This is the config needed for current version of QEMU.
 # This works for both kvm and tcg.
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 00f1d5ca8da..12d08cf2b1b 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -19,8 +19,7 @@
 import time
 import datetime
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.accel import kvm_available
-from qemu.machine import QEMUMachine
+from qemu.machine import kvm_available, QEMUMachine
 import subprocess
 import hashlib
 import argparse
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 03/25] python: create utils sub-package
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
  2021-05-12 23:12 ` [PATCH v6 01/25] iotests/297: add --namespace-packages to mypy arguments John Snow
  2021-05-12 23:12 ` [PATCH v6 02/25] python: create qemu packages John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-18 15:12   ` Cleber Rosa
  2021-05-21 17:18   ` Willian Rampazzo
  2021-05-12 23:12 ` [PATCH v6 04/25] python: add qemu package installer John Snow
                   ` (22 subsequent siblings)
  25 siblings, 2 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Create a space for miscellaneous things that don't belong strictly in
"qemu.machine" nor "qemu.qmp" packages.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/machine/__init__.py           |  8 --------
 python/qemu/utils/__init__.py             | 23 +++++++++++++++++++++++
 python/qemu/{machine => utils}/accel.py   |  0
 tests/acceptance/avocado_qemu/__init__.py |  4 ++--
 tests/acceptance/virtio-gpu.py            |  2 +-
 tests/vm/aarch64vm.py                     |  2 +-
 tests/vm/basevm.py                        |  3 ++-
 7 files changed, 29 insertions(+), 13 deletions(-)
 create mode 100644 python/qemu/utils/__init__.py
 rename python/qemu/{machine => utils}/accel.py (100%)

diff --git a/python/qemu/machine/__init__.py b/python/qemu/machine/__init__.py
index 0ac6c1e36e3..98302ea31e7 100644
--- a/python/qemu/machine/__init__.py
+++ b/python/qemu/machine/__init__.py
@@ -8,10 +8,6 @@
  - QEMUQtestMachine: VM class, with a qtest socket.
 
 - QEMUQtestProtocol: Connect to, send/receive qtest messages.
-
-- list_accel: List available accelerators
-- kvm_available: Probe for KVM support
-- tcg_available: Probe for TCG support
 """
 
 # Copyright (C) 2020-2021 John Snow for Red Hat Inc.
@@ -26,15 +22,11 @@
 # the COPYING file in the top-level directory.
 #
 
-from .accel import kvm_available, list_accel, tcg_available
 from .machine import QEMUMachine
 from .qtest import QEMUQtestMachine, QEMUQtestProtocol
 
 
 __all__ = (
-    'list_accel',
-    'kvm_available',
-    'tcg_available',
     'QEMUMachine',
     'QEMUQtestProtocol',
     'QEMUQtestMachine',
diff --git a/python/qemu/utils/__init__.py b/python/qemu/utils/__init__.py
new file mode 100644
index 00000000000..edf807a93e5
--- /dev/null
+++ b/python/qemu/utils/__init__.py
@@ -0,0 +1,23 @@
+"""
+QEMU development and testing utilities
+
+This library provides a small handful of utilities for performing various tasks
+not directly related to the launching of a VM.
+
+The only module included at present is accel; its public functions are
+repeated here for your convenience:
+
+- list_accel: List available accelerators
+- kvm_available: Probe for KVM support
+- tcg_available: Prove for TCG support
+"""
+
+# pylint: disable=import-error
+from .accel import kvm_available, list_accel, tcg_available
+
+
+__all__ = (
+    'list_accel',
+    'kvm_available',
+    'tcg_available',
+)
diff --git a/python/qemu/machine/accel.py b/python/qemu/utils/accel.py
similarity index 100%
rename from python/qemu/machine/accel.py
rename to python/qemu/utils/accel.py
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index ff7bf81f1a9..5f60892c2c4 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -41,8 +41,8 @@
 sys.path.append(os.path.join(SOURCE_DIR, 'python'))
 
 from qemu.machine import QEMUMachine
-from qemu.machine import kvm_available
-from qemu.machine import tcg_available
+from qemu.utils import kvm_available
+from qemu.utils import tcg_available
 
 def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
diff --git a/tests/acceptance/virtio-gpu.py b/tests/acceptance/virtio-gpu.py
index 0685e30bcae..e7979343e93 100644
--- a/tests/acceptance/virtio-gpu.py
+++ b/tests/acceptance/virtio-gpu.py
@@ -10,7 +10,7 @@
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import is_readable_executable_file
 
-from qemu.machine import kvm_available
+from qemu.utils import kvm_available
 
 import os
 import socket
diff --git a/tests/vm/aarch64vm.py b/tests/vm/aarch64vm.py
index 39ff99b0859..b00cce07eb8 100644
--- a/tests/vm/aarch64vm.py
+++ b/tests/vm/aarch64vm.py
@@ -14,7 +14,7 @@
 import sys
 import subprocess
 import basevm
-from qemu.machine import kvm_available
+from qemu.utils import kvm_available
 
 # This is the config needed for current version of QEMU.
 # This works for both kvm and tcg.
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 12d08cf2b1b..a3867fdf88e 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -19,7 +19,8 @@
 import time
 import datetime
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.machine import kvm_available, QEMUMachine
+from qemu.machine import QEMUMachine
+from qemu.utils import kvm_available
 import subprocess
 import hashlib
 import argparse
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 04/25] python: add qemu package installer
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (2 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 03/25] python: create utils sub-package John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-21  4:00   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 05/25] python: add VERSION file John Snow
                   ` (21 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Add setup.cfg and setup.py, necessary for installing a package via
pip. Add a ReST document (PACKAGE.rst) explaining the basics of what
this package is for and who to contact for more information. This
document will be used as the landing page for the package on PyPI.

I am not yet using a pyproject.toml style package manifest, because
"editable" installs are not defined (yet?) by PEP-517/518.

I consider editable installs crucial for development, though they have
(apparently) always been somewhat poorly defined.

Pip now (19.2 and later) now supports editable installs for projects
using pyproject.toml manifests, but might require the use of the
--no-use-pep517 flag, which somewhat defeats the point.

For now, while the dust settles, stick with the de-facto
setup.py/setup.cfg combination supported by setuptools. It will be worth
re-evaluating this point again in the future when our supported build
platforms all ship a fairly modern pip.

Additional reading on this matter:

https://github.com/pypa/packaging-problems/issues/256
https://github.com/pypa/pip/issues/6334
https://github.com/pypa/pip/issues/6375
https://github.com/pypa/pip/issues/6434
https://github.com/pypa/pip/issues/6438

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/PACKAGE.rst | 33 +++++++++++++++++++++++++++++++++
 python/setup.cfg   | 19 +++++++++++++++++++
 python/setup.py    | 23 +++++++++++++++++++++++
 3 files changed, 75 insertions(+)
 create mode 100644 python/PACKAGE.rst
 create mode 100644 python/setup.cfg
 create mode 100755 python/setup.py

diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
new file mode 100644
index 00000000000..1bbfe1b58e2
--- /dev/null
+++ b/python/PACKAGE.rst
@@ -0,0 +1,33 @@
+QEMU Python Tooling
+===================
+
+This package provides QEMU tooling used by the QEMU project to build,
+configure, and test QEMU. It is not a fully-fledged SDK and it is subject
+to change at any time.
+
+Usage
+-----
+
+The ``qemu.qmp`` subpackage provides a library for communicating with
+QMP servers. The ``qemu.machine`` subpackage offers rudimentary
+facilities for launching and managing QEMU processes. Refer to each
+package's documentation
+(``>>> help(qemu.qmp)``, ``>>> help(qemu.machine)``)
+for more information.
+
+Contributing
+------------
+
+This package is maintained by John Snow <jsnow@redhat.com> as part of
+the QEMU source tree. Contributions are welcome and follow the `QEMU
+patch submission process
+<https://wiki.qemu.org/Contribute/SubmitAPatch>`_, which involves
+sending patches to the QEMU development mailing list.
+
+John maintains a `GitLab staging branch
+<https://gitlab.com/jsnow/qemu/-/tree/python>`_, and there is an
+official `GitLab mirror <https://gitlab.com/qemu-project/qemu>`_.
+
+Please report bugs on the `QEMU issue tracker
+<https://gitlab.com/qemu-project/qemu/-/issues>`_ and tag ``@jsnow`` in
+the report.
diff --git a/python/setup.cfg b/python/setup.cfg
new file mode 100644
index 00000000000..dd71640fc2f
--- /dev/null
+++ b/python/setup.cfg
@@ -0,0 +1,19 @@
+[metadata]
+name = qemu
+maintainer = QEMU Developer Team
+maintainer_email = qemu-devel@nongnu.org
+url = https://www.qemu.org/
+download_url = https://www.qemu.org/download/
+description = QEMU Python Build, Debug and SDK tooling.
+long_description = file:PACKAGE.rst
+long_description_content_type = text/x-rst
+classifiers =
+    Development Status :: 3 - Alpha
+    License :: OSI Approved :: GNU General Public License v2 (GPLv2)
+    Natural Language :: English
+    Operating System :: OS Independent
+    Programming Language :: Python :: 3 :: Only
+
+[options]
+python_requires = >= 3.6
+packages = find_namespace:
diff --git a/python/setup.py b/python/setup.py
new file mode 100755
index 00000000000..2014f81b757
--- /dev/null
+++ b/python/setup.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+"""
+QEMU tooling installer script
+Copyright (c) 2020-2021 John Snow for Red Hat, Inc.
+"""
+
+import setuptools
+import pkg_resources
+
+
+def main():
+    """
+    QEMU tooling installer
+    """
+
+    # https://medium.com/@daveshawley/safely-using-setup-cfg-for-metadata-1babbe54c108
+    pkg_resources.require('setuptools>=39.2')
+
+    setuptools.setup()
+
+
+if __name__ == '__main__':
+    main()
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 05/25] python: add VERSION file
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (3 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 04/25] python: add qemu package installer John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 06/25] python: add directory structure README.rst files John Snow
                   ` (20 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Python infrastructure as it exists today is not capable reliably of
single-sourcing a package version from a parent directory. The authors
of pip are working to correct this, but as of today this is not possible.

The problem is that when using pip to build and install a python
package, it copies files over to a temporary directory and performs its
build there. This loses access to any information in the parent
directory, including git itself.

Further, Python versions have a standard (PEP 440) that may or may not
follow QEMU's versioning. In general, it does; but naturally QEMU does
not follow PEP 440. To avoid any automatically-generated conflict, a
manual version file is preferred.


I am proposing:

- Python tooling follows the QEMU version, indirectly, but with a major
  version of 0 to indicate that the API is not expected to be
  stable. This would mean version 0.5.2.0, 0.5.1.1, 0.5.3.0, etc.

- In the event that a Python package needs to be updated independently
  of the QEMU version, a pre-release alpha version should be preferred,
  but *only* after inclusion to the qemu development or stable branches.

  e.g. 0.5.2.0a1, 0.5.2.0a2, and so on should be preferred prior to
  5.2.0's release.

- The Python core tooling makes absolutely no version compatibility
  checks or constraints. It *may* work with releases of QEMU from the
  past or future, but it is not required to.

  i.e., "qemu.machine" will, for now, remain in lock-step with QEMU.

- We reserve the right to split the qemu package into independently
  versioned subpackages at a later date. This might allow for us to
  begin versioning QMP independently from QEMU at a later date, if
  we so choose.


Implement this versioning scheme by adding a VERSION file and setting it
to 0.6.0.0a1.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 python/VERSION   | 1 +
 python/setup.cfg | 1 +
 2 files changed, 2 insertions(+)
 create mode 100644 python/VERSION

diff --git a/python/VERSION b/python/VERSION
new file mode 100644
index 00000000000..c19f3b832b7
--- /dev/null
+++ b/python/VERSION
@@ -0,0 +1 @@
+0.6.1.0a1
diff --git a/python/setup.cfg b/python/setup.cfg
index dd71640fc2f..e7f8ab23815 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -1,5 +1,6 @@
 [metadata]
 name = qemu
+version = file:VERSION
 maintainer = QEMU Developer Team
 maintainer_email = qemu-devel@nongnu.org
 url = https://www.qemu.org/
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 06/25] python: add directory structure README.rst files
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (4 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 05/25] python: add VERSION file John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25  2:33   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 07/25] python: add MANIFEST.in John Snow
                   ` (19 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Add short readmes to python/, python/qemu/, python/qemu/machine,
python/qemu/qmp, and python/qemu/utils that explain the directory
hierarchy. These readmes are visible when browsing the source on
e.g. gitlab/github and are designed to help new developers/users quickly
make sense of the source tree.

They are not designed for inclusion in a published manual.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst              | 41 ++++++++++++++++++++++++++++++++++
 python/qemu/README.rst         |  8 +++++++
 python/qemu/machine/README.rst |  9 ++++++++
 python/qemu/qmp/README.rst     |  9 ++++++++
 python/qemu/utils/README.rst   |  7 ++++++
 5 files changed, 74 insertions(+)
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/qemu/machine/README.rst
 create mode 100644 python/qemu/qmp/README.rst
 create mode 100644 python/qemu/utils/README.rst

diff --git a/python/README.rst b/python/README.rst
new file mode 100644
index 00000000000..7a0dc5dff4a
--- /dev/null
+++ b/python/README.rst
@@ -0,0 +1,41 @@
+QEMU Python Tooling
+===================
+
+This directory houses Python tooling used by the QEMU project to build,
+configure, and test QEMU. It is organized by namespace (``qemu``), and
+then by package (e.g. ``qemu/machine``, ``qemu/qmp``, etc).
+
+``setup.py`` is used by ``pip`` to install this tooling to the current
+environment. ``setup.cfg`` provides the packaging configuration used by
+setup.py in a setuptools specific format. You will generally invoke it
+by doing one of the following:
+
+1. ``pip3 install .`` will install these packages to your current
+   environment. If you are inside a virtual environment, they will
+   install there. If you are not, it will attempt to install to the
+   global environment, which is not recommended.
+
+2. ``pip3 install --user .`` will install these packages to your user's
+   local python packages. If you are inside of a virtual environment,
+   this will fail.
+
+If you append the ``-e`` argument, pip will install in "editable" mode;
+which installs a version of the package that installs a forwarder
+pointing to these files, such that the package always reflects the
+latest version in your git tree.
+
+See `Installing packages using pip and virtual environments
+<https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/>`_
+for more information.
+
+
+Files in this directory
+-----------------------
+
+- ``qemu/`` Python package source directory.
+- ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
+- ``README.rst`` you are here!
+- ``VERSION`` contains the PEP-440 compliant version used to describe
+  this package; it is referenced by ``setup.cfg``.
+- ``setup.cfg`` houses setuptools package configuration.
+- ``setup.py`` is the setuptools installer used by pip; See above.
diff --git a/python/qemu/README.rst b/python/qemu/README.rst
new file mode 100644
index 00000000000..d04943f526c
--- /dev/null
+++ b/python/qemu/README.rst
@@ -0,0 +1,8 @@
+QEMU Python Namespace
+=====================
+
+This directory serves as the root of a `Python PEP 420 implicit
+namespace package <https://www.python.org/dev/peps/pep-0420/>`_.
+
+Each directory below is assumed to be an installable Python package that
+is available under the ``qemu.<package>`` namespace.
diff --git a/python/qemu/machine/README.rst b/python/qemu/machine/README.rst
new file mode 100644
index 00000000000..ac2b4fffb42
--- /dev/null
+++ b/python/qemu/machine/README.rst
@@ -0,0 +1,9 @@
+qemu.machine package
+====================
+
+This package provides core utilities used for testing and debugging
+QEMU. It is used by the iotests, vm tests, acceptance tests, and several
+other utilities in the ./scripts directory. It is not a fully-fledged
+SDK and it is subject to change at any time.
+
+See the documentation in ``__init__.py`` for more information.
diff --git a/python/qemu/qmp/README.rst b/python/qemu/qmp/README.rst
new file mode 100644
index 00000000000..c21951491cf
--- /dev/null
+++ b/python/qemu/qmp/README.rst
@@ -0,0 +1,9 @@
+qemu.qmp package
+================
+
+This package provides a library used for connecting to and communicating
+with QMP servers. It is used extensively by iotests, vm tests,
+acceptance tests, and other utilities in the ./scripts directory. It is
+not a fully-fledged SDK and is subject to change at any time.
+
+See the documentation in ``__init__.py`` for more information.
diff --git a/python/qemu/utils/README.rst b/python/qemu/utils/README.rst
new file mode 100644
index 00000000000..975fbf4d7de
--- /dev/null
+++ b/python/qemu/utils/README.rst
@@ -0,0 +1,7 @@
+qemu.utils package
+==================
+
+This package provides miscellaneous utilities used for testing and
+debugging QEMU. It is used primarily by the vm and acceptance tests.
+
+See the documentation in ``__init__.py`` for more information.
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 07/25] python: add MANIFEST.in
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (5 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 06/25] python: add directory structure README.rst files John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25  2:42   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 08/25] python: Add pipenv support John Snow
                   ` (18 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

When creating a source distribution via 'python3 setup.py sdist', the
VERSION and PACKAGE.rst files aren't bundled by default. Create a
MANIFEST.in file that instructs the build tools to include these so that
installation from source dists won't fail.

(This invocation is required by 'tox', as well as by the tooling needed
to upload packages to PyPI.)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst  | 2 ++
 python/MANIFEST.in | 2 ++
 2 files changed, 4 insertions(+)
 create mode 100644 python/MANIFEST.in

diff --git a/python/README.rst b/python/README.rst
index 7a0dc5dff4a..86364367261 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -33,6 +33,8 @@ Files in this directory
 -----------------------
 
 - ``qemu/`` Python package source directory.
+- ``MANIFEST.in`` is read by python setuptools, it specifies additional files
+  that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
 - ``README.rst`` you are here!
 - ``VERSION`` contains the PEP-440 compliant version used to describe
diff --git a/python/MANIFEST.in b/python/MANIFEST.in
new file mode 100644
index 00000000000..2b1ee8b4e72
--- /dev/null
+++ b/python/MANIFEST.in
@@ -0,0 +1,2 @@
+include VERSION
+include PACKAGE.rst
\ No newline at end of file
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 08/25] python: Add pipenv support
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (6 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 07/25] python: add MANIFEST.in John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 09/25] python: add pylint import exceptions John Snow
                   ` (17 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

pipenv is a tool used for managing virtual environments with pinned,
explicit dependencies. It is used for precisely recreating python
virtual environments.

pipenv uses two files to do this:

(1) Pipfile, which is similar in purpose and scope to what setup.cfg
lists. It specifies the requisite minimum to get a functional
environment for using this package.

(2) Pipfile.lock, which is similar in purpose to `pip freeze >
requirements.txt`. It specifies a canonical virtual environment used for
deployment or testing. This ensures that all users have repeatable
results.

The primary benefit of using this tool is to ensure *rock solid*
repeatable CI results with a known set of packages. Although I endeavor
to support as many versions as I can, the fluid nature of the Python
toolchain often means tailoring code for fairly specific versions.

Note that pipenv is *not* required to install or use this module; this is
purely for the sake of repeatable testing by CI or developers.

Here, a "blank" pipfile is added with no dependencies, but specifies
Python 3.6 for the virtual environment.

Pipfile will specify our version minimums, while Pipfile.lock specifies
an exact loadout of packages that were known to operate correctly. This
latter file provides the real value for easy setup of container images
and CI environments.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 python/README.rst |  3 +++
 python/Pipfile    | 11 +++++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 python/Pipfile

diff --git a/python/README.rst b/python/README.rst
index 86364367261..e27ba0130ba 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -36,6 +36,9 @@ Files in this directory
 - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
   that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
+- ``Pipfile`` is used by Pipenv to generate ``Pipfile.lock``.
+- ``Pipfile.lock`` is a set of pinned package dependencies that this package
+  is tested under in our CI suite. It is used by ``make venv-check``.
 - ``README.rst`` you are here!
 - ``VERSION`` contains the PEP-440 compliant version used to describe
   this package; it is referenced by ``setup.cfg``.
diff --git a/python/Pipfile b/python/Pipfile
new file mode 100644
index 00000000000..9534830b5eb
--- /dev/null
+++ b/python/Pipfile
@@ -0,0 +1,11 @@
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple"
+verify_ssl = true
+
+[dev-packages]
+
+[packages]
+
+[requires]
+python_version = "3.6"
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 09/25] python: add pylint import exceptions
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (7 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 08/25] python: Add pipenv support John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 10/25] python: move pylintrc into setup.cfg John Snow
                   ` (16 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Pylint 2.5.x - 2.7.x have regressions that make import checking
inconsistent, see:

https://github.com/PyCQA/pylint/issues/3609
https://github.com/PyCQA/pylint/issues/3624
https://github.com/PyCQA/pylint/issues/3651

Pinning to 2.4.4 is worse, because it mandates versions of shared
dependencies that are too old for features we want in isort and mypy.
Oh well.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/machine/__init__.py | 3 +++
 python/qemu/machine/machine.py  | 2 +-
 python/qemu/machine/qtest.py    | 2 +-
 3 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/python/qemu/machine/__init__.py b/python/qemu/machine/__init__.py
index 98302ea31e7..728f27adbed 100644
--- a/python/qemu/machine/__init__.py
+++ b/python/qemu/machine/__init__.py
@@ -22,6 +22,9 @@
 # the COPYING file in the top-level directory.
 #
 
+# pylint: disable=import-error
+# see: https://github.com/PyCQA/pylint/issues/3624
+# see: https://github.com/PyCQA/pylint/issues/3651
 from .machine import QEMUMachine
 from .qtest import QEMUQtestMachine, QEMUQtestProtocol
 
diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py
index dea343afeba..6fe41d83cb5 100644
--- a/python/qemu/machine/machine.py
+++ b/python/qemu/machine/machine.py
@@ -38,7 +38,7 @@
     Type,
 )
 
-from qemu.qmp import (
+from qemu.qmp import (  # pylint: disable=import-error
     QEMUMonitorProtocol,
     QMPMessage,
     QMPReturnValue,
diff --git a/python/qemu/machine/qtest.py b/python/qemu/machine/qtest.py
index 53926e434a7..c3adf4e3012 100644
--- a/python/qemu/machine/qtest.py
+++ b/python/qemu/machine/qtest.py
@@ -26,7 +26,7 @@
     TextIO,
 )
 
-from qemu.qmp import SocketAddrT
+from qemu.qmp import SocketAddrT  # pylint: disable=import-error
 
 from .machine import QEMUMachine
 
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 10/25] python: move pylintrc into setup.cfg
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (8 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 09/25] python: add pylint import exceptions John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 11/25] python: add pylint to pipenv John Snow
                   ` (15 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Delete the empty settings now that it's sharing a home with settings for
other tools.

pylint can now be run from this folder as "pylint qemu".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/machine/pylintrc | 58 ------------------------------------
 python/setup.cfg             | 29 ++++++++++++++++++
 2 files changed, 29 insertions(+), 58 deletions(-)
 delete mode 100644 python/qemu/machine/pylintrc

diff --git a/python/qemu/machine/pylintrc b/python/qemu/machine/pylintrc
deleted file mode 100644
index 3f69205000d..00000000000
--- a/python/qemu/machine/pylintrc
+++ /dev/null
@@ -1,58 +0,0 @@
-[MASTER]
-
-[MESSAGES CONTROL]
-
-# Disable the message, report, category or checker with the given id(s). You
-# can either give multiple identifiers separated by comma (,) or put this
-# option multiple times (only on the command line, not in the configuration
-# file where it should appear only once). You can also use "--disable=all" to
-# disable everything first and then reenable specific checks. For example, if
-# you want to run only the similarities checker, you can use "--disable=all
-# --enable=similarities". If you want to run only the classes checker, but have
-# no Warning level messages displayed, use "--disable=all --enable=classes
-# --disable=W".
-disable=too-many-arguments,
-        too-many-instance-attributes,
-        too-many-public-methods,
-
-[REPORTS]
-
-[REFACTORING]
-
-[MISCELLANEOUS]
-
-[LOGGING]
-
-[BASIC]
-
-# Good variable names which should always be accepted, separated by a comma.
-good-names=i,
-           j,
-           k,
-           ex,
-           Run,
-           _,
-           fd,
-           c,
-[VARIABLES]
-
-[STRING]
-
-[SPELLING]
-
-[FORMAT]
-
-[SIMILARITIES]
-
-# Ignore imports when computing similarities.
-ignore-imports=yes
-
-[TYPECHECK]
-
-[CLASSES]
-
-[IMPORTS]
-
-[DESIGN]
-
-[EXCEPTIONS]
diff --git a/python/setup.cfg b/python/setup.cfg
index e7f8ab23815..20b24372a4a 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -18,3 +18,32 @@ classifiers =
 [options]
 python_requires = >= 3.6
 packages = find_namespace:
+
+[pylint.messages control]
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=too-many-arguments,
+        too-many-instance-attributes,
+        too-many-public-methods,
+
+[pylint.basic]
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _,
+           fd,
+           c,
+
+[pylint.similarities]
+# Ignore imports when computing similarities.
+ignore-imports=yes
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 11/25] python: add pylint to pipenv
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (9 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 10/25] python: move pylintrc into setup.cfg John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25  3:33   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 12/25] python: move flake8 config to setup.cfg John Snow
                   ` (14 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

We are specifying >= pylint 2.7.x for several reasons:

1. For setup.cfg support, added in pylint 2.5.x
2. To specify a version that has incompatibly dropped
   bad-whitespace checks (2.6.x)
3. 2.7.x fixes "unsubscriptable" warnings in Python 3.9

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile      |   1 +
 python/Pipfile.lock | 130 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 131 insertions(+)
 create mode 100644 python/Pipfile.lock

diff --git a/python/Pipfile b/python/Pipfile
index 9534830b5eb..aaee00581eb 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -4,6 +4,7 @@ url = "https://pypi.org/simple"
 verify_ssl = true
 
 [dev-packages]
+pylint = ">=2.7.0"
 
 [packages]
 
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
new file mode 100644
index 00000000000..b5e556d265b
--- /dev/null
+++ b/python/Pipfile.lock
@@ -0,0 +1,130 @@
+{
+    "_meta": {
+        "hash": {
+            "sha256": "2016d2670f203ed4ffac9f59cdfbc540f209ad76eff28e484f245f401d524f5c"
+        },
+        "pipfile-spec": 6,
+        "requires": {
+            "python_version": "3.6"
+        },
+        "sources": [
+            {
+                "name": "pypi",
+                "url": "https://pypi.org/simple",
+                "verify_ssl": true
+            }
+        ]
+    },
+    "default": {},
+    "develop": {
+        "astroid": {
+            "hashes": [
+                "sha256:6b0ed1af831570e500e2437625979eaa3b36011f66ddfc4ce930128610258ca9",
+                "sha256:cd80bf957c49765dce6d92c43163ff9d2abc43132ce64d4b1b47717c6d2522df"
+            ],
+            "markers": "python_version >= '3.6'",
+            "version": "==2.5.2"
+        },
+        "isort": {
+            "hashes": [
+                "sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
+                "sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d"
+            ],
+            "markers": "python_version >= '3.6' and python_version < '4.0'",
+            "version": "==5.8.0"
+        },
+        "lazy-object-proxy": {
+            "hashes": [
+                "sha256:17e0967ba374fc24141738c69736da90e94419338fd4c7c7bef01ee26b339653",
+                "sha256:1fee665d2638491f4d6e55bd483e15ef21f6c8c2095f235fef72601021e64f61",
+                "sha256:22ddd618cefe54305df49e4c069fa65715be4ad0e78e8d252a33debf00f6ede2",
+                "sha256:24a5045889cc2729033b3e604d496c2b6f588c754f7a62027ad4437a7ecc4837",
+                "sha256:410283732af311b51b837894fa2f24f2c0039aa7f220135192b38fcc42bd43d3",
+                "sha256:4732c765372bd78a2d6b2150a6e99d00a78ec963375f236979c0626b97ed8e43",
+                "sha256:489000d368377571c6f982fba6497f2aa13c6d1facc40660963da62f5c379726",
+                "sha256:4f60460e9f1eb632584c9685bccea152f4ac2130e299784dbaf9fae9f49891b3",
+                "sha256:5743a5ab42ae40caa8421b320ebf3a998f89c85cdc8376d6b2e00bd12bd1b587",
+                "sha256:85fb7608121fd5621cc4377a8961d0b32ccf84a7285b4f1d21988b2eae2868e8",
+                "sha256:9698110e36e2df951c7c36b6729e96429c9c32b3331989ef19976592c5f3c77a",
+                "sha256:9d397bf41caad3f489e10774667310d73cb9c4258e9aed94b9ec734b34b495fd",
+                "sha256:b579f8acbf2bdd9ea200b1d5dea36abd93cabf56cf626ab9c744a432e15c815f",
+                "sha256:b865b01a2e7f96db0c5d12cfea590f98d8c5ba64ad222300d93ce6ff9138bcad",
+                "sha256:bf34e368e8dd976423396555078def5cfc3039ebc6fc06d1ae2c5a65eebbcde4",
+                "sha256:c6938967f8528b3668622a9ed3b31d145fab161a32f5891ea7b84f6b790be05b",
+                "sha256:d1c2676e3d840852a2de7c7d5d76407c772927addff8d742b9808fe0afccebdf",
+                "sha256:d7124f52f3bd259f510651450e18e0fd081ed82f3c08541dffc7b94b883aa981",
+                "sha256:d900d949b707778696fdf01036f58c9876a0d8bfe116e8d220cfd4b15f14e741",
+                "sha256:ebfd274dcd5133e0afae738e6d9da4323c3eb021b3e13052d8cbd0e457b1256e",
+                "sha256:ed361bb83436f117f9917d282a456f9e5009ea12fd6de8742d1a4752c3017e93",
+                "sha256:f5144c75445ae3ca2057faac03fda5a902eff196702b0a24daf1d6ce0650514b"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'",
+            "version": "==1.6.0"
+        },
+        "mccabe": {
+            "hashes": [
+                "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
+                "sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
+            ],
+            "version": "==0.6.1"
+        },
+        "pylint": {
+            "hashes": [
+                "sha256:0e21d3b80b96740909d77206d741aa3ce0b06b41be375d92e1f3244a274c1f8a",
+                "sha256:d09b0b07ba06bcdff463958f53f23df25e740ecd81895f7d2699ec04bbd8dc3b"
+            ],
+            "index": "pypi",
+            "version": "==2.7.2"
+        },
+        "toml": {
+            "hashes": [
+                "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
+                "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"
+            ],
+            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==0.10.2"
+        },
+        "typed-ast": {
+            "hashes": [
+                "sha256:07d49388d5bf7e863f7fa2f124b1b1d89d8aa0e2f7812faff0a5658c01c59aa1",
+                "sha256:14bf1522cdee369e8f5581238edac09150c765ec1cb33615855889cf33dcb92d",
+                "sha256:240296b27397e4e37874abb1df2a608a92df85cf3e2a04d0d4d61055c8305ba6",
+                "sha256:36d829b31ab67d6fcb30e185ec996e1f72b892255a745d3a82138c97d21ed1cd",
+                "sha256:37f48d46d733d57cc70fd5f30572d11ab8ed92da6e6b28e024e4a3edfb456e37",
+                "sha256:4c790331247081ea7c632a76d5b2a265e6d325ecd3179d06e9cf8d46d90dd151",
+                "sha256:5dcfc2e264bd8a1db8b11a892bd1647154ce03eeba94b461effe68790d8b8e07",
+                "sha256:7147e2a76c75f0f64c4319886e7639e490fee87c9d25cb1d4faef1d8cf83a440",
+                "sha256:7703620125e4fb79b64aa52427ec192822e9f45d37d4b6625ab37ef403e1df70",
+                "sha256:8368f83e93c7156ccd40e49a783a6a6850ca25b556c0fa0240ed0f659d2fe496",
+                "sha256:84aa6223d71012c68d577c83f4e7db50d11d6b1399a9c779046d75e24bed74ea",
+                "sha256:85f95aa97a35bdb2f2f7d10ec5bbdac0aeb9dafdaf88e17492da0504de2e6400",
+                "sha256:8db0e856712f79c45956da0c9a40ca4246abc3485ae0d7ecc86a20f5e4c09abc",
+                "sha256:9044ef2df88d7f33692ae3f18d3be63dec69c4fb1b5a4a9ac950f9b4ba571606",
+                "sha256:963c80b583b0661918718b095e02303d8078950b26cc00b5e5ea9ababe0de1fc",
+                "sha256:987f15737aba2ab5f3928c617ccf1ce412e2e321c77ab16ca5a293e7bbffd581",
+                "sha256:9ec45db0c766f196ae629e509f059ff05fc3148f9ffd28f3cfe75d4afb485412",
+                "sha256:9fc0b3cb5d1720e7141d103cf4819aea239f7d136acf9ee4a69b047b7986175a",
+                "sha256:a2c927c49f2029291fbabd673d51a2180038f8cd5a5b2f290f78c4516be48be2",
+                "sha256:a38878a223bdd37c9709d07cd357bb79f4c760b29210e14ad0fb395294583787",
+                "sha256:b4fcdcfa302538f70929eb7b392f536a237cbe2ed9cba88e3bf5027b39f5f77f",
+                "sha256:c0c74e5579af4b977c8b932f40a5464764b2f86681327410aa028a22d2f54937",
+                "sha256:c1c876fd795b36126f773db9cbb393f19808edd2637e00fd6caba0e25f2c7b64",
+                "sha256:c9aadc4924d4b5799112837b226160428524a9a45f830e0d0f184b19e4090487",
+                "sha256:cc7b98bf58167b7f2db91a4327da24fb93368838eb84a44c472283778fc2446b",
+                "sha256:cf54cfa843f297991b7388c281cb3855d911137223c6b6d2dd82a47ae5125a41",
+                "sha256:d003156bb6a59cda9050e983441b7fa2487f7800d76bdc065566b7d728b4581a",
+                "sha256:d175297e9533d8d37437abc14e8a83cbc68af93cc9c1c59c2c292ec59a0697a3",
+                "sha256:d746a437cdbca200622385305aedd9aef68e8a645e385cc483bdc5e488f07166",
+                "sha256:e683e409e5c45d5c9082dc1daf13f6374300806240719f95dc783d1fc942af10"
+            ],
+            "markers": "python_version < '3.8' and implementation_name == 'cpython'",
+            "version": "==1.4.2"
+        },
+        "wrapt": {
+            "hashes": [
+                "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
+            ],
+            "version": "==1.12.1"
+        }
+    }
+}
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 12/25] python: move flake8 config to setup.cfg
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (10 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 11/25] python: add pylint to pipenv John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 13/25] python: add excluded dirs to flake8 config John Snow
                   ` (13 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Update the comment concerning the flake8 exception to match commit
42c0dd12, whose commit message stated:

A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 python/qemu/machine/.flake8 | 2 --
 python/setup.cfg            | 3 +++
 2 files changed, 3 insertions(+), 2 deletions(-)
 delete mode 100644 python/qemu/machine/.flake8

diff --git a/python/qemu/machine/.flake8 b/python/qemu/machine/.flake8
deleted file mode 100644
index 45d8146f3f5..00000000000
--- a/python/qemu/machine/.flake8
+++ /dev/null
@@ -1,2 +0,0 @@
-[flake8]
-extend-ignore = E722  # Pylint handles this, but smarter.
\ No newline at end of file
diff --git a/python/setup.cfg b/python/setup.cfg
index 20b24372a4a..9ecb2902006 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -19,6 +19,9 @@ classifiers =
 python_requires = >= 3.6
 packages = find_namespace:
 
+[flake8]
+extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 13/25] python: add excluded dirs to flake8 config
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (11 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 12/25] python: move flake8 config to setup.cfg John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 15:50   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 14/25] python: Add flake8 to pipenv John Snow
                   ` (12 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Instruct flake8 to avoid certain well-known directories created by
python tooling that it ought not check.

Note that at-present, nothing actually creates a ".venv" directory; but
it is in such widespread usage as a de-facto location for a developer's
virtual environment that it should be excluded anyway. A forthcoming
commit canonizes this with a "make venv" command.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/setup.cfg | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/python/setup.cfg b/python/setup.cfg
index 9ecb2902006..f21a1c42fc0 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -21,6 +21,8 @@ packages = find_namespace:
 
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
+exclude = __pycache__,
+          .venv,
 
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 14/25] python: Add flake8 to pipenv
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (12 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 13/25] python: add excluded dirs to flake8 config John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 15/25] python: move mypy.ini into setup.cfg John Snow
                   ` (11 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

flake8 3.5.x does not support the --extend-ignore syntax used in the
.flake8 file to gracefully extend default ignores, so 3.6.x is our
minimum requirement. There is no known upper bound.

flake8 can be run from the python/ directory with no arguments.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 python/Pipfile      |  1 +
 python/Pipfile.lock | 55 ++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 53 insertions(+), 3 deletions(-)

diff --git a/python/Pipfile b/python/Pipfile
index aaee00581eb..cc43445ef8f 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -4,6 +4,7 @@ url = "https://pypi.org/simple"
 verify_ssl = true
 
 [dev-packages]
+flake8 = ">=3.6.0"
 pylint = ">=2.7.0"
 
 [packages]
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index b5e556d265b..dd2fafadd61 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "2016d2670f203ed4ffac9f59cdfbc540f209ad76eff28e484f245f401d524f5c"
+            "sha256": "4a4a9c6e09d9c5ce670fe4d067052a743dc3eff93c4790f9da31a0db2a66f8fc"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -25,6 +25,22 @@
             "markers": "python_version >= '3.6'",
             "version": "==2.5.2"
         },
+        "flake8": {
+            "hashes": [
+                "sha256:12d05ab02614b6aee8df7c36b97d1a3b2372761222b19b58621355e82acddcff",
+                "sha256:78873e372b12b093da7b5e5ed302e8ad9e988b38b063b61ad937f26ca58fc5f0"
+            ],
+            "index": "pypi",
+            "version": "==3.9.0"
+        },
+        "importlib-metadata": {
+            "hashes": [
+                "sha256:1cedf994a9b6885dcbb7ed40b24c332b1de3956319f4b1a0f07c0621d453accc",
+                "sha256:c9c1b6c7dbc62084f3e6a614a194eb16ded7947736c18e3300125d5c0a7a8b3c"
+            ],
+            "markers": "python_version < '3.8'",
+            "version": "==3.9.1"
+        },
         "isort": {
             "hashes": [
                 "sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
@@ -68,6 +84,22 @@
             ],
             "version": "==0.6.1"
         },
+        "pycodestyle": {
+            "hashes": [
+                "sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068",
+                "sha256:c389c1d06bf7904078ca03399a4816f974a1d590090fecea0c63ec26ebaf1cef"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==2.7.0"
+        },
+        "pyflakes": {
+            "hashes": [
+                "sha256:7893783d01b8a89811dd72d7dfd4d84ff098e5eed95cfa8905b22bbffe52efc3",
+                "sha256:f5bc8ecabc05bb9d291eb5203d6810b49040f6ff446a756326104746cc00c1db"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==2.3.1"
+        },
         "pylint": {
             "hashes": [
                 "sha256:0e21d3b80b96740909d77206d741aa3ce0b06b41be375d92e1f3244a274c1f8a",
@@ -81,7 +113,7 @@
                 "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
                 "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"
             ],
-            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2'",
             "version": "==0.10.2"
         },
         "typed-ast": {
@@ -117,14 +149,31 @@
                 "sha256:d746a437cdbca200622385305aedd9aef68e8a645e385cc483bdc5e488f07166",
                 "sha256:e683e409e5c45d5c9082dc1daf13f6374300806240719f95dc783d1fc942af10"
             ],
-            "markers": "python_version < '3.8' and implementation_name == 'cpython'",
+            "markers": "implementation_name == 'cpython' and python_version < '3.8'",
             "version": "==1.4.2"
         },
+        "typing-extensions": {
+            "hashes": [
+                "sha256:7cb407020f00f7bfc3cb3e7881628838e69d8f3fcab2f64742a5e76b2f841918",
+                "sha256:99d4073b617d30288f569d3f13d2bd7548c3a7e4c8de87db09a9d29bb3a4a60c",
+                "sha256:dafc7639cde7f1b6e1acc0f457842a83e722ccca8eef5270af2d74792619a89f"
+            ],
+            "markers": "python_version < '3.8'",
+            "version": "==3.7.4.3"
+        },
         "wrapt": {
             "hashes": [
                 "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
             ],
             "version": "==1.12.1"
+        },
+        "zipp": {
+            "hashes": [
+                "sha256:3607921face881ba3e026887d8150cca609d517579abe052ac81fc5aeffdbd76",
+                "sha256:51cb66cc54621609dd593d1787f286ee42a5c0adbb4b29abea5a63edc3e03098"
+            ],
+            "markers": "python_version >= '3.6'",
+            "version": "==3.4.1"
         }
     }
 }
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 15/25] python: move mypy.ini into setup.cfg
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (13 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 14/25] python: Add flake8 to pipenv John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 15:52   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 16/25] python: add mypy to pipenv John Snow
                   ` (10 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

mypy supports reading its configuration values from a central project
configuration file; do so.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/mypy.ini  | 4 ----
 python/setup.cfg | 5 +++++
 2 files changed, 5 insertions(+), 4 deletions(-)
 delete mode 100644 python/mypy.ini

diff --git a/python/mypy.ini b/python/mypy.ini
deleted file mode 100644
index 1a581c5f1ea..00000000000
--- a/python/mypy.ini
+++ /dev/null
@@ -1,4 +0,0 @@
-[mypy]
-strict = True
-python_version = 3.6
-warn_unused_configs = True
diff --git a/python/setup.cfg b/python/setup.cfg
index f21a1c42fc0..a3fcf67f8d2 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -24,6 +24,11 @@ extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
 exclude = __pycache__,
           .venv,
 
+[mypy]
+strict = True
+python_version = 3.6
+warn_unused_configs = True
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 16/25] python: add mypy to pipenv
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (14 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 15/25] python: move mypy.ini into setup.cfg John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 17/25] python: move .isort.cfg into setup.cfg John Snow
                   ` (9 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

0.730 appears to be about the oldest version that works with the
features we want, including nice human readable output (to make sure
iotest 297 passes), and type-parameterized Popen generics.

0.770, however, supports adding 'strict' to the config file, so require
at least 0.770.

Now that we are checking a namespace package, we need to tell mypy to
allow PEP420 namespaces, so modify the mypy config as part of the move.

mypy can now be run from the python root by typing 'mypy -p qemu'.

A note on mypy invocation: Running it as "mypy qemu/" changes the import
path detection mechanisms in mypy slightly, and it will fail. See
https://github.com/python/mypy/issues/8584 for a decent entry point with
more breadcrumbs on the various behaviors that contribute to this subtle
difference.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 python/Pipfile      |  1 +
 python/Pipfile.lock | 37 ++++++++++++++++++++++++++++++++++++-
 python/setup.cfg    |  1 +
 3 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/python/Pipfile b/python/Pipfile
index cc43445ef8f..bb2e34b381d 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -5,6 +5,7 @@ verify_ssl = true
 
 [dev-packages]
 flake8 = ">=3.6.0"
+mypy = ">=0.770"
 pylint = ">=2.7.0"
 
 [packages]
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index dd2fafadd61..2b1567b359c 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "4a4a9c6e09d9c5ce670fe4d067052a743dc3eff93c4790f9da31a0db2a66f8fc"
+            "sha256": "5e5b76eb2f6d8e833b79d9f083f08e0087b3e663275a00b84a6799419aa8a776"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -84,6 +84,41 @@
             ],
             "version": "==0.6.1"
         },
+        "mypy": {
+            "hashes": [
+                "sha256:0d0a87c0e7e3a9becdfbe936c981d32e5ee0ccda3e0f07e1ef2c3d1a817cf73e",
+                "sha256:25adde9b862f8f9aac9d2d11971f226bd4c8fbaa89fb76bdadb267ef22d10064",
+                "sha256:28fb5479c494b1bab244620685e2eb3c3f988d71fd5d64cc753195e8ed53df7c",
+                "sha256:2f9b3407c58347a452fc0736861593e105139b905cca7d097e413453a1d650b4",
+                "sha256:33f159443db0829d16f0a8d83d94df3109bb6dd801975fe86bacb9bf71628e97",
+                "sha256:3f2aca7f68580dc2508289c729bd49ee929a436208d2b2b6aab15745a70a57df",
+                "sha256:499c798053cdebcaa916eef8cd733e5584b5909f789de856b482cd7d069bdad8",
+                "sha256:4eec37370483331d13514c3f55f446fc5248d6373e7029a29ecb7b7494851e7a",
+                "sha256:552a815579aa1e995f39fd05dde6cd378e191b063f031f2acfe73ce9fb7f9e56",
+                "sha256:5873888fff1c7cf5b71efbe80e0e73153fe9212fafdf8e44adfe4c20ec9f82d7",
+                "sha256:61a3d5b97955422964be6b3baf05ff2ce7f26f52c85dd88db11d5e03e146a3a6",
+                "sha256:674e822aa665b9fd75130c6c5f5ed9564a38c6cea6a6432ce47eafb68ee578c5",
+                "sha256:7ce3175801d0ae5fdfa79b4f0cfed08807af4d075b402b7e294e6aa72af9aa2a",
+                "sha256:9743c91088d396c1a5a3c9978354b61b0382b4e3c440ce83cf77994a43e8c521",
+                "sha256:9f94aac67a2045ec719ffe6111df543bac7874cee01f41928f6969756e030564",
+                "sha256:a26f8ec704e5a7423c8824d425086705e381b4f1dfdef6e3a1edab7ba174ec49",
+                "sha256:abf7e0c3cf117c44d9285cc6128856106183938c68fd4944763003decdcfeb66",
+                "sha256:b09669bcda124e83708f34a94606e01b614fa71931d356c1f1a5297ba11f110a",
+                "sha256:cd07039aa5df222037005b08fbbfd69b3ab0b0bd7a07d7906de75ae52c4e3119",
+                "sha256:d23e0ea196702d918b60c8288561e722bf437d82cb7ef2edcd98cfa38905d506",
+                "sha256:d65cc1df038ef55a99e617431f0553cd77763869eebdf9042403e16089fe746c",
+                "sha256:d7da2e1d5f558c37d6e8c1246f1aec1e7349e4913d8fb3cb289a35de573fe2eb"
+            ],
+            "index": "pypi",
+            "version": "==0.812"
+        },
+        "mypy-extensions": {
+            "hashes": [
+                "sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d",
+                "sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"
+            ],
+            "version": "==0.4.3"
+        },
         "pycodestyle": {
             "hashes": [
                 "sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068",
diff --git a/python/setup.cfg b/python/setup.cfg
index a3fcf67f8d2..cdd8477292b 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -28,6 +28,7 @@ exclude = __pycache__,
 strict = True
 python_version = 3.6
 warn_unused_configs = True
+namespace_packages = True
 
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 17/25] python: move .isort.cfg into setup.cfg
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (15 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 16/25] python: add mypy to pipenv John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 18/25] python/qemu: add isort to pipenv John Snow
                   ` (8 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 python/.isort.cfg | 7 -------
 python/setup.cfg  | 8 ++++++++
 2 files changed, 8 insertions(+), 7 deletions(-)
 delete mode 100644 python/.isort.cfg

diff --git a/python/.isort.cfg b/python/.isort.cfg
deleted file mode 100644
index 6d0fd6cc0d3..00000000000
--- a/python/.isort.cfg
+++ /dev/null
@@ -1,7 +0,0 @@
-[settings]
-force_grid_wrap=4
-force_sort_within_sections=True
-include_trailing_comma=True
-line_length=72
-lines_after_imports=2
-multi_line_output=3
\ No newline at end of file
diff --git a/python/setup.cfg b/python/setup.cfg
index cdd8477292b..8a2c200a581 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -58,3 +58,11 @@ good-names=i,
 [pylint.similarities]
 # Ignore imports when computing similarities.
 ignore-imports=yes
+
+[isort]
+force_grid_wrap=4
+force_sort_within_sections=True
+include_trailing_comma=True
+line_length=72
+lines_after_imports=2
+multi_line_output=3
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 18/25] python/qemu: add isort to pipenv
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (16 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 17/25] python: move .isort.cfg into setup.cfg John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 15:56   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 19/25] python/qemu: add qemu package itself " John Snow
                   ` (7 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
certain "from ..." clauses that are not related to imports.

isort < 5.1.1 has a bug where it does not handle comments near import
statements correctly.

Require 5.1.2 or greater.

isort can be run with 'isort -c qemu' from the python root.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile      | 1 +
 python/Pipfile.lock | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/python/Pipfile b/python/Pipfile
index bb2e34b381d..fb7c8d142ee 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -5,6 +5,7 @@ verify_ssl = true
 
 [dev-packages]
 flake8 = ">=3.6.0"
+isort = ">=5.1.2"
 mypy = ">=0.770"
 pylint = ">=2.7.0"
 
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 2b1567b359c..030d5683147 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "5e5b76eb2f6d8e833b79d9f083f08e0087b3e663275a00b84a6799419aa8a776"
+            "sha256": "e1f54e4d7fc287bdff731659614f5f9bbea2f1aee122ba04506ab26c4007440e"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -46,7 +46,7 @@
                 "sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
                 "sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d"
             ],
-            "markers": "python_version >= '3.6' and python_version < '4.0'",
+            "index": "pypi",
             "version": "==5.8.0"
         },
         "lazy-object-proxy": {
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 19/25] python/qemu: add qemu package itself to pipenv
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (17 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 18/25] python/qemu: add isort to pipenv John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-12 23:12 ` [PATCH v6 20/25] python: add devel package requirements to setuptools John Snow
                   ` (6 subsequent siblings)
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

This adds the python qemu packages themselves to the pipenv manifest.
'pipenv sync' will create a virtual environment sufficient to use the SDK.
'pipenv sync --dev' will create a virtual environment sufficient to use
and test the SDK (with pylint, mypy, isort, flake8, etc.)

The qemu packages are installed in 'editable' mode; all changes made to
the python package inside the git tree will be reflected in the
installed package without reinstallation. This includes changes made
via git pull and so on.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 python/Pipfile      | 1 +
 python/Pipfile.lock | 9 +++++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/python/Pipfile b/python/Pipfile
index fb7c8d142ee..fb23455eadd 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -10,6 +10,7 @@ mypy = ">=0.770"
 pylint = ">=2.7.0"
 
 [packages]
+qemu = {editable = true,path = "."}
 
 [requires]
 python_version = "3.6"
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 030d5683147..5d3de43609d 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "e1f54e4d7fc287bdff731659614f5f9bbea2f1aee122ba04506ab26c4007440e"
+            "sha256": "986164b4c690953890066f288b48c3d84c63df86fc8fa30a26e9001d5b0968e0"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -15,7 +15,12 @@
             }
         ]
     },
-    "default": {},
+    "default": {
+        "qemu": {
+            "editable": true,
+            "path": "."
+        }
+    },
     "develop": {
         "astroid": {
             "hashes": [
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 20/25] python: add devel package requirements to setuptools
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (18 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 19/25] python/qemu: add qemu package itself " John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 16:13   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 21/25] python: add avocado-framework and tests John Snow
                   ` (5 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

setuptools doesn't have a formal understanding of development requires,
but it has an optional feataures section. Fine; add a "devel" feature
and add the requirements to it.

To avoid duplication, we can modify pipenv to install qemu[devel]
instead. This enables us to run invocations like "pip install -e
.[devel]" and test the package on bleeding-edge packages beyond those
specified in Pipfile.lock.

Importantly, this also allows us to install the qemu development
packages in a non-networked mode: `pip3 install --no-index -e .[devel]`
will now fail if the proper development dependencies are not already
met. This can be useful for automated build scripts where fetching
network packages may be undesirable.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile      |  5 +----
 python/Pipfile.lock | 14 +++++++++-----
 python/setup.cfg    |  9 +++++++++
 3 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/python/Pipfile b/python/Pipfile
index fb23455eadd..e7acb8cefa4 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -4,10 +4,7 @@ url = "https://pypi.org/simple"
 verify_ssl = true
 
 [dev-packages]
-flake8 = ">=3.6.0"
-isort = ">=5.1.2"
-mypy = ">=0.770"
-pylint = ">=2.7.0"
+qemu = {editable = true, extras = ["devel"], path = "."}
 
 [packages]
 qemu = {editable = true,path = "."}
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 5d3de43609d..18f3bba08f2 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "986164b4c690953890066f288b48c3d84c63df86fc8fa30a26e9001d5b0968e0"
+            "sha256": "eff562a688ebc6f3ffe67494dbb804b883e2159ad81c4d55d96da9f7aec13e91"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -35,7 +35,7 @@
                 "sha256:12d05ab02614b6aee8df7c36b97d1a3b2372761222b19b58621355e82acddcff",
                 "sha256:78873e372b12b093da7b5e5ed302e8ad9e988b38b063b61ad937f26ca58fc5f0"
             ],
-            "index": "pypi",
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
             "version": "==3.9.0"
         },
         "importlib-metadata": {
@@ -51,7 +51,7 @@
                 "sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
                 "sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d"
             ],
-            "index": "pypi",
+            "markers": "python_version >= '3.6' and python_version < '4.0'",
             "version": "==5.8.0"
         },
         "lazy-object-proxy": {
@@ -114,7 +114,7 @@
                 "sha256:d65cc1df038ef55a99e617431f0553cd77763869eebdf9042403e16089fe746c",
                 "sha256:d7da2e1d5f558c37d6e8c1246f1aec1e7349e4913d8fb3cb289a35de573fe2eb"
             ],
-            "index": "pypi",
+            "markers": "python_version >= '3.5'",
             "version": "==0.812"
         },
         "mypy-extensions": {
@@ -145,9 +145,13 @@
                 "sha256:0e21d3b80b96740909d77206d741aa3ce0b06b41be375d92e1f3244a274c1f8a",
                 "sha256:d09b0b07ba06bcdff463958f53f23df25e740ecd81895f7d2699ec04bbd8dc3b"
             ],
-            "index": "pypi",
+            "markers": "python_version ~= '3.6'",
             "version": "==2.7.2"
         },
+        "qemu": {
+            "editable": true,
+            "path": "."
+        },
         "toml": {
             "hashes": [
                 "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
diff --git a/python/setup.cfg b/python/setup.cfg
index 8a2c200a581..9d941386921 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -19,6 +19,15 @@ classifiers =
 python_requires = >= 3.6
 packages = find_namespace:
 
+[options.extras_require]
+# Run `pipenv lock` when changing these requirements.
+devel =
+    flake8 >= 3.6.0
+    isort >= 5.1.2
+    mypy >= 0.770
+    pylint >= 2.7.0
+
+
 [flake8]
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
 exclude = __pycache__,
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 21/25] python: add avocado-framework and tests
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (19 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 20/25] python: add devel package requirements to setuptools John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 18:58   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 22/25] python: add Makefile for some common tasks John Snow
                   ` (4 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Try using avocado to manage our various tests; even though right now
they're only invoking shell scripts and not really running any
python-native code.

Create tests/, and add shell scripts which call out to mypy, flake8,
pylint and isort to enforce the standards in this directory.

Add avocado-framework to the setup.cfg development dependencies, and add
avocado.cfg to store some preferences for how we'd like the test output
to look.

Finally, add avocado-framework to the Pipfile environment and lock the
new dependencies. We are using avocado >= 87.0 here to take advantage of
some features that Cleber has helpfully added to make the test output
here *very* friendly and easy to read for developers that might chance
upon the output in Gitlab CI.

[Note: ALL of the dependencies get updated to the most modern versions
that exist at the time of this writing. No way around it that I have
seen. Not ideal, but so it goes.]

Provided you have the right development dependencies (mypy, flake8,
isort, pylint, and now avocado-framework) You should be able to run
"avocado --config avocado.cfg run tests/" from the python folder to run
all of these linters with the correct arguments.

(A forthcoming commit adds the much easier 'make check'.)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst      |   2 +
 python/Pipfile.lock    | 104 ++++++++++++++++++++++-------------------
 python/avocado.cfg     |  10 ++++
 python/setup.cfg       |   1 +
 python/tests/flake8.sh |   2 +
 python/tests/isort.sh  |   2 +
 python/tests/mypy.sh   |   2 +
 python/tests/pylint.sh |   2 +
 8 files changed, 77 insertions(+), 48 deletions(-)
 create mode 100644 python/avocado.cfg
 create mode 100755 python/tests/flake8.sh
 create mode 100755 python/tests/isort.sh
 create mode 100755 python/tests/mypy.sh
 create mode 100755 python/tests/pylint.sh

diff --git a/python/README.rst b/python/README.rst
index e27ba0130ba..e107bd12a69 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -33,6 +33,8 @@ Files in this directory
 -----------------------
 
 - ``qemu/`` Python package source directory.
+- ``tests/`` Python package tests directory.
+- ``avocado.cfg`` Configuration for the Avocado test-runner.
 - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
   that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 18f3bba08f2..2995ede77cd 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -24,27 +24,35 @@
     "develop": {
         "astroid": {
             "hashes": [
-                "sha256:6b0ed1af831570e500e2437625979eaa3b36011f66ddfc4ce930128610258ca9",
-                "sha256:cd80bf957c49765dce6d92c43163ff9d2abc43132ce64d4b1b47717c6d2522df"
+                "sha256:4db03ab5fc3340cf619dbc25e42c2cc3755154ce6009469766d7143d1fc2ee4e",
+                "sha256:8a398dfce302c13f14bab13e2b14fe385d32b73f4e4853b9bdfb64598baa1975"
             ],
-            "markers": "python_version >= '3.6'",
-            "version": "==2.5.2"
+            "markers": "python_version ~= '3.6'",
+            "version": "==2.5.6"
+        },
+        "avocado-framework": {
+            "hashes": [
+                "sha256:3fca7226d7d164f124af8a741e7fa658ff4345a0738ddc32907631fd688b38ed",
+                "sha256:48ac254c0ae2ef0c0ceeb38e3d3df0388718eda8f48b3ab55b30b252839f42b1"
+            ],
+            "markers": "python_version >= '3.5'",
+            "version": "==87.0"
         },
         "flake8": {
             "hashes": [
-                "sha256:12d05ab02614b6aee8df7c36b97d1a3b2372761222b19b58621355e82acddcff",
-                "sha256:78873e372b12b093da7b5e5ed302e8ad9e988b38b063b61ad937f26ca58fc5f0"
+                "sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b",
+                "sha256:bf8fd333346d844f616e8d47905ef3a3384edae6b4e9beb0c5101e25e3110907"
             ],
             "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
-            "version": "==3.9.0"
+            "version": "==3.9.2"
         },
         "importlib-metadata": {
             "hashes": [
-                "sha256:1cedf994a9b6885dcbb7ed40b24c332b1de3956319f4b1a0f07c0621d453accc",
-                "sha256:c9c1b6c7dbc62084f3e6a614a194eb16ded7947736c18e3300125d5c0a7a8b3c"
+                "sha256:8c501196e49fb9df5df43833bdb1e4328f64847763ec8a50703148b73784d581",
+                "sha256:d7eb1dea6d6a6086f8be21784cc9e3bcfa55872b52309bc5fad53a8ea444465d"
             ],
             "markers": "python_version < '3.8'",
-            "version": "==3.9.1"
+            "version": "==4.0.1"
         },
         "isort": {
             "hashes": [
@@ -142,11 +150,11 @@
         },
         "pylint": {
             "hashes": [
-                "sha256:0e21d3b80b96740909d77206d741aa3ce0b06b41be375d92e1f3244a274c1f8a",
-                "sha256:d09b0b07ba06bcdff463958f53f23df25e740ecd81895f7d2699ec04bbd8dc3b"
+                "sha256:586d8fa9b1891f4b725f587ef267abe2a1bad89d6b184520c7f07a253dd6e217",
+                "sha256:f7e2072654a6b6afdf5e2fb38147d3e2d2d43c89f648637baab63e026481279b"
             ],
             "markers": "python_version ~= '3.6'",
-            "version": "==2.7.2"
+            "version": "==2.8.2"
         },
         "qemu": {
             "editable": true,
@@ -162,48 +170,48 @@
         },
         "typed-ast": {
             "hashes": [
-                "sha256:07d49388d5bf7e863f7fa2f124b1b1d89d8aa0e2f7812faff0a5658c01c59aa1",
-                "sha256:14bf1522cdee369e8f5581238edac09150c765ec1cb33615855889cf33dcb92d",
-                "sha256:240296b27397e4e37874abb1df2a608a92df85cf3e2a04d0d4d61055c8305ba6",
-                "sha256:36d829b31ab67d6fcb30e185ec996e1f72b892255a745d3a82138c97d21ed1cd",
-                "sha256:37f48d46d733d57cc70fd5f30572d11ab8ed92da6e6b28e024e4a3edfb456e37",
-                "sha256:4c790331247081ea7c632a76d5b2a265e6d325ecd3179d06e9cf8d46d90dd151",
-                "sha256:5dcfc2e264bd8a1db8b11a892bd1647154ce03eeba94b461effe68790d8b8e07",
-                "sha256:7147e2a76c75f0f64c4319886e7639e490fee87c9d25cb1d4faef1d8cf83a440",
-                "sha256:7703620125e4fb79b64aa52427ec192822e9f45d37d4b6625ab37ef403e1df70",
-                "sha256:8368f83e93c7156ccd40e49a783a6a6850ca25b556c0fa0240ed0f659d2fe496",
-                "sha256:84aa6223d71012c68d577c83f4e7db50d11d6b1399a9c779046d75e24bed74ea",
-                "sha256:85f95aa97a35bdb2f2f7d10ec5bbdac0aeb9dafdaf88e17492da0504de2e6400",
-                "sha256:8db0e856712f79c45956da0c9a40ca4246abc3485ae0d7ecc86a20f5e4c09abc",
-                "sha256:9044ef2df88d7f33692ae3f18d3be63dec69c4fb1b5a4a9ac950f9b4ba571606",
-                "sha256:963c80b583b0661918718b095e02303d8078950b26cc00b5e5ea9ababe0de1fc",
-                "sha256:987f15737aba2ab5f3928c617ccf1ce412e2e321c77ab16ca5a293e7bbffd581",
-                "sha256:9ec45db0c766f196ae629e509f059ff05fc3148f9ffd28f3cfe75d4afb485412",
-                "sha256:9fc0b3cb5d1720e7141d103cf4819aea239f7d136acf9ee4a69b047b7986175a",
-                "sha256:a2c927c49f2029291fbabd673d51a2180038f8cd5a5b2f290f78c4516be48be2",
-                "sha256:a38878a223bdd37c9709d07cd357bb79f4c760b29210e14ad0fb395294583787",
-                "sha256:b4fcdcfa302538f70929eb7b392f536a237cbe2ed9cba88e3bf5027b39f5f77f",
-                "sha256:c0c74e5579af4b977c8b932f40a5464764b2f86681327410aa028a22d2f54937",
-                "sha256:c1c876fd795b36126f773db9cbb393f19808edd2637e00fd6caba0e25f2c7b64",
-                "sha256:c9aadc4924d4b5799112837b226160428524a9a45f830e0d0f184b19e4090487",
-                "sha256:cc7b98bf58167b7f2db91a4327da24fb93368838eb84a44c472283778fc2446b",
-                "sha256:cf54cfa843f297991b7388c281cb3855d911137223c6b6d2dd82a47ae5125a41",
-                "sha256:d003156bb6a59cda9050e983441b7fa2487f7800d76bdc065566b7d728b4581a",
-                "sha256:d175297e9533d8d37437abc14e8a83cbc68af93cc9c1c59c2c292ec59a0697a3",
-                "sha256:d746a437cdbca200622385305aedd9aef68e8a645e385cc483bdc5e488f07166",
-                "sha256:e683e409e5c45d5c9082dc1daf13f6374300806240719f95dc783d1fc942af10"
+                "sha256:01ae5f73431d21eead5015997ab41afa53aa1fbe252f9da060be5dad2c730ace",
+                "sha256:067a74454df670dcaa4e59349a2e5c81e567d8d65458d480a5b3dfecec08c5ff",
+                "sha256:0fb71b8c643187d7492c1f8352f2c15b4c4af3f6338f21681d3681b3dc31a266",
+                "sha256:1b3ead4a96c9101bef08f9f7d1217c096f31667617b58de957f690c92378b528",
+                "sha256:2068531575a125b87a41802130fa7e29f26c09a2833fea68d9a40cf33902eba6",
+                "sha256:209596a4ec71d990d71d5e0d312ac935d86930e6eecff6ccc7007fe54d703808",
+                "sha256:2c726c276d09fc5c414693a2de063f521052d9ea7c240ce553316f70656c84d4",
+                "sha256:398e44cd480f4d2b7ee8d98385ca104e35c81525dd98c519acff1b79bdaac363",
+                "sha256:52b1eb8c83f178ab787f3a4283f68258525f8d70f778a2f6dd54d3b5e5fb4341",
+                "sha256:5feca99c17af94057417d744607b82dd0a664fd5e4ca98061480fd8b14b18d04",
+                "sha256:7538e495704e2ccda9b234b82423a4038f324f3a10c43bc088a1636180f11a41",
+                "sha256:760ad187b1041a154f0e4d0f6aae3e40fdb51d6de16e5c99aedadd9246450e9e",
+                "sha256:777a26c84bea6cd934422ac2e3b78863a37017618b6e5c08f92ef69853e765d3",
+                "sha256:95431a26309a21874005845c21118c83991c63ea800dd44843e42a916aec5899",
+                "sha256:9ad2c92ec681e02baf81fdfa056fe0d818645efa9af1f1cd5fd6f1bd2bdfd805",
+                "sha256:9c6d1a54552b5330bc657b7ef0eae25d00ba7ffe85d9ea8ae6540d2197a3788c",
+                "sha256:aee0c1256be6c07bd3e1263ff920c325b59849dc95392a05f258bb9b259cf39c",
+                "sha256:af3d4a73793725138d6b334d9d247ce7e5f084d96284ed23f22ee626a7b88e39",
+                "sha256:b36b4f3920103a25e1d5d024d155c504080959582b928e91cb608a65c3a49e1a",
+                "sha256:b9574c6f03f685070d859e75c7f9eeca02d6933273b5e69572e5ff9d5e3931c3",
+                "sha256:bff6ad71c81b3bba8fa35f0f1921fb24ff4476235a6e94a26ada2e54370e6da7",
+                "sha256:c190f0899e9f9f8b6b7863debfb739abcb21a5c054f911ca3596d12b8a4c4c7f",
+                "sha256:c907f561b1e83e93fad565bac5ba9c22d96a54e7ea0267c708bffe863cbe4075",
+                "sha256:cae53c389825d3b46fb37538441f75d6aecc4174f615d048321b716df2757fb0",
+                "sha256:dd4a21253f42b8d2b48410cb31fe501d32f8b9fbeb1f55063ad102fe9c425e40",
+                "sha256:dde816ca9dac1d9c01dd504ea5967821606f02e510438120091b84e852367428",
+                "sha256:f2362f3cb0f3172c42938946dbc5b7843c2a28aec307c49100c8b38764eb6927",
+                "sha256:f328adcfebed9f11301eaedfa48e15bdece9b519fb27e6a8c01aa52a17ec31b3",
+                "sha256:f8afcf15cc511ada719a88e013cec87c11aff7b91f019295eb4530f96fe5ef2f",
+                "sha256:fb1bbeac803adea29cedd70781399c99138358c26d05fcbd23c13016b7f5ec65"
             ],
             "markers": "implementation_name == 'cpython' and python_version < '3.8'",
-            "version": "==1.4.2"
+            "version": "==1.4.3"
         },
         "typing-extensions": {
             "hashes": [
-                "sha256:7cb407020f00f7bfc3cb3e7881628838e69d8f3fcab2f64742a5e76b2f841918",
-                "sha256:99d4073b617d30288f569d3f13d2bd7548c3a7e4c8de87db09a9d29bb3a4a60c",
-                "sha256:dafc7639cde7f1b6e1acc0f457842a83e722ccca8eef5270af2d74792619a89f"
+                "sha256:0ac0f89795dd19de6b97debb0c6af1c70987fd80a2d62d1958f7e56fcc31b497",
+                "sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342",
+                "sha256:779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84"
             ],
             "markers": "python_version < '3.8'",
-            "version": "==3.7.4.3"
+            "version": "==3.10.0.0"
         },
         "wrapt": {
             "hashes": [
diff --git a/python/avocado.cfg b/python/avocado.cfg
new file mode 100644
index 00000000000..10dc6fb6054
--- /dev/null
+++ b/python/avocado.cfg
@@ -0,0 +1,10 @@
+[simpletests]
+# Don't show stdout/stderr in the test *summary*
+status.failure_fields = ['status']
+
+[job]
+# Don't show the full debug.log output; only select stdout/stderr.
+output.testlogs.logfiles = ['stdout', 'stderr']
+
+# Show full stdout/stderr only on tests that FAIL
+output.testlogs.statuses = ['FAIL']
diff --git a/python/setup.cfg b/python/setup.cfg
index 9d941386921..364b68434ca 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -26,6 +26,7 @@ devel =
     isort >= 5.1.2
     mypy >= 0.770
     pylint >= 2.7.0
+    avocado-framework >= 87.0
 
 
 [flake8]
diff --git a/python/tests/flake8.sh b/python/tests/flake8.sh
new file mode 100755
index 00000000000..51e0788462b
--- /dev/null
+++ b/python/tests/flake8.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m flake8
diff --git a/python/tests/isort.sh b/python/tests/isort.sh
new file mode 100755
index 00000000000..4480405bfb0
--- /dev/null
+++ b/python/tests/isort.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m isort -c qemu/
diff --git a/python/tests/mypy.sh b/python/tests/mypy.sh
new file mode 100755
index 00000000000..5f980f563bb
--- /dev/null
+++ b/python/tests/mypy.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m mypy -p qemu
diff --git a/python/tests/pylint.sh b/python/tests/pylint.sh
new file mode 100755
index 00000000000..4b10b34db7c
--- /dev/null
+++ b/python/tests/pylint.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m pylint qemu/
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 22/25] python: add Makefile for some common tasks
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (20 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 21/25] python: add avocado-framework and tests John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 19:24   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 23/25] python: add .gitignore John Snow
                   ` (3 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Add "make venv" to create the pipenv-managed virtual environment that
contains our explicitly pinned dependencies.

Add "make check" to run the python linters [in the host execution
environment].

Add "make venv-check" which combines the above two: create/update the
venv, then run the linters in that explicitly managed environment.

Add "make develop" which canonizes the runes needed to get both the
linting pre-requisites (the "[devel]" part), and the editable
live-install (the "-e" part) of these python libraries.

make clean: delete miscellaneous python packaging output possibly
created by pipenv, pip, or other python packaging utilities

make distclean: delete the above, the .venv, and the editable "qemu"
package forwarder (qemu.egg-info) if there is one.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst |  3 +++
 python/Makefile   | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
 create mode 100644 python/Makefile

diff --git a/python/README.rst b/python/README.rst
index e107bd12a69..3e09d20c23c 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -35,6 +35,9 @@ Files in this directory
 - ``qemu/`` Python package source directory.
 - ``tests/`` Python package tests directory.
 - ``avocado.cfg`` Configuration for the Avocado test-runner.
+  Used by ``make check`` et al.
+- ``Makefile`` provides some common testing/installation invocations.
+  Try ``make help`` to see available targets.
 - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
   that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
diff --git a/python/Makefile b/python/Makefile
new file mode 100644
index 00000000000..184f59e5634
--- /dev/null
+++ b/python/Makefile
@@ -0,0 +1,42 @@
+.PHONY: help venv venv-check check clean distclean develop
+
+help:
+	@echo "python packaging help:"
+	@echo ""
+	@echo "make venv:       Create pipenv's virtual environment."
+	@echo "    NOTE: Requires Python 3.6 and pipenv."
+	@echo "          Will download packages from PyPI."
+	@echo "    Hint: (On Fedora): 'sudo dnf install python36 pipenv'"
+	@echo ""
+	@echo "make venv-check: run linters using pipenv's virtual environment."
+	@echo "    Hint: If you don't know which test to run, run this one!"
+	@echo ""
+	@echo "make develop:    Install deps for 'make check', and"
+	@echo "                 the qemu libs in editable/development mode."
+	@echo ""
+	@echo "make check:      run linters using the current environment."
+	@echo ""
+	@echo "make clean:      remove build output."
+	@echo ""
+	@echo "make distclean:  remove venv files, qemu package forwarder, and"
+	@echo "                 everything from 'make clean'."
+
+venv: .venv
+.venv: Pipfile.lock
+	@PIPENV_VENV_IN_PROJECT=1 pipenv sync --dev --keep-outdated
+	@touch .venv
+
+venv-check: venv
+	@pipenv run make check
+
+develop:
+	pip3 install -e .[devel]
+
+check:
+	@avocado --config avocado.cfg run tests/
+
+clean:
+	rm -rf build/ dist/
+
+distclean: clean
+	rm -rf qemu.egg-info/ .venv/
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 23/25] python: add .gitignore
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (21 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 22/25] python: add Makefile for some common tasks John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 19:36   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 24/25] gitlab: add python linters to CI John Snow
                   ` (2 subsequent siblings)
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Ignore *Python* build and package output (build, dist, qemu.egg-info);
these files are not created as part of a QEMU build.

Ignore miscellaneous cached python confetti (__pycache__, *.pyc,
.mypy_cache).

Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/.gitignore | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
 create mode 100644 python/.gitignore

diff --git a/python/.gitignore b/python/.gitignore
new file mode 100644
index 00000000000..e27c99e009c
--- /dev/null
+++ b/python/.gitignore
@@ -0,0 +1,19 @@
+# python bytecode cache
+*.pyc
+__pycache__/
+
+# linter/tooling cache
+.mypy_cache/
+.cache/
+
+# python packaging
+build/
+dist/
+qemu.egg-info/
+
+# editor config
+.idea/
+.vscode/
+
+# virtual environments (pipenv et al)
+.venv/
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 24/25] gitlab: add python linters to CI
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (22 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 23/25] python: add .gitignore John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 19:55   ` Cleber Rosa
  2021-05-12 23:12 ` [PATCH v6 25/25] python: add tox support John Snow
  2021-05-17 18:52 ` [PATCH v6 00/25] python: create installable package John Snow
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Add python3.6 to the fedora container image: we need it to run the
linters against that explicit version to make sure we don't break our
minimum version promise.

Add pipenv so that we can fetch precise versions of pip packages we need
to guarantee test reproducability.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 .gitlab-ci.yml                         | 12 ++++++++++++
 tests/docker/dockerfiles/fedora.docker |  2 ++
 2 files changed, 14 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index dcb6317aace..a371c0c7163 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -779,6 +779,18 @@ check-patch:
     GIT_DEPTH: 1000
   allow_failure: true
 
+
+check-python:
+  stage: test
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
+  script:
+    - cd python
+    - make venv-check
+  variables:
+    GIT_DEPTH: 1000
+  needs:
+    job: amd64-fedora-container
+
 check-dco:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 915fdc1845e..6908d69ac37 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -84,6 +84,7 @@ ENV PACKAGES \
     numactl-devel \
     perl \
     perl-Test-Harness \
+    pipenv \
     pixman-devel \
     python3 \
     python3-PyYAML \
@@ -93,6 +94,7 @@ ENV PACKAGES \
     python3-pip \
     python3-sphinx \
     python3-virtualenv \
+    python3.6 \
     rdma-core-devel \
     SDL2-devel \
     snappy-devel \
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* [PATCH v6 25/25] python: add tox support
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (23 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 24/25] gitlab: add python linters to CI John Snow
@ 2021-05-12 23:12 ` John Snow
  2021-05-25 20:15   ` Cleber Rosa
  2021-05-17 18:52 ` [PATCH v6 00/25] python: create installable package John Snow
  25 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

This is intended to be a manually run, non-CI script.

Use tox to test the linters against all python versions from 3.6 to
3.9. This will only work if you actually have those versions installed
locally, but Fedora makes this easy:

> sudo dnf install python36 python37 python38 python39

Unlike the pipenv tests (make venv-check), this pulls "whichever"
versions of the python packages, so they are unpinned and may break as
time goes on. In the case that breakages are found, setup.cfg should be
amended accordingly to avoid the bad dependant versions, or the code
should be amended to work around the issue.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst |  2 ++
 python/.gitignore |  1 +
 python/Makefile   |  7 ++++++-
 python/setup.cfg  |  1 +
 python/tox.ini    | 13 +++++++++++++
 5 files changed, 23 insertions(+), 1 deletion(-)
 create mode 100644 python/tox.ini

diff --git a/python/README.rst b/python/README.rst
index 3e09d20c23c..7360dee32be 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -49,3 +49,5 @@ Files in this directory
   this package; it is referenced by ``setup.cfg``.
 - ``setup.cfg`` houses setuptools package configuration.
 - ``setup.py`` is the setuptools installer used by pip; See above.
+- ``tox.ini`` houses configuration for tox, which runs tests against
+  several Python versions to test compatibility.
diff --git a/python/.gitignore b/python/.gitignore
index e27c99e009c..d92e3f4bcca 100644
--- a/python/.gitignore
+++ b/python/.gitignore
@@ -17,3 +17,4 @@ qemu.egg-info/
 
 # virtual environments (pipenv et al)
 .venv/
+.tox/
diff --git a/python/Makefile b/python/Makefile
index 184f59e5634..a01db823318 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -16,6 +16,8 @@ help:
 	@echo ""
 	@echo "make check:      run linters using the current environment."
 	@echo ""
+	@echo "make check-tox:  run linters using multiple python versions."
+	@echo ""
 	@echo "make clean:      remove build output."
 	@echo ""
 	@echo "make distclean:  remove venv files, qemu package forwarder, and"
@@ -35,8 +37,11 @@ develop:
 check:
 	@avocado --config avocado.cfg run tests/
 
+check-tox:
+	@tox
+
 clean:
 	rm -rf build/ dist/
 
 distclean: clean
-	rm -rf qemu.egg-info/ .venv/
+	rm -rf qemu.egg-info/ .venv/ .tox/
diff --git a/python/setup.cfg b/python/setup.cfg
index 364b68434ca..a7c5f636790 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -33,6 +33,7 @@ devel =
 extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
 exclude = __pycache__,
           .venv,
+          .tox,
 
 [mypy]
 strict = True
diff --git a/python/tox.ini b/python/tox.ini
new file mode 100644
index 00000000000..d8fe503b477
--- /dev/null
+++ b/python/tox.ini
@@ -0,0 +1,13 @@
+# tox (https://tox.readthedocs.io/) is a tool for running tests
+# in multiple virtualenvs. This configuration file will run the
+# test suite on all supported python versions. To use it, "pip install tox"
+# and then run "tox" from this directory.
+
+[tox]
+envlist = py36, py37, py38, py39
+
+[testenv]
+allowlist_externals = make
+deps = .[devel]
+commands =
+    make check
-- 
2.30.2



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 00/25] python: create installable package
  2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
                   ` (24 preceding siblings ...)
  2021-05-12 23:12 ` [PATCH v6 25/25] python: add tox support John Snow
@ 2021-05-17 18:52 ` John Snow
  25 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-17 18:52 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, Willian Rampazzo,
	Willian Rampazzo, Cleber Rosa, Alex Bennée, Beraldo Leal

On 5/12/21 7:12 PM, John Snow wrote:
> Based-on: <20210512214642.2803189-1-jsnow@redhat.com>
> CI: https://gitlab.com/jsnow/qemu/-/pipelines/302010131
> GitLab: https://gitlab.com/jsnow/qemu/-/tree/python-package-mk3
> MR: https://gitlab.com/jsnow/qemu/-/merge_requests/4
> 
Patchset updated and rebased on top of new linting pre-req series.
(Gitlab branch and MR rebased and updated.)

Based-on: <20210517184808.3562549-1-jsnow@redhat.com>

> I invite you to leave review comments on my mock merge request on
> gitlab, submitted against my own mirror. I will, as always, also respond
> to feedback on-list.
> 
> ABOUT
> =====
> 
> This series factors the python/qemu directory as an installable
> package. It does not yet actually change the mechanics of how any other
> python source in the tree actually consumes it (yet), beyond the import
> path -- some import statements change in a few places.
> 
> RATIONALE
> =========
> 
> The primary motivation of this series is primarily to formalize our
> dependencies on mypy, flake8, isort, and pylint alongside versions that
> are known to work. It does this using the setup.cfg and setup.py
> files. It also adds explicitly pinned versions (using Pipfile.lock) of
> these dependencies that should behave in a repeatable and known way for
> developers and CI environments both. Lastly, it enables those CI checks
> such that we can enforce Python coding quality checks via the CI tests.
> 
> An auxiliary motivation is that this package is formatted in such a way
> that it COULD be uploaded to https://pypi.org/project/qemu and installed
> independently of qemu.git with `pip install qemu`, but that button
> remains *unpushed* and this series *will not* cause any such
> releases. We have time to debate finer points like API guarantees and
> versioning even after this series is merged.
> 
> Other bits of interest
> ----------------------
> 
> With the python tooling as a proper package, you can install this
> package in editable or production mode to a virtual environment, your
> local user environment, or your system packages. The primary benefit of
> this is to gain access to QMP tooling regardless of CWD, without needing
> to battle sys.path (and confounding other python analysis tools).
> 
> For example: when developing, you may go to qemu/python/ and run `make
> venv` followed by `pipenv shell` to activate a virtual environment that
> contains the qemu python packages. These packages will always reflect
> the current version of the source files in the tree. When you are
> finished, you can simply exit the shell (^d) to remove these packages
> from your python environment.
> 
> When not developing, you could install a version of this package to your
> environment outright to gain access to the QMP and QEMUMachine classes
> for lightweight scripting and testing by using pip: "pip install
> [--user] ."
> 
> TESTING THIS SERIES
> ===================
> 
> First of all, nothing should change. Without any intervention,
> everything should behave exactly as it did before. The only new
> information here comes from how to interact with and run the linters
> that will be enforcing code quality standards in this subdirectory.
> 
> There are various invocations available that will test subtly different
> combinations using subtly different environments. I am assuming some
> light knowledge of Python environments and installing Python packages
> here. If you have questions, I would be delighted to answer them.
> 
> To test the new tests, CD to ./python/ first, and then:
> 
> 0. Try "make" or "make help" to get a sense of this series.
> 
> 1. Try "make venv && pipenv shell" to get a venv with the package
>     installed to it in editable mode. Ctrl+d exits this venv shell. While
>     in this shell, any python script that uses "from qemu.[qmp|machine]
>     import ..." should work correctly regardless of where the script is,
>     or what your CWD is.
> 
>     This will pull some packages from PyPI and install them into the
>     virtual environment, leaving your normal environment untouched.
> 
>     You will need Python 3.6 and pipenv installed on your system to do
>     this step. For Fedora: "dnf install python36 pipenv" will do the
>     trick. If you don't have this, skip down to #4 and onwards.
> 
> 2. Try "make check" while still in the shell to run the Python linters
> using the venv built in the previous step. This will run avocado, which
> will in turn execute mypy, flake8, isort and pylint with the correct
> arguments.
> 
> 3. Having exited the shell from above, try "make venv-check". This will
> create and update the venv if needed, then run 'make check' within the
> context of that shell. It should pass as long as the above did. You
> should be able to run "make distclean" prior to running "make
> venv-check" and have the entire process work start to finish.
> 
> 4. Still outside of the venv, you may try running "make check". This
> will not install anything, but unless you have the right Python
> dependencies installed, these tests may fail for you. You might try
> using "pip install --user .[devel]" to install the development packages
> needed to run the tests successfully to your local user's python
> environment. Once done, you will probably want to "pip uninstall qemu"
> to remove the qemu packages to avoid them interfering with other things.
> 
> 5. "make distclean" will delete the venv and any temporary files that
> may have been created by packaging, installing, testing, etc.
> 
> 6. You may also (if you wish) create your own environment using a Python
>     other than 3.6, bypassing pipenv. This may be useful for environments
>     in which you simply do not have Python 3.6 readily available, or
>     explicitly wish to test against a different version.
> 
>     Using the Python of your choice:
> 
>     - `> make distclean` (Remove the .venv, if you created one.)
>     - `> /usr/bin/python3.9 -m venv .venv`
>     - `> source .venv/bin/activate` (or activate.[fish|csh])
>     - `> make develop`  (Installs linter deps to this venv)
>     - `> make check`    (Runs tests using current env)
>     - `> deactivate`    (Leaves the venv)
> 
>     This *should* pass, and if it doesn't, it's a bug. Due to
>     dependencies changing like shifting sands, it's impossible to promise
>     support for every last combination of dependency and python
>     version. This is why the "make venv-check" target uses Python 3 and a
>     very explicit set of packages instead. Still, this form *should*
>     pass.
> 
> 7. If you are running Fedora or a distro that packages multiple Python
> versions, you can try running "make check-tox" to run the linters under
> all Python versions from Python 3.6 to 3.9. This will create a series of
> virtual environments under the .tox folder, leaving your existing
> environment unscathed. This should pass, provided you have the requisite
> Python versions on hand. When you are done, "make distclean" should
> remove any testing files.
> 
> CHANGELOG
> =========
> 
> V6
> --
> 
> ```
> 001/25:[----] [--] 'iotests/297: add --namespace-packages to mypy arguments'
> 002/25:[----] [-C] 'python: create qemu packages'
> 003/25:[----] [--] 'python: create utils sub-package'
> 004/25:[0005] [FC] 'python: add qemu package installer'
> 005/25:[0002] [FC] 'python: add VERSION file'
> 006/25:[----] [--] 'python: add directory structure README.rst files'
> 007/25:[down] 'python: add MANIFEST.in'
> 008/25:[0001] [FC] 'python: Add pipenv support'
> 009/25:[----] [--] 'python: add pylint import exceptions'
> 010/25:[----] [--] 'python: move pylintrc into setup.cfg'
> 011/25:[0083] [FC] 'python: add pylint to pipenv'
> 012/25:[----] [--] 'python: move flake8 config to setup.cfg'
> 013/25:[----] [--] 'python: add excluded dirs to flake8 config'
> 014/25:[0038] [FC] 'python: Add flake8 to pipenv'
> 015/25:[----] [--] 'python: move mypy.ini into setup.cfg'
> 016/25:[0050] [FC] 'python: add mypy to pipenv'
> 017/25:[----] [--] 'python: move .isort.cfg into setup.cfg'
> 018/25:[0006] [FC] 'python/qemu: add isort to pipenv'
> 019/25:[0004] [FC] 'python/qemu: add qemu package itself to pipenv'
> 020/25:[0014] [FC] 'python: add devel package requirements to setuptools'
> 021/25:[down] 'python: add avocado-framework and tests'
> 022/25:[0015] [FC] 'python: add Makefile for some common tasks'
> 023/25:[0018] [FC] 'python: add .gitignore'
> 024/25:[0004] [FC] 'gitlab: add python linters to CI'
> 025/25:[down] 'python: add tox support'
> ```
> 
> - 04: Changed maintainer blurb in PACKAGE.rst
> - 05: VERSION is now 0.6.1a0
> - 07: New, needed for both tox and using Pipenv to install the qemu
>        package(s) in editable mode.
> - 08: Add README entries explaining the purpose of the pipenv files.
> - 11: pylint dependency was bumped to 2.7.0.
> - 14: pipenv changes.
> - 16: pipenv changes.
> - 18: isort pinned at >= 5.1.2 now; see commit message.
> - 19: pipenv changes.
> - 20: Context differences from isort, pylint, and pipenv changes.
> - 22:
>     - Add 'make develop'
>     - Add more README.rst entries.
>     - Change pytest invocation to avocado.
> - 23:
>     - Add comments to gitignore, split ignores out by function/category
>     - Ignore vscode files. (I like vscode now!)
> - 24: Use the 'needs' feature of Gitlab CI and move the test to 'test'
>    stage
> 
> STATUS
> =======
> 
> ```
> + [01] iotests-297-add-namespace      # [RB] CR [SOB] JS
> + [02] python-create-qemu-packages    # [SOB] JS
> + [03] python-create-utils-sub        # [SOB] JS
> + [04] python-add-qemu-package        # [SOB] JS
> + [05] python-add-version-file        # [RB] CR [SOB] JS
> + [06] python-add-directory-structure # [SOB] JS
> + [07] python-add-manifest-in         # [SOB] JS
> + [08] python-add-pipenv-support      # [RB] CR [SOB] JS
> + [09] python-add-pylint-import       # [RB] CR [SOB] JS
> + [10] python-move-pylintrc-into      # [RB] CR [TB] CR [SOB] JS
> + [11] python-add-pylint-to-pipenv    # [SOB] JS
> + [12] python-move-flake8-config-to   # [RB] CR [SOB] JS
> + [13] python-add-excluded-dirs-to    # [SOB] JS
> + [14] python-add-flake8-to-pipenv    # [RB] CR [TB] CR [SOB] JS
> + [15] python-move-mypy-ini-into      # [SOB] JS
> + [16] python-add-mypy-to-pipenv      # [RB] CR [TB] CR [SOB] JS
> + [17] python-move-isort-cfg-into     # [RB] CR [SOB] JS
> + [18] python-qemu-add-isort-to       # [SOB] JS
> + [19] python-qemu-add-qemu-package   # [RB] CR [TB] CR [SOB] JS
> + [20] python-add-devel-package       # [SOB] JS
> + [21] python-add-avocado-framework   # [SOB] JS
> + [22] python-add-makefile-for-some   # [SOB] JS
> + [23] python-add-gitignore           # [SOB] JS
> + [24] gitlab-add-python-linters-to   # [SOB] JS
>> [25] python-add-tox-support         # [SOB] JS
> ```
> 
> John Snow (25):
>    iotests/297: add --namespace-packages to mypy arguments
>    python: create qemu packages
>    python: create utils sub-package
>    python: add qemu package installer
>    python: add VERSION file
>    python: add directory structure README.rst files
>    python: add MANIFEST.in
>    python: Add pipenv support
>    python: add pylint import exceptions
>    python: move pylintrc into setup.cfg
>    python: add pylint to pipenv
>    python: move flake8 config to setup.cfg
>    python: add excluded dirs to flake8 config
>    python: Add flake8 to pipenv
>    python: move mypy.ini into setup.cfg
>    python: add mypy to pipenv
>    python: move .isort.cfg into setup.cfg
>    python/qemu: add isort to pipenv
>    python/qemu: add qemu package itself to pipenv
>    python: add devel package requirements to setuptools
>    python: add avocado-framework and tests
>    python: add Makefile for some common tasks
>    python: add .gitignore
>    gitlab: add python linters to CI
>    python: add tox support
> 
>   python/PACKAGE.rst                          |  33 +++
>   python/README.rst                           |  53 +++++
>   python/qemu/README.rst                      |   8 +
>   python/qemu/machine/README.rst              |   9 +
>   python/qemu/qmp/README.rst                  |   9 +
>   python/qemu/utils/README.rst                |   7 +
>   .gitlab-ci.yml                              |  12 +
>   python/.gitignore                           |  20 ++
>   python/MANIFEST.in                          |   2 +
>   python/Makefile                             |  47 ++++
>   python/Pipfile                              |  13 ++
>   python/Pipfile.lock                         | 231 ++++++++++++++++++++
>   python/VERSION                              |   1 +
>   python/avocado.cfg                          |  10 +
>   python/mypy.ini                             |   4 -
>   python/qemu/.flake8                         |   2 -
>   python/qemu/.isort.cfg                      |   7 -
>   python/qemu/__init__.py                     |  11 -
>   python/qemu/machine/__init__.py             |  36 +++
>   python/qemu/{ => machine}/console_socket.py |   0
>   python/qemu/{ => machine}/machine.py        |  16 +-
>   python/qemu/{ => machine}/qtest.py          |   3 +-
>   python/qemu/pylintrc                        |  58 -----
>   python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
>   python/qemu/utils/__init__.py               |  23 ++
>   python/qemu/{ => utils}/accel.py            |   0
>   python/setup.cfg                            |  79 +++++++
>   python/setup.py                             |  23 ++
>   python/tests/flake8.sh                      |   2 +
>   python/tests/isort.sh                       |   2 +
>   python/tests/mypy.sh                        |   2 +
>   python/tests/pylint.sh                      |   2 +
>   python/tox.ini                              |  13 ++
>   tests/acceptance/avocado_qemu/__init__.py   |   4 +-
>   tests/acceptance/virtio-gpu.py              |   2 +-
>   tests/docker/dockerfiles/fedora.docker      |   2 +
>   tests/qemu-iotests/297                      |   1 +
>   tests/qemu-iotests/300                      |   4 +-
>   tests/qemu-iotests/iotests.py               |   2 +-
>   tests/vm/aarch64vm.py                       |   2 +-
>   tests/vm/basevm.py                          |   2 +-
>   41 files changed, 672 insertions(+), 97 deletions(-)
>   create mode 100644 python/PACKAGE.rst
>   create mode 100644 python/README.rst
>   create mode 100644 python/qemu/README.rst
>   create mode 100644 python/qemu/machine/README.rst
>   create mode 100644 python/qemu/qmp/README.rst
>   create mode 100644 python/qemu/utils/README.rst
>   create mode 100644 python/.gitignore
>   create mode 100644 python/MANIFEST.in
>   create mode 100644 python/Makefile
>   create mode 100644 python/Pipfile
>   create mode 100644 python/Pipfile.lock
>   create mode 100644 python/VERSION
>   create mode 100644 python/avocado.cfg
>   delete mode 100644 python/mypy.ini
>   delete mode 100644 python/qemu/.flake8
>   delete mode 100644 python/qemu/.isort.cfg
>   delete mode 100644 python/qemu/__init__.py
>   create mode 100644 python/qemu/machine/__init__.py
>   rename python/qemu/{ => machine}/console_socket.py (100%)
>   rename python/qemu/{ => machine}/machine.py (98%)
>   rename python/qemu/{ => machine}/qtest.py (98%)
>   delete mode 100644 python/qemu/pylintrc
>   rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
>   create mode 100644 python/qemu/utils/__init__.py
>   rename python/qemu/{ => utils}/accel.py (100%)
>   create mode 100644 python/setup.cfg
>   create mode 100755 python/setup.py
>   create mode 100755 python/tests/flake8.sh
>   create mode 100755 python/tests/isort.sh
>   create mode 100755 python/tests/mypy.sh
>   create mode 100755 python/tests/pylint.sh
>   create mode 100644 python/tox.ini
> 



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 02/25] python: create qemu packages
  2021-05-12 23:12 ` [PATCH v6 02/25] python: create qemu packages John Snow
@ 2021-05-18 14:31   ` Cleber Rosa
  2021-05-21 17:15   ` Willian Rampazzo
  1 sibling, 0 replies; 64+ messages in thread
From: Cleber Rosa @ 2021-05-18 14:31 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz,
	Philippe Mathieu-Daudé,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 2914 bytes --]

On Wed, May 12, 2021 at 07:12:18PM -0400, John Snow wrote:
> move python/qemu/*.py to python/qemu/[machine, qmp]/*.py and update import
> directives across the tree.
> 
> This is done to create a PEP420 namespace package, in which we may
> create subpackages. To do this, the namespace directory ("qemu") should
> not have any modules in it. Those files will go into new 'machine' and 'qmp'
> subpackages instead.
> 
> Implement machine/__init__.py making the top-level classes and functions
> from its various modules available directly inside the package. Change
> qmp.py to qmp/__init__.py similarly, such that all of the useful QMP
> library classes are available directly from "qemu.qmp" instead of
> "qemu.qmp.qmp".
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> 
> 
> ---
> 
> Note for reviewers: in the next patch, I add a utils sub-package and
> move qemu/machine/accel.py to qemu/utils/accel.py. If we like it that
> way, we can squash it in here if we want, or just leave it as its own
> follow-up patch, I am just leaving it as something that will be easy to
> un-do or change for now.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/{qemu => }/.isort.cfg                |  0
>  python/qemu/__init__.py                     | 11 ------
>  python/qemu/{ => machine}/.flake8           |  0
>  python/qemu/machine/__init__.py             | 41 +++++++++++++++++++++
>  python/qemu/{ => machine}/accel.py          |  0
>  python/qemu/{ => machine}/console_socket.py |  0
>  python/qemu/{ => machine}/machine.py        | 16 +++++---
>  python/qemu/{ => machine}/pylintrc          |  0
>  python/qemu/{ => machine}/qtest.py          |  3 +-
>  python/qemu/{qmp.py => qmp/__init__.py}     | 12 +++++-
>  tests/acceptance/avocado_qemu/__init__.py   |  4 +-
>  tests/acceptance/virtio-gpu.py              |  2 +-
>  tests/qemu-iotests/300                      |  4 +-
>  tests/qemu-iotests/iotests.py               |  2 +-
>  tests/vm/aarch64vm.py                       |  2 +-
>  tests/vm/basevm.py                          |  3 +-
>  16 files changed, 73 insertions(+), 27 deletions(-)
>  rename python/{qemu => }/.isort.cfg (100%)
>  delete mode 100644 python/qemu/__init__.py
>  rename python/qemu/{ => machine}/.flake8 (100%)
>  create mode 100644 python/qemu/machine/__init__.py
>  rename python/qemu/{ => machine}/accel.py (100%)
>  rename python/qemu/{ => machine}/console_socket.py (100%)
>  rename python/qemu/{ => machine}/machine.py (98%)
>  rename python/qemu/{ => machine}/pylintrc (100%)
>  rename python/qemu/{ => machine}/qtest.py (99%)
>  rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
>

Hi John,

Thanks for this!  I went through it and it LGTM.  I've tested it with
iotests, acceptance and with a vm-build-fedora.

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 03/25] python: create utils sub-package
  2021-05-12 23:12 ` [PATCH v6 03/25] python: create utils sub-package John Snow
@ 2021-05-18 15:12   ` Cleber Rosa
  2021-05-18 15:20     ` John Snow
  2021-05-21 17:18   ` Willian Rampazzo
  1 sibling, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-18 15:12 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz,
	Philippe Mathieu-Daudé,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

On Wed, May 12, 2021 at 07:12:19PM -0400, John Snow wrote:
> Create a space for miscellaneous things that don't belong strictly in
> "qemu.machine" nor "qemu.qmp" packages.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/qemu/machine/__init__.py           |  8 --------
>  python/qemu/utils/__init__.py             | 23 +++++++++++++++++++++++
>  python/qemu/{machine => utils}/accel.py   |  0
>  tests/acceptance/avocado_qemu/__init__.py |  4 ++--
>  tests/acceptance/virtio-gpu.py            |  2 +-
>  tests/vm/aarch64vm.py                     |  2 +-
>  tests/vm/basevm.py                        |  3 ++-
>  7 files changed, 29 insertions(+), 13 deletions(-)
>  create mode 100644 python/qemu/utils/__init__.py
>  rename python/qemu/{machine => utils}/accel.py (100%)
> 

As you mentioned in the previous patch notes, I would not mind a
squash here.  Either way:

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 03/25] python: create utils sub-package
  2021-05-18 15:12   ` Cleber Rosa
@ 2021-05-18 15:20     ` John Snow
  0 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-18 15:20 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz,
	Philippe Mathieu-Daudé,
	Beraldo Leal

On 5/18/21 11:12 AM, Cleber Rosa wrote:
> As you mentioned in the previous patch notes, I would not mind a
> squash here.  Either way:
> 
> Reviewed-by: Cleber Rosa<crosa@redhat.com>
> Tested-by: Cleber Rosa<crosa@redhat.com>

I will definitely do so, then. :)

Thanks for the reviews thus far!

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 04/25] python: add qemu package installer
  2021-05-12 23:12 ` [PATCH v6 04/25] python: add qemu package installer John Snow
@ 2021-05-21  4:00   ` Cleber Rosa
  2021-05-21 15:52     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-21  4:00 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 6467 bytes --]

On Wed, May 12, 2021 at 07:12:20PM -0400, John Snow wrote:
> Add setup.cfg and setup.py, necessary for installing a package via
> pip. Add a ReST document (PACKAGE.rst) explaining the basics of what
> this package is for and who to contact for more information. This
> document will be used as the landing page for the package on PyPI.
> 
> I am not yet using a pyproject.toml style package manifest, because
> "editable" installs are not defined (yet?) by PEP-517/518.
> 
> I consider editable installs crucial for development, though they have
> (apparently) always been somewhat poorly defined.
>

They are crucial for development indeed, so I agree with your overall
choices here.

> Pip now (19.2 and later) now supports editable installs for projects
> using pyproject.toml manifests, but might require the use of the
> --no-use-pep517 flag, which somewhat defeats the point.
>

Just to make it clear for other people reading this, it means that
even with the very latest pip release (21.1.1), you *must* have a
setup.py or setup.cfg file to use editable (development) installs[1]
You can *not* rely solely on a pyproject.toml setup.

[1] https://github.com/pypa/pip/pull/9547/commits/7a95720e796a5e56481c1cc20b6ce6249c50f357

> For now, while the dust settles, stick with the de-facto
> setup.py/setup.cfg combination supported by setuptools. It will be worth
> re-evaluating this point again in the future when our supported build
> platforms all ship a fairly modern pip.
>

Agreed, but let's not hold our breath given that even pip 21.1.1 still
doesn't support that.  My guesstimate is 6-12 months for the feature to
be developed/merged, and let's say another 12 months for our supported
build platforms to ship it.

> Additional reading on this matter:
> 
> https://github.com/pypa/packaging-problems/issues/256
> https://github.com/pypa/pip/issues/6334
> https://github.com/pypa/pip/issues/6375
> https://github.com/pypa/pip/issues/6434
> https://github.com/pypa/pip/issues/6438
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/PACKAGE.rst | 33 +++++++++++++++++++++++++++++++++
>  python/setup.cfg   | 19 +++++++++++++++++++
>  python/setup.py    | 23 +++++++++++++++++++++++
>  3 files changed, 75 insertions(+)
>  create mode 100644 python/PACKAGE.rst
>  create mode 100644 python/setup.cfg
>  create mode 100755 python/setup.py
>
> diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
> new file mode 100644
> index 00000000000..1bbfe1b58e2
> --- /dev/null
> +++ b/python/PACKAGE.rst
> @@ -0,0 +1,33 @@
> +QEMU Python Tooling
> +===================
> +
> +This package provides QEMU tooling used by the QEMU project to build,
> +configure, and test QEMU. It is not a fully-fledged SDK and it is subject
> +to change at any time.
> +
> +Usage
> +-----
> +
> +The ``qemu.qmp`` subpackage provides a library for communicating with
> +QMP servers. The ``qemu.machine`` subpackage offers rudimentary
> +facilities for launching and managing QEMU processes. Refer to each
> +package's documentation
> +(``>>> help(qemu.qmp)``, ``>>> help(qemu.machine)``)
> +for more information.
> +
> +Contributing
> +------------
> +
> +This package is maintained by John Snow <jsnow@redhat.com> as part of
> +the QEMU source tree. Contributions are welcome and follow the `QEMU
> +patch submission process
> +<https://wiki.qemu.org/Contribute/SubmitAPatch>`_, which involves
> +sending patches to the QEMU development mailing list.
> +
> +John maintains a `GitLab staging branch
> +<https://gitlab.com/jsnow/qemu/-/tree/python>`_, and there is an
> +official `GitLab mirror <https://gitlab.com/qemu-project/qemu>`_.
> +
> +Please report bugs on the `QEMU issue tracker
> +<https://gitlab.com/qemu-project/qemu/-/issues>`_ and tag ``@jsnow`` in
> +the report.
> diff --git a/python/setup.cfg b/python/setup.cfg
> new file mode 100644
> index 00000000000..dd71640fc2f
> --- /dev/null
> +++ b/python/setup.cfg
> @@ -0,0 +1,19 @@
> +[metadata]
> +name = qemu
> +maintainer = QEMU Developer Team
> +maintainer_email = qemu-devel@nongnu.org
> +url = https://www.qemu.org/
> +download_url = https://www.qemu.org/download/
> +description = QEMU Python Build, Debug and SDK tooling.
> +long_description = file:PACKAGE.rst
> +long_description_content_type = text/x-rst
> +classifiers =
> +    Development Status :: 3 - Alpha
> +    License :: OSI Approved :: GNU General Public License v2 (GPLv2)
> +    Natural Language :: English
> +    Operating System :: OS Independent
> +    Programming Language :: Python :: 3 :: Only
> +
> +[options]
> +python_requires = >= 3.6
> +packages = find_namespace:
> diff --git a/python/setup.py b/python/setup.py
> new file mode 100755
> index 00000000000..2014f81b757
> --- /dev/null
> +++ b/python/setup.py
> @@ -0,0 +1,23 @@
> +#!/usr/bin/env python3
> +"""
> +QEMU tooling installer script
> +Copyright (c) 2020-2021 John Snow for Red Hat, Inc.
> +"""
> +
> +import setuptools
> +import pkg_resources
> +
> +
> +def main():
> +    """
> +    QEMU tooling installer
> +    """
> +
> +    # https://medium.com/@daveshawley/safely-using-setup-cfg-for-metadata-1babbe54c108
> +    pkg_resources.require('setuptools>=39.2')
> +
> +    setuptools.setup()
> +
> +
> +if __name__ == '__main__':
> +    main()
> -- 
> 2.30.2
> 
> 

BTW, about the need to have a "setup.py", before pip 21.1.1:

  $ rm setup.py
  $ pip install -e .
  ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/cleber/src/qemu/python

On pip 21.1.1:

   $ pip install -e . 
   Obtaining file:///home/cleber/src/qemu/python
   Installing collected packages: qemu
     Running setup.py develop for qemu
   Successfully installed qemu-0.0.0

Side note: The "Running setup.py ..." message given by pip 21.1.1,
even though there is *not* a "setup.py" is rather confusing.

Anyway, we may be able to drop setup.py either when we find pip 21.1.1
or later in our "common build environments", or if we require people
hacking on the Python module to "pip install --upgrade pip".

I'll be repeating myself here, but I believe you made the right
choices at this time, and based on my testing I can successfully
install/develop using "python setup.py" and "pip", so:

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 04/25] python: add qemu package installer
  2021-05-21  4:00   ` Cleber Rosa
@ 2021-05-21 15:52     ` John Snow
  0 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-21 15:52 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/21/21 12:00 AM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:20PM -0400, John Snow wrote:
>> Add setup.cfg and setup.py, necessary for installing a package via
>> pip. Add a ReST document (PACKAGE.rst) explaining the basics of what
>> this package is for and who to contact for more information. This
>> document will be used as the landing page for the package on PyPI.
>>
>> I am not yet using a pyproject.toml style package manifest, because
>> "editable" installs are not defined (yet?) by PEP-517/518.
>>
>> I consider editable installs crucial for development, though they have
>> (apparently) always been somewhat poorly defined.
>>
> 
> They are crucial for development indeed, so I agree with your overall
> choices here.
> 
>> Pip now (19.2 and later) now supports editable installs for projects
>> using pyproject.toml manifests, but might require the use of the
>> --no-use-pep517 flag, which somewhat defeats the point.
>>
> 
> Just to make it clear for other people reading this, it means that
> even with the very latest pip release (21.1.1), you *must* have a
> setup.py or setup.cfg file to use editable (development) installs[1]
> You can *not* rely solely on a pyproject.toml setup.
> 
> [1] https://github.com/pypa/pip/pull/9547/commits/7a95720e796a5e56481c1cc20b6ce6249c50f357
> 
>> For now, while the dust settles, stick with the de-facto
>> setup.py/setup.cfg combination supported by setuptools. It will be worth
>> re-evaluating this point again in the future when our supported build
>> platforms all ship a fairly modern pip.
>>
> 
> Agreed, but let's not hold our breath given that even pip 21.1.1 still
> doesn't support that.  My guesstimate is 6-12 months for the feature to
> be developed/merged, and let's say another 12 months for our supported
> build platforms to ship it.
> 
>> Additional reading on this matter:
>>
>> https://github.com/pypa/packaging-problems/issues/256
>> https://github.com/pypa/pip/issues/6334
>> https://github.com/pypa/pip/issues/6375
>> https://github.com/pypa/pip/issues/6434
>> https://github.com/pypa/pip/issues/6438
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/PACKAGE.rst | 33 +++++++++++++++++++++++++++++++++
>>   python/setup.cfg   | 19 +++++++++++++++++++
>>   python/setup.py    | 23 +++++++++++++++++++++++
>>   3 files changed, 75 insertions(+)
>>   create mode 100644 python/PACKAGE.rst
>>   create mode 100644 python/setup.cfg
>>   create mode 100755 python/setup.py
>>
>> diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
>> new file mode 100644
>> index 00000000000..1bbfe1b58e2
>> --- /dev/null
>> +++ b/python/PACKAGE.rst
>> @@ -0,0 +1,33 @@
>> +QEMU Python Tooling
>> +===================
>> +
>> +This package provides QEMU tooling used by the QEMU project to build,
>> +configure, and test QEMU. It is not a fully-fledged SDK and it is subject
>> +to change at any time.
>> +
>> +Usage
>> +-----
>> +
>> +The ``qemu.qmp`` subpackage provides a library for communicating with
>> +QMP servers. The ``qemu.machine`` subpackage offers rudimentary
>> +facilities for launching and managing QEMU processes. Refer to each
>> +package's documentation
>> +(``>>> help(qemu.qmp)``, ``>>> help(qemu.machine)``)
>> +for more information.
>> +
>> +Contributing
>> +------------
>> +
>> +This package is maintained by John Snow <jsnow@redhat.com> as part of
>> +the QEMU source tree. Contributions are welcome and follow the `QEMU
>> +patch submission process
>> +<https://wiki.qemu.org/Contribute/SubmitAPatch>`_, which involves
>> +sending patches to the QEMU development mailing list.
>> +
>> +John maintains a `GitLab staging branch
>> +<https://gitlab.com/jsnow/qemu/-/tree/python>`_, and there is an
>> +official `GitLab mirror <https://gitlab.com/qemu-project/qemu>`_.
>> +
>> +Please report bugs on the `QEMU issue tracker
>> +<https://gitlab.com/qemu-project/qemu/-/issues>`_ and tag ``@jsnow`` in
>> +the report.
>> diff --git a/python/setup.cfg b/python/setup.cfg
>> new file mode 100644
>> index 00000000000..dd71640fc2f
>> --- /dev/null
>> +++ b/python/setup.cfg
>> @@ -0,0 +1,19 @@
>> +[metadata]
>> +name = qemu
>> +maintainer = QEMU Developer Team
>> +maintainer_email = qemu-devel@nongnu.org
>> +url = https://www.qemu.org/
>> +download_url = https://www.qemu.org/download/
>> +description = QEMU Python Build, Debug and SDK tooling.
>> +long_description = file:PACKAGE.rst
>> +long_description_content_type = text/x-rst
>> +classifiers =
>> +    Development Status :: 3 - Alpha
>> +    License :: OSI Approved :: GNU General Public License v2 (GPLv2)
>> +    Natural Language :: English
>> +    Operating System :: OS Independent
>> +    Programming Language :: Python :: 3 :: Only
>> +
>> +[options]
>> +python_requires = >= 3.6
>> +packages = find_namespace:
>> diff --git a/python/setup.py b/python/setup.py
>> new file mode 100755
>> index 00000000000..2014f81b757
>> --- /dev/null
>> +++ b/python/setup.py
>> @@ -0,0 +1,23 @@
>> +#!/usr/bin/env python3
>> +"""
>> +QEMU tooling installer script
>> +Copyright (c) 2020-2021 John Snow for Red Hat, Inc.
>> +"""
>> +
>> +import setuptools
>> +import pkg_resources
>> +
>> +
>> +def main():
>> +    """
>> +    QEMU tooling installer
>> +    """
>> +
>> +    # https://medium.com/@daveshawley/safely-using-setup-cfg-for-metadata-1babbe54c108
>> +    pkg_resources.require('setuptools>=39.2')
>> +
>> +    setuptools.setup()
>> +
>> +
>> +if __name__ == '__main__':
>> +    main()
>> -- 
>> 2.30.2
>>
>>
> 
> BTW, about the need to have a "setup.py", before pip 21.1.1:
> 
>    $ rm setup.py
>    $ pip install -e .
>    ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/cleber/src/qemu/python
> 
> On pip 21.1.1:
> 
>     $ pip install -e .
>     Obtaining file:///home/cleber/src/qemu/python
>     Installing collected packages: qemu
>       Running setup.py develop for qemu
>     Successfully installed qemu-0.0.0
> 
> Side note: The "Running setup.py ..." message given by pip 21.1.1,
> even though there is *not* a "setup.py" is rather confusing.
> 
> Anyway, we may be able to drop setup.py either when we find pip 21.1.1
> or later in our "common build environments", or if we require people
> hacking on the Python module to "pip install --upgrade pip".
> 
> I'll be repeating myself here, but I believe you made the right
> choices at this time, and based on my testing I can successfully
> install/develop using "python setup.py" and "pip", so:
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> 

Great :)

I figured it would be helpful to outline the reasons in the commit 
message, at least, so that anyone who is curious about it in the future 
can find out exactly what we considered the blockers to be.

I'll update the commit message with an edited version of your reply 
here, just to add more information to the historical record.

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 02/25] python: create qemu packages
  2021-05-12 23:12 ` [PATCH v6 02/25] python: create qemu packages John Snow
  2021-05-18 14:31   ` Cleber Rosa
@ 2021-05-21 17:15   ` Willian Rampazzo
  1 sibling, 0 replies; 64+ messages in thread
From: Willian Rampazzo @ 2021-05-21 17:15 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Cleber Rosa, Max Reitz, Philippe Mathieu-Daudé,
	Beraldo Leal

On Wed, May 12, 2021 at 8:13 PM John Snow <jsnow@redhat.com> wrote:
>
> move python/qemu/*.py to python/qemu/[machine, qmp]/*.py and update import
> directives across the tree.
>
> This is done to create a PEP420 namespace package, in which we may
> create subpackages. To do this, the namespace directory ("qemu") should
> not have any modules in it. Those files will go into new 'machine' and 'qmp'
> subpackages instead.
>
> Implement machine/__init__.py making the top-level classes and functions
> from its various modules available directly inside the package. Change
> qmp.py to qmp/__init__.py similarly, such that all of the useful QMP
> library classes are available directly from "qemu.qmp" instead of
> "qemu.qmp.qmp".
>
> Signed-off-by: John Snow <jsnow@redhat.com>
>
>
> ---
>
> Note for reviewers: in the next patch, I add a utils sub-package and
> move qemu/machine/accel.py to qemu/utils/accel.py. If we like it that
> way, we can squash it in here if we want, or just leave it as its own
> follow-up patch, I am just leaving it as something that will be easy to
> un-do or change for now.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/{qemu => }/.isort.cfg                |  0
>  python/qemu/__init__.py                     | 11 ------
>  python/qemu/{ => machine}/.flake8           |  0
>  python/qemu/machine/__init__.py             | 41 +++++++++++++++++++++
>  python/qemu/{ => machine}/accel.py          |  0
>  python/qemu/{ => machine}/console_socket.py |  0
>  python/qemu/{ => machine}/machine.py        | 16 +++++---
>  python/qemu/{ => machine}/pylintrc          |  0
>  python/qemu/{ => machine}/qtest.py          |  3 +-
>  python/qemu/{qmp.py => qmp/__init__.py}     | 12 +++++-
>  tests/acceptance/avocado_qemu/__init__.py   |  4 +-
>  tests/acceptance/virtio-gpu.py              |  2 +-
>  tests/qemu-iotests/300                      |  4 +-
>  tests/qemu-iotests/iotests.py               |  2 +-
>  tests/vm/aarch64vm.py                       |  2 +-
>  tests/vm/basevm.py                          |  3 +-
>  16 files changed, 73 insertions(+), 27 deletions(-)
>  rename python/{qemu => }/.isort.cfg (100%)
>  delete mode 100644 python/qemu/__init__.py
>  rename python/qemu/{ => machine}/.flake8 (100%)
>  create mode 100644 python/qemu/machine/__init__.py
>  rename python/qemu/{ => machine}/accel.py (100%)
>  rename python/qemu/{ => machine}/console_socket.py (100%)
>  rename python/qemu/{ => machine}/machine.py (98%)
>  rename python/qemu/{ => machine}/pylintrc (100%)
>  rename python/qemu/{ => machine}/qtest.py (99%)
>  rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
>

This improves a lot the organization of the code.

Reviewed-by: Willian Rampazzo <willianr@redhat.com>



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 03/25] python: create utils sub-package
  2021-05-12 23:12 ` [PATCH v6 03/25] python: create utils sub-package John Snow
  2021-05-18 15:12   ` Cleber Rosa
@ 2021-05-21 17:18   ` Willian Rampazzo
  2021-05-21 17:39     ` John Snow
  1 sibling, 1 reply; 64+ messages in thread
From: Willian Rampazzo @ 2021-05-21 17:18 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Cleber Rosa, Max Reitz, Philippe Mathieu-Daudé,
	Beraldo Leal

On Wed, May 12, 2021 at 8:13 PM John Snow <jsnow@redhat.com> wrote:
>
> Create a space for miscellaneous things that don't belong strictly in
> "qemu.machine" nor "qemu.qmp" packages.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/qemu/machine/__init__.py           |  8 --------
>  python/qemu/utils/__init__.py             | 23 +++++++++++++++++++++++
>  python/qemu/{machine => utils}/accel.py   |  0
>  tests/acceptance/avocado_qemu/__init__.py |  4 ++--
>  tests/acceptance/virtio-gpu.py            |  2 +-
>  tests/vm/aarch64vm.py                     |  2 +-
>  tests/vm/basevm.py                        |  3 ++-
>  7 files changed, 29 insertions(+), 13 deletions(-)
>  create mode 100644 python/qemu/utils/__init__.py
>  rename python/qemu/{machine => utils}/accel.py (100%)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 03/25] python: create utils sub-package
  2021-05-21 17:18   ` Willian Rampazzo
@ 2021-05-21 17:39     ` John Snow
  0 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-21 17:39 UTC (permalink / raw)
  To: Willian Rampazzo
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Cleber Rosa, Max Reitz, Philippe Mathieu-Daudé,
	Beraldo Leal

On 5/21/21 1:18 PM, Willian Rampazzo wrote:
> On Wed, May 12, 2021 at 8:13 PM John Snow <jsnow@redhat.com> wrote:
>>
>> Create a space for miscellaneous things that don't belong strictly in
>> "qemu.machine" nor "qemu.qmp" packages.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/qemu/machine/__init__.py           |  8 --------
>>   python/qemu/utils/__init__.py             | 23 +++++++++++++++++++++++
>>   python/qemu/{machine => utils}/accel.py   |  0
>>   tests/acceptance/avocado_qemu/__init__.py |  4 ++--
>>   tests/acceptance/virtio-gpu.py            |  2 +-
>>   tests/vm/aarch64vm.py                     |  2 +-
>>   tests/vm/basevm.py                        |  3 ++-
>>   7 files changed, 29 insertions(+), 13 deletions(-)
>>   create mode 100644 python/qemu/utils/__init__.py
>>   rename python/qemu/{machine => utils}/accel.py (100%)
>>
> 
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> 

Thanks :) I've since squashed patches 2/3, so I will add your R-B to 
that combined patch.

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 06/25] python: add directory structure README.rst files
  2021-05-12 23:12 ` [PATCH v6 06/25] python: add directory structure README.rst files John Snow
@ 2021-05-25  2:33   ` Cleber Rosa
  2021-05-25 17:14     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25  2:33 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 5989 bytes --]

On Wed, May 12, 2021 at 07:12:22PM -0400, John Snow wrote:
> Add short readmes to python/, python/qemu/, python/qemu/machine,
> python/qemu/qmp, and python/qemu/utils that explain the directory
> hierarchy. These readmes are visible when browsing the source on
> e.g. gitlab/github and are designed to help new developers/users quickly
> make sense of the source tree.
> 
> They are not designed for inclusion in a published manual.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/README.rst              | 41 ++++++++++++++++++++++++++++++++++
>  python/qemu/README.rst         |  8 +++++++
>  python/qemu/machine/README.rst |  9 ++++++++
>  python/qemu/qmp/README.rst     |  9 ++++++++
>  python/qemu/utils/README.rst   |  7 ++++++
>  5 files changed, 74 insertions(+)
>  create mode 100644 python/README.rst
>  create mode 100644 python/qemu/README.rst
>  create mode 100644 python/qemu/machine/README.rst
>  create mode 100644 python/qemu/qmp/README.rst
>  create mode 100644 python/qemu/utils/README.rst
> 
> diff --git a/python/README.rst b/python/README.rst
> new file mode 100644
> index 00000000000..7a0dc5dff4a
> --- /dev/null
> +++ b/python/README.rst
> @@ -0,0 +1,41 @@
> +QEMU Python Tooling
> +===================
> +
> +This directory houses Python tooling used by the QEMU project to build,
> +configure, and test QEMU. It is organized by namespace (``qemu``), and
> +then by package (e.g. ``qemu/machine``, ``qemu/qmp``, etc).
> +
> +``setup.py`` is used by ``pip`` to install this tooling to the current
> +environment. ``setup.cfg`` provides the packaging configuration used by
> +setup.py in a setuptools specific format. You will generally invoke it

For consistency, ``setup.py`` here?  Also, I guess ``setuptools`` as it
falls in the same category of ``pip``.

> +by doing one of the following:
> +
> +1. ``pip3 install .`` will install these packages to your current
> +   environment. If you are inside a virtual environment, they will
> +   install there. If you are not, it will attempt to install to the
> +   global environment, which is not recommended.

Maybe some **emphasis** on **not**?

> +
> +2. ``pip3 install --user .`` will install these packages to your user's
> +   local python packages. If you are inside of a virtual environment,
> +   this will fail.
> +

Maybe note that, if you are inside of a virtual environment, option #1
will probably be what users doing "--user" in a venv actually want.

> +If you append the ``-e`` argument, pip will install in "editable" mode;
> +which installs a version of the package that installs a forwarder
> +pointing to these files, such that the package always reflects the
> +latest version in your git tree.
> +
> +See `Installing packages using pip and virtual environments
> +<https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/>`_
> +for more information.
> +
> +
> +Files in this directory
> +-----------------------
> +
> +- ``qemu/`` Python package source directory.
> +- ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
> +- ``README.rst`` you are here!
> +- ``VERSION`` contains the PEP-440 compliant version used to describe
> +  this package; it is referenced by ``setup.cfg``.
> +- ``setup.cfg`` houses setuptools package configuration.
> +- ``setup.py`` is the setuptools installer used by pip; See above.

Not only used by pip... but I understand the reason for limiting the
amount of information given here.

> diff --git a/python/qemu/README.rst b/python/qemu/README.rst
> new file mode 100644
> index 00000000000..d04943f526c
> --- /dev/null
> +++ b/python/qemu/README.rst
> @@ -0,0 +1,8 @@
> +QEMU Python Namespace
> +=====================
> +
> +This directory serves as the root of a `Python PEP 420 implicit
> +namespace package <https://www.python.org/dev/peps/pep-0420/>`_.
> +
> +Each directory below is assumed to be an installable Python package that
> +is available under the ``qemu.<package>`` namespace.
> diff --git a/python/qemu/machine/README.rst b/python/qemu/machine/README.rst
> new file mode 100644
> index 00000000000..ac2b4fffb42
> --- /dev/null
> +++ b/python/qemu/machine/README.rst
> @@ -0,0 +1,9 @@
> +qemu.machine package
> +====================
> +
> +This package provides core utilities used for testing and debugging
> +QEMU. It is used by the iotests, vm tests, acceptance tests, and several
> +other utilities in the ./scripts directory. It is not a fully-fledged
> +SDK and it is subject to change at any time.
> +
> +See the documentation in ``__init__.py`` for more information.
> diff --git a/python/qemu/qmp/README.rst b/python/qemu/qmp/README.rst
> new file mode 100644
> index 00000000000..c21951491cf
> --- /dev/null
> +++ b/python/qemu/qmp/README.rst
> @@ -0,0 +1,9 @@
> +qemu.qmp package
> +================
> +
> +This package provides a library used for connecting to and communicating
> +with QMP servers. It is used extensively by iotests, vm tests,
> +acceptance tests, and other utilities in the ./scripts directory. It is
> +not a fully-fledged SDK and is subject to change at any time.
> +
> +See the documentation in ``__init__.py`` for more information.
> diff --git a/python/qemu/utils/README.rst b/python/qemu/utils/README.rst
> new file mode 100644
> index 00000000000..975fbf4d7de
> --- /dev/null
> +++ b/python/qemu/utils/README.rst
> @@ -0,0 +1,7 @@
> +qemu.utils package
> +==================
> +
> +This package provides miscellaneous utilities used for testing and
> +debugging QEMU. It is used primarily by the vm and acceptance tests.
> +
> +See the documentation in ``__init__.py`` for more information.
> -- 
> 2.30.2
> 
> 

With the ``setup.py`` and ``setuptools`` for consistency sake
mentioned in my first comment, all other comments are suggestions, so:

Reviewed-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 07/25] python: add MANIFEST.in
  2021-05-12 23:12 ` [PATCH v6 07/25] python: add MANIFEST.in John Snow
@ 2021-05-25  2:42   ` Cleber Rosa
  2021-05-25 14:12     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25  2:42 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 1102 bytes --]

On Wed, May 12, 2021 at 07:12:23PM -0400, John Snow wrote:
> When creating a source distribution via 'python3 setup.py sdist', the
> VERSION and PACKAGE.rst files aren't bundled by default. Create a
> MANIFEST.in file that instructs the build tools to include these so that
> installation from source dists won't fail.
> 
> (This invocation is required by 'tox', as well as by the tooling needed
> to upload packages to PyPI.)
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/README.rst  | 2 ++
>  python/MANIFEST.in | 2 ++
>  2 files changed, 4 insertions(+)
>  create mode 100644 python/MANIFEST.in
>

I was about to propose mypy.ini to be included here, but given
that it's merged into setup.cfg later in this series:

Reviewed-by: Cleber Rosa <crosa@redhat.com>

---

Note to self (and to you) when generating the sdist, I get:

   ...
   package init file 'qemu/__init__.py' not found (or not a regular file)
   package init file 'dist/__init__.py' not found (or not a regular file)
   ...

Which may not be too harmful, but deserves investigation.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 11/25] python: add pylint to pipenv
  2021-05-12 23:12 ` [PATCH v6 11/25] python: add pylint to pipenv John Snow
@ 2021-05-25  3:33   ` Cleber Rosa
  0 siblings, 0 replies; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25  3:33 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On Wed, May 12, 2021 at 07:12:27PM -0400, John Snow wrote:
> We are specifying >= pylint 2.7.x for several reasons:
> 
> 1. For setup.cfg support, added in pylint 2.5.x
> 2. To specify a version that has incompatibly dropped
>    bad-whitespace checks (2.6.x)
> 3. 2.7.x fixes "unsubscriptable" warnings in Python 3.9
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/Pipfile      |   1 +
>  python/Pipfile.lock | 130 ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 131 insertions(+)
>  create mode 100644 python/Pipfile.lock
> 

Works as expected:

   $ pipenv run pylint --version 
   pylint 2.7.2
   astroid 2.5.2
   Python 3.6.13 (default, Feb 25 2021, 00:00:00) 
   [GCC 11.0.0 20210210 (Red Hat 11.0.0-0)]

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 07/25] python: add MANIFEST.in
  2021-05-25  2:42   ` Cleber Rosa
@ 2021-05-25 14:12     ` John Snow
  0 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-25 14:12 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/24/21 10:42 PM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:23PM -0400, John Snow wrote:
>> When creating a source distribution via 'python3 setup.py sdist', the
>> VERSION and PACKAGE.rst files aren't bundled by default. Create a
>> MANIFEST.in file that instructs the build tools to include these so that
>> installation from source dists won't fail.
>>
>> (This invocation is required by 'tox', as well as by the tooling needed
>> to upload packages to PyPI.)
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/README.rst  | 2 ++
>>   python/MANIFEST.in | 2 ++
>>   2 files changed, 4 insertions(+)
>>   create mode 100644 python/MANIFEST.in
>>
> 
> I was about to propose mypy.ini to be included here, but given
> that it's merged into setup.cfg later in this series:
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 
> ---
> 
> Note to self (and to you) when generating the sdist, I get:
> 
>     ...
>     package init file 'qemu/__init__.py' not found (or not a regular file)

I think this one is fine, it's just a consequence of PEP420.

>     package init file 'dist/__init__.py' not found (or not a regular file)

This is kinda weird, but it only happens if you run sdist a *second* 
time, and it has created a dist/ folder. It shouldn't be trying to 
package its own dist folder ... !

After a 60 second look I don't know how to rectify this behavior, I will 
look into it.

>     ...
> 
> Which may not be too harmful, but deserves investigation.
> 

fwiw clean run vs re-run:

--- run1	2021-05-25 10:10:37.166975879 -0400
+++ run2	2021-05-25 10:10:48.527966676 -0400
@@ -1,11 +1,10 @@
  running sdist
  running egg_info
-creating qemu.egg-info
  writing qemu.egg-info/PKG-INFO
  writing dependency_links to qemu.egg-info/dependency_links.txt
  writing top-level names to qemu.egg-info/top_level.txt
-writing manifest file 'qemu.egg-info/SOURCES.txt'
  package init file 'qemu/__init__.py' not found (or not a regular file)
+package init file 'dist/__init__.py' not found (or not a regular file)
  reading manifest file 'qemu.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'qemu.egg-info/SOURCES.txt'
@@ -35,6 +34,5 @@
  copying qemu/utils/__init__.py -> qemu-0.6.1.0a1/qemu/utils
  copying qemu/utils/accel.py -> qemu-0.6.1.0a1/qemu/utils
  Writing qemu-0.6.1.0a1/setup.cfg
-creating dist
  Creating tar archive
  removing 'qemu-0.6.1.0a1' (and everything under it)



at the very least, it doesn't look like it does anything actively 
harmful, but it seems to want to find sources in the dist, so I should 
steer it towards exclusively the qemu/ folder.

Thanks for this, Cleber!

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 13/25] python: add excluded dirs to flake8 config
  2021-05-12 23:12 ` [PATCH v6 13/25] python: add excluded dirs to flake8 config John Snow
@ 2021-05-25 15:50   ` Cleber Rosa
  2021-05-25 17:18     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 15:50 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

On Wed, May 12, 2021 at 07:12:29PM -0400, John Snow wrote:
> Instruct flake8 to avoid certain well-known directories created by
> python tooling that it ought not check.
> 
> Note that at-present, nothing actually creates a ".venv" directory; but
> it is in such widespread usage as a de-facto location for a developer's
> virtual environment that it should be excluded anyway. A forthcoming
> commit canonizes this with a "make venv" command.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/setup.cfg | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/python/setup.cfg b/python/setup.cfg
> index 9ecb2902006..f21a1c42fc0 100644
> --- a/python/setup.cfg
> +++ b/python/setup.cfg
> @@ -21,6 +21,8 @@ packages = find_namespace:
>  
>  [flake8]
>  extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
> +exclude = __pycache__,
> +          .venv,
>

Given that the default set of exclusions (version control system
files) are not expected here, it LGTM to reset it with these.

Reviewed-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 15/25] python: move mypy.ini into setup.cfg
  2021-05-12 23:12 ` [PATCH v6 15/25] python: move mypy.ini into setup.cfg John Snow
@ 2021-05-25 15:52   ` Cleber Rosa
  0 siblings, 0 replies; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 15:52 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]

On Wed, May 12, 2021 at 07:12:31PM -0400, John Snow wrote:
> mypy supports reading its configuration values from a central project
> configuration file; do so.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/mypy.ini  | 4 ----
>  python/setup.cfg | 5 +++++
>  2 files changed, 5 insertions(+), 4 deletions(-)
>  delete mode 100644 python/mypy.ini

Reviewed-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 18/25] python/qemu: add isort to pipenv
  2021-05-12 23:12 ` [PATCH v6 18/25] python/qemu: add isort to pipenv John Snow
@ 2021-05-25 15:56   ` Cleber Rosa
  2021-05-25 17:21     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 15:56 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

On Wed, May 12, 2021 at 07:12:34PM -0400, John Snow wrote:
> isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
> certain "from ..." clauses that are not related to imports.
> 
> isort < 5.1.1 has a bug where it does not handle comments near import
> statements correctly.
> 
> Require 5.1.2 or greater.
> 
> isort can be run with 'isort -c qemu' from the python root.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/Pipfile      | 1 +
>  python/Pipfile.lock | 4 ++--
>  2 files changed, 3 insertions(+), 2 deletions(-)

Reviewed-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 20/25] python: add devel package requirements to setuptools
  2021-05-12 23:12 ` [PATCH v6 20/25] python: add devel package requirements to setuptools John Snow
@ 2021-05-25 16:13   ` Cleber Rosa
  2021-05-25 17:43     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 16:13 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

On Wed, May 12, 2021 at 07:12:36PM -0400, John Snow wrote:
> setuptools doesn't have a formal understanding of development requires,
> but it has an optional feataures section. Fine; add a "devel" feature
> and add the requirements to it.
> 
> To avoid duplication, we can modify pipenv to install qemu[devel]
> instead. This enables us to run invocations like "pip install -e
> .[devel]" and test the package on bleeding-edge packages beyond those
> specified in Pipfile.lock.
> 
> Importantly, this also allows us to install the qemu development
> packages in a non-networked mode: `pip3 install --no-index -e .[devel]`
> will now fail if the proper development dependencies are not already
> met. This can be useful for automated build scripts where fetching
> network packages may be undesirable.
>

This is a fairly exotic feature of setuptools, with very very few
packages that I know about using it.  With most users (I believe)
relying on pipenv to get the exact packages, the setuptools/pip use
case may fall into obscurity IMO.

So my suggestion is: consider better exposing the fact that this is
available (a documentation section perhaps).

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/Pipfile      |  5 +----
>  python/Pipfile.lock | 14 +++++++++-----
>  python/setup.cfg    |  9 +++++++++
>  3 files changed, 19 insertions(+), 9 deletions(-)
>

Either way,

Reviewed-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 06/25] python: add directory structure README.rst files
  2021-05-25  2:33   ` Cleber Rosa
@ 2021-05-25 17:14     ` John Snow
  2021-05-25 20:30       ` Cleber Rosa
  0 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-25 17:14 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/24/21 10:33 PM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:22PM -0400, John Snow wrote:
>> Add short readmes to python/, python/qemu/, python/qemu/machine,
>> python/qemu/qmp, and python/qemu/utils that explain the directory
>> hierarchy. These readmes are visible when browsing the source on
>> e.g. gitlab/github and are designed to help new developers/users quickly
>> make sense of the source tree.
>>
>> They are not designed for inclusion in a published manual.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/README.rst              | 41 ++++++++++++++++++++++++++++++++++
>>   python/qemu/README.rst         |  8 +++++++
>>   python/qemu/machine/README.rst |  9 ++++++++
>>   python/qemu/qmp/README.rst     |  9 ++++++++
>>   python/qemu/utils/README.rst   |  7 ++++++
>>   5 files changed, 74 insertions(+)
>>   create mode 100644 python/README.rst
>>   create mode 100644 python/qemu/README.rst
>>   create mode 100644 python/qemu/machine/README.rst
>>   create mode 100644 python/qemu/qmp/README.rst
>>   create mode 100644 python/qemu/utils/README.rst
>>
>> diff --git a/python/README.rst b/python/README.rst
>> new file mode 100644
>> index 00000000000..7a0dc5dff4a
>> --- /dev/null
>> +++ b/python/README.rst
>> @@ -0,0 +1,41 @@
>> +QEMU Python Tooling
>> +===================
>> +
>> +This directory houses Python tooling used by the QEMU project to build,
>> +configure, and test QEMU. It is organized by namespace (``qemu``), and
>> +then by package (e.g. ``qemu/machine``, ``qemu/qmp``, etc).
>> +
>> +``setup.py`` is used by ``pip`` to install this tooling to the current
>> +environment. ``setup.cfg`` provides the packaging configuration used by
>> +setup.py in a setuptools specific format. You will generally invoke it
> 
> For consistency, ``setup.py`` here?  Also, I guess ``setuptools`` as it
> falls in the same category of ``pip``.
> 

Kinda-sorta, but `pip` is a command line executable and setuptools 
isn't. Along those lines I'll fix setup.py but leave setuptools as-is.

>> +by doing one of the following:
>> +
>> +1. ``pip3 install .`` will install these packages to your current
>> +   environment. If you are inside a virtual environment, they will
>> +   install there. If you are not, it will attempt to install to the
>> +   global environment, which is not recommended.
> 
> Maybe some **emphasis** on **not**?
> 

Sure :)

>> +
>> +2. ``pip3 install --user .`` will install these packages to your user's
>> +   local python packages. If you are inside of a virtual environment,
>> +   this will fail.
>> +
> 
> Maybe note that, if you are inside of a virtual environment, option #1
> will probably be what users doing "--user" in a venv actually want.
> 

Yes. It's frequently annoying how this works, because it's hard to relay 
succinctly :)

I think at least newer versions of pip give you good warnings when you 
use --user for virtual environments at least.

>> +If you append the ``-e`` argument, pip will install in "editable" mode;
>> +which installs a version of the package that installs a forwarder
>> +pointing to these files, such that the package always reflects the
>> +latest version in your git tree.
>> +
>> +See `Installing packages using pip and virtual environments
>> +<https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/>`_
>> +for more information.
>> +
>> +
>> +Files in this directory
>> +-----------------------
>> +
>> +- ``qemu/`` Python package source directory.
>> +- ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
>> +- ``README.rst`` you are here!
>> +- ``VERSION`` contains the PEP-440 compliant version used to describe
>> +  this package; it is referenced by ``setup.cfg``.
>> +- ``setup.cfg`` houses setuptools package configuration.
>> +- ``setup.py`` is the setuptools installer used by pip; See above.
> 
> Not only used by pip... but I understand the reason for limiting the
> amount of information given here.
> 

Yes ... suggesting broadly that I don't really support using 
setuptools/setup.py alone to install the package, but instead expect and 
consider 'pip' to be the canonical/supported interface.

There are sometimes minor differences between how they handle things, so 
I wanted less emphasis on setuptools et al.

>> diff --git a/python/qemu/README.rst b/python/qemu/README.rst
>> new file mode 100644
>> index 00000000000..d04943f526c
>> --- /dev/null
>> +++ b/python/qemu/README.rst
>> @@ -0,0 +1,8 @@
>> +QEMU Python Namespace
>> +=====================
>> +
>> +This directory serves as the root of a `Python PEP 420 implicit
>> +namespace package <https://www.python.org/dev/peps/pep-0420/>`_.
>> +
>> +Each directory below is assumed to be an installable Python package that
>> +is available under the ``qemu.<package>`` namespace.
>> diff --git a/python/qemu/machine/README.rst b/python/qemu/machine/README.rst
>> new file mode 100644
>> index 00000000000..ac2b4fffb42
>> --- /dev/null
>> +++ b/python/qemu/machine/README.rst
>> @@ -0,0 +1,9 @@
>> +qemu.machine package
>> +====================
>> +
>> +This package provides core utilities used for testing and debugging
>> +QEMU. It is used by the iotests, vm tests, acceptance tests, and several
>> +other utilities in the ./scripts directory. It is not a fully-fledged
>> +SDK and it is subject to change at any time.
>> +
>> +See the documentation in ``__init__.py`` for more information.
>> diff --git a/python/qemu/qmp/README.rst b/python/qemu/qmp/README.rst
>> new file mode 100644
>> index 00000000000..c21951491cf
>> --- /dev/null
>> +++ b/python/qemu/qmp/README.rst
>> @@ -0,0 +1,9 @@
>> +qemu.qmp package
>> +================
>> +
>> +This package provides a library used for connecting to and communicating
>> +with QMP servers. It is used extensively by iotests, vm tests,
>> +acceptance tests, and other utilities in the ./scripts directory. It is
>> +not a fully-fledged SDK and is subject to change at any time.
>> +
>> +See the documentation in ``__init__.py`` for more information.
>> diff --git a/python/qemu/utils/README.rst b/python/qemu/utils/README.rst
>> new file mode 100644
>> index 00000000000..975fbf4d7de
>> --- /dev/null
>> +++ b/python/qemu/utils/README.rst
>> @@ -0,0 +1,7 @@
>> +qemu.utils package
>> +==================
>> +
>> +This package provides miscellaneous utilities used for testing and
>> +debugging QEMU. It is used primarily by the vm and acceptance tests.
>> +
>> +See the documentation in ``__init__.py`` for more information.
>> -- 
>> 2.30.2
>>
>>
> 
> With the ``setup.py`` and ``setuptools`` for consistency sake
> mentioned in my first comment, all other comments are suggestions, so:
> 

I took one of those two review comments, and acted on one of two of your 
suggestions.

> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 

So technically I didn't meet your criteria for taking this RB :) Let me 
know if I can still apply it:

diff --git a/python/README.rst b/python/README.rst
index 7a0dc5dff4a..38b0c83f321 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -7,17 +7,17 @@ then by package (e.g. ``qemu/machine``, ``qemu/qmp``, 
etc).

  ``setup.py`` is used by ``pip`` to install this tooling to the current
  environment. ``setup.cfg`` provides the packaging configuration used by
-setup.py in a setuptools specific format. You will generally invoke it
-by doing one of the following:
+``setup.py`` in a setuptools specific format. You will generally invoke
+it by doing one of the following:

  1. ``pip3 install .`` will install these packages to your current
     environment. If you are inside a virtual environment, they will
     install there. If you are not, it will attempt to install to the
-   global environment, which is not recommended.
+   global environment, which is **not recommended**.

  2. ``pip3 install --user .`` will install these packages to your user's
     local python packages. If you are inside of a virtual environment,
-   this will fail.
+   this will fail; you likely want the first invocation above.

  If you append the ``-e`` argument, pip will install in "editable" mode;
  which installs a version of the package that installs a forwarder



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 13/25] python: add excluded dirs to flake8 config
  2021-05-25 15:50   ` Cleber Rosa
@ 2021-05-25 17:18     ` John Snow
  0 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-25 17:18 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/25/21 11:50 AM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:29PM -0400, John Snow wrote:
>> Instruct flake8 to avoid certain well-known directories created by
>> python tooling that it ought not check.
>>
>> Note that at-present, nothing actually creates a ".venv" directory; but
>> it is in such widespread usage as a de-facto location for a developer's
>> virtual environment that it should be excluded anyway. A forthcoming
>> commit canonizes this with a "make venv" command.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/setup.cfg | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/python/setup.cfg b/python/setup.cfg
>> index 9ecb2902006..f21a1c42fc0 100644
>> --- a/python/setup.cfg
>> +++ b/python/setup.cfg
>> @@ -21,6 +21,8 @@ packages = find_namespace:
>>   
>>   [flake8]
>>   extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
>> +exclude = __pycache__,
>> +          .venv,
>>
> 
> Given that the default set of exclusions (version control system
> files) are not expected here, it LGTM to reset it with these.
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 

Yep. Admittedly, I just excluded things that caused me trouble in 
practice. I could narrow the flake8 invocation to specify *only* the 
qemu/ folder, but in the event we added Python tests or other scripts I 
actually felt like leaving it "open" would be fine.

(This way, flake8 runs like how I expect it to by just running "flake8" 
with no arguments, which I saw as nice.)

The goal was something like: the less specific I make the flake8 
invocation, the better, even though I expect the linter to be run only 
by "make check" eventually, as a courtesy to mid-advanced users, keeping 
those canonical invocations simple felt good.

Thanks!

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 18/25] python/qemu: add isort to pipenv
  2021-05-25 15:56   ` Cleber Rosa
@ 2021-05-25 17:21     ` John Snow
  2021-05-25 20:34       ` Cleber Rosa
  0 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-25 17:21 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/25/21 11:56 AM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:34PM -0400, John Snow wrote:
>> isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
>> certain "from ..." clauses that are not related to imports.
>>
>> isort < 5.1.1 has a bug where it does not handle comments near import
>> statements correctly.
>>
>> Require 5.1.2 or greater.
>>
>> isort can be run with 'isort -c qemu' from the python root.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/Pipfile      | 1 +
>>   python/Pipfile.lock | 4 ++--
>>   2 files changed, 3 insertions(+), 2 deletions(-)
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 

Thanks. I have also updated the commit message a little bit:

isort can be run (in "check" mode) with 'isort -c qemu' from the python
root. isort can also be used to fix/rewrite import order automatically
by using 'isort qemu'.

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 20/25] python: add devel package requirements to setuptools
  2021-05-25 16:13   ` Cleber Rosa
@ 2021-05-25 17:43     ` John Snow
  2021-05-25 20:38       ` Cleber Rosa
  0 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-25 17:43 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/25/21 12:13 PM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:36PM -0400, John Snow wrote:
>> setuptools doesn't have a formal understanding of development requires,
>> but it has an optional feataures section. Fine; add a "devel" feature
>> and add the requirements to it.
>>
>> To avoid duplication, we can modify pipenv to install qemu[devel]
>> instead. This enables us to run invocations like "pip install -e
>> .[devel]" and test the package on bleeding-edge packages beyond those
>> specified in Pipfile.lock.
>>
>> Importantly, this also allows us to install the qemu development
>> packages in a non-networked mode: `pip3 install --no-index -e .[devel]`
>> will now fail if the proper development dependencies are not already
>> met. This can be useful for automated build scripts where fetching
>> network packages may be undesirable.
>>
> 
> This is a fairly exotic feature of setuptools, with very very few
> packages that I know about using it.  With most users (I believe)
> relying on pipenv to get the exact packages, the setuptools/pip use
> case may fall into obscurity IMO.
> 

Fair enough.

The intent is:

- Pipenv is more for CI, to deploy a consistent set of frozen packages 
that are known to behave in an extremely stable manner. My hope is to 
avoid breaking changes introduced unknowingly by pylint et al.

- pip install qemu[devel] is intended more for external/normal use by 
developers. It grabs the latest and greatest and it may indeed break as 
dependencies change beyond my awareness.


Some packages like aiohttp use that optional dependency feature to 
install optional modules -- `pip install aiohttp[speedups]` installs 
optional dependencies that allow that module to work much faster, but 
aren't required.

Since these linting tools aren't *required* just to *use* the package, I 
am doing users a courtesy by listing them as optional. That way, they 
aren't pulled in when using "pip install qemu", and if I have to pin on 
specific sub-versions etc, it won't include conflict dependencies for 
people using other projects that DO declare a hard requirement on those 
packages.

I can amend the PACKAGE.rst file to mention this usage, though it's only 
useful for folks developing the package.

(Still, part of the ploy here is to attract outside help on developing 
the QEMU SDK, pull requests welcome etc, so it's worth a documentation 
blurb for now.)

> So my suggestion is: consider better exposing the fact that this is
> available (a documentation section perhaps).
> 
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/Pipfile      |  5 +----
>>   python/Pipfile.lock | 14 +++++++++-----
>>   python/setup.cfg    |  9 +++++++++
>>   3 files changed, 19 insertions(+), 9 deletions(-)
>>
> 
> Either way,
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 

Thanks! I am taking your R-B and I have applied the following diff.

Note that the PACKAGE.rst blurb references qemu[devel] instead because 
the PACKAGE.rst file is what is displayed theoretically on PyPI. That 
exact invocation will fail currently, because it's not on PyPI yet.

A little weird, but I *think* it's correct.


diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
index 1bbfe1b58e2..05ea7789fc1 100644
--- a/python/PACKAGE.rst
+++ b/python/PACKAGE.rst
@@ -31,3 +31,7 @@ official `GitLab mirror 
<https://gitlab.com/qemu-project/qemu>`_.
  Please report bugs on the `QEMU issue tracker
  <https://gitlab.com/qemu-project/qemu/-/issues>`_ and tag ``@jsnow`` in
  the report.
+
+Optional packages necessary for running code quality analysis for this
+package can be installed with the optional dependency group "devel":
+``pip install qemu[devel]``.
diff --git a/python/README.rst b/python/README.rst
index bf9bbca979a..954870973d0 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -24,6 +24,10 @@ which installs a version of the package that installs 
a forwarder
  pointing to these files, such that the package always reflects the
  latest version in your git tree.

+Installing ".[devel]" instead of "." will additionally pull in required
+packages for testing this package. They are not runtime requirements,
+and are not needed to simply use these libraries.
+
  See `Installing packages using pip and virtual environments
 
<https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/>`_
  for more information.



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 21/25] python: add avocado-framework and tests
  2021-05-12 23:12 ` [PATCH v6 21/25] python: add avocado-framework and tests John Snow
@ 2021-05-25 18:58   ` Cleber Rosa
  0 siblings, 0 replies; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 18:58 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 2689 bytes --]

On Wed, May 12, 2021 at 07:12:37PM -0400, John Snow wrote:
> Try using avocado to manage our various tests; even though right now
> they're only invoking shell scripts and not really running any
> python-native code.
> 
> Create tests/, and add shell scripts which call out to mypy, flake8,
> pylint and isort to enforce the standards in this directory.
> 
> Add avocado-framework to the setup.cfg development dependencies, and add
> avocado.cfg to store some preferences for how we'd like the test output
> to look.
> 
> Finally, add avocado-framework to the Pipfile environment and lock the
> new dependencies. We are using avocado >= 87.0 here to take advantage of
> some features that Cleber has helpfully added to make the test output
> here *very* friendly and easy to read for developers that might chance
> upon the output in Gitlab CI.
> 
> [Note: ALL of the dependencies get updated to the most modern versions
> that exist at the time of this writing. No way around it that I have
> seen. Not ideal, but so it goes.]
> 
> Provided you have the right development dependencies (mypy, flake8,
> isort, pylint, and now avocado-framework) You should be able to run
> "avocado --config avocado.cfg run tests/" from the python folder to run
> all of these linters with the correct arguments.
> 
> (A forthcoming commit adds the much easier 'make check'.)
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/README.rst      |   2 +
>  python/Pipfile.lock    | 104 ++++++++++++++++++++++-------------------
>  python/avocado.cfg     |  10 ++++
>  python/setup.cfg       |   1 +
>  python/tests/flake8.sh |   2 +
>  python/tests/isort.sh  |   2 +
>  python/tests/mypy.sh   |   2 +
>  python/tests/pylint.sh |   2 +
>  8 files changed, 77 insertions(+), 48 deletions(-)
>  create mode 100644 python/avocado.cfg
>  create mode 100755 python/tests/flake8.sh
>  create mode 100755 python/tests/isort.sh
>  create mode 100755 python/tests/mypy.sh
>  create mode 100755 python/tests/pylint.sh
> 

With the patches from your "Python: delint python library" series:

   $ pipenv run avocado --config avocado.cfg run tests/
   JOB ID     : b27b48eded8b405c6672e61e3d1561407fca9d5e
   JOB LOG    : /home/cleber/avocado/job-results/job-2021-05-25T14.56-b27b48e/job.log
    (1/4) tests/flake8.sh: PASS (0.67 s)
    (2/4) tests/isort.sh: PASS (0.37 s)
    (3/4) tests/mypy.sh: PASS (0.39 s)
    (4/4) tests/pylint.sh: PASS (4.85 s)
   RESULTS    : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB TIME   : 6.75 s

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 22/25] python: add Makefile for some common tasks
  2021-05-12 23:12 ` [PATCH v6 22/25] python: add Makefile for some common tasks John Snow
@ 2021-05-25 19:24   ` Cleber Rosa
  2021-05-25 19:45     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 19:24 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 4014 bytes --]

On Wed, May 12, 2021 at 07:12:38PM -0400, John Snow wrote:
> Add "make venv" to create the pipenv-managed virtual environment that
> contains our explicitly pinned dependencies.
> 
> Add "make check" to run the python linters [in the host execution
> environment].
> 
> Add "make venv-check" which combines the above two: create/update the
> venv, then run the linters in that explicitly managed environment.
> 
> Add "make develop" which canonizes the runes needed to get both the
> linting pre-requisites (the "[devel]" part), and the editable
> live-install (the "-e" part) of these python libraries.
> 
> make clean: delete miscellaneous python packaging output possibly
> created by pipenv, pip, or other python packaging utilities
> 
> make distclean: delete the above, the .venv, and the editable "qemu"
> package forwarder (qemu.egg-info) if there is one.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/README.rst |  3 +++
>  python/Makefile   | 42 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 45 insertions(+)
>  create mode 100644 python/Makefile
> 
> diff --git a/python/README.rst b/python/README.rst
> index e107bd12a69..3e09d20c23c 100644
> --- a/python/README.rst
> +++ b/python/README.rst
> @@ -35,6 +35,9 @@ Files in this directory
>  - ``qemu/`` Python package source directory.
>  - ``tests/`` Python package tests directory.
>  - ``avocado.cfg`` Configuration for the Avocado test-runner.
> +  Used by ``make check`` et al.
> +- ``Makefile`` provides some common testing/installation invocations.
> +  Try ``make help`` to see available targets.
>  - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
>    that should be included by a source distribution.
>  - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
> diff --git a/python/Makefile b/python/Makefile
> new file mode 100644
> index 00000000000..184f59e5634
> --- /dev/null
> +++ b/python/Makefile
> @@ -0,0 +1,42 @@
> +.PHONY: help venv venv-check check clean distclean develop
> +
> +help:
> +	@echo "python packaging help:"
> +	@echo ""
> +	@echo "make venv:       Create pipenv's virtual environment."
> +	@echo "    NOTE: Requires Python 3.6 and pipenv."
> +	@echo "          Will download packages from PyPI."
> +	@echo "    Hint: (On Fedora): 'sudo dnf install python36 pipenv'"
> +	@echo ""
> +	@echo "make venv-check: run linters using pipenv's virtual environment."
> +	@echo "    Hint: If you don't know which test to run, run this one!"
> +	@echo ""
> +	@echo "make develop:    Install deps for 'make check', and"
> +	@echo "                 the qemu libs in editable/development mode."
> +	@echo ""
> +	@echo "make check:      run linters using the current environment."
> +	@echo ""

Let's observe how this will be used (or misused).  I fear most people
will jump into `make check`, even though you have described `make
venv-check` as the primary choice.

We have a precedent with `make check-acceptance` that will create a
venv and use it by default, so we can consider that as a fallback
strategy based on user feedback.

> +	@echo "make clean:      remove build output."
> +	@echo ""
> +	@echo "make distclean:  remove venv files, qemu package forwarder, and"
> +	@echo "                 everything from 'make clean'."
> +
> +venv: .venv
> +.venv: Pipfile.lock
> +	@PIPENV_VENV_IN_PROJECT=1 pipenv sync --dev --keep-outdated
> +	@touch .venv
> +
> +venv-check: venv
> +	@pipenv run make check
> +
> +develop:
> +	pip3 install -e .[devel]
> +
> +check:
> +	@avocado --config avocado.cfg run tests/
> +
> +clean:
> +	rm -rf build/ dist/
> +

Usually `python3 setup.py clean --all` would be the better choice here,
but, it doesn't clean `dist/`, so I'm OK with this.

> +distclean: clean
> +	rm -rf qemu.egg-info/ .venv/
> -- 
> 2.30.2
> 
> 

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 23/25] python: add .gitignore
  2021-05-12 23:12 ` [PATCH v6 23/25] python: add .gitignore John Snow
@ 2021-05-25 19:36   ` Cleber Rosa
  2021-05-25 20:10     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 19:36 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

On Wed, May 12, 2021 at 07:12:39PM -0400, John Snow wrote:
> Ignore *Python* build and package output (build, dist, qemu.egg-info);
> these files are not created as part of a QEMU build.
> 
> Ignore miscellaneous cached python confetti (__pycache__, *.pyc,
> .mypy_cache).
> 
> Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/.gitignore | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
>  create mode 100644 python/.gitignore
> 
> diff --git a/python/.gitignore b/python/.gitignore
> new file mode 100644
> index 00000000000..e27c99e009c
> --- /dev/null
> +++ b/python/.gitignore
> @@ -0,0 +1,19 @@
> +# python bytecode cache
> +*.pyc

This is a duplicate from the parent .gitignore, so I would avoid it.

> +__pycache__/

And this one is interesting because, the only thing that *should* be
in __pycache__ dirs is .pyc files (covered by the parent .gitignore
file).

So, I get the same behavior without these two entries here, so I would
skip them.  Let me know if you have any reason for explicitly
including them.

- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 22/25] python: add Makefile for some common tasks
  2021-05-25 19:24   ` Cleber Rosa
@ 2021-05-25 19:45     ` John Snow
  2021-05-25 20:39       ` Cleber Rosa
  0 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-25 19:45 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/25/21 3:24 PM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:38PM -0400, John Snow wrote:
>> Add "make venv" to create the pipenv-managed virtual environment that
>> contains our explicitly pinned dependencies.
>>
>> Add "make check" to run the python linters [in the host execution
>> environment].
>>
>> Add "make venv-check" which combines the above two: create/update the
>> venv, then run the linters in that explicitly managed environment.
>>
>> Add "make develop" which canonizes the runes needed to get both the
>> linting pre-requisites (the "[devel]" part), and the editable
>> live-install (the "-e" part) of these python libraries.
>>
>> make clean: delete miscellaneous python packaging output possibly
>> created by pipenv, pip, or other python packaging utilities
>>
>> make distclean: delete the above, the .venv, and the editable "qemu"
>> package forwarder (qemu.egg-info) if there is one.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/README.rst |  3 +++
>>   python/Makefile   | 42 ++++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 45 insertions(+)
>>   create mode 100644 python/Makefile
>>
>> diff --git a/python/README.rst b/python/README.rst
>> index e107bd12a69..3e09d20c23c 100644
>> --- a/python/README.rst
>> +++ b/python/README.rst
>> @@ -35,6 +35,9 @@ Files in this directory
>>   - ``qemu/`` Python package source directory.
>>   - ``tests/`` Python package tests directory.
>>   - ``avocado.cfg`` Configuration for the Avocado test-runner.
>> +  Used by ``make check`` et al.
>> +- ``Makefile`` provides some common testing/installation invocations.
>> +  Try ``make help`` to see available targets.
>>   - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
>>     that should be included by a source distribution.
>>   - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
>> diff --git a/python/Makefile b/python/Makefile
>> new file mode 100644
>> index 00000000000..184f59e5634
>> --- /dev/null
>> +++ b/python/Makefile
>> @@ -0,0 +1,42 @@
>> +.PHONY: help venv venv-check check clean distclean develop
>> +
>> +help:
>> +	@echo "python packaging help:"
>> +	@echo ""
>> +	@echo "make venv:       Create pipenv's virtual environment."
>> +	@echo "    NOTE: Requires Python 3.6 and pipenv."
>> +	@echo "          Will download packages from PyPI."
>> +	@echo "    Hint: (On Fedora): 'sudo dnf install python36 pipenv'"
>> +	@echo ""
>> +	@echo "make venv-check: run linters using pipenv's virtual environment."
>> +	@echo "    Hint: If you don't know which test to run, run this one!"
>> +	@echo ""
>> +	@echo "make develop:    Install deps for 'make check', and"
>> +	@echo "                 the qemu libs in editable/development mode."
>> +	@echo ""
>> +	@echo "make check:      run linters using the current environment."
>> +	@echo ""
> 
> Let's observe how this will be used (or misused).  I fear most people
> will jump into `make check`, even though you have described `make
> venv-check` as the primary choice.
> 
> We have a precedent with `make check-acceptance` that will create a
> venv and use it by default, so we can consider that as a fallback
> strategy based on user feedback.
> 

Right, I see. Though, I did intentionally want to make it clear which of 
these invocations created an environment and which did not.

Unlike the acceptance tests, it might make sense to run these tests both 
inside and outside of that venv, so I opted to make the default "make" 
target "make help".

The Gitlab CI will run the right one, after all -- and I do still expect 
the regular 'make check' to pass, so I am not as sure that it's a 
crucial failure if someone runs the "wrong one".

>> +	@echo "make clean:      remove build output."
>> +	@echo ""
>> +	@echo "make distclean:  remove venv files, qemu package forwarder, and"
>> +	@echo "                 everything from 'make clean'."
>> +
>> +venv: .venv
>> +.venv: Pipfile.lock
>> +	@PIPENV_VENV_IN_PROJECT=1 pipenv sync --dev --keep-outdated
>> +	@touch .venv
>> +
>> +venv-check: venv
>> +	@pipenv run make check
>> +
>> +develop:
>> +	pip3 install -e .[devel]
>> +
>> +check:
>> +	@avocado --config avocado.cfg run tests/
>> +
>> +clean:
>> +	rm -rf build/ dist/
>> +
> 
> Usually `python3 setup.py clean --all` would be the better choice here,
> but, it doesn't clean `dist/`, so I'm OK with this.
> 

Hm, I should probably move the 'dist' down into 'distclean' anyway, and 
I will replace the clean invocation with the one you suggest.

>> +distclean: clean
>> +	rm -rf qemu.egg-info/ .venv/
>> -- 
>> 2.30.2
>>
>>
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> 



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 24/25] gitlab: add python linters to CI
  2021-05-12 23:12 ` [PATCH v6 24/25] gitlab: add python linters to CI John Snow
@ 2021-05-25 19:55   ` Cleber Rosa
  2021-05-25 20:33     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 19:55 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 2193 bytes --]

On Wed, May 12, 2021 at 07:12:40PM -0400, John Snow wrote:
> Add python3.6 to the fedora container image: we need it to run the
> linters against that explicit version to make sure we don't break our
> minimum version promise.
> 
> Add pipenv so that we can fetch precise versions of pip packages we need
> to guarantee test reproducability.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  .gitlab-ci.yml                         | 12 ++++++++++++
>  tests/docker/dockerfiles/fedora.docker |  2 ++
>  2 files changed, 14 insertions(+)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index dcb6317aace..a371c0c7163 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -779,6 +779,18 @@ check-patch:
>      GIT_DEPTH: 1000
>    allow_failure: true
>  
> +
> +check-python:
> +  stage: test
> +  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
> +  script:
> +    - cd python
> +    - make venv-check
> +  variables:
> +    GIT_DEPTH: 1000
> +  needs:
> +    job: amd64-fedora-container
> +
>  check-dco:
>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
> index 915fdc1845e..6908d69ac37 100644
> --- a/tests/docker/dockerfiles/fedora.docker
> +++ b/tests/docker/dockerfiles/fedora.docker
> @@ -84,6 +84,7 @@ ENV PACKAGES \
>      numactl-devel \
>      perl \
>      perl-Test-Harness \
> +    pipenv \
>      pixman-devel \
>      python3 \
>      python3-PyYAML \
> @@ -93,6 +94,7 @@ ENV PACKAGES \
>      python3-pip \
>      python3-sphinx \
>      python3-virtualenv \
> +    python3.6 \

I personally would prefer having a different container image for this
job.  Because it would:

1. Be super simple (FROM fedora:33 / dnf -y install python3.6 pipenv)
2. Not carry all this unnecessary baggage
3. Not risk building QEMU with Python 3.6 (suppose the ./configure
   probe changes unintentionally)

But, AFAICT there is no precedent in requiring new images for
different types of checks.  So, unless someone else complains loudly,
I'm OK with this.

Reviewed-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 23/25] python: add .gitignore
  2021-05-25 19:36   ` Cleber Rosa
@ 2021-05-25 20:10     ` John Snow
  2021-05-25 20:42       ` Cleber Rosa
  0 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-25 20:10 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/25/21 3:36 PM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:39PM -0400, John Snow wrote:
>> Ignore *Python* build and package output (build, dist, qemu.egg-info);
>> these files are not created as part of a QEMU build.
>>
>> Ignore miscellaneous cached python confetti (__pycache__, *.pyc,
>> .mypy_cache).
>>
>> Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/.gitignore | 19 +++++++++++++++++++
>>   1 file changed, 19 insertions(+)
>>   create mode 100644 python/.gitignore
>>
>> diff --git a/python/.gitignore b/python/.gitignore
>> new file mode 100644
>> index 00000000000..e27c99e009c
>> --- /dev/null
>> +++ b/python/.gitignore
>> @@ -0,0 +1,19 @@
>> +# python bytecode cache
>> +*.pyc
> 
> This is a duplicate from the parent .gitignore, so I would avoid it.
> 
>> +__pycache__/
> 
> And this one is interesting because, the only thing that *should* be
> in __pycache__ dirs is .pyc files (covered by the parent .gitignore
> file).
> 
> So, I get the same behavior without these two entries here, so I would
> skip them.  Let me know if you have any reason for explicitly
> including them.
> 
> - Cleber.
> 

Hm, not really ... Just completeness, I suppose, since this directory is 
becoming increasingly separate from the rest of the tree.

It isn't crucial, it just seemed like a weird omission if they weren't 
listed here. *shrug*

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 25/25] python: add tox support
  2021-05-12 23:12 ` [PATCH v6 25/25] python: add tox support John Snow
@ 2021-05-25 20:15   ` Cleber Rosa
  2021-05-25 20:25     ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 20:15 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

On Wed, May 12, 2021 at 07:12:41PM -0400, John Snow wrote:
> This is intended to be a manually run, non-CI script.
> 
> Use tox to test the linters against all python versions from 3.6 to
> 3.9. This will only work if you actually have those versions installed
> locally, but Fedora makes this easy:
> 
> > sudo dnf install python36 python37 python38 python39
> 
> Unlike the pipenv tests (make venv-check), this pulls "whichever"
> versions of the python packages, so they are unpinned and may break as
> time goes on. In the case that breakages are found, setup.cfg should be
> amended accordingly to avoid the bad dependant versions, or the code
> should be amended to work around the issue.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/README.rst |  2 ++
>  python/.gitignore |  1 +
>  python/Makefile   |  7 ++++++-
>  python/setup.cfg  |  1 +
>  python/tox.ini    | 13 +++++++++++++
>  5 files changed, 23 insertions(+), 1 deletion(-)
>  create mode 100644 python/tox.ini
> 

This works as intended for me.  A couple of notes / suggestions
for future improvements:

 * `dnf install tox` pulled all the Python versions available (I
   assume as suggestions) automatically

 * tox.ini can be folded into setup.cfg

 * a custom container image with all those Python versions may be
   handy for running both the pipenv based job (along with the
   suggestions on the previous patch) and an on-demand,
   "allow_failure" tox based CI job.

Other than those suggestions, this LGTM!

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 25/25] python: add tox support
  2021-05-25 20:15   ` Cleber Rosa
@ 2021-05-25 20:25     ` John Snow
  2021-05-25 20:46       ` Cleber Rosa
  0 siblings, 1 reply; 64+ messages in thread
From: John Snow @ 2021-05-25 20:25 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/25/21 4:15 PM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:41PM -0400, John Snow wrote:
>> This is intended to be a manually run, non-CI script.
>>
>> Use tox to test the linters against all python versions from 3.6 to
>> 3.9. This will only work if you actually have those versions installed
>> locally, but Fedora makes this easy:
>>
>>> sudo dnf install python36 python37 python38 python39
>>
>> Unlike the pipenv tests (make venv-check), this pulls "whichever"
>> versions of the python packages, so they are unpinned and may break as
>> time goes on. In the case that breakages are found, setup.cfg should be
>> amended accordingly to avoid the bad dependant versions, or the code
>> should be amended to work around the issue.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/README.rst |  2 ++
>>   python/.gitignore |  1 +
>>   python/Makefile   |  7 ++++++-
>>   python/setup.cfg  |  1 +
>>   python/tox.ini    | 13 +++++++++++++
>>   5 files changed, 23 insertions(+), 1 deletion(-)
>>   create mode 100644 python/tox.ini
>>
> 
> This works as intended for me.  A couple of notes / suggestions
> for future improvements:
> 
>   * `dnf install tox` pulled all the Python versions available (I
>     assume as suggestions) automatically
> 
>   * tox.ini can be folded into setup.cfg
> 

Done!

>   * a custom container image with all those Python versions may be
>     handy for running both the pipenv based job (along with the
>     suggestions on the previous patch) and an on-demand,
>     "allow_failure" tox based CI job.
> 

Yeah, I was thinking this would be good, too!

I think at this point, it's going to be a follow-up, though. Because 
ideally, yes, this SHOULD pass -- it's just that it needs a fairly 
particular environment to run in, which is annoying, so it's here as an 
optional-ish thing for now.

Maybe I'll make a new fedora:latest container that's meant solely for 
testing python stuff, because it's just such a convenient distro for it.

Later, though.

> Other than those suggestions, this LGTM!
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> 

🎉



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 06/25] python: add directory structure README.rst files
  2021-05-25 17:14     ` John Snow
@ 2021-05-25 20:30       ` Cleber Rosa
  0 siblings, 0 replies; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 20:30 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 9256 bytes --]

On Tue, May 25, 2021 at 01:14:50PM -0400, John Snow wrote:
> On 5/24/21 10:33 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:22PM -0400, John Snow wrote:
> > > Add short readmes to python/, python/qemu/, python/qemu/machine,
> > > python/qemu/qmp, and python/qemu/utils that explain the directory
> > > hierarchy. These readmes are visible when browsing the source on
> > > e.g. gitlab/github and are designed to help new developers/users quickly
> > > make sense of the source tree.
> > > 
> > > They are not designed for inclusion in a published manual.
> > > 
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > >   python/README.rst              | 41 ++++++++++++++++++++++++++++++++++
> > >   python/qemu/README.rst         |  8 +++++++
> > >   python/qemu/machine/README.rst |  9 ++++++++
> > >   python/qemu/qmp/README.rst     |  9 ++++++++
> > >   python/qemu/utils/README.rst   |  7 ++++++
> > >   5 files changed, 74 insertions(+)
> > >   create mode 100644 python/README.rst
> > >   create mode 100644 python/qemu/README.rst
> > >   create mode 100644 python/qemu/machine/README.rst
> > >   create mode 100644 python/qemu/qmp/README.rst
> > >   create mode 100644 python/qemu/utils/README.rst
> > > 
> > > diff --git a/python/README.rst b/python/README.rst
> > > new file mode 100644
> > > index 00000000000..7a0dc5dff4a
> > > --- /dev/null
> > > +++ b/python/README.rst
> > > @@ -0,0 +1,41 @@
> > > +QEMU Python Tooling
> > > +===================
> > > +
> > > +This directory houses Python tooling used by the QEMU project to build,
> > > +configure, and test QEMU. It is organized by namespace (``qemu``), and
> > > +then by package (e.g. ``qemu/machine``, ``qemu/qmp``, etc).
> > > +
> > > +``setup.py`` is used by ``pip`` to install this tooling to the current
> > > +environment. ``setup.cfg`` provides the packaging configuration used by
> > > +setup.py in a setuptools specific format. You will generally invoke it
> > 
> > For consistency, ``setup.py`` here?  Also, I guess ``setuptools`` as it
> > falls in the same category of ``pip``.
> > 
> 
> Kinda-sorta, but `pip` is a command line executable and setuptools isn't.
> Along those lines I'll fix setup.py but leave setuptools as-is.
>

OK, fair enough.

> > > +by doing one of the following:
> > > +
> > > +1. ``pip3 install .`` will install these packages to your current
> > > +   environment. If you are inside a virtual environment, they will
> > > +   install there. If you are not, it will attempt to install to the
> > > +   global environment, which is not recommended.
> > 
> > Maybe some **emphasis** on **not**?
> > 
> 
> Sure :)
> 
> > > +
> > > +2. ``pip3 install --user .`` will install these packages to your user's
> > > +   local python packages. If you are inside of a virtual environment,
> > > +   this will fail.
> > > +
> > 
> > Maybe note that, if you are inside of a virtual environment, option #1
> > will probably be what users doing "--user" in a venv actually want.
> > 
> 
> Yes. It's frequently annoying how this works, because it's hard to relay
> succinctly :)
> 
> I think at least newer versions of pip give you good warnings when you use
> --user for virtual environments at least.
>

True.

> > > +If you append the ``-e`` argument, pip will install in "editable" mode;
> > > +which installs a version of the package that installs a forwarder
> > > +pointing to these files, such that the package always reflects the
> > > +latest version in your git tree.
> > > +
> > > +See `Installing packages using pip and virtual environments
> > > +<https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/>`_
> > > +for more information.
> > > +
> > > +
> > > +Files in this directory
> > > +-----------------------
> > > +
> > > +- ``qemu/`` Python package source directory.
> > > +- ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
> > > +- ``README.rst`` you are here!
> > > +- ``VERSION`` contains the PEP-440 compliant version used to describe
> > > +  this package; it is referenced by ``setup.cfg``.
> > > +- ``setup.cfg`` houses setuptools package configuration.
> > > +- ``setup.py`` is the setuptools installer used by pip; See above.
> > 
> > Not only used by pip... but I understand the reason for limiting the
> > amount of information given here.
> > 
> 
> Yes ... suggesting broadly that I don't really support using
> setuptools/setup.py alone to install the package, but instead expect and
> consider 'pip' to be the canonical/supported interface.
> 
> There are sometimes minor differences between how they handle things, so I
> wanted less emphasis on setuptools et al.
>

Sure, I can definitely side with you here.

> > > diff --git a/python/qemu/README.rst b/python/qemu/README.rst
> > > new file mode 100644
> > > index 00000000000..d04943f526c
> > > --- /dev/null
> > > +++ b/python/qemu/README.rst
> > > @@ -0,0 +1,8 @@
> > > +QEMU Python Namespace
> > > +=====================
> > > +
> > > +This directory serves as the root of a `Python PEP 420 implicit
> > > +namespace package <https://www.python.org/dev/peps/pep-0420/>`_.
> > > +
> > > +Each directory below is assumed to be an installable Python package that
> > > +is available under the ``qemu.<package>`` namespace.
> > > diff --git a/python/qemu/machine/README.rst b/python/qemu/machine/README.rst
> > > new file mode 100644
> > > index 00000000000..ac2b4fffb42
> > > --- /dev/null
> > > +++ b/python/qemu/machine/README.rst
> > > @@ -0,0 +1,9 @@
> > > +qemu.machine package
> > > +====================
> > > +
> > > +This package provides core utilities used for testing and debugging
> > > +QEMU. It is used by the iotests, vm tests, acceptance tests, and several
> > > +other utilities in the ./scripts directory. It is not a fully-fledged
> > > +SDK and it is subject to change at any time.
> > > +
> > > +See the documentation in ``__init__.py`` for more information.
> > > diff --git a/python/qemu/qmp/README.rst b/python/qemu/qmp/README.rst
> > > new file mode 100644
> > > index 00000000000..c21951491cf
> > > --- /dev/null
> > > +++ b/python/qemu/qmp/README.rst
> > > @@ -0,0 +1,9 @@
> > > +qemu.qmp package
> > > +================
> > > +
> > > +This package provides a library used for connecting to and communicating
> > > +with QMP servers. It is used extensively by iotests, vm tests,
> > > +acceptance tests, and other utilities in the ./scripts directory. It is
> > > +not a fully-fledged SDK and is subject to change at any time.
> > > +
> > > +See the documentation in ``__init__.py`` for more information.
> > > diff --git a/python/qemu/utils/README.rst b/python/qemu/utils/README.rst
> > > new file mode 100644
> > > index 00000000000..975fbf4d7de
> > > --- /dev/null
> > > +++ b/python/qemu/utils/README.rst
> > > @@ -0,0 +1,7 @@
> > > +qemu.utils package
> > > +==================
> > > +
> > > +This package provides miscellaneous utilities used for testing and
> > > +debugging QEMU. It is used primarily by the vm and acceptance tests.
> > > +
> > > +See the documentation in ``__init__.py`` for more information.
> > > -- 
> > > 2.30.2
> > > 
> > > 
> > 
> > With the ``setup.py`` and ``setuptools`` for consistency sake
> > mentioned in my first comment, all other comments are suggestions, so:
> > 
> 
> I took one of those two review comments, and acted on one of two of your
> suggestions.
> 
> > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > 
> 
> So technically I didn't meet your criteria for taking this RB :) Let me know
> if I can still apply it:
> 
> diff --git a/python/README.rst b/python/README.rst
> index 7a0dc5dff4a..38b0c83f321 100644
> --- a/python/README.rst
> +++ b/python/README.rst
> @@ -7,17 +7,17 @@ then by package (e.g. ``qemu/machine``, ``qemu/qmp``,
> etc).
> 
>  ``setup.py`` is used by ``pip`` to install this tooling to the current
>  environment. ``setup.cfg`` provides the packaging configuration used by
> -setup.py in a setuptools specific format. You will generally invoke it
> -by doing one of the following:
> +``setup.py`` in a setuptools specific format. You will generally invoke
> +it by doing one of the following:
>

OK.

>  1. ``pip3 install .`` will install these packages to your current
>     environment. If you are inside a virtual environment, they will
>     install there. If you are not, it will attempt to install to the
> -   global environment, which is not recommended.
> +   global environment, which is **not recommended**.
>

Nice :)

>  2. ``pip3 install --user .`` will install these packages to your user's
>     local python packages. If you are inside of a virtual environment,
> -   this will fail.
> +   this will fail; you likely want the first invocation above.
> 
>  If you append the ``-e`` argument, pip will install in "editable" mode;
>  which installs a version of the package that installs a forwarder

Cool!  And, just to make sure, I'll give you not one but two:

Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>

:)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 24/25] gitlab: add python linters to CI
  2021-05-25 19:55   ` Cleber Rosa
@ 2021-05-25 20:33     ` John Snow
  0 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-25 20:33 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/25/21 3:55 PM, Cleber Rosa wrote:
> On Wed, May 12, 2021 at 07:12:40PM -0400, John Snow wrote:
>> Add python3.6 to the fedora container image: we need it to run the
>> linters against that explicit version to make sure we don't break our
>> minimum version promise.
>>
>> Add pipenv so that we can fetch precise versions of pip packages we need
>> to guarantee test reproducability.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   .gitlab-ci.yml                         | 12 ++++++++++++
>>   tests/docker/dockerfiles/fedora.docker |  2 ++
>>   2 files changed, 14 insertions(+)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index dcb6317aace..a371c0c7163 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -779,6 +779,18 @@ check-patch:
>>       GIT_DEPTH: 1000
>>     allow_failure: true
>>   
>> +
>> +check-python:
>> +  stage: test
>> +  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
>> +  script:
>> +    - cd python
>> +    - make venv-check
>> +  variables:
>> +    GIT_DEPTH: 1000
>> +  needs:
>> +    job: amd64-fedora-container
>> +
>>   check-dco:
>>     stage: build
>>     image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>> diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
>> index 915fdc1845e..6908d69ac37 100644
>> --- a/tests/docker/dockerfiles/fedora.docker
>> +++ b/tests/docker/dockerfiles/fedora.docker
>> @@ -84,6 +84,7 @@ ENV PACKAGES \
>>       numactl-devel \
>>       perl \
>>       perl-Test-Harness \
>> +    pipenv \
>>       pixman-devel \
>>       python3 \
>>       python3-PyYAML \
>> @@ -93,6 +94,7 @@ ENV PACKAGES \
>>       python3-pip \
>>       python3-sphinx \
>>       python3-virtualenv \
>> +    python3.6 \
> 
> I personally would prefer having a different container image for this
> job.  Because it would:
> 
> 1. Be super simple (FROM fedora:33 / dnf -y install python3.6 pipenv)
> 2. Not carry all this unnecessary baggage
> 3. Not risk building QEMU with Python 3.6 (suppose the ./configure
>     probe changes unintentionally)
> 
> But, AFAICT there is no precedent in requiring new images for
> different types of checks.  So, unless someone else complains loudly,
> I'm OK with this.
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 

Hm....

I'll try. You're right that it would make the pipeline a lot simpler 
because it'd have less cruft.

but thank you for the R-B in the meantime ;)

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 18/25] python/qemu: add isort to pipenv
  2021-05-25 17:21     ` John Snow
@ 2021-05-25 20:34       ` Cleber Rosa
  0 siblings, 0 replies; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 20:34 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

On Tue, May 25, 2021 at 01:21:25PM -0400, John Snow wrote:
> On 5/25/21 11:56 AM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:34PM -0400, John Snow wrote:
> > > isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
> > > certain "from ..." clauses that are not related to imports.
> > > 
> > > isort < 5.1.1 has a bug where it does not handle comments near import
> > > statements correctly.
> > > 
> > > Require 5.1.2 or greater.
> > > 
> > > isort can be run with 'isort -c qemu' from the python root.
> > > 
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > >   python/Pipfile      | 1 +
> > >   python/Pipfile.lock | 4 ++--
> > >   2 files changed, 3 insertions(+), 2 deletions(-)
> > 
> > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > 
> 
> Thanks. I have also updated the commit message a little bit:
> 
> isort can be run (in "check" mode) with 'isort -c qemu' from the python
> root. isort can also be used to fix/rewrite import order automatically
> by using 'isort qemu'.
> 
> --js

OK, sounds even better!

- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 20/25] python: add devel package requirements to setuptools
  2021-05-25 17:43     ` John Snow
@ 2021-05-25 20:38       ` Cleber Rosa
  0 siblings, 0 replies; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 20:38 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 4996 bytes --]

On Tue, May 25, 2021 at 01:43:42PM -0400, John Snow wrote:
> On 5/25/21 12:13 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:36PM -0400, John Snow wrote:
> > > setuptools doesn't have a formal understanding of development requires,
> > > but it has an optional feataures section. Fine; add a "devel" feature
> > > and add the requirements to it.
> > > 
> > > To avoid duplication, we can modify pipenv to install qemu[devel]
> > > instead. This enables us to run invocations like "pip install -e
> > > .[devel]" and test the package on bleeding-edge packages beyond those
> > > specified in Pipfile.lock.
> > > 
> > > Importantly, this also allows us to install the qemu development
> > > packages in a non-networked mode: `pip3 install --no-index -e .[devel]`
> > > will now fail if the proper development dependencies are not already
> > > met. This can be useful for automated build scripts where fetching
> > > network packages may be undesirable.
> > > 
> > 
> > This is a fairly exotic feature of setuptools, with very very few
> > packages that I know about using it.  With most users (I believe)
> > relying on pipenv to get the exact packages, the setuptools/pip use
> > case may fall into obscurity IMO.
> > 
> 
> Fair enough.
> 
> The intent is:
> 
> - Pipenv is more for CI, to deploy a consistent set of frozen packages that
> are known to behave in an extremely stable manner. My hope is to avoid
> breaking changes introduced unknowingly by pylint et al.
> 
> - pip install qemu[devel] is intended more for external/normal use by
> developers. It grabs the latest and greatest and it may indeed break as
> dependencies change beyond my awareness.
> 
> 
> Some packages like aiohttp use that optional dependency feature to install
> optional modules -- `pip install aiohttp[speedups]` installs optional
> dependencies that allow that module to work much faster, but aren't
> required.
> 
> Since these linting tools aren't *required* just to *use* the package, I am
> doing users a courtesy by listing them as optional. That way, they aren't
> pulled in when using "pip install qemu", and if I have to pin on specific
> sub-versions etc, it won't include conflict dependencies for people using
> other projects that DO declare a hard requirement on those packages.
> 
> I can amend the PACKAGE.rst file to mention this usage, though it's only
> useful for folks developing the package.
> 
> (Still, part of the ploy here is to attract outside help on developing the
> QEMU SDK, pull requests welcome etc, so it's worth a documentation blurb for
> now.)
>

Yep, I agree with your reasoning here.  I just felt like an extra bit
of documentation would do the trick.

> > So my suggestion is: consider better exposing the fact that this is
> > available (a documentation section perhaps).
> > 
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > >   python/Pipfile      |  5 +----
> > >   python/Pipfile.lock | 14 +++++++++-----
> > >   python/setup.cfg    |  9 +++++++++
> > >   3 files changed, 19 insertions(+), 9 deletions(-)
> > > 
> > 
> > Either way,
> > 
> > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > 
> 
> Thanks! I am taking your R-B and I have applied the following diff.
> 
> Note that the PACKAGE.rst blurb references qemu[devel] instead because the
> PACKAGE.rst file is what is displayed theoretically on PyPI. That exact
> invocation will fail currently, because it's not on PyPI yet.
> 
> A little weird, but I *think* it's correct.
> 
> 
> diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
> index 1bbfe1b58e2..05ea7789fc1 100644
> --- a/python/PACKAGE.rst
> +++ b/python/PACKAGE.rst
> @@ -31,3 +31,7 @@ official `GitLab mirror
> <https://gitlab.com/qemu-project/qemu>`_.
>  Please report bugs on the `QEMU issue tracker
>  <https://gitlab.com/qemu-project/qemu/-/issues>`_ and tag ``@jsnow`` in
>  the report.
> +
> +Optional packages necessary for running code quality analysis for this
> +package can be installed with the optional dependency group "devel":
> +``pip install qemu[devel]``.
> diff --git a/python/README.rst b/python/README.rst
> index bf9bbca979a..954870973d0 100644
> --- a/python/README.rst
> +++ b/python/README.rst
> @@ -24,6 +24,10 @@ which installs a version of the package that installs a
> forwarder
>  pointing to these files, such that the package always reflects the
>  latest version in your git tree.
> 
> +Installing ".[devel]" instead of "." will additionally pull in required
> +packages for testing this package. They are not runtime requirements,
> +and are not needed to simply use these libraries.
> +
>  See `Installing packages using pip and virtual environments
> 
> <https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/>`_
>  for more information.

Looks great to me.  And, let me be clear about it:

Reviewed-by: Cleber Rosa <crosa@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 22/25] python: add Makefile for some common tasks
  2021-05-25 19:45     ` John Snow
@ 2021-05-25 20:39       ` Cleber Rosa
  0 siblings, 0 replies; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 20:39 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 5072 bytes --]

On Tue, May 25, 2021 at 03:45:26PM -0400, John Snow wrote:
> On 5/25/21 3:24 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:38PM -0400, John Snow wrote:
> > > Add "make venv" to create the pipenv-managed virtual environment that
> > > contains our explicitly pinned dependencies.
> > > 
> > > Add "make check" to run the python linters [in the host execution
> > > environment].
> > > 
> > > Add "make venv-check" which combines the above two: create/update the
> > > venv, then run the linters in that explicitly managed environment.
> > > 
> > > Add "make develop" which canonizes the runes needed to get both the
> > > linting pre-requisites (the "[devel]" part), and the editable
> > > live-install (the "-e" part) of these python libraries.
> > > 
> > > make clean: delete miscellaneous python packaging output possibly
> > > created by pipenv, pip, or other python packaging utilities
> > > 
> > > make distclean: delete the above, the .venv, and the editable "qemu"
> > > package forwarder (qemu.egg-info) if there is one.
> > > 
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > >   python/README.rst |  3 +++
> > >   python/Makefile   | 42 ++++++++++++++++++++++++++++++++++++++++++
> > >   2 files changed, 45 insertions(+)
> > >   create mode 100644 python/Makefile
> > > 
> > > diff --git a/python/README.rst b/python/README.rst
> > > index e107bd12a69..3e09d20c23c 100644
> > > --- a/python/README.rst
> > > +++ b/python/README.rst
> > > @@ -35,6 +35,9 @@ Files in this directory
> > >   - ``qemu/`` Python package source directory.
> > >   - ``tests/`` Python package tests directory.
> > >   - ``avocado.cfg`` Configuration for the Avocado test-runner.
> > > +  Used by ``make check`` et al.
> > > +- ``Makefile`` provides some common testing/installation invocations.
> > > +  Try ``make help`` to see available targets.
> > >   - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
> > >     that should be included by a source distribution.
> > >   - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
> > > diff --git a/python/Makefile b/python/Makefile
> > > new file mode 100644
> > > index 00000000000..184f59e5634
> > > --- /dev/null
> > > +++ b/python/Makefile
> > > @@ -0,0 +1,42 @@
> > > +.PHONY: help venv venv-check check clean distclean develop
> > > +
> > > +help:
> > > +	@echo "python packaging help:"
> > > +	@echo ""
> > > +	@echo "make venv:       Create pipenv's virtual environment."
> > > +	@echo "    NOTE: Requires Python 3.6 and pipenv."
> > > +	@echo "          Will download packages from PyPI."
> > > +	@echo "    Hint: (On Fedora): 'sudo dnf install python36 pipenv'"
> > > +	@echo ""
> > > +	@echo "make venv-check: run linters using pipenv's virtual environment."
> > > +	@echo "    Hint: If you don't know which test to run, run this one!"
> > > +	@echo ""
> > > +	@echo "make develop:    Install deps for 'make check', and"
> > > +	@echo "                 the qemu libs in editable/development mode."
> > > +	@echo ""
> > > +	@echo "make check:      run linters using the current environment."
> > > +	@echo ""
> > 
> > Let's observe how this will be used (or misused).  I fear most people
> > will jump into `make check`, even though you have described `make
> > venv-check` as the primary choice.
> > 
> > We have a precedent with `make check-acceptance` that will create a
> > venv and use it by default, so we can consider that as a fallback
> > strategy based on user feedback.
> > 
> 
> Right, I see. Though, I did intentionally want to make it clear which of
> these invocations created an environment and which did not.
> 
> Unlike the acceptance tests, it might make sense to run these tests both
> inside and outside of that venv, so I opted to make the default "make"
> target "make help".
> 
> The Gitlab CI will run the right one, after all -- and I do still expect the
> regular 'make check' to pass, so I am not as sure that it's a crucial
> failure if someone runs the "wrong one".
> 
> > > +	@echo "make clean:      remove build output."
> > > +	@echo ""
> > > +	@echo "make distclean:  remove venv files, qemu package forwarder, and"
> > > +	@echo "                 everything from 'make clean'."
> > > +
> > > +venv: .venv
> > > +.venv: Pipfile.lock
> > > +	@PIPENV_VENV_IN_PROJECT=1 pipenv sync --dev --keep-outdated
> > > +	@touch .venv
> > > +
> > > +venv-check: venv
> > > +	@pipenv run make check
> > > +
> > > +develop:
> > > +	pip3 install -e .[devel]
> > > +
> > > +check:
> > > +	@avocado --config avocado.cfg run tests/
> > > +
> > > +clean:
> > > +	rm -rf build/ dist/
> > > +
> > 
> > Usually `python3 setup.py clean --all` would be the better choice here,
> > but, it doesn't clean `dist/`, so I'm OK with this.
> > 
> 
> Hm, I should probably move the 'dist' down into 'distclean' anyway, and I
> will replace the clean invocation with the one you suggest.
>

OK then, makes sense to me.

- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 23/25] python: add .gitignore
  2021-05-25 20:10     ` John Snow
@ 2021-05-25 20:42       ` Cleber Rosa
  2021-05-25 23:54         ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 20:42 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

On Tue, May 25, 2021 at 04:10:55PM -0400, John Snow wrote:
> On 5/25/21 3:36 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:39PM -0400, John Snow wrote:
> > > Ignore *Python* build and package output (build, dist, qemu.egg-info);
> > > these files are not created as part of a QEMU build.
> > > 
> > > Ignore miscellaneous cached python confetti (__pycache__, *.pyc,
> > > .mypy_cache).
> > > 
> > > Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).
> > > 
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > >   python/.gitignore | 19 +++++++++++++++++++
> > >   1 file changed, 19 insertions(+)
> > >   create mode 100644 python/.gitignore
> > > 
> > > diff --git a/python/.gitignore b/python/.gitignore
> > > new file mode 100644
> > > index 00000000000..e27c99e009c
> > > --- /dev/null
> > > +++ b/python/.gitignore
> > > @@ -0,0 +1,19 @@
> > > +# python bytecode cache
> > > +*.pyc
> > 
> > This is a duplicate from the parent .gitignore, so I would avoid it.
> > 
> > > +__pycache__/
> > 
> > And this one is interesting because, the only thing that *should* be
> > in __pycache__ dirs is .pyc files (covered by the parent .gitignore
> > file).
> > 
> > So, I get the same behavior without these two entries here, so I would
> > skip them.  Let me know if you have any reason for explicitly
> > including them.
> > 
> > - Cleber.
> > 
> 
> Hm, not really ... Just completeness, I suppose, since this directory is
> becoming increasingly separate from the rest of the tree.
> 
> It isn't crucial, it just seemed like a weird omission if they weren't
> listed here. *shrug*
> 
> --js

And still, this dir is part of the overall tree.  Honestly, without
any change in behavior, I'd *not* add those two ignore rules.

Cheers,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 25/25] python: add tox support
  2021-05-25 20:25     ` John Snow
@ 2021-05-25 20:46       ` Cleber Rosa
  2021-05-25 22:15         ` John Snow
  0 siblings, 1 reply; 64+ messages in thread
From: Cleber Rosa @ 2021-05-25 20:46 UTC (permalink / raw)
  To: John Snow
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

[-- Attachment #1: Type: text/plain, Size: 2535 bytes --]

On Tue, May 25, 2021 at 04:25:37PM -0400, John Snow wrote:
> On 5/25/21 4:15 PM, Cleber Rosa wrote:
> > On Wed, May 12, 2021 at 07:12:41PM -0400, John Snow wrote:
> > > This is intended to be a manually run, non-CI script.
> > > 
> > > Use tox to test the linters against all python versions from 3.6 to
> > > 3.9. This will only work if you actually have those versions installed
> > > locally, but Fedora makes this easy:
> > > 
> > > > sudo dnf install python36 python37 python38 python39
> > > 
> > > Unlike the pipenv tests (make venv-check), this pulls "whichever"
> > > versions of the python packages, so they are unpinned and may break as
> > > time goes on. In the case that breakages are found, setup.cfg should be
> > > amended accordingly to avoid the bad dependant versions, or the code
> > > should be amended to work around the issue.
> > > 
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > >   python/README.rst |  2 ++
> > >   python/.gitignore |  1 +
> > >   python/Makefile   |  7 ++++++-
> > >   python/setup.cfg  |  1 +
> > >   python/tox.ini    | 13 +++++++++++++
> > >   5 files changed, 23 insertions(+), 1 deletion(-)
> > >   create mode 100644 python/tox.ini
> > > 
> > 
> > This works as intended for me.  A couple of notes / suggestions
> > for future improvements:
> > 
> >   * `dnf install tox` pulled all the Python versions available (I
> >     assume as suggestions) automatically
> > 
> >   * tox.ini can be folded into setup.cfg
> > 
> 
> Done!
>

Nice!

> >   * a custom container image with all those Python versions may be
> >     handy for running both the pipenv based job (along with the
> >     suggestions on the previous patch) and an on-demand,
> >     "allow_failure" tox based CI job.
> > 
> 
> Yeah, I was thinking this would be good, too!
> 
> I think at this point, it's going to be a follow-up, though. Because
> ideally, yes, this SHOULD pass -- it's just that it needs a fairly
> particular environment to run in, which is annoying, so it's here as an
> optional-ish thing for now.
> 
> Maybe I'll make a new fedora:latest container that's meant solely for
> testing python stuff, because it's just such a convenient distro for it.
> 
> Later, though.
>

Sure.

> > Other than those suggestions, this LGTM!
> > 
> > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > Tested-by: Cleber Rosa <crosa@redhat.com>
> > 
> 
> 🎉

\o/ (with 3 characters, because I'm unable to find the right codepoint)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 25/25] python: add tox support
  2021-05-25 20:46       ` Cleber Rosa
@ 2021-05-25 22:15         ` John Snow
  0 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-25 22:15 UTC (permalink / raw)
  To: Cleber Rosa; +Cc: qemu-devel

On 5/25/21 4:46 PM, Cleber Rosa wrote:
> On Tue, May 25, 2021 at 04:25:37PM -0400, John Snow wrote:
>> On 5/25/21 4:15 PM, Cleber Rosa wrote:
>>> On Wed, May 12, 2021 at 07:12:41PM -0400, John Snow wrote:
>>>> This is intended to be a manually run, non-CI script.
>>>>
>>>> Use tox to test the linters against all python versions from 3.6 to
>>>> 3.9. This will only work if you actually have those versions installed
>>>> locally, but Fedora makes this easy:
>>>>
>>>>> sudo dnf install python36 python37 python38 python39
>>>>
>>>> Unlike the pipenv tests (make venv-check), this pulls "whichever"
>>>> versions of the python packages, so they are unpinned and may break as
>>>> time goes on. In the case that breakages are found, setup.cfg should be
>>>> amended accordingly to avoid the bad dependant versions, or the code
>>>> should be amended to work around the issue.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>>    python/README.rst |  2 ++
>>>>    python/.gitignore |  1 +
>>>>    python/Makefile   |  7 ++++++-
>>>>    python/setup.cfg  |  1 +
>>>>    python/tox.ini    | 13 +++++++++++++
>>>>    5 files changed, 23 insertions(+), 1 deletion(-)
>>>>    create mode 100644 python/tox.ini
>>>>
>>>
>>> This works as intended for me.  A couple of notes / suggestions
>>> for future improvements:
>>>
>>>    * `dnf install tox` pulled all the Python versions available (I
>>>      assume as suggestions) automatically
>>>
>>>    * tox.ini can be folded into setup.cfg
>>>
>>
>> Done!
>>
> 
> Nice!
> 
>>>    * a custom container image with all those Python versions may be
>>>      handy for running both the pipenv based job (along with the
>>>      suggestions on the previous patch) and an on-demand,
>>>      "allow_failure" tox based CI job.
>>>
>>
>> Yeah, I was thinking this would be good, too!
>>
>> I think at this point, it's going to be a follow-up, though. Because
>> ideally, yes, this SHOULD pass -- it's just that it needs a fairly
>> particular environment to run in, which is annoying, so it's here as an
>> optional-ish thing for now.
>>
>> Maybe I'll make a new fedora:latest container that's meant solely for
>> testing python stuff, because it's just such a convenient distro for it.
>>
>> Later, though.
>>
> 
> Sure.
> 
>>> Other than those suggestions, this LGTM!
>>>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>
>>
>> 🎉
> 
> \o/ (with 3 characters, because I'm unable to find the right codepoint)
> 

if you're on fedora using the ime-bus stuff, try ctrl+shift+e to enter 
EMOJI MODE and then type 'tada' and then hit 'space' to search.

🥳



^ permalink raw reply	[flat|nested] 64+ messages in thread

* Re: [PATCH v6 23/25] python: add .gitignore
  2021-05-25 20:42       ` Cleber Rosa
@ 2021-05-25 23:54         ` John Snow
  0 siblings, 0 replies; 64+ messages in thread
From: John Snow @ 2021-05-25 23:54 UTC (permalink / raw)
  To: Cleber Rosa
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Willian Rampazzo, Max Reitz, Alex Bennée,
	Beraldo Leal

On 5/25/21 4:42 PM, Cleber Rosa wrote:
> On Tue, May 25, 2021 at 04:10:55PM -0400, John Snow wrote:
>> On 5/25/21 3:36 PM, Cleber Rosa wrote:
>>> On Wed, May 12, 2021 at 07:12:39PM -0400, John Snow wrote:
>>>> Ignore *Python* build and package output (build, dist, qemu.egg-info);
>>>> these files are not created as part of a QEMU build.
>>>>
>>>> Ignore miscellaneous cached python confetti (__pycache__, *.pyc,
>>>> .mypy_cache).
>>>>
>>>> Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>>    python/.gitignore | 19 +++++++++++++++++++
>>>>    1 file changed, 19 insertions(+)
>>>>    create mode 100644 python/.gitignore
>>>>
>>>> diff --git a/python/.gitignore b/python/.gitignore
>>>> new file mode 100644
>>>> index 00000000000..e27c99e009c
>>>> --- /dev/null
>>>> +++ b/python/.gitignore
>>>> @@ -0,0 +1,19 @@
>>>> +# python bytecode cache
>>>> +*.pyc
>>>
>>> This is a duplicate from the parent .gitignore, so I would avoid it.
>>>
>>>> +__pycache__/
>>>
>>> And this one is interesting because, the only thing that *should* be
>>> in __pycache__ dirs is .pyc files (covered by the parent .gitignore
>>> file).
>>>
>>> So, I get the same behavior without these two entries here, so I would
>>> skip them.  Let me know if you have any reason for explicitly
>>> including them.
>>>
>>> - Cleber.
>>>
>>
>> Hm, not really ... Just completeness, I suppose, since this directory is
>> becoming increasingly separate from the rest of the tree.
>>
>> It isn't crucial, it just seemed like a weird omission if they weren't
>> listed here. *shrug*
>>
>> --js
> 
> And still, this dir is part of the overall tree.

For now 😇

                                                    Honestly, without
> any change in behavior, I'd *not* add those two ignore rules.
> 

If you insist.

--js



^ permalink raw reply	[flat|nested] 64+ messages in thread

end of thread, other threads:[~2021-05-25 23:56 UTC | newest]

Thread overview: 64+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
2021-05-12 23:12 ` [PATCH v6 01/25] iotests/297: add --namespace-packages to mypy arguments John Snow
2021-05-12 23:12 ` [PATCH v6 02/25] python: create qemu packages John Snow
2021-05-18 14:31   ` Cleber Rosa
2021-05-21 17:15   ` Willian Rampazzo
2021-05-12 23:12 ` [PATCH v6 03/25] python: create utils sub-package John Snow
2021-05-18 15:12   ` Cleber Rosa
2021-05-18 15:20     ` John Snow
2021-05-21 17:18   ` Willian Rampazzo
2021-05-21 17:39     ` John Snow
2021-05-12 23:12 ` [PATCH v6 04/25] python: add qemu package installer John Snow
2021-05-21  4:00   ` Cleber Rosa
2021-05-21 15:52     ` John Snow
2021-05-12 23:12 ` [PATCH v6 05/25] python: add VERSION file John Snow
2021-05-12 23:12 ` [PATCH v6 06/25] python: add directory structure README.rst files John Snow
2021-05-25  2:33   ` Cleber Rosa
2021-05-25 17:14     ` John Snow
2021-05-25 20:30       ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 07/25] python: add MANIFEST.in John Snow
2021-05-25  2:42   ` Cleber Rosa
2021-05-25 14:12     ` John Snow
2021-05-12 23:12 ` [PATCH v6 08/25] python: Add pipenv support John Snow
2021-05-12 23:12 ` [PATCH v6 09/25] python: add pylint import exceptions John Snow
2021-05-12 23:12 ` [PATCH v6 10/25] python: move pylintrc into setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 11/25] python: add pylint to pipenv John Snow
2021-05-25  3:33   ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 12/25] python: move flake8 config to setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 13/25] python: add excluded dirs to flake8 config John Snow
2021-05-25 15:50   ` Cleber Rosa
2021-05-25 17:18     ` John Snow
2021-05-12 23:12 ` [PATCH v6 14/25] python: Add flake8 to pipenv John Snow
2021-05-12 23:12 ` [PATCH v6 15/25] python: move mypy.ini into setup.cfg John Snow
2021-05-25 15:52   ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 16/25] python: add mypy to pipenv John Snow
2021-05-12 23:12 ` [PATCH v6 17/25] python: move .isort.cfg into setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 18/25] python/qemu: add isort to pipenv John Snow
2021-05-25 15:56   ` Cleber Rosa
2021-05-25 17:21     ` John Snow
2021-05-25 20:34       ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 19/25] python/qemu: add qemu package itself " John Snow
2021-05-12 23:12 ` [PATCH v6 20/25] python: add devel package requirements to setuptools John Snow
2021-05-25 16:13   ` Cleber Rosa
2021-05-25 17:43     ` John Snow
2021-05-25 20:38       ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 21/25] python: add avocado-framework and tests John Snow
2021-05-25 18:58   ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 22/25] python: add Makefile for some common tasks John Snow
2021-05-25 19:24   ` Cleber Rosa
2021-05-25 19:45     ` John Snow
2021-05-25 20:39       ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 23/25] python: add .gitignore John Snow
2021-05-25 19:36   ` Cleber Rosa
2021-05-25 20:10     ` John Snow
2021-05-25 20:42       ` Cleber Rosa
2021-05-25 23:54         ` John Snow
2021-05-12 23:12 ` [PATCH v6 24/25] gitlab: add python linters to CI John Snow
2021-05-25 19:55   ` Cleber Rosa
2021-05-25 20:33     ` John Snow
2021-05-12 23:12 ` [PATCH v6 25/25] python: add tox support John Snow
2021-05-25 20:15   ` Cleber Rosa
2021-05-25 20:25     ` John Snow
2021-05-25 20:46       ` Cleber Rosa
2021-05-25 22:15         ` John Snow
2021-05-17 18:52 ` [PATCH v6 00/25] python: create installable package John Snow

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.