Re: [PATCH v3 00/15] python: create installable package

[John Snow <jsnow@redhat.com>]

Ping O:-)

Looking for feedback from at least Cleber and Eduardo before I barge 
ahead and send a PR to include this on master. Additional packaging and 
versioning feedback from Dan would be nice.

(I know we have a very busy two weeks here; I will continue pinging, but 
I have every intention of merging this prior to 5.2.)

--js

On 10/20/20 3:35 PM, John Snow wrote:
> Based-on: https://gitlab.com/jsnow/qemu/-/tree/python
> 
> This series factors the python/qemu directory as an installable
> package. It does not yet actually change the mechanics of how any other
> python source in the tree actually consumes it (yet), beyond the import
> path.
>  > The point of this series is primarily to formalize our dependencies on
> mypy, flake8, isort, and pylint alongside versions that are known to
> work. It also adds explicitly pinned versions of these dependencies that
> should behave in a repeatable and known way for developers and CI
> environments both.
> 
> With the python tooling as a proper package, you can install this
> package in editable or production mode to a virtual environment, your
> local user environment, or your system packages. The primary benefit of
> this is to gain access to QMP tooling regardless of CWD, without needing
> to battle sys.path.
> 
> For example: when developing, you may go to qemu/python/ and invoke
> `pipenv shell` to activate a virtual environment that contains the qemu
> packages.  This package will always reflect the current version of the
> source files in the tree. When you are finished, you can simply exit the
> shell to remove these packages from your python environment.
> 
> When not developing, you could install a version of this package to your
> environment outright to gain access to the QMP and QEMUMachine classes
> for lightweight scripting and testing by using pip: "pip install [--user] ."
> 
> Finally, this package is formatted in such a way that it COULD be
> uploaded to https://pypi.org/project/qemu and installed independently of
> qemu.git with `pip install qemu`, but that button remains unpushed.
> 
> TESTING THIS SERIES:
> 
> CD to qemu/python first, and then:
> 
> 1. Try "pipenv shell" to get a venv with the package installed to it in
> editable mode. Ctrl+d exits this venv shell. While in this shell, any
> python script that uses "from qemu.[qmp|machine] import ..." should work
> correctly regardless of your CWD.
> 
> 2. Try "pipenv sync --dev" to create/update the venv with the
> development packages without actually entering the venv. This should
> install isort, mypy, flake8 and pylint to the venv.
> 
> 3. After the above sync, try "pipenv shell" again, and from the python
> project root, try any of the following:
> 
>    - pylint qemu
>    - flake8 qemu
>    - isort -c qemu
>    - mypy qemu
> 
> 4. Leave any venv you are in, and from the project root, try the
> following commands:
> 
>    - pipenv run pylint qemu
>    - pipenv run flake8 qemu
>    - pipenv run isort -c qemu
>    - pipenv run mypy qemu
> 
> V3:
>   - Changed "qemu.core" to "qemu.qmp" and "qemu.machine",
>     Partly to accommodate forthcoming work which would benefit from a separate
>     qemu.qmp namespace.
>   - Changed the initial version from 5.2.0a1 to 0.5.2.0a1, to allow for
>     more rapid development while we smooth out the initial kinks.
>   - 001: Renamed patch title; differences implement the new names.
>   - 002: Readme changes for above.
>   - 003: Version change.
>   - 004: New readme for the new qmp directory.
>   - 006: A few more import exceptions for pylint, hopefully temporary.
>   - 009: Updated flake8 config comment to match qapi's
>   - 012: Added namespace_package configuration value
> 
> 001/15:[down] 'python: create qemu packages'
> 002/15:[0009] [FC] 'python: add qemu package installer'
> 003/15:[0002] [FC] 'python: add VERSION file'
> 004/15:[0015] [FC] 'python: add directory structure README.rst files'
> 005/15:[----] [--] 'python: Add pipenv support'
> 006/15:[down] 'python: add pylint import exceptions'
> 007/15:[----] [--] 'python: move pylintrc into setup.cfg'
> 008/15:[----] [--] 'python: add pylint to pipenv'
> 009/15:[0002] [FC] 'python: move flake8 config to setup.cfg'
> 010/15:[----] [--] 'python: Add flake8 to pipenv'
> 011/15:[----] [-C] 'python: move mypy.ini into setup.cfg'
> 012/15:[0001] [FC] 'python: add mypy to pipenv'
> 013/15:[----] [--] 'python: move .isort.cfg into setup.cfg'
> 014/15:[----] [--] 'python/qemu: add isort to pipenv'
> 015/15:[----] [--] 'python/qemu: add qemu package itself to pipenv'
> 
> John Snow (15):
>    python: create qemu packages
>    python: add qemu package installer
>    python: add VERSION file
>    python: add directory structure README.rst files
>    python: Add pipenv support
>    python: add pylint import exceptions
>    python: move pylintrc into setup.cfg
>    python: add pylint to pipenv
>    python: move flake8 config to setup.cfg
>    python: Add flake8 to pipenv
>    python: move mypy.ini into setup.cfg
>    python: add mypy to pipenv
>    python: move .isort.cfg into setup.cfg
>    python/qemu: add isort to pipenv
>    python/qemu: add qemu package itself to pipenv
> 
>   python/PACKAGE.rst                          |  26 +++
>   python/README.rst                           |  27 +++
>   python/qemu/README.rst                      |   8 +
>   python/qemu/machine/README.rst              |   9 +
>   python/qemu/qmp/README.rst                  |   9 +
>   python/Pipfile                              |  16 ++
>   python/Pipfile.lock                         | 207 ++++++++++++++++++++
>   python/VERSION                              |   1 +
>   python/mypy.ini                             |   4 -
>   python/qemu/.flake8                         |   2 -
>   python/qemu/.isort.cfg                      |   7 -
>   python/qemu/__init__.py                     |  11 --
>   python/qemu/machine/__init__.py             |  44 +++++
>   python/qemu/{ => machine}/accel.py          |   0
>   python/qemu/{ => machine}/console_socket.py |   0
>   python/qemu/{ => machine}/machine.py        |  16 +-
>   python/qemu/{ => machine}/qtest.py          |   3 +-
>   python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
>   python/{qemu/pylintrc => setup.cfg}         |  67 ++++---
>   python/setup.py                             |  23 +++
>   tests/acceptance/boot_linux.py              |   3 +-
>   tests/qemu-iotests/297                      |   2 +-
>   tests/qemu-iotests/300                      |   4 +-
>   tests/qemu-iotests/iotests.py               |   2 +-
>   tests/vm/basevm.py                          |   3 +-
>   25 files changed, 437 insertions(+), 69 deletions(-)
>   create mode 100644 python/PACKAGE.rst
>   create mode 100644 python/README.rst
>   create mode 100644 python/qemu/README.rst
>   create mode 100644 python/qemu/machine/README.rst
>   create mode 100644 python/qemu/qmp/README.rst
>   create mode 100644 python/Pipfile
>   create mode 100644 python/Pipfile.lock
>   create mode 100644 python/VERSION
>   delete mode 100644 python/mypy.ini
>   delete mode 100644 python/qemu/.flake8
>   delete mode 100644 python/qemu/.isort.cfg
>   delete mode 100644 python/qemu/__init__.py
>   create mode 100644 python/qemu/machine/__init__.py
>   rename python/qemu/{ => machine}/accel.py (100%)
>   rename python/qemu/{ => machine}/console_socket.py (100%)
>   rename python/qemu/{ => machine}/machine.py (98%)
>   rename python/qemu/{ => machine}/qtest.py (98%)
>   rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
>   rename python/{qemu/pylintrc => setup.cfg} (50%)
>   mode change 100644 => 100755
>   create mode 100755 python/setup.py
>