Re: [PATCH v6 00/25] python: create installable package

From: John Snow <jsnow@redhat.com>
To: qemu-devel@nongnu.org
Cc: "Kevin Wolf" <kwolf@redhat.com>, "Fam Zheng" <fam@euphon.net>,
	"Thomas Huth" <thuth@redhat.com>,
	"Eduardo Habkost" <ehabkost@redhat.com>,
	qemu-block@nongnu.org,
	"Philippe Mathieu-Daudé" <philmd@redhat.com>,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
	"Max Reitz" <mreitz@redhat.com>,
	"Willian Rampazzo" <willianr@redhat.com>,
	"Willian Rampazzo" <wrampazz@redhat.com>,
	"Cleber Rosa" <crosa@redhat.com>,
	"Alex Bennée" <alex.bennee@linaro.org>,
	"Beraldo Leal" <bleal@redhat.com>
Subject: Re: [PATCH v6 00/25] python: create installable package
Date: Mon, 17 May 2021 14:52:33 -0400	[thread overview]
Message-ID: <d59b6295-58a4-90bd-f898-75d8fc5326ed@redhat.com> (raw)
In-Reply-To: <20210512231241.2816122-1-jsnow@redhat.com>

On 5/12/21 7:12 PM, John Snow wrote:
> Based-on: <20210512214642.2803189-1-jsnow@redhat.com>
> CI: https://gitlab.com/jsnow/qemu/-/pipelines/302010131
> GitLab: https://gitlab.com/jsnow/qemu/-/tree/python-package-mk3
> MR: https://gitlab.com/jsnow/qemu/-/merge_requests/4
> 
Patchset updated and rebased on top of new linting pre-req series.
(Gitlab branch and MR rebased and updated.)

Based-on: <20210517184808.3562549-1-jsnow@redhat.com>

> I invite you to leave review comments on my mock merge request on
> gitlab, submitted against my own mirror. I will, as always, also respond
> to feedback on-list.
> 
> ABOUT
> =====
> 
> This series factors the python/qemu directory as an installable
> package. It does not yet actually change the mechanics of how any other
> python source in the tree actually consumes it (yet), beyond the import
> path -- some import statements change in a few places.
> 
> RATIONALE
> =========
> 
> The primary motivation of this series is primarily to formalize our
> dependencies on mypy, flake8, isort, and pylint alongside versions that
> are known to work. It does this using the setup.cfg and setup.py
> files. It also adds explicitly pinned versions (using Pipfile.lock) of
> these dependencies that should behave in a repeatable and known way for
> developers and CI environments both. Lastly, it enables those CI checks
> such that we can enforce Python coding quality checks via the CI tests.
> 
> An auxiliary motivation is that this package is formatted in such a way
> that it COULD be uploaded to https://pypi.org/project/qemu and installed
> independently of qemu.git with `pip install qemu`, but that button
> remains *unpushed* and this series *will not* cause any such
> releases. We have time to debate finer points like API guarantees and
> versioning even after this series is merged.
> 
> Other bits of interest
> ----------------------
> 
> With the python tooling as a proper package, you can install this
> package in editable or production mode to a virtual environment, your
> local user environment, or your system packages. The primary benefit of
> this is to gain access to QMP tooling regardless of CWD, without needing
> to battle sys.path (and confounding other python analysis tools).
> 
> For example: when developing, you may go to qemu/python/ and run `make
> venv` followed by `pipenv shell` to activate a virtual environment that
> contains the qemu python packages. These packages will always reflect
> the current version of the source files in the tree. When you are
> finished, you can simply exit the shell (^d) to remove these packages
> from your python environment.
> 
> When not developing, you could install a version of this package to your
> environment outright to gain access to the QMP and QEMUMachine classes
> for lightweight scripting and testing by using pip: "pip install
> [--user] ."
> 
> TESTING THIS SERIES
> ===================
> 
> First of all, nothing should change. Without any intervention,
> everything should behave exactly as it did before. The only new
> information here comes from how to interact with and run the linters
> that will be enforcing code quality standards in this subdirectory.
> 
> There are various invocations available that will test subtly different
> combinations using subtly different environments. I am assuming some
> light knowledge of Python environments and installing Python packages
> here. If you have questions, I would be delighted to answer them.
> 
> To test the new tests, CD to ./python/ first, and then:
> 
> 0. Try "make" or "make help" to get a sense of this series.
> 
> 1. Try "make venv && pipenv shell" to get a venv with the package
>     installed to it in editable mode. Ctrl+d exits this venv shell. While
>     in this shell, any python script that uses "from qemu.[qmp|machine]
>     import ..." should work correctly regardless of where the script is,
>     or what your CWD is.
> 
>     This will pull some packages from PyPI and install them into the
>     virtual environment, leaving your normal environment untouched.
> 
>     You will need Python 3.6 and pipenv installed on your system to do
>     this step. For Fedora: "dnf install python36 pipenv" will do the
>     trick. If you don't have this, skip down to #4 and onwards.
> 
> 2. Try "make check" while still in the shell to run the Python linters
> using the venv built in the previous step. This will run avocado, which
> will in turn execute mypy, flake8, isort and pylint with the correct
> arguments.
> 
> 3. Having exited the shell from above, try "make venv-check". This will
> create and update the venv if needed, then run 'make check' within the
> context of that shell. It should pass as long as the above did. You
> should be able to run "make distclean" prior to running "make
> venv-check" and have the entire process work start to finish.
> 
> 4. Still outside of the venv, you may try running "make check". This
> will not install anything, but unless you have the right Python
> dependencies installed, these tests may fail for you. You might try
> using "pip install --user .[devel]" to install the development packages
> needed to run the tests successfully to your local user's python
> environment. Once done, you will probably want to "pip uninstall qemu"
> to remove the qemu packages to avoid them interfering with other things.
> 
> 5. "make distclean" will delete the venv and any temporary files that
> may have been created by packaging, installing, testing, etc.
> 
> 6. You may also (if you wish) create your own environment using a Python
>     other than 3.6, bypassing pipenv. This may be useful for environments
>     in which you simply do not have Python 3.6 readily available, or
>     explicitly wish to test against a different version.
> 
>     Using the Python of your choice:
> 
>     - `> make distclean` (Remove the .venv, if you created one.)
>     - `> /usr/bin/python3.9 -m venv .venv`
>     - `> source .venv/bin/activate` (or activate.[fish|csh])
>     - `> make develop`  (Installs linter deps to this venv)
>     - `> make check`    (Runs tests using current env)
>     - `> deactivate`    (Leaves the venv)
> 
>     This *should* pass, and if it doesn't, it's a bug. Due to
>     dependencies changing like shifting sands, it's impossible to promise
>     support for every last combination of dependency and python
>     version. This is why the "make venv-check" target uses Python 3 and a
>     very explicit set of packages instead. Still, this form *should*
>     pass.
> 
> 7. If you are running Fedora or a distro that packages multiple Python
> versions, you can try running "make check-tox" to run the linters under
> all Python versions from Python 3.6 to 3.9. This will create a series of
> virtual environments under the .tox folder, leaving your existing
> environment unscathed. This should pass, provided you have the requisite
> Python versions on hand. When you are done, "make distclean" should
> remove any testing files.
> 
> CHANGELOG
> =========
> 
> V6
> --
> 
> ```
> 001/25:[----] [--] 'iotests/297: add --namespace-packages to mypy arguments'
> 002/25:[----] [-C] 'python: create qemu packages'
> 003/25:[----] [--] 'python: create utils sub-package'
> 004/25:[0005] [FC] 'python: add qemu package installer'
> 005/25:[0002] [FC] 'python: add VERSION file'
> 006/25:[----] [--] 'python: add directory structure README.rst files'
> 007/25:[down] 'python: add MANIFEST.in'
> 008/25:[0001] [FC] 'python: Add pipenv support'
> 009/25:[----] [--] 'python: add pylint import exceptions'
> 010/25:[----] [--] 'python: move pylintrc into setup.cfg'
> 011/25:[0083] [FC] 'python: add pylint to pipenv'
> 012/25:[----] [--] 'python: move flake8 config to setup.cfg'
> 013/25:[----] [--] 'python: add excluded dirs to flake8 config'
> 014/25:[0038] [FC] 'python: Add flake8 to pipenv'
> 015/25:[----] [--] 'python: move mypy.ini into setup.cfg'
> 016/25:[0050] [FC] 'python: add mypy to pipenv'
> 017/25:[----] [--] 'python: move .isort.cfg into setup.cfg'
> 018/25:[0006] [FC] 'python/qemu: add isort to pipenv'
> 019/25:[0004] [FC] 'python/qemu: add qemu package itself to pipenv'
> 020/25:[0014] [FC] 'python: add devel package requirements to setuptools'
> 021/25:[down] 'python: add avocado-framework and tests'
> 022/25:[0015] [FC] 'python: add Makefile for some common tasks'
> 023/25:[0018] [FC] 'python: add .gitignore'
> 024/25:[0004] [FC] 'gitlab: add python linters to CI'
> 025/25:[down] 'python: add tox support'
> ```
> 
> - 04: Changed maintainer blurb in PACKAGE.rst
> - 05: VERSION is now 0.6.1a0
> - 07: New, needed for both tox and using Pipenv to install the qemu
>        package(s) in editable mode.
> - 08: Add README entries explaining the purpose of the pipenv files.
> - 11: pylint dependency was bumped to 2.7.0.
> - 14: pipenv changes.
> - 16: pipenv changes.
> - 18: isort pinned at >= 5.1.2 now; see commit message.
> - 19: pipenv changes.
> - 20: Context differences from isort, pylint, and pipenv changes.
> - 22:
>     - Add 'make develop'
>     - Add more README.rst entries.
>     - Change pytest invocation to avocado.
> - 23:
>     - Add comments to gitignore, split ignores out by function/category
>     - Ignore vscode files. (I like vscode now!)
> - 24: Use the 'needs' feature of Gitlab CI and move the test to 'test'
>    stage
> 
> STATUS
> =======
> 
> ```
> + [01] iotests-297-add-namespace      # [RB] CR [SOB] JS
> + [02] python-create-qemu-packages    # [SOB] JS
> + [03] python-create-utils-sub        # [SOB] JS
> + [04] python-add-qemu-package        # [SOB] JS
> + [05] python-add-version-file        # [RB] CR [SOB] JS
> + [06] python-add-directory-structure # [SOB] JS
> + [07] python-add-manifest-in         # [SOB] JS
> + [08] python-add-pipenv-support      # [RB] CR [SOB] JS
> + [09] python-add-pylint-import       # [RB] CR [SOB] JS
> + [10] python-move-pylintrc-into      # [RB] CR [TB] CR [SOB] JS
> + [11] python-add-pylint-to-pipenv    # [SOB] JS
> + [12] python-move-flake8-config-to   # [RB] CR [SOB] JS
> + [13] python-add-excluded-dirs-to    # [SOB] JS
> + [14] python-add-flake8-to-pipenv    # [RB] CR [TB] CR [SOB] JS
> + [15] python-move-mypy-ini-into      # [SOB] JS
> + [16] python-add-mypy-to-pipenv      # [RB] CR [TB] CR [SOB] JS
> + [17] python-move-isort-cfg-into     # [RB] CR [SOB] JS
> + [18] python-qemu-add-isort-to       # [SOB] JS
> + [19] python-qemu-add-qemu-package   # [RB] CR [TB] CR [SOB] JS
> + [20] python-add-devel-package       # [SOB] JS
> + [21] python-add-avocado-framework   # [SOB] JS
> + [22] python-add-makefile-for-some   # [SOB] JS
> + [23] python-add-gitignore           # [SOB] JS
> + [24] gitlab-add-python-linters-to   # [SOB] JS
>> [25] python-add-tox-support         # [SOB] JS
> ```
> 
> John Snow (25):
>    iotests/297: add --namespace-packages to mypy arguments
>    python: create qemu packages
>    python: create utils sub-package
>    python: add qemu package installer
>    python: add VERSION file
>    python: add directory structure README.rst files
>    python: add MANIFEST.in
>    python: Add pipenv support
>    python: add pylint import exceptions
>    python: move pylintrc into setup.cfg
>    python: add pylint to pipenv
>    python: move flake8 config to setup.cfg
>    python: add excluded dirs to flake8 config
>    python: Add flake8 to pipenv
>    python: move mypy.ini into setup.cfg
>    python: add mypy to pipenv
>    python: move .isort.cfg into setup.cfg
>    python/qemu: add isort to pipenv
>    python/qemu: add qemu package itself to pipenv
>    python: add devel package requirements to setuptools
>    python: add avocado-framework and tests
>    python: add Makefile for some common tasks
>    python: add .gitignore
>    gitlab: add python linters to CI
>    python: add tox support
> 
>   python/PACKAGE.rst                          |  33 +++
>   python/README.rst                           |  53 +++++
>   python/qemu/README.rst                      |   8 +
>   python/qemu/machine/README.rst              |   9 +
>   python/qemu/qmp/README.rst                  |   9 +
>   python/qemu/utils/README.rst                |   7 +
>   .gitlab-ci.yml                              |  12 +
>   python/.gitignore                           |  20 ++
>   python/MANIFEST.in                          |   2 +
>   python/Makefile                             |  47 ++++
>   python/Pipfile                              |  13 ++
>   python/Pipfile.lock                         | 231 ++++++++++++++++++++
>   python/VERSION                              |   1 +
>   python/avocado.cfg                          |  10 +
>   python/mypy.ini                             |   4 -
>   python/qemu/.flake8                         |   2 -
>   python/qemu/.isort.cfg                      |   7 -
>   python/qemu/__init__.py                     |  11 -
>   python/qemu/machine/__init__.py             |  36 +++
>   python/qemu/{ => machine}/console_socket.py |   0
>   python/qemu/{ => machine}/machine.py        |  16 +-
>   python/qemu/{ => machine}/qtest.py          |   3 +-
>   python/qemu/pylintrc                        |  58 -----
>   python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
>   python/qemu/utils/__init__.py               |  23 ++
>   python/qemu/{ => utils}/accel.py            |   0
>   python/setup.cfg                            |  79 +++++++
>   python/setup.py                             |  23 ++
>   python/tests/flake8.sh                      |   2 +
>   python/tests/isort.sh                       |   2 +
>   python/tests/mypy.sh                        |   2 +
>   python/tests/pylint.sh                      |   2 +
>   python/tox.ini                              |  13 ++
>   tests/acceptance/avocado_qemu/__init__.py   |   4 +-
>   tests/acceptance/virtio-gpu.py              |   2 +-
>   tests/docker/dockerfiles/fedora.docker      |   2 +
>   tests/qemu-iotests/297                      |   1 +
>   tests/qemu-iotests/300                      |   4 +-
>   tests/qemu-iotests/iotests.py               |   2 +-
>   tests/vm/aarch64vm.py                       |   2 +-
>   tests/vm/basevm.py                          |   2 +-
>   41 files changed, 672 insertions(+), 97 deletions(-)
>   create mode 100644 python/PACKAGE.rst
>   create mode 100644 python/README.rst
>   create mode 100644 python/qemu/README.rst
>   create mode 100644 python/qemu/machine/README.rst
>   create mode 100644 python/qemu/qmp/README.rst
>   create mode 100644 python/qemu/utils/README.rst
>   create mode 100644 python/.gitignore
>   create mode 100644 python/MANIFEST.in
>   create mode 100644 python/Makefile
>   create mode 100644 python/Pipfile
>   create mode 100644 python/Pipfile.lock
>   create mode 100644 python/VERSION
>   create mode 100644 python/avocado.cfg
>   delete mode 100644 python/mypy.ini
>   delete mode 100644 python/qemu/.flake8
>   delete mode 100644 python/qemu/.isort.cfg
>   delete mode 100644 python/qemu/__init__.py
>   create mode 100644 python/qemu/machine/__init__.py
>   rename python/qemu/{ => machine}/console_socket.py (100%)
>   rename python/qemu/{ => machine}/machine.py (98%)
>   rename python/qemu/{ => machine}/qtest.py (98%)
>   delete mode 100644 python/qemu/pylintrc
>   rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
>   create mode 100644 python/qemu/utils/__init__.py
>   rename python/qemu/{ => utils}/accel.py (100%)
>   create mode 100644 python/setup.cfg
>   create mode 100755 python/setup.py
>   create mode 100755 python/tests/flake8.sh
>   create mode 100755 python/tests/isort.sh
>   create mode 100755 python/tests/mypy.sh
>   create mode 100755 python/tests/pylint.sh
>   create mode 100644 python/tox.ini
>