All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v6 00/25] python: create installable package
@ 2021-05-12 23:12 John Snow
  2021-05-12 23:12 ` [PATCH v6 01/25] iotests/297: add --namespace-packages to mypy arguments John Snow
                   ` (25 more replies)
  0 siblings, 26 replies; 64+ messages in thread
From: John Snow @ 2021-05-12 23:12 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Thomas Huth, Eduardo Habkost, qemu-block,
	Alex Bennée, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Max Reitz, John Snow,
	Willian Rampazzo, Cleber Rosa, Willian Rampazzo,
	Philippe Mathieu-Daudé,
	Beraldo Leal

Based-on: <20210512214642.2803189-1-jsnow@redhat.com>
CI: https://gitlab.com/jsnow/qemu/-/pipelines/302010131
GitLab: https://gitlab.com/jsnow/qemu/-/tree/python-package-mk3
MR: https://gitlab.com/jsnow/qemu/-/merge_requests/4

I invite you to leave review comments on my mock merge request on
gitlab, submitted against my own mirror. I will, as always, also respond
to feedback on-list.

ABOUT
=====

This series factors the python/qemu directory as an installable
package. It does not yet actually change the mechanics of how any other
python source in the tree actually consumes it (yet), beyond the import
path -- some import statements change in a few places.

RATIONALE
=========

The primary motivation of this series is primarily to formalize our
dependencies on mypy, flake8, isort, and pylint alongside versions that
are known to work. It does this using the setup.cfg and setup.py
files. It also adds explicitly pinned versions (using Pipfile.lock) of
these dependencies that should behave in a repeatable and known way for
developers and CI environments both. Lastly, it enables those CI checks
such that we can enforce Python coding quality checks via the CI tests.

An auxiliary motivation is that this package is formatted in such a way
that it COULD be uploaded to https://pypi.org/project/qemu and installed
independently of qemu.git with `pip install qemu`, but that button
remains *unpushed* and this series *will not* cause any such
releases. We have time to debate finer points like API guarantees and
versioning even after this series is merged.

Other bits of interest
----------------------

With the python tooling as a proper package, you can install this
package in editable or production mode to a virtual environment, your
local user environment, or your system packages. The primary benefit of
this is to gain access to QMP tooling regardless of CWD, without needing
to battle sys.path (and confounding other python analysis tools).

For example: when developing, you may go to qemu/python/ and run `make
venv` followed by `pipenv shell` to activate a virtual environment that
contains the qemu python packages. These packages will always reflect
the current version of the source files in the tree. When you are
finished, you can simply exit the shell (^d) to remove these packages
from your python environment.

When not developing, you could install a version of this package to your
environment outright to gain access to the QMP and QEMUMachine classes
for lightweight scripting and testing by using pip: "pip install
[--user] ."

TESTING THIS SERIES
===================

First of all, nothing should change. Without any intervention,
everything should behave exactly as it did before. The only new
information here comes from how to interact with and run the linters
that will be enforcing code quality standards in this subdirectory.

There are various invocations available that will test subtly different
combinations using subtly different environments. I am assuming some
light knowledge of Python environments and installing Python packages
here. If you have questions, I would be delighted to answer them.

To test the new tests, CD to ./python/ first, and then:

0. Try "make" or "make help" to get a sense of this series.

1. Try "make venv && pipenv shell" to get a venv with the package
   installed to it in editable mode. Ctrl+d exits this venv shell. While
   in this shell, any python script that uses "from qemu.[qmp|machine]
   import ..." should work correctly regardless of where the script is,
   or what your CWD is.

   This will pull some packages from PyPI and install them into the
   virtual environment, leaving your normal environment untouched.

   You will need Python 3.6 and pipenv installed on your system to do
   this step. For Fedora: "dnf install python36 pipenv" will do the
   trick. If you don't have this, skip down to #4 and onwards.

2. Try "make check" while still in the shell to run the Python linters
using the venv built in the previous step. This will run avocado, which
will in turn execute mypy, flake8, isort and pylint with the correct
arguments.

3. Having exited the shell from above, try "make venv-check". This will
create and update the venv if needed, then run 'make check' within the
context of that shell. It should pass as long as the above did. You
should be able to run "make distclean" prior to running "make
venv-check" and have the entire process work start to finish.

4. Still outside of the venv, you may try running "make check". This
will not install anything, but unless you have the right Python
dependencies installed, these tests may fail for you. You might try
using "pip install --user .[devel]" to install the development packages
needed to run the tests successfully to your local user's python
environment. Once done, you will probably want to "pip uninstall qemu"
to remove the qemu packages to avoid them interfering with other things.

5. "make distclean" will delete the venv and any temporary files that
may have been created by packaging, installing, testing, etc.

6. You may also (if you wish) create your own environment using a Python
   other than 3.6, bypassing pipenv. This may be useful for environments
   in which you simply do not have Python 3.6 readily available, or
   explicitly wish to test against a different version.

   Using the Python of your choice:

   - `> make distclean` (Remove the .venv, if you created one.)
   - `> /usr/bin/python3.9 -m venv .venv`
   - `> source .venv/bin/activate` (or activate.[fish|csh])
   - `> make develop`  (Installs linter deps to this venv)
   - `> make check`    (Runs tests using current env)
   - `> deactivate`    (Leaves the venv)

   This *should* pass, and if it doesn't, it's a bug. Due to
   dependencies changing like shifting sands, it's impossible to promise
   support for every last combination of dependency and python
   version. This is why the "make venv-check" target uses Python 3 and a
   very explicit set of packages instead. Still, this form *should*
   pass.

7. If you are running Fedora or a distro that packages multiple Python
versions, you can try running "make check-tox" to run the linters under
all Python versions from Python 3.6 to 3.9. This will create a series of
virtual environments under the .tox folder, leaving your existing
environment unscathed. This should pass, provided you have the requisite
Python versions on hand. When you are done, "make distclean" should
remove any testing files.

CHANGELOG
=========

V6
--

```
001/25:[----] [--] 'iotests/297: add --namespace-packages to mypy arguments'
002/25:[----] [-C] 'python: create qemu packages'
003/25:[----] [--] 'python: create utils sub-package'
004/25:[0005] [FC] 'python: add qemu package installer'
005/25:[0002] [FC] 'python: add VERSION file'
006/25:[----] [--] 'python: add directory structure README.rst files'
007/25:[down] 'python: add MANIFEST.in'
008/25:[0001] [FC] 'python: Add pipenv support'
009/25:[----] [--] 'python: add pylint import exceptions'
010/25:[----] [--] 'python: move pylintrc into setup.cfg'
011/25:[0083] [FC] 'python: add pylint to pipenv'
012/25:[----] [--] 'python: move flake8 config to setup.cfg'
013/25:[----] [--] 'python: add excluded dirs to flake8 config'
014/25:[0038] [FC] 'python: Add flake8 to pipenv'
015/25:[----] [--] 'python: move mypy.ini into setup.cfg'
016/25:[0050] [FC] 'python: add mypy to pipenv'
017/25:[----] [--] 'python: move .isort.cfg into setup.cfg'
018/25:[0006] [FC] 'python/qemu: add isort to pipenv'
019/25:[0004] [FC] 'python/qemu: add qemu package itself to pipenv'
020/25:[0014] [FC] 'python: add devel package requirements to setuptools'
021/25:[down] 'python: add avocado-framework and tests'
022/25:[0015] [FC] 'python: add Makefile for some common tasks'
023/25:[0018] [FC] 'python: add .gitignore'
024/25:[0004] [FC] 'gitlab: add python linters to CI'
025/25:[down] 'python: add tox support'
```

- 04: Changed maintainer blurb in PACKAGE.rst
- 05: VERSION is now 0.6.1a0
- 07: New, needed for both tox and using Pipenv to install the qemu
      package(s) in editable mode.
- 08: Add README entries explaining the purpose of the pipenv files.
- 11: pylint dependency was bumped to 2.7.0.
- 14: pipenv changes.
- 16: pipenv changes.
- 18: isort pinned at >= 5.1.2 now; see commit message.
- 19: pipenv changes.
- 20: Context differences from isort, pylint, and pipenv changes.
- 22:
   - Add 'make develop'
   - Add more README.rst entries.
   - Change pytest invocation to avocado.
- 23:
   - Add comments to gitignore, split ignores out by function/category
   - Ignore vscode files. (I like vscode now!)
- 24: Use the 'needs' feature of Gitlab CI and move the test to 'test'
  stage

STATUS
=======

```
+ [01] iotests-297-add-namespace      # [RB] CR [SOB] JS
+ [02] python-create-qemu-packages    # [SOB] JS
+ [03] python-create-utils-sub        # [SOB] JS
+ [04] python-add-qemu-package        # [SOB] JS
+ [05] python-add-version-file        # [RB] CR [SOB] JS
+ [06] python-add-directory-structure # [SOB] JS
+ [07] python-add-manifest-in         # [SOB] JS
+ [08] python-add-pipenv-support      # [RB] CR [SOB] JS
+ [09] python-add-pylint-import       # [RB] CR [SOB] JS
+ [10] python-move-pylintrc-into      # [RB] CR [TB] CR [SOB] JS
+ [11] python-add-pylint-to-pipenv    # [SOB] JS
+ [12] python-move-flake8-config-to   # [RB] CR [SOB] JS
+ [13] python-add-excluded-dirs-to    # [SOB] JS
+ [14] python-add-flake8-to-pipenv    # [RB] CR [TB] CR [SOB] JS
+ [15] python-move-mypy-ini-into      # [SOB] JS
+ [16] python-add-mypy-to-pipenv      # [RB] CR [TB] CR [SOB] JS
+ [17] python-move-isort-cfg-into     # [RB] CR [SOB] JS
+ [18] python-qemu-add-isort-to       # [SOB] JS
+ [19] python-qemu-add-qemu-package   # [RB] CR [TB] CR [SOB] JS
+ [20] python-add-devel-package       # [SOB] JS
+ [21] python-add-avocado-framework   # [SOB] JS
+ [22] python-add-makefile-for-some   # [SOB] JS
+ [23] python-add-gitignore           # [SOB] JS
+ [24] gitlab-add-python-linters-to   # [SOB] JS
> [25] python-add-tox-support         # [SOB] JS
```

John Snow (25):
  iotests/297: add --namespace-packages to mypy arguments
  python: create qemu packages
  python: create utils sub-package
  python: add qemu package installer
  python: add VERSION file
  python: add directory structure README.rst files
  python: add MANIFEST.in
  python: Add pipenv support
  python: add pylint import exceptions
  python: move pylintrc into setup.cfg
  python: add pylint to pipenv
  python: move flake8 config to setup.cfg
  python: add excluded dirs to flake8 config
  python: Add flake8 to pipenv
  python: move mypy.ini into setup.cfg
  python: add mypy to pipenv
  python: move .isort.cfg into setup.cfg
  python/qemu: add isort to pipenv
  python/qemu: add qemu package itself to pipenv
  python: add devel package requirements to setuptools
  python: add avocado-framework and tests
  python: add Makefile for some common tasks
  python: add .gitignore
  gitlab: add python linters to CI
  python: add tox support

 python/PACKAGE.rst                          |  33 +++
 python/README.rst                           |  53 +++++
 python/qemu/README.rst                      |   8 +
 python/qemu/machine/README.rst              |   9 +
 python/qemu/qmp/README.rst                  |   9 +
 python/qemu/utils/README.rst                |   7 +
 .gitlab-ci.yml                              |  12 +
 python/.gitignore                           |  20 ++
 python/MANIFEST.in                          |   2 +
 python/Makefile                             |  47 ++++
 python/Pipfile                              |  13 ++
 python/Pipfile.lock                         | 231 ++++++++++++++++++++
 python/VERSION                              |   1 +
 python/avocado.cfg                          |  10 +
 python/mypy.ini                             |   4 -
 python/qemu/.flake8                         |   2 -
 python/qemu/.isort.cfg                      |   7 -
 python/qemu/__init__.py                     |  11 -
 python/qemu/machine/__init__.py             |  36 +++
 python/qemu/{ => machine}/console_socket.py |   0
 python/qemu/{ => machine}/machine.py        |  16 +-
 python/qemu/{ => machine}/qtest.py          |   3 +-
 python/qemu/pylintrc                        |  58 -----
 python/qemu/{qmp.py => qmp/__init__.py}     |  12 +-
 python/qemu/utils/__init__.py               |  23 ++
 python/qemu/{ => utils}/accel.py            |   0
 python/setup.cfg                            |  79 +++++++
 python/setup.py                             |  23 ++
 python/tests/flake8.sh                      |   2 +
 python/tests/isort.sh                       |   2 +
 python/tests/mypy.sh                        |   2 +
 python/tests/pylint.sh                      |   2 +
 python/tox.ini                              |  13 ++
 tests/acceptance/avocado_qemu/__init__.py   |   4 +-
 tests/acceptance/virtio-gpu.py              |   2 +-
 tests/docker/dockerfiles/fedora.docker      |   2 +
 tests/qemu-iotests/297                      |   1 +
 tests/qemu-iotests/300                      |   4 +-
 tests/qemu-iotests/iotests.py               |   2 +-
 tests/vm/aarch64vm.py                       |   2 +-
 tests/vm/basevm.py                          |   2 +-
 41 files changed, 672 insertions(+), 97 deletions(-)
 create mode 100644 python/PACKAGE.rst
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/qemu/machine/README.rst
 create mode 100644 python/qemu/qmp/README.rst
 create mode 100644 python/qemu/utils/README.rst
 create mode 100644 python/.gitignore
 create mode 100644 python/MANIFEST.in
 create mode 100644 python/Makefile
 create mode 100644 python/Pipfile
 create mode 100644 python/Pipfile.lock
 create mode 100644 python/VERSION
 create mode 100644 python/avocado.cfg
 delete mode 100644 python/mypy.ini
 delete mode 100644 python/qemu/.flake8
 delete mode 100644 python/qemu/.isort.cfg
 delete mode 100644 python/qemu/__init__.py
 create mode 100644 python/qemu/machine/__init__.py
 rename python/qemu/{ => machine}/console_socket.py (100%)
 rename python/qemu/{ => machine}/machine.py (98%)
 rename python/qemu/{ => machine}/qtest.py (98%)
 delete mode 100644 python/qemu/pylintrc
 rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
 create mode 100644 python/qemu/utils/__init__.py
 rename python/qemu/{ => utils}/accel.py (100%)
 create mode 100644 python/setup.cfg
 create mode 100755 python/setup.py
 create mode 100755 python/tests/flake8.sh
 create mode 100755 python/tests/isort.sh
 create mode 100755 python/tests/mypy.sh
 create mode 100755 python/tests/pylint.sh
 create mode 100644 python/tox.ini

-- 
2.30.2




^ permalink raw reply	[flat|nested] 64+ messages in thread

end of thread, other threads:[~2021-05-25 23:56 UTC | newest]

Thread overview: 64+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
2021-05-12 23:12 ` [PATCH v6 01/25] iotests/297: add --namespace-packages to mypy arguments John Snow
2021-05-12 23:12 ` [PATCH v6 02/25] python: create qemu packages John Snow
2021-05-18 14:31   ` Cleber Rosa
2021-05-21 17:15   ` Willian Rampazzo
2021-05-12 23:12 ` [PATCH v6 03/25] python: create utils sub-package John Snow
2021-05-18 15:12   ` Cleber Rosa
2021-05-18 15:20     ` John Snow
2021-05-21 17:18   ` Willian Rampazzo
2021-05-21 17:39     ` John Snow
2021-05-12 23:12 ` [PATCH v6 04/25] python: add qemu package installer John Snow
2021-05-21  4:00   ` Cleber Rosa
2021-05-21 15:52     ` John Snow
2021-05-12 23:12 ` [PATCH v6 05/25] python: add VERSION file John Snow
2021-05-12 23:12 ` [PATCH v6 06/25] python: add directory structure README.rst files John Snow
2021-05-25  2:33   ` Cleber Rosa
2021-05-25 17:14     ` John Snow
2021-05-25 20:30       ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 07/25] python: add MANIFEST.in John Snow
2021-05-25  2:42   ` Cleber Rosa
2021-05-25 14:12     ` John Snow
2021-05-12 23:12 ` [PATCH v6 08/25] python: Add pipenv support John Snow
2021-05-12 23:12 ` [PATCH v6 09/25] python: add pylint import exceptions John Snow
2021-05-12 23:12 ` [PATCH v6 10/25] python: move pylintrc into setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 11/25] python: add pylint to pipenv John Snow
2021-05-25  3:33   ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 12/25] python: move flake8 config to setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 13/25] python: add excluded dirs to flake8 config John Snow
2021-05-25 15:50   ` Cleber Rosa
2021-05-25 17:18     ` John Snow
2021-05-12 23:12 ` [PATCH v6 14/25] python: Add flake8 to pipenv John Snow
2021-05-12 23:12 ` [PATCH v6 15/25] python: move mypy.ini into setup.cfg John Snow
2021-05-25 15:52   ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 16/25] python: add mypy to pipenv John Snow
2021-05-12 23:12 ` [PATCH v6 17/25] python: move .isort.cfg into setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 18/25] python/qemu: add isort to pipenv John Snow
2021-05-25 15:56   ` Cleber Rosa
2021-05-25 17:21     ` John Snow
2021-05-25 20:34       ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 19/25] python/qemu: add qemu package itself " John Snow
2021-05-12 23:12 ` [PATCH v6 20/25] python: add devel package requirements to setuptools John Snow
2021-05-25 16:13   ` Cleber Rosa
2021-05-25 17:43     ` John Snow
2021-05-25 20:38       ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 21/25] python: add avocado-framework and tests John Snow
2021-05-25 18:58   ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 22/25] python: add Makefile for some common tasks John Snow
2021-05-25 19:24   ` Cleber Rosa
2021-05-25 19:45     ` John Snow
2021-05-25 20:39       ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 23/25] python: add .gitignore John Snow
2021-05-25 19:36   ` Cleber Rosa
2021-05-25 20:10     ` John Snow
2021-05-25 20:42       ` Cleber Rosa
2021-05-25 23:54         ` John Snow
2021-05-12 23:12 ` [PATCH v6 24/25] gitlab: add python linters to CI John Snow
2021-05-25 19:55   ` Cleber Rosa
2021-05-25 20:33     ` John Snow
2021-05-12 23:12 ` [PATCH v6 25/25] python: add tox support John Snow
2021-05-25 20:15   ` Cleber Rosa
2021-05-25 20:25     ` John Snow
2021-05-25 20:46       ` Cleber Rosa
2021-05-25 22:15         ` John Snow
2021-05-17 18:52 ` [PATCH v6 00/25] python: create installable package John Snow

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.