[PATCH 0/7] python: create installable package

* [PATCH 0/7] python: create installable package
@ 2020-06-03  0:15 John Snow
  2020-06-03  0:15 ` [PATCH 1/7] python/qemu: create qemu.lib module John Snow
                   ` (8 more replies)
  0 siblings, 9 replies; 30+ messages in thread
From: John Snow @ 2020-06-03  0:15 UTC (permalink / raw)
  To: qemu-devel
  Cc: Kevin Wolf, Fam Zheng, Vladimir Sementsov-Ogievskiy,
	Eduardo Habkost, qemu-block, Philippe Mathieu-Daudé,
	Markus Armbruster, Max Reitz, John Snow, Stefan Hajnoczi,
	Cleber Rosa, Alex Bennée

Based-on: 20200602214528.12107-1-jsnow@redhat.com

This series factors the python/qemu directory as an installable
module. As a developer, you can install this to your virtual environment
and then always have access to the classes contained within without
needing to wrangle python import path problems.

When developing, you could go to qemu/python/ and invoke `pipenv shell`
to activate a virtual environment within which you could type `pip
install -e .` to install a special development version of this package
to your virtual environment. This package will always reflect the most
recent version of the source files in the tree.

When not developing, you could install a version of this package to your
environment outright to gain access to the QMP and QEMUMachine classes
for lightweight scripting and testing.

This package is formatted in such a way that it COULD be uploaded to
https://pypi.org/project/qemu and installed independently of qemu.git
with `pip install qemu`, but of course that button remains unpushed.

There are a few major questions to answer first:

- What versioning scheme should we use? See patch 2.

- Should we use a namespace package or not?
  - Namespaced: 'qemu.machine', 'qemu.monitor' etc may be separately
    versioned, packaged and distributed packages. Third party authors
    may register 'qemu.xxx' to create plugins within the namespace as
    they see fit.

  - Non-namespaced: 'qemu' is one giant glob package, packaged and
    versioned in unison. We control this package exclusively.

- How do we eliminate sys.path hacks from the rest of the QEMU tree?
  (Background: sys.path hacks generally impede the function of static
  code quality analysis tools like mypy and pylint.)

  - Simplest: parent scripts (or developer) needs to set PYTHONPATH.

  - Harder: Python scripts should all be written to assume package form,
    all tests and CI that use Python should execute within a VENV.

  In either case, we lose the ability (for many scripts) to "just run" a
  script out of the source tree if it depends on other QEMU Python
  files. This is annoying, but as the complexity of the Python lib
  grows, it is unavoidable.

  In the VENV case, we at least establish a simple paradigm: the scripts
  should work in their "installed" forms; and the rest of the build and
  test infrastructure should use this VENV to automatically handle
  dependencies and path requirements. This should allow us to move many
  of our existing python scripts with "hidden" dependencies into a
  proper python module hierarchy and test for regressions with mypy,
  flake8, pylint, etc.

  (We could even establish e.g. Sphinx versions as a dependency for our
  build kit here and make sure it's installed to the VENV.)

  Pros: Almost all scripts can be moved under python/qemu/* and checked
  with CQA tools. imports are written the same no matter where you are
  (Use the fully qualified names, e.g. qemu.core.qmp.QMPMessage).
  Regressions in scripts are caught *much* faster.

  Downsides: Kind of annoying; most scripts now require you to install a
  devkit forwarder (pip3 install --user .) or be inside of an activated
  venv. Not too bad if you know python at all, but it's certainly less
  plug-n-play.

- What's our backwards compatibility policy if we start shipping this?

  Proposed: Attempt to maintain API compatibility (after stabilizing the
  library). Incompatible changes should probably cause a semver bump.

  Each published release makes no attempt to support any version of QEMU
  other than the one it was released against. We publish this on the tin
  in big red letters.

TESTING THIS PACKAGE OUT:

1. You can install to your local user's environment normally by
navigating to qemu/python/ and typing "pip3 install --user ."

2. If you are in a VENV, use "pip install ."

3. To install in development mode (Where the installed package always
reflects the most recent version of the files automatically), use "pip3
install -e ." or "pip install -e ." as appropriate (See above)

John Snow (7):
  python/qemu: create qemu.lib module
  python/qemu: formalize as package
  python/qemu: add README.rst
  python/qemu: Add pipenv support
  python/qemu: add pylint to pipenv
  python/qemu: Add flake8 to pipenv
  python/qemu: add mypy to pipenv

 python/README.rst                         |   6 +
 python/qemu/README.rst                    |   8 +
 python/Pipfile                            |  14 ++
 python/Pipfile.lock                       | 195 ++++++++++++++++++++++
 python/qemu/__init__.py                   |  11 --
 python/qemu/{ => core}/.flake8            |   0
 python/qemu/core/__init__.py              |  57 +++++++
 python/qemu/{ => core}/accel.py           |   0
 python/qemu/{ => core}/machine.py         |   0
 python/qemu/{ => core}/pylintrc           |   0
 python/qemu/{ => core}/qmp.py             |   0
 python/qemu/{ => core}/qtest.py           |   0
 python/setup.py                           |  50 ++++++
 scripts/device-crash-test                 |   2 +-
 scripts/qmp/qemu-ga-client                |   2 +-
 scripts/qmp/qmp                           |   2 +-
 scripts/qmp/qmp-shell                     |   2 +-
 scripts/qmp/qom-fuse                      |   2 +-
 scripts/qmp/qom-get                       |   2 +-
 scripts/qmp/qom-list                      |   2 +-
 scripts/qmp/qom-set                       |   2 +-
 scripts/qmp/qom-tree                      |   2 +-
 scripts/render_block_graph.py             |   6 +-
 scripts/simplebench/bench_block_job.py    |   4 +-
 tests/acceptance/avocado_qemu/__init__.py |   2 +-
 tests/acceptance/boot_linux.py            |   3 +-
 tests/acceptance/virtio_check_params.py   |   2 +-
 tests/acceptance/virtio_version.py        |   2 +-
 tests/migration/guestperf/engine.py       |   2 +-
 tests/qemu-iotests/235                    |   2 +-
 tests/qemu-iotests/297                    |   2 +-
 tests/qemu-iotests/iotests.py             |   4 +-
 tests/vm/basevm.py                        |   6 +-
 33 files changed, 355 insertions(+), 39 deletions(-)
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/Pipfile
 create mode 100644 python/Pipfile.lock
 delete mode 100644 python/qemu/__init__.py
 rename python/qemu/{ => core}/.flake8 (100%)
 create mode 100644 python/qemu/core/__init__.py
 rename python/qemu/{ => core}/accel.py (100%)
 rename python/qemu/{ => core}/machine.py (100%)
 rename python/qemu/{ => core}/pylintrc (100%)
 rename python/qemu/{ => core}/qmp.py (100%)
 rename python/qemu/{ => core}/qtest.py (100%)
 create mode 100755 python/setup.py

-- 
2.21.3

^ permalink raw reply	[flat|nested] 30+ messages in thread