[PATCH v2 00/15] python: create installable package

[PATCH v2 00/15] python: create installable package

[John Snow]

Based-on: https://gitlab.com/jsnow/qemu/-/tree/python

This series factors the python/qemu directory as an installable
module. It does not yet actually change the mechanics of how any other
python source in the tree actually consumes it (yet), beyond the import
path.

The point of this series is primarily to formalize our dependencies on
mypy, flake8, isort, and pylint alongside versions that are known to
work. It also adds explicitly pinned versions of these dependencies that
should behave in a repeatable and known way for developers and CI
environments both.

With the python tooling as a proper package, you can install this
package in editable or production mode to a virtual environment, your
local user environment, or your system packages. The primary benefit of
this is to gain access to QMP tooling regardless of CWD, without needing
to battle sys.path.

For example: when developing, you may go to qemu/python/ and invoke
`pipenv shell` to activate a virtual environment that contains the qemu
packages.  This package will always reflect the current version of the
source files in the tree. When you are finished, you can simply exit the
shell to remove these packages from your python environment.

When not developing, you could install a version of this package to your
environment outright to gain access to the QMP and QEMUMachine classes
for lightweight scripting and testing by using pip: "pip install [--user] ."

Finally, this package is formatted in such a way that it COULD be
uploaded to https://pypi.org/project/qemu and installed independently of
qemu.git with `pip install qemu`, but that button remains unpushed.

TESTING THIS SERIES:

CD to qemu/python first, and then:

1. Try "pipenv shell" to get a venv with the package installed to it in
editable mode. Ctrl+d exits this venv shell. While in this shell, any
python script that uses "from qemu.core import ..." should work
correctly regardless of your CWD.

2. Try "pipenv sync --dev" to create/update the venv with the
development packages without actually entering the venv. This should
install isort, mypy, flake8 and pylint to the venv.

3. After the above sync, try "pipenv shell" again, and from the python
project root, try any of the following:

  - pylint qemu
  - flake8 qemu
  - isort -c qemu
  - mypy qemu

4. Leave any venv you are in, and from the project root, try the
following commands:

  - pipenv run pylint qemu
  - pipenv run flake8 qemu
  - pipenv run isort -c qemu
  - pipenv run mypy qemu

John Snow (15):
  python: create qemu.core package
  python: add qemu package installer
  python: add VERSION file
  python: add directory structure README.rst files
  python: Add pipenv support
  python: add pylint exceptions to __init__.py
  python: move pylintrc into setup.cfg
  python: add pylint to pipenv
  python: move flake8 config to setup.cfg
  python: Add flake8 to pipenv
  python: move mypy.ini into setup.cfg
  python: add mypy to pipenv
  python: move .isort.cfg into setup.cfg
  python/qemu: add isort to pipenv
  python/qemu: add qemu package itself to pipenv

 python/PACKAGE.rst                        |  23 +++
 python/README.rst                         |  27 +++
 python/qemu/README.rst                    |   8 +
 python/qemu/core/README.rst               |   9 +
 python/Pipfile                            |  16 ++
 python/Pipfile.lock                       | 207 ++++++++++++++++++++++
 python/VERSION                            |   1 +
 python/mypy.ini                           |   4 -
 python/qemu/.flake8                       |   2 -
 python/qemu/.isort.cfg                    |   7 -
 python/qemu/__init__.py                   |  11 --
 python/qemu/core/__init__.py              |  47 +++++
 python/qemu/{ => core}/accel.py           |   0
 python/qemu/{ => core}/console_socket.py  |   0
 python/qemu/{ => core}/machine.py         |   0
 python/qemu/{ => core}/qmp.py             |   0
 python/qemu/{ => core}/qtest.py           |   0
 python/{qemu/pylintrc => setup.cfg}       |  66 +++----
 python/setup.py                           |  23 +++
 scripts/device-crash-test                 |   2 +-
 scripts/qmp/qemu-ga-client                |   2 +-
 scripts/qmp/qmp                           |   2 +-
 scripts/qmp/qmp-shell                     |   2 +-
 scripts/qmp/qom-fuse                      |   2 +-
 scripts/qmp/qom-get                       |   2 +-
 scripts/qmp/qom-list                      |   2 +-
 scripts/qmp/qom-set                       |   2 +-
 scripts/qmp/qom-tree                      |   2 +-
 scripts/render_block_graph.py             |   6 +-
 scripts/simplebench/bench_block_job.py    |   4 +-
 tests/acceptance/avocado_qemu/__init__.py |   2 +-
 tests/acceptance/boot_linux.py            |   3 +-
 tests/acceptance/virtio_check_params.py   |   2 +-
 tests/acceptance/virtio_version.py        |   2 +-
 tests/migration/guestperf/engine.py       |   2 +-
 tests/qemu-iotests/235                    |   2 +-
 tests/qemu-iotests/297                    |   2 +-
 tests/qemu-iotests/300                    |   4 +-
 tests/qemu-iotests/iotests.py             |   4 +-
 tests/vm/basevm.py                        |   6 +-
 40 files changed, 424 insertions(+), 84 deletions(-)
 create mode 100644 python/PACKAGE.rst
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/qemu/core/README.rst
 create mode 100644 python/Pipfile
 create mode 100644 python/Pipfile.lock
 create mode 100644 python/VERSION
 delete mode 100644 python/mypy.ini
 delete mode 100644 python/qemu/.flake8
 delete mode 100644 python/qemu/.isort.cfg
 delete mode 100644 python/qemu/__init__.py
 create mode 100644 python/qemu/core/__init__.py
 rename python/qemu/{ => core}/accel.py (100%)
 rename python/qemu/{ => core}/console_socket.py (100%)
 rename python/qemu/{ => core}/machine.py (100%)
 rename python/qemu/{ => core}/qmp.py (100%)
 rename python/qemu/{ => core}/qtest.py (100%)
 rename python/{qemu/pylintrc => setup.cfg} (51%)
 mode change 100644 => 100755
 create mode 100755 python/setup.py

-- 
2.26.2

[PATCH v2 01/15] python: create qemu.core package

[John Snow]

move python/qemu/*.py to python/qemu/core/*.py and update import
directives across the tree.

This is done to create a PEP420 namespace package, in which we may
create subpackages. To do this, the namespace directory ("qemu") should
not have any modules in it. Those files will go in a new 'core'
subpackage instead.

Bolster the core/__init__.py module, making the top-level classes and
functions from this package available directly inside the `qemu.core`
namespace.

This facilitates the convenient shorthand:

> from qemu.core import QEMUQtestMachine, QEMUMonitorProtocol

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/{qemu => }/.isort.cfg              |  0
 python/qemu/__init__.py                   | 11 ------
 python/qemu/{ => core}/.flake8            |  0
 python/qemu/core/__init__.py              | 44 +++++++++++++++++++++++
 python/qemu/{ => core}/accel.py           |  0
 python/qemu/{ => core}/console_socket.py  |  0
 python/qemu/{ => core}/machine.py         |  0
 python/qemu/{ => core}/pylintrc           |  0
 python/qemu/{ => core}/qmp.py             |  0
 python/qemu/{ => core}/qtest.py           |  0
 scripts/device-crash-test                 |  2 +-
 scripts/qmp/qemu-ga-client                |  2 +-
 scripts/qmp/qmp                           |  2 +-
 scripts/qmp/qmp-shell                     |  2 +-
 scripts/qmp/qom-fuse                      |  2 +-
 scripts/qmp/qom-get                       |  2 +-
 scripts/qmp/qom-list                      |  2 +-
 scripts/qmp/qom-set                       |  2 +-
 scripts/qmp/qom-tree                      |  2 +-
 scripts/render_block_graph.py             |  6 ++--
 scripts/simplebench/bench_block_job.py    |  4 +--
 tests/acceptance/avocado_qemu/__init__.py |  2 +-
 tests/acceptance/boot_linux.py            |  3 +-
 tests/acceptance/virtio_check_params.py   |  2 +-
 tests/acceptance/virtio_version.py        |  2 +-
 tests/migration/guestperf/engine.py       |  2 +-
 tests/qemu-iotests/235                    |  2 +-
 tests/qemu-iotests/297                    |  2 +-
 tests/qemu-iotests/300                    |  4 +--
 tests/qemu-iotests/iotests.py             |  4 +--
 tests/vm/basevm.py                        |  6 ++--
 31 files changed, 71 insertions(+), 41 deletions(-)
 rename python/{qemu => }/.isort.cfg (100%)
 delete mode 100644 python/qemu/__init__.py
 rename python/qemu/{ => core}/.flake8 (100%)
 create mode 100644 python/qemu/core/__init__.py
 rename python/qemu/{ => core}/accel.py (100%)
 rename python/qemu/{ => core}/console_socket.py (100%)
 rename python/qemu/{ => core}/machine.py (100%)
 rename python/qemu/{ => core}/pylintrc (100%)
 rename python/qemu/{ => core}/qmp.py (100%)
 rename python/qemu/{ => core}/qtest.py (100%)

diff --git a/python/qemu/.isort.cfg b/python/.isort.cfg
similarity index 100%
rename from python/qemu/.isort.cfg
rename to python/.isort.cfg
diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
deleted file mode 100644
index 4ca06c34a4..0000000000
--- a/python/qemu/__init__.py
+++ /dev/null
@@ -1,11 +0,0 @@
-# QEMU library
-#
-# Copyright (C) 2015-2016 Red Hat Inc.
-# Copyright (C) 2012 IBM Corp.
-#
-# Authors:
-#  Fam Zheng <famz@redhat.com>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.  See
-# the COPYING file in the top-level directory.
-#
diff --git a/python/qemu/.flake8 b/python/qemu/core/.flake8
similarity index 100%
rename from python/qemu/.flake8
rename to python/qemu/core/.flake8
diff --git a/python/qemu/core/__init__.py b/python/qemu/core/__init__.py
new file mode 100644
index 0000000000..bf23ccd839
--- /dev/null
+++ b/python/qemu/core/__init__.py
@@ -0,0 +1,44 @@
+"""
+QEMU development and testing library.
+
+This library provides a few high-level classes for driving QEMU from a
+test suite, not intended for production use.
+
+- QEMUMachine: Configure and Boot a QEMU VM
+ - QEMUQtestMachine: VM class, with a qtest socket.
+
+- QEMUMonitorProtocol: Connect to, send/receive QMP messages.
+- QEMUQtestProtocol: Connect to, send/receive qtest message.
+
+- list_accel: List available accelerators
+- kvm_available: Probe for KVM support
+- tcg_available: Probe for TCG support
+"""
+
+# Copyright (C) 2020 John Snow for Red Hat Inc.
+# Copyright (C) 2015-2016 Red Hat Inc.
+# Copyright (C) 2012 IBM Corp.
+#
+# Authors:
+#  John Snow <jsnow@redhat.com>
+#  Fam Zheng <fam@euphon.net>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.  See
+# the COPYING file in the top-level directory.
+#
+
+from .accel import kvm_available, list_accel, tcg_available
+from .machine import QEMUMachine
+from .qmp import QEMUMonitorProtocol
+from .qtest import QEMUQtestMachine, QEMUQtestProtocol
+
+
+__all__ = (
+    'list_accel',
+    'kvm_available',
+    'tcg_available',
+    'QEMUMonitorProtocol',
+    'QEMUMachine',
+    'QEMUQtestProtocol',
+    'QEMUQtestMachine',
+)
diff --git a/python/qemu/accel.py b/python/qemu/core/accel.py
similarity index 100%
rename from python/qemu/accel.py
rename to python/qemu/core/accel.py
diff --git a/python/qemu/console_socket.py b/python/qemu/core/console_socket.py
similarity index 100%
rename from python/qemu/console_socket.py
rename to python/qemu/core/console_socket.py
diff --git a/python/qemu/machine.py b/python/qemu/core/machine.py
similarity index 100%
rename from python/qemu/machine.py
rename to python/qemu/core/machine.py
diff --git a/python/qemu/pylintrc b/python/qemu/core/pylintrc
similarity index 100%
rename from python/qemu/pylintrc
rename to python/qemu/core/pylintrc
diff --git a/python/qemu/qmp.py b/python/qemu/core/qmp.py
similarity index 100%
rename from python/qemu/qmp.py
rename to python/qemu/core/qmp.py
diff --git a/python/qemu/qtest.py b/python/qemu/core/qtest.py
similarity index 100%
rename from python/qemu/qtest.py
rename to python/qemu/core/qtest.py
diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 866baf7058..951092b6c7 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -35,7 +35,7 @@ import argparse
 from itertools import chain
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
-from qemu.machine import QEMUMachine
+from qemu.core import QEMUMachine
 
 logger = logging.getLogger('device-crash-test')
 dbg = logger.debug
diff --git a/scripts/qmp/qemu-ga-client b/scripts/qmp/qemu-ga-client
index ce122984a9..b1e8add6f5 100755
--- a/scripts/qmp/qemu-ga-client
+++ b/scripts/qmp/qemu-ga-client
@@ -42,7 +42,7 @@ import base64
 import random
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu import qmp
+from qemu.core import qmp
 
 
 class QemuGuestAgent(qmp.QEMUMonitorProtocol):
diff --git a/scripts/qmp/qmp b/scripts/qmp/qmp
index 8e52e4a54d..eaf4b0333a 100755
--- a/scripts/qmp/qmp
+++ b/scripts/qmp/qmp
@@ -13,7 +13,7 @@
 import sys, os
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.core import QEMUMonitorProtocol
 
 def print_response(rsp, prefix=[]):
     if type(rsp) == list:
diff --git a/scripts/qmp/qmp-shell b/scripts/qmp/qmp-shell
index c5eef06f3f..e819382f68 100755
--- a/scripts/qmp/qmp-shell
+++ b/scripts/qmp/qmp-shell
@@ -75,7 +75,7 @@ import atexit
 import re
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu import qmp
+from qemu.core import qmp
 
 class QMPCompleter(list):
     def complete(self, text, state):
diff --git a/scripts/qmp/qom-fuse b/scripts/qmp/qom-fuse
index 7c7cff8edf..d998524ec4 100755
--- a/scripts/qmp/qom-fuse
+++ b/scripts/qmp/qom-fuse
@@ -19,7 +19,7 @@ import os, posix, sys
 from errno import *
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.core import QEMUMonitorProtocol
 
 fuse.fuse_python_api = (0, 2)
 
diff --git a/scripts/qmp/qom-get b/scripts/qmp/qom-get
index 666df71832..cc97ae09e4 100755
--- a/scripts/qmp/qom-get
+++ b/scripts/qmp/qom-get
@@ -15,7 +15,7 @@ import sys
 import os
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.core import QEMUMonitorProtocol
 
 cmd, args = sys.argv[0], sys.argv[1:]
 socket_path = None
diff --git a/scripts/qmp/qom-list b/scripts/qmp/qom-list
index 5074fd939f..2b5e1881b0 100755
--- a/scripts/qmp/qom-list
+++ b/scripts/qmp/qom-list
@@ -15,7 +15,7 @@ import sys
 import os
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.core import QEMUMonitorProtocol
 
 cmd, args = sys.argv[0], sys.argv[1:]
 socket_path = None
diff --git a/scripts/qmp/qom-set b/scripts/qmp/qom-set
index 240a78187f..e001eebe6c 100755
--- a/scripts/qmp/qom-set
+++ b/scripts/qmp/qom-set
@@ -15,7 +15,7 @@ import sys
 import os
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.core import QEMUMonitorProtocol
 
 cmd, args = sys.argv[0], sys.argv[1:]
 socket_path = None
diff --git a/scripts/qmp/qom-tree b/scripts/qmp/qom-tree
index 25b0781323..bd677fd28a 100755
--- a/scripts/qmp/qom-tree
+++ b/scripts/qmp/qom-tree
@@ -17,7 +17,7 @@ import sys
 import os
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import QEMUMonitorProtocol
+from qemu.core import QEMUMonitorProtocol
 
 cmd, args = sys.argv[0], sys.argv[1:]
 socket_path = None
diff --git a/scripts/render_block_graph.py b/scripts/render_block_graph.py
index da6acf050d..0362185837 100755
--- a/scripts/render_block_graph.py
+++ b/scripts/render_block_graph.py
@@ -25,10 +25,8 @@
 from graphviz import Digraph
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
-from qemu.qmp import (
-    QEMUMonitorProtocol,
-    QMPResponseError,
-)
+from qemu.core import QEMUMonitorProtocol
+from qemu.core.qmp import QMPResponseError
 
 
 def perm(arr):
diff --git a/scripts/simplebench/bench_block_job.py b/scripts/simplebench/bench_block_job.py
index 9808d696cf..9ec5c0a42b 100755
--- a/scripts/simplebench/bench_block_job.py
+++ b/scripts/simplebench/bench_block_job.py
@@ -25,8 +25,8 @@
 import json
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.machine import QEMUMachine
-from qemu.qmp import QMPConnectError
+from qemu.core import QEMUMachine
+from qemu.core.qmp import QMPConnectError
 
 
 def bench_block_job(cmd, cmd_args, qemu_args):
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index 4cda037187..99ef507141 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -32,7 +32,7 @@
 
 sys.path.append(os.path.join(SOURCE_DIR, 'python'))
 
-from qemu.machine import QEMUMachine
+from qemu.core import QEMUMachine
 
 def is_readable_executable_file(path):
     return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
index 0055dc7cee..3e08383f9d 100644
--- a/tests/acceptance/boot_linux.py
+++ b/tests/acceptance/boot_linux.py
@@ -12,8 +12,7 @@
 
 from avocado_qemu import Test, BUILD_DIR
 
-from qemu.accel import kvm_available
-from qemu.accel import tcg_available
+from qemu.core import kvm_available, tcg_available
 
 from avocado.utils import cloudinit
 from avocado.utils import network
diff --git a/tests/acceptance/virtio_check_params.py b/tests/acceptance/virtio_check_params.py
index 87e6c839d1..51d9d169cc 100644
--- a/tests/acceptance/virtio_check_params.py
+++ b/tests/acceptance/virtio_check_params.py
@@ -23,7 +23,7 @@
 import logging
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.machine import QEMUMachine
+from qemu.core import QEMUMachine
 from avocado_qemu import Test
 from avocado import skip
 
diff --git a/tests/acceptance/virtio_version.py b/tests/acceptance/virtio_version.py
index 33593c29dd..dba149ec4f 100644
--- a/tests/acceptance/virtio_version.py
+++ b/tests/acceptance/virtio_version.py
@@ -12,7 +12,7 @@
 import os
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.machine import QEMUMachine
+from qemu.core import QEMUMachine
 from avocado_qemu import Test
 
 # Virtio Device IDs:
diff --git a/tests/migration/guestperf/engine.py b/tests/migration/guestperf/engine.py
index fd63c66601..fef02e157c 100644
--- a/tests/migration/guestperf/engine.py
+++ b/tests/migration/guestperf/engine.py
@@ -29,7 +29,7 @@
 
 sys.path.append(os.path.join(os.path.dirname(__file__),
                              '..', '..', '..', 'python'))
-from qemu.machine import QEMUMachine
+from qemu.core import QEMUMachine
 
 
 class Engine(object):
diff --git a/tests/qemu-iotests/235 b/tests/qemu-iotests/235
index d1b10ac36b..4fc4e60041 100755
--- a/tests/qemu-iotests/235
+++ b/tests/qemu-iotests/235
@@ -25,7 +25,7 @@ from iotests import qemu_img_create, qemu_io, file_path, log
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 
-from qemu.machine import QEMUMachine
+from qemu.core import QEMUMachine
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 
diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index 5c5420712b..8236875222 100755
--- a/tests/qemu-iotests/297
+++ b/tests/qemu-iotests/297
@@ -36,7 +36,7 @@ MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
     --disallow-any-generics --disallow-incomplete-defs \
     --disallow-untyped-decorators --no-implicit-optional \
     --warn-redundant-casts --warn-unused-ignores \
-    --no-implicit-reexport iotests.py
+    --no-implicit-reexport --namespace-packages iotests.py
 
 # success, all done
 echo "*** done"
diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
index 5b75121b84..31a4e7d412 100755
--- a/tests/qemu-iotests/300
+++ b/tests/qemu-iotests/300
@@ -23,7 +23,7 @@ import random
 import re
 from typing import Dict, List, Optional, Union
 import iotests
-import qemu
+from qemu.core import machine
 
 BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
 
@@ -454,7 +454,7 @@ class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
         # the failed migration
         try:
             self.vm_b.shutdown()
-        except qemu.machine.AbnormalShutdown:
+        except machine.AbnormalShutdown:
             pass
 
     def test_aliased_bitmap_name_too_long(self) -> None:
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 63d2ace93c..0038fec992 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -37,8 +37,8 @@
 
 # pylint: disable=import-error, wrong-import-position
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu import qtest
-from qemu.qmp import QMPMessage
+from qemu.core import qtest
+from qemu.core.qmp import QMPMessage
 
 # Use this logger for logging messages directly from the iotests module
 logger = logging.getLogger('qemu.iotests')
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 3fac20e929..dd3d070e3b 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -18,9 +18,6 @@
 import logging
 import time
 import datetime
-sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.accel import kvm_available
-from qemu.machine import QEMUMachine
 import subprocess
 import hashlib
 import argparse
@@ -31,6 +28,9 @@
 import traceback
 import shlex
 
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
+from qemu.core import kvm_available, QEMUMachine
+
 SSH_KEY_FILE = os.path.join(os.path.dirname(__file__),
                "..", "keys", "id_rsa")
 SSH_PUB_KEY_FILE = os.path.join(os.path.dirname(__file__),
-- 
2.26.2

[PATCH v2 02/15] python: add qemu package installer

[John Snow]

Add setup.cfg and setup.py, necessary for installing a package via
pip. Add a rst document explaining the basics of what this package is
for and who to contact for more information. This document will be used
as the landing page for the package on PyPI.

I am not yet using a pyproject.toml style package manifest, because
using pyproject.toml (and PEP-517) style packages means that pip is not
able to install in "editable" or "develop" mode, which I consider
necessary for the development of this package.

Use a light-weight setup.py instead.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/PACKAGE.rst | 23 +++++++++++++++++++++++
 python/setup.cfg   | 18 ++++++++++++++++++
 python/setup.py    | 23 +++++++++++++++++++++++
 3 files changed, 64 insertions(+)
 create mode 100644 python/PACKAGE.rst
 create mode 100755 python/setup.cfg
 create mode 100755 python/setup.py

diff --git a/python/PACKAGE.rst b/python/PACKAGE.rst
new file mode 100644
index 0000000000..6e82f05606
--- /dev/null
+++ b/python/PACKAGE.rst
@@ -0,0 +1,23 @@
+QEMU Python Tooling
+===================
+
+This package provides QEMU tooling used by the QEMU project to build,
+configure, and test QEMU. It is not a fully-fledged SDK and it is subject
+to change at any time.
+
+Usage
+-----
+
+The ``qemu.core`` subpackage offers rudimentary facilities for launching
+QEMU and communicating with it via QMP. Refer to the module documentation
+(``>>> help(qemu.core)``) for more information.
+
+Contributing
+------------
+
+This package is maintained by John Snow <jsnow@redhat.com> as part of
+the QEMU source tree. Contributions are welcome and follow the `QEMU
+patch submission process
+<https://wiki.qemu.org/Contribute/SubmitAPatch>`_. There is a `Gitlab
+mirror <https://gitlab.com/jsnow/qemu/-/tree/python>`_, but
+contributions must be sent to the list for inclusion.
diff --git a/python/setup.cfg b/python/setup.cfg
new file mode 100755
index 0000000000..12b99a796e
--- /dev/null
+++ b/python/setup.cfg
@@ -0,0 +1,18 @@
+[metadata]
+name = qemu
+maintainer = QEMU Developer Team
+maintainer_email = qemu-devel@nongnu.org
+url = https://www.qemu.org/
+download_url = https://www.qemu.org/download/
+description = QEMU Python Build, Debug and SDK tooling.
+long_description = file:PACKAGE.rst
+long_description_content_type = text/x-rst
+classifiers =
+    Development Status :: 3 - Alpha
+    License :: OSI Approved :: GNU General Public License v2 (GPLv2)
+    Natural Language :: English
+    Operating System :: OS Independent
+
+[options]
+python_requires = >= 3.6
+packages = find_namespace:
diff --git a/python/setup.py b/python/setup.py
new file mode 100755
index 0000000000..e93d0075d1
--- /dev/null
+++ b/python/setup.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+"""
+QEMU tooling installer script
+Copyright (c) 2020 John Snow for Red Hat, Inc.
+"""
+
+import setuptools
+import pkg_resources
+
+
+def main():
+    """
+    QEMU tooling installer
+    """
+
+    # https://medium.com/@daveshawley/safely-using-setup-cfg-for-metadata-1babbe54c108
+    pkg_resources.require('setuptools>=39.2')
+
+    setuptools.setup()
+
+
+if __name__ == '__main__':
+    main()
-- 
2.26.2

[PATCH v2 03/15] python: add VERSION file

[John Snow]

Python infrastructure as it exists today is not capable reliably of
single-sourcing a package version from a parent directory. The authors
of pip are working to correct this, but as of today this is not possible
to my knowledge.

The problem is that when using pip to build and install a python
package, it copies files over to a temporary directory and performs its
build there. This loses access to any information in the parent
directory, including git itself.

Further, Python versions have a standard (PEP 440) that may or may not
follow QEMU's versioning. In general, it does; but naturally QEMU does
not follow PEP 440. To avoid any automatically-generated conflict, a
manual version file is preferred.


I am proposing:

- Python core tooling synchronizes with the QEMU version directly
  (5.2.0, 5.1.1, 5.3.0, etc.)

- In the event that a Python package needs to be updated independently
  of the QEMU version, a pre-release alpha version should be preferred,
  but *only* after inclusion to the qemu development or stable branches.

  e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's
  release.

- The Python core tooling makes absolutely no version compatibility
  checks or constraints. It *may* work with releases of QEMU from the
  past or future, but it is not required to.

  i.e., "qemu.core" will always remain in lock-step with QEMU.

- We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp
  and begin indepedently versioning such a package separately from the
  QEMU version it accompanies.


Implement this versioning scheme by adding a VERSION file and setting it
to 5.2.0a1.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/VERSION   | 1 +
 python/setup.cfg | 1 +
 2 files changed, 2 insertions(+)
 create mode 100644 python/VERSION

diff --git a/python/VERSION b/python/VERSION
new file mode 100644
index 0000000000..2e81039c82
--- /dev/null
+++ b/python/VERSION
@@ -0,0 +1 @@
+5.2.0a1
diff --git a/python/setup.cfg b/python/setup.cfg
index 12b99a796e..260f7f4e94 100755
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -1,5 +1,6 @@
 [metadata]
 name = qemu
+version = file:VERSION
 maintainer = QEMU Developer Team
 maintainer_email = qemu-devel@nongnu.org
 url = https://www.qemu.org/
-- 
2.26.2

[PATCH v2 04/15] python: add directory structure README.rst files

[John Snow]

Add short readmes to python/, python/qemu/, and python/qemu/core that
explain the directory hierarchy. These readmes are visible when browsing
the source on e.g. gitlab/github and are designed to help new
developers/users quickly make sense of the source tree.

They are not designed for inclusion in a published manual.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst           | 27 +++++++++++++++++++++++++++
 python/qemu/README.rst      |  8 ++++++++
 python/qemu/core/README.rst |  9 +++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 python/README.rst
 create mode 100644 python/qemu/README.rst
 create mode 100644 python/qemu/core/README.rst

diff --git a/python/README.rst b/python/README.rst
new file mode 100644
index 0000000000..fcc0552ec4
--- /dev/null
+++ b/python/README.rst
@@ -0,0 +1,27 @@
+QEMU Python Tooling
+===================
+
+This directory houses Python tooling used by the QEMU project to build,
+configure, and test QEMU. It is organized by namespace (``qemu``), and
+then by package (``qemu/core``).
+
+``setup.py`` is used by ``pip`` to install this tooling to the current
+environment. You will generally invoke it by doing one of the following:
+
+1. ``pip3 install .`` will install these packages to your current
+   environment. If you are inside a virtual environment, they will
+   install there. If you are not, it will attempt to install to the
+   global environment, which is not recommended.
+
+2. ``pip3 install --user .`` will install these packages to your user's
+   local python packages. If you are inside of a virtual environment,
+   this will fail.
+
+If you amend the ``-e`` argument, pip will install in "editable" mode;
+which installs a version of the package that uses symlinks to these
+files, such that the package always reflects the latest version in your
+git tree.
+
+See `Installing packages using pip and virtual environments
+<https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/>`_
+for more information.
diff --git a/python/qemu/README.rst b/python/qemu/README.rst
new file mode 100644
index 0000000000..31209c80a5
--- /dev/null
+++ b/python/qemu/README.rst
@@ -0,0 +1,8 @@
+QEMU Python Namespace
+=====================
+
+This directory serves as the root of a `Python PEP 420 implicit
+namespace package <<https://www.python.org/dev/peps/pep-0420/>`_.
+
+Each directory below is assumed to be an installable Python package that
+is available under the ``qemu.<package>`` namespace.
diff --git a/python/qemu/core/README.rst b/python/qemu/core/README.rst
new file mode 100644
index 0000000000..91668e00bd
--- /dev/null
+++ b/python/qemu/core/README.rst
@@ -0,0 +1,9 @@
+qemu.core Package
+=================
+
+This package provides core utilities used for testing and debugging
+QEMU. It is used by the iotests, vm tests, and several other utilities
+in the ./scripts directory. It is not a fully-fledged SDK and it is
+subject to change at any time.
+
+See the documentation in ``__init__.py`` for more information.
-- 
2.26.2

[PATCH v2 05/15] python: Add pipenv support

[John Snow]

pipenv is a tool used for managing virtual environments with pinned,
explicit dependencies. It is used for precisely recreating python
virtual environments.

pipenv uses two files to do this:

(1) Pipfile, which is similar in purpose and scope to what setup.py
lists. It specifies the requisite minimum to get a functional
environment for using this package.

(2) Pipfile.lock, which is similar in purpose to `pip freeze >
requirements.txt`. It specifies a canonical virtual environment used for
deployment or testing. This ensures that all users have repeatable
results.

The primary benefit of using this tool is to ensure repeatable CI
results with a known set of packages. Although I endeavor to support as
many versions as I can, the fluid nature of the Python toolchain often
means tailoring code for fairly specific versions.

Note that pipenv is *not* required to install or use this module; this is
purely for the sake of repeatable testing by CI or developers.

Here, a "blank" pipfile is added with no dependencies, but specifies
Python 3.6 for the virtual environment.

Pipfile will specify our version minimums, while Pipfile.lock specifies
an exact loudout of packages that were known to operate correctly. This
latter file provides the real value for easy setup of container images
and CI environments.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100644 python/Pipfile

diff --git a/python/Pipfile b/python/Pipfile
new file mode 100644
index 0000000000..9534830b5e
--- /dev/null
+++ b/python/Pipfile
@@ -0,0 +1,11 @@
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple"
+verify_ssl = true
+
+[dev-packages]
+
+[packages]
+
+[requires]
+python_version = "3.6"
-- 
2.26.2

[PATCH v2 06/15] python: add pylint exceptions to __init__.py

[John Snow]

Pylint 2.5.x and 2.6.x have regressions that make import checking
inconsistent, see:

https: //github.com/PyCQA/pylint/issues/3609
https: //github.com/PyCQA/pylint/issues/3624
https: //github.com/PyCQA/pylint/issues/3651
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/core/__init__.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/python/qemu/core/__init__.py b/python/qemu/core/__init__.py
index bf23ccd839..804a414dfb 100644
--- a/python/qemu/core/__init__.py
+++ b/python/qemu/core/__init__.py
@@ -27,6 +27,9 @@
 # the COPYING file in the top-level directory.
 #
 
+# pylint: disable=import-error
+# see: https://github.com/PyCQA/pylint/issues/3624
+# see: https://github.com/PyCQA/pylint/issues/3651
 from .accel import kvm_available, list_accel, tcg_available
 from .machine import QEMUMachine
 from .qmp import QEMUMonitorProtocol
-- 
2.26.2

[PATCH v2 07/15] python: move pylintrc into setup.cfg

[John Snow]

Delete the empty settings now that it's sharing a home with settings for
other tools.

pylint can now be run from this folder as "pylint qemu".
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/core/pylintrc | 58 ---------------------------------------
 python/setup.cfg          | 29 ++++++++++++++++++++
 2 files changed, 29 insertions(+), 58 deletions(-)
 delete mode 100644 python/qemu/core/pylintrc

diff --git a/python/qemu/core/pylintrc b/python/qemu/core/pylintrc
deleted file mode 100644
index 3f69205000..0000000000
--- a/python/qemu/core/pylintrc
+++ /dev/null
@@ -1,58 +0,0 @@
-[MASTER]
-
-[MESSAGES CONTROL]
-
-# Disable the message, report, category or checker with the given id(s). You
-# can either give multiple identifiers separated by comma (,) or put this
-# option multiple times (only on the command line, not in the configuration
-# file where it should appear only once). You can also use "--disable=all" to
-# disable everything first and then reenable specific checks. For example, if
-# you want to run only the similarities checker, you can use "--disable=all
-# --enable=similarities". If you want to run only the classes checker, but have
-# no Warning level messages displayed, use "--disable=all --enable=classes
-# --disable=W".
-disable=too-many-arguments,
-        too-many-instance-attributes,
-        too-many-public-methods,
-
-[REPORTS]
-
-[REFACTORING]
-
-[MISCELLANEOUS]
-
-[LOGGING]
-
-[BASIC]
-
-# Good variable names which should always be accepted, separated by a comma.
-good-names=i,
-           j,
-           k,
-           ex,
-           Run,
-           _,
-           fd,
-           c,
-[VARIABLES]
-
-[STRING]
-
-[SPELLING]
-
-[FORMAT]
-
-[SIMILARITIES]
-
-# Ignore imports when computing similarities.
-ignore-imports=yes
-
-[TYPECHECK]
-
-[CLASSES]
-
-[IMPORTS]
-
-[DESIGN]
-
-[EXCEPTIONS]
diff --git a/python/setup.cfg b/python/setup.cfg
index 260f7f4e94..a65802d85e 100755
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -17,3 +17,32 @@ classifiers =
 [options]
 python_requires = >= 3.6
 packages = find_namespace:
+
+[pylint.messages control]
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=too-many-arguments,
+        too-many-instance-attributes,
+        too-many-public-methods,
+
+[pylint.basic]
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _,
+           fd,
+           c,
+
+[pylint.similarities]
+# Ignore imports when computing similarities.
+ignore-imports=yes
-- 
2.26.2

[PATCH v2 08/15] python: add pylint to pipenv

[John Snow]

We are specifying >= pylint 2.6.x for two reasons:

1. For setup.cfg support, added in pylint 2.5.x
2. To clarify that we are using a version that has incompatibly dropped
bad-whitespace checks.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile      |   1 +
 python/Pipfile.lock | 127 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 128 insertions(+)
 create mode 100644 python/Pipfile.lock

diff --git a/python/Pipfile b/python/Pipfile
index 9534830b5e..1e58986c89 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -4,6 +4,7 @@ url = "https://pypi.org/simple"
 verify_ssl = true
 
 [dev-packages]
+pylint = ">=2.6.0"
 
 [packages]
 
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
new file mode 100644
index 0000000000..71e0d40ffb
--- /dev/null
+++ b/python/Pipfile.lock
@@ -0,0 +1,127 @@
+{
+    "_meta": {
+        "hash": {
+            "sha256": "b7ac1f2ad73bc166244c0378298afba64951a16bb749b81a9668dc41f40f941c"
+        },
+        "pipfile-spec": 6,
+        "requires": {
+            "python_version": "3.6"
+        },
+        "sources": [
+            {
+                "name": "pypi",
+                "url": "https://pypi.org/simple",
+                "verify_ssl": true
+            }
+        ]
+    },
+    "default": {},
+    "develop": {
+        "astroid": {
+            "hashes": [
+                "sha256:2f4078c2a41bf377eea06d71c9d2ba4eb8f6b1af2135bec27bbbb7d8f12bb703",
+                "sha256:bc58d83eb610252fd8de6363e39d4f1d0619c894b0ed24603b881c02e64c7386"
+            ],
+            "markers": "python_version >= '3.5'",
+            "version": "==2.4.2"
+        },
+        "isort": {
+            "hashes": [
+                "sha256:dcab1d98b469a12a1a624ead220584391648790275560e1a43e54c5dceae65e7",
+                "sha256:dcaeec1b5f0eca77faea2a35ab790b4f3680ff75590bfcb7145986905aab2f58"
+            ],
+            "markers": "python_version >= '3.6' and python_version < '4.0'",
+            "version": "==5.6.4"
+        },
+        "lazy-object-proxy": {
+            "hashes": [
+                "sha256:0c4b206227a8097f05c4dbdd323c50edf81f15db3b8dc064d08c62d37e1a504d",
+                "sha256:194d092e6f246b906e8f70884e620e459fc54db3259e60cf69a4d66c3fda3449",
+                "sha256:1be7e4c9f96948003609aa6c974ae59830a6baecc5376c25c92d7d697e684c08",
+                "sha256:4677f594e474c91da97f489fea5b7daa17b5517190899cf213697e48d3902f5a",
+                "sha256:48dab84ebd4831077b150572aec802f303117c8cc5c871e182447281ebf3ac50",
+                "sha256:5541cada25cd173702dbd99f8e22434105456314462326f06dba3e180f203dfd",
+                "sha256:59f79fef100b09564bc2df42ea2d8d21a64fdcda64979c0fa3db7bdaabaf6239",
+                "sha256:8d859b89baf8ef7f8bc6b00aa20316483d67f0b1cbf422f5b4dc56701c8f2ffb",
+                "sha256:9254f4358b9b541e3441b007a0ea0764b9d056afdeafc1a5569eee1cc6c1b9ea",
+                "sha256:9651375199045a358eb6741df3e02a651e0330be090b3bc79f6d0de31a80ec3e",
+                "sha256:97bb5884f6f1cdce0099f86b907aa41c970c3c672ac8b9c8352789e103cf3156",
+                "sha256:9b15f3f4c0f35727d3a0fba4b770b3c4ebbb1fa907dbcc046a1d2799f3edd142",
+                "sha256:a2238e9d1bb71a56cd710611a1614d1194dc10a175c1e08d75e1a7bcc250d442",
+                "sha256:a6ae12d08c0bf9909ce12385803a543bfe99b95fe01e752536a60af2b7797c62",
+                "sha256:ca0a928a3ddbc5725be2dd1cf895ec0a254798915fb3a36af0964a0a4149e3db",
+                "sha256:cb2c7c57005a6804ab66f106ceb8482da55f5314b7fcb06551db1edae4ad1531",
+                "sha256:d74bb8693bf9cf75ac3b47a54d716bbb1a92648d5f781fc799347cfc95952383",
+                "sha256:d945239a5639b3ff35b70a88c5f2f491913eb94871780ebfabb2568bd58afc5a",
+                "sha256:eba7011090323c1dadf18b3b689845fd96a61ba0a1dfbd7f24b921398affc357",
+                "sha256:efa1909120ce98bbb3777e8b6f92237f5d5c8ea6758efea36a473e1d38f7d3e4",
+                "sha256:f3900e8a5de27447acbf900b4750b0ddfd7ec1ea7fbaf11dfa911141bc522af0"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==1.4.3"
+        },
+        "mccabe": {
+            "hashes": [
+                "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
+                "sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
+            ],
+            "version": "==0.6.1"
+        },
+        "pylint": {
+            "hashes": [
+                "sha256:bb4a908c9dadbc3aac18860550e870f58e1a02c9f2c204fdf5693d73be061210",
+                "sha256:bfe68f020f8a0fece830a22dd4d5dddb4ecc6137db04face4c3420a46a52239f"
+            ],
+            "index": "pypi",
+            "version": "==2.6.0"
+        },
+        "six": {
+            "hashes": [
+                "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259",
+                "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2'",
+            "version": "==1.15.0"
+        },
+        "toml": {
+            "hashes": [
+                "sha256:926b612be1e5ce0634a2ca03470f95169cf16f939018233a670519cb4ac58b0f",
+                "sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88"
+            ],
+            "version": "==0.10.1"
+        },
+        "typed-ast": {
+            "hashes": [
+                "sha256:0666aa36131496aed8f7be0410ff974562ab7eeac11ef351def9ea6fa28f6355",
+                "sha256:0c2c07682d61a629b68433afb159376e24e5b2fd4641d35424e462169c0a7919",
+                "sha256:249862707802d40f7f29f6e1aad8d84b5aa9e44552d2cc17384b209f091276aa",
+                "sha256:24995c843eb0ad11a4527b026b4dde3da70e1f2d8806c99b7b4a7cf491612652",
+                "sha256:269151951236b0f9a6f04015a9004084a5ab0d5f19b57de779f908621e7d8b75",
+                "sha256:4083861b0aa07990b619bd7ddc365eb7fa4b817e99cf5f8d9cf21a42780f6e01",
+                "sha256:498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d",
+                "sha256:4e3e5da80ccbebfff202a67bf900d081906c358ccc3d5e3c8aea42fdfdfd51c1",
+                "sha256:6daac9731f172c2a22ade6ed0c00197ee7cc1221aa84cfdf9c31defeb059a907",
+                "sha256:715ff2f2df46121071622063fc7543d9b1fd19ebfc4f5c8895af64a77a8c852c",
+                "sha256:73d785a950fc82dd2a25897d525d003f6378d1cb23ab305578394694202a58c3",
+                "sha256:8c8aaad94455178e3187ab22c8b01a3837f8ee50e09cf31f1ba129eb293ec30b",
+                "sha256:8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614",
+                "sha256:aaee9905aee35ba5905cfb3c62f3e83b3bec7b39413f0a7f19be4e547ea01ebb",
+                "sha256:bcd3b13b56ea479b3650b82cabd6b5343a625b0ced5429e4ccad28a8973f301b",
+                "sha256:c9e348e02e4d2b4a8b2eedb48210430658df6951fa484e59de33ff773fbd4b41",
+                "sha256:d205b1b46085271b4e15f670058ce182bd1199e56b317bf2ec004b6a44f911f6",
+                "sha256:d43943ef777f9a1c42bf4e552ba23ac77a6351de620aa9acf64ad54933ad4d34",
+                "sha256:d5d33e9e7af3b34a40dc05f498939f0ebf187f07c385fd58d591c533ad8562fe",
+                "sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4",
+                "sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"
+            ],
+            "markers": "implementation_name == 'cpython' and python_version < '3.8'",
+            "version": "==1.4.1"
+        },
+        "wrapt": {
+            "hashes": [
+                "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
+            ],
+            "version": "==1.12.1"
+        }
+    }
+}
-- 
2.26.2

[PATCH v2 09/15] python: move flake8 config to setup.cfg

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/core/.flake8 | 2 --
 python/setup.cfg         | 3 +++
 2 files changed, 3 insertions(+), 2 deletions(-)
 delete mode 100644 python/qemu/core/.flake8

diff --git a/python/qemu/core/.flake8 b/python/qemu/core/.flake8
deleted file mode 100644
index 45d8146f3f..0000000000
--- a/python/qemu/core/.flake8
+++ /dev/null
@@ -1,2 +0,0 @@
-[flake8]
-extend-ignore = E722  # Pylint handles this, but smarter.
\ No newline at end of file
diff --git a/python/setup.cfg b/python/setup.cfg
index a65802d85e..103371ad5e 100755
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -18,6 +18,9 @@ classifiers =
 python_requires = >= 3.6
 packages = find_namespace:
 
+[flake8]
+extend-ignore = E722  # Pylint handles this, but smarter.
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
-- 
2.26.2

[PATCH v2 10/15] python: Add flake8 to pipenv

[John Snow]

flake8 3.5.x does not support the --extend-ignore syntax used in the
.flake8 file to gracefully extend default ignores, so 3.6.x is our
minimum requirement. There is no known upper bound.

flake8 can be run from the python/ directory with no arguments.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile      |  1 +
 python/Pipfile.lock | 42 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/python/Pipfile b/python/Pipfile
index 1e58986c89..d1f7045f68 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -4,6 +4,7 @@ url = "https://pypi.org/simple"
 verify_ssl = true
 
 [dev-packages]
+flake8 = ">=3.6.0"
 pylint = ">=2.6.0"
 
 [packages]
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 71e0d40ffb..eb5ae75a0c 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "b7ac1f2ad73bc166244c0378298afba64951a16bb749b81a9668dc41f40f941c"
+            "sha256": "9f6d4857a6c72ad40fc1ec1e58cdb76f187a2986ac4156f0027e5eb798ec69a9"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -25,6 +25,22 @@
             "markers": "python_version >= '3.5'",
             "version": "==2.4.2"
         },
+        "flake8": {
+            "hashes": [
+                "sha256:749dbbd6bfd0cf1318af27bf97a14e28e5ff548ef8e5b1566ccfb25a11e7c839",
+                "sha256:aadae8761ec651813c24be05c6f7b4680857ef6afaae4651a4eccaef97ce6c3b"
+            ],
+            "index": "pypi",
+            "version": "==3.8.4"
+        },
+        "importlib-metadata": {
+            "hashes": [
+                "sha256:77a540690e24b0305878c37ffd421785a6f7e53c8b5720d211b211de8d0e95da",
+                "sha256:cefa1a2f919b866c5beb7c9f7b0ebb4061f30a8a9bf16d609b000e2dfaceb9c3"
+            ],
+            "markers": "python_version < '3.8'",
+            "version": "==2.0.0"
+        },
         "isort": {
             "hashes": [
                 "sha256:dcab1d98b469a12a1a624ead220584391648790275560e1a43e54c5dceae65e7",
@@ -67,6 +83,22 @@
             ],
             "version": "==0.6.1"
         },
+        "pycodestyle": {
+            "hashes": [
+                "sha256:2295e7b2f6b5bd100585ebcb1f616591b652db8a741695b3d8f5d28bdc934367",
+                "sha256:c58a7d2815e0e8d7972bf1803331fb0152f867bd89adf8a01dfd55085434192e"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==2.6.0"
+        },
+        "pyflakes": {
+            "hashes": [
+                "sha256:0d94e0e05a19e57a99444b6ddcf9a6eb2e5c68d3ca1e98e90707af8152c90a92",
+                "sha256:35b2d75ee967ea93b55750aa9edbbf72813e06a66ba54438df2cfac9e3c27fc8"
+            ],
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
+            "version": "==2.2.0"
+        },
         "pylint": {
             "hashes": [
                 "sha256:bb4a908c9dadbc3aac18860550e870f58e1a02c9f2c204fdf5693d73be061210",
@@ -122,6 +154,14 @@
                 "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
             ],
             "version": "==1.12.1"
+        },
+        "zipp": {
+            "hashes": [
+                "sha256:64ad89efee774d1897a58607895d80789c59778ea02185dd846ac38394a8642b",
+                "sha256:eed8ec0b8d1416b2ca33516a37a08892442f3954dee131e92cfd92d8fe3e7066"
+            ],
+            "markers": "python_version >= '3.6'",
+            "version": "==3.3.0"
         }
     }
 }
-- 
2.26.2

[PATCH v2 11/15] python: move mypy.ini into setup.cfg

[John Snow]

mypy supports reading its configuration values from a central project
configuration file; do so.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/mypy.ini  | 4 ----
 python/setup.cfg | 5 +++++
 2 files changed, 5 insertions(+), 4 deletions(-)
 delete mode 100644 python/mypy.ini

diff --git a/python/mypy.ini b/python/mypy.ini
deleted file mode 100644
index 1a581c5f1e..0000000000
--- a/python/mypy.ini
+++ /dev/null
@@ -1,4 +0,0 @@
-[mypy]
-strict = True
-python_version = 3.6
-warn_unused_configs = True
diff --git a/python/setup.cfg b/python/setup.cfg
index 103371ad5e..13a7a8cd9c 100755
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -21,6 +21,11 @@ packages = find_namespace:
 [flake8]
 extend-ignore = E722  # Pylint handles this, but smarter.
 
+[mypy]
+strict = True
+python_version = 3.6
+warn_unused_configs = True
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
-- 
2.26.2

[PATCH v2 12/15] python: add mypy to pipenv

[John Snow]

0.730 appears to be about the oldest version that works with the
features we want, including nice human readable output (to make sure
iotest 297 passes), and type-parameterized Popen generics.

0.770, however, supports adding 'strict' to the config file, so require
at least 0.770.

mypy can be run from the python root by typing 'mypy qemu'.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile      |  1 +
 python/Pipfile.lock | 37 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/python/Pipfile b/python/Pipfile
index d1f7045f68..51c537b0d1 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -5,6 +5,7 @@ verify_ssl = true
 
 [dev-packages]
 flake8 = ">=3.6.0"
+mypy = ">=0.770"
 pylint = ">=2.6.0"
 
 [packages]
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index eb5ae75a0c..376f95a6a8 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "9f6d4857a6c72ad40fc1ec1e58cdb76f187a2986ac4156f0027e5eb798ec69a9"
+            "sha256": "65f010a8f2e55e9870d2b7e0d8af129516097d23abf2504f396552748b067ade"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -83,6 +83,33 @@
             ],
             "version": "==0.6.1"
         },
+        "mypy": {
+            "hashes": [
+                "sha256:0a0d102247c16ce93c97066443d11e2d36e6cc2a32d8ccc1f705268970479324",
+                "sha256:0d34d6b122597d48a36d6c59e35341f410d4abfa771d96d04ae2c468dd201abc",
+                "sha256:2170492030f6faa537647d29945786d297e4862765f0b4ac5930ff62e300d802",
+                "sha256:2842d4fbd1b12ab422346376aad03ff5d0805b706102e475e962370f874a5122",
+                "sha256:2b21ba45ad9ef2e2eb88ce4aeadd0112d0f5026418324176fd494a6824b74975",
+                "sha256:72060bf64f290fb629bd4a67c707a66fd88ca26e413a91384b18db3876e57ed7",
+                "sha256:af4e9ff1834e565f1baa74ccf7ae2564ae38c8df2a85b057af1dbbc958eb6666",
+                "sha256:bd03b3cf666bff8d710d633d1c56ab7facbdc204d567715cb3b9f85c6e94f669",
+                "sha256:c614194e01c85bb2e551c421397e49afb2872c88b5830e3554f0519f9fb1c178",
+                "sha256:cf4e7bf7f1214826cf7333627cb2547c0db7e3078723227820d0a2490f117a01",
+                "sha256:da56dedcd7cd502ccd3c5dddc656cb36113dd793ad466e894574125945653cea",
+                "sha256:e86bdace26c5fe9cf8cb735e7cedfe7850ad92b327ac5d797c656717d2ca66de",
+                "sha256:e97e9c13d67fbe524be17e4d8025d51a7dca38f90de2e462243ab8ed8a9178d1",
+                "sha256:eea260feb1830a627fb526d22fbb426b750d9f5a47b624e8d5e7e004359b219c"
+            ],
+            "index": "pypi",
+            "version": "==0.790"
+        },
+        "mypy-extensions": {
+            "hashes": [
+                "sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d",
+                "sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"
+            ],
+            "version": "==0.4.3"
+        },
         "pycodestyle": {
             "hashes": [
                 "sha256:2295e7b2f6b5bd100585ebcb1f616591b652db8a741695b3d8f5d28bdc934367",
@@ -149,6 +176,14 @@
             "markers": "implementation_name == 'cpython' and python_version < '3.8'",
             "version": "==1.4.1"
         },
+        "typing-extensions": {
+            "hashes": [
+                "sha256:7cb407020f00f7bfc3cb3e7881628838e69d8f3fcab2f64742a5e76b2f841918",
+                "sha256:99d4073b617d30288f569d3f13d2bd7548c3a7e4c8de87db09a9d29bb3a4a60c",
+                "sha256:dafc7639cde7f1b6e1acc0f457842a83e722ccca8eef5270af2d74792619a89f"
+            ],
+            "version": "==3.7.4.3"
+        },
         "wrapt": {
             "hashes": [
                 "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
-- 
2.26.2

[PATCH v2 13/15] python: move .isort.cfg into setup.cfg

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/.isort.cfg | 7 -------
 python/setup.cfg  | 8 ++++++++
 2 files changed, 8 insertions(+), 7 deletions(-)
 delete mode 100644 python/.isort.cfg

diff --git a/python/.isort.cfg b/python/.isort.cfg
deleted file mode 100644
index 6d0fd6cc0d..0000000000
--- a/python/.isort.cfg
+++ /dev/null
@@ -1,7 +0,0 @@
-[settings]
-force_grid_wrap=4
-force_sort_within_sections=True
-include_trailing_comma=True
-line_length=72
-lines_after_imports=2
-multi_line_output=3
\ No newline at end of file
diff --git a/python/setup.cfg b/python/setup.cfg
index 13a7a8cd9c..04ad3d91bc 100755
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -54,3 +54,11 @@ good-names=i,
 [pylint.similarities]
 # Ignore imports when computing similarities.
 ignore-imports=yes
+
+[isort]
+force_grid_wrap=4
+force_sort_within_sections=True
+include_trailing_comma=True
+line_length=72
+lines_after_imports=2
+multi_line_output=3
-- 
2.26.2

[PATCH v2 14/15] python/qemu: add isort to pipenv

[John Snow]

isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
certain "from ..." clauses that are not related to imports.

Require 5.0.5 or greater.

isort can be run with 'isort -c qemu' from the python root.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile      | 1 +
 python/Pipfile.lock | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/python/Pipfile b/python/Pipfile
index 51c537b0d1..75b96f29d8 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -5,6 +5,7 @@ verify_ssl = true
 
 [dev-packages]
 flake8 = ">=3.6.0"
+isort = ">=5.0.5"
 mypy = ">=0.770"
 pylint = ">=2.6.0"
 
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 376f95a6a8..74563b444c 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "65f010a8f2e55e9870d2b7e0d8af129516097d23abf2504f396552748b067ade"
+            "sha256": "b89c7a1b8a414f2a4cd708964123fb427d55419ee0b39e088bf2e7d4fbc11979"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -46,7 +46,7 @@
                 "sha256:dcab1d98b469a12a1a624ead220584391648790275560e1a43e54c5dceae65e7",
                 "sha256:dcaeec1b5f0eca77faea2a35ab790b4f3680ff75590bfcb7145986905aab2f58"
             ],
-            "markers": "python_version >= '3.6' and python_version < '4.0'",
+            "index": "pypi",
             "version": "==5.6.4"
         },
         "lazy-object-proxy": {
-- 
2.26.2

[PATCH v2 15/15] python/qemu: add qemu package itself to pipenv

[John Snow]

This adds the python qemu packages themselves to the pipenv manifest.
'pipenv sync' will create a virtual environment sufficient to use the SDK.
'pipenv sync --dev' will create a virtual environment sufficient to use
and test the SDK (with pylint, mypy, isort, flake8, etc.)

The qemu packages are installed in 'editable' mode; all changes made to
the python package inside the git tree will be reflected in the
installed package without reinstallation. This includes changes made
via git pull and so on.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Pipfile      | 1 +
 python/Pipfile.lock | 9 +++++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/python/Pipfile b/python/Pipfile
index 75b96f29d8..214fb175e7 100644
--- a/python/Pipfile
+++ b/python/Pipfile
@@ -10,6 +10,7 @@ mypy = ">=0.770"
 pylint = ">=2.6.0"
 
 [packages]
+qemu = {editable = true,path = "."}
 
 [requires]
 python_version = "3.6"
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
index 74563b444c..386ca54ea6 100644
--- a/python/Pipfile.lock
+++ b/python/Pipfile.lock
@@ -1,7 +1,7 @@
 {
     "_meta": {
         "hash": {
-            "sha256": "b89c7a1b8a414f2a4cd708964123fb427d55419ee0b39e088bf2e7d4fbc11979"
+            "sha256": "e38d142c3fadc2f2ed849e86f7ebd14e25974dc12228751490aef5a9ee074f2f"
         },
         "pipfile-spec": 6,
         "requires": {
@@ -15,7 +15,12 @@
             }
         ]
     },
-    "default": {},
+    "default": {
+        "qemu": {
+            "editable": true,
+            "path": "."
+        }
+    },
     "develop": {
         "astroid": {
             "hashes": [
-- 
2.26.2

Re: [PATCH v2 03/15] python: add VERSION file

[John Snow]

On 10/14/20 10:29 AM, John Snow wrote:
> Python infrastructure as it exists today is not capable reliably of
> single-sourcing a package version from a parent directory. The authors
> of pip are working to correct this, but as of today this is not possible
> to my knowledge.
> 
> The problem is that when using pip to build and install a python
> package, it copies files over to a temporary directory and performs its
> build there. This loses access to any information in the parent
> directory, including git itself.
> 

See also:

https://github.com/pypa/pip/issues/7549
https://github.com/pypa/pip/issues/7555

> Further, Python versions have a standard (PEP 440) that may or may not
> follow QEMU's versioning. In general, it does; but naturally QEMU does
> not follow PEP 440. To avoid any automatically-generated conflict, a
> manual version file is preferred.
> 
> 
> I am proposing:
> 
> - Python core tooling synchronizes with the QEMU version directly
>    (5.2.0, 5.1.1, 5.3.0, etc.)
> 
> - In the event that a Python package needs to be updated independently
>    of the QEMU version, a pre-release alpha version should be preferred,
>    but *only* after inclusion to the qemu development or stable branches.
> 
>    e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's
>    release.
> 
> - The Python core tooling makes absolutely no version compatibility
>    checks or constraints. It *may* work with releases of QEMU from the
>    past or future, but it is not required to.
> 
>    i.e., "qemu.core" will always remain in lock-step with QEMU.
> 
> - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp
>    and begin indepedently versioning such a package separately from the
>    QEMU version it accompanies.
> 
> 
> Implement this versioning scheme by adding a VERSION file and setting it
> to 5.2.0a1.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   python/VERSION   | 1 +
>   python/setup.cfg | 1 +
>   2 files changed, 2 insertions(+)
>   create mode 100644 python/VERSION
> 
> diff --git a/python/VERSION b/python/VERSION
> new file mode 100644
> index 0000000000..2e81039c82
> --- /dev/null
> +++ b/python/VERSION
> @@ -0,0 +1 @@
> +5.2.0a1
> diff --git a/python/setup.cfg b/python/setup.cfg
> index 12b99a796e..260f7f4e94 100755
> --- a/python/setup.cfg
> +++ b/python/setup.cfg
> @@ -1,5 +1,6 @@
>   [metadata]
>   name = qemu
> +version = file:VERSION
>   maintainer = QEMU Developer Team
>   maintainer_email = qemu-devel@nongnu.org
>   url = https://www.qemu.org/
> 

Re: [PATCH v2 01/15] python: create qemu.core package

[Philippe Mathieu-Daudé]

On 10/14/20 4:29 PM, John Snow wrote:
> move python/qemu/*.py to python/qemu/core/*.py and update import
> directives across the tree.
> 
> This is done to create a PEP420 namespace package, in which we may
> create subpackages. To do this, the namespace directory ("qemu") should
> not have any modules in it. Those files will go in a new 'core'
> subpackage instead.
> 
> Bolster the core/__init__.py module, making the top-level classes and
> functions from this package available directly inside the `qemu.core`
> namespace.
> 
> This facilitates the convenient shorthand:
> 
>> from qemu.core import QEMUQtestMachine, QEMUMonitorProtocol
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   python/{qemu => }/.isort.cfg              |  0
>   python/qemu/__init__.py                   | 11 ------
>   python/qemu/{ => core}/.flake8            |  0
>   python/qemu/core/__init__.py              | 44 +++++++++++++++++++++++
>   python/qemu/{ => core}/accel.py           |  0
>   python/qemu/{ => core}/console_socket.py  |  0
>   python/qemu/{ => core}/machine.py         |  0
>   python/qemu/{ => core}/pylintrc           |  0
>   python/qemu/{ => core}/qmp.py             |  0
>   python/qemu/{ => core}/qtest.py           |  0
>   scripts/device-crash-test                 |  2 +-
>   scripts/qmp/qemu-ga-client                |  2 +-
>   scripts/qmp/qmp                           |  2 +-
>   scripts/qmp/qmp-shell                     |  2 +-
>   scripts/qmp/qom-fuse                      |  2 +-
>   scripts/qmp/qom-get                       |  2 +-
>   scripts/qmp/qom-list                      |  2 +-
>   scripts/qmp/qom-set                       |  2 +-
>   scripts/qmp/qom-tree                      |  2 +-
>   scripts/render_block_graph.py             |  6 ++--
>   scripts/simplebench/bench_block_job.py    |  4 +--
>   tests/acceptance/avocado_qemu/__init__.py |  2 +-
>   tests/acceptance/boot_linux.py            |  3 +-
>   tests/acceptance/virtio_check_params.py   |  2 +-
>   tests/acceptance/virtio_version.py        |  2 +-
>   tests/migration/guestperf/engine.py       |  2 +-
>   tests/qemu-iotests/235                    |  2 +-
>   tests/qemu-iotests/297                    |  2 +-
>   tests/qemu-iotests/300                    |  4 +--
>   tests/qemu-iotests/iotests.py             |  4 +--
>   tests/vm/basevm.py                        |  6 ++--
>   31 files changed, 71 insertions(+), 41 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH v2 04/15] python: add directory structure README.rst files

[Philippe Mathieu-Daudé]

On 10/14/20 4:29 PM, John Snow wrote:
> Add short readmes to python/, python/qemu/, and python/qemu/core that
> explain the directory hierarchy. These readmes are visible when browsing

Maybe readmes -> READMEs

Otherwise:
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> the source on e.g. gitlab/github and are designed to help new
> developers/users quickly make sense of the source tree.
> 
> They are not designed for inclusion in a published manual.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   python/README.rst           | 27 +++++++++++++++++++++++++++
>   python/qemu/README.rst      |  8 ++++++++
>   python/qemu/core/README.rst |  9 +++++++++
>   3 files changed, 44 insertions(+)
>   create mode 100644 python/README.rst
>   create mode 100644 python/qemu/README.rst
>   create mode 100644 python/qemu/core/README.rst

Re: [PATCH v2 13/15] python: move .isort.cfg into setup.cfg

[Philippe Mathieu-Daudé]

On 10/14/20 4:29 PM, John Snow wrote:
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   python/.isort.cfg | 7 -------
>   python/setup.cfg  | 8 ++++++++
>   2 files changed, 8 insertions(+), 7 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH v2 04/15] python: add directory structure README.rst files

[John Snow]

On 10/14/20 2:05 PM, Philippe Mathieu-Daudé wrote:
> On 10/14/20 4:29 PM, John Snow wrote:
>> Add short readmes to python/, python/qemu/, and python/qemu/core that
>> explain the directory hierarchy. These readmes are visible when browsing
> 
> Maybe readmes -> READMEs
> 

If you want it to match the filename, "README files".

--js

> Otherwise:
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> 
>> the source on e.g. gitlab/github and are designed to help new
>> developers/users quickly make sense of the source tree.
>>
>> They are not designed for inclusion in a published manual.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/README.rst           | 27 +++++++++++++++++++++++++++
>>   python/qemu/README.rst      |  8 ++++++++
>>   python/qemu/core/README.rst |  9 +++++++++
>>   3 files changed, 44 insertions(+)
>>   create mode 100644 python/README.rst
>>   create mode 100644 python/qemu/README.rst
>>   create mode 100644 python/qemu/core/README.rst
> 

Re: [PATCH v2 03/15] python: add VERSION file

[Andrea Bolognani]

On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote:
> Python infrastructure as it exists today is not capable reliably of
> single-sourcing a package version from a parent directory. The authors
> of pip are working to correct this, but as of today this is not possible
> to my knowledge.
> 
> The problem is that when using pip to build and install a python
> package, it copies files over to a temporary directory and performs its
> build there. This loses access to any information in the parent
> directory, including git itself.
> 
> Further, Python versions have a standard (PEP 440) that may or may not
> follow QEMU's versioning. In general, it does; but naturally QEMU does
> not follow PEP 440. To avoid any automatically-generated conflict, a
> manual version file is preferred.
> 
> 
> I am proposing:
> 
> - Python core tooling synchronizes with the QEMU version directly
>   (5.2.0, 5.1.1, 5.3.0, etc.)
> 
> - In the event that a Python package needs to be updated independently
>   of the QEMU version, a pre-release alpha version should be preferred,
>   but *only* after inclusion to the qemu development or stable branches.
> 
>   e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's
>   release.
> 
> - The Python core tooling makes absolutely no version compatibility
>   checks or constraints. It *may* work with releases of QEMU from the
>   past or future, but it is not required to.
> 
>   i.e., "qemu.core" will always remain in lock-step with QEMU.
> 
> - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp
>   and begin indepedently versioning such a package separately from the
>   QEMU version it accompanies.

I think this need to be considered very carefully.

I'm not overly familiar with the Python ecosystem but it would appear
that, despite PEP 440 not mandating this, many (most?) of the
packages uploaded to PyPi are using semantic versioning.

With that in mind, I think it would be unwise for qemu.* not to do
the same; in particular, using a version number that's not <1.0.0 for
a package that is very much in flux will almost certainly break
people's expectations, and is also not something that you can easily
take back at a later time.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [PATCH v2 03/15] python: add VERSION file

[Daniel P. Berrangé]

On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote:
> On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote:
> > Python infrastructure as it exists today is not capable reliably of
> > single-sourcing a package version from a parent directory. The authors
> > of pip are working to correct this, but as of today this is not possible
> > to my knowledge.
> > 
> > The problem is that when using pip to build and install a python
> > package, it copies files over to a temporary directory and performs its
> > build there. This loses access to any information in the parent
> > directory, including git itself.
> > 
> > Further, Python versions have a standard (PEP 440) that may or may not
> > follow QEMU's versioning. In general, it does; but naturally QEMU does
> > not follow PEP 440. To avoid any automatically-generated conflict, a
> > manual version file is preferred.
> > 
> > 
> > I am proposing:
> > 
> > - Python core tooling synchronizes with the QEMU version directly
> >   (5.2.0, 5.1.1, 5.3.0, etc.)
> > 
> > - In the event that a Python package needs to be updated independently
> >   of the QEMU version, a pre-release alpha version should be preferred,
> >   but *only* after inclusion to the qemu development or stable branches.
> > 
> >   e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's
> >   release.
> > 
> > - The Python core tooling makes absolutely no version compatibility
> >   checks or constraints. It *may* work with releases of QEMU from the
> >   past or future, but it is not required to.
> > 
> >   i.e., "qemu.core" will always remain in lock-step with QEMU.
> > 
> > - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp
> >   and begin indepedently versioning such a package separately from the
> >   QEMU version it accompanies.
> 
> I think this need to be considered very carefully.
> 
> I'm not overly familiar with the Python ecosystem but it would appear
> that, despite PEP 440 not mandating this, many (most?) of the
> packages uploaded to PyPi are using semantic versioning.

  https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme

Semver is the recommended approach, but they explicitly list date
based versioning as a valid alternative

  "Semantic versioning is not a suitable choice for all projects, 
   such as those with a regular time based release cadence and a 
   deprecation process that provides warnings for a number of 
   releases prior to removal of a feature."

That paragraph describes QEMU's scenario.

NB, historically we've made arbitrary changes to the python code
since it was not considered public API. If we make it official
public API, then we would actually need to start following our
deprecation process for the python code too.

> With that in mind, I think it would be unwise for qemu.* not to do
> the same; in particular, using a version number that's not <1.0.0 for
> a package that is very much in flux will almost certainly break
> people's expectations, and is also not something that you can easily
> take back at a later time.

I don't think it is that big a deal, and there is clear benefit to
having the python code version match the QEMU version that it is
the companioon to.

Ultimately the versioning scheme just impacts on the version string
conditionals people list for their dependancies. Apps consuming QEMU
can handle any of the version schemes without much difference.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 03/15] python: add VERSION file

[John Snow]

On 10/19/20 6:02 AM, Daniel P. Berrangé wrote:
> On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote:
>> On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote:
>>> Python infrastructure as it exists today is not capable reliably of
>>> single-sourcing a package version from a parent directory. The authors
>>> of pip are working to correct this, but as of today this is not possible
>>> to my knowledge.
>>>
>>> The problem is that when using pip to build and install a python
>>> package, it copies files over to a temporary directory and performs its
>>> build there. This loses access to any information in the parent
>>> directory, including git itself.
>>>
>>> Further, Python versions have a standard (PEP 440) that may or may not
>>> follow QEMU's versioning. In general, it does; but naturally QEMU does
>>> not follow PEP 440. To avoid any automatically-generated conflict, a
>>> manual version file is preferred.
>>>
>>>
>>> I am proposing:
>>>
>>> - Python core tooling synchronizes with the QEMU version directly
>>>    (5.2.0, 5.1.1, 5.3.0, etc.)
>>>
>>> - In the event that a Python package needs to be updated independently
>>>    of the QEMU version, a pre-release alpha version should be preferred,
>>>    but *only* after inclusion to the qemu development or stable branches.
>>>
>>>    e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's
>>>    release.
>>>
>>> - The Python core tooling makes absolutely no version compatibility
>>>    checks or constraints. It *may* work with releases of QEMU from the
>>>    past or future, but it is not required to.
>>>
>>>    i.e., "qemu.core" will always remain in lock-step with QEMU.
>>>
>>> - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp
>>>    and begin indepedently versioning such a package separately from the
>>>    QEMU version it accompanies.
>>
>> I think this need to be considered very carefully.
>>
>> I'm not overly familiar with the Python ecosystem but it would appear
>> that, despite PEP 440 not mandating this, many (most?) of the
>> packages uploaded to PyPi are using semantic versioning.
> 
>    https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme
> 
> Semver is the recommended approach, but they explicitly list date
> based versioning as a valid alternative
> 
>    "Semantic versioning is not a suitable choice for all projects,
>     such as those with a regular time based release cadence and a
>     deprecation process that provides warnings for a number of
>     releases prior to removal of a feature."
> 
> That paragraph describes QEMU's scenario.
> 
> NB, historically we've made arbitrary changes to the python code
> since it was not considered public API. If we make it official
> public API, then we would actually need to start following our
> deprecation process for the python code too.
> 

I think our deprecation process is not tightly compatible with how 
Python programmers at-large expect packages to work. Semver is more or 
less the norm, despite the fact that it isn't explicitly required.

setting requirements in requirements.txt, setup.[cfg|py], Pipfile, etc 
often hinge on a major version, e.g.

qemu >= 5.0
qemu >= 3.0, < 6.0

would both be common forms of describing a requirement.

Pinning specific versions is considered bad form, but in the context of 
releasing a package, I often see maintainers hedging their bets and 
preventing upgrades across a major version line.

For that reason I am a little weary of adopting the deprecation policy 
as it exists in QEMU directly, and would propose a modification for my 
purposes here:

- Features must be marked as deprecated
- They must remain in a deprecated state for [at least] 2 releases
- Deprecated features may not be removed until a major version change.

In practice, this modification is a change from "2 releases" to "at 
least 2".

However, I didn't intend to pay any mind to the deprecation policy 
"yet", as I have the package metadata listing the package status as 
"Alpha", see below:

>> With that in mind, I think it would be unwise for qemu.* not to do
>> the same; in particular, using a version number that's not <1.0.0 for
>> a package that is very much in flux will almost certainly break
>> people's expectations, and is also not something that you can easily
>> take back at a later time.
> 
> I don't think it is that big a deal, and there is clear benefit to
> having the python code version match the QEMU version that it is
> the companioon to.
> 

Do you think it's fine if I start versioning at, say, "0.5.2", I could 
ignore a deprecation policy for now? I have a lot of changes I want to 
make and expect a lot of breaking changes very quickly.

I just wanted to try -- somehow -- to conjure up a relationship to the 
QEMU package it's designed to work with.

> Ultimately the versioning scheme just impacts on the version string
> conditionals people list for their dependancies. Apps consuming QEMU
> can handle any of the version schemes without much difference.
> 
> Regards,
> Daniel
> 

Thanks for your input. This is the trickiest part of the process for me.

I believe there is value in distributing these tools for other 
developers to help them prototype and experiment with new features, but 
realize it's a tightrope walk because we're flying dangerously close to 
providing a management utility that needs to care about cross-version 
compatibility and so on.

I have no interest in providing stringent cross-version compatibility, 
but at the same time, libraries like QMP are not really at risk of 
changing all that much, actually. It will *probably* work for most QEMU 
versions.

I have some further patches where I clean up the scripts in 
./scripts/qmp and move them to ./python/qemu/qmp -- and that work is 
making me consider a rework to this series. I think that rework matches 
the spirit of an earlier suggestion of yours:

- move ./python/qemu/core/qmp.py to ./python/qemu/qmp/qmp.py
- (in a follow-up series) move ./scripts/qmp/* to ./python/qemu/qmp/*
- rename ./python/qemu/core/ to ./python/qemu/machine
     (move accel.py, machine.py and qtest.py to a 'machine' pkg.)


In effect, we'd then have:

qemu.machine -- primarily a test interface for QEMU instances
qemu.qmp -- fairly ageless QMP tools/lib for talking to QEMU

and the different levels of support and "use at your own risk" become 
slightly more clear between the two packages.

--js

Re: [PATCH v2 03/15] python: add VERSION file

[Andrea Bolognani]

On Mon, 2020-10-19 at 11:02 +0100, Daniel P. Berrangé wrote:
> On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote:
> > I think this need to be considered very carefully.
> > 
> > I'm not overly familiar with the Python ecosystem but it would appear
> > that, despite PEP 440 not mandating this, many (most?) of the
> > packages uploaded to PyPi are using semantic versioning.
> 
>   https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme
> 
> Semver is the recommended approach, but they explicitly list date
> based versioning as a valid alternative
> 
>   "Semantic versioning is not a suitable choice for all projects, 
>    such as those with a regular time based release cadence and a 
>    deprecation process that provides warnings for a number of 
>    releases prior to removal of a feature."
> 
> That paragraph describes QEMU's scenario.

The section on date based versioning continues with

  A key advantage of date based versioning is that it is
  straightforward to tell how old the base feature set of a
  particular release is given just the version number.

  Version numbers for date based projects typically take the form of
  YEAR.MONTH (for example, 12.04, 15.10).

The problem with QEMU's version numbers is that, while they are date
based, they still *look* like semver, so it wouldn't be at all
unreasonable for the user to expect that they also *behave* like
semver.

This is not much of a problem when it comes to the main binary, but
it is certainly much more confusing when you start using the same
version number for a Python library.

> > With that in mind, I think it would be unwise for qemu.* not to do
> > the same; in particular, using a version number that's not <1.0.0 for
> > a package that is very much in flux will almost certainly break
> > people's expectations, and is also not something that you can easily
> > take back at a later time.
> 
> I don't think it is that big a deal, and there is clear benefit to
> having the python code version match the QEMU version that it is
> the companioon to.
> 
> Ultimately the versioning scheme just impacts on the version string
> conditionals people list for their dependancies. Apps consuming QEMU
> can handle any of the version schemes without much difference.

The problem comes from the expectations: a Python programmer, who is
used to semver due to its prominence on PyPi, when deciding whether
to move from qemu.core 4.2.0 to 5.0.0 might expect to need code
changes to cope with API-breaking changes - where in fact there are
none, and at the same time might expect upgrading to 5.2.0 from 5.0.0
to be completely straightforward when in reality a feature their
application depends on might have been removed after the usual
deprecation period.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [PATCH v2 03/15] python: add VERSION file

[Daniel P. Berrangé]

On Tue, Oct 20, 2020 at 10:52:14AM +0200, Andrea Bolognani wrote:
> On Mon, 2020-10-19 at 11:02 +0100, Daniel P. Berrangé wrote:
> > On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote:
> > > With that in mind, I think it would be unwise for qemu.* not to do
> > > the same; in particular, using a version number that's not <1.0.0 for
> > > a package that is very much in flux will almost certainly break
> > > people's expectations, and is also not something that you can easily
> > > take back at a later time.
> > 
> > I don't think it is that big a deal, and there is clear benefit to
> > having the python code version match the QEMU version that it is
> > the companioon to.
> > 
> > Ultimately the versioning scheme just impacts on the version string
> > conditionals people list for their dependancies. Apps consuming QEMU
> > can handle any of the version schemes without much difference.
> 
> The problem comes from the expectations: a Python programmer, who is
> used to semver due to its prominence on PyPi, when deciding whether
> to move from qemu.core 4.2.0 to 5.0.0 might expect to need code
> changes to cope with API-breaking changes - where in fact there are
> none, and at the same time might expect upgrading to 5.2.0 from 5.0.0
> to be completely straightforward when in reality a feature their
> application depends on might have been removed after the usual
> deprecation period.

The QEMU python modules are not like other python modules though,
precisely because they are talking to QEMU. If we are shipping
QEMU python releases on the same schedule as QEMU, then we can
expect the normal ase to be updating both QEMU & Python together.
So regardless of versioning in the python code, the QMP code they
are talking to is liable to have removed deprecated features they
are using.  IMHO the upgrade issue is largely a problem of docs
and testing, semver is no magic bullet.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 03/15] python: add VERSION file

[Andrea Bolognani]

On Tue, 2020-10-20 at 10:06 +0100, Daniel P. Berrangé wrote:
> The QEMU python modules are not like other python modules though,
> precisely because they are talking to QEMU. If we are shipping
> QEMU python releases on the same schedule as QEMU, then we can
> expect the normal ase to be updating both QEMU & Python together.

Once you start uploading the Python packages to PyPi, you really have
no way to ensure this will be the case.

-- 
Andrea Bolognani / Red Hat / Virtualization