[PATCH 0/7] Python: Drop support for Python 3.6

[PATCH 0/7] Python: Drop support for Python 3.6

[John Snow]

Howdy, this series increases our minimum python version to 3.7.

CI: https://gitlab.com/jsnow/qemu/-/pipelines/771780626
    (All green!)
GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37

Patches 1 and 2 are loose pre-requisites; I'd like to merge them into
qemu.git within the week whether or not we take this series. I'd
appreciate an "ACK" on those specifically. They're just riding along
here because they make this series a bit nicer.

Patches 3-6 are the hard pre-requisites, and 7 does the dirty work.

The motivation for this series is that Python 3.6 was EOL at the end of
2021; upstream tools are beginning to drop support for it, including
setuptools, pylint, mypy, etc. As time goes by, it becomes more
difficult to support and test against the full range of Python versions
that QEMU supports. The closer we get to Python 3.12, the harder it will
be to cover that full spread of versions.

The qemu.qmp library and the avocado testing framework both have
motivations for dropping 3.6 support, but are committed to not doing so
until QEMU drops support.

So, I'd like to talk about doing it.

RFC:
 - Patch 5 is just a proof-of-concept; it's not admissable as-is and I
   am aware of that.
 - Cleber, I need to update your ansible scripts. How do I test them?
 - Are we testing on CentOS 9 Stream yet? If so, we can move doc
   building from CentOS 8 to CentOS 9 to maintain that coverage.
   (See patch 6 for details.)

Thanks!
--js

John Snow (7):
  python: support pylint 2.16
  Python: drop pipenv
  configure: Look for auxiliary Python installations
  configure: Add nice hint to Python failure message
  testing: Add Python >= 3.7 to Centos, OpenSuSE
  CI: Stop building docs on centos8
  Python: Drop support for Python 3.6

 docs/conf.py                                  |   4 +-
 python/README.rst                             |   3 -
 configure                                     |  40 +-
 .gitlab-ci.d/buildtest.yml                    |   2 +-
 .gitlab-ci.d/static_checks.yml                |   4 +-
 python/.gitignore                             |   4 +-
 python/Makefile                               |  57 ++-
 python/Pipfile                                |  13 -
 python/Pipfile.lock                           | 347 ------------------
 python/qemu/qmp/protocol.py                   |   2 +-
 python/qemu/qmp/qmp_client.py                 |   2 +-
 python/qemu/utils/qemu_ga_client.py           |   6 +-
 python/setup.cfg                              |  11 +-
 python/tests/minreqs.txt                      |  45 +++
 scripts/qapi/mypy.ini                         |   2 +-
 tests/docker/dockerfiles/centos8.docker       |   1 +
 tests/docker/dockerfiles/opensuse-leap.docker |   1 +
 tests/docker/dockerfiles/python.docker        |   1 -
 tests/qemu-iotests/iotests.py                 |   4 +-
 .../tests/migrate-bitmaps-postcopy-test       |   2 +-
 20 files changed, 135 insertions(+), 416 deletions(-)
 delete mode 100644 python/Pipfile
 delete mode 100644 python/Pipfile.lock
 create mode 100644 python/tests/minreqs.txt

-- 
2.39.0

[PATCH 1/7] python: support pylint 2.16

[John Snow]

Pylint 2.16 adds a few new checks that cause the optional check-tox CI
job to fail.

1. The superfluous-parens check seems to be a bit more aggressive,
2. broad-exception-raised is new; it discourages "raise Exception".

Fix these minor issues and turn the lights green.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/qmp/protocol.py                            | 2 +-
 python/qemu/qmp/qmp_client.py                          | 2 +-
 python/qemu/utils/qemu_ga_client.py                    | 6 +++---
 tests/qemu-iotests/iotests.py                          | 4 ++--
 tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 2 +-
 5 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/python/qemu/qmp/protocol.py b/python/qemu/qmp/protocol.py
index 6d3d739daa7..22e60298d28 100644
--- a/python/qemu/qmp/protocol.py
+++ b/python/qemu/qmp/protocol.py
@@ -207,7 +207,7 @@ class AsyncProtocol(Generic[T]):
     logger = logging.getLogger(__name__)
 
     # Maximum allowable size of read buffer
-    _limit = (64 * 1024)
+    _limit = 64 * 1024
 
     # -------------------------
     # Section: Public interface
diff --git a/python/qemu/qmp/qmp_client.py b/python/qemu/qmp/qmp_client.py
index b5772e7f32b..9d73ae6e7ad 100644
--- a/python/qemu/qmp/qmp_client.py
+++ b/python/qemu/qmp/qmp_client.py
@@ -198,7 +198,7 @@ async def run(self, address='/tmp/qemu.socket'):
     logger = logging.getLogger(__name__)
 
     # Read buffer limit; 10MB like libvirt default
-    _limit = (10 * 1024 * 1024)
+    _limit = 10 * 1024 * 1024
 
     # Type alias for pending execute() result items
     _PendingT = Union[Message, ExecInterruptedError]
diff --git a/python/qemu/utils/qemu_ga_client.py b/python/qemu/utils/qemu_ga_client.py
index 8c38a7ac9c0..d8411bb2d0b 100644
--- a/python/qemu/utils/qemu_ga_client.py
+++ b/python/qemu/utils/qemu_ga_client.py
@@ -155,7 +155,7 @@ def ping(self, timeout: Optional[float]) -> bool:
 
     def fsfreeze(self, cmd: str) -> object:
         if cmd not in ['status', 'freeze', 'thaw']:
-            raise Exception('Invalid command: ' + cmd)
+            raise ValueError('Invalid command: ' + cmd)
         # Can be int (freeze, thaw) or GuestFsfreezeStatus (status)
         return getattr(self.qga, 'fsfreeze' + '_' + cmd)()
 
@@ -167,7 +167,7 @@ def fstrim(self, minimum: int) -> Dict[str, object]:
 
     def suspend(self, mode: str) -> None:
         if mode not in ['disk', 'ram', 'hybrid']:
-            raise Exception('Invalid mode: ' + mode)
+            raise ValueError('Invalid mode: ' + mode)
 
         try:
             getattr(self.qga, 'suspend' + '_' + mode)()
@@ -178,7 +178,7 @@ def suspend(self, mode: str) -> None:
 
     def shutdown(self, mode: str = 'powerdown') -> None:
         if mode not in ['powerdown', 'halt', 'reboot']:
-            raise Exception('Invalid mode: ' + mode)
+            raise ValueError('Invalid mode: ' + mode)
 
         try:
             self.qga.shutdown(mode=mode)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 94aeb3f3b20..3e82c634cfe 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -720,7 +720,7 @@ def __exit__(self, exc_type, value, traceback):
         signal.setitimer(signal.ITIMER_REAL, 0)
         return False
     def timeout(self, signum, frame):
-        raise Exception(self.errmsg)
+        raise TimeoutError(self.errmsg)
 
 def file_pattern(name):
     return "{0}-{1}".format(os.getpid(), name)
@@ -804,7 +804,7 @@ def remote_filename(path):
     elif imgproto == 'ssh':
         return "ssh://%s@127.0.0.1:22%s" % (os.environ.get('USER'), path)
     else:
-        raise Exception("Protocol %s not supported" % (imgproto))
+        raise ValueError("Protocol %s not supported" % (imgproto))
 
 class VM(qtest.QEMUQtestMachine):
     '''A QEMU VM'''
diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
index fc9c4b4ef41..dda55fad284 100755
--- a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
+++ b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
@@ -84,7 +84,7 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
                 e['vm'] = 'SRC'
             for e in self.vm_b_events:
                 e['vm'] = 'DST'
-            events = (self.vm_a_events + self.vm_b_events)
+            events = self.vm_a_events + self.vm_b_events
             events = [(e['timestamp']['seconds'],
                        e['timestamp']['microseconds'],
                        e['vm'],
-- 
2.39.0

[PATCH 2/7] Python: drop pipenv

[John Snow]

The pipenv tool was nice in theory, but in practice it's just too hard
to update selectively, and it makes using it a pain. The qemu.qmp repo
dropped pipenv support a while back and it's been functioning just fine,
so I'm backporting that change here to qemu.git.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst                      |   3 -
 .gitlab-ci.d/static_checks.yml         |   4 +-
 python/.gitignore                      |   4 +-
 python/Makefile                        |  53 ++--
 python/Pipfile                         |  13 -
 python/Pipfile.lock                    | 347 -------------------------
 python/setup.cfg                       |   4 +-
 python/tests/minreqs.txt               |  45 ++++
 tests/docker/dockerfiles/python.docker |   1 -
 9 files changed, 86 insertions(+), 388 deletions(-)
 delete mode 100644 python/Pipfile
 delete mode 100644 python/Pipfile.lock
 create mode 100644 python/tests/minreqs.txt

diff --git a/python/README.rst b/python/README.rst
index 9c1fceaee73..d62e71528d2 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -77,9 +77,6 @@ Files in this directory
 - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
   that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
-- ``Pipfile`` is used by Pipenv to generate ``Pipfile.lock``.
-- ``Pipfile.lock`` is a set of pinned package dependencies that this package
-  is tested under in our CI suite. It is used by ``make check-pipenv``.
 - ``README.rst`` you are here!
 - ``VERSION`` contains the PEP-440 compliant version used to describe
   this package; it is referenced by ``setup.cfg``.
diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
index 289ad1359e3..b4cbdbce2ab 100644
--- a/.gitlab-ci.d/static_checks.yml
+++ b/.gitlab-ci.d/static_checks.yml
@@ -23,12 +23,12 @@ check-dco:
   before_script:
     - apk -U add git
 
-check-python-pipenv:
+check-python-minreqs:
   extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
-    - make -C python check-pipenv
+    - make -C python check-minreqs
   variables:
     GIT_DEPTH: 1
   needs:
diff --git a/python/.gitignore b/python/.gitignore
index 904f324bb11..c3ceb1ca0ab 100644
--- a/python/.gitignore
+++ b/python/.gitignore
@@ -11,8 +11,8 @@ qemu.egg-info/
 .idea/
 .vscode/
 
-# virtual environments (pipenv et al)
-.venv/
+# virtual environments
+.min-venv/
 .tox/
 .dev-venv/
 
diff --git a/python/Makefile b/python/Makefile
index b170708398a..c5bd6ff83ac 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -1,15 +1,16 @@
 QEMU_VENV_DIR=.dev-venv
+QEMU_MINVENV_DIR=.min-venv
 QEMU_TOX_EXTRA_ARGS ?=
 
 .PHONY: help
 help:
 	@echo "python packaging help:"
 	@echo ""
-	@echo "make check-pipenv:"
-	@echo "    Run tests in pipenv's virtual environment."
+	@echo "make check-minreqs:"
+	@echo "    Run tests in the minreqs virtual environment."
 	@echo "    These tests use the oldest dependencies."
-	@echo "    Requires: Python 3.6 and pipenv."
-	@echo "    Hint (Fedora): 'sudo dnf install python3.6 pipenv'"
+	@echo "    Requires: Python 3.6"
+	@echo "    Hint (Fedora): 'sudo dnf install python3.6'"
 	@echo ""
 	@echo "make check-tox:"
 	@echo "    Run tests against multiple python versions."
@@ -33,8 +34,8 @@ help:
 	@echo "    and install the qemu package in editable mode."
 	@echo "    (Can be used in or outside of a venv.)"
 	@echo ""
-	@echo "make pipenv"
-	@echo "    Creates pipenv's virtual environment (.venv)"
+	@echo "make min-venv"
+	@echo "    Creates the minreqs virtual environment ($(QEMU_MINVENV_DIR))"
 	@echo ""
 	@echo "make dev-venv"
 	@echo "    Creates a simple venv for check-dev. ($(QEMU_VENV_DIR))"
@@ -43,21 +44,38 @@ help:
 	@echo "    Remove package build output."
 	@echo ""
 	@echo "make distclean:"
-	@echo "    remove pipenv/venv files, qemu package forwarder,"
+	@echo "    remove venv files, qemu package forwarder,"
 	@echo "    built distribution files, and everything from 'make clean'."
 	@echo ""
 	@echo -e "Have a nice day ^_^\n"
 
-.PHONY: pipenv
-pipenv: .venv
-.venv: Pipfile.lock
-	@PIPENV_VENV_IN_PROJECT=1 pipenv sync --dev --keep-outdated
-	rm -f pyproject.toml
-	@touch .venv
+.PHONY: pipenv check-pipenv
+pipenv check-pipenv:
+	@echo "pipenv was dropped; try 'make check-minreqs' or 'make min-venv'"
+	@exit 1
 
-.PHONY: check-pipenv
-check-pipenv: pipenv
-	@pipenv run make check
+.PHONY: min-venv
+min-venv: $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate
+$(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate: setup.cfg tests/minreqs.txt
+	@echo "VENV $(QEMU_MINVENV_DIR)"
+	@python3.6 -m venv $(QEMU_MINVENV_DIR)
+	@(								\
+		echo "ACTIVATE $(QEMU_MINVENV_DIR)";			\
+		. $(QEMU_MINVENV_DIR)/bin/activate;			\
+		echo "INSTALL -r tests/minreqs.txt $(QEMU_MINVENV_DIR)";\
+		pip install -r tests/minreqs.txt 1>/dev/null;		\
+		echo "INSTALL -e qemu $(QEMU_MINVENV_DIR)";		\
+		pip install -e . 1>/dev/null;				\
+	)
+	@touch $(QEMU_MINVENV_DIR)
+
+.PHONY: check-minreqs
+check-minreqs: min-venv
+	@(							\
+		echo "ACTIVATE $(QEMU_MINVENV_DIR)";		\
+		. $(QEMU_MINVENV_DIR)/bin/activate;		\
+		make check;					\
+	)
 
 .PHONY: dev-venv
 dev-venv: $(QEMU_VENV_DIR) $(QEMU_VENV_DIR)/bin/activate
@@ -106,6 +124,7 @@ clean:
 
 .PHONY: distclean
 distclean: clean
-	rm -rf qemu.egg-info/ .venv/ .tox/ $(QEMU_VENV_DIR) dist/
+	rm -rf qemu.egg-info/ .eggs/ dist/
+	rm -rf $(QEMU_VENV_DIR) $(QEMU_MINVENV_DIR) .tox/
 	rm -f .coverage .coverage.*
 	rm -rf htmlcov/
diff --git a/python/Pipfile b/python/Pipfile
deleted file mode 100644
index e7acb8cefa4..00000000000
--- a/python/Pipfile
+++ /dev/null
@@ -1,13 +0,0 @@
-[[source]]
-name = "pypi"
-url = "https://pypi.org/simple"
-verify_ssl = true
-
-[dev-packages]
-qemu = {editable = true, extras = ["devel"], path = "."}
-
-[packages]
-qemu = {editable = true,path = "."}
-
-[requires]
-python_version = "3.6"
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
deleted file mode 100644
index ce46404ce08..00000000000
--- a/python/Pipfile.lock
+++ /dev/null
@@ -1,347 +0,0 @@
-{
-    "_meta": {
-        "hash": {
-            "sha256": "f1a25654d884a5b450e38d78b1f2e3ebb9073e421cc4358d4bbb83ac251a5670"
-        },
-        "pipfile-spec": 6,
-        "requires": {
-            "python_version": "3.6"
-        },
-        "sources": [
-            {
-                "name": "pypi",
-                "url": "https://pypi.org/simple",
-                "verify_ssl": true
-            }
-        ]
-    },
-    "default": {
-        "qemu": {
-            "editable": true,
-            "path": "."
-        }
-    },
-    "develop": {
-        "appdirs": {
-            "hashes": [
-                "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41",
-                "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128"
-            ],
-            "version": "==1.4.4"
-        },
-        "astroid": {
-            "hashes": [
-                "sha256:09bdb456e02564731f8b5957cdd0c98a7f01d2db5e90eb1d794c353c28bfd705",
-                "sha256:6a8a51f64dae307f6e0c9db752b66a7951e282389d8362cc1d39a56f3feeb31d"
-            ],
-            "index": "pypi",
-            "version": "==2.6.0"
-        },
-        "avocado-framework": {
-            "hashes": [
-                "sha256:244cb569f8eb4e50a22ac82e1a2b2bba2458999f4281efbe2651bd415d59c65b",
-                "sha256:6f15998b67ecd0e7dde790c4de4dd249d6df52dfe6d5cc4e2dd6596df51c3583"
-            ],
-            "index": "pypi",
-            "version": "==90.0"
-        },
-        "distlib": {
-            "hashes": [
-                "sha256:106fef6dc37dd8c0e2c0a60d3fca3e77460a48907f335fa28420463a6f799736",
-                "sha256:23e223426b28491b1ced97dc3bbe183027419dfc7982b4fa2f05d5f3ff10711c"
-            ],
-            "index": "pypi",
-            "version": "==0.3.2"
-        },
-        "filelock": {
-            "hashes": [
-                "sha256:18d82244ee114f543149c66a6e0c14e9c4f8a1044b5cdaadd0f82159d6a6ff59",
-                "sha256:929b7d63ec5b7d6b71b0fa5ac14e030b3f70b75747cef1b10da9b879fef15836"
-            ],
-            "index": "pypi",
-            "version": "==3.0.12"
-        },
-        "flake8": {
-            "hashes": [
-                "sha256:6a35f5b8761f45c5513e3405f110a86bea57982c3b75b766ce7b65217abe1670",
-                "sha256:c01f8a3963b3571a8e6bd7a4063359aff90749e160778e03817cd9b71c9e07d2"
-            ],
-            "index": "pypi",
-            "version": "==3.6.0"
-        },
-        "fusepy": {
-            "hashes": [
-                "sha256:10f5c7f5414241bffecdc333c4d3a725f1d6605cae6b4eaf86a838ff49cdaf6c",
-                "sha256:a9f3a3699080ddcf0919fd1eb2cf743e1f5859ca54c2018632f939bdfac269ee"
-            ],
-            "index": "pypi",
-            "version": "==2.0.4"
-        },
-        "importlib-metadata": {
-            "hashes": [
-                "sha256:90bb658cdbbf6d1735b6341ce708fc7024a3e14e99ffdc5783edea9f9b077f83",
-                "sha256:dc15b2969b4ce36305c51eebe62d418ac7791e9a157911d58bfb1f9ccd8e2070"
-            ],
-            "markers": "python_version < '3.8'",
-            "version": "==1.7.0"
-        },
-        "importlib-resources": {
-            "hashes": [
-                "sha256:54161657e8ffc76596c4ede7080ca68cb02962a2e074a2586b695a93a925d36e",
-                "sha256:e962bff7440364183203d179d7ae9ad90cb1f2b74dcb84300e88ecc42dca3351"
-            ],
-            "index": "pypi",
-            "version": "==5.1.4"
-        },
-        "isort": {
-            "hashes": [
-                "sha256:408e4d75d84f51b64d0824894afee44469eba34a4caee621dc53799f80d71ccc",
-                "sha256:64022dea6a06badfa09b300b4dfe8ba968114a737919e8ed50aea1c288f078aa"
-            ],
-            "index": "pypi",
-            "version": "==5.1.2"
-        },
-        "lazy-object-proxy": {
-            "hashes": [
-                "sha256:17e0967ba374fc24141738c69736da90e94419338fd4c7c7bef01ee26b339653",
-                "sha256:1fee665d2638491f4d6e55bd483e15ef21f6c8c2095f235fef72601021e64f61",
-                "sha256:22ddd618cefe54305df49e4c069fa65715be4ad0e78e8d252a33debf00f6ede2",
-                "sha256:24a5045889cc2729033b3e604d496c2b6f588c754f7a62027ad4437a7ecc4837",
-                "sha256:410283732af311b51b837894fa2f24f2c0039aa7f220135192b38fcc42bd43d3",
-                "sha256:4732c765372bd78a2d6b2150a6e99d00a78ec963375f236979c0626b97ed8e43",
-                "sha256:489000d368377571c6f982fba6497f2aa13c6d1facc40660963da62f5c379726",
-                "sha256:4f60460e9f1eb632584c9685bccea152f4ac2130e299784dbaf9fae9f49891b3",
-                "sha256:5743a5ab42ae40caa8421b320ebf3a998f89c85cdc8376d6b2e00bd12bd1b587",
-                "sha256:85fb7608121fd5621cc4377a8961d0b32ccf84a7285b4f1d21988b2eae2868e8",
-                "sha256:9698110e36e2df951c7c36b6729e96429c9c32b3331989ef19976592c5f3c77a",
-                "sha256:9d397bf41caad3f489e10774667310d73cb9c4258e9aed94b9ec734b34b495fd",
-                "sha256:b579f8acbf2bdd9ea200b1d5dea36abd93cabf56cf626ab9c744a432e15c815f",
-                "sha256:b865b01a2e7f96db0c5d12cfea590f98d8c5ba64ad222300d93ce6ff9138bcad",
-                "sha256:bf34e368e8dd976423396555078def5cfc3039ebc6fc06d1ae2c5a65eebbcde4",
-                "sha256:c6938967f8528b3668622a9ed3b31d145fab161a32f5891ea7b84f6b790be05b",
-                "sha256:d1c2676e3d840852a2de7c7d5d76407c772927addff8d742b9808fe0afccebdf",
-                "sha256:d7124f52f3bd259f510651450e18e0fd081ed82f3c08541dffc7b94b883aa981",
-                "sha256:d900d949b707778696fdf01036f58c9876a0d8bfe116e8d220cfd4b15f14e741",
-                "sha256:ebfd274dcd5133e0afae738e6d9da4323c3eb021b3e13052d8cbd0e457b1256e",
-                "sha256:ed361bb83436f117f9917d282a456f9e5009ea12fd6de8742d1a4752c3017e93",
-                "sha256:f5144c75445ae3ca2057faac03fda5a902eff196702b0a24daf1d6ce0650514b"
-            ],
-            "index": "pypi",
-            "version": "==1.6.0"
-        },
-        "mccabe": {
-            "hashes": [
-                "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
-                "sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
-            ],
-            "version": "==0.6.1"
-        },
-        "mypy": {
-            "hashes": [
-                "sha256:00cb1964a7476e871d6108341ac9c1a857d6bd20bf5877f4773ac5e9d92cd3cd",
-                "sha256:127de5a9b817a03a98c5ae8a0c46a20dc44442af6dcfa2ae7f96cb519b312efa",
-                "sha256:1f3976a945ad7f0a0727aafdc5651c2d3278e3c88dee94e2bf75cd3386b7b2f4",
-                "sha256:2f8c098f12b402c19b735aec724cc9105cc1a9eea405d08814eb4b14a6fb1a41",
-                "sha256:4ef13b619a289aa025f2273e05e755f8049bb4eaba6d703a425de37d495d178d",
-                "sha256:5d142f219bf8c7894dfa79ebfb7d352c4c63a325e75f10dfb4c3db9417dcd135",
-                "sha256:62eb5dd4ea86bda8ce386f26684f7f26e4bfe6283c9f2b6ca6d17faf704dcfad",
-                "sha256:64c36eb0936d0bfb7d8da49f92c18e312ad2e3ed46e5548ae4ca997b0d33bd59",
-                "sha256:75eed74d2faf2759f79c5f56f17388defd2fc994222312ec54ee921e37b31ad4",
-                "sha256:974bebe3699b9b46278a7f076635d219183da26e1a675c1f8243a69221758273",
-                "sha256:a5e5bb12b7982b179af513dddb06fca12285f0316d74f3964078acbfcf4c68f2",
-                "sha256:d31291df31bafb997952dc0a17ebb2737f802c754aed31dd155a8bfe75112c57",
-                "sha256:d3b4941de44341227ece1caaf5b08b23e42ad4eeb8b603219afb11e9d4cfb437",
-                "sha256:eadb865126da4e3c4c95bdb47fe1bb087a3e3ea14d39a3b13224b8a4d9f9a102"
-            ],
-            "index": "pypi",
-            "version": "==0.780"
-        },
-        "mypy-extensions": {
-            "hashes": [
-                "sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d",
-                "sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"
-            ],
-            "version": "==0.4.3"
-        },
-        "packaging": {
-            "hashes": [
-                "sha256:5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5",
-                "sha256:67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"
-            ],
-            "index": "pypi",
-            "version": "==20.9"
-        },
-        "pluggy": {
-            "hashes": [
-                "sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0",
-                "sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"
-            ],
-            "index": "pypi",
-            "version": "==0.13.1"
-        },
-        "py": {
-            "hashes": [
-                "sha256:21b81bda15b66ef5e1a777a21c4dcd9c20ad3efd0b3f817e7a809035269e1bd3",
-                "sha256:3b80836aa6d1feeaa108e046da6423ab8f6ceda6468545ae8d02d9d58d18818a"
-            ],
-            "index": "pypi",
-            "version": "==1.10.0"
-        },
-        "pycodestyle": {
-            "hashes": [
-                "sha256:74abc4e221d393ea5ce1f129ea6903209940c1ecd29e002e8c6933c2b21026e0",
-                "sha256:cbc619d09254895b0d12c2c691e237b2e91e9b2ecf5e84c26b35400f93dcfb83",
-                "sha256:cbfca99bd594a10f674d0cd97a3d802a1fdef635d4361e1a2658de47ed261e3a"
-            ],
-            "version": "==2.4.0"
-        },
-        "pyflakes": {
-            "hashes": [
-                "sha256:9a7662ec724d0120012f6e29d6248ae3727d821bba522a0e6b356eff19126a49",
-                "sha256:f661252913bc1dbe7fcfcbf0af0db3f42ab65aabd1a6ca68fe5d466bace94dae"
-            ],
-            "version": "==2.0.0"
-        },
-        "pygments": {
-            "hashes": [
-                "sha256:a18f47b506a429f6f4b9df81bb02beab9ca21d0a5fee38ed15aef65f0545519f",
-                "sha256:d66e804411278594d764fc69ec36ec13d9ae9147193a1740cd34d272ca383b8e"
-            ],
-            "index": "pypi",
-            "version": "==2.9.0"
-        },
-        "pylint": {
-            "hashes": [
-                "sha256:082a6d461b54f90eea49ca90fff4ee8b6e45e8029e5dbd72f6107ef84f3779c0",
-                "sha256:a01cd675eccf6e25b3bdb42be184eb46aaf89187d612ba0fb5f93328ed6b0fd5"
-            ],
-            "index": "pypi",
-            "version": "==2.8.0"
-        },
-        "pyparsing": {
-            "hashes": [
-                "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
-                "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
-            ],
-            "index": "pypi",
-            "version": "==2.4.7"
-        },
-        "qemu": {
-            "editable": true,
-            "path": "."
-        },
-        "setuptools": {
-            "hashes": [
-                "sha256:22c7348c6d2976a52632c67f7ab0cdf40147db7789f9aed18734643fe9cf3373",
-                "sha256:4ce92f1e1f8f01233ee9952c04f6b81d1e02939d6e1b488428154974a4d0783e"
-            ],
-            "markers": "python_version >= '3.6'",
-            "version": "==59.6.0"
-        },
-        "six": {
-            "hashes": [
-                "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
-                "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
-            ],
-            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
-            "version": "==1.16.0"
-        },
-        "toml": {
-            "hashes": [
-                "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
-                "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"
-            ],
-            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
-            "version": "==0.10.2"
-        },
-        "tox": {
-            "hashes": [
-                "sha256:c60692d92fe759f46c610ac04c03cf0169432d1ff8e981e8ae63e068d0954fc3",
-                "sha256:f179cb4043d7dc1339425dd49ab1dd8c916246b0d9173143c1b0af7498a03ab0"
-            ],
-            "index": "pypi",
-            "version": "==3.18.0"
-        },
-        "typed-ast": {
-            "hashes": [
-                "sha256:01ae5f73431d21eead5015997ab41afa53aa1fbe252f9da060be5dad2c730ace",
-                "sha256:067a74454df670dcaa4e59349a2e5c81e567d8d65458d480a5b3dfecec08c5ff",
-                "sha256:0fb71b8c643187d7492c1f8352f2c15b4c4af3f6338f21681d3681b3dc31a266",
-                "sha256:1b3ead4a96c9101bef08f9f7d1217c096f31667617b58de957f690c92378b528",
-                "sha256:2068531575a125b87a41802130fa7e29f26c09a2833fea68d9a40cf33902eba6",
-                "sha256:209596a4ec71d990d71d5e0d312ac935d86930e6eecff6ccc7007fe54d703808",
-                "sha256:2c726c276d09fc5c414693a2de063f521052d9ea7c240ce553316f70656c84d4",
-                "sha256:398e44cd480f4d2b7ee8d98385ca104e35c81525dd98c519acff1b79bdaac363",
-                "sha256:52b1eb8c83f178ab787f3a4283f68258525f8d70f778a2f6dd54d3b5e5fb4341",
-                "sha256:5feca99c17af94057417d744607b82dd0a664fd5e4ca98061480fd8b14b18d04",
-                "sha256:7538e495704e2ccda9b234b82423a4038f324f3a10c43bc088a1636180f11a41",
-                "sha256:760ad187b1041a154f0e4d0f6aae3e40fdb51d6de16e5c99aedadd9246450e9e",
-                "sha256:777a26c84bea6cd934422ac2e3b78863a37017618b6e5c08f92ef69853e765d3",
-                "sha256:95431a26309a21874005845c21118c83991c63ea800dd44843e42a916aec5899",
-                "sha256:9ad2c92ec681e02baf81fdfa056fe0d818645efa9af1f1cd5fd6f1bd2bdfd805",
-                "sha256:9c6d1a54552b5330bc657b7ef0eae25d00ba7ffe85d9ea8ae6540d2197a3788c",
-                "sha256:aee0c1256be6c07bd3e1263ff920c325b59849dc95392a05f258bb9b259cf39c",
-                "sha256:af3d4a73793725138d6b334d9d247ce7e5f084d96284ed23f22ee626a7b88e39",
-                "sha256:b36b4f3920103a25e1d5d024d155c504080959582b928e91cb608a65c3a49e1a",
-                "sha256:b9574c6f03f685070d859e75c7f9eeca02d6933273b5e69572e5ff9d5e3931c3",
-                "sha256:bff6ad71c81b3bba8fa35f0f1921fb24ff4476235a6e94a26ada2e54370e6da7",
-                "sha256:c190f0899e9f9f8b6b7863debfb739abcb21a5c054f911ca3596d12b8a4c4c7f",
-                "sha256:c907f561b1e83e93fad565bac5ba9c22d96a54e7ea0267c708bffe863cbe4075",
-                "sha256:cae53c389825d3b46fb37538441f75d6aecc4174f615d048321b716df2757fb0",
-                "sha256:dd4a21253f42b8d2b48410cb31fe501d32f8b9fbeb1f55063ad102fe9c425e40",
-                "sha256:dde816ca9dac1d9c01dd504ea5967821606f02e510438120091b84e852367428",
-                "sha256:f2362f3cb0f3172c42938946dbc5b7843c2a28aec307c49100c8b38764eb6927",
-                "sha256:f328adcfebed9f11301eaedfa48e15bdece9b519fb27e6a8c01aa52a17ec31b3",
-                "sha256:f8afcf15cc511ada719a88e013cec87c11aff7b91f019295eb4530f96fe5ef2f",
-                "sha256:fb1bbeac803adea29cedd70781399c99138358c26d05fcbd23c13016b7f5ec65"
-            ],
-            "markers": "python_version < '3.8' and implementation_name == 'cpython'",
-            "version": "==1.4.3"
-        },
-        "typing-extensions": {
-            "hashes": [
-                "sha256:0ac0f89795dd19de6b97debb0c6af1c70987fd80a2d62d1958f7e56fcc31b497",
-                "sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342",
-                "sha256:779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84"
-            ],
-            "index": "pypi",
-            "version": "==3.10.0.0"
-        },
-        "urwid": {
-            "hashes": [
-                "sha256:588bee9c1cb208d0906a9f73c613d2bd32c3ed3702012f51efe318a3f2127eae"
-            ],
-            "index": "pypi",
-            "version": "==2.1.2"
-        },
-        "urwid-readline": {
-            "hashes": [
-                "sha256:018020cbc864bb5ed87be17dc26b069eae2755cb29f3a9c569aac3bded1efaf4"
-            ],
-            "index": "pypi",
-            "version": "==0.13"
-        },
-        "virtualenv": {
-            "hashes": [
-                "sha256:14fdf849f80dbb29a4eb6caa9875d476ee2a5cf76a5f5415fa2f1606010ab467",
-                "sha256:2b0126166ea7c9c3661f5b8e06773d28f83322de7a3ff7d06f0aed18c9de6a76"
-            ],
-            "index": "pypi",
-            "version": "==20.4.7"
-        },
-        "wrapt": {
-            "hashes": [
-                "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
-            ],
-            "version": "==1.12.1"
-        },
-        "zipp": {
-            "hashes": [
-                "sha256:3607921face881ba3e026887d8150cca609d517579abe052ac81fc5aeffdbd76",
-                "sha256:51cb66cc54621609dd593d1787f286ee42a5c0adbb4b29abea5a63edc3e03098"
-            ],
-            "index": "pypi",
-            "version": "==3.4.1"
-        }
-    }
-}
diff --git a/python/setup.cfg b/python/setup.cfg
index 56418157065..9e923d97628 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -33,9 +33,7 @@ packages =
 * = py.typed
 
 [options.extras_require]
-# For the devel group, When adding new dependencies or bumping the minimum
-# version, use e.g. "pipenv install --dev pylint==3.0.0".
-# Subsequently, edit 'Pipfile' to remove e.g. 'pylint = "==3.0.0'.
+# Remember to update tests/minreqs.txt if changing anything below:
 devel =
     avocado-framework >= 90.0
     flake8 >= 3.6.0
diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
new file mode 100644
index 00000000000..dfb8abb155f
--- /dev/null
+++ b/python/tests/minreqs.txt
@@ -0,0 +1,45 @@
+# This file lists the ***oldest possible dependencies*** needed to run
+# "make check" successfully under ***Python 3.6***. It is used primarily
+# by GitLab CI to ensure that our stated minimum versions in setup.cfg
+# are truthful and regularly validated.
+#
+# This file should not contain any dependencies that are not expressed
+# by the [devel] section of setup.cfg, except for transitive
+# dependencies which must be enumerated here explicitly to eliminate
+# dependency resolution ambiguity.
+#
+# When adding new dependencies, pin the very oldest non-yanked version
+# on PyPI that allows the test suite to pass.
+
+# Dependencies for the TUI addon (Required for successful linting)
+urwid==2.1.2
+urwid-readline==0.13
+Pygments==2.9.0
+
+# Dependencies for FUSE support for qom-fuse
+fusepy==2.0.4
+
+# Test-runners, utilities, etc.
+avocado-framework==90.0
+
+# Linters
+flake8==3.6.0
+isort==5.1.2
+mypy==0.780
+pylint==2.8.0
+
+# Transitive flake8 dependencies
+mccabe==0.6.0
+pycodestyle==2.4.0
+pyflakes==2.0.0
+
+# Transitive mypy dependencies
+mypy-extensions==0.4.3
+typed-ast==1.4.0
+typing-extensions==3.7.4
+
+# Transitive pylint dependencies
+astroid==2.5.4
+lazy-object-proxy==1.4.0
+toml==0.10.0
+wrapt==1.12.1
diff --git a/tests/docker/dockerfiles/python.docker b/tests/docker/dockerfiles/python.docker
index 56d88417df4..175c10a34e8 100644
--- a/tests/docker/dockerfiles/python.docker
+++ b/tests/docker/dockerfiles/python.docker
@@ -7,7 +7,6 @@ MAINTAINER John Snow <jsnow@redhat.com>
 ENV PACKAGES \
     gcc \
     make \
-    pipenv \
     python3 \
     python3-pip \
     python3-tox \
-- 
2.39.0

[PATCH 3/7] configure: Look for auxiliary Python installations

[John Snow]

At the moment, we look for just "python3" and "python", which is good
enough almost all of the time. But ... if you are on a platform that
uses an older Python by default and only offers a newer Python as an
option, you'll have to specify --python=/usr/bin/foo every time.

We can be kind and instead make a cursory attempt to locate a suitable
Python binary ourselves, looking for the remaining well-known binaries.

This configure loop will prefer, in order:

1. Whatever is specified in $PYTHON
2. python3
3. python
4. python3.11 down through python3.6

Notes:

- Python virtual environment provides binaries for "python3", "python",
  and whichever version you used to create the venv,
  e.g. "python3.8". If configure is invoked from inside of a venv, this
  configure loop will not "break out" of that venv unless that venv is
  created using an explicitly non-suitable version of Python that we
  cannot use.

- In the event that no suitable python is found, the first python found
  is the version used to generate the human-readable error message.

- The error message isn't printed right away to allow later
  configuration code to pick up an explicitly configured python.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 33 +++++++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 8 deletions(-)

diff --git a/configure b/configure
index 64960c6000f..ea8c973d13b 100755
--- a/configure
+++ b/configure
@@ -592,20 +592,39 @@ esac
 
 : ${make=${MAKE-make}}
 
-# We prefer python 3.x. A bare 'python' is traditionally
-# python 2.x, but some distros have it as python 3.x, so
-# we check that too
+
+check_py_version() {
+    # We require python >= 3.6.
+    # NB: a True python conditional creates a non-zero return code (Failure)
+    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
+}
+
 python=
+first_python=
 explicit_python=no
-for binary in "${PYTHON-python3}" python
+# A bare 'python' is traditionally python 2.x, but some distros
+# have it as python 3.x, so check in both places.
+for binary in "${PYTHON-python3}" python python3.{11..6}
 do
     if has "$binary"
     then
         python=$(command -v "$binary")
-        break
+        if test -z "$first_python"; then
+           first_python=$python
+        fi
+        if check_py_version "$python"; then
+            # This one is good.
+            first_python=
+            break
+        fi
     fi
 done
 
+# If first_python is set, we didn't find a suitable binary.
+# Use this one for possible future error messages.
+if test -n "$first_python"; then
+    python="$first_python"
+fi
 
 # Check for ancillary tools used in testing
 genisoimage=
@@ -1037,9 +1056,7 @@ then
     error_exit "GNU make ($make) not found"
 fi
 
-# Note that if the Python conditional here evaluates True we will exit
-# with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
+if ! check_py_version "$python"; then
   error_exit "Cannot use '$python', Python >= 3.6 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
-- 
2.39.0

[PATCH 4/7] configure: Add nice hint to Python failure message

[John Snow]

If we begin requiring Python 3.7+, a few platforms are going to need to
install an additional package.

This is at least mildly annoying to the user (and I hate negative
attention), so solve the user's problem for them before they get a
chance to become irritated while searching on Google for how to install
newer Python packages.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index ea8c973d13b..bf512273f44 100755
--- a/configure
+++ b/configure
@@ -1058,7 +1058,10 @@ fi
 
 if ! check_py_version "$python"; then
   error_exit "Cannot use '$python', Python >= 3.6 is required." \
-      "Use --python=/path/to/python to specify a supported Python."
+             "Use --python=/path/to/python to specify a supported Python." \
+             "Maybe try:" \
+             "  openSUSE Leap 15.3+: zypper install python39" \
+             "  CentOS 8: dnf install python38"
 fi
 
 # Suppress writing compiled files
-- 
2.39.0

[PATCH 5/7] testing: Add Python >= 3.7 to Centos, OpenSuSE

[John Snow]

This is just a proof-of-concept patch, as these files are lcitool
generated. The real fix will involve updating the lcitool configuration
and updating these files that way.

Note that this requires OpenSuSE Leap 15.3; 15.2 won't cut it.

This is just to prove that bumping our dependency works.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/docker/dockerfiles/centos8.docker       | 1 +
 tests/docker/dockerfiles/opensuse-leap.docker | 1 +
 2 files changed, 2 insertions(+)

diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index fbc953c6dcc..a3bfddf382d 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -95,6 +95,7 @@ RUN dnf distro-sync -y && \
         pkgconfig \
         pulseaudio-libs-devel \
         python3 \
+        python38 \
         python3-PyYAML \
         python3-numpy \
         python3-pillow \
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 4b2c02d6abf..9e688c1d441 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -89,6 +89,7 @@ RUN zypper update -y && \
            pam-devel \
            pcre-devel-static \
            pkgconfig \
+           python39 \
            python3-Pillow \
            python3-PyYAML \
            python3-Sphinx \
-- 
2.39.0

[PATCH 6/7] CI: Stop building docs on centos8

[John Snow]

CentOS 8 does not ship with a sphinx new enough for our purposes (It
necessarily uses Python 3.6), so drop this from this build. We can
resume building docs on CentOS 9 if we wish, but we also currently test
and build docs on Fedora, Ubuntu, Alpine and Debian.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 .gitlab-ci.d/buildtest.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 0aa149a3524..0eb7f6606d4 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -166,7 +166,7 @@ build-system-centos:
   variables:
     IMAGE: centos8
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
-      --enable-modules --enable-trace-backends=dtrace --enable-docs
+      --enable-modules --enable-trace-backends=dtrace
       --enable-vfio-user-server
     TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
-- 
2.39.0

[PATCH 7/7] Python: Drop support for Python 3.6

[John Snow]

Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
begun dropping support for this version and it is becoming more
cumbersome to support. Avocado-framework and qemu.qmp each have their
own reasons for wanting to drop Python 3.6.

Since it is safe to under our supported platform policy, do so.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py             |  4 ++--
 configure                |  8 ++++----
 python/Makefile          | 10 +++++-----
 python/setup.cfg         |  7 +++----
 python/tests/minreqs.txt |  2 +-
 scripts/qapi/mypy.ini    |  2 +-
 6 files changed, 16 insertions(+), 17 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index 73a287a4f27..d40448f35d9 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -37,9 +37,9 @@
 # In newer versions of Sphinx this will display nicely; in older versions
 # Sphinx will also produce a Python backtrace but at least the information
 # gets printed...
-if sys.version_info < (3,6):
+if sys.version_info < (3,7):
     raise ConfigError(
-        "QEMU requires a Sphinx that uses Python 3.6 or better\n")
+        "QEMU requires a Sphinx that uses Python 3.7 or better\n")
 
 # The per-manual conf.py will set qemu_docdir for a single-manual build;
 # otherwise set it here if this is an entire-manual-set build.
diff --git a/configure b/configure
index bf512273f44..de27994ddc8 100755
--- a/configure
+++ b/configure
@@ -594,9 +594,9 @@ esac
 
 
 check_py_version() {
-    # We require python >= 3.6.
+    # We require python >= 3.7.
     # NB: a True python conditional creates a non-zero return code (Failure)
-    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
+    "$1" -c 'import sys; sys.exit(sys.version_info < (3,7))'
 }
 
 python=
@@ -604,7 +604,7 @@ first_python=
 explicit_python=no
 # A bare 'python' is traditionally python 2.x, but some distros
 # have it as python 3.x, so check in both places.
-for binary in "${PYTHON-python3}" python python3.{11..6}
+for binary in "${PYTHON-python3}" python python3.{11..7}
 do
     if has "$binary"
     then
@@ -1057,7 +1057,7 @@ then
 fi
 
 if ! check_py_version "$python"; then
-  error_exit "Cannot use '$python', Python >= 3.6 is required." \
+  error_exit "Cannot use '$python', Python >= 3.7 is required." \
              "Use --python=/path/to/python to specify a supported Python." \
              "Maybe try:" \
              "  openSUSE Leap 15.3+: zypper install python39" \
diff --git a/python/Makefile b/python/Makefile
index c5bd6ff83ac..f660d999143 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -9,14 +9,14 @@ help:
 	@echo "make check-minreqs:"
 	@echo "    Run tests in the minreqs virtual environment."
 	@echo "    These tests use the oldest dependencies."
-	@echo "    Requires: Python 3.6"
-	@echo "    Hint (Fedora): 'sudo dnf install python3.6'"
+	@echo "    Requires: Python 3.7"
+	@echo "    Hint (Fedora): 'sudo dnf install python3.7'"
 	@echo ""
 	@echo "make check-tox:"
 	@echo "    Run tests against multiple python versions."
 	@echo "    These tests use the newest dependencies."
-	@echo "    Requires: Python 3.6 - 3.10, and tox."
-	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.10'"
+	@echo "    Requires: Python 3.7 - 3.11, and tox."
+	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.11'"
 	@echo "    The variable QEMU_TOX_EXTRA_ARGS can be use to pass extra"
 	@echo "    arguments to tox".
 	@echo ""
@@ -58,7 +58,7 @@ pipenv check-pipenv:
 min-venv: $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate
 $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate: setup.cfg tests/minreqs.txt
 	@echo "VENV $(QEMU_MINVENV_DIR)"
-	@python3.6 -m venv $(QEMU_MINVENV_DIR)
+	@python3.7 -m venv $(QEMU_MINVENV_DIR)
 	@(								\
 		echo "ACTIVATE $(QEMU_MINVENV_DIR)";			\
 		. $(QEMU_MINVENV_DIR)/bin/activate;			\
diff --git a/python/setup.cfg b/python/setup.cfg
index 9e923d97628..1e8392a045c 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -14,7 +14,6 @@ classifiers =
     Natural Language :: English
     Operating System :: OS Independent
     Programming Language :: Python :: 3 :: Only
-    Programming Language :: Python :: 3.6
     Programming Language :: Python :: 3.7
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
@@ -23,7 +22,7 @@ classifiers =
     Typing :: Typed
 
 [options]
-python_requires = >= 3.6
+python_requires = >= 3.7
 packages =
     qemu.qmp
     qemu.machine
@@ -76,7 +75,7 @@ exclude = __pycache__,
 
 [mypy]
 strict = True
-python_version = 3.6
+python_version = 3.7
 warn_unused_configs = True
 namespace_packages = True
 warn_unused_ignores = False
@@ -158,7 +157,7 @@ multi_line_output=3
 # of python available on your system to run this test.
 
 [tox:tox]
-envlist = py36, py37, py38, py39, py310, py311
+envlist = py37, py38, py39, py310, py311
 skip_missing_interpreters = true
 
 [testenv]
diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
index dfb8abb155f..55cc6b41d85 100644
--- a/python/tests/minreqs.txt
+++ b/python/tests/minreqs.txt
@@ -1,5 +1,5 @@
 # This file lists the ***oldest possible dependencies*** needed to run
-# "make check" successfully under ***Python 3.6***. It is used primarily
+# "make check" successfully under ***Python 3.7***. It is used primarily
 # by GitLab CI to ensure that our stated minimum versions in setup.cfg
 # are truthful and regularly validated.
 #
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 66253564297..3463307ddc7 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -1,7 +1,7 @@
 [mypy]
 strict = True
 disallow_untyped_calls = False
-python_version = 3.6
+python_version = 3.7
 
 [mypy-qapi.schema]
 disallow_untyped_defs = False
-- 
2.39.0

Re: [PATCH 5/7] testing: Add Python >= 3.7 to Centos, OpenSuSE

[Thomas Huth]

On 09/02/2023 16.40, John Snow wrote:
> This is just a proof-of-concept patch, as these files are lcitool
> generated. The real fix will involve updating the lcitool configuration
> and updating these files that way.

I think it would be good to have a RFC or DONOTMERGE in the patch title, so 
that it is clear right from the start that this should not get merged.

> Note that this requires OpenSuSE Leap 15.3; 15.2 won't cut it.

Looking at https://en.wikipedia.org/wiki/OpenSUSE#Version_history it seems 
like even 15.3 is EOL already, so I think we can switch to 15.4 immediately.

  Thomas

Re: [PATCH 6/7] CI: Stop building docs on centos8

[Thomas Huth]

On 09/02/2023 16.40, John Snow wrote:
> CentOS 8 does not ship with a sphinx new enough for our purposes (It
> necessarily uses Python 3.6), so drop this from this build. We can
> resume building docs on CentOS 9 if we wish, but we also currently test
> and build docs on Fedora, Ubuntu, Alpine and Debian.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 0aa149a3524..0eb7f6606d4 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -166,7 +166,7 @@ build-system-centos:
>     variables:
>       IMAGE: centos8
>       CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
> -      --enable-modules --enable-trace-backends=dtrace --enable-docs
> +      --enable-modules --enable-trace-backends=dtrace
>         --enable-vfio-user-server
>       TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
>         x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 1/7] python: support pylint 2.16

[Philippe Mathieu-Daudé]

On 9/2/23 16:40, John Snow wrote:
> Pylint 2.16 adds a few new checks that cause the optional check-tox CI
> job to fail.
> 
> 1. The superfluous-parens check seems to be a bit more aggressive,
> 2. broad-exception-raised is new; it discourages "raise Exception".
> 
> Fix these minor issues and turn the lights green.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   python/qemu/qmp/protocol.py                            | 2 +-
>   python/qemu/qmp/qmp_client.py                          | 2 +-
>   python/qemu/utils/qemu_ga_client.py                    | 6 +++---
>   tests/qemu-iotests/iotests.py                          | 4 ++--
>   tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 2 +-
>   5 files changed, 8 insertions(+), 8 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 6/7] CI: Stop building docs on centos8

[Philippe Mathieu-Daudé]

On 9/2/23 16:40, John Snow wrote:
> CentOS 8 does not ship with a sphinx new enough for our purposes (It
> necessarily uses Python 3.6), so drop this from this build. We can
> resume building docs on CentOS 9 if we wish, but we also currently test
> and build docs on Fedora, Ubuntu, Alpine and Debian.

I guess remember this wide collection is on purpose, since sphinx
had different dependency versions and was failing differently on each
distrib. Peter might remember.

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 0aa149a3524..0eb7f6606d4 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -166,7 +166,7 @@ build-system-centos:
>     variables:
>       IMAGE: centos8

If so (wide distrib collection), a "Please add --enable-docs when
upgrading to centos9" comment here.

>       CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
> -      --enable-modules --enable-trace-backends=dtrace --enable-docs
> +      --enable-modules --enable-trace-backends=dtrace
>         --enable-vfio-user-server
>       TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
>         x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu

Re: [PATCH 5/7] testing: Add Python >= 3.7 to Centos, OpenSuSE

[John Snow]

On Thu, Feb 9, 2023 at 10:56 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 09/02/2023 16.40, John Snow wrote:
> > This is just a proof-of-concept patch, as these files are lcitool
> > generated. The real fix will involve updating the lcitool configuration
> > and updating these files that way.
>
> I think it would be good to have a RFC or DONOTMERGE in the patch title, so
> that it is clear right from the start that this should not get merged.

Apologies. I just assumed there'd be no risk of this passing review :)
If there's any point in me respinning this with the proof-of-concept
again, I'll add a DO-NOT-MERGE tag.

>
> > Note that this requires OpenSuSE Leap 15.3; 15.2 won't cut it.
>
> Looking at https://en.wikipedia.org/wiki/OpenSUSE#Version_history it seems
> like even 15.3 is EOL already, so I think we can switch to 15.4 immediately.
>

Noted. Some of the patches in this series are a bit old, admittedly,
so I'll re-review these requirements.

I'm hoping to hear from Dan/Alex on the subject of coverage, because I
don't really know what our current strategy is or what our coverage
ought to be. If CI minutes were infinitely free, I'd suggest we test
both CentOS 8 and 9 simultaneously, alongside with every currently
supported OpenSuSE version, but I imagine that's not really truly on
the table.

I imagine we want to have a mixture of "cutting edge" with "oldest we
still support" to try and cover the spread of likely scenarios, but I
am not sure what the strategies for which to keep at which versions
are.

>   Thomas
>

Re: [PATCH 7/7] Python: Drop support for Python 3.6

[Daniel P. Berrangé]

On Thu, Feb 09, 2023 at 10:40:34AM -0500, John Snow wrote:
> Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
> begun dropping support for this version and it is becoming more
> cumbersome to support. Avocado-framework and qemu.qmp each have their
> own reasons for wanting to drop Python 3.6.
>
> Since it is safe to under our supported platform policy, do so.

Upstream EOL dates are essentially irrelevant from our platform
support policy for deciding min versions

QEMU aims to target released OS distributions, and their vendors
may choose to support software for arbitrarily longer periods of
time than upstream. This extended support is one of the key
value adds of OS distributions.

To justify dropping python 3.6, the commit message needs to
illustrate that all our targetted distros are capable of
supporting the new proposed new min version. 

> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  docs/conf.py             |  4 ++--
>  configure                |  8 ++++----
>  python/Makefile          | 10 +++++-----
>  python/setup.cfg         |  7 +++----
>  python/tests/minreqs.txt |  2 +-
>  scripts/qapi/mypy.ini    |  2 +-
>  6 files changed, 16 insertions(+), 17 deletions(-)


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 7/7] Python: Drop support for Python 3.6

[John Snow]

On Thu, Feb 9, 2023 at 11:27 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Feb 09, 2023 at 10:40:34AM -0500, John Snow wrote:
> > Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
> > begun dropping support for this version and it is becoming more
> > cumbersome to support. Avocado-framework and qemu.qmp each have their
> > own reasons for wanting to drop Python 3.6.
> >
> > Since it is safe to under our supported platform policy, do so.
>
> Upstream EOL dates are essentially irrelevant from our platform
> support policy for deciding min versions
>

It's relevant because as other Python packages drop support, the
burden of support is pushed onto Cleber and I, as explained in the
cover letter. It is not a justification in and of itself, but it
summarizes the ecosystem conditions that prompt the desire for the
change.

> QEMU aims to target released OS distributions, and their vendors
> may choose to support software for arbitrarily longer periods of
> time than upstream. This extended support is one of the key
> value adds of OS distributions.
>
> To justify dropping python 3.6, the commit message needs to
> illustrate that all our targetted distros are capable of
> supporting the new proposed new min version.

That is the point of this series: illustrating that it is indeed safe
to drop Python 3.6. I am not clear on what you are actually requesting
as a change.

> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  docs/conf.py             |  4 ++--
> >  configure                |  8 ++++----
> >  python/Makefile          | 10 +++++-----
> >  python/setup.cfg         |  7 +++----
> >  python/tests/minreqs.txt |  2 +-
> >  scripts/qapi/mypy.ini    |  2 +-
> >  6 files changed, 16 insertions(+), 17 deletions(-)
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: [PATCH 1/7] python: support pylint 2.16

[Beraldo Leal]

On Thu, Feb 09, 2023 at 10:40:28AM -0500, John Snow wrote:
> Pylint 2.16 adds a few new checks that cause the optional check-tox CI
> job to fail.
> 
> 1. The superfluous-parens check seems to be a bit more aggressive,
> 2. broad-exception-raised is new; it discourages "raise Exception".
> 
> Fix these minor issues and turn the lights green.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/qemu/qmp/protocol.py                            | 2 +-
>  python/qemu/qmp/qmp_client.py                          | 2 +-
>  python/qemu/utils/qemu_ga_client.py                    | 6 +++---
>  tests/qemu-iotests/iotests.py                          | 4 ++--
>  tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 2 +-
>  5 files changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/python/qemu/qmp/protocol.py b/python/qemu/qmp/protocol.py
> index 6d3d739daa7..22e60298d28 100644
> --- a/python/qemu/qmp/protocol.py
> +++ b/python/qemu/qmp/protocol.py
> @@ -207,7 +207,7 @@ class AsyncProtocol(Generic[T]):
>      logger = logging.getLogger(__name__)
>  
>      # Maximum allowable size of read buffer
> -    _limit = (64 * 1024)
> +    _limit = 64 * 1024
>  
>      # -------------------------
>      # Section: Public interface
> diff --git a/python/qemu/qmp/qmp_client.py b/python/qemu/qmp/qmp_client.py
> index b5772e7f32b..9d73ae6e7ad 100644
> --- a/python/qemu/qmp/qmp_client.py
> +++ b/python/qemu/qmp/qmp_client.py
> @@ -198,7 +198,7 @@ async def run(self, address='/tmp/qemu.socket'):
>      logger = logging.getLogger(__name__)
>  
>      # Read buffer limit; 10MB like libvirt default
> -    _limit = (10 * 1024 * 1024)
> +    _limit = 10 * 1024 * 1024
>  
>      # Type alias for pending execute() result items
>      _PendingT = Union[Message, ExecInterruptedError]
> diff --git a/python/qemu/utils/qemu_ga_client.py b/python/qemu/utils/qemu_ga_client.py
> index 8c38a7ac9c0..d8411bb2d0b 100644
> --- a/python/qemu/utils/qemu_ga_client.py
> +++ b/python/qemu/utils/qemu_ga_client.py
> @@ -155,7 +155,7 @@ def ping(self, timeout: Optional[float]) -> bool:
>  
>      def fsfreeze(self, cmd: str) -> object:
>          if cmd not in ['status', 'freeze', 'thaw']:
> -            raise Exception('Invalid command: ' + cmd)
> +            raise ValueError('Invalid command: ' + cmd)
>          # Can be int (freeze, thaw) or GuestFsfreezeStatus (status)
>          return getattr(self.qga, 'fsfreeze' + '_' + cmd)()
>  
> @@ -167,7 +167,7 @@ def fstrim(self, minimum: int) -> Dict[str, object]:
>  
>      def suspend(self, mode: str) -> None:
>          if mode not in ['disk', 'ram', 'hybrid']:
> -            raise Exception('Invalid mode: ' + mode)
> +            raise ValueError('Invalid mode: ' + mode)
>  
>          try:
>              getattr(self.qga, 'suspend' + '_' + mode)()
> @@ -178,7 +178,7 @@ def suspend(self, mode: str) -> None:
>  
>      def shutdown(self, mode: str = 'powerdown') -> None:
>          if mode not in ['powerdown', 'halt', 'reboot']:
> -            raise Exception('Invalid mode: ' + mode)
> +            raise ValueError('Invalid mode: ' + mode)
>  
>          try:
>              self.qga.shutdown(mode=mode)
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 94aeb3f3b20..3e82c634cfe 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -720,7 +720,7 @@ def __exit__(self, exc_type, value, traceback):
>          signal.setitimer(signal.ITIMER_REAL, 0)
>          return False
>      def timeout(self, signum, frame):
> -        raise Exception(self.errmsg)
> +        raise TimeoutError(self.errmsg)
>  
>  def file_pattern(name):
>      return "{0}-{1}".format(os.getpid(), name)
> @@ -804,7 +804,7 @@ def remote_filename(path):
>      elif imgproto == 'ssh':
>          return "ssh://%s@127.0.0.1:22%s" % (os.environ.get('USER'), path)
>      else:
> -        raise Exception("Protocol %s not supported" % (imgproto))
> +        raise ValueError("Protocol %s not supported" % (imgproto))
>  
>  class VM(qtest.QEMUQtestMachine):
>      '''A QEMU VM'''
> diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
> index fc9c4b4ef41..dda55fad284 100755
> --- a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
> +++ b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
> @@ -84,7 +84,7 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
>                  e['vm'] = 'SRC'
>              for e in self.vm_b_events:
>                  e['vm'] = 'DST'
> -            events = (self.vm_a_events + self.vm_b_events)
> +            events = self.vm_a_events + self.vm_b_events
>              events = [(e['timestamp']['seconds'],
>                         e['timestamp']['microseconds'],
>                         e['vm'],
> -- 
> 2.39.0
>

Reviewed-by: Beraldo Leal <bleal@redhat.com>

--
Beraldo

Re: [PATCH 7/7] Python: Drop support for Python 3.6

[Daniel P. Berrangé]

On Thu, Feb 09, 2023 at 11:40:57AM -0500, John Snow wrote:
> On Thu, Feb 9, 2023 at 11:27 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Thu, Feb 09, 2023 at 10:40:34AM -0500, John Snow wrote:
> > > Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
> > > begun dropping support for this version and it is becoming more
> > > cumbersome to support. Avocado-framework and qemu.qmp each have their
> > > own reasons for wanting to drop Python 3.6.
> > >
> > > Since it is safe to under our supported platform policy, do so.
> >
> > Upstream EOL dates are essentially irrelevant from our platform
> > support policy for deciding min versions
> >
> 
> It's relevant because as other Python packages drop support, the
> burden of support is pushed onto Cleber and I, as explained in the
> cover letter. It is not a justification in and of itself, but it
> summarizes the ecosystem conditions that prompt the desire for the
> change.
> > QEMU aims to target released OS distributions, and their vendors
> > may choose to support software for arbitrarily longer periods of
> > time than upstream. This extended support is one of the key
> > value adds of OS distributions.
> >
> > To justify dropping python 3.6, the commit message needs to
> > illustrate that all our targetted distros are capable of
> > supporting the new proposed new min version.
> 
> That is the point of this series: illustrating that it is indeed safe
> to drop Python 3.6. I am not clear on what you are actually requesting
> as a change.

Essentially it should explain what versions are available in each
distro. See for example commit 5890258aeeba303704ec1adca415e46067800777.



With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 7/7] Python: Drop support for Python 3.6

[John Snow]

On Thu, Feb 9, 2023 at 1:03 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Feb 09, 2023 at 11:40:57AM -0500, John Snow wrote:
> > On Thu, Feb 9, 2023 at 11:27 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > On Thu, Feb 09, 2023 at 10:40:34AM -0500, John Snow wrote:
> > > > Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
> > > > begun dropping support for this version and it is becoming more
> > > > cumbersome to support. Avocado-framework and qemu.qmp each have their
> > > > own reasons for wanting to drop Python 3.6.
> > > >
> > > > Since it is safe to under our supported platform policy, do so.
> > >
> > > Upstream EOL dates are essentially irrelevant from our platform
> > > support policy for deciding min versions
> > >
> >
> > It's relevant because as other Python packages drop support, the
> > burden of support is pushed onto Cleber and I, as explained in the
> > cover letter. It is not a justification in and of itself, but it
> > summarizes the ecosystem conditions that prompt the desire for the
> > change.
> > > QEMU aims to target released OS distributions, and their vendors
> > > may choose to support software for arbitrarily longer periods of
> > > time than upstream. This extended support is one of the key
> > > value adds of OS distributions.
> > >
> > > To justify dropping python 3.6, the commit message needs to
> > > illustrate that all our targetted distros are capable of
> > > supporting the new proposed new min version.
> >
> > That is the point of this series: illustrating that it is indeed safe
> > to drop Python 3.6. I am not clear on what you are actually requesting
> > as a change.
>
> Essentially it should explain what versions are available in each
> distro. See for example commit 5890258aeeba303704ec1adca415e46067800777.

Ah, OK.

Fedora 36: 3.10.5
Ubuntu 20.04: 3.8.10
Debian 11: 3.9.2
OpenSUSE Leap 15.4 uses 3.6.15, but 3.9 and 3.10 are available with a
simple zypper invocation.
CentOS Stream 8 uses 3.6.8, but 3.8 and 3.9 are available.
CentOS Stream 9 uses 3.9.13.
Alpine: Not sure, but tests pass.
FreeBSD 12: 3.7 (tests/vm/freebsd already uses this version.)
OpenBSD: ?
NetBSD: ?

I can update the commit message, but I do have questions for you in
patch 5 concerning the docker configurations and our test matrix that
I want to work out before a respin.

>
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: [PATCH 6/7] CI: Stop building docs on centos8

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

Il gio 9 feb 2023, 16:40 John Snow <jsnow@redhat.com> ha scritto:

> CentOS 8 does not ship with a sphinx new enough for our purposes (It
> necessarily uses Python 3.6), so drop this from this build. We can
> resume building docs on CentOS 9 if we wish, but we also currently test
> and build docs on Fedora, Ubuntu, Alpine and Debian.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
>

It's possible to teach lcitool to use pip instead to install docutils,
sphinx and sphinx-rtd-theme
Once we can drop the previous patch this one can go as well; as soon as the
last lcitool prerequisite is in (
https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/341) I will pick the
rest of this series, if that's okay for you.

Paolo

---
>  .gitlab-ci.d/buildtest.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 0aa149a3524..0eb7f6606d4 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -166,7 +166,7 @@ build-system-centos:
>    variables:
>      IMAGE: centos8
>      CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
> -      --enable-modules --enable-trace-backends=dtrace --enable-docs
> +      --enable-modules --enable-trace-backends=dtrace
>        --enable-vfio-user-server
>      TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
>        x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
> --
> 2.39.0
>
>

[-- Attachment #2: Type: text/html, Size: 2299 bytes --]

Re: [PATCH 6/7] CI: Stop building docs on centos8

[John Snow]

On Thu, Feb 9, 2023 at 5:43 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
>
> Il gio 9 feb 2023, 16:40 John Snow <jsnow@redhat.com> ha scritto:
>>
>> CentOS 8 does not ship with a sphinx new enough for our purposes (It
>> necessarily uses Python 3.6), so drop this from this build. We can
>> resume building docs on CentOS 9 if we wish, but we also currently test
>> and build docs on Fedora, Ubuntu, Alpine and Debian.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>
>
> It's possible to teach lcitool to use pip instead to install docutils, sphinx and sphinx-rtd-theme
> Once we can drop the previous patch this one can go as well; as soon as the last lcitool prerequisite is in (https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/341) I will pick the rest of this series, if that's okay for you.

OK - Dan requested extra version information in the commit message to
7, which I am researching a bit more exhaustively now. I can reply to
that patch with the requested info and if you'd like to pick 1-4 & 7,
you can be my guest.

I will say that once upon a time, Peter Maydell expressed a preference
to use the version(s) of sphinx managed by the distro instead of
swallowing it into the python-managed parts of the codebase. It seemed
like a reasonable request at the time. We can revisit it once I get
the patches in to add a venv at configure time. (Which I was intending
to visit next.)

>
> Paolo
>

Re: [PATCH 6/7] CI: Stop building docs on centos8

[Paolo Bonzini]

On 2/10/23 00:32, John Snow wrote:
>> It's possible to teach lcitool to use pip instead to install
>> docutils, sphinx and sphinx-rtd-theme
> 
> I will say that once upon a time, Peter Maydell expressed a
> preference to use the version(s) of sphinx managed by the distro
> instead of swallowing it into the python-managed parts of the
> codebase. It seemed like a reasonable request at the time.

Note that this is not about using pip in a venv set up by e.g. 
configure; simply, lcitool falls back to pip if a package is not 
available as an RPM/DEB/APK, so it's a matter of telling it that the 
python3.8 runtime in CentOS 8 does not have either docutils or sphinx.

In other words, since lcitool takes care of building the container 
dockerfile, this "pip install" would be done in the container phase and 
configure in general cannot care less whether sphinx comes from an RPM 
or a "pip install".

Paolo

> We can
> revisit it once I get the patches in to add a venv at configure time.
> (Which I was intending to visit next.)

Re: [PATCH 5/7] testing: Add Python >= 3.7 to Centos, OpenSuSE

[Daniel P. Berrangé]

On Thu, Feb 09, 2023 at 10:40:32AM -0500, John Snow wrote:
> This is just a proof-of-concept patch, as these files are lcitool
> generated. The real fix will involve updating the lcitool configuration
> and updating these files that way.

Paolo has been working on this

  https://lists.gnu.org/archive/html/qemu-devel/2023-01/msg03547.html

though I requested we adapt it a little:

  https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg02237.html
  https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg02239.html

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 3/7] configure: Look for auxiliary Python installations

[Eric Blake]

On Thu, Feb 09, 2023 at 10:40:30AM -0500, John Snow wrote:
> At the moment, we look for just "python3" and "python", which is good
> enough almost all of the time. But ... if you are on a platform that
> uses an older Python by default and only offers a newer Python as an
> option, you'll have to specify --python=/usr/bin/foo every time.
> 
> We can be kind and instead make a cursory attempt to locate a suitable
> Python binary ourselves, looking for the remaining well-known binaries.
> 
> This configure loop will prefer, in order:
> 
> 1. Whatever is specified in $PYTHON
> 2. python3
> 3. python
> 4. python3.11 down through python3.6

Makes sense.


>  python=
> +first_python=
>  explicit_python=no
> -for binary in "${PYTHON-python3}" python
> +# A bare 'python' is traditionally python 2.x, but some distros
> +# have it as python 3.x, so check in both places.
> +for binary in "${PYTHON-python3}" python python3.{11..6}

This does not match your commit message. If $PYTHON is set but fails,
you never check python3.  Pre-existing, but now that you're calling it
out as intended, it may be better to write the list prefix as:

for binary in $PYTHON python3 python ...

except that it mishandles $PYTHON containing space, so you want the
quotes, but you don't want to test an empty binary or waste time
testing python3 twice, so more precise could be:

for binary in "${PYTHON-python3}" ${PYTHON:+python3} python ...

Meanwhioe, your use of {11.6} is a bashism, but configure is /bin/sh.
It would be nice if you could use $(seq -f python3.%g 11 -1 6), but
that's probably too specific to GNU Coreutils and won't work on other
platforms; and open-coding it in a shell loop isn't going to be any
prettier.  So you'll be safest if you just manually spell it out:

python3.11 python3.10 ...

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [PATCH 3/7] configure: Look for auxiliary Python installations

[John Snow]

On Fri, Feb 10, 2023 at 2:45 PM Eric Blake <eblake@redhat.com> wrote:
>
> On Thu, Feb 09, 2023 at 10:40:30AM -0500, John Snow wrote:
> > At the moment, we look for just "python3" and "python", which is good
> > enough almost all of the time. But ... if you are on a platform that
> > uses an older Python by default and only offers a newer Python as an
> > option, you'll have to specify --python=/usr/bin/foo every time.
> >
> > We can be kind and instead make a cursory attempt to locate a suitable
> > Python binary ourselves, looking for the remaining well-known binaries.
> >
> > This configure loop will prefer, in order:
> >
> > 1. Whatever is specified in $PYTHON
> > 2. python3
> > 3. python
> > 4. python3.11 down through python3.6
>
> Makes sense.
>
>
> >  python=
> > +first_python=
> >  explicit_python=no
> > -for binary in "${PYTHON-python3}" python
> > +# A bare 'python' is traditionally python 2.x, but some distros
> > +# have it as python 3.x, so check in both places.
> > +for binary in "${PYTHON-python3}" python python3.{11..6}
>
> This does not match your commit message. If $PYTHON is set but fails,
> you never check python3.  Pre-existing, but now that you're calling it
> out as intended, it may be better to write the list prefix as:
>
> for binary in $PYTHON python3 python ...
>
> except that it mishandles $PYTHON containing space, so you want the
> quotes, but you don't want to test an empty binary or waste time
> testing python3 twice, so more precise could be:
>
> for binary in "${PYTHON-python3}" ${PYTHON:+python3} python ...

Yep. I wonder if it's worth helping people work around broken
environment variables at all, but, eh, you already solved the more
difficult case, so I can include it.

(Well, pending other discussions, I s'pose.)

>
> Meanwhioe, your use of {11.6} is a bashism, but configure is /bin/sh.
> It would be nice if you could use $(seq -f python3.%g 11 -1 6), but
> that's probably too specific to GNU Coreutils and won't work on other
> platforms; and open-coding it in a shell loop isn't going to be any
> prettier.  So you'll be safest if you just manually spell it out:
>
> python3.11 python3.10 ...

Oh, icky.  Well, OK. Thanks for pointing it out before it became a
problem somewhere.

>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3266
> Virtualization:  qemu.org | libvirt.org
>

Re: [PATCH 5/7] testing: Add Python >= 3.7 to Centos, OpenSuSE

[John Snow]

On Fri, Feb 10, 2023 at 5:38 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Feb 09, 2023 at 10:40:32AM -0500, John Snow wrote:
> > This is just a proof-of-concept patch, as these files are lcitool
> > generated. The real fix will involve updating the lcitool configuration
> > and updating these files that way.
>
> Paolo has been working on this
>
>   https://lists.gnu.org/archive/html/qemu-devel/2023-01/msg03547.html
>
> though I requested we adapt it a little:
>
>   https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg02237.html
>   https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg02239.html
>
> With regards,
> Daniel

Ah, I had no idea that Paolo was specifically working on 3.7 as a
minimum requisite; I hadn't actually seen this or talked with him
about this in some time. With this series I was mostly occupied with
the simpler question of "Is this even possible?"

... I'll just trust that the lcitool changes are in good hands with Paolo.

Thanks for the links to the relevant threads :~)

--js

Re: [PATCH 6/7] CI: Stop building docs on centos8

[John Snow]

On Fri, Feb 10, 2023 at 5:30 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 2/10/23 00:32, John Snow wrote:
> >> It's possible to teach lcitool to use pip instead to install
> >> docutils, sphinx and sphinx-rtd-theme
> >
> > I will say that once upon a time, Peter Maydell expressed a
> > preference to use the version(s) of sphinx managed by the distro
> > instead of swallowing it into the python-managed parts of the
> > codebase. It seemed like a reasonable request at the time.
>
> Note that this is not about using pip in a venv set up by e.g.
> configure; simply, lcitool falls back to pip if a package is not
> available as an RPM/DEB/APK, so it's a matter of telling it that the
> python3.8 runtime in CentOS 8 does not have either docutils or sphinx.
>
> In other words, since lcitool takes care of building the container
> dockerfile, this "pip install" would be done in the container phase and
> configure in general cannot care less whether sphinx comes from an RPM
> or a "pip install".
>
> Paolo
>

Understood - I am just recalling changes I proposed to 'configure'
several years back to specifically allow it to find and utilize
versions of sphinx installed via pip, but Peter Maydell wasn't as sure
of the idea, believing that QEMU should stick to only the system
packages. I wasn't talking about specifically *creating* the
environments with pip, this time.

(But, I'd still like to, but realize that's a separate discussion.)