[PATCH v2 0/7] Python: Drop support for Python 3.6

[PATCH v2 0/7] Python: Drop support for Python 3.6

[John Snow]

Howdy, this series increases our minimum python version to 3.7.

CI: https://gitlab.com/jsnow/qemu/-/pipelines/771780626
    (All green!)
GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37

Patches 1 and 2 are loose pre-requisites; I'd like to merge them into
qemu.git within the week whether or not we take this series. I'd
appreciate an "ACK" on those specifically. They're just riding along
here because they make this series a bit nicer.

Patches 3-6 are the hard pre-requisites, and 7 does the dirty work.

The motivation for this series is that Python 3.6 was EOL at the end of
2021; upstream tools are beginning to drop support for it, including
setuptools, pylint, mypy, etc. As time goes by, it becomes more
difficult to support and test against the full range of Python versions
that QEMU supports. The closer we get to Python 3.12, the harder it will
be to cover that full spread of versions.

The qemu.qmp library and the avocado testing framework both have
motivations for dropping 3.6 support, but are committed to not doing so
until QEMU drops support.

So, I'd like to talk about doing it.

V2:
- Added R-Bs to patch 1
- Updated commit message for patch 7 with explicit version info
- Added DO-NOT-MERGE to patch 5's title
- Tested tests/vm/freebsd, netbsd, and openbsd in addition to full CI

RFC:
 - Patch 5 is just a proof-of-concept; we need to update lcitool instead.
 - Cleber, I need to update your ansible scripts. How do I test them?

Thanks!
--js

John Snow (7):
  python: support pylint 2.16
  python: drop pipenv
  configure: Look for auxiliary Python installations
  configure: Add nice hint to Python failure message
  DO-NOT-MERGE: testing: Add Python >= 3.7 to Centos, OpenSuSE
  CI: Stop building docs on centos8
  Python: Drop support for Python 3.6

 docs/conf.py                                  |   4 +-
 python/README.rst                             |   3 -
 configure                                     |  40 +-
 .gitlab-ci.d/buildtest.yml                    |   2 +-
 .gitlab-ci.d/static_checks.yml                |   4 +-
 python/.gitignore                             |   4 +-
 python/Makefile                               |  57 ++-
 python/Pipfile                                |  13 -
 python/Pipfile.lock                           | 347 ------------------
 python/qemu/qmp/protocol.py                   |   2 +-
 python/qemu/qmp/qmp_client.py                 |   2 +-
 python/qemu/utils/qemu_ga_client.py           |   6 +-
 python/setup.cfg                              |  11 +-
 python/tests/minreqs.txt                      |  45 +++
 scripts/qapi/mypy.ini                         |   2 +-
 tests/docker/dockerfiles/centos8.docker       |   1 +
 tests/docker/dockerfiles/opensuse-leap.docker |   1 +
 tests/docker/dockerfiles/python.docker        |   1 -
 tests/qemu-iotests/iotests.py                 |   4 +-
 .../tests/migrate-bitmaps-postcopy-test       |   2 +-
 20 files changed, 135 insertions(+), 416 deletions(-)
 delete mode 100644 python/Pipfile
 delete mode 100644 python/Pipfile.lock
 create mode 100644 python/tests/minreqs.txt

-- 
2.39.0

[PATCH v2 1/7] python: support pylint 2.16

[John Snow]

Pylint 2.16 adds a few new checks that cause the optional check-tox CI
job to fail.

1. The superfluous-parens check seems to be a bit more aggressive,
2. broad-exception-raised is new; it discourages "raise Exception".

Fix these minor issues and turn the lights green.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/qemu/qmp/protocol.py                            | 2 +-
 python/qemu/qmp/qmp_client.py                          | 2 +-
 python/qemu/utils/qemu_ga_client.py                    | 6 +++---
 tests/qemu-iotests/iotests.py                          | 4 ++--
 tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 2 +-
 5 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/python/qemu/qmp/protocol.py b/python/qemu/qmp/protocol.py
index 6d3d739daa7..22e60298d28 100644
--- a/python/qemu/qmp/protocol.py
+++ b/python/qemu/qmp/protocol.py
@@ -207,7 +207,7 @@ class AsyncProtocol(Generic[T]):
     logger = logging.getLogger(__name__)
 
     # Maximum allowable size of read buffer
-    _limit = (64 * 1024)
+    _limit = 64 * 1024
 
     # -------------------------
     # Section: Public interface
diff --git a/python/qemu/qmp/qmp_client.py b/python/qemu/qmp/qmp_client.py
index b5772e7f32b..9d73ae6e7ad 100644
--- a/python/qemu/qmp/qmp_client.py
+++ b/python/qemu/qmp/qmp_client.py
@@ -198,7 +198,7 @@ async def run(self, address='/tmp/qemu.socket'):
     logger = logging.getLogger(__name__)
 
     # Read buffer limit; 10MB like libvirt default
-    _limit = (10 * 1024 * 1024)
+    _limit = 10 * 1024 * 1024
 
     # Type alias for pending execute() result items
     _PendingT = Union[Message, ExecInterruptedError]
diff --git a/python/qemu/utils/qemu_ga_client.py b/python/qemu/utils/qemu_ga_client.py
index 8c38a7ac9c0..d8411bb2d0b 100644
--- a/python/qemu/utils/qemu_ga_client.py
+++ b/python/qemu/utils/qemu_ga_client.py
@@ -155,7 +155,7 @@ def ping(self, timeout: Optional[float]) -> bool:
 
     def fsfreeze(self, cmd: str) -> object:
         if cmd not in ['status', 'freeze', 'thaw']:
-            raise Exception('Invalid command: ' + cmd)
+            raise ValueError('Invalid command: ' + cmd)
         # Can be int (freeze, thaw) or GuestFsfreezeStatus (status)
         return getattr(self.qga, 'fsfreeze' + '_' + cmd)()
 
@@ -167,7 +167,7 @@ def fstrim(self, minimum: int) -> Dict[str, object]:
 
     def suspend(self, mode: str) -> None:
         if mode not in ['disk', 'ram', 'hybrid']:
-            raise Exception('Invalid mode: ' + mode)
+            raise ValueError('Invalid mode: ' + mode)
 
         try:
             getattr(self.qga, 'suspend' + '_' + mode)()
@@ -178,7 +178,7 @@ def suspend(self, mode: str) -> None:
 
     def shutdown(self, mode: str = 'powerdown') -> None:
         if mode not in ['powerdown', 'halt', 'reboot']:
-            raise Exception('Invalid mode: ' + mode)
+            raise ValueError('Invalid mode: ' + mode)
 
         try:
             self.qga.shutdown(mode=mode)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 94aeb3f3b20..3e82c634cfe 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -720,7 +720,7 @@ def __exit__(self, exc_type, value, traceback):
         signal.setitimer(signal.ITIMER_REAL, 0)
         return False
     def timeout(self, signum, frame):
-        raise Exception(self.errmsg)
+        raise TimeoutError(self.errmsg)
 
 def file_pattern(name):
     return "{0}-{1}".format(os.getpid(), name)
@@ -804,7 +804,7 @@ def remote_filename(path):
     elif imgproto == 'ssh':
         return "ssh://%s@127.0.0.1:22%s" % (os.environ.get('USER'), path)
     else:
-        raise Exception("Protocol %s not supported" % (imgproto))
+        raise ValueError("Protocol %s not supported" % (imgproto))
 
 class VM(qtest.QEMUQtestMachine):
     '''A QEMU VM'''
diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
index fc9c4b4ef41..dda55fad284 100755
--- a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
+++ b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
@@ -84,7 +84,7 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
                 e['vm'] = 'SRC'
             for e in self.vm_b_events:
                 e['vm'] = 'DST'
-            events = (self.vm_a_events + self.vm_b_events)
+            events = self.vm_a_events + self.vm_b_events
             events = [(e['timestamp']['seconds'],
                        e['timestamp']['microseconds'],
                        e['vm'],
-- 
2.39.0

[PATCH v2 2/7] python: drop pipenv

[John Snow]

The pipenv tool was nice in theory, but in practice it's just too hard
to update selectively, and it makes using it a pain. The qemu.qmp repo
dropped pipenv support a while back and it's been functioning just fine,
so I'm backporting that change here to qemu.git.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst                      |   3 -
 .gitlab-ci.d/static_checks.yml         |   4 +-
 python/.gitignore                      |   4 +-
 python/Makefile                        |  53 ++--
 python/Pipfile                         |  13 -
 python/Pipfile.lock                    | 347 -------------------------
 python/setup.cfg                       |   4 +-
 python/tests/minreqs.txt               |  45 ++++
 tests/docker/dockerfiles/python.docker |   1 -
 9 files changed, 86 insertions(+), 388 deletions(-)
 delete mode 100644 python/Pipfile
 delete mode 100644 python/Pipfile.lock
 create mode 100644 python/tests/minreqs.txt

diff --git a/python/README.rst b/python/README.rst
index 9c1fceaee73..d62e71528d2 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -77,9 +77,6 @@ Files in this directory
 - ``MANIFEST.in`` is read by python setuptools, it specifies additional files
   that should be included by a source distribution.
 - ``PACKAGE.rst`` is used as the README file that is visible on PyPI.org.
-- ``Pipfile`` is used by Pipenv to generate ``Pipfile.lock``.
-- ``Pipfile.lock`` is a set of pinned package dependencies that this package
-  is tested under in our CI suite. It is used by ``make check-pipenv``.
 - ``README.rst`` you are here!
 - ``VERSION`` contains the PEP-440 compliant version used to describe
   this package; it is referenced by ``setup.cfg``.
diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
index 289ad1359e3..b4cbdbce2ab 100644
--- a/.gitlab-ci.d/static_checks.yml
+++ b/.gitlab-ci.d/static_checks.yml
@@ -23,12 +23,12 @@ check-dco:
   before_script:
     - apk -U add git
 
-check-python-pipenv:
+check-python-minreqs:
   extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
-    - make -C python check-pipenv
+    - make -C python check-minreqs
   variables:
     GIT_DEPTH: 1
   needs:
diff --git a/python/.gitignore b/python/.gitignore
index 904f324bb11..c3ceb1ca0ab 100644
--- a/python/.gitignore
+++ b/python/.gitignore
@@ -11,8 +11,8 @@ qemu.egg-info/
 .idea/
 .vscode/
 
-# virtual environments (pipenv et al)
-.venv/
+# virtual environments
+.min-venv/
 .tox/
 .dev-venv/
 
diff --git a/python/Makefile b/python/Makefile
index b170708398a..c5bd6ff83ac 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -1,15 +1,16 @@
 QEMU_VENV_DIR=.dev-venv
+QEMU_MINVENV_DIR=.min-venv
 QEMU_TOX_EXTRA_ARGS ?=
 
 .PHONY: help
 help:
 	@echo "python packaging help:"
 	@echo ""
-	@echo "make check-pipenv:"
-	@echo "    Run tests in pipenv's virtual environment."
+	@echo "make check-minreqs:"
+	@echo "    Run tests in the minreqs virtual environment."
 	@echo "    These tests use the oldest dependencies."
-	@echo "    Requires: Python 3.6 and pipenv."
-	@echo "    Hint (Fedora): 'sudo dnf install python3.6 pipenv'"
+	@echo "    Requires: Python 3.6"
+	@echo "    Hint (Fedora): 'sudo dnf install python3.6'"
 	@echo ""
 	@echo "make check-tox:"
 	@echo "    Run tests against multiple python versions."
@@ -33,8 +34,8 @@ help:
 	@echo "    and install the qemu package in editable mode."
 	@echo "    (Can be used in or outside of a venv.)"
 	@echo ""
-	@echo "make pipenv"
-	@echo "    Creates pipenv's virtual environment (.venv)"
+	@echo "make min-venv"
+	@echo "    Creates the minreqs virtual environment ($(QEMU_MINVENV_DIR))"
 	@echo ""
 	@echo "make dev-venv"
 	@echo "    Creates a simple venv for check-dev. ($(QEMU_VENV_DIR))"
@@ -43,21 +44,38 @@ help:
 	@echo "    Remove package build output."
 	@echo ""
 	@echo "make distclean:"
-	@echo "    remove pipenv/venv files, qemu package forwarder,"
+	@echo "    remove venv files, qemu package forwarder,"
 	@echo "    built distribution files, and everything from 'make clean'."
 	@echo ""
 	@echo -e "Have a nice day ^_^\n"
 
-.PHONY: pipenv
-pipenv: .venv
-.venv: Pipfile.lock
-	@PIPENV_VENV_IN_PROJECT=1 pipenv sync --dev --keep-outdated
-	rm -f pyproject.toml
-	@touch .venv
+.PHONY: pipenv check-pipenv
+pipenv check-pipenv:
+	@echo "pipenv was dropped; try 'make check-minreqs' or 'make min-venv'"
+	@exit 1
 
-.PHONY: check-pipenv
-check-pipenv: pipenv
-	@pipenv run make check
+.PHONY: min-venv
+min-venv: $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate
+$(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate: setup.cfg tests/minreqs.txt
+	@echo "VENV $(QEMU_MINVENV_DIR)"
+	@python3.6 -m venv $(QEMU_MINVENV_DIR)
+	@(								\
+		echo "ACTIVATE $(QEMU_MINVENV_DIR)";			\
+		. $(QEMU_MINVENV_DIR)/bin/activate;			\
+		echo "INSTALL -r tests/minreqs.txt $(QEMU_MINVENV_DIR)";\
+		pip install -r tests/minreqs.txt 1>/dev/null;		\
+		echo "INSTALL -e qemu $(QEMU_MINVENV_DIR)";		\
+		pip install -e . 1>/dev/null;				\
+	)
+	@touch $(QEMU_MINVENV_DIR)
+
+.PHONY: check-minreqs
+check-minreqs: min-venv
+	@(							\
+		echo "ACTIVATE $(QEMU_MINVENV_DIR)";		\
+		. $(QEMU_MINVENV_DIR)/bin/activate;		\
+		make check;					\
+	)
 
 .PHONY: dev-venv
 dev-venv: $(QEMU_VENV_DIR) $(QEMU_VENV_DIR)/bin/activate
@@ -106,6 +124,7 @@ clean:
 
 .PHONY: distclean
 distclean: clean
-	rm -rf qemu.egg-info/ .venv/ .tox/ $(QEMU_VENV_DIR) dist/
+	rm -rf qemu.egg-info/ .eggs/ dist/
+	rm -rf $(QEMU_VENV_DIR) $(QEMU_MINVENV_DIR) .tox/
 	rm -f .coverage .coverage.*
 	rm -rf htmlcov/
diff --git a/python/Pipfile b/python/Pipfile
deleted file mode 100644
index e7acb8cefa4..00000000000
--- a/python/Pipfile
+++ /dev/null
@@ -1,13 +0,0 @@
-[[source]]
-name = "pypi"
-url = "https://pypi.org/simple"
-verify_ssl = true
-
-[dev-packages]
-qemu = {editable = true, extras = ["devel"], path = "."}
-
-[packages]
-qemu = {editable = true,path = "."}
-
-[requires]
-python_version = "3.6"
diff --git a/python/Pipfile.lock b/python/Pipfile.lock
deleted file mode 100644
index ce46404ce08..00000000000
--- a/python/Pipfile.lock
+++ /dev/null
@@ -1,347 +0,0 @@
-{
-    "_meta": {
-        "hash": {
-            "sha256": "f1a25654d884a5b450e38d78b1f2e3ebb9073e421cc4358d4bbb83ac251a5670"
-        },
-        "pipfile-spec": 6,
-        "requires": {
-            "python_version": "3.6"
-        },
-        "sources": [
-            {
-                "name": "pypi",
-                "url": "https://pypi.org/simple",
-                "verify_ssl": true
-            }
-        ]
-    },
-    "default": {
-        "qemu": {
-            "editable": true,
-            "path": "."
-        }
-    },
-    "develop": {
-        "appdirs": {
-            "hashes": [
-                "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41",
-                "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128"
-            ],
-            "version": "==1.4.4"
-        },
-        "astroid": {
-            "hashes": [
-                "sha256:09bdb456e02564731f8b5957cdd0c98a7f01d2db5e90eb1d794c353c28bfd705",
-                "sha256:6a8a51f64dae307f6e0c9db752b66a7951e282389d8362cc1d39a56f3feeb31d"
-            ],
-            "index": "pypi",
-            "version": "==2.6.0"
-        },
-        "avocado-framework": {
-            "hashes": [
-                "sha256:244cb569f8eb4e50a22ac82e1a2b2bba2458999f4281efbe2651bd415d59c65b",
-                "sha256:6f15998b67ecd0e7dde790c4de4dd249d6df52dfe6d5cc4e2dd6596df51c3583"
-            ],
-            "index": "pypi",
-            "version": "==90.0"
-        },
-        "distlib": {
-            "hashes": [
-                "sha256:106fef6dc37dd8c0e2c0a60d3fca3e77460a48907f335fa28420463a6f799736",
-                "sha256:23e223426b28491b1ced97dc3bbe183027419dfc7982b4fa2f05d5f3ff10711c"
-            ],
-            "index": "pypi",
-            "version": "==0.3.2"
-        },
-        "filelock": {
-            "hashes": [
-                "sha256:18d82244ee114f543149c66a6e0c14e9c4f8a1044b5cdaadd0f82159d6a6ff59",
-                "sha256:929b7d63ec5b7d6b71b0fa5ac14e030b3f70b75747cef1b10da9b879fef15836"
-            ],
-            "index": "pypi",
-            "version": "==3.0.12"
-        },
-        "flake8": {
-            "hashes": [
-                "sha256:6a35f5b8761f45c5513e3405f110a86bea57982c3b75b766ce7b65217abe1670",
-                "sha256:c01f8a3963b3571a8e6bd7a4063359aff90749e160778e03817cd9b71c9e07d2"
-            ],
-            "index": "pypi",
-            "version": "==3.6.0"
-        },
-        "fusepy": {
-            "hashes": [
-                "sha256:10f5c7f5414241bffecdc333c4d3a725f1d6605cae6b4eaf86a838ff49cdaf6c",
-                "sha256:a9f3a3699080ddcf0919fd1eb2cf743e1f5859ca54c2018632f939bdfac269ee"
-            ],
-            "index": "pypi",
-            "version": "==2.0.4"
-        },
-        "importlib-metadata": {
-            "hashes": [
-                "sha256:90bb658cdbbf6d1735b6341ce708fc7024a3e14e99ffdc5783edea9f9b077f83",
-                "sha256:dc15b2969b4ce36305c51eebe62d418ac7791e9a157911d58bfb1f9ccd8e2070"
-            ],
-            "markers": "python_version < '3.8'",
-            "version": "==1.7.0"
-        },
-        "importlib-resources": {
-            "hashes": [
-                "sha256:54161657e8ffc76596c4ede7080ca68cb02962a2e074a2586b695a93a925d36e",
-                "sha256:e962bff7440364183203d179d7ae9ad90cb1f2b74dcb84300e88ecc42dca3351"
-            ],
-            "index": "pypi",
-            "version": "==5.1.4"
-        },
-        "isort": {
-            "hashes": [
-                "sha256:408e4d75d84f51b64d0824894afee44469eba34a4caee621dc53799f80d71ccc",
-                "sha256:64022dea6a06badfa09b300b4dfe8ba968114a737919e8ed50aea1c288f078aa"
-            ],
-            "index": "pypi",
-            "version": "==5.1.2"
-        },
-        "lazy-object-proxy": {
-            "hashes": [
-                "sha256:17e0967ba374fc24141738c69736da90e94419338fd4c7c7bef01ee26b339653",
-                "sha256:1fee665d2638491f4d6e55bd483e15ef21f6c8c2095f235fef72601021e64f61",
-                "sha256:22ddd618cefe54305df49e4c069fa65715be4ad0e78e8d252a33debf00f6ede2",
-                "sha256:24a5045889cc2729033b3e604d496c2b6f588c754f7a62027ad4437a7ecc4837",
-                "sha256:410283732af311b51b837894fa2f24f2c0039aa7f220135192b38fcc42bd43d3",
-                "sha256:4732c765372bd78a2d6b2150a6e99d00a78ec963375f236979c0626b97ed8e43",
-                "sha256:489000d368377571c6f982fba6497f2aa13c6d1facc40660963da62f5c379726",
-                "sha256:4f60460e9f1eb632584c9685bccea152f4ac2130e299784dbaf9fae9f49891b3",
-                "sha256:5743a5ab42ae40caa8421b320ebf3a998f89c85cdc8376d6b2e00bd12bd1b587",
-                "sha256:85fb7608121fd5621cc4377a8961d0b32ccf84a7285b4f1d21988b2eae2868e8",
-                "sha256:9698110e36e2df951c7c36b6729e96429c9c32b3331989ef19976592c5f3c77a",
-                "sha256:9d397bf41caad3f489e10774667310d73cb9c4258e9aed94b9ec734b34b495fd",
-                "sha256:b579f8acbf2bdd9ea200b1d5dea36abd93cabf56cf626ab9c744a432e15c815f",
-                "sha256:b865b01a2e7f96db0c5d12cfea590f98d8c5ba64ad222300d93ce6ff9138bcad",
-                "sha256:bf34e368e8dd976423396555078def5cfc3039ebc6fc06d1ae2c5a65eebbcde4",
-                "sha256:c6938967f8528b3668622a9ed3b31d145fab161a32f5891ea7b84f6b790be05b",
-                "sha256:d1c2676e3d840852a2de7c7d5d76407c772927addff8d742b9808fe0afccebdf",
-                "sha256:d7124f52f3bd259f510651450e18e0fd081ed82f3c08541dffc7b94b883aa981",
-                "sha256:d900d949b707778696fdf01036f58c9876a0d8bfe116e8d220cfd4b15f14e741",
-                "sha256:ebfd274dcd5133e0afae738e6d9da4323c3eb021b3e13052d8cbd0e457b1256e",
-                "sha256:ed361bb83436f117f9917d282a456f9e5009ea12fd6de8742d1a4752c3017e93",
-                "sha256:f5144c75445ae3ca2057faac03fda5a902eff196702b0a24daf1d6ce0650514b"
-            ],
-            "index": "pypi",
-            "version": "==1.6.0"
-        },
-        "mccabe": {
-            "hashes": [
-                "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
-                "sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
-            ],
-            "version": "==0.6.1"
-        },
-        "mypy": {
-            "hashes": [
-                "sha256:00cb1964a7476e871d6108341ac9c1a857d6bd20bf5877f4773ac5e9d92cd3cd",
-                "sha256:127de5a9b817a03a98c5ae8a0c46a20dc44442af6dcfa2ae7f96cb519b312efa",
-                "sha256:1f3976a945ad7f0a0727aafdc5651c2d3278e3c88dee94e2bf75cd3386b7b2f4",
-                "sha256:2f8c098f12b402c19b735aec724cc9105cc1a9eea405d08814eb4b14a6fb1a41",
-                "sha256:4ef13b619a289aa025f2273e05e755f8049bb4eaba6d703a425de37d495d178d",
-                "sha256:5d142f219bf8c7894dfa79ebfb7d352c4c63a325e75f10dfb4c3db9417dcd135",
-                "sha256:62eb5dd4ea86bda8ce386f26684f7f26e4bfe6283c9f2b6ca6d17faf704dcfad",
-                "sha256:64c36eb0936d0bfb7d8da49f92c18e312ad2e3ed46e5548ae4ca997b0d33bd59",
-                "sha256:75eed74d2faf2759f79c5f56f17388defd2fc994222312ec54ee921e37b31ad4",
-                "sha256:974bebe3699b9b46278a7f076635d219183da26e1a675c1f8243a69221758273",
-                "sha256:a5e5bb12b7982b179af513dddb06fca12285f0316d74f3964078acbfcf4c68f2",
-                "sha256:d31291df31bafb997952dc0a17ebb2737f802c754aed31dd155a8bfe75112c57",
-                "sha256:d3b4941de44341227ece1caaf5b08b23e42ad4eeb8b603219afb11e9d4cfb437",
-                "sha256:eadb865126da4e3c4c95bdb47fe1bb087a3e3ea14d39a3b13224b8a4d9f9a102"
-            ],
-            "index": "pypi",
-            "version": "==0.780"
-        },
-        "mypy-extensions": {
-            "hashes": [
-                "sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d",
-                "sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"
-            ],
-            "version": "==0.4.3"
-        },
-        "packaging": {
-            "hashes": [
-                "sha256:5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5",
-                "sha256:67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"
-            ],
-            "index": "pypi",
-            "version": "==20.9"
-        },
-        "pluggy": {
-            "hashes": [
-                "sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0",
-                "sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"
-            ],
-            "index": "pypi",
-            "version": "==0.13.1"
-        },
-        "py": {
-            "hashes": [
-                "sha256:21b81bda15b66ef5e1a777a21c4dcd9c20ad3efd0b3f817e7a809035269e1bd3",
-                "sha256:3b80836aa6d1feeaa108e046da6423ab8f6ceda6468545ae8d02d9d58d18818a"
-            ],
-            "index": "pypi",
-            "version": "==1.10.0"
-        },
-        "pycodestyle": {
-            "hashes": [
-                "sha256:74abc4e221d393ea5ce1f129ea6903209940c1ecd29e002e8c6933c2b21026e0",
-                "sha256:cbc619d09254895b0d12c2c691e237b2e91e9b2ecf5e84c26b35400f93dcfb83",
-                "sha256:cbfca99bd594a10f674d0cd97a3d802a1fdef635d4361e1a2658de47ed261e3a"
-            ],
-            "version": "==2.4.0"
-        },
-        "pyflakes": {
-            "hashes": [
-                "sha256:9a7662ec724d0120012f6e29d6248ae3727d821bba522a0e6b356eff19126a49",
-                "sha256:f661252913bc1dbe7fcfcbf0af0db3f42ab65aabd1a6ca68fe5d466bace94dae"
-            ],
-            "version": "==2.0.0"
-        },
-        "pygments": {
-            "hashes": [
-                "sha256:a18f47b506a429f6f4b9df81bb02beab9ca21d0a5fee38ed15aef65f0545519f",
-                "sha256:d66e804411278594d764fc69ec36ec13d9ae9147193a1740cd34d272ca383b8e"
-            ],
-            "index": "pypi",
-            "version": "==2.9.0"
-        },
-        "pylint": {
-            "hashes": [
-                "sha256:082a6d461b54f90eea49ca90fff4ee8b6e45e8029e5dbd72f6107ef84f3779c0",
-                "sha256:a01cd675eccf6e25b3bdb42be184eb46aaf89187d612ba0fb5f93328ed6b0fd5"
-            ],
-            "index": "pypi",
-            "version": "==2.8.0"
-        },
-        "pyparsing": {
-            "hashes": [
-                "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
-                "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
-            ],
-            "index": "pypi",
-            "version": "==2.4.7"
-        },
-        "qemu": {
-            "editable": true,
-            "path": "."
-        },
-        "setuptools": {
-            "hashes": [
-                "sha256:22c7348c6d2976a52632c67f7ab0cdf40147db7789f9aed18734643fe9cf3373",
-                "sha256:4ce92f1e1f8f01233ee9952c04f6b81d1e02939d6e1b488428154974a4d0783e"
-            ],
-            "markers": "python_version >= '3.6'",
-            "version": "==59.6.0"
-        },
-        "six": {
-            "hashes": [
-                "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
-                "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
-            ],
-            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
-            "version": "==1.16.0"
-        },
-        "toml": {
-            "hashes": [
-                "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b",
-                "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"
-            ],
-            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
-            "version": "==0.10.2"
-        },
-        "tox": {
-            "hashes": [
-                "sha256:c60692d92fe759f46c610ac04c03cf0169432d1ff8e981e8ae63e068d0954fc3",
-                "sha256:f179cb4043d7dc1339425dd49ab1dd8c916246b0d9173143c1b0af7498a03ab0"
-            ],
-            "index": "pypi",
-            "version": "==3.18.0"
-        },
-        "typed-ast": {
-            "hashes": [
-                "sha256:01ae5f73431d21eead5015997ab41afa53aa1fbe252f9da060be5dad2c730ace",
-                "sha256:067a74454df670dcaa4e59349a2e5c81e567d8d65458d480a5b3dfecec08c5ff",
-                "sha256:0fb71b8c643187d7492c1f8352f2c15b4c4af3f6338f21681d3681b3dc31a266",
-                "sha256:1b3ead4a96c9101bef08f9f7d1217c096f31667617b58de957f690c92378b528",
-                "sha256:2068531575a125b87a41802130fa7e29f26c09a2833fea68d9a40cf33902eba6",
-                "sha256:209596a4ec71d990d71d5e0d312ac935d86930e6eecff6ccc7007fe54d703808",
-                "sha256:2c726c276d09fc5c414693a2de063f521052d9ea7c240ce553316f70656c84d4",
-                "sha256:398e44cd480f4d2b7ee8d98385ca104e35c81525dd98c519acff1b79bdaac363",
-                "sha256:52b1eb8c83f178ab787f3a4283f68258525f8d70f778a2f6dd54d3b5e5fb4341",
-                "sha256:5feca99c17af94057417d744607b82dd0a664fd5e4ca98061480fd8b14b18d04",
-                "sha256:7538e495704e2ccda9b234b82423a4038f324f3a10c43bc088a1636180f11a41",
-                "sha256:760ad187b1041a154f0e4d0f6aae3e40fdb51d6de16e5c99aedadd9246450e9e",
-                "sha256:777a26c84bea6cd934422ac2e3b78863a37017618b6e5c08f92ef69853e765d3",
-                "sha256:95431a26309a21874005845c21118c83991c63ea800dd44843e42a916aec5899",
-                "sha256:9ad2c92ec681e02baf81fdfa056fe0d818645efa9af1f1cd5fd6f1bd2bdfd805",
-                "sha256:9c6d1a54552b5330bc657b7ef0eae25d00ba7ffe85d9ea8ae6540d2197a3788c",
-                "sha256:aee0c1256be6c07bd3e1263ff920c325b59849dc95392a05f258bb9b259cf39c",
-                "sha256:af3d4a73793725138d6b334d9d247ce7e5f084d96284ed23f22ee626a7b88e39",
-                "sha256:b36b4f3920103a25e1d5d024d155c504080959582b928e91cb608a65c3a49e1a",
-                "sha256:b9574c6f03f685070d859e75c7f9eeca02d6933273b5e69572e5ff9d5e3931c3",
-                "sha256:bff6ad71c81b3bba8fa35f0f1921fb24ff4476235a6e94a26ada2e54370e6da7",
-                "sha256:c190f0899e9f9f8b6b7863debfb739abcb21a5c054f911ca3596d12b8a4c4c7f",
-                "sha256:c907f561b1e83e93fad565bac5ba9c22d96a54e7ea0267c708bffe863cbe4075",
-                "sha256:cae53c389825d3b46fb37538441f75d6aecc4174f615d048321b716df2757fb0",
-                "sha256:dd4a21253f42b8d2b48410cb31fe501d32f8b9fbeb1f55063ad102fe9c425e40",
-                "sha256:dde816ca9dac1d9c01dd504ea5967821606f02e510438120091b84e852367428",
-                "sha256:f2362f3cb0f3172c42938946dbc5b7843c2a28aec307c49100c8b38764eb6927",
-                "sha256:f328adcfebed9f11301eaedfa48e15bdece9b519fb27e6a8c01aa52a17ec31b3",
-                "sha256:f8afcf15cc511ada719a88e013cec87c11aff7b91f019295eb4530f96fe5ef2f",
-                "sha256:fb1bbeac803adea29cedd70781399c99138358c26d05fcbd23c13016b7f5ec65"
-            ],
-            "markers": "python_version < '3.8' and implementation_name == 'cpython'",
-            "version": "==1.4.3"
-        },
-        "typing-extensions": {
-            "hashes": [
-                "sha256:0ac0f89795dd19de6b97debb0c6af1c70987fd80a2d62d1958f7e56fcc31b497",
-                "sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342",
-                "sha256:779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84"
-            ],
-            "index": "pypi",
-            "version": "==3.10.0.0"
-        },
-        "urwid": {
-            "hashes": [
-                "sha256:588bee9c1cb208d0906a9f73c613d2bd32c3ed3702012f51efe318a3f2127eae"
-            ],
-            "index": "pypi",
-            "version": "==2.1.2"
-        },
-        "urwid-readline": {
-            "hashes": [
-                "sha256:018020cbc864bb5ed87be17dc26b069eae2755cb29f3a9c569aac3bded1efaf4"
-            ],
-            "index": "pypi",
-            "version": "==0.13"
-        },
-        "virtualenv": {
-            "hashes": [
-                "sha256:14fdf849f80dbb29a4eb6caa9875d476ee2a5cf76a5f5415fa2f1606010ab467",
-                "sha256:2b0126166ea7c9c3661f5b8e06773d28f83322de7a3ff7d06f0aed18c9de6a76"
-            ],
-            "index": "pypi",
-            "version": "==20.4.7"
-        },
-        "wrapt": {
-            "hashes": [
-                "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
-            ],
-            "version": "==1.12.1"
-        },
-        "zipp": {
-            "hashes": [
-                "sha256:3607921face881ba3e026887d8150cca609d517579abe052ac81fc5aeffdbd76",
-                "sha256:51cb66cc54621609dd593d1787f286ee42a5c0adbb4b29abea5a63edc3e03098"
-            ],
-            "index": "pypi",
-            "version": "==3.4.1"
-        }
-    }
-}
diff --git a/python/setup.cfg b/python/setup.cfg
index 56418157065..9e923d97628 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -33,9 +33,7 @@ packages =
 * = py.typed
 
 [options.extras_require]
-# For the devel group, When adding new dependencies or bumping the minimum
-# version, use e.g. "pipenv install --dev pylint==3.0.0".
-# Subsequently, edit 'Pipfile' to remove e.g. 'pylint = "==3.0.0'.
+# Remember to update tests/minreqs.txt if changing anything below:
 devel =
     avocado-framework >= 90.0
     flake8 >= 3.6.0
diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
new file mode 100644
index 00000000000..dfb8abb155f
--- /dev/null
+++ b/python/tests/minreqs.txt
@@ -0,0 +1,45 @@
+# This file lists the ***oldest possible dependencies*** needed to run
+# "make check" successfully under ***Python 3.6***. It is used primarily
+# by GitLab CI to ensure that our stated minimum versions in setup.cfg
+# are truthful and regularly validated.
+#
+# This file should not contain any dependencies that are not expressed
+# by the [devel] section of setup.cfg, except for transitive
+# dependencies which must be enumerated here explicitly to eliminate
+# dependency resolution ambiguity.
+#
+# When adding new dependencies, pin the very oldest non-yanked version
+# on PyPI that allows the test suite to pass.
+
+# Dependencies for the TUI addon (Required for successful linting)
+urwid==2.1.2
+urwid-readline==0.13
+Pygments==2.9.0
+
+# Dependencies for FUSE support for qom-fuse
+fusepy==2.0.4
+
+# Test-runners, utilities, etc.
+avocado-framework==90.0
+
+# Linters
+flake8==3.6.0
+isort==5.1.2
+mypy==0.780
+pylint==2.8.0
+
+# Transitive flake8 dependencies
+mccabe==0.6.0
+pycodestyle==2.4.0
+pyflakes==2.0.0
+
+# Transitive mypy dependencies
+mypy-extensions==0.4.3
+typed-ast==1.4.0
+typing-extensions==3.7.4
+
+# Transitive pylint dependencies
+astroid==2.5.4
+lazy-object-proxy==1.4.0
+toml==0.10.0
+wrapt==1.12.1
diff --git a/tests/docker/dockerfiles/python.docker b/tests/docker/dockerfiles/python.docker
index 56d88417df4..175c10a34e8 100644
--- a/tests/docker/dockerfiles/python.docker
+++ b/tests/docker/dockerfiles/python.docker
@@ -7,7 +7,6 @@ MAINTAINER John Snow <jsnow@redhat.com>
 ENV PACKAGES \
     gcc \
     make \
-    pipenv \
     python3 \
     python3-pip \
     python3-tox \
-- 
2.39.0

[PATCH v2 3/7] configure: Look for auxiliary Python installations

[John Snow]

At the moment, we look for just "python3" and "python", which is good
enough almost all of the time. But ... if you are on a platform that
uses an older Python by default and only offers a newer Python as an
option, you'll have to specify --python=/usr/bin/foo every time.

We can be kind and instead make a cursory attempt to locate a suitable
Python binary ourselves, looking for the remaining well-known binaries.

This configure loop will prefer, in order:

1. Whatever is specified in $PYTHON
2. python3
3. python
4. python3.11 down through python3.6

Notes:

- Python virtual environment provides binaries for "python3", "python",
  and whichever version you used to create the venv,
  e.g. "python3.8". If configure is invoked from inside of a venv, this
  configure loop will not "break out" of that venv unless that venv is
  created using an explicitly non-suitable version of Python that we
  cannot use.

- In the event that no suitable python is found, the first python found
  is the version used to generate the human-readable error message.

- The error message isn't printed right away to allow later
  configuration code to pick up an explicitly configured python.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 33 +++++++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 8 deletions(-)

diff --git a/configure b/configure
index 64960c6000f..ea8c973d13b 100755
--- a/configure
+++ b/configure
@@ -592,20 +592,39 @@ esac
 
 : ${make=${MAKE-make}}
 
-# We prefer python 3.x. A bare 'python' is traditionally
-# python 2.x, but some distros have it as python 3.x, so
-# we check that too
+
+check_py_version() {
+    # We require python >= 3.6.
+    # NB: a True python conditional creates a non-zero return code (Failure)
+    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
+}
+
 python=
+first_python=
 explicit_python=no
-for binary in "${PYTHON-python3}" python
+# A bare 'python' is traditionally python 2.x, but some distros
+# have it as python 3.x, so check in both places.
+for binary in "${PYTHON-python3}" python python3.{11..6}
 do
     if has "$binary"
     then
         python=$(command -v "$binary")
-        break
+        if test -z "$first_python"; then
+           first_python=$python
+        fi
+        if check_py_version "$python"; then
+            # This one is good.
+            first_python=
+            break
+        fi
     fi
 done
 
+# If first_python is set, we didn't find a suitable binary.
+# Use this one for possible future error messages.
+if test -n "$first_python"; then
+    python="$first_python"
+fi
 
 # Check for ancillary tools used in testing
 genisoimage=
@@ -1037,9 +1056,7 @@ then
     error_exit "GNU make ($make) not found"
 fi
 
-# Note that if the Python conditional here evaluates True we will exit
-# with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
+if ! check_py_version "$python"; then
   error_exit "Cannot use '$python', Python >= 3.6 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
-- 
2.39.0

Re: [PATCH v2 3/7] configure: Look for auxiliary Python installations

[Thomas Huth]

On 10/02/2023 01.31, John Snow wrote:
> At the moment, we look for just "python3" and "python", which is good
> enough almost all of the time. But ... if you are on a platform that
> uses an older Python by default and only offers a newer Python as an
> option, you'll have to specify --python=/usr/bin/foo every time.
> 
> We can be kind and instead make a cursory attempt to locate a suitable
> Python binary ourselves, looking for the remaining well-known binaries.
> 
> This configure loop will prefer, in order:
> 
> 1. Whatever is specified in $PYTHON
> 2. python3
> 3. python
> 4. python3.11 down through python3.6
> 
> Notes:
> 
> - Python virtual environment provides binaries for "python3", "python",
>    and whichever version you used to create the venv,
>    e.g. "python3.8". If configure is invoked from inside of a venv, this
>    configure loop will not "break out" of that venv unless that venv is
>    created using an explicitly non-suitable version of Python that we
>    cannot use.
> 
> - In the event that no suitable python is found, the first python found
>    is the version used to generate the human-readable error message.
> 
> - The error message isn't printed right away to allow later
>    configuration code to pick up an explicitly configured python.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   configure | 33 +++++++++++++++++++++++++--------
>   1 file changed, 25 insertions(+), 8 deletions(-)
> 
> diff --git a/configure b/configure
> index 64960c6000f..ea8c973d13b 100755
> --- a/configure
> +++ b/configure
> @@ -592,20 +592,39 @@ esac
>   
>   : ${make=${MAKE-make}}
>   
> -# We prefer python 3.x. A bare 'python' is traditionally
> -# python 2.x, but some distros have it as python 3.x, so
> -# we check that too
> +
> +check_py_version() {
> +    # We require python >= 3.6.
> +    # NB: a True python conditional creates a non-zero return code (Failure)
> +    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
> +}
> +
>   python=
> +first_python=
>   explicit_python=no
> -for binary in "${PYTHON-python3}" python
> +# A bare 'python' is traditionally python 2.x, but some distros
> +# have it as python 3.x, so check in both places.
> +for binary in "${PYTHON-python3}" python python3.{11..6}
>   do
>       if has "$binary"
>       then
>           python=$(command -v "$binary")
> -        break
> +        if test -z "$first_python"; then
> +           first_python=$python
> +        fi
> +        if check_py_version "$python"; then
> +            # This one is good.
> +            first_python=
> +            break
> +        fi
>       fi
>   done
>   
> +# If first_python is set, we didn't find a suitable binary.
> +# Use this one for possible future error messages.
> +if test -n "$first_python"; then
> +    python="$first_python"
> +fi

>   # Check for ancillary tools used in testing
>   genisoimage=
> @@ -1037,9 +1056,7 @@ then
>       error_exit "GNU make ($make) not found"
>   fi
>   
> -# Note that if the Python conditional here evaluates True we will exit
> -# with status 1 which is a shell 'false' value.
> -if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
> +if ! check_py_version "$python"; then
>     error_exit "Cannot use '$python', Python >= 3.6 is required." \
>         "Use --python=/path/to/python to specify a supported Python."
>   fi

I like the idea!

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v2 3/7] configure: Look for auxiliary Python installations

[Paolo Bonzini]

On 2/10/23 01:31, John Snow wrote:
> At the moment, we look for just "python3" and "python", which is good
> enough almost all of the time. But ... if you are on a platform that
> uses an older Python by default and only offers a newer Python as an
> option, you'll have to specify --python=/usr/bin/foo every time.
> 
> We can be kind and instead make a cursory attempt to locate a suitable
> Python binary ourselves, looking for the remaining well-known binaries.
> 
> This configure loop will prefer, in order:
> 
> 1. Whatever is specified in $PYTHON
> 2. python3
> 3. python
> 4. python3.11 down through python3.6
> 
> Notes:
> 
> - Python virtual environment provides binaries for "python3", "python",
>    and whichever version you used to create the venv,
>    e.g. "python3.8". If configure is invoked from inside of a venv, this
>    configure loop will not "break out" of that venv unless that venv is
>    created using an explicitly non-suitable version of Python that we
>    cannot use.
> 
> - In the event that no suitable python is found, the first python found
>    is the version used to generate the human-readable error message.
> 
> - The error message isn't printed right away to allow later
>    configuration code to pick up an explicitly configured python.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   configure | 33 +++++++++++++++++++++++++--------
>   1 file changed, 25 insertions(+), 8 deletions(-)
> 
> diff --git a/configure b/configure
> index 64960c6000f..ea8c973d13b 100755
> --- a/configure
> +++ b/configure
> @@ -592,20 +592,39 @@ esac
>   
>   : ${make=${MAKE-make}}
>   
> -# We prefer python 3.x. A bare 'python' is traditionally
> -# python 2.x, but some distros have it as python 3.x, so
> -# we check that too
> +
> +check_py_version() {
> +    # We require python >= 3.6.
> +    # NB: a True python conditional creates a non-zero return code (Failure)
> +    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
> +}
> +
>   python=
> +first_python=
>   explicit_python=no
> -for binary in "${PYTHON-python3}" python
> +# A bare 'python' is traditionally python 2.x, but some distros
> +# have it as python 3.x, so check in both places.
> +for binary in "${PYTHON-python3}" python python3.{11..6}

This is not available in e.g. dash, so we need to use {11,10,9,8,7,6}.
Just a nit, I can fix it myself.

Paolo

Re: [PATCH v2 3/7] configure: Look for auxiliary Python installations

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 3289 bytes --]

On Fri, Feb 10, 2023, 5:46 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 2/10/23 01:31, John Snow wrote:
> > At the moment, we look for just "python3" and "python", which is good
> > enough almost all of the time. But ... if you are on a platform that
> > uses an older Python by default and only offers a newer Python as an
> > option, you'll have to specify --python=/usr/bin/foo every time.
> >
> > We can be kind and instead make a cursory attempt to locate a suitable
> > Python binary ourselves, looking for the remaining well-known binaries.
> >
> > This configure loop will prefer, in order:
> >
> > 1. Whatever is specified in $PYTHON
> > 2. python3
> > 3. python
> > 4. python3.11 down through python3.6
> >
> > Notes:
> >
> > - Python virtual environment provides binaries for "python3", "python",
> >    and whichever version you used to create the venv,
> >    e.g. "python3.8". If configure is invoked from inside of a venv, this
> >    configure loop will not "break out" of that venv unless that venv is
> >    created using an explicitly non-suitable version of Python that we
> >    cannot use.
> >
> > - In the event that no suitable python is found, the first python found
> >    is the version used to generate the human-readable error message.
> >
> > - The error message isn't printed right away to allow later
> >    configuration code to pick up an explicitly configured python.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >   configure | 33 +++++++++++++++++++++++++--------
> >   1 file changed, 25 insertions(+), 8 deletions(-)
> >
> > diff --git a/configure b/configure
> > index 64960c6000f..ea8c973d13b 100755
> > --- a/configure
> > +++ b/configure
> > @@ -592,20 +592,39 @@ esac
> >
> >   : ${make=${MAKE-make}}
> >
> > -# We prefer python 3.x. A bare 'python' is traditionally
> > -# python 2.x, but some distros have it as python 3.x, so
> > -# we check that too
> > +
> > +check_py_version() {
> > +    # We require python >= 3.6.
> > +    # NB: a True python conditional creates a non-zero return code
> (Failure)
> > +    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
> > +}
> > +
> >   python=
> > +first_python=
> >   explicit_python=no
> > -for binary in "${PYTHON-python3}" python
> > +# A bare 'python' is traditionally python 2.x, but some distros
> > +# have it as python 3.x, so check in both places.
> > +for binary in "${PYTHON-python3}" python python3.{11..6}
>
> This is not available in e.g. dash, so we need to use {11,10,9,8,7,6}.
> Just a nit, I can fix it myself.
>

What platforms use dash by default?

Did I not see a failure because nothing that uses dash iterated that far
down in the list?

Anyway, you've got my blessing to change it, of course.


> Paolo
>

PS, while you're here, how does this new loop interfere with your "custom
python specified" flag for meson? I think meson uses the version of python
*it* detects and not the configure script identified one, right? Does that
mean that e.g. the qapi generator gets run with the system default/meson
version and not the config version?

Do I need to adjust this loop to consider more binaries as "explicitly
specified"?

(PPS: English needs "paraphrasing quotes" as punctuation. Maybe like double
quotes except they're wiggly.)

--js

>

[-- Attachment #2: Type: text/html, Size: 4962 bytes --]

Re: [PATCH v2 3/7] configure: Look for auxiliary Python installations

[Peter Maydell]

On Fri, 10 Feb 2023 at 15:28, John Snow <jsnow@redhat.com> wrote:
>
>
>
> On Fri, Feb 10, 2023, 5:46 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>> On 2/10/23 01:31, John Snow wrote:
>> > +for binary in "${PYTHON-python3}" python python3.{11..6}
>>
>> This is not available in e.g. dash, so we need to use {11,10,9,8,7,6}.
>> Just a nit, I can fix it myself.
>
>
> What platforms use dash by default?

Ubuntu and Debian default to /bin/sh being dash. The BSDs will
also have some non-bash /bin/sh I expect.

shellcheck will catch this non-portability:

$ cat /tmp/foo.sh
#!/bin/sh

echo {11..6}
$ shellcheck /tmp/foo.sh

In /tmp/foo.sh line 3:
echo {11..6}
     ^-----^ SC3009 (warning): In POSIX sh, brace expansion is undefined.

For more information:
  https://www.shellcheck.net/wiki/SC3009 -- In POSIX sh, brace expansion is u...


though if you try it on configure right now there are a ton
of unaddressed warnings (some of which are harmless and some
of which are probably technically portability bugs) so it's
a bit tricky to use to sanity-check new changes.

thanks
-- PMM

Re: [PATCH v2 3/7] configure: Look for auxiliary Python installations

[Paolo Bonzini]

On Fri, Feb 10, 2023 at 4:28 PM John Snow <jsnow@redhat.com> wrote:
> PS, while you're here, how does this new loop interfere with your "custom python specified" flag for meson? I think meson uses the version of python *it* detects and not the configure script identified one, right? Does that mean that e.g. the qapi generator gets run with the system default/meson version and not the config version?

Yes, if neither --python nor --meson are specified, then it could
happen that a different python is used during ninja's execution vs.
what is used for "other stuff" (docker cross compilers and other
Makefile invocations of $(PYTHON)).

The meson version of Python is guaranteed to be at least 3.7 as soon
as we update to 0.63.x (which will be Real Soon Now), but it's ugly.
The main issue I anticipate could be a problem when running from a
virtual environment, so perhaps we can force usage of the internal
meson if neither --python nor --meson are specified, and VIRTUAL_ENV
is set and $VIRTUAL_ENV/bin/meson does not exist?

diff --git a/configure b/configure
index 06bcd9031903..001a79a90170 100755
--- a/configure
+++ b/configure
@@ -870,8 +870,18 @@ fi
 # Suppress writing compiled files
 python="$python -B"

+has_meson() {
+  if test "${VIRTUAL_ENV:+set}" = set; then
+    # Ensure that Meson and Python come from the same virtual environment
+    test -x "$(VIRTUAL_ENV}/bin/meson" &&
+      test "$(command -v meson)" -ef "$(VIRTUAL_ENV}/bin/meson"
+  else
+    has meson
+  fi
+}
+
 if test -z "$meson"; then
-    if test "$explicit_python" = no && has meson && version_ge
"$(meson --version)" 0.63.0; then
+    if test "$explicit_python" = no && has_meson && version_ge
"$(meson --version)" 0.63.0; then
         meson=meson
     elif test "$git_submodules_action" != 'ignore' ; then
         meson=git

I will include it when posting the final series.

> Do I need to adjust this loop to consider more binaries as "explicitly specified"?

I don't think it's a huge problem. Outside virtual environments, the
most likely setting is that Meson uses python3 which in turn is the
most recent python3.X, so it should be fine overall.

Though part of me thinks that your new loop is slightly overengineered
and we should just require /usr/bin/env python3 and call it a day.

Paolo

Re: [PATCH v2 3/7] configure: Look for auxiliary Python installations

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2579 bytes --]

On Fri, Feb 10, 2023, 11:17 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On Fri, Feb 10, 2023 at 4:28 PM John Snow <jsnow@redhat.com> wrote:
> > PS, while you're here, how does this new loop interfere with your
> "custom python specified" flag for meson? I think meson uses the version of
> python *it* detects and not the configure script identified one, right?
> Does that mean that e.g. the qapi generator gets run with the system
> default/meson version and not the config version?
>
> Yes, if neither --python nor --meson are specified, then it could
> happen that a different python is used during ninja's execution vs.
> what is used for "other stuff" (docker cross compilers and other
> Makefile invocations of $(PYTHON)).
>
> The meson version of Python is guaranteed to be at least 3.7 as soon
> as we update to 0.63.x (which will be Real Soon Now), but it's ugly.
> The main issue I anticipate could be a problem when running from a
> virtual environment, so perhaps we can force usage of the internal
> meson if neither --python nor --meson are specified, and VIRTUAL_ENV
> is set and $VIRTUAL_ENV/bin/meson does not exist?
>
> diff --git a/configure b/configure
> index 06bcd9031903..001a79a90170 100755
> --- a/configure
> +++ b/configure
> @@ -870,8 +870,18 @@ fi
>  # Suppress writing compiled files
>  python="$python -B"
>
> +has_meson() {
> +  if test "${VIRTUAL_ENV:+set}" = set; then
> +    # Ensure that Meson and Python come from the same virtual environment
> +    test -x "$(VIRTUAL_ENV}/bin/meson" &&
> +      test "$(command -v meson)" -ef "$(VIRTUAL_ENV}/bin/meson"
> +  else
> +    has meson
> +  fi
> +}
> +
>  if test -z "$meson"; then
> -    if test "$explicit_python" = no && has meson && version_ge
> "$(meson --version)" 0.63.0; then
> +    if test "$explicit_python" = no && has_meson && version_ge
> "$(meson --version)" 0.63.0; then
>          meson=meson
>      elif test "$git_submodules_action" != 'ignore' ; then
>          meson=git
>
> I will include it when posting the final series.
>
> > Do I need to adjust this loop to consider more binaries as "explicitly
> specified"?
>
> I don't think it's a huge problem. Outside virtual environments, the
> most likely setting is that Meson uses python3 which in turn is the
> most recent python3.X, so it should be fine overall.
>
> Though part of me thinks that your new loop is slightly overengineered
> and we should just require /usr/bin/env python3 and call it a day.
>

Well, but that'd be a problem for CentOS 8, wouldn't it? python3 is gonna
resolve to python3.6.


> Paolo
>
>

[-- Attachment #2: Type: text/html, Size: 3589 bytes --]

Re: [PATCH v2 3/7] configure: Look for auxiliary Python installations

[Paolo Bonzini]

On Fri, Feb 10, 2023 at 5:22 PM John Snow <jsnow@redhat.com> wrote:
>> Though part of me thinks that your new loop is slightly overengineered
>> and we should just require /usr/bin/env python3 and call it a day.
>
> Well, but that'd be a problem for CentOS 8, wouldn't it? python3 is gonna resolve to python3.6.

The user would have to specify --python=/usr/bin/python3.8, or could
also set up "alternatives" so that python3 resolves to modular Python
(3.8 or newer). I think it's a fair requirement for users of
enterprise distributions, and it works because it forces usage of
QEMU's meson submodule.

My lcitool update does the former by placing ENV PYTHON
"/usr/bin/python3.8" in the Dockerfile.

Paolo

Re: [PATCH v2 3/7] configure: Look for auxiliary Python installations

[Eric Blake]

On Fri, Feb 10, 2023 at 10:28:42AM -0500, John Snow wrote:
> > >   python=
> > > +first_python=
> > >   explicit_python=no
> > > -for binary in "${PYTHON-python3}" python
> > > +# A bare 'python' is traditionally python 2.x, but some distros
> > > +# have it as python 3.x, so check in both places.
> > > +for binary in "${PYTHON-python3}" python python3.{11..6}
> >
> > This is not available in e.g. dash, so we need to use {11,10,9,8,7,6}.
> > Just a nit, I can fix it myself.

Shoot - I didn't notice v2 before I reviewed v1, where I pointed out
the same problem.  But note that dash doesn't understand ANY brace
expansion; {11,10,9} is no better than {11..9}.

The list is also not testing python3 when $PYTHON is set.  See my
suggestion for fixing that in my mail on v1.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[PATCH v2 4/7] configure: Add nice hint to Python failure message

[John Snow]

If we begin requiring Python 3.7+, a few platforms are going to need to
install an additional package.

This is at least mildly annoying to the user (and I hate negative
attention), so solve the user's problem for them before they get a
chance to become irritated while searching on Google for how to install
newer Python packages.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index ea8c973d13b..bf512273f44 100755
--- a/configure
+++ b/configure
@@ -1058,7 +1058,10 @@ fi
 
 if ! check_py_version "$python"; then
   error_exit "Cannot use '$python', Python >= 3.6 is required." \
-      "Use --python=/path/to/python to specify a supported Python."
+             "Use --python=/path/to/python to specify a supported Python." \
+             "Maybe try:" \
+             "  openSUSE Leap 15.3+: zypper install python39" \
+             "  CentOS 8: dnf install python38"
 fi
 
 # Suppress writing compiled files
-- 
2.39.0

Re: [PATCH v2 4/7] configure: Add nice hint to Python failure message

[Thomas Huth]

On 10/02/2023 01.31, John Snow wrote:
> If we begin requiring Python 3.7+, a few platforms are going to need to
> install an additional package.
> 
> This is at least mildly annoying to the user (and I hate negative
> attention), so solve the user's problem for them before they get a
> chance to become irritated while searching on Google for how to install
> newer Python packages.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   configure | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/configure b/configure
> index ea8c973d13b..bf512273f44 100755
> --- a/configure
> +++ b/configure
> @@ -1058,7 +1058,10 @@ fi
>   
>   if ! check_py_version "$python"; then
>     error_exit "Cannot use '$python', Python >= 3.6 is required." \
> -      "Use --python=/path/to/python to specify a supported Python."
> +             "Use --python=/path/to/python to specify a supported Python." \
> +             "Maybe try:" \
> +             "  openSUSE Leap 15.3+: zypper install python39" \
> +             "  CentOS 8: dnf install python38"

IMHO the "Python > 3.6" is already pretty clear, and the hints that you 
provide here will expire pretty fast (unless you bump them regularly), so 
I'd rather drop this patch. Just my 0.02 €.

  Thomas

Re: [PATCH v2 4/7] configure: Add nice hint to Python failure message

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2167 bytes --]

On Fri, Feb 10, 2023, 2:45 AM Thomas Huth <thuth@redhat.com> wrote:

> On 10/02/2023 01.31, John Snow wrote:
> > If we begin requiring Python 3.7+, a few platforms are going to need to
> > install an additional package.
> >
> > This is at least mildly annoying to the user (and I hate negative
> > attention), so solve the user's problem for them before they get a
> > chance to become irritated while searching on Google for how to install
> > newer Python packages.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >   configure | 5 ++++-
> >   1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/configure b/configure
> > index ea8c973d13b..bf512273f44 100755
> > --- a/configure
> > +++ b/configure
> > @@ -1058,7 +1058,10 @@ fi
> >
> >   if ! check_py_version "$python"; then
> >     error_exit "Cannot use '$python', Python >= 3.6 is required." \
> > -      "Use --python=/path/to/python to specify a supported Python."
> > +             "Use --python=/path/to/python to specify a supported
> Python." \
> > +             "Maybe try:" \
> > +             "  openSUSE Leap 15.3+: zypper install python39" \
> > +             "  CentOS 8: dnf install python38"
>
> IMHO the "Python > 3.6" is already pretty clear, and the hints that you
> provide here will expire pretty fast (unless you bump them regularly), so
> I'd rather drop this patch. Just my 0.02 €.
>
>   Thomas
>

I figured that when they expired that they also just wouldn't... get
printed anymore. Just trying my best to minimize disruption as a courtesy,
and as a demonstration of how gentle the deprecation could be.

I get it though, it *will* rot ... It's the kind of thing that I'd want to
have in for maybe a release or two before just removing it, once everyone
has a chance to catch on and learn the simple remedy.

(I don't want anyone with 3.6 on their system to be unaware of how to
mitigate this issue, and quickly.)

I could replace it with a more generic hint, too; like "Try looking for
python3y or python3.y packages in your distro's software repository" that
would rot at a slower pace.

--js

>

[-- Attachment #2: Type: text/html, Size: 3218 bytes --]

[PATCH v2 5/7] DO-NOT-MERGE: testing: Add Python >= 3.7 to Centos, OpenSuSE

[John Snow]

This is just a proof-of-concept patch, as these files are lcitool
generated. The real fix will involve updating the lcitool configuration
and updating these files that way.

This is just to prove that bumping our dependency works.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/docker/dockerfiles/centos8.docker       | 1 +
 tests/docker/dockerfiles/opensuse-leap.docker | 1 +
 2 files changed, 2 insertions(+)

diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index fbc953c6dcc..a3bfddf382d 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -95,6 +95,7 @@ RUN dnf distro-sync -y && \
         pkgconfig \
         pulseaudio-libs-devel \
         python3 \
+        python38 \
         python3-PyYAML \
         python3-numpy \
         python3-pillow \
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 4b2c02d6abf..9e688c1d441 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -89,6 +89,7 @@ RUN zypper update -y && \
            pam-devel \
            pcre-devel-static \
            pkgconfig \
+           python39 \
            python3-Pillow \
            python3-PyYAML \
            python3-Sphinx \
-- 
2.39.0

[PATCH v2 6/7] CI: Stop building docs on centos8

[John Snow]

CentOS 8 does not ship with a sphinx new enough for our purposes (It
necessarily uses Python 3.6), so drop this from this build. We can
resume building docs on CentOS 9 if we wish, but we also currently test
and build docs on Fedora, Ubuntu, Alpine and Debian.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 .gitlab-ci.d/buildtest.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 0aa149a3524..0eb7f6606d4 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -166,7 +166,7 @@ build-system-centos:
   variables:
     IMAGE: centos8
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
-      --enable-modules --enable-trace-backends=dtrace --enable-docs
+      --enable-modules --enable-trace-backends=dtrace
       --enable-vfio-user-server
     TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
-- 
2.39.0

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Philippe Mathieu-Daudé]

On 10/2/23 01:31, John Snow wrote:
> CentOS 8 does not ship with a sphinx new enough for our purposes (It
> necessarily uses Python 3.6), so drop this from this build. We can
> resume building docs on CentOS 9 if we wish, but we also currently test
> and build docs on Fedora, Ubuntu, Alpine and Debian.

This wish ...

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 0aa149a3524..0eb7f6606d4 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -166,7 +166,7 @@ build-system-centos:
>     variables:
>       IMAGE: centos8

... can be remembered by adding a "Please add --enable-docs when
upgrading to centos9" comment here.

>       CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
> -      --enable-modules --enable-trace-backends=dtrace --enable-docs
> +      --enable-modules --enable-trace-backends=dtrace
>         --enable-vfio-user-server
>       TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
>         x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Peter Maydell]

On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote:
>
> CentOS 8 does not ship with a sphinx new enough for our purposes (It
> necessarily uses Python 3.6), so drop this from this build. We can
> resume building docs on CentOS 9 if we wish, but we also currently test
> and build docs on Fedora, Ubuntu, Alpine and Debian.

This confuses me. We work fine with Python 3.6 today.
Either:
 * CentOS 8 has fallen off the end of our "supported build platforms"
   list -- if so, we don't need to be CI'ing anything on it.
 * CentOS 8 is still a supported platform -- in this case building
   the docs is something we need to continue to support, and we
   can't drop Python 3.6 until all of our supported build
   platforms have a newer Python available.

thanks
-- PMM

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2878 bytes --]

On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote:
> >
> > CentOS 8 does not ship with a sphinx new enough for our purposes (It
> > necessarily uses Python 3.6), so drop this from this build. We can
> > resume building docs on CentOS 9 if we wish, but we also currently test
> > and build docs on Fedora, Ubuntu, Alpine and Debian.
>
> This confuses me. We work fine with Python 3.6 today.
>

That won't last - Many tools such as mypy, pylint and flake8 which I use to
manage our python codebase have been dropping support for 3.6 and I've had
to implement an increasing number of workarounds to help keep it possible
to test 3.6 via CI while also ensuring our newest platforms work as dev
environments.

Our testing matrix for Python is novel and thorough enough that it's
revealed  several bugs in other downstream Python distributions for Debian
and Fedora, and dozens of bugs for the linters themselves.

I'm concerned that when 3.7 is EOL'd in just a few months that the support
and testing gap is going to get even uglier.

In addition, Avocado framework has been asking me to drop 3.6 so they can
follow suit, for much of the same reasons - maintenance cost.

The typing work we've been pursuing for our testing and qapi libraries
would also be greatly helped by dropping 3.6, but it's really the
maintenance argument that's driving this.

Anyway, just trying to explain that it's not just a frivolous thing I'm
doing for the sake of itself.

Either:
>  * CentOS 8 has fallen off the end of our "supported build platforms"
>    list -- if so, we don't need to be CI'ing anything on it.
>  * CentOS 8 is still a supported platform -- in this case building
>    the docs is something we need to continue to support, and we
>    can't drop Python 3.6 until all of our supported build
>    platforms have a newer Python available.
>

The argument I'm making is:

- CentOS 8 is a supported build platform
- All platforms *do* have a Python modern enough to allow us to drop 3.6
- CentOS's repo version of sphinx is hardcoded to use the older 3.6, though
- You expressed a preference to me in the past to NOT use a pip installed
version of sphinx in preference to the system version in "configure"
- It's still possible to build docs on CentOS 8 after this patchset, you
just need a pip version.
- We've used the justification that our build platform promise does not
necessarily extend to docs and tests in the past.
- So just skip docs building for CentOS 8, only in the CI.

If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want
to go back to discussing the possibility of using sphinx versions from pip.

Paolo mentioned wanting to do similar in reply to another patch on this
series (#5) to preserve docs building on the platform.


> thanks
> -- PMM
>
>

[-- Attachment #2: Type: text/html, Size: 4381 bytes --]

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Peter Maydell]

On Fri, 10 Feb 2023 at 16:01, John Snow <jsnow@redhat.com> wrote:
> On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote:
>> This confuses me. We work fine with Python 3.6 today.
>
>
> That won't last - Many tools such as mypy, pylint and flake8 which I use to manage our python codebase have been dropping support for 3.6 and I've had to implement an increasing number of workarounds to help keep it possible to test 3.6 via CI while also ensuring our newest platforms work as dev environments.

Something somewhere seems kind of out-of-sync here. Either:
 * Python is deprecating old versions too quickly and
   dropping support for them too fast
 * CentOS is retaining old versions of Python when it needs to
   ship newer ones
 * Our policy for what distros we support is overly lax
   and causing us to try to support ancient platforms
   that we shouldn't be trying to support

because "use the system version of foo" should not be a big
deal -- it's not like we're trying to support decades-old
hosts here: Centos 8 was released in 2019, which is less than
five years ago.

> The argument I'm making is:
>
> - CentOS 8 is a supported build platform
> - All platforms *do* have a Python modern enough to allow us to drop 3.6
> - CentOS's repo version of sphinx is hardcoded to use the older 3.6, though
> - You expressed a preference to me in the past to NOT use a pip installed version of sphinx in preference to the system version in "configure"
> - It's still possible to build docs on CentOS 8 after this patchset, you just need a pip version.
> - We've used the justification that our build platform promise does not necessarily extend to docs and tests in the past.
> - So just skip docs building for CentOS 8, only in the CI.
>
> If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want to go back to discussing the possibility of using sphinx versions from pip.

If Python 3.6 is EOL, shouldn't CentOS have shipped an updated
version of Sphinx to go with their updated Python ?

I'm not super-happy about dropping the docs build requirement,
but I guess it's probably the least-worst choice.

thanks
-- PMM

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Fri, Feb 10, 2023 at 04:32:19PM +0000, Peter Maydell wrote:
> On Fri, 10 Feb 2023 at 16:01, John Snow <jsnow@redhat.com> wrote:
> > On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> >> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote:
> >> This confuses me. We work fine with Python 3.6 today.
> >
> >
> > That won't last - Many tools such as mypy, pylint and flake8 which I use to manage our python codebase have been dropping support for 3.6 and I've had to implement an increasing number of workarounds to help keep it possible to test 3.6 via CI while also ensuring our newest platforms work as dev environments.
> 
> Something somewhere seems kind of out-of-sync here. Either:
>  * Python is deprecating old versions too quickly and
>    dropping support for them too fast

Nope, they're fine in declaring EOL whenever they like. There's
no requirement for upstreams to support arbitrary old versions
for any length of time.

>  * CentOS is retaining old versions of Python when it needs to
>    ship newer ones

It is also totally OK for an distro to ship and support software
versions which are EOL upstream. In fact for enterprise distros
you can generally assume that *all* the software versions they
ship are probably EOL or nearly so. The main value of enterprise
distros is that they provide long term support, where upstreams
are not willing to.

>  * Our policy for what distros we support is overly lax
>    and causing us to try to support ancient platforms
>    that we shouldn't be trying to support

I don't think so. Users of distros are not anywhere near
as aggressive at updating their installations as users
are. The number of users of RHEL-8 will dwarf that of
RHEL-9 by a large margin for a decent amount of time
yet.

The challenge here is that upstream developers want to
use shiny new features from latest upstream, and at the
same time don't want to keep back compat with older
versions that users see in their real world deployed
distros, because that is a burden.

Ideally upstream developers would never care about old
versions and only ever target the very latest upstream.
Meanwhile for users, apps would ideally support every
version of every OS that's ever existed.

Somewhere in the middle we have to strike a bargain
that spreads the pain fairly between the two groups,
accepting that this is going to be a burden for both
to some degre. Our distro support policy is a decent
attempt at doing that IMHO and has worked pretty well
IMHO.


We don't have to drop python 3.6. It is a choice because
of a desire to be able to use some shiny new python
features without caring about back compat.

Similarly we don't have to use the new meson which drops
support for python 3.6, it is again a choice because we
want to rely on some new meson features.

QEMU could easily carry on supporting python 3.6 until
the affected distros drop out ofo our support matrix, but
we would have to opt out of using certain new features,
or put more effort into back compat.

Personally I'm still on the side of carrying on supporting
3.6 per our distro support policy, but if broad consensus
is to drop 3.6 I'm not going push back anymore.

> because "use the system version of foo" should not be a big
> deal -- it's not like we're trying to support decades-old
> hosts here: Centos 8 was released in 2019, which is less than
> five years ago.

Yeah, it isn't very old at all, and in terms of deployments
will dominate over CentOS/RHEL 9 users.

> > The argument I'm making is:
> >
> > - CentOS 8 is a supported build platform
> > - All platforms *do* have a Python modern enough to allow us to drop 3.6
> > - CentOS's repo version of sphinx is hardcoded to use the older 3.6, though
> > - You expressed a preference to me in the past to NOT use a pip installed version of sphinx in preference to the system version in "configure"
> > - It's still possible to build docs on CentOS 8 after this patchset, you just need a pip version.
> > - We've used the justification that our build platform promise does not necessarily extend to docs and tests in the past.
> > - So just skip docs building for CentOS 8, only in the CI.
> >
> > If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want to go back to discussing the possibility of using sphinx versions from pip.
> 
> If Python 3.6 is EOL, shouldn't CentOS have shipped an updated
> version of Sphinx to go with their updated Python ?

CentOS isn't dropping python 3.6 support. It will exist for the
lifetime of CentOS 8.

It just decided to also ship some newer python versions as an
opt-in. These newer python versions are just the minimal runtime
though. CentOS is not shipping all the add-on python modules it
has with 3.6.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Peter Maydell]

On Fri, 10 Feb 2023 at 16:52, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Feb 10, 2023 at 04:32:19PM +0000, Peter Maydell wrote:
> > On Fri, 10 Feb 2023 at 16:01, John Snow <jsnow@redhat.com> wrote:
> > > On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > >> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote:
> > >> This confuses me. We work fine with Python 3.6 today.
> > >
> > >
> > > That won't last - Many tools such as mypy, pylint and flake8 which I use to manage our python codebase have been dropping support for 3.6 and I've had to implement an increasing number of workarounds to help keep it possible to test 3.6 via CI while also ensuring our newest platforms work as dev environments.
> >
> > Something somewhere seems kind of out-of-sync here. Either:
> >  * Python is deprecating old versions too quickly and
> >    dropping support for them too fast
>
> Nope, they're fine in declaring EOL whenever they like. There's
> no requirement for upstreams to support arbitrary old versions
> for any length of time.

To be clear, yes, absolutely any other software project can
set whatever lifecycle it chooses to do, and similarly distros
can do what they wish to do. But somewhere along the line
something's gone sideways here, because all these choices
people are making seem reasonable and yet here we are :-)
I put this bit in here because one response to "upstream's
idea of how long it wants to support something is very short"
would be "this isn't a dependency that it's suitable for
QEMU as a project to have, because the mismatch is too great".
(This would obviously be super-painful in this specific case...)

> >  * CentOS is retaining old versions of Python when it needs to
> >    ship newer ones
>
> It is also totally OK for an distro to ship and support software
> versions which are EOL upstream. In fact for enterprise distros
> you can generally assume that *all* the software versions they
> ship are probably EOL or nearly so. The main value of enterprise
> distros is that they provide long term support, where upstreams
> are not willing to.

But we as a project also have the choice, if it seems to us
that Distro X is shipping absolutely ancient versions of
dependencies, to say "we don't support distro X, because its
life cycle policies are too far out of sync with ours", or to
say "you're going to need to provide the dependencies
yourself, here's a suggestion". (We effectively do that
already with macos and Windows.)

> >  * Our policy for what distros we support is overly lax
> >    and causing us to try to support ancient platforms
> >    that we shouldn't be trying to support
>
> I don't think so. Users of distros are not anywhere near
> as aggressive at updating their installations as users
> are. The number of users of RHEL-8 will dwarf that of
> RHEL-9 by a large margin for a decent amount of time
> yet.

Right. All of these things together seem to say:
 * Python is not an unreasonable thing for the project
   to depend on
 * CentOS 8 is not an unreasonable distro for us to
   want to continue to support
 * Therefore we should continue to work with the Python
   that ships with CentOS 8...

[snip]

> We don't have to drop python 3.6. It is a choice because
> of a desire to be able to use some shiny new python
> features without caring about back compat.
>
> Similarly we don't have to use the new meson which drops
> support for python 3.6, it is again a choice because we
> want to rely on some new meson features.
>
> QEMU could easily carry on supporting python 3.6 until
> the affected distros drop out ofo our support matrix, but
> we would have to opt out of using certain new features,
> or put more effort into back compat.
>
> Personally I'm still on the side of carrying on supporting
> 3.6 per our distro support policy, but if broad consensus
> is to drop 3.6 I'm not going push back anymore.

This is really where I'm at as well -- we set our distro
support policy, and we know that means that sometimes
we have to deal with continuing to support older versions
of dependencies even when it might be easier for us if we
could require newer versions. I'm reluctant to say that
Python gets a special derogation from that policy.

thanks
-- PMM

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Paolo Bonzini]

On 2/10/23 18:15, Peter Maydell wrote:
> Right. All of these things together seem to say:
>   * Python is not an unreasonable thing for the project
>     to depend on
>   * CentOS 8 is not an unreasonable distro for us to
>     want to continue to support
>   * Therefore we should continue to work with the Python
>     that ships with CentOS 8...
> 
> [snip]
> 
>> We don't have to drop python 3.6. It is a choice because
>> of a desire to be able to use some shiny new python
>> features without caring about back compat.
>>
>> Similarly we don't have to use the new meson which drops
>> support for python 3.6, it is again a choice because we
>> want to rely on some new meson features.
>>
>> QEMU could easily carry on supporting python 3.6 until
>> the affected distros drop out ofo our support matrix, but
>> we would have to opt out of using certain new features,
>> or put more effort into back compat.
>>
>> Personally I'm still on the side of carrying on supporting
>> 3.6 per our distro support policy, but if broad consensus
>> is to drop 3.6 I'm not going push back anymore.
> 
> This is really where I'm at as well -- we set our distro
> support policy, and we know that means that sometimes
> we have to deal with continuing to support older versions
> of dependencies even when it might be easier for us if we
> could require newer versions.

There are four possibilities:

* we change the support policy and stop supporting CentOS 8 and SLE 15, 
not a good idea since a lot of people have not migrated to CentOS 9 yet.

* we keep supporting Python 3.6 until CentOS 8 and SLE 15 stop being 
supported.  This means several more years since SLE 16 hasn't even been 
released.

* we support Python 3.6 just for building documentation, i.e. we are 
careful not to use Python 3.7+ features in our Sphinx extensions but are 
free to use them elsewhere.  CentOS 8 users can install sphinx from 
either RPMs (which will use Python 3.6) or pip (which will use Python 3.8).

* we only support Python 3.7+, which means CentOS 8 CI and users have to 
either install Sphinx from pip or disable documentation.

The only difference between the last two is documentation and CI 
configuration.  The code is exactly the same.

> I'm reluctant to say that
> Python gets a special derogation from that policy.

Note that its not the Python runtime but the Python dependencies, for 
which we already install avocado and some Python development tools in a 
virtual environment.  So, the questions are:

* to what extent can we rely on pip packages (preinstalled by the user) 
instead of the distro packages?

* to what extent should QEMU install its own dependencies via pip in a 
virtual environment instead of requiring the user to preinstall them?

Right now the answers for both are "avocado gets an exception for 
tests/, Python development tools such as mypy get an exception for python/".

The Python 3.7+ series (not this one by John) currently adds "sphinx 
gets an exception to the first answer only".

In the future I would like to unify virtual environment generation for 
tests/ and python/ and move it to configure.  If desirable, this would 
make it possible to implement something like "the user need not care 
about Python dependencies, configure can (but need not) install them all 
via pip".  Distros would still package the dependencies, but users would 
have a slightly easier time building QEMU.

Thanks,

Paolo

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Fri, Feb 10, 2023 at 07:27:41PM +0100, Paolo Bonzini wrote:
> On 2/10/23 18:15, Peter Maydell wrote:
> There are four possibilities:
> 
> * we change the support policy and stop supporting CentOS 8 and SLE 15, not
> a good idea since a lot of people have not migrated to CentOS 9 yet.
> 
> * we keep supporting Python 3.6 until CentOS 8 and SLE 15 stop being
> supported.  This means several more years since SLE 16 hasn't even been
> released.

The massive gap from SLE 15 to 16 is something our support policy
wasn't anticipating, though it probably should have done, since a
4-5 year gap is actually normal if looking at previous SLE versions.

I was thinking too much about RHEL which has a short 3 year gap,
but also rebases software versions in .y releases, so often the
dependancies aren't as old as the 3 year life of RHEL would
suggest.

We need to finese our support policy to give us more flexibility
wrt the very long life / long gap enterprise distros.

> * we support Python 3.6 just for building documentation, i.e. we are careful
> not to use Python 3.7+ features in our Sphinx extensions but are free to use
> them elsewhere.  CentOS 8 users can install sphinx from either RPMs (which
> will use Python 3.6) or pip (which will use Python 3.8).
> 
> * we only support Python 3.7+, which means CentOS 8 CI and users have to
> either install Sphinx from pip or disable documentation.
> 
> The only difference between the last two is documentation and CI
> configuration.  The code is exactly the same.

While it is good to mention the idea of 3.6 just for docs, I don't
think it is really a good idea to put into practice.

IMHO there's compelling benefit in having a clear & simple story for
the min versions of dependancies, both for us as maintainers, and
for people consuming it. So if we want 3.7, we should apply it
universally without caveats / carve-outs.

> > I'm reluctant to say that
> > Python gets a special derogation from that policy.
> 
> Note that its not the Python runtime but the Python dependencies, for which
> we already install avocado and some Python development tools in a virtual
> environment.  So, the questions are:
> 
> * to what extent can we rely on pip packages (preinstalled by the user)
> instead of the distro packages?
> 
> * to what extent should QEMU install its own dependencies via pip in a
> virtual environment instead of requiring the user to preinstall them?

FWIW, when I introduced scripts/git-submodule.sh script to handle
automatically checking out git submodules during build, there was
quite a bit of negative feedback from some users/contributors who
don't want  stuff being fetched off the net during builds.

Typically their scenario was that they have a QEMU checkout on
one machine with net access. That filesystem was exported to
another machine (or multiple other machines) to perform the
actual build(s) and those build machines lacked net access.

This motiviated the introduction of the configure arg, to tell
us *NOT* to attempt to checkout submodules, but merely validate
that they exist at the right checkout hash. The user would
manually checkout the submodules on the hosts which had net
access.

This situation is not too dis-similar to what distros have in
their package build systems which often block net access.

None of this means we can't use a private virtualenv for all of
QEMU python bits.

Just that if we go this route, then we're putting a greater
burden on people whose distro doesn't have the required packages,
and whose build env lacks net access.

They'll have to populate a venv manually in some way and get it
onto their build env.  The number of people with this situation
though likely is small enough, that it is likely an acceptable
burden / tradeoff in general.  The burden would only apply to the
pieces of build that are actually mandatory for build, the most
prominent of which are meson, and the QAPI generation code. Bits
like avocado, mypy, flake8, etc are all optional extra tests.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

[...]

> We don't have to drop python 3.6. It is a choice because
> of a desire to be able to use some shiny new python
> features without caring about back compat.

I read this on Friday, and decided to let it sit until after the
weekend.  Well, it's now Tuesday, and to be frank, it's still as
offensively flippant as it was on Friday.  It shows either ignorance of
or cavalier disregard for the sheer amount of work some of us have had
to put into keeping old versions of Python viable.

The latter would be quite unlike you, so it must be the former.

John has sunk *man-months* into keeping old versions of Python viable.
I've put in a lot less myself, but still enough to be painfully aware of
it.  I figure Cleber and Beraldo are somewhere in between.

Insinuating John's proposal is motivated by "a desire to be able to use
some shiny new python features without caring about back compat"
disrespects all this work.

We should have a sober discussion on which versions of Python to work
with, and the tradeoffs involved.  But before I engage in that, I insist
on resetting the frame: no, this is not about shiny, new Python
features.  It is about stopping the bleeding.  It is about reducing what
feels more and more like bullshit work to me, so we can actually
accomplish stuff that matters.

And let's give the people who have been doing the actual work the
benefit of the doubt.

[...]

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Thomas Huth]

On 14/02/2023 08.40, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> [...]
> 
>> We don't have to drop python 3.6. It is a choice because
>> of a desire to be able to use some shiny new python
>> features without caring about back compat.
> 
> I read this on Friday, and decided to let it sit until after the
> weekend.  Well, it's now Tuesday, and to be frank, it's still as
> offensively flippant as it was on Friday.  It shows either ignorance of
> or cavalier disregard for the sheer amount of work some of us have had
> to put into keeping old versions of Python viable.

I'm a complete python ignorant, too, so I'm a little bit surprised of the 
amount of pain that these scripts are causing.

No matter of that fact, I think Peter still has a point that we have a real 
conflict here with our current support policy. So this either means that 
Python was the wrong choice for our needs (since it is moving too fast and 
causing too much friction), or we should really rethink our support policy.

I guess we're too deep into the Python rabbit hole already, and I'm not 
aware of any other good solutions (back to Perl scripts? No, thanks!), so 
it's likely quite impossible to tune that knob.

Thus we should maybe really start talking about our support policy now. I 
think the main problem is likely the sentence "Support for the previous 
major version will be dropped 2 years after the new major version is 
released". Maybe we should shorten that time frame to 1 year. The 2 years 
caused some confusions in the past already, since e.g. Debian only supports 
the previous major release for only one more year, and macOS also releases a 
major version each year ... so IMHO we could shorten the time frame for the 
previous major release to 1 year instead. People then could still continue 
building QEMU on CentOS 8, but they have to be aware that they might install 
other software like Sphinx manually if they want to continue using QEMU with 
docs there. What do you think?

  Thomas

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 14/02/2023 08.40, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> [...]
>> 
>>> We don't have to drop python 3.6. It is a choice because
>>> of a desire to be able to use some shiny new python
>>> features without caring about back compat.
>> I read this on Friday, and decided to let it sit until after the
>> weekend.  Well, it's now Tuesday, and to be frank, it's still as
>> offensively flippant as it was on Friday.  It shows either ignorance of
>> or cavalier disregard for the sheer amount of work some of us have had
>> to put into keeping old versions of Python viable.
>
> I'm a complete python ignorant, too, so I'm a little bit surprised of
> the amount of pain that these scripts are causing.
>
> No matter of that fact, I think Peter still has a point that we have a
> real conflict here with our current support policy. So this either
> means that Python was the wrong choice for our needs (since it is
> moving too fast and causing too much friction), or we should really
> rethink our support policy.
>
> I guess we're too deep into the Python rabbit hole already, and I'm
> not aware of any other good solutions (back to Perl scripts? No,
> thanks!), so it's likely quite impossible to tune that knob.
>
> Thus we should maybe really start talking about our support policy
> now. I think the main problem is likely the sentence "Support for the
> previous major version will be dropped 2 years after the new major
> version is released". Maybe we should shorten that time frame to 1
> year.

I think this should be a fair approach. Generally I recommend avoiding
installing a new LTS until at least the first tick release has ironed
out the obvious bugs. A year seems like a fair grace period to update to
the next LTS. Those that like sitting on old maintained LTS releases are
less likely to be tracking the bleeding edge of development anyway and
likely are happy with their distro provided packages.

BTW my next testing/next finally updates the last few Ubuntu 20.04 to
22.04 systems which also allows removing a few tsan and clang hacks in
the process. Progress might not be a straight line but it averages out
in that approximate direction ;-)

> The 2 years caused some confusions in the past already, since
> e.g. Debian only supports the previous major release for only one more
> year, and macOS also releases a major version each year ... so IMHO we
> could shorten the time frame for the previous major release to 1 year
> instead. People then could still continue building QEMU on CentOS 8,
> but they have to be aware that they might install other software like
> Sphinx manually if they want to continue using QEMU with docs there.
> What do you think?

Works for me at least.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Tue, Feb 14, 2023 at 09:35:44AM +0100, Thomas Huth wrote:
> On 14/02/2023 08.40, Markus Armbruster wrote:
> > Daniel P. Berrangé <berrange@redhat.com> writes:
> > 
> > [...]
> > 
> > > We don't have to drop python 3.6. It is a choice because
> > > of a desire to be able to use some shiny new python
> > > features without caring about back compat.
> > 
> > I read this on Friday, and decided to let it sit until after the
> > weekend.  Well, it's now Tuesday, and to be frank, it's still as
> > offensively flippant as it was on Friday.  It shows either ignorance of
> > or cavalier disregard for the sheer amount of work some of us have had
> > to put into keeping old versions of Python viable.
> 
> I'm a complete python ignorant, too, so I'm a little bit surprised of the
> amount of pain that these scripts are causing.
> 
> No matter of that fact, I think Peter still has a point that we have a real
> conflict here with our current support policy. So this either means that
> Python was the wrong choice for our needs (since it is moving too fast and
> causing too much friction), or we should really rethink our support policy.
> 
> I guess we're too deep into the Python rabbit hole already, and I'm not
> aware of any other good solutions (back to Perl scripts? No, thanks!), so
> it's likely quite impossible to tune that knob.

I still believe python is a probably the best thing for what we're using
it for. Certainly would not suggest shell or perl, and using a compiled
language would add its own complications for cross compilation.

> Thus we should maybe really start talking about our support policy now. I
> think the main problem is likely the sentence "Support for the previous
> major version will be dropped 2 years after the new major version is
> released". Maybe we should shorten that time frame to 1 year. The 2 years
> caused some confusions in the past already, since e.g. Debian only supports
> the previous major release for only one more year, and macOS also releases a
> major version each year ... so IMHO we could shorten the time frame for the
> previous major release to 1 year instead. People then could still continue
> building QEMU on CentOS 8, but they have to be aware that they might install
> other software like Sphinx manually if they want to continue using QEMU with
> docs there. What do you think?


I think perhaps the problem is not in the length of time defined by
our support policy, but rather that we're facing a rather different
reality to the one we've historically been used it, where distros
are no longer critical dependancies and our support policy does not
reflect that.


For any C/C++ application, wanting to target the versions shipped in a
distro has been pretty much normal practice. C has not ever come with
a standard package manager toolset, the distros service that role. The
distros also aren't generally a fan of shipping multiple versions of
C libs in parallel.


Pretty much every non-C library though is different. They all have
their own package manager service / tools (perl has cpan, pytyhon has
PyPi/pip, ruby has gems. With latest compiled languages like Go/Rust,
this has gone one step further and is natively integrated into the
compiler toolchain as standard.


IOW, for everything except C, it has become increasingly normal
practice to ignore the distro and dynamically download all the deps
your application needs into a self contained local environment.
Now, the distros aren't especially a fan of this new world, since
they still prefer to unbundle all these deps, but I think that
approach is increasingly difficult for them to achieve because the
majority of upstreams don't care for the distro versions.


Thus what we're experiancing is a clash between the traditional
way that C applications/libraries deal with their deps, vs the
way pretty much every other language deals with their deps in
the modern world. It has come up now because we're making much
more use of python now, than we did in the past.


Our support policy is written from the POV of the C world, and
merely reducing the length of time we support a distro does not
address the different world view of Python.

Should we instead try to be more explicit about the different
needs of the non-C dependencies ?

We could for example say

 * For native library/application dependancies we aim to
   support the two most recent distro versions, for 2 years
   overlap

 * For non-native library/applications dependancies we aim
   to support only the most recent distro version. Users
   of older distros may need to dynamically fetch newer
   deps.

The python 3.8 runtime would be considered a native dep, so fall
under the 2 distro versions rule. This is fine with CentOS 8,
since it provides newer python runtime versions.

The python libraries, or tools written in python (meson), would
fall under the second rule, and so only need to target one distro
version. This would be compatible with CentOS 8, as the users would
be expected to download extra python components (or we do it on
their behalf).

For the second rule, rather than saying most recent distro versions,
possibly we might want to carve out an exclusion for LTS distros too.
ie, explicitly don't care about versions of non-native bits in RHEL
at all, beyond availability of the base (python) runtime.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Feb 14, 2023 at 09:35:44AM +0100, Thomas Huth wrote:
>> On 14/02/2023 08.40, Markus Armbruster wrote:
>> > Daniel P. Berrangé <berrange@redhat.com> writes:
>> > 
>> > [...]
>> > 
>> > > We don't have to drop python 3.6. It is a choice because
>> > > of a desire to be able to use some shiny new python
>> > > features without caring about back compat.
>> > 
>> > I read this on Friday, and decided to let it sit until after the
>> > weekend.  Well, it's now Tuesday, and to be frank, it's still as
>> > offensively flippant as it was on Friday.  It shows either ignorance of
>> > or cavalier disregard for the sheer amount of work some of us have had
>> > to put into keeping old versions of Python viable.
>> 
>> I'm a complete python ignorant, too, so I'm a little bit surprised of the
>> amount of pain that these scripts are causing.
>> 
>> No matter of that fact, I think Peter still has a point that we have a real
>> conflict here with our current support policy. So this either means that
>> Python was the wrong choice for our needs (since it is moving too fast and
>> causing too much friction), or we should really rethink our support policy.
>> 
>> I guess we're too deep into the Python rabbit hole already, and I'm not
>> aware of any other good solutions (back to Perl scripts? No, thanks!), so
>> it's likely quite impossible to tune that knob.
>
> I still believe python is a probably the best thing for what we're using
> it for. Certainly would not suggest shell or perl, and using a compiled
> language would add its own complications for cross compilation.
>
>> Thus we should maybe really start talking about our support policy now. I
>> think the main problem is likely the sentence "Support for the previous
>> major version will be dropped 2 years after the new major version is
>> released". Maybe we should shorten that time frame to 1 year. The 2 years

It's actually "2 years after the new major version is released or when
the vendor itself drops support, whichever comes first."

>> caused some confusions in the past already, since e.g. Debian only supports
>> the previous major release for only one more year, and macOS also releases a
>> major version each year ... so IMHO we could shorten the time frame for the
>> previous major release to 1 year instead. People then could still continue
>> building QEMU on CentOS 8, but they have to be aware that they might install
>> other software like Sphinx manually if they want to continue using QEMU with
>> docs there. What do you think?
>
>
> I think perhaps the problem is not in the length of time defined by
> our support policy, but rather that we're facing a rather different
> reality to the one we've historically been used it, where distros
> are no longer critical dependancies and our support policy does not
> reflect that.
>
>
> For any C/C++ application, wanting to target the versions shipped in a
> distro has been pretty much normal practice. C has not ever come with
> a standard package manager toolset, the distros service that role. The
> distros also aren't generally a fan of shipping multiple versions of
> C libs in parallel.
>
>
> Pretty much every non-C library though is different. They all have
> their own package manager service / tools (perl has cpan, pytyhon has
> PyPi/pip, ruby has gems. With latest compiled languages like Go/Rust,
> this has gone one step further and is natively integrated into the
> compiler toolchain as standard.
>
>
> IOW, for everything except C, it has become increasingly normal
> practice to ignore the distro and dynamically download all the deps
> your application needs into a self contained local environment.
> Now, the distros aren't especially a fan of this new world, since
> they still prefer to unbundle all these deps, but I think that
> approach is increasingly difficult for them to achieve because the
> majority of upstreams don't care for the distro versions.
>
>
> Thus what we're experiancing is a clash between the traditional
> way that C applications/libraries deal with their deps, vs the
> way pretty much every other language deals with their deps in
> the modern world. It has come up now because we're making much
> more use of python now, than we did in the past.

Yes.

The traditional way of building applications is to examine the
environment, and configure the application accordingly.  If you depend
on libfoo, configure looks for (a supported version of) libfoo, and the
source code deals with differences between versions, if any.  libfoo is
expected to bend over backwards to avoid differences.

The newfangled way of building applications is to set up a controlled
environment.  No need to configure the application.

Pros and cons, no need to rehash them here, I believe.

> Our support policy is written from the POV of the C world, and
> merely reducing the length of time we support a distro does not
> address the different world view of Python.
>
> Should we instead try to be more explicit about the different
> needs of the non-C dependencies ?
>
> We could for example say
>
>  * For native library/application dependancies we aim to
>    support the two most recent distro versions, for 2 years
>    overlap
>
>  * For non-native library/applications dependancies we aim
>    to support only the most recent distro version. Users
>    of older distros may need to dynamically fetch newer
>    deps.

Who does the fetching, users manually, or the build process
automatically?

> The python 3.8 runtime would be considered a native dep, so fall
> under the 2 distro versions rule. This is fine with CentOS 8,
> since it provides newer python runtime versions.
>
> The python libraries, or tools written in python (meson), would
> fall under the second rule, and so only need to target one distro
> version. This would be compatible with CentOS 8, as the users would
> be expected to download extra python components (or we do it on
> their behalf).
>
> For the second rule, rather than saying most recent distro versions,
> possibly we might want to carve out an exclusion for LTS distros too.
> ie, explicitly don't care about versions of non-native bits in RHEL
> at all, beyond availability of the base (python) runtime.

Interesting idea.  Can anyone think of reasons not to do this?

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Thu, Feb 16, 2023 at 02:08:33AM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> > Our support policy is written from the POV of the C world, and
> > merely reducing the length of time we support a distro does not
> > address the different world view of Python.
> >
> > Should we instead try to be more explicit about the different
> > needs of the non-C dependencies ?
> >
> > We could for example say
> >
> >  * For native library/application dependancies we aim to
> >    support the two most recent distro versions, for 2 years
> >    overlap
> >
> >  * For non-native library/applications dependancies we aim
> >    to support only the most recent distro version. Users
> >    of older distros may need to dynamically fetch newer
> >    deps.
> 
> Who does the fetching, users manually, or the build process
> automatically?

I expect both cases need supporting.

In the case of distro builds, they will have to fetch the
deps ahead of time, because the build environments usually
won't have any network access. Some contributors have also
previously objected to the build system fetching stuff off
the network, but they're a small minority.

For friendliness to developers, the build process would be
best to fetch automatically, if they haven't been provided
upfront.

> > The python 3.8 runtime would be considered a native dep, so fall
> > under the 2 distro versions rule. This is fine with CentOS 8,
> > since it provides newer python runtime versions.
> >
> > The python libraries, or tools written in python (meson), would
> > fall under the second rule, and so only need to target one distro
> > version. This would be compatible with CentOS 8, as the users would
> > be expected to download extra python components (or we do it on
> > their behalf).
> >
> > For the second rule, rather than saying most recent distro versions,
> > possibly we might want to carve out an exclusion for LTS distros too.
> > ie, explicitly don't care about versions of non-native bits in RHEL
> > at all, beyond availability of the base (python) runtime.
> 
> Interesting idea.  Can anyone think of reasons not to do this?

It is probably even more compelling when looking at SLES, which is
having an even larger gap between releases than RHEL does.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Peter Maydell]

On Tue, 14 Feb 2023 at 07:40, Markus Armbruster <armbru@redhat.com> wrote:
> I read this on Friday, and decided to let it sit until after the
> weekend.  Well, it's now Tuesday, and to be frank, it's still as
> offensively flippant as it was on Friday.  It shows either ignorance of
> or cavalier disregard for the sheer amount of work some of us have had
> to put into keeping old versions of Python viable.

From my point of view it is definitely ignorance. I simply
didn't expect that a serious major scripting language
would be making it this hard to write code that is
backwards compatible with older but not ancient versions.

thanks
-- PMM

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Kevin Wolf]

Am 14.02.2023 um 08:40 hat Markus Armbruster geschrieben:
> I read this on Friday, and decided to let it sit until after the
> weekend.  Well, it's now Tuesday, and to be frank, it's still as
> offensively flippant as it was on Friday.  It shows either ignorance of
> or cavalier disregard for the sheer amount of work some of us have had
> to put into keeping old versions of Python viable.
> 
> The latter would be quite unlike you, so it must be the former.

Honest question, Markus, because I haven't been following as much what
is happening in Python recently: What are the biggest pain points in
this context?

Has Python started removing features from new versions more aggressively
so that we have to update the code so it can run on newer versions, and
still keep compatibility paths for older versions that don't have the
replacement yet?

Kevin

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Markus Armbruster]

Kevin Wolf <kwolf@redhat.com> writes:

> Am 14.02.2023 um 08:40 hat Markus Armbruster geschrieben:
>> I read this on Friday, and decided to let it sit until after the
>> weekend.  Well, it's now Tuesday, and to be frank, it's still as
>> offensively flippant as it was on Friday.  It shows either ignorance of
>> or cavalier disregard for the sheer amount of work some of us have had
>> to put into keeping old versions of Python viable.
>> 
>> The latter would be quite unlike you, so it must be the former.
>
> Honest question, Markus, because I haven't been following as much what
> is happening in Python recently: What are the biggest pain points in
> this context?
>
> Has Python started removing features from new versions more aggressively
> so that we have to update the code so it can run on newer versions, and
> still keep compatibility paths for older versions that don't have the
> replacement yet?

A caveat before I answer: I'm aware of four non-trivial uses of Python,
(QAPI generator, Avocado, the qemu/qmp package, meson), but my direct
experience is limited to just QAPI.

And another preliminary remark: I think there's confusion about the
actual impact of dropping 3.6.  But attempting to clarify that is a
topic for another message (Message-ID: <87v8k2ycjb.fsf@pond.sub.org>,
just sent).

I've personally observed roughly three kinds of pain.  In order of
(sharply) increasing intensity:

1. Python language and core libraries

   Python still evolves at a good pace.

   We can sidestep new features.  We can't sidestep bugs.  In either
   case, we need to become aware of the issue first.

   For features, diligent reading of documentation suffices, but for
   bugs, it's testing or bust.

   Things have exploded into John's face often enough to "motivate" him
   to spend a huge chunk of his time (man-months) on building and
   maintaining infrastructure for testing with all the Python versions.
   Naturally, the wider the version range, the bigger the test matrix,
   and the deeper the time sink.

   The range of versions we're trying to target is actually wider than
   Python's deprecation grace period.  This means we sometimes get to do
   things in both the old and the new way (because neither works across
   the whole range of versions), or look for ways to sidestep the mess
   somehow.

   Python programs are nicely portable between host systems.  Between
   more than a few Python versions, not so much.

2. Python tooling and non-core libraries

   Much of these don't give a rat's ass for anything but the latest few
   versions, let alone for EOLed ones like 3.6.  Likely because they
   decided it would be a fool's errand.

   For instance, Python packaging has changed beyond recognition since
   3.6.  There is no single way to package that spans the range of
   versions we try to target.  Bites the qemu/qmp package.

   Linters are also a moving target, and keeping lint tests pass for a
   wide range of linters version involves lots of largely pointless
   fiddling.  Another dimension in the test matrix.

3. Python typing is too immature in anything but latest versions

   This is kind of a special case of 2., but it's a big one for *me*, so
   I'm making it an item of its own.

   Type hints have been evolving substantially, and it doesn't look like
   this is going to stop any time soon.

   Trying to keep typing tests pass for a wide range of mypy versions is
   even less practical than for other linters.

   Can we simply skip type checking unless we have a sufficiently recent
   mypy?  After all, type hints are designed to be transparent at
   runtime, so code with new type hints should still run on old Python,
   shouldn't it?  Nope.  Doesn't.

   Kevin and others pointed out that the QAPI generator code is harder
   to work with than it should be, in good part because they have to
   spend too much time understanding possible types.  In response to
   this valid complaint, we embarked on the known non-trivial project to
   add type hints to the QAPI code generator.  This may have been a
   mistake.  We had no idea just how much trouble type hints would give
   us when combined with a range of Python versions this wide.  But here
   we are, 2+ years older and somewhat better informed, and our informed
   desire is to narrow the version range as much as practical.

   Ditching 3.6 feels eminently practical.  Let's do it.

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Tue, Feb 14, 2023 at 08:40:20AM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> [...]
> 
> > We don't have to drop python 3.6. It is a choice because
> > of a desire to be able to use some shiny new python
> > features without caring about back compat.
> 
> I read this on Friday, and decided to let it sit until after the
> weekend.  Well, it's now Tuesday, and to be frank, it's still as
> offensively flippant as it was on Friday.  It shows either ignorance of
> or cavalier disregard for the sheer amount of work some of us have had
> to put into keeping old versions of Python viable.
> 
> The latter would be quite unlike you, so it must be the former.

I'm sorry, I don't mean it to be offensive. I'm genuinely not seeing
from the descriptions in the series what the functional benefits are
from dropping 3.6.

> John has sunk *man-months* into keeping old versions of Python viable.
> I've put in a lot less myself, but still enough to be painfully aware of
> it.  I figure Cleber and Beraldo are somewhere in between
> 
> Insinuating John's proposal is motivated by "a desire to be able to use
> some shiny new python features without caring about back compat"
> disrespects all this work.

I'm writing my comments based on what is described in the cover letter
as the motivations for the change:

[quote]
The motivation for this series is that Python 3.6 was EOL at the end of
2021; upstream tools are beginning to drop support for it, including
setuptools, pylint, mypy, etc. As time goes by, it becomes more
difficult to support and test against the full range of Python versions
that QEMU supports. The closer we get to Python 3.12, the harder it will
be to cover that full spread of versions.
[/quote]

this is all about new/eol versions of software upstream, and I don't
think that's a justification. QEMU explicitly aims to use distro provided
versions and upstream EOL status is not relevant in that context. Even
if using "pip" to install it is possible to limit yourself to upstream
releases which still support 3.6.

There is the separate issue of Meson dropping python 3.6 which motivates
Paolo's series. Again though, we don't have to increase our minimum meson
version, because meson is working today. It is our choice to to increase
it to use latest available meson features. At some point we can decide
what we have is good enough and we don't have to keep chasing the latest
features. Maybe we're not there yet, but we should think about when that
would be.

[quote]
The qemu.qmp library and the avocado testing framework both have
motivations for dropping 3.6 support, but are committed to not doing so
until QEMU drops support.
[/quote]

I suspect that this is more of a driver for the drop of 3.6, but I
don't see any details.

IOW overall justification come across as wanting to use new features,
and follow upstream EOL, without any real detail of what we're going
to gain from a functional POV.

> We should have a sober discussion on which versions of Python to work
> with, and the tradeoffs involved.  But before I engage in that, I insist
> on resetting the frame: no, this is not about shiny, new Python
> features.  It is about stopping the bleeding.  It is about reducing what
> feels more and more like bullshit work to me, so we can actually
> accomplish stuff that matters.

Every applications developer chooses an arbitrary cut off points for
minimum software versions, depending on their particular needs. With
our support policy we tried to express a reasonable tradeoff between
keeping back compat, and being able to adopt new features.

Obviously that tradeoff is not currently looking acceptable on the
python side, but it is not clear why that is ?

Can someone simply explain what we wil see as the benefit from dropping
3.6 / adopting 3.7 as the baseline ? 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Paolo Bonzini]

On Tue, Feb 14, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> [quote]
> The motivation for this series is that Python 3.6 was EOL at the end of
> 2021; upstream tools are beginning to drop support for it, including
> setuptools, pylint, mypy, etc. As time goes by, it becomes more
> difficult to support and test against the full range of Python versions
> that QEMU supports. The closer we get to Python 3.12, the harder it will
> be to cover that full spread of versions.
> [/quote]
>
> this is all about new/eol versions of software upstream, and I don't
> think that's a justification. QEMU explicitly aims to use distro provided
> versions and upstream EOL status is not relevant in that context. Even
> if using "pip" to install it is possible to limit yourself to upstream
> releases which still support 3.6.
>
> There is the separate issue of Meson dropping python 3.6 which motivates
> Paolo's series. Again though, we don't have to increase our minimum meson
> version, because meson is working today. It is our choice to to increase
> it to use latest available meson features. At some point we can decide
> what we have is good enough and we don't have to keep chasing the latest
> features. Maybe we're not there yet, but we should think about when that
> would be.

In the case of Meson, the main advantage is moving _all_ of the
emulator configury out of the configure script.  This requires
add_global_dependencies which was added in 0.63.  So in that case it
is indeed mostly about shiny new features and it's not absolutely
necessary.

In the case of Python the issue is not the interpreter per se, though
there are a couple new feature in Python 3.7 that are quite nice (for
example improved data classes[1] or context variables[2]). The main
problem as far as I understood (and have seen in my experience) is
linting tools. New versions fix bugs that caused false positives, but
also become more strict at the same time. The newer versions at the
same time are very quick at dropping support for old versions of
Python; while older versions sometimes throw deprecation warnings on
new versions of Python. This makes it very hard to support a single
version of, say, mypy that works on all versions from RHEL8 and SLE15
to Fedora 38 and Ubuntu 23.04.

[1] https://peps.python.org/pep-0557/
[2] https://peps.python.org/pep-0567/

In fact this issue is the reason why RHEL9 does not package any of
these tools and does not run them as part of building RPMs even though
in principle it would be a good idea; it's too much of a pain to have
a single version that works across all the packages in the
distribution.

Regarding your other suggestion:

> * For non-native library/applications dependancies we aim
>   to support only the most recent distro version. Users
>   of older distros may need to dynamically fetch newer
>   deps.

I think this is a good idea, but one issue with "only supporting the
most recent distro version" is SUSE. While the most recent version of
SLE is about 5 years old, there is no next version in sight---SUSE
instead is working on their "Adaptable Linux Platform", but it's still
in the prototype stage[3]. So alternatively we could put a 4 or 5 year
cutoff after which you need to fetch newer deps. Considering the
delays between freeze and release of distros like RHEL or SLE, in
practice we would probably keep Python versions supported for 6-7
years.

A 4 year cutoff in practice means that we would be able to drop Python
3.6 support for QEMU 7.1 (RHEL8 becomes 4 year old next May, while SLE
is already over the threshold). In practice this means waiting until
next April for John/Markus/myself, which I think is fair.

Note that at least for now Meson need not to follow this rule; it is
distributed with QEMU because configure must execute it before Make
sets up the Python venv. This may change in the future of course,
depending on the direction that the configure script takes with
respect to setting up the venv, but I wouldn't hold my breath.
However, the minimum required version of Meson of course must respect
the oldest supported version of Python.

Paolo

[3] https://www.suse.com/c/the-first-prototype-of-adaptable-linux-platform-is-live/

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Tue, Feb 14, 2023 at 03:03:54PM +0100, Paolo Bonzini wrote:
> On Tue, Feb 14, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > [quote]
> > The motivation for this series is that Python 3.6 was EOL at the end of
> > 2021; upstream tools are beginning to drop support for it, including
> > setuptools, pylint, mypy, etc. As time goes by, it becomes more
> > difficult to support and test against the full range of Python versions
> > that QEMU supports. The closer we get to Python 3.12, the harder it will
> > be to cover that full spread of versions.
> > [/quote]
> >
> > this is all about new/eol versions of software upstream, and I don't
> > think that's a justification. QEMU explicitly aims to use distro provided
> > versions and upstream EOL status is not relevant in that context. Even
> > if using "pip" to install it is possible to limit yourself to upstream
> > releases which still support 3.6.
> >
> > There is the separate issue of Meson dropping python 3.6 which motivates
> > Paolo's series. Again though, we don't have to increase our minimum meson
> > version, because meson is working today. It is our choice to to increase
> > it to use latest available meson features. At some point we can decide
> > what we have is good enough and we don't have to keep chasing the latest
> > features. Maybe we're not there yet, but we should think about when that
> > would be.
> 
> In the case of Meson, the main advantage is moving _all_ of the
> emulator configury out of the configure script.  This requires
> add_global_dependencies which was added in 0.63.  So in that case it
> is indeed mostly about shiny new features and it's not absolutely
> necessary.
> 
> In the case of Python the issue is not the interpreter per se, though
> there are a couple new feature in Python 3.7 that are quite nice (for
> example improved data classes[1] or context variables[2]). The main
> problem as far as I understood (and have seen in my experience) is
> linting tools. New versions fix bugs that caused false positives, but
> also become more strict at the same time. The newer versions at the
> same time are very quick at dropping support for old versions of
> Python; while older versions sometimes throw deprecation warnings on
> new versions of Python. This makes it very hard to support a single
> version of, say, mypy that works on all versions from RHEL8 and SLE15
> to Fedora 38 and Ubuntu 23.04.
> 
> [1] https://peps.python.org/pep-0557/
> [2] https://peps.python.org/pep-0567/
> 
> In fact this issue is the reason why RHEL9 does not package any of
> these tools and does not run them as part of building RPMs even though
> in principle it would be a good idea; it's too much of a pain to have
> a single version that works across all the packages in the
> distribution.
> 
> Regarding your other suggestion:
> 
> > * For non-native library/applications dependancies we aim
> >   to support only the most recent distro version. Users
> >   of older distros may need to dynamically fetch newer
> >   deps.
> 
> I think this is a good idea, but one issue with "only supporting the
> most recent distro version" is SUSE. While the most recent version of
> SLE is about 5 years old, there is no next version in sight---SUSE
> instead is working on their "Adaptable Linux Platform", but it's still
> in the prototype stage[3]. So alternatively we could put a 4 or 5 year
> cutoff after which you need to fetch newer deps. Considering the
> delays between freeze and release of distros like RHEL or SLE, in
> practice we would probably keep Python versions supported for 6-7
> years.

Yeah, that kind of problem with very old SUSE would push towards
simply excluding the LTS distros, or excluding them if they're
older than N years, and expect users of such old distros to
download newer python modules, etc.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Kevin Wolf]

Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben:
> In the case of Python the issue is not the interpreter per se, though
> there are a couple new feature in Python 3.7 that are quite nice (for
> example improved data classes[1] or context variables[2]). The main
> problem as far as I understood (and have seen in my experience) is
> linting tools. New versions fix bugs that caused false positives, but
> also become more strict at the same time. The newer versions at the
> same time are very quick at dropping support for old versions of
> Python; while older versions sometimes throw deprecation warnings on
> new versions of Python. This makes it very hard to support a single
> version of, say, mypy that works on all versions from RHEL8 and SLE15
> to Fedora 38 and Ubuntu 23.04.

Why do we have to support a single version of mypy? What is wrong with
running an old mypy version with old Python version, and a newer mypy
with newer Python versions?

Sure, they will complain about different things, but it doesn't feel
that different from supporting multiple C compilers in various versions.

Kevin

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1577 bytes --]

Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto:

> Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben:
> > In the case of Python the issue is not the interpreter per se, though
> > there are a couple new feature in Python 3.7 that are quite nice (for
> > example improved data classes[1] or context variables[2]). The main
> > problem as far as I understood (and have seen in my experience) is
> > linting tools. New versions fix bugs that caused false positives, but
> > also become more strict at the same time. The newer versions at the
> > same time are very quick at dropping support for old versions of
> > Python; while older versions sometimes throw deprecation warnings on
> > new versions of Python. This makes it very hard to support a single
> > version of, say, mypy that works on all versions from RHEL8 and SLE15
> > to Fedora 38 and Ubuntu 23.04.
>
> Why do we have to support a single version of mypy? What is wrong with
> running an old mypy version with old Python version, and a newer mypy
> with newer Python versions?
>
> Sure, they will complain about different things, but it doesn't feel
> that different from supporting multiple C compilers in various versions.
>

It's more like having to support only C++03 on RHEL 8 and only C++20 in
Fedora 37, without even being able to use a preprocessor.

For example old versions might not understand some type annotations and
will fail mypy altogether, therefore even with newer versions you can't
annotate the whole source and have to fall back to non-strict mode.

Paolo


> Kevin
>
>

[-- Attachment #2: Type: text/html, Size: 2295 bytes --]

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Kevin Wolf]

Am 14.02.2023 um 21:52 hat Paolo Bonzini geschrieben:
> Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto:
> 
> > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben:
> > > In the case of Python the issue is not the interpreter per se, though
> > > there are a couple new feature in Python 3.7 that are quite nice (for
> > > example improved data classes[1] or context variables[2]). The main
> > > problem as far as I understood (and have seen in my experience) is
> > > linting tools. New versions fix bugs that caused false positives, but
> > > also become more strict at the same time. The newer versions at the
> > > same time are very quick at dropping support for old versions of
> > > Python; while older versions sometimes throw deprecation warnings on
> > > new versions of Python. This makes it very hard to support a single
> > > version of, say, mypy that works on all versions from RHEL8 and SLE15
> > > to Fedora 38 and Ubuntu 23.04.
> >
> > Why do we have to support a single version of mypy? What is wrong with
> > running an old mypy version with old Python version, and a newer mypy
> > with newer Python versions?
> >
> > Sure, they will complain about different things, but it doesn't feel
> > that different from supporting multiple C compilers in various versions.
> >
> 
> It's more like having to support only C++03 on RHEL 8 and only C++20 in
> Fedora 37, without even being able to use a preprocessor.
> 
> For example old versions might not understand some type annotations and
> will fail mypy altogether, therefore even with newer versions you can't
> annotate the whole source and have to fall back to non-strict mode.

Sure, using old versions restricts which features can be used. But I
suppose C++03 is (for the most part) a subset of C++20, and annotations
understood by old mypy are a subset of annotation understood by new
mypy? Which would make it something for the "shiny new feature"
department.

Don't get me wrong, I was among the first people advocating for using
mypy and I still think any improvement in type checking is something we
should take when we can. But while it may a very desirable new feature,
it still is one.

So I assume the real problem sits somewhere else? Or do we get lots of
places where new mypy requires something that old mypy complains about
and vice versa?

Kevin

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Tue, Feb 14, 2023 at 09:52:44PM +0100, Paolo Bonzini wrote:
> Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto:
> 
> > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben:
> > > In the case of Python the issue is not the interpreter per se, though
> > > there are a couple new feature in Python 3.7 that are quite nice (for
> > > example improved data classes[1] or context variables[2]). The main
> > > problem as far as I understood (and have seen in my experience) is
> > > linting tools. New versions fix bugs that caused false positives, but
> > > also become more strict at the same time. The newer versions at the
> > > same time are very quick at dropping support for old versions of
> > > Python; while older versions sometimes throw deprecation warnings on
> > > new versions of Python. This makes it very hard to support a single
> > > version of, say, mypy that works on all versions from RHEL8 and SLE15
> > > to Fedora 38 and Ubuntu 23.04.
> >
> > Why do we have to support a single version of mypy? What is wrong with
> > running an old mypy version with old Python version, and a newer mypy
> > with newer Python versions?
> >
> > Sure, they will complain about different things, but it doesn't feel
> > that different from supporting multiple C compilers in various versions.
> >
> 
> It's more like having to support only C++03 on RHEL 8 and only C++20 in
> Fedora 37, without even being able to use a preprocessor.
> 
> For example old versions might not understand some type annotations and
> will fail mypy altogether, therefore even with newer versions you can't
> annotate the whole source and have to fall back to non-strict mode.

In terms of back compatibility, is there a distinction to be
made between mypy compat and the python runtime compat ?

If we add annotations wanted by new mypy, and old mypy doesn't
understand them, that's not a huge problem. We can simply declare
that we don't support old mypy, and skip the validation tests if
old mypy is installed. The mypy results are targetted at upstream
maintainers primarily, not people consuming QEMU, unless they are
backporting huge amounts of code and need to validate it. IOW it
should be sufficient to test once with an arbitrary version of
mypy of our choosing.

If we add annotations wanted by new mypy, and old python runtime
barfs, then that's a significant problem, which would require us
to either bump the min python or avoid the new mypy annotations.



The same could be asked for the other linting tools we use like
pylint / flake8. Is it sufficient to declare a min versions for
those tools and skip the tests if not satisfied, while still
retaining ability to execute the code on 3.6 ?

Or are there some core python runtime features we also want to
take advantage of at the same time ?


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Feb 14, 2023 at 09:52:44PM +0100, Paolo Bonzini wrote:
>> Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto:
>> 
>> > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben:
>> > > In the case of Python the issue is not the interpreter per se, though
>> > > there are a couple new feature in Python 3.7 that are quite nice (for
>> > > example improved data classes[1] or context variables[2]). The main
>> > > problem as far as I understood (and have seen in my experience) is
>> > > linting tools. New versions fix bugs that caused false positives, but
>> > > also become more strict at the same time. The newer versions at the
>> > > same time are very quick at dropping support for old versions of
>> > > Python; while older versions sometimes throw deprecation warnings on
>> > > new versions of Python. This makes it very hard to support a single
>> > > version of, say, mypy that works on all versions from RHEL8 and SLE15
>> > > to Fedora 38 and Ubuntu 23.04.
>> >
>> > Why do we have to support a single version of mypy? What is wrong with
>> > running an old mypy version with old Python version, and a newer mypy
>> > with newer Python versions?
>> >
>> > Sure, they will complain about different things, but it doesn't feel
>> > that different from supporting multiple C compilers in various versions.
>> >
>> 
>> It's more like having to support only C++03 on RHEL 8 and only C++20 in
>> Fedora 37, without even being able to use a preprocessor.
>> 
>> For example old versions might not understand some type annotations and
>> will fail mypy altogether, therefore even with newer versions you can't
>> annotate the whole source and have to fall back to non-strict mode.
>
> In terms of back compatibility, is there a distinction to be
> made between mypy compat and the python runtime compat ?
>
> If we add annotations wanted by new mypy, and old mypy doesn't
> understand them, that's not a huge problem. We can simply declare
> that we don't support old mypy, and skip the validation tests if
> old mypy is installed.

In theory, type hints are transparent at run time.  In practice, use of
modern type hints is known to break 3.6 at run time.  I don't remember
the details offhand, but John should be able to dig them up if you're
interested.

So, it should not be a problem, but it is.

Checking types only with sufficiently new mypy still makes sense
regardless.

>                        The mypy results are targetted at upstream
> maintainers primarily, not people consuming QEMU, unless they are
> backporting huge amounts of code and need to validate it. IOW it
> should be sufficient to test once with an arbitrary version of
> mypy of our choosing.
>
> If we add annotations wanted by new mypy, and old python runtime
> barfs, then that's a significant problem, which would require us
> to either bump the min python or avoid the new mypy annotations.

Yup.

Avoiding new mypy leaves us with underpowered type checking.

If dropping 3.6 had serious drawbacks, then we'd have to weigh type
checking improvements against them.  However, the only drawback we can
see is "CentOS 8 provides no suitable Sphinx", which we can either
accept or mitigate, I think.

> The same could be asked for the other linting tools we use like
> pylint / flake8. Is it sufficient to declare a min versions for
> those tools and skip the tests if not satisfied, while still
> retaining ability to execute the code on 3.6 ?

Yes, let's try to narrow the version range for linting tools.

> Or are there some core python runtime features we also want to
> take advantage of at the same time ?

Those are gravy.

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Thu, Feb 16, 2023 at 02:46:18AM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Tue, Feb 14, 2023 at 09:52:44PM +0100, Paolo Bonzini wrote:
> >> Il mar 14 feb 2023, 18:26 Kevin Wolf <kwolf@redhat.com> ha scritto:
> >> 
> >> > Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben:
> >> > > In the case of Python the issue is not the interpreter per se, though
> >> > > there are a couple new feature in Python 3.7 that are quite nice (for
> >> > > example improved data classes[1] or context variables[2]). The main
> >> > > problem as far as I understood (and have seen in my experience) is
> >> > > linting tools. New versions fix bugs that caused false positives, but
> >> > > also become more strict at the same time. The newer versions at the
> >> > > same time are very quick at dropping support for old versions of
> >> > > Python; while older versions sometimes throw deprecation warnings on
> >> > > new versions of Python. This makes it very hard to support a single
> >> > > version of, say, mypy that works on all versions from RHEL8 and SLE15
> >> > > to Fedora 38 and Ubuntu 23.04.
> >> >
> >> > Why do we have to support a single version of mypy? What is wrong with
> >> > running an old mypy version with old Python version, and a newer mypy
> >> > with newer Python versions?
> >> >
> >> > Sure, they will complain about different things, but it doesn't feel
> >> > that different from supporting multiple C compilers in various versions.
> >> >
> >> 
> >> It's more like having to support only C++03 on RHEL 8 and only C++20 in
> >> Fedora 37, without even being able to use a preprocessor.
> >> 
> >> For example old versions might not understand some type annotations and
> >> will fail mypy altogether, therefore even with newer versions you can't
> >> annotate the whole source and have to fall back to non-strict mode.
> >
> > In terms of back compatibility, is there a distinction to be
> > made between mypy compat and the python runtime compat ?
> >
> > If we add annotations wanted by new mypy, and old mypy doesn't
> > understand them, that's not a huge problem. We can simply declare
> > that we don't support old mypy, and skip the validation tests if
> > old mypy is installed.
> 
> In theory, type hints are transparent at run time.  In practice, use of
> modern type hints is known to break 3.6 at run time.  I don't remember
> the details offhand, but John should be able to dig them up if you're
> interested.
> 
> So, it should not be a problem, but it is.

Ok, this is a pretty compelling motivating factor for dropping
3.6, as it is clear demonstration that we're being held back
by the unfortunate lack of runtime compatibility.

For most of the other problems we can simply mandate a new
enough version of the add on tool, but if using new mypy
requires annotations that break at runtime on 3.6 that's
a painful blocker.



With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[John Snow]

On Tue, Feb 14, 2023 at 12:26 PM Kevin Wolf <kwolf@redhat.com> wrote:
>
> Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben:
> > In the case of Python the issue is not the interpreter per se, though
> > there are a couple new feature in Python 3.7 that are quite nice (for
> > example improved data classes[1] or context variables[2]). The main
> > problem as far as I understood (and have seen in my experience) is
> > linting tools. New versions fix bugs that caused false positives, but
> > also become more strict at the same time. The newer versions at the
> > same time are very quick at dropping support for old versions of
> > Python; while older versions sometimes throw deprecation warnings on
> > new versions of Python. This makes it very hard to support a single
> > version of, say, mypy that works on all versions from RHEL8 and SLE15
> > to Fedora 38 and Ubuntu 23.04.
>
> Why do we have to support a single version of mypy? What is wrong with
> running an old mypy version with old Python version, and a newer mypy
> with newer Python versions?

Well, the problem is, ...

>
> Sure, they will complain about different things, but it doesn't feel
> that different from supporting multiple C compilers in various versions.

...well, it's this.

The first dimension of the test matrix is the version of mypy itself.
The second dimension of the test matrix is the version of Python mypy
runs under. A given version of mypy can run under multiple versions of
Python and may indeed have different behaviors under each.
The third dimension of this test matrix is the version(s) of Python
our code is targeting; for instance we configure mypy to understand
that we require Python 3.6.

Trying to cast the net wide on *all of these* gets tough; the very
latest versions of mypy don't support 3.6 at all, so you'll have cases
where the mypy that just so happens to come with your Fedora
installation just won't work properly with our code anymore, which has
to test under 3.6 appropriately.

In general, the majority of python package upstreams I am aware of
simply pin a python version and a mypy version and leave it at that.
Compatibility concerns for most of the upstreams just do not really
ever consider that you'd be running *and* testing against a large
spread of versions. I've gone the extra mile and I run mypy and pylint
under all versions of python from 3.6 to 3.11 to ensure that developer
workstations can run the same linting tests and feel assured that if
it passes locally, it will pass on the CI and vice-versa. Our matrix
is generally quite a bit larger than most upstreams, so I am playing a
lot of whack-a-mole to keep things functioning consistently across the
versions. I will probably even add support for Python 3.12 alpha
*soon* because it's already available in the Fedora repo, and it will
be good to find any compatibility issues before that version is
officially released. (I'll need to do this for the qemu.qmp package,
which should have support for 3.12 on the day 3.12 releases and not
sometime afterwards.)

I know the "check-tox" test was annoying upstream as it sometimes
turned yellow because a new pylint version was released, but that's
how I stay ahead of breaking changes before they're on local
workstations.

We could avoid at least one of the reasons for dropping 3.6 support by
saying "Tough cookies, you'll use precisely this version of mypy and
precisely this python interpreter, or you'll get nothing" and that
does relieve a huge amount of pressure as-is. But, as a courtesy, I do
go out of my way -- where possible -- to ensure that developers can
use whichever versions of tools their distro is providing them so that
they don't have to mess around with that stuff. Unfortunately, that
means when upstreams start dropping support for older things, it gets
hairier and quite a bit more painful.

I think 3.6 being the first version that offers full-throated type
hint support has unique pain in this circumstance because a lot of the
details did not get hammered out until 3.7 or later; in general the
amount of workarounds I have had to apply because something type
checks only in 3.7+ but not 3.6 is fairly high compared to the number
of times I've found that a different version was the odd one out. I
don't expect this to become a recurring desire where I start to whine
about old Python versions before our support window would otherwise
let me drop them; I think this is actually just a unique pain point of
this one version if we keep static typing.

That's the tooling issue, anyway.

>
> Kevin
>

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Feb 14, 2023 at 12:26 PM Kevin Wolf <kwolf@redhat.com> wrote:
>>
>> Am 14.02.2023 um 15:03 hat Paolo Bonzini geschrieben:
>> > In the case of Python the issue is not the interpreter per se, though
>> > there are a couple new feature in Python 3.7 that are quite nice (for
>> > example improved data classes[1] or context variables[2]). The main
>> > problem as far as I understood (and have seen in my experience) is
>> > linting tools. New versions fix bugs that caused false positives, but
>> > also become more strict at the same time. The newer versions at the
>> > same time are very quick at dropping support for old versions of
>> > Python; while older versions sometimes throw deprecation warnings on
>> > new versions of Python. This makes it very hard to support a single
>> > version of, say, mypy that works on all versions from RHEL8 and SLE15
>> > to Fedora 38 and Ubuntu 23.04.
>>
>> Why do we have to support a single version of mypy? What is wrong with
>> running an old mypy version with old Python version, and a newer mypy
>> with newer Python versions?
>
> Well, the problem is, ...
>
>>
>> Sure, they will complain about different things, but it doesn't feel
>> that different from supporting multiple C compilers in various versions.

Different C compilers often produce different, useful warnings, so
compiling with several of them can get you more useful warnings.

However, you get pretty much all the useful warnings just from the
latest versions, at least in my experience.  You keep the old ones
working not for getting additional useful warnings, but to compile on
additional hosts.  As the version range widens, warnings from old
versions might even become such an annoyance that switching them off
becomes sensible.

> ...well, it's this.
>
> The first dimension of the test matrix is the version of mypy itself.
> The second dimension of the test matrix is the version of Python mypy
> runs under. A given version of mypy can run under multiple versions of
> Python and may indeed have different behaviors under each.
> The third dimension of this test matrix is the version(s) of Python
> our code is targeting; for instance we configure mypy to understand
> that we require Python 3.6.
>
> Trying to cast the net wide on *all of these* gets tough; the very
> latest versions of mypy don't support 3.6 at all, so you'll have cases
> where the mypy that just so happens to come with your Fedora
> installation just won't work properly with our code anymore, which has
> to test under 3.6 appropriately.
>
> In general, the majority of python package upstreams I am aware of
> simply pin a python version and a mypy version and leave it at that.

When pretty much everybody else doesn't do something we're doing, there
are two obvious explanations: (1) we're so smart we discovered something
worth doing pretty much first, or (2) the thing we're doing is a bad
idea in general.

In this case, my money is on (2).  It may still be a good idea for us
(because "in general").  But that notion needs to be supported with
concrete reasons, such as ...

> Compatibility concerns for most of the upstreams just do not really
> ever consider that you'd be running *and* testing against a large
> spread of versions. I've gone the extra mile and I run mypy and pylint
> under all versions of python from 3.6 to 3.11 to ensure that developer
> workstations can run the same linting tests and feel assured that if
> it passes locally, it will pass on the CI and vice-versa. Our matrix
> is generally quite a bit larger than most upstreams, so I am playing a
> lot of whack-a-mole to keep things functioning consistently across the
> versions. I will probably even add support for Python 3.12 alpha
> *soon* because it's already available in the Fedora repo, and it will
> be good to find any compatibility issues before that version is
> officially released. (I'll need to do this for the qemu.qmp package,
> which should have support for 3.12 on the day 3.12 releases and not
> sometime afterwards.)
>
> I know the "check-tox" test was annoying upstream as it sometimes
> turned yellow because a new pylint version was released, but that's
> how I stay ahead of breaking changes before they're on local
> workstations.
>
> We could avoid at least one of the reasons for dropping 3.6 support by
> saying "Tough cookies, you'll use precisely this version of mypy and
> precisely this python interpreter, or you'll get nothing" and that
> does relieve a huge amount of pressure as-is.

... this one:

>                                               But, as a courtesy, I do
> go out of my way -- where possible -- to ensure that developers can
> use whichever versions of tools their distro is providing them so that
> they don't have to mess around with that stuff. Unfortunately, that
> means when upstreams start dropping support for older things, it gets
> hairier and quite a bit more painful.

I appreciate the courtesy.  But it feels like it's gotten rather costly,
and getting costlier.

> I think 3.6 being the first version that offers full-throated type
> hint support has unique pain in this circumstance because a lot of the
> details did not get hammered out until 3.7 or later; in general the
> amount of workarounds I have had to apply because something type
> checks only in 3.7+ but not 3.6 is fairly high compared to the number
> of times I've found that a different version was the odd one out. I
> don't expect this to become a recurring desire where I start to whine
> about old Python versions before our support window would otherwise
> let me drop them; I think this is actually just a unique pain point of
> this one version if we keep static typing.
>
> That's the tooling issue, anyway.

Perhaps dropping just 3.6 from the test matrix will get us back to a
reasonable cost / benefit ratio.  But I'd like to encourage you to have
a sober look at the test matrix, and not shy away from cutting more.

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Daniel P. Berrangé]

On Tue, Feb 14, 2023 at 03:03:54PM +0100, Paolo Bonzini wrote:
> On Tue, Feb 14, 2023 at 12:49 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > [quote]
> > The motivation for this series is that Python 3.6 was EOL at the end of
> > 2021; upstream tools are beginning to drop support for it, including
> > setuptools, pylint, mypy, etc. As time goes by, it becomes more
> > difficult to support and test against the full range of Python versions
> > that QEMU supports. The closer we get to Python 3.12, the harder it will
> > be to cover that full spread of versions.
> > [/quote]
> >
> > this is all about new/eol versions of software upstream, and I don't
> > think that's a justification. QEMU explicitly aims to use distro provided
> > versions and upstream EOL status is not relevant in that context. Even
> > if using "pip" to install it is possible to limit yourself to upstream
> > releases which still support 3.6.
> >
> > There is the separate issue of Meson dropping python 3.6 which motivates
> > Paolo's series. Again though, we don't have to increase our minimum meson
> > version, because meson is working today. It is our choice to to increase
> > it to use latest available meson features. At some point we can decide
> > what we have is good enough and we don't have to keep chasing the latest
> > features. Maybe we're not there yet, but we should think about when that
> > would be.
> 
> In the case of Meson, the main advantage is moving _all_ of the
> emulator configury out of the configure script.  This requires
> add_global_dependencies which was added in 0.63.  So in that case it
> is indeed mostly about shiny new features and it's not absolutely
> necessary.

I forgot to mention in my previous reply, I feel like the ability
to finish the configure -> meson conversion is a pretty compelling
motivation to adopt the new meson. The hybrid configure/meson
state we've been in has worked better than I expected it would
at first, but none the less, the more we can get out of configure
the better it will be for ongoing maint burden.

So yes, its shiny new features, but they're pretty compelling
features as they allow us to finish the job we started. We've
clear precedent all over QEMU codebase that half-finished
conversions harm our ability to maintain the project.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Feb 14, 2023 at 08:40:20AM +0100, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> [...]
>> 
>> > We don't have to drop python 3.6. It is a choice because
>> > of a desire to be able to use some shiny new python
>> > features without caring about back compat.
>> 
>> I read this on Friday, and decided to let it sit until after the
>> weekend.  Well, it's now Tuesday, and to be frank, it's still as
>> offensively flippant as it was on Friday.  It shows either ignorance of
>> or cavalier disregard for the sheer amount of work some of us have had
>> to put into keeping old versions of Python viable.
>> 
>> The latter would be quite unlike you, so it must be the former.
>
> I'm sorry, I don't mean it to be offensive. I'm genuinely not seeing
> from the descriptions in the series what the functional benefits are
> from dropping 3.6.

Apology accepted, and point well taken.

>> John has sunk *man-months* into keeping old versions of Python viable.
>> I've put in a lot less myself, but still enough to be painfully aware of
>> it.  I figure Cleber and Beraldo are somewhere in between
>> 
>> Insinuating John's proposal is motivated by "a desire to be able to use
>> some shiny new python features without caring about back compat"
>> disrespects all this work.
>
> I'm writing my comments based on what is described in the cover letter
> as the motivations for the change:
>
> [quote]
> The motivation for this series is that Python 3.6 was EOL at the end of
> 2021; upstream tools are beginning to drop support for it, including
> setuptools, pylint, mypy, etc. As time goes by, it becomes more
> difficult to support and test against the full range of Python versions
> that QEMU supports. The closer we get to Python 3.12, the harder it will
> be to cover that full spread of versions.
> [/quote]
>
> this is all about new/eol versions of software upstream, and I don't
> think that's a justification. QEMU explicitly aims to use distro provided
> versions and upstream EOL status is not relevant in that context. Even
> if using "pip" to install it is possible to limit yourself to upstream
> releases which still support 3.6.
>
> There is the separate issue of Meson dropping python 3.6 which motivates
> Paolo's series. Again though, we don't have to increase our minimum meson
> version, because meson is working today. It is our choice to to increase
> it to use latest available meson features. At some point we can decide
> what we have is good enough and we don't have to keep chasing the latest
> features. Maybe we're not there yet, but we should think about when that
> would be.
>
> [quote]
> The qemu.qmp library and the avocado testing framework both have
> motivations for dropping 3.6 support, but are committed to not doing so
> until QEMU drops support.
> [/quote]
>
> I suspect that this is more of a driver for the drop of 3.6, but I
> don't see any details.
>
> IOW overall justification come across as wanting to use new features,
> and follow upstream EOL, without any real detail of what we're going
> to gain from a functional POV.

I think what we gain is there: "As time goes by, it becomes more
difficult to support and test against the full range of Python versions
that QEMU supports. The closer we get to Python 3.12, the harder it will
be to cover that full spread of versions."  In other words, the gain is
"we make something that's hard and getting harder easier".

What isn't in the cover letter, only in commit message 6+7/7, i.e. this
patch and the next one, is what we pay for it: basically nothing (next
patch's commit message) except for the issue of Sphinx in CentOS 8 (this
patch's commit message).  Putting at least a summary it in the cover
letter would have been better.

John did explain this again and in more detail in reply to Peter's "This
confuses me.  We work fine with Python 3.6 today."  Quote:

    That won't last - Many tools such as mypy, pylint and flake8 which I use to
    manage our python codebase have been dropping support for 3.6 and I've had
    to implement an increasing number of workarounds to help keep it possible
    to test 3.6 via CI while also ensuring our newest platforms work as dev
    environments.

    Our testing matrix for Python is novel and thorough enough that it's
    revealed  several bugs in other downstream Python distributions for Debian
    and Fedora, and dozens of bugs for the linters themselves.

    I'm concerned that when 3.7 is EOL'd in just a few months that the support
    and testing gap is going to get even uglier.

    [...]

    The argument I'm making is:

    - CentOS 8 is a supported build platform
    - All platforms *do* have a Python modern enough to allow us to drop 3.6
    - CentOS's repo version of sphinx is hardcoded to use the older 3.6, though
    - You expressed a preference to me in the past to NOT use a pip installed
    version of sphinx in preference to the system version in "configure"
    - It's still possible to build docs on CentOS 8 after this patchset, you
    just need a pip version.
    - We've used the justification that our build platform promise does not
    necessarily extend to docs and tests in the past.
    - So just skip docs building for CentOS 8, only in the CI.

    If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want
    to go back to discussing the possibility of using sphinx versions from pip.

You even quoted most of it in your message :)

The second paragraph is evidence we've been doing something basically
nobody else does.

>> We should have a sober discussion on which versions of Python to work
>> with, and the tradeoffs involved.  But before I engage in that, I insist
>> on resetting the frame: no, this is not about shiny, new Python
>> features.  It is about stopping the bleeding.  It is about reducing what
>> feels more and more like bullshit work to me, so we can actually
>> accomplish stuff that matters.
>
> Every applications developer chooses an arbitrary cut off points for
> minimum software versions, depending on their particular needs. With
> our support policy we tried to express a reasonable tradeoff between
> keeping back compat, and being able to adopt new features.
>
> Obviously that tradeoff is not currently looking acceptable on the
> python side, but it is not clear why that is ?
>
> Can someone simply explain what we wil see as the benefit from dropping
> 3.6 / adopting 3.7 as the baseline ? 

Let me start with the costs, then move on to the benefits.

Every single supported build platform already provides Python >= 3.7,
and all the Python tools we use continue to work with it, with one
exception: Sphinx on CentOS 8.

Solutions for this issue proposed so far:

1. Do nothing.  To build documentation on CentOS 8, you have to pip
   install a working version of Sphinx, which is easy enough.  As long
   as we don't in CI, we have to disable building docs there (this
   patch).

2. Have the build system pip install it for you.  A bit of a paradigm
   shift: so far our build system doesn't install anything.

3. Split the Python dependency for Sphinx off the general one.  On
   CentOS 8, we'd use 3.6 to run Sphinx, and 3.7+ for everything else.
   On all other systems, we'd use the same 3.7+ for everything.

So the cost is our choice of "you have to pip install Sphinx to build
docs on CentOS 8" / "build system pip installs Sphinx" / "special Python
dependency for Sphinx".  Feels quite modest to me.

The main benefit is we reduce the bleeding caused by trying to support
Python all the way back to 3.6 when basically nobody else does.  Linters
in particular.  How much of a benefit is this?  I'd like you all to
trust John it's plenty big enough to justify the (modest!) cost.  He's
been doing much of the bleeding, after all.

New features provided by 3.7 will be nice to have (especially for
typing), but that's not why we're doing this.

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 6710 bytes --]

On Fri, Feb 10, 2023, 11:32 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Fri, 10 Feb 2023 at 16:01, John Snow <jsnow@redhat.com> wrote:
> > On Fri, Feb 10, 2023, 5:41 AM Peter Maydell <peter.maydell@linaro.org>
> wrote:
> >> On Fri, 10 Feb 2023 at 00:31, John Snow <jsnow@redhat.com> wrote:
> >> This confuses me. We work fine with Python 3.6 today.
> >
> >
> > That won't last - Many tools such as mypy, pylint and flake8 which I use
> to manage our python codebase have been dropping support for 3.6 and I've
> had to implement an increasing number of workarounds to help keep it
> possible to test 3.6 via CI while also ensuring our newest platforms work
> as dev environments.
>
> Something somewhere seems kind of out-of-sync here. Either:
>  * Python is deprecating old versions too quickly and
>    dropping support for them too fast
>

I think this is true. The Python upstream ecosystem moves extremely fast
and breaks backwards compatibility for tooling often. It is sometimes tough
to mediate that flow rate difference.

Part of the reason I added that "check-tox" CI test that is currently
optional to run is specifically so it can find problems with cutting edge
tooling - and it happens often. (It seems like every freeze, I find out a
new upstream version of something breaks the testing, like clockwork.)

On the other hand, Python 3.6 did launch some six years ago - 2016-12-22.
It's probably reasonable most cutting edge tooling for f37/f38 no longer
supports developing for a version this old.

I can't deny that the Python package ecosystem is too cavalier about
dropping for support for things too quickly, though.

 * CentOS is retaining old versions of Python when it needs to
>    ship newer ones
>

I'm not clear on the exact policies for Stream 8/9, but in our traditional
releases we'd not ever upgrade this - we'd only do micro updates and
backports.

Instead, Stream 8/9 (and OpenSUSE too) offer optional Python versions that
install side-by-side.

So sphinx (as packaged as an rpm) continues to use the platform Python -
3.6 - but you *can* launch the "system sphinx" with your newer Python
interpreter - if the version of sphinx shipped happens to cope with a newer
interpreter.

(If you run `python3.7 -m sphinx`, for example - it might work, but only if
your rpm-packaged sphinx isn't using any features of Python that were
dropped in 3.7. I'd wager it probably does work most of the time.)

 * Our policy for what distros we support is overly lax
>    and causing us to try to support ancient platforms
>    that we shouldn't be trying to support
>

Possibly. Part of the idea behind RHEL8/CentOS8 is that you'll get security
fixes, but it should otherwise be a stable platform for production.

Supporting this as a development environment for cutting edge versions does
feel like a mismatch in visions at times.

RHEL8 launched 2018-11-14, so is it reasonable to expect that cutting edge
versions of QEMU 4 years in the future will build on it no problem?

Meanwhile, OpenSUSE 15.4 released just eight months ago and still ships
Python 3.6 as its default/platform Python. They don't ship a distro with a
modern Python as default at all... Even the upcoming 15.5 release still
appears to use Python 3.6 as the platform default.

(Like RHEL, they offer optional additional interpreters if you want to run
upstream software written within the last two years.)

At least QEMU's stated support for RHEL8 runs out in another year or so. I
wonder what OpenSUSE is planning to do.


>
>
> because "use the system version of foo" should not be a big
> deal -- it's not like we're trying to support decades-old
> hosts here: Centos 8 was released in 2019, which is less than
> five years ago.
>

Yeah, I have nothing cute to say to this. It's unfortunate that it's hard
to support RHEL8 and Fedora 37 simultaneously as development environments
for our devs.

I do take great pains to reduce disruptions to everyone as I keep the
python code afloat. I hope I have been successful these past few years.

However, there are packaged versions of modern Python available for any
platform we support that otherwise defaults to 3.6, so it's not a huge
affair to continue supporting these older platforms.

It'll just be as if you need one extra dependency on those systems. (Maybe
two if you want to build docs.)

I think that trade-off is fair; as it's only a one-time expense to
developers still working on older platforms.


> > The argument I'm making is:
> >
> > - CentOS 8 is a supported build platform
> > - All platforms *do* have a Python modern enough to allow us to drop 3.6
> > - CentOS's repo version of sphinx is hardcoded to use the older 3.6,
> though
> > - You expressed a preference to me in the past to NOT use a pip
> installed version of sphinx in preference to the system version in
> "configure"
> > - It's still possible to build docs on CentOS 8 after this patchset, you
> just need a pip version.
> > - We've used the justification that our build platform promise does not
> necessarily extend to docs and tests in the past.
> > - So just skip docs building for CentOS 8, only in the CI.
> >
> > If you believe docs in CI for CentOS 8 is a hard deal breaker, then I
> want to go back to discussing the possibility of using sphinx versions from
> pip.
>
> If Python 3.6 is EOL, shouldn't CentOS have shipped an updated
> version of Sphinx to go with their updated Python ?
>

It's EOL upstream, but I suppose the RHEL/CentOS view is that we'll support
it with our own security fixes/backports past its upstream death. I don't
know the exact versioning policies, but generally they never do a version
bump. I don't expect them to ever upgrade the platform default python from
3.6, because it will likely involve hundreds of version bumps to other
packages as a result.


> I'm not super-happy about dropping the docs build requirement,
> but I guess it's probably the least-worst choice.
>

I think Paolo is advocating for allowing the build to use a pip-installed
version of sphinx, and configuring our docker containers to set up that
environment.

We'd keep CentOS 8 building docs just fine as a result.

It would mean that individual contributors that wanted to build docs on
CentOS 8 would need to install a sphinx version with pip, though.

(The problem with just allowing sphinx to be a black box and continuing to
happily use the 3.6-based versions is that we are using QAPIDoc extensions
from our own codebase, which would lock those to 3.6. A big motivator for
Markus is dropping some 3.6 kludges we're carrying for qapi, so I looked to
the opposite solution - nudging Sphinx forward instead.)


> thanks
> -- PMM
>

Sorry for the essay!
--js

>

[-- Attachment #2: Type: text/html, Size: 10072 bytes --]

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Peter Maydell]

On Fri, 10 Feb 2023 at 17:55, John Snow <jsnow@redhat.com> wrote:
> (The problem with just allowing sphinx to be a black box and
> continuing to happily use the 3.6-based versions is that we are
> using QAPIDoc extensions from our own codebase, which would lock
> those to 3.6. A big motivator for Markus is dropping some 3.6
> kludges we're carrying for qapi, so I looked to the opposite
> solution - nudging Sphinx forward instead.)

Mmm. Where on the pain spectrum does "allow python 3.8
because CentosOS ships that, except that where our python
code gets run via Sphinx that part of the codebase must
retain 3.6 compatibility" sit ?

thanks
-- PMM

Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

Il ven 10 feb 2023, 19:09 Peter Maydell <peter.maydell@linaro.org> ha
scritto:

> On Fri, 10 Feb 2023 at 17:55, John Snow <jsnow@redhat.com> wrote:
> > (The problem with just allowing sphinx to be a black box and
> > continuing to happily use the 3.6-based versions is that we are
> > using QAPIDoc extensions from our own codebase, which would lock
> > those to 3.6. A big motivator for Markus is dropping some 3.6
> > kludges we're carrying for qapi, so I looked to the opposite
> > solution - nudging Sphinx forward instead.)
>
> Mmm. Where on the pain spectrum does "allow python 3.8
> because CentosOS ships that, except that where our python
> code gets run via Sphinx that part of the codebase must
> retain 3.6 compatibility" sit ?
>

As far as the build system is concerned no changes are required, however
Python 3.7 support was requested by Markus for scripts/qapi/ for quite some
time. So it would not be painful to implement but it would remove most of
the benefits of dropping support for 3.6.

Paolo


> thanks
> -- PMM
>
>

[-- Attachment #2: Type: text/html, Size: 1764 bytes --]

[PATCH v2 7/7] Python: Drop support for Python 3.6

[John Snow]

Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
begun dropping support for this version and it is becoming more
cumbersome to support. Avocado-framework and qemu.qmp each have their
own reasons for wanting to drop Python 3.6, but won't until QEMU does.

Versions of Python available in our supported build platforms as of today,
with optional versions available in parentheses:

openSUSE Leap 15.4: 3.6.15 (3.9.10, 3.10.2)
CentOS Stream 8:    3.6.8  (3.8.13, 3.9.16)
CentOS Stream 9:    3.9.13
Fedora 36:          3.10
Fedora 37:          3.11
Debian 11:          3.9.2
Alpine 3.14, 3.15:  3.9.16
Alpine 3.16, 3.17:  3.10.10
Ubuntu 20.04 LTS:   3.8.10
Ubuntu 22.04 LTS:   3.10.4
NetBSD 9.3:         3.9.13*
FreeBSD 12.4:       3.9.16
FreeBSD 13.1:       3.9.16
OpenBSD 7.2:        3.9.16

Note: Our VM tests install 3.7 specifically for freebsd and netbsd; the
default for "python" or "python3" in FreeBSD is 3.9.16. NetBSD does not
appear to have a default meta-package, but offers several options, the
lowest of which is 3.7.15. "python39" appears to be a pre-requisite to
one of the other packages we request in tests/vm/netbsd.

Since it is safe to under our supported platform policy, bump our
minimum supported version of Python to 3.7.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py             |  4 ++--
 configure                |  8 ++++----
 python/Makefile          | 10 +++++-----
 python/setup.cfg         |  7 +++----
 python/tests/minreqs.txt |  2 +-
 scripts/qapi/mypy.ini    |  2 +-
 6 files changed, 16 insertions(+), 17 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index 73a287a4f27..d40448f35d9 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -37,9 +37,9 @@
 # In newer versions of Sphinx this will display nicely; in older versions
 # Sphinx will also produce a Python backtrace but at least the information
 # gets printed...
-if sys.version_info < (3,6):
+if sys.version_info < (3,7):
     raise ConfigError(
-        "QEMU requires a Sphinx that uses Python 3.6 or better\n")
+        "QEMU requires a Sphinx that uses Python 3.7 or better\n")
 
 # The per-manual conf.py will set qemu_docdir for a single-manual build;
 # otherwise set it here if this is an entire-manual-set build.
diff --git a/configure b/configure
index bf512273f44..de27994ddc8 100755
--- a/configure
+++ b/configure
@@ -594,9 +594,9 @@ esac
 
 
 check_py_version() {
-    # We require python >= 3.6.
+    # We require python >= 3.7.
     # NB: a True python conditional creates a non-zero return code (Failure)
-    "$1" -c 'import sys; sys.exit(sys.version_info < (3,6))'
+    "$1" -c 'import sys; sys.exit(sys.version_info < (3,7))'
 }
 
 python=
@@ -604,7 +604,7 @@ first_python=
 explicit_python=no
 # A bare 'python' is traditionally python 2.x, but some distros
 # have it as python 3.x, so check in both places.
-for binary in "${PYTHON-python3}" python python3.{11..6}
+for binary in "${PYTHON-python3}" python python3.{11..7}
 do
     if has "$binary"
     then
@@ -1057,7 +1057,7 @@ then
 fi
 
 if ! check_py_version "$python"; then
-  error_exit "Cannot use '$python', Python >= 3.6 is required." \
+  error_exit "Cannot use '$python', Python >= 3.7 is required." \
              "Use --python=/path/to/python to specify a supported Python." \
              "Maybe try:" \
              "  openSUSE Leap 15.3+: zypper install python39" \
diff --git a/python/Makefile b/python/Makefile
index c5bd6ff83ac..f660d999143 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -9,14 +9,14 @@ help:
 	@echo "make check-minreqs:"
 	@echo "    Run tests in the minreqs virtual environment."
 	@echo "    These tests use the oldest dependencies."
-	@echo "    Requires: Python 3.6"
-	@echo "    Hint (Fedora): 'sudo dnf install python3.6'"
+	@echo "    Requires: Python 3.7"
+	@echo "    Hint (Fedora): 'sudo dnf install python3.7'"
 	@echo ""
 	@echo "make check-tox:"
 	@echo "    Run tests against multiple python versions."
 	@echo "    These tests use the newest dependencies."
-	@echo "    Requires: Python 3.6 - 3.10, and tox."
-	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.10'"
+	@echo "    Requires: Python 3.7 - 3.11, and tox."
+	@echo "    Hint (Fedora): 'sudo dnf install python3-tox python3.11'"
 	@echo "    The variable QEMU_TOX_EXTRA_ARGS can be use to pass extra"
 	@echo "    arguments to tox".
 	@echo ""
@@ -58,7 +58,7 @@ pipenv check-pipenv:
 min-venv: $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate
 $(QEMU_MINVENV_DIR) $(QEMU_MINVENV_DIR)/bin/activate: setup.cfg tests/minreqs.txt
 	@echo "VENV $(QEMU_MINVENV_DIR)"
-	@python3.6 -m venv $(QEMU_MINVENV_DIR)
+	@python3.7 -m venv $(QEMU_MINVENV_DIR)
 	@(								\
 		echo "ACTIVATE $(QEMU_MINVENV_DIR)";			\
 		. $(QEMU_MINVENV_DIR)/bin/activate;			\
diff --git a/python/setup.cfg b/python/setup.cfg
index 9e923d97628..1e8392a045c 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -14,7 +14,6 @@ classifiers =
     Natural Language :: English
     Operating System :: OS Independent
     Programming Language :: Python :: 3 :: Only
-    Programming Language :: Python :: 3.6
     Programming Language :: Python :: 3.7
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
@@ -23,7 +22,7 @@ classifiers =
     Typing :: Typed
 
 [options]
-python_requires = >= 3.6
+python_requires = >= 3.7
 packages =
     qemu.qmp
     qemu.machine
@@ -76,7 +75,7 @@ exclude = __pycache__,
 
 [mypy]
 strict = True
-python_version = 3.6
+python_version = 3.7
 warn_unused_configs = True
 namespace_packages = True
 warn_unused_ignores = False
@@ -158,7 +157,7 @@ multi_line_output=3
 # of python available on your system to run this test.
 
 [tox:tox]
-envlist = py36, py37, py38, py39, py310, py311
+envlist = py37, py38, py39, py310, py311
 skip_missing_interpreters = true
 
 [testenv]
diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
index dfb8abb155f..55cc6b41d85 100644
--- a/python/tests/minreqs.txt
+++ b/python/tests/minreqs.txt
@@ -1,5 +1,5 @@
 # This file lists the ***oldest possible dependencies*** needed to run
-# "make check" successfully under ***Python 3.6***. It is used primarily
+# "make check" successfully under ***Python 3.7***. It is used primarily
 # by GitLab CI to ensure that our stated minimum versions in setup.cfg
 # are truthful and regularly validated.
 #
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 66253564297..3463307ddc7 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -1,7 +1,7 @@
 [mypy]
 strict = True
 disallow_untyped_calls = False
-python_version = 3.6
+python_version = 3.7
 
 [mypy-qapi.schema]
 disallow_untyped_defs = False
-- 
2.39.0

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Markus Armbruster]

First, a plea.  Supporting 3.6 has made a few of us prisoners dragging
ball and chain.  So, please, *please* let us cut of these leg irons!

The series does not include follow-up cleanups.  Fine with me.

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[John Snow]

On Thu, Feb 9, 2023 at 7:31 PM John Snow <jsnow@redhat.com> wrote:
>
> Howdy, this series increases our minimum python version to 3.7.
>
> CI: https://gitlab.com/jsnow/qemu/-/pipelines/771780626
>     (All green!)
> GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37
>
> Patches 1 and 2 are loose pre-requisites; I'd like to merge them into
> qemu.git within the week whether or not we take this series. I'd
> appreciate an "ACK" on those specifically. They're just riding along
> here because they make this series a bit nicer.
>
> Patches 3-6 are the hard pre-requisites, and 7 does the dirty work.
>
> The motivation for this series is that Python 3.6 was EOL at the end of
> 2021; upstream tools are beginning to drop support for it, including
> setuptools, pylint, mypy, etc. As time goes by, it becomes more
> difficult to support and test against the full range of Python versions
> that QEMU supports. The closer we get to Python 3.12, the harder it will
> be to cover that full spread of versions.
>
> The qemu.qmp library and the avocado testing framework both have
> motivations for dropping 3.6 support, but are committed to not doing so
> until QEMU drops support.
>
> So, I'd like to talk about doing it.
>
> V2:
> - Added R-Bs to patch 1
> - Updated commit message for patch 7 with explicit version info
> - Added DO-NOT-MERGE to patch 5's title
> - Tested tests/vm/freebsd, netbsd, and openbsd in addition to full CI
>
> RFC:
>  - Patch 5 is just a proof-of-concept; we need to update lcitool instead.
>  - Cleber, I need to update your ansible scripts. How do I test them?
>
> Thanks!
> --js
>
> John Snow (7):
>   python: support pylint 2.16
>   python: drop pipenv

Hi, I've staged these first two patches to my Python branch.

(Kevin, Hanna; is that acceptable? I touch some iotests to do some
trivial linting whack-a-mole.)

--js

>   configure: Look for auxiliary Python installations
>   configure: Add nice hint to Python failure message
>   DO-NOT-MERGE: testing: Add Python >= 3.7 to Centos, OpenSuSE
>   CI: Stop building docs on centos8
>   Python: Drop support for Python 3.6
>
>  docs/conf.py                                  |   4 +-
>  python/README.rst                             |   3 -
>  configure                                     |  40 +-
>  .gitlab-ci.d/buildtest.yml                    |   2 +-
>  .gitlab-ci.d/static_checks.yml                |   4 +-
>  python/.gitignore                             |   4 +-
>  python/Makefile                               |  57 ++-
>  python/Pipfile                                |  13 -
>  python/Pipfile.lock                           | 347 ------------------
>  python/qemu/qmp/protocol.py                   |   2 +-
>  python/qemu/qmp/qmp_client.py                 |   2 +-
>  python/qemu/utils/qemu_ga_client.py           |   6 +-
>  python/setup.cfg                              |  11 +-
>  python/tests/minreqs.txt                      |  45 +++
>  scripts/qapi/mypy.ini                         |   2 +-
>  tests/docker/dockerfiles/centos8.docker       |   1 +
>  tests/docker/dockerfiles/opensuse-leap.docker |   1 +
>  tests/docker/dockerfiles/python.docker        |   1 -
>  tests/qemu-iotests/iotests.py                 |   4 +-
>  .../tests/migrate-bitmaps-postcopy-test       |   2 +-
>  20 files changed, 135 insertions(+), 416 deletions(-)
>  delete mode 100644 python/Pipfile
>  delete mode 100644 python/Pipfile.lock
>  create mode 100644 python/tests/minreqs.txt
>
> --
> 2.39.0
>
>

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Kevin Wolf]

Am 14.02.2023 um 19:35 hat John Snow geschrieben:
> On Thu, Feb 9, 2023 at 7:31 PM John Snow <jsnow@redhat.com> wrote:
> >
> > Howdy, this series increases our minimum python version to 3.7.
> >
> > CI: https://gitlab.com/jsnow/qemu/-/pipelines/771780626
> >     (All green!)
> > GL: https://gitlab.com/jsnow/qemu/-/commits/python-require-37
> >
> > Patches 1 and 2 are loose pre-requisites; I'd like to merge them into
> > qemu.git within the week whether or not we take this series. I'd
> > appreciate an "ACK" on those specifically. They're just riding along
> > here because they make this series a bit nicer.
> >
> > Patches 3-6 are the hard pre-requisites, and 7 does the dirty work.
> >
> > The motivation for this series is that Python 3.6 was EOL at the end of
> > 2021; upstream tools are beginning to drop support for it, including
> > setuptools, pylint, mypy, etc. As time goes by, it becomes more
> > difficult to support and test against the full range of Python versions
> > that QEMU supports. The closer we get to Python 3.12, the harder it will
> > be to cover that full spread of versions.
> >
> > The qemu.qmp library and the avocado testing framework both have
> > motivations for dropping 3.6 support, but are committed to not doing so
> > until QEMU drops support.
> >
> > So, I'd like to talk about doing it.
> >
> > V2:
> > - Added R-Bs to patch 1
> > - Updated commit message for patch 7 with explicit version info
> > - Added DO-NOT-MERGE to patch 5's title
> > - Tested tests/vm/freebsd, netbsd, and openbsd in addition to full CI
> >
> > RFC:
> >  - Patch 5 is just a proof-of-concept; we need to update lcitool instead.
> >  - Cleber, I need to update your ansible scripts. How do I test them?
> >
> > Thanks!
> > --js
> >
> > John Snow (7):
> >   python: support pylint 2.16
> >   python: drop pipenv
> 
> Hi, I've staged these first two patches to my Python branch.
> 
> (Kevin, Hanna; is that acceptable? I touch some iotests to do some
> trivial linting whack-a-mole.)

Yes, of course.

Kevin

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Markus Armbruster]

The discussion under PATCH 6 makes me think there's a bit of confusion
about the actual impact of dropping support for Python 3.6.  Possibly
because it's spelled out in the commit message of PATCH 7.  Let me
summarize it in one sentence:

    *** All supported host systems continue to work ***

Evidence: CI remains green.

On some supported host systems, different packages need to be installed.
On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
instead of 3.6.8.  Let me stress again: same repository, different
package.  No downsides I can see.

The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
version of Sphinx that works with Python 3.7 or newer.  This series
proposes to simply stop building the docs there, unless the user
provides a suitable version of Sphinx (which is easy enough with pip).
That's PATCH 7.

Paolo thinks we could also stay on 3.6 just for Sphinx, with a bit of
care.  That's constructive patch review.

All the angst about CentOS falling off the end of our "supported build
platforms" list is not actually warranted by this series :)

Some of the discussion is valuable regardless.  For instance, the points
Daniel made about traditional distro dependencies (still used with C
tool chains), and distro-agnostic dependencies baked into modern tool
chains.  This series is not (and does not try to be) the final answer to
the question how to handle Python dependencies in QEMU.

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Peter Maydell]

On Wed, 15 Feb 2023 at 19:05, Markus Armbruster <armbru@redhat.com> wrote:
>
> The discussion under PATCH 6 makes me think there's a bit of confusion
> about the actual impact of dropping support for Python 3.6.  Possibly
> because it's spelled out in the commit message of PATCH 7.  Let me
> summarize it in one sentence:
>
>     *** All supported host systems continue to work ***
>
> Evidence: CI remains green.
>
> On some supported host systems, different packages need to be installed.
> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> instead of 3.6.8.  Let me stress again: same repository, different
> package.  No downsides I can see.

Yes; I never had any issues with this part of it. If there was
a "Sphinx that also used that Python" in that repo, the answer
would be easy.

> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> version of Sphinx that works with Python 3.7 or newer.  This series
> proposes to simply stop building the docs there, unless the user
> provides a suitable version of Sphinx (which is easy enough with pip).
> That's PATCH 7.

Yes; this brings CentOS 8 down from "fully supported" to "kinda
supported but not for everything", which is less than ideal.
I think the level of not-idealness of that side of the scales
is probably clear enough. The difficulty I think for those who
haven't had their arms deep in QEMU's Python code is not having
the background info to be able to weigh up how heavy the other side
of the tradeoff scales is (since the naive "well, just keep writing
Python 3.6 for the moment" take is clearly wrong).

thanks
-- PMM

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 15 Feb 2023 at 19:05, Markus Armbruster <armbru@redhat.com> wrote:
>>
>> The discussion under PATCH 6 makes me think there's a bit of confusion
>> about the actual impact of dropping support for Python 3.6.  Possibly
>> because it's spelled out in the commit message of PATCH 7.  Let me
>> summarize it in one sentence:
>>
>>     *** All supported host systems continue to work ***
>>
>> Evidence: CI remains green.
>>
>> On some supported host systems, different packages need to be installed.
>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
>> instead of 3.6.8.  Let me stress again: same repository, different
>> package.  No downsides I can see.
>
> Yes; I never had any issues with this part of it. If there was
> a "Sphinx that also used that Python" in that repo, the answer
> would be easy.
>
>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
>> version of Sphinx that works with Python 3.7 or newer.  This series
>> proposes to simply stop building the docs there, unless the user
>> provides a suitable version of Sphinx (which is easy enough with pip).
>> That's PATCH 7.
>
> Yes; this brings CentOS 8 down from "fully supported" to "kinda
> supported but not for everything", which is less than ideal.

I acknowledge there's a difference between "you need to dnf install
python-sphinx to be able to build docs" and "you need to pip install
sphinx to be able to build docs", and that the difference is negligible
in some scenarios, and a show stopper in others.  I wasn't fully aware
of the latter.

> I think the level of not-idealness of that side of the scales
> is probably clear enough. The difficulty I think for those who
> haven't had their arms deep in QEMU's Python code is not having
> the background info to be able to weigh up how heavy the other side
> of the tradeoff scales is (since the naive "well, just keep writing
> Python 3.6 for the moment" take is clearly wrong).

Fair point.  I hope the situation is more clear now.

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Thomas Huth]

On 15/02/2023 20.05, Markus Armbruster wrote:
> The discussion under PATCH 6 makes me think there's a bit of confusion
> about the actual impact of dropping support for Python 3.6.  Possibly
> because it's spelled out in the commit message of PATCH 7.  Let me
> summarize it in one sentence:
> 
>      *** All supported host systems continue to work ***
> 
> Evidence: CI remains green.

The CI remains green since one of the patches disabled the building of the 
docs on CentOS 8. That's not how I'd describe "continue to work", at least 
not in the same extend as before.

> On some supported host systems, different packages need to be installed.
> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> instead of 3.6.8.  Let me stress again: same repository, different
> package.  No downsides I can see.
> 
> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> version of Sphinx that works with Python 3.7 or newer.  This series
> proposes to simply stop building the docs there, unless the user
> provides a suitable version of Sphinx (which is easy enough with pip).

I think we've all understood that. The thing that you obviously did not 
understood: This breaks our support statement.
I'm pretty sure that you could also build the whole QEMU suite successfully 
on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a 
newer version of GCC and some of the required libraries first. But that's 
not how we understand our support statement.

Sure, you can argue that you can use "pip install" to get a newer version of 
Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is 
that really that much different from installing a newer version of GCC and 
libraries on an ancient distro that we do not officially support anymore? 
I'd say no. You also have to consider that not every build host has access 
to the internet, maybe some companies only have an internal mirror of the 
distro packages in their intranet (I remember some discussion about such a 
case in the past) - so while you were perfectly fine to build the whole of 
QEMU on a CentOS 8 there before this change, you could now not build parts 
of QEMU anymore there due to the missing possibility to run "pip install" 
without full internet connection.

And sure, you can argue that it's "just" the documentation. But IMHO that's 
still an essential part of the QEMU build, and it used to work before, so it 
feels wrong to cut that now out. And also, if we start with the 
documentation now, what's next? If for example scripts/shaderinclude.py 
stops working with Python 3.6, do we then simply say: "Oh, it's fine, you 
can still build all the other targets that work without this script, just 
not the ones anymore that need it"?

> All the angst about CentOS falling off the end of our "supported build
> platforms" list is not actually warranted by this series :)

Using the term "angst" for the concerns of your fellows here is quite 
cheeky. It's not about "angst", it's about a discussion that our support 
policy might need to be adjusted if we do this step. So instead of writing 
such sentences, I'd rather would like to see you posting a patch for 
docs/about/build-platforms.rst for constructive further discussion instead.

  Thanks,
   Thomas

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Markus Armbruster]

Thomas Huth <thuth@redhat.com> writes:

> On 15/02/2023 20.05, Markus Armbruster wrote:
>> The discussion under PATCH 6 makes me think there's a bit of confusion
>> about the actual impact of dropping support for Python 3.6.  Possibly
>> because it's spelled out in the commit message of PATCH 7.  Let me
>> summarize it in one sentence:
>> 
>>      *** All supported host systems continue to work ***
>> 
>> Evidence: CI remains green.
>
> The CI remains green since one of the patches disabled the building of the 
> docs on CentOS 8. That's not how I'd describe "continue to work", at least 
> not in the same extend as before.

Thus the exception ...

>> On some supported host systems, different packages need to be installed.
>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
>> instead of 3.6.8.  Let me stress again: same repository, different
>> package.  No downsides I can see.

... right here:

>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
>> version of Sphinx that works with Python 3.7 or newer.  This series
>> proposes to simply stop building the docs there, unless the user
>> provides a suitable version of Sphinx (which is easy enough with pip).
>
> I think we've all understood that. The thing that you obviously did not 
> understood: This breaks our support statement.
> I'm pretty sure that you could also build the whole QEMU suite successfully 
> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a 
> newer version of GCC and some of the required libraries first. But that's 
> not how we understand our support statement.
>
> Sure, you can argue that you can use "pip install" to get a newer version of 
> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is 
> that really that much different from installing a newer version of GCC and 
> libraries on an ancient distro that we do not officially support anymore? 
> I'd say no. You also have to consider that not every build host has access 
> to the internet, maybe some companies only have an internal mirror of the 
> distro packages in their intranet (I remember some discussion about such a 
> case in the past) - so while you were perfectly fine to build the whole of 
> QEMU on a CentOS 8 there before this change, you could now not build parts 
> of QEMU anymore there due to the missing possibility to run "pip install" 
> without full internet connection.
>
> And sure, you can argue that it's "just" the documentation. But IMHO that's 
> still an essential part of the QEMU build, and it used to work before, so it 
> feels wrong to cut that now out. And also, if we start with the 
> documentation now, what's next? If for example scripts/shaderinclude.py 
> stops working with Python 3.6, do we then simply say: "Oh, it's fine, you 
> can still build all the other targets that work without this script, just 
> not the ones anymore that need it"?

My view on all this is a bit more pragmatic.

For a human developer, the difference between "dnf install
python-sphinx" and "pip install sphinx" is, in my opinion, close to
negligible.  Really no comparison to "git-clone GCC and bootstap it".
You seem to disagree with that.

For automated builds in general, and distro packaging in particular, the
difference is real, and could even be a show stopper.  But who's
packaging bleeding edge QEMU on CentOS 8?  I suspect the only automated
builds are our own CI, where the difference is real, but hardly a show
stopper.  John's patch is the stupidest solution that could possibly
work for us: disable doc building on CentOS 8.  Alternative solutions
have been proposed, and that's fair.  Again, you seem to think this
issue is a lot more serious.

So maybe this breaks our support statement for a sufficiently rigid
interpretation of our support statement.  Not the way interpreted it,
but if it's the way it is to be interpreted, I stand corrected.

But then I'd like us to be a bit more pragmatic.  Is minor and graceful
degradation for systems close to the trailing edge really so
unacceptably terrible that we have to bend over backwards to avoid it?

>> All the angst about CentOS falling off the end of our "supported build
>> platforms" list is not actually warranted by this series :)
>
> Using the term "angst" for the concerns of your fellows here is quite 
> cheeky. It's not about "angst", it's about a discussion that our support 
> policy might need to be adjusted if we do this step. So instead of writing 
> such sentences, I'd rather would like to see you posting a patch for 
> docs/about/build-platforms.rst for constructive further discussion instead.

The phrasing of this sentence was ill-advised.  If it caused offense, I
apologize.

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Thomas Huth]

On 17/02/2023 10.06, Markus Armbruster wrote:
> Thomas Huth <thuth@redhat.com> writes:
...
> My view on all this is a bit more pragmatic.
> 
> For a human developer, the difference between "dnf install
> python-sphinx" and "pip install sphinx" is, in my opinion, close to
> negligible.  Really no comparison to "git-clone GCC and bootstap it".
> You seem to disagree with that.

Honestly, being a Python ignorant, I completely messed up my system with 
"pip" already a couple of times, especially if the instructions forgot to 
tell me to use the "--user" switch. So yes, I tend to disagree ;-)

> For automated builds in general, and distro packaging in particular, the
> difference is real, and could even be a show stopper.  But who's
> packaging bleeding edge QEMU on CentOS 8?  I suspect the only automated
> builds are our own CI, where the difference is real, but hardly a show
> stopper.

If we've got the feeling that nobody out there really builds QEMU on older 
long-term distros anymore, then why the heck are we still trying to support 
this according to our support statement?

> But then I'd like us to be a bit more pragmatic.  Is minor and graceful
> degradation for systems close to the trailing edge really so
> unacceptably terrible that we have to bend over backwards to avoid it?

Let's just get our support statement adjusted - it was written with good 
intention originally, but apparently this is causing too much pain, so we 
should adjust it instead of suffering in upstream development.

>>> All the angst about CentOS falling off the end of our "supported build
>>> platforms" list is not actually warranted by this series :)
>>
>> Using the term "angst" for the concerns of your fellows here is quite
>> cheeky. It's not about "angst", it's about a discussion that our support
>> policy might need to be adjusted if we do this step. So instead of writing
>> such sentences, I'd rather would like to see you posting a patch for
>> docs/about/build-platforms.rst for constructive further discussion instead.
> 
> The phrasing of this sentence was ill-advised.  If it caused offense, I
> apologize.

Ok, thanks. And just to make it clear again: I certainly do not object 
dropping the support for Python 3.6 - I just want to make sure that we 
adjust our support statement if the current version is causing too much pain 
for us. Sorry if I got that across in the wrong way.

  Thomas

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Peter Maydell]

On Fri, 17 Feb 2023 at 09:56, Thomas Huth <thuth@redhat.com> wrote:
>
> On 17/02/2023 10.06, Markus Armbruster wrote:
> > Thomas Huth <thuth@redhat.com> writes:
> ...
> > My view on all this is a bit more pragmatic.
> >
> > For a human developer, the difference between "dnf install
> > python-sphinx" and "pip install sphinx" is, in my opinion, close to
> > negligible.  Really no comparison to "git-clone GCC and bootstap it".
> > You seem to disagree with that.
>
> Honestly, being a Python ignorant, I completely messed up my system with
> "pip" already a couple of times, especially if the instructions forgot to
> tell me to use the "--user" switch. So yes, I tend to disagree ;-)

Seconded. I trust my distro package manager and I know how it works,
and I know how to uninstall a package later if I want to revert what
I've done. I do not know or trust what the heck pip is doing or where it's
trying to install anything, because it's not a tool I habitually
use. I can't remember if I've managed to mess up the system with it,
but I've definitely had the experience of "install stuff with pip,
do a distro upgrade later, the pip installed stuff is all busted".

> > For automated builds in general, and distro packaging in particular, the
> > difference is real, and could even be a show stopper.  But who's
> > packaging bleeding edge QEMU on CentOS 8?  I suspect the only automated
> > builds are our own CI, where the difference is real, but hardly a show
> > stopper.
>
> If we've got the feeling that nobody out there really builds QEMU on older
> long-term distros anymore, then why the heck are we still trying to support
> this according to our support statement?

I don't think anybody is *packaging* new QEMU on an old distro.
I do think we have users who do ad-hoc from-source builds.

thanks
-- PMM

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Daniel P. Berrangé]

On Fri, Feb 17, 2023 at 03:37:16PM +0000, Peter Maydell wrote:
> On Fri, 17 Feb 2023 at 09:56, Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 17/02/2023 10.06, Markus Armbruster wrote:
> > > Thomas Huth <thuth@redhat.com> writes:
> > ...
> > > My view on all this is a bit more pragmatic.
> > >
> > > For a human developer, the difference between "dnf install
> > > python-sphinx" and "pip install sphinx" is, in my opinion, close to
> > > negligible.  Really no comparison to "git-clone GCC and bootstap it".
> > > You seem to disagree with that.
> >
> > Honestly, being a Python ignorant, I completely messed up my system with
> > "pip" already a couple of times, especially if the instructions forgot to
> > tell me to use the "--user" switch. So yes, I tend to disagree ;-)
> 
> Seconded. I trust my distro package manager and I know how it works,
> and I know how to uninstall a package later if I want to revert what
> I've done. I do not know or trust what the heck pip is doing or where it's
> trying to install anything, because it's not a tool I habitually
> use. I can't remember if I've managed to mess up the system with it,
> but I've definitely had the experience of "install stuff with pip,
> do a distro upgrade later, the pip installed stuff is all busted".
> 
> > > For automated builds in general, and distro packaging in particular, the
> > > difference is real, and could even be a show stopper.  But who's
> > > packaging bleeding edge QEMU on CentOS 8?  I suspect the only automated
> > > builds are our own CI, where the difference is real, but hardly a show
> > > stopper.
> >
> > If we've got the feeling that nobody out there really builds QEMU on older
> > long-term distros anymore, then why the heck are we still trying to support
> > this according to our support statement?
> 
> I don't think anybody is *packaging* new QEMU on an old distro.

I recall that at one time the openvz folks where packaging new QEMU on
RHEL-7 for a while after we had already dropped RHEL-7 as a target.
That's the trouble with enterprise distros, their usage sticks around
way longer than any of us would care to admit.

> I do think we have users who do ad-hoc from-source builds.

We'll certainly have contributors using it as a dev platform from
the corporate world.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Daniel P. Berrangé]

On Fri, Feb 17, 2023 at 10:06:49AM +0100, Markus Armbruster wrote:
> Thomas Huth <thuth@redhat.com> writes:
> 
> > On 15/02/2023 20.05, Markus Armbruster wrote:
> >> The discussion under PATCH 6 makes me think there's a bit of confusion
> >> about the actual impact of dropping support for Python 3.6.  Possibly
> >> because it's spelled out in the commit message of PATCH 7.  Let me
> >> summarize it in one sentence:
> >> 
> >>      *** All supported host systems continue to work ***
> >> 
> >> Evidence: CI remains green.
> >
> > The CI remains green since one of the patches disabled the building of the 
> > docs on CentOS 8. That's not how I'd describe "continue to work", at least 
> > not in the same extend as before.
> 
> Thus the exception ...
> 
> >> On some supported host systems, different packages need to be installed.
> >> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> >> instead of 3.6.8.  Let me stress again: same repository, different
> >> package.  No downsides I can see.
> 
> ... right here:
> 
> >> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> >> version of Sphinx that works with Python 3.7 or newer.  This series
> >> proposes to simply stop building the docs there, unless the user
> >> provides a suitable version of Sphinx (which is easy enough with pip).
> >
> > I think we've all understood that. The thing that you obviously did not 
> > understood: This breaks our support statement.
> > I'm pretty sure that you could also build the whole QEMU suite successfully 
> > on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a 
> > newer version of GCC and some of the required libraries first. But that's 
> > not how we understand our support statement.
> >
> > Sure, you can argue that you can use "pip install" to get a newer version of 
> > Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is 
> > that really that much different from installing a newer version of GCC and 
> > libraries on an ancient distro that we do not officially support anymore? 
> > I'd say no. You also have to consider that not every build host has access 
> > to the internet, maybe some companies only have an internal mirror of the 
> > distro packages in their intranet (I remember some discussion about such a 
> > case in the past) - so while you were perfectly fine to build the whole of 
> > QEMU on a CentOS 8 there before this change, you could now not build parts 
> > of QEMU anymore there due to the missing possibility to run "pip install" 
> > without full internet connection.
> >
> > And sure, you can argue that it's "just" the documentation. But IMHO that's 
> > still an essential part of the QEMU build, and it used to work before, so it 
> > feels wrong to cut that now out. And also, if we start with the 
> > documentation now, what's next? If for example scripts/shaderinclude.py 
> > stops working with Python 3.6, do we then simply say: "Oh, it's fine, you 
> > can still build all the other targets that work without this script, just 
> > not the ones anymore that need it"?
> 
> My view on all this is a bit more pragmatic.
> 
> For a human developer, the difference between "dnf install
> python-sphinx" and "pip install sphinx" is, in my opinion, close to
> negligible.  Really no comparison to "git-clone GCC and bootstap it".
> You seem to disagree with that.

The command for installing the package is certainly not
significantly different.

What I think is lacking with pip is what happens after that
point - the ongoing management of the installed packages, or
rather than lack thereof.

If doing user installs, my $HOME/.local/lib ends up with multiple
orphaned previously installed versions. After each python version
upgrade, pip ceases to report about existance previously installed
bits at all, alone ever upgrade them.

Then there is the issue that pip will refuse to install a package,
if it already exists either from pip or the distro install, even
if the version that already exists doesn't satisfy the version
required. This would impact if the user has 'flake8'  N installed,
and QEMU wanted version N+1 installed, both using the same python
runtime version.

This is not the situation we're in with centos8/docutils, but
it is plausible in future once we decide we'll no longer rely
on exclusively on distro provided python packages, and instead
use pip if we ever want a newer version of something.


The python answer to this IIUC, is to use virtualenvs to isolate
everything. If we use keep the virtualenv under the QEMU build
root, then any time you clean your build root the virtualenv
content needs re-populating. At least pip has the sense to use
a cache, so $HOME/.cache/pip keeps a copy of everything you've
previously downloaded (presumably they age out the cache at
some time/size?), to avoid spending time re-loading all the
time. 


NB, I'm not saying the above is a blocker against use of pip.
Just that there are real differences between use of pip and
the distro packaging tools, beyond the initial install command.


If we're going to use pip, then using virtualenvs looks likely
to be very desirable to avoid conflicts between versions of
pkgs QEMU wants vs what the user might already have installed,
as well as to avoid leaving cruft behind in $HOME/.local/lib.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[John Snow]

On Thu, Feb 16, 2023 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 15/02/2023 20.05, Markus Armbruster wrote:
> > The discussion under PATCH 6 makes me think there's a bit of confusion
> > about the actual impact of dropping support for Python 3.6.  Possibly
> > because it's spelled out in the commit message of PATCH 7.  Let me
> > summarize it in one sentence:
> >
> >      *** All supported host systems continue to work ***
> >
> > Evidence: CI remains green.
>
> The CI remains green since one of the patches disabled the building of the
> docs on CentOS 8. That's not how I'd describe "continue to work", at least
> not in the same extend as before.
>
> > On some supported host systems, different packages need to be installed.
> > On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> > instead of 3.6.8.  Let me stress again: same repository, different
> > package.  No downsides I can see.
> >
> > The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> > version of Sphinx that works with Python 3.7 or newer.  This series
> > proposes to simply stop building the docs there, unless the user
> > provides a suitable version of Sphinx (which is easy enough with pip).
>
> I think we've all understood that. The thing that you obviously did not
> understood: This breaks our support statement.
> I'm pretty sure that you could also build the whole QEMU suite successfully
> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a
> newer version of GCC and some of the required libraries first. But that's
> not how we understand our support statement.
>
> Sure, you can argue that you can use "pip install" to get a newer version of
> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is
> that really that much different from installing a newer version of GCC and
> libraries on an ancient distro that we do not officially support anymore?
> I'd say no. You also have to consider that not every build host has access
> to the internet, maybe some companies only have an internal mirror of the
> distro packages in their intranet (I remember some discussion about such a
> case in the past) - so while you were perfectly fine to build the whole of
> QEMU on a CentOS 8 there before this change, you could now not build parts
> of QEMU anymore there due to the missing possibility to run "pip install"
> without full internet connection.

There are good points elsewhere in this thread and I am taking notes,
but this critique caught my eye as something I was not specifically
planning around, so I wanted to get an elaboration here if I may.

Do we have a support statement for this? I find this critique somewhat
surprising -- If we don't have internet, how did we get the other 20
to 30 dependencies needed to build QEMU? To what extent are we
*required* to preserve a build that works without internet access?

you generally need internet to run "dnf install", as you would to "pip
install", so how does this distinction exclude one but not the other?

If you mean to say: "The build cannot rely on using internet-connected
pip to configure an environment just-in-time during a build because
internet may not be present" -- I completely agree, as this is a
necessity for e.g. RHEL packaging downstream. That requirement won't
be violated by me.

--js

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Thomas Huth]

On 17/02/2023 21.46, John Snow wrote:
> On Thu, Feb 16, 2023 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 15/02/2023 20.05, Markus Armbruster wrote:
>>> The discussion under PATCH 6 makes me think there's a bit of confusion
>>> about the actual impact of dropping support for Python 3.6.  Possibly
>>> because it's spelled out in the commit message of PATCH 7.  Let me
>>> summarize it in one sentence:
>>>
>>>       *** All supported host systems continue to work ***
>>>
>>> Evidence: CI remains green.
>>
>> The CI remains green since one of the patches disabled the building of the
>> docs on CentOS 8. That's not how I'd describe "continue to work", at least
>> not in the same extend as before.
>>
>>> On some supported host systems, different packages need to be installed.
>>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
>>> instead of 3.6.8.  Let me stress again: same repository, different
>>> package.  No downsides I can see.
>>>
>>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
>>> version of Sphinx that works with Python 3.7 or newer.  This series
>>> proposes to simply stop building the docs there, unless the user
>>> provides a suitable version of Sphinx (which is easy enough with pip).
>>
>> I think we've all understood that. The thing that you obviously did not
>> understood: This breaks our support statement.
>> I'm pretty sure that you could also build the whole QEMU suite successfully
>> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a
>> newer version of GCC and some of the required libraries first. But that's
>> not how we understand our support statement.
>>
>> Sure, you can argue that you can use "pip install" to get a newer version of
>> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is
>> that really that much different from installing a newer version of GCC and
>> libraries on an ancient distro that we do not officially support anymore?
>> I'd say no. You also have to consider that not every build host has access
>> to the internet, maybe some companies only have an internal mirror of the
>> distro packages in their intranet (I remember some discussion about such a
>> case in the past) - so while you were perfectly fine to build the whole of
>> QEMU on a CentOS 8 there before this change, you could now not build parts
>> of QEMU anymore there due to the missing possibility to run "pip install"
>> without full internet connection.
> 
> There are good points elsewhere in this thread and I am taking notes,
> but this critique caught my eye as something I was not specifically
> planning around, so I wanted to get an elaboration here if I may.
> 
> Do we have a support statement for this? I find this critique somewhat
> surprising -- If we don't have internet, how did we get the other 20
> to 30 dependencies needed to build QEMU? To what extent are we
> *required* to preserve a build that works without internet access?

It's not written in stone, but I saw it this way: If I have a complete 
mirror of a distro repository in my intrAnet, I can use that mirror to set 
up a QEMU build host system that has no access to the internet. Or maybe 
think of a DVD image(s) with all distro packages that you use to install a 
host without network access (and you copy the QEMU tarball there via USB 
stick). I think it's not that uncommon to have such scenarios out there.

For example, do you remember that SDL 1.2 discussion a some years ago? See:

  https://www.mail-archive.com/qemu-devel@nongnu.org/msg631628.html

It was not exactly the same situation, since those folks were even unable to 
install a SDL2-devel package on their pre-installed hosts, though it was 
theoretically available as an update in their distro, but I think it gives 
an impression of what people are using and expecting out there... (and no, 
I'm not happy with this, I'd also rather love if we could move faster in the 
QEMU project sometimes).

  Thomas

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[John Snow]

On Mon, Feb 20, 2023 at 1:16 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 17/02/2023 21.46, John Snow wrote:
> > On Thu, Feb 16, 2023 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
> >>
> >> On 15/02/2023 20.05, Markus Armbruster wrote:
> >>> The discussion under PATCH 6 makes me think there's a bit of confusion
> >>> about the actual impact of dropping support for Python 3.6.  Possibly
> >>> because it's spelled out in the commit message of PATCH 7.  Let me
> >>> summarize it in one sentence:
> >>>
> >>>       *** All supported host systems continue to work ***
> >>>
> >>> Evidence: CI remains green.
> >>
> >> The CI remains green since one of the patches disabled the building of the
> >> docs on CentOS 8. That's not how I'd describe "continue to work", at least
> >> not in the same extend as before.
> >>
> >>> On some supported host systems, different packages need to be installed.
> >>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
> >>> instead of 3.6.8.  Let me stress again: same repository, different
> >>> package.  No downsides I can see.
> >>>
> >>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
> >>> version of Sphinx that works with Python 3.7 or newer.  This series
> >>> proposes to simply stop building the docs there, unless the user
> >>> provides a suitable version of Sphinx (which is easy enough with pip).
> >>
> >> I think we've all understood that. The thing that you obviously did not
> >> understood: This breaks our support statement.
> >> I'm pretty sure that you could also build the whole QEMU suite successfully
> >> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a
> >> newer version of GCC and some of the required libraries first. But that's
> >> not how we understand our support statement.
> >>
> >> Sure, you can argue that you can use "pip install" to get a newer version of
> >> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is
> >> that really that much different from installing a newer version of GCC and
> >> libraries on an ancient distro that we do not officially support anymore?
> >> I'd say no. You also have to consider that not every build host has access
> >> to the internet, maybe some companies only have an internal mirror of the
> >> distro packages in their intranet (I remember some discussion about such a
> >> case in the past) - so while you were perfectly fine to build the whole of
> >> QEMU on a CentOS 8 there before this change, you could now not build parts
> >> of QEMU anymore there due to the missing possibility to run "pip install"
> >> without full internet connection.
> >
> > There are good points elsewhere in this thread and I am taking notes,
> > but this critique caught my eye as something I was not specifically
> > planning around, so I wanted to get an elaboration here if I may.
> >
> > Do we have a support statement for this? I find this critique somewhat
> > surprising -- If we don't have internet, how did we get the other 20
> > to 30 dependencies needed to build QEMU? To what extent are we
> > *required* to preserve a build that works without internet access?
>
> It's not written in stone, but I saw it this way: If I have a complete
> mirror of a distro repository in my intrAnet, I can use that mirror to set
> up a QEMU build host system that has no access to the internet. Or maybe
> think of a DVD image(s) with all distro packages that you use to install a
> host without network access (and you copy the QEMU tarball there via USB
> stick). I think it's not that uncommon to have such scenarios out there.
>
> For example, do you remember that SDL 1.2 discussion a some years ago? See:
>
>   https://www.mail-archive.com/qemu-devel@nongnu.org/msg631628.html
>
> It was not exactly the same situation, since those folks were even unable to
> install a SDL2-devel package on their pre-installed hosts, though it was
> theoretically available as an update in their distro, but I think it gives
> an impression of what people are using and expecting out there... (and no,
> I'm not happy with this, I'd also rather love if we could move faster in the
> QEMU project sometimes).
>
>   Thomas

Well, in this case I believe our support policy generally is written
to require a fully up-to-date version of the LTS distros, e.g. we
don't really test against "release day" 16.04, in the same way we
don't offer support for RHEL 8.0, just the latest point release. I
don't want to march things forward and break things for people for no
reason, but at a certain point, I have to ask: Why do people expect
software written three to four years after the release of their
operating system to not only run, but compile on that system -- with
no updates or internet? I think it's (unfortunately) reasonable to
expect that if you want to run a stable OS with no changes for years
that at a certain point, brand new releases may start requiring a few
hoops for you to jump through.

Or, in other words: If you can get code from 2019 onto a machine from
2016 to attempt to compile, you can also get the dependencies from the
future, too.

Still; with regards to the "offline building" thing specifically, it's
my intent to preserve the ability to build QEMU offline *provided* you
have the necessary dependencies in place already. For the Python case
under consideration, it would just be that you have your distro's
python38/python39 package installed. I consider this fundamentally no
different to other dependencies. For docs building it's a bit hairier;
you would indeed need a pip version installed prior to going offline.
The loss of docs doesn't fail the build, though; they aren't
*technically* required.

I think really all we need is the ability to know a priori what we
need to build QEMU before going offline without any last second
surprises during configure, make, or make check. Right? Or do we
really want to say "Any preparation that might be needed from outside
your system's repository *at all* is entirely prohibited"?

Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Thomas Huth]

On 20/02/2023 20.56, John Snow wrote:
> On Mon, Feb 20, 2023 at 1:16 AM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 17/02/2023 21.46, John Snow wrote:
>>> On Thu, Feb 16, 2023 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
>>>>
>>>> On 15/02/2023 20.05, Markus Armbruster wrote:
>>>>> The discussion under PATCH 6 makes me think there's a bit of confusion
>>>>> about the actual impact of dropping support for Python 3.6.  Possibly
>>>>> because it's spelled out in the commit message of PATCH 7.  Let me
>>>>> summarize it in one sentence:
>>>>>
>>>>>        *** All supported host systems continue to work ***
>>>>>
>>>>> Evidence: CI remains green.
>>>>
>>>> The CI remains green since one of the patches disabled the building of the
>>>> docs on CentOS 8. That's not how I'd describe "continue to work", at least
>>>> not in the same extend as before.
>>>>
>>>>> On some supported host systems, different packages need to be installed.
>>>>> On CentOS 8, for instance, we need to install Python 3.8.13 or 3.9.16
>>>>> instead of 3.6.8.  Let me stress again: same repository, different
>>>>> package.  No downsides I can see.
>>>>>
>>>>> The *one* exception is Sphinx on CentOS 8.  CentOS 8 does not ship a
>>>>> version of Sphinx that works with Python 3.7 or newer.  This series
>>>>> proposes to simply stop building the docs there, unless the user
>>>>> provides a suitable version of Sphinx (which is easy enough with pip).
>>>>
>>>> I think we've all understood that. The thing that you obviously did not
>>>> understood: This breaks our support statement.
>>>> I'm pretty sure that you could also build the whole QEMU suite successfully
>>>> on an ancient CentOS 7 or Ubuntu 18.04 system if you manually install a
>>>> newer version of GCC and some of the required libraries first. But that's
>>>> not how we understand our support statement.
>>>>
>>>> Sure, you can argue that you can use "pip install" to get a newer version of
>>>> Sphinx on RHEL 8 / CentOS 8 to continue building the docs there - but is
>>>> that really that much different from installing a newer version of GCC and
>>>> libraries on an ancient distro that we do not officially support anymore?
>>>> I'd say no. You also have to consider that not every build host has access
>>>> to the internet, maybe some companies only have an internal mirror of the
>>>> distro packages in their intranet (I remember some discussion about such a
>>>> case in the past) - so while you were perfectly fine to build the whole of
>>>> QEMU on a CentOS 8 there before this change, you could now not build parts
>>>> of QEMU anymore there due to the missing possibility to run "pip install"
>>>> without full internet connection.
>>>
>>> There are good points elsewhere in this thread and I am taking notes,
>>> but this critique caught my eye as something I was not specifically
>>> planning around, so I wanted to get an elaboration here if I may.
>>>
>>> Do we have a support statement for this? I find this critique somewhat
>>> surprising -- If we don't have internet, how did we get the other 20
>>> to 30 dependencies needed to build QEMU? To what extent are we
>>> *required* to preserve a build that works without internet access?
>>
>> It's not written in stone, but I saw it this way: If I have a complete
>> mirror of a distro repository in my intrAnet, I can use that mirror to set
>> up a QEMU build host system that has no access to the internet. Or maybe
>> think of a DVD image(s) with all distro packages that you use to install a
>> host without network access (and you copy the QEMU tarball there via USB
>> stick). I think it's not that uncommon to have such scenarios out there.
>>
>> For example, do you remember that SDL 1.2 discussion a some years ago? See:
>>
>>    https://www.mail-archive.com/qemu-devel@nongnu.org/msg631628.html
>>
>> It was not exactly the same situation, since those folks were even unable to
>> install a SDL2-devel package on their pre-installed hosts, though it was
>> theoretically available as an update in their distro, but I think it gives
>> an impression of what people are using and expecting out there... (and no,
>> I'm not happy with this, I'd also rather love if we could move faster in the
>> QEMU project sometimes).
>>
>>    Thomas
> 
> Well, in this case I believe our support policy generally is written
> to require a fully up-to-date version of the LTS distros, e.g. we
> don't really test against "release day" 16.04, in the same way we
> don't offer support for RHEL 8.0, just the latest point release.

Yes, sure, that's what I meant with "not exactly the same situation" ... it 
was just an example of people trying to compile QEMU offline.

> I think really all we need is the ability to know a priori what we
> need to build QEMU before going offline without any last second
> surprises during configure, make, or make check. Right?

I think it should be OK with the patch that Paolo suggested for the support 
policy and maybe a note somewhere that you have to make sure to install a 
newer Sphinx with pip in case you still want to build the docs on older 
enterprise distros...

  Thomas

Proposed way forward Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Daniel P. Berrangé]

On Thu, Feb 09, 2023 at 07:31:40PM -0500, John Snow wrote:
> Howdy, this series increases our minimum python version to 3.7.

..snip...

So based on the discussion in this thread I see the following as the
key take aways:

 * Use of newer Meson feature

   We need newer meson in order to continue/finish the conversion
   of configure shell script into meson.build file logic. The newer
   meson has python 3.7 as a minimum required.

   The switch to meson was/is a significant strategic goal for
   improving QEMU build system maintainability which is thus
   highly desirable to continue without arbitrary delay.

 * Use of newer python parameter type hinting annotations

   Newer mypy supports a wider range of python type hinting
   annotations. While these ought to only impact the need for
   a new mypy for offline static analysis, in practice they
   also impact the runtime execution of the python code
   breaking compatibility with python 3.6. This unfortunate
   but out of our control.

   The wider use of type hinting in our python code is improving
   its maintainability, because it makes it significantly simpler
   for contributors to understand the behaviour of the code.


 * Use of newer python static analysis tools in general
 
   This applies to thinks like pylint, flak8, mypy (any more?)
   The issues reported by newer versions are not neccessarily
   always a strict superset of issues reports by older versions.
   Thus you can't assume that a passing result from new tools
   implies a passing result from the old tools. As such supporting
   all older versions is a cost/burden with little to no upside.

   With the exception of mypy, these aren't believed to have a
   direct bearing on the minimum python runtime version, but this
   would still technically be a deviation from our current support
   policy.


The first two points feel like pretty decent functional reasons to
justify an increase in the minimum python version. They will both
unlock the use of features that will have a tangible benefit to the
maintainability of QEMU in key areas.

If we do the first two points, then we can justify the 3rd point too
as having negligible additional downside.


The question is how to reconcile this with the platform support policy.
The policy has two goals. First to give us as maintainers a clear rule
as to when it is acceptable to increase minimum versions. The intent
is we don't need to debate the merits of each proposed change, just
validate that it follows our stated support policy. The second goal
is to give consumers of QEMU guidance as to what their platform has
to be able to provide in order to build and deploy QEMU.

The long life enterprise distros are the pain point in unlocking the
new of python features. So the proposal is that, at a minimum, we
augment the current policy with words to the effect that:

  * For long life cycle distributions, QEMU will follow normal
    policy for platform level dependancies / native code. For
    Python modules, QEMU may choose to require a newer versions
    than are available from the distribution package manager.

Potentially we could go further and reserve the right to
mandate newer python versions for any OS distro, because once
we have the facility for doing this with enterprise distros,
that shoudl just work anywhere. Perhaps start with targetted
exception at long life distros though. Either would unblock
our immediate needs wrt droppping python 3.6.

Finally, in terms of putting this into practice

  * If the OS platform doesn't provide the sufficiently new
    python modules, QEMU should default to downloading and
    installing them using pip, into a virtualenv under the
    QEMU build root so we don't impact other stuff the user
    might have in $HOME/.local/lib/python*. IIUC, the use
    of virtualenvs has been desired for a while already.

  * QEMU must provide a mechanism to disable any automatic
    downloading with pip, to allow users to pre-populate the
    addon modules if desired, to cope with scenarios where
    network access isn't available. Should be straightforward.

  * CI should continue to test all features, and be able
    to install the extra pieces with pip, if relevant for
    the distro. This should already be doable with the
    lcitool stuff Paolo has got working, which seemlessly
    uses pip on distros if needed.


So basically I'm suggesting we take a combination of John's series
here, and Paolo's series. John's last patch would change so that our
dockerfiles grab docutils with pip instead of disabling docs build
on centos8. Then look at introducing the use of virtualenvs for auto
installing any python mods the OS doesn't satisfy.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Proposed way forward Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Thomas Huth]

On 17/02/2023 12.37, Daniel P. Berrangé wrote:
...
> The long life enterprise distros are the pain point in unlocking the
> new of python features. So the proposal is that, at a minimum, we
> augment the current policy with words to the effect that:
> 
>    * For long life cycle distributions, QEMU will follow normal
>      policy for platform level dependancies / native code. For
>      Python modules, QEMU may choose to require a newer versions
>      than are available from the distribution package manager.

Sounds reasonable to me. But I think we still should also add a sentence 
where we limit the total amount of time that we promise to support a 
long-term distro. Otherwise we'll also get problems with other way too 
backlevel native code dependencies at one point in time.

I just sent a patch for discussion, shortly before I noticed your mail here. 
Feel free to grab the ideas from there into your patch (if you're planning 
to send one), or let me know if I should try to include the Python-related 
sentences in mine.

  Thomas

Re: Proposed way forward Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Daniel P. Berrangé]

On Fri, Feb 17, 2023 at 02:46:53PM +0100, Thomas Huth wrote:
> On 17/02/2023 12.37, Daniel P. Berrangé wrote:
> ...
> > The long life enterprise distros are the pain point in unlocking the
> > new of python features. So the proposal is that, at a minimum, we
> > augment the current policy with words to the effect that:
> > 
> >    * For long life cycle distributions, QEMU will follow normal
> >      policy for platform level dependancies / native code. For
> >      Python modules, QEMU may choose to require a newer versions
> >      than are available from the distribution package manager.
> 
> Sounds reasonable to me. But I think we still should also add a sentence
> where we limit the total amount of time that we promise to support a
> long-term distro. Otherwise we'll also get problems with other way too
> backlevel native code dependencies at one point in time.

Oh sure, having something time related mention makes sense.

> I just sent a patch for discussion, shortly before I noticed your mail here.
> Feel free to grab the ideas from there into your patch (if you're planning
> to send one), or let me know if I should try to include the Python-related
> sentences in mine.

I wasn't planning to imminently send a change. Especially now that
both yourself an Paolo have proposed similar competing additions,
lets just iterate on one of those 2 proposals.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Proposed way forward Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

[Paolo Bonzini]

On 2/17/23 12:37, Daniel P. Berrangé wrote:
> 
> So basically I'm suggesting we take a combination of John's series
> here, and Paolo's series. John's last patch would change so that our
> dockerfiles grab docutils with pip instead of disabling docs build
> on centos8. Then look at introducing the use of virtualenvs for auto
> installing any python mods the OS doesn't satisfy.

Sounds good.  I'll post a v3 of this series shortly, in the meanwhile 
I've already posted a suggested update to the support policy, but don't 
plan to drop support for Python 3.6 in 7.0 so that we can give a heads 
up in the release notes.

John is looking at building the virtualenv at configure time, which will 
have the characteristics that you detailed, so both the minimum version 
bump and the virtualenv change would happen in time for 7.1.

Paolo