[RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[John Snow]

GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
           (All green, except Python self-tests, see below)

This patch series creates a mandatory python virtual environment
("venv") during configure time and uses it to ensure the availability of
meson and sphinx.

See https://www.qemu.org/2023/03/24/python/ for details. The summary is
that the goal of this series is to ensure that the `python` used to run
meson is the same `python` used to run Sphinx, tests, and any build-time
python scripting we have. As it stands, meson and sphinx (and their
extensions) *may* run in a different python environment than the one
configured and chosen by the user at configure/build time.

The effective change of this series is that QEMU will now
unconditionally create a venv at configure-time and will ensure that
meson (and sphinx, if docs are enabled) are available through that venv.

Some important points as a pre-emptive "FAQ":

- This venv is unconditionally created and lives at {build_dir}/pyvenv.

- The python interpreter used by this venv is always the one identified
  by configure. (Which in turn is always the one specified by --python
  or $PYTHON)

- *almost* all python scripts in qemu.git executed as part of the build
  system, meson, sphinx, avocado tests, vm tests or CI are always
  executed within this venv.

  (iotests are not yet integrated; I plan to tackle this separately as a
  follow-up in order to have a more tightly focused scope on that
  series.)

- It remains possible to build and test fully offline.
  (In most cases, you just need meson and sphinx from your distro's repo.)

- Distribution packaged 'meson' and 'sphinx' are still utilized whenever
  possible as the highest preference.

- Vendored versions of e.g. 'meson' are always preferred to PyPI
  versions for speed, repeatability and ensuring tarball builds work
  as-is offline.

  (Sphinx will not be vendored, just like it already isn't.)

- Missing dependencies, when possible, are fetched and installed
  on-demand automatically to make developer environments "just work".

- Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
  Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere

- No new dependencies (...for most platforms. Debian and NetBSD get an
  asterisk.)

- The meson git submodule is unused after this series and can be removed.

For reviewers, here's how the series is broken up:

Patch 1 is a testing pre-req. Note that even with this patch,
'check-python-minreqs' and 'check-python-tox' CI jobs will both still
fail on origin/master because this series requires 3.7+, but
origin/master is currently still 3.6+.

- python: update pylint configuration

Patches 2-8 add the mkvenv script. The first patch checks in the barest
essentials, and each subsequent patch adds a workaround or feature one
at a time.

- python: add mkvenv.py
- mkvenv: add console script entry point generation
- mkvenv: Add better error message for missing pyexapt module
- mkvenv: generate console entry shims from inside the venv
- mkvenv: work around broken pip installations on Debian 10
- mkvenv: add nested venv workaround
- mkvenv: add ensure subcommand

Patches 9-11 modify our testing configuration to add new dependencies as
needed.

- tests/docker: add python3-venv dependency
- tests/vm: Configure netbsd to use Python 3.10
- tests/vm: add py310-expat to NetBSD

Patch 12 changes how we package release tarballs.

- scripts/make-release: download meson==0.61.5 .whl

Patches 13-16 wire mkvenv into configure and tests.

- configure: create a python venv unconditionally
- configure: use 'mkvenv ensure meson' to bootstrap meson
- configure: add --enable-pypi and --disable-pypi
- tests: Use configure-provided pyvenv for tests

Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
changes could be folded earlier in the series (like the diagnose()
patch), but I'm keeping it separate for review for now.

- configure: move --enable-docs and --disable-docs back to configure
- mkvenv: add diagnose() method for ensure() failures
- configure: use --diagnose option with meson ensure
- configure: bootstrap sphinx with mkvenv

That's all for now, seeya!
--js

John Snow (20):
  python: update pylint configuration
  python: add mkvenv.py
  mkvenv: add console script entry point generation
  mkvenv: Add better error message for missing pyexpat module
  mkvenv: generate console entry shims from inside the venv
  mkvenv: work around broken pip installations on Debian 10
  mkvenv: add nested venv workaround
  mkvenv: add ensure subcommand
  tests/docker: add python3-venv dependency
  tests/vm: Configure netbsd to use Python 3.10
  tests/vm: add py310-expat to NetBSD
  scripts/make-release: download meson==0.61.5 .whl
  configure: create a python venv unconditionally
  configure: use 'mkvenv ensure meson' to bootstrap meson
  configure: add --enable-pypi and --disable-pypi
  tests: Use configure-provided pyvenv for tests
  configure: move --enable-docs and --disable-docs back to configure
  mkvenv: add diagnose() method for ensure() failures
  configure: use --diagnose option with meson ensure
  configure: bootstrap sphinx with mkvenv

 docs/devel/acpi-bits.rst                      |   6 +-
 docs/devel/testing.rst                        |  14 +-
 configure                                     | 139 +--
 .gitlab-ci.d/buildtest-template.yml           |   4 +-
 .gitlab-ci.d/buildtest.yml                    |   6 +-
 python/scripts/mkvenv.py                      | 871 ++++++++++++++++++
 python/setup.cfg                              |  10 +
 python/tests/flake8.sh                        |   1 +
 python/tests/isort.sh                         |   1 +
 python/tests/mypy.sh                          |   1 +
 python/tests/pylint.sh                        |   1 +
 .../org.centos/stream/8/x86_64/test-avocado   |   4 +-
 scripts/device-crash-test                     |   2 +-
 scripts/make-release                          |  11 +
 tests/Makefile.include                        |  10 +-
 .../dockerfiles/debian-all-test-cross.docker  |   3 +-
 .../dockerfiles/debian-hexagon-cross.docker   |   3 +-
 .../dockerfiles/debian-riscv64-cross.docker   |   3 +-
 .../dockerfiles/debian-tricore-cross.docker   |   3 +-
 tests/requirements.txt                        |   7 +-
 tests/vm/netbsd                               |   2 +
 21 files changed, 1016 insertions(+), 86 deletions(-)
 create mode 100644 python/scripts/mkvenv.py

-- 
2.39.2

[RFC PATCH v3 01/20] python: update pylint configuration

[John Snow]

Pylint 2.17.x decided that SocketAddrT was a bad name for a Type Alias for some
reason. Sure, fine, whatever.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/setup.cfg | 1 +
 1 file changed, 1 insertion(+)

diff --git a/python/setup.cfg b/python/setup.cfg
index 9e923d9762..3f36502954 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -132,6 +132,7 @@ good-names=i,
            fd,  # fd = os.open(...)
            c,   # for c in string: ...
            T,   # for TypeVars. See pylint#3401
+           SocketAddrT,  # Not sure why this is invalid.
 
 [pylint.similarities]
 # Ignore imports when computing similarities.
-- 
2.39.2

[RFC PATCH v3 02/20] python: add mkvenv.py

[John Snow]

This script will be responsible for building a lightweight Python
virtual environment at configure time. It works with Python 3.7 or
newer.

It has been designed to:
- work *offline*, no PyPI required.
- work *quickly*, The fast path is only ~65ms on my machine.
- work *robustly*, with multiple fallbacks to keep things working.
- work *cooperatively*, using system packages where possible.
  (You can use your distro's meson, no problem.)

Due to its unique position in the build chain, it exists outside of the
installable python packages in-tree and *must* be runnable without any
third party dependencies.

Under normal circumstances, the only dependency required to execute this
script is Python 3.7+ itself. The script is *faster* by several seconds
when setuptools and pip are installed in the host environment, which is
probably the case for a typical multi-purpose developer workstation.

In the event that pip/setuptools are missing or not usable, additional
dependencies may be required on some distributions which remove certain
Python stdlib modules to package them separately:

- Debian may require python3-venv to provide "ensurepip"
- NetBSD may require py310-expat to provide "pyexpat" *
  (* Or whichever version is current for NetBSD.)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 277 +++++++++++++++++++++++++++++++++++++++
 python/setup.cfg         |   9 ++
 python/tests/flake8.sh   |   1 +
 python/tests/isort.sh    |   1 +
 python/tests/mypy.sh     |   1 +
 python/tests/pylint.sh   |   1 +
 6 files changed, 290 insertions(+)
 create mode 100644 python/scripts/mkvenv.py

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
new file mode 100644
index 0000000000..1dfcc0198a
--- /dev/null
+++ b/python/scripts/mkvenv.py
@@ -0,0 +1,277 @@
+"""
+mkvenv - QEMU pyvenv bootstrapping utility
+
+usage: mkvenv [-h] command ...
+
+QEMU pyvenv bootstrapping utility
+
+options:
+  -h, --help  show this help message and exit
+
+Commands:
+  command     Description
+    create    create a venv
+
+--------------------------------------------------
+
+usage: mkvenv create [-h] target
+
+positional arguments:
+  target      Target directory to install virtual environment into.
+
+options:
+  -h, --help  show this help message and exit
+
+"""
+
+# Copyright (C) 2022-2023 Red Hat, Inc.
+#
+# Authors:
+#  John Snow <jsnow@redhat.com>
+#  Paolo Bonzini <pbonzini@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import argparse
+from importlib.util import find_spec
+import logging
+import os
+from pathlib import Path
+import subprocess
+import sys
+import traceback
+from types import SimpleNamespace
+from typing import Any, Optional, Union
+import venv
+
+
+# Do not add any mandatory dependencies from outside the stdlib:
+# This script *must* be usable standalone!
+
+logger = logging.getLogger("mkvenv")
+
+
+class Ouch(RuntimeError):
+    """An Exception class we can't confuse with a builtin."""
+
+
+class QemuEnvBuilder(venv.EnvBuilder):
+    """
+    An extension of venv.EnvBuilder for building QEMU's configure-time venv.
+
+    As of this commit, it does not yet do anything particularly
+    different than the standard venv-creation utility. The next several
+    commits will gradually change that in small commits that highlight
+    each feature individually.
+
+    Parameters for base class init:
+      - system_site_packages: bool = False
+      - clear: bool = False
+      - symlinks: bool = False
+      - upgrade: bool = False
+      - with_pip: bool = False
+      - prompt: Optional[str] = None
+      - upgrade_deps: bool = False             (Since 3.9)
+    """
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        logger.debug("QemuEnvBuilder.__init__(...)")
+        super().__init__(*args, **kwargs)
+
+        # The EnvBuilder class is cute and toggles this setting off
+        # before post_setup, but we need it to decide if we want to
+        # generate shims or not.
+        self._system_site_packages = self.system_site_packages
+
+    def post_setup(self, context: SimpleNamespace) -> None:
+        logger.debug("post_setup(...)")
+
+        # print the python executable to stdout for configure.
+        print(context.env_exe)
+
+
+def need_ensurepip() -> bool:
+    """
+    Tests for the presence of setuptools and pip.
+
+    :return: `True` if we do not detect both packages.
+    """
+    # Don't try to actually import them, it's fraught with danger:
+    # https://github.com/pypa/setuptools/issues/2993
+    if find_spec("setuptools") and find_spec("pip"):
+        return False
+    return True
+
+
+def check_ensurepip(with_pip: bool) -> None:
+    """
+    Check that we have ensurepip.
+
+    Raise a fatal exception with a helpful hint if it isn't available.
+    """
+    if with_pip and not find_spec("ensurepip"):
+        msg = (
+            "Python's ensurepip module is not found.\n"
+            "It's normally part of the Python standard library, "
+            "maybe your distribution packages it separately?\n"
+            "Either install ensurepip, or alleviate the need for it in the "
+            "first place by installing pip and setuptools for "
+            f"'{sys.executable}'.\n"
+            "(Hint: Debian puts ensurepip in its python3-venv package.)"
+        )
+        raise Ouch(msg)
+
+
+def make_venv(  # pylint: disable=too-many-arguments
+    env_dir: Union[str, Path],
+    system_site_packages: bool = False,
+    clear: bool = True,
+    symlinks: Optional[bool] = None,
+    with_pip: Optional[bool] = None,
+) -> None:
+    """
+    Create a venv using `QemuEnvBuilder`.
+
+    This is analogous to the `venv.create` module-level convenience
+    function that is part of the Python stdblib, except it uses
+    `QemuEnvBuilder` instead.
+
+    :param env_dir: The directory to create/install to.
+    :param system_site_packages:
+        Allow inheriting packages from the system installation.
+    :param clear: When True, fully remove any prior venv and files.
+    :param symlinks:
+        Whether to use symlinks to the target interpreter or not. If
+        left unspecified, it will use symlinks except on Windows to
+        match behavior with the "venv" CLI tool.
+    :param with_pip:
+        Whether to run "ensurepip" or not. If unspecified, this will
+        default to False if system_site_packages is True and a usable
+        version of pip is found.
+    """
+    logging.debug(
+        "%s: make_venv(env_dir=%s, system_site_packages=%s, "
+        "clear=%s, symlinks=%s, with_pip=%s)",
+        __file__,
+        str(env_dir),
+        system_site_packages,
+        clear,
+        symlinks,
+        with_pip,
+    )
+
+    print(f"MKVENV {str(env_dir)}", file=sys.stderr)
+
+    # ensurepip is slow: venv creation can be very fast for cases where
+    # we allow the use of system_site_packages. Toggle ensure_pip on only
+    # in the cases where we really need it.
+    if with_pip is None:
+        with_pip = True if not system_site_packages else need_ensurepip()
+        logger.debug("with_pip unset, choosing %s", with_pip)
+
+    check_ensurepip(with_pip)
+
+    if symlinks is None:
+        # Default behavior of standard venv CLI
+        symlinks = os.name != "nt"
+
+    builder = QemuEnvBuilder(
+        system_site_packages=system_site_packages,
+        clear=clear,
+        symlinks=symlinks,
+        with_pip=with_pip,
+    )
+    try:
+        logger.debug("Invoking builder.create()")
+        try:
+            builder.create(str(env_dir))
+        except SystemExit as exc:
+            # Some versions of the venv module raise SystemExit; *nasty*!
+            # We want the exception that prompted it. It might be a subprocess
+            # error that has output we *really* want to see.
+            logger.debug("Intercepted SystemExit from EnvBuilder.create()")
+            raise exc.__cause__ or exc.__context__ or exc
+        logger.debug("builder.create() finished")
+    except subprocess.CalledProcessError as exc:
+        print(f"cmd: {exc.cmd}", file=sys.stderr)
+        print(f"returncode: {exc.returncode}", file=sys.stderr)
+
+        def _stringify(data: Optional[Union[str, bytes]]) -> Optional[str]:
+            if isinstance(data, bytes):
+                return data.decode()
+            return data
+
+        if exc.stdout:
+            print(
+                "========== stdout ==========",
+                _stringify(exc.stdout),
+                "============================",
+                sep="\n",
+                file=sys.stderr,
+            )
+        if exc.stderr:
+            print(
+                "========== stderr ==========",
+                _stringify(exc.stderr),
+                "============================",
+                sep="\n",
+                file=sys.stderr,
+            )
+        raise Ouch("VENV creation subprocess failed.") from exc
+
+
+def _add_create_subcommand(subparsers: Any) -> None:
+    subparser = subparsers.add_parser("create", help="create a venv")
+    subparser.add_argument(
+        "target",
+        type=str,
+        action="store",
+        help="Target directory to install virtual environment into.",
+    )
+
+
+def main() -> int:
+    """CLI interface to make_qemu_venv. See module docstring."""
+    if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"):
+        # You're welcome.
+        logging.basicConfig(level=logging.DEBUG)
+    elif os.environ.get("V"):
+        logging.basicConfig(level=logging.INFO)
+
+    parser = argparse.ArgumentParser(
+        prog="mkvenv",
+        description="QEMU pyvenv bootstrapping utility",
+    )
+    subparsers = parser.add_subparsers(
+        title="Commands",
+        dest="command",
+        required=True,
+        metavar="command",
+        help="Description",
+    )
+
+    _add_create_subcommand(subparsers)
+
+    args = parser.parse_args()
+    try:
+        if args.command == "create":
+            make_venv(
+                args.target,
+                system_site_packages=True,
+                clear=True,
+            )
+        logger.debug("mkvenv.py %s: exiting", args.command)
+    except Ouch as exc:
+        print("\n*** Ouch! ***\n", file=sys.stderr)
+        print(str(exc), "\n\n", file=sys.stderr)
+        return 1
+    except:  # pylint: disable=bare-except
+        print("mkvenv did not complete successfully:", file=sys.stderr)
+        traceback.print_exc()
+        return 2
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/python/setup.cfg b/python/setup.cfg
index 3f36502954..5b25f810fa 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -103,6 +103,15 @@ ignore_missing_imports = True
 [mypy-pygments]
 ignore_missing_imports = True
 
+[mypy-importlib.metadata]
+ignore_missing_imports = True
+
+[mypy-importlib_metadata]
+ignore_missing_imports = True
+
+[mypy-pkg_resources]
+ignore_missing_imports = True
+
 [pylint.messages control]
 # Disable the message, report, category or checker with the given id(s). You
 # can either give multiple identifiers separated by comma (,) or put this
diff --git a/python/tests/flake8.sh b/python/tests/flake8.sh
index 1cd7d40fad..e013699645 100755
--- a/python/tests/flake8.sh
+++ b/python/tests/flake8.sh
@@ -1,2 +1,3 @@
 #!/bin/sh -e
 python3 -m flake8 qemu/
+python3 -m flake8 scripts/
diff --git a/python/tests/isort.sh b/python/tests/isort.sh
index 4480405bfb..66c2f7df0f 100755
--- a/python/tests/isort.sh
+++ b/python/tests/isort.sh
@@ -1,2 +1,3 @@
 #!/bin/sh -e
 python3 -m isort -c qemu/
+python3 -m isort -c scripts/
diff --git a/python/tests/mypy.sh b/python/tests/mypy.sh
index 5f980f563b..a33a3f58ab 100755
--- a/python/tests/mypy.sh
+++ b/python/tests/mypy.sh
@@ -1,2 +1,3 @@
 #!/bin/sh -e
 python3 -m mypy -p qemu
+python3 -m mypy scripts/
diff --git a/python/tests/pylint.sh b/python/tests/pylint.sh
index 03d64705a1..2b68da90df 100755
--- a/python/tests/pylint.sh
+++ b/python/tests/pylint.sh
@@ -1,3 +1,4 @@
 #!/bin/sh -e
 # See commit message for environment variable explainer.
 SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint qemu/
+SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint scripts/
-- 
2.39.2

[RFC PATCH v3 03/20] mkvenv: add console script entry point generation

[John Snow]

When creating a virtual environment that inherits system packages,
script entry points (like "meson", "sphinx-build", etc) are not
re-generated with the correct shebang. When you are *inside* of the
venv, this is not a problem, but if you are *outside* of it, you will
not have a script that engages the virtual environment appropriately.

Add a mechanism that generates new entry points for pre-existing
packages so that we can use these scripts to run "meson",
"sphinx-build", "pip", unambiguously inside the venv.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 179 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 172 insertions(+), 7 deletions(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 1dfcc0198a..f355cb54fb 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -14,13 +14,14 @@
 
 --------------------------------------------------
 
-usage: mkvenv create [-h] target
+usage: mkvenv create [-h] [--gen GEN] target
 
 positional arguments:
   target      Target directory to install virtual environment into.
 
 options:
   -h, --help  show this help message and exit
+  --gen GEN   Regenerate console_scripts for given packages, if found.
 
 """
 
@@ -38,11 +39,20 @@
 import logging
 import os
 from pathlib import Path
+import re
+import stat
 import subprocess
 import sys
 import traceback
 from types import SimpleNamespace
-from typing import Any, Optional, Union
+from typing import (
+    Any,
+    Dict,
+    Iterator,
+    Optional,
+    Sequence,
+    Union,
+)
 import venv
 
 
@@ -60,10 +70,9 @@ class QemuEnvBuilder(venv.EnvBuilder):
     """
     An extension of venv.EnvBuilder for building QEMU's configure-time venv.
 
-    As of this commit, it does not yet do anything particularly
-    different than the standard venv-creation utility. The next several
-    commits will gradually change that in small commits that highlight
-    each feature individually.
+    The only functional change is that it adds the ability to regenerate
+    console_script shims for packages available via system_site
+    packages.
 
     Parameters for base class init:
       - system_site_packages: bool = False
@@ -77,6 +86,7 @@ class QemuEnvBuilder(venv.EnvBuilder):
 
     def __init__(self, *args: Any, **kwargs: Any) -> None:
         logger.debug("QemuEnvBuilder.__init__(...)")
+        self.script_packages = kwargs.pop("script_packages", ())
         super().__init__(*args, **kwargs)
 
         # The EnvBuilder class is cute and toggles this setting off
@@ -87,6 +97,12 @@ def __init__(self, *args: Any, **kwargs: Any) -> None:
     def post_setup(self, context: SimpleNamespace) -> None:
         logger.debug("post_setup(...)")
 
+        # Generate console_script entry points for system packages:
+        if self._system_site_packages:
+            generate_console_scripts(
+                context.env_exe, context.bin_path, self.script_packages
+            )
+
         # print the python executable to stdout for configure.
         print(context.env_exe)
 
@@ -129,6 +145,7 @@ def make_venv(  # pylint: disable=too-many-arguments
     clear: bool = True,
     symlinks: Optional[bool] = None,
     with_pip: Optional[bool] = None,
+    script_packages: Sequence[str] = (),
 ) -> None:
     """
     Create a venv using `QemuEnvBuilder`.
@@ -149,16 +166,20 @@ def make_venv(  # pylint: disable=too-many-arguments
         Whether to run "ensurepip" or not. If unspecified, this will
         default to False if system_site_packages is True and a usable
         version of pip is found.
+    :param script_packages:
+        A sequence of package names to generate console entry point
+        shims for, when system_site_packages is True.
     """
     logging.debug(
         "%s: make_venv(env_dir=%s, system_site_packages=%s, "
-        "clear=%s, symlinks=%s, with_pip=%s)",
+        "clear=%s, symlinks=%s, with_pip=%s, script_packages=%s)",
         __file__,
         str(env_dir),
         system_site_packages,
         clear,
         symlinks,
         with_pip,
+        script_packages,
     )
 
     print(f"MKVENV {str(env_dir)}", file=sys.stderr)
@@ -181,6 +202,7 @@ def make_venv(  # pylint: disable=too-many-arguments
         clear=clear,
         symlinks=symlinks,
         with_pip=with_pip,
+        script_packages=script_packages,
     )
     try:
         logger.debug("Invoking builder.create()")
@@ -221,8 +243,147 @@ def _stringify(data: Optional[Union[str, bytes]]) -> Optional[str]:
         raise Ouch("VENV creation subprocess failed.") from exc
 
 
+def _gen_importlib(packages: Sequence[str]) -> Iterator[Dict[str, str]]:
+    # pylint: disable=import-outside-toplevel
+    try:
+        # First preference: Python 3.8+ stdlib
+        from importlib.metadata import (
+            PackageNotFoundError,
+            distribution,
+        )
+    except ImportError as exc:
+        logger.debug("%s", str(exc))
+        # Second preference: Commonly available PyPI backport
+        from importlib_metadata import (
+            PackageNotFoundError,
+            distribution,
+        )
+
+    # Borrowed from CPython (Lib/importlib/metadata/__init__.py)
+    pattern = re.compile(
+        r"(?P<module>[\w.]+)\s*"
+        r"(:\s*(?P<attr>[\w.]+)\s*)?"
+        r"((?P<extras>\[.*\])\s*)?$"
+    )
+
+    def _generator() -> Iterator[Dict[str, str]]:
+        for package in packages:
+            try:
+                entry_points = distribution(package).entry_points
+            except PackageNotFoundError:
+                continue
+
+            # The EntryPoints type is only available in 3.10+,
+            # treat this as a vanilla list and filter it ourselves.
+            entry_points = filter(
+                lambda ep: ep.group == "console_scripts", entry_points
+            )
+
+            for entry_point in entry_points:
+                # Python 3.8 doesn't have 'module' or 'attr' attributes
+                if not (
+                    hasattr(entry_point, "module")
+                    and hasattr(entry_point, "attr")
+                ):
+                    match = pattern.match(entry_point.value)
+                    assert match is not None
+                    module = match.group("module")
+                    attr = match.group("attr")
+                else:
+                    module = entry_point.module
+                    attr = entry_point.attr
+                yield {
+                    "name": entry_point.name,
+                    "module": module,
+                    "import_name": attr,
+                    "func": attr,
+                }
+
+    return _generator()
+
+
+def _gen_pkg_resources(packages: Sequence[str]) -> Iterator[Dict[str, str]]:
+    # pylint: disable=import-outside-toplevel
+    # Bundled with setuptools; has a good chance of being available.
+    import pkg_resources
+
+    def _generator() -> Iterator[Dict[str, str]]:
+        for package in packages:
+            try:
+                eps = pkg_resources.get_entry_map(package, "console_scripts")
+            except pkg_resources.DistributionNotFound:
+                continue
+
+            for entry_point in eps.values():
+                yield {
+                    "name": entry_point.name,
+                    "module": entry_point.module_name,
+                    "import_name": ".".join(entry_point.attrs),
+                    "func": ".".join(entry_point.attrs),
+                }
+
+    return _generator()
+
+
+# Borrowed/adapted from pip's vendored version of distutils:
+SCRIPT_TEMPLATE = r"""#!{python_path:s}
+# -*- coding: utf-8 -*-
+import re
+import sys
+from {module:s} import {import_name:s}
+if __name__ == '__main__':
+    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
+    sys.exit({func:s}())
+"""
+
+
+def generate_console_scripts(
+    python_path: str, bin_path: str, packages: Sequence[str]
+) -> None:
+    """
+    Generate script shims for console_script entry points in @packages.
+    """
+    if not packages:
+        return
+
+    def _get_entry_points() -> Iterator[Dict[str, str]]:
+        """Python 3.7 compatibility shim for iterating entry points."""
+        # Python 3.8+, or Python 3.7 with importlib_metadata installed.
+        try:
+            return _gen_importlib(packages)
+        except ImportError as exc:
+            logger.debug("%s", str(exc))
+
+        # Python 3.7 with setuptools installed.
+        try:
+            return _gen_pkg_resources(packages)
+        except ImportError as exc:
+            logger.debug("%s", str(exc))
+            raise Ouch(
+                "Neither importlib.metadata nor pkg_resources found, "
+                "can't generate console script shims.\n"
+                "Use Python 3.8+, or install importlib-metadata or setuptools."
+            ) from exc
+
+    for entry_point in _get_entry_points():
+        script_path = os.path.join(bin_path, entry_point["name"])
+        script = SCRIPT_TEMPLATE.format(python_path=python_path, **entry_point)
+        with open(script_path, "w", encoding="UTF-8") as file:
+            file.write(script)
+        mode = os.stat(script_path).st_mode | stat.S_IEXEC
+        os.chmod(script_path, mode)
+
+        logger.debug("wrote '%s'", script_path)
+
+
 def _add_create_subcommand(subparsers: Any) -> None:
     subparser = subparsers.add_parser("create", help="create a venv")
+    subparser.add_argument(
+        "--gen",
+        type=str,
+        action="append",
+        help="Regenerate console_scripts for given packages, if found.",
+    )
     subparser.add_argument(
         "target",
         type=str,
@@ -256,10 +417,14 @@ def main() -> int:
     args = parser.parse_args()
     try:
         if args.command == "create":
+            script_packages = []
+            for element in args.gen or ():
+                script_packages.extend(element.split(","))
             make_venv(
                 args.target,
                 system_site_packages=True,
                 clear=True,
+                script_packages=script_packages,
             )
         logger.debug("mkvenv.py %s: exiting", args.command)
     except Ouch as exc:
-- 
2.39.2

[RFC PATCH v3 04/20] mkvenv: Add better error message for missing pyexpat module

[John Snow]

NetBSD debundles pyexpat from python, but ensurepip needs pyexpat. Try
our best to offer a helpful error message instead of just failing
catastrophically.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index f355cb54fb..2172774403 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -126,7 +126,10 @@ def check_ensurepip(with_pip: bool) -> None:
 
     Raise a fatal exception with a helpful hint if it isn't available.
     """
-    if with_pip and not find_spec("ensurepip"):
+    if not with_pip:
+        return
+
+    if not find_spec("ensurepip"):
         msg = (
             "Python's ensurepip module is not found.\n"
             "It's normally part of the Python standard library, "
@@ -138,6 +141,19 @@ def check_ensurepip(with_pip: bool) -> None:
         )
         raise Ouch(msg)
 
+    # ensurepip uses pyexpat, which can also go missing on us:
+    if not find_spec("pyexpat"):
+        msg = (
+            "Python's pyexpat module is not found.\n"
+            "It's normally part of the Python standard library, "
+            "maybe your distribution packages it separately?\n"
+            "Either install pyexpat, or alleviate the need for it in the "
+            "first place by installing pip and setuptools for "
+            f"'{sys.executable}'.\n\n"
+            "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)"
+        )
+        raise Ouch(msg)
+
 
 def make_venv(  # pylint: disable=too-many-arguments
     env_dir: Union[str, Path],
-- 
2.39.2

[RFC PATCH v3 05/20] mkvenv: generate console entry shims from inside the venv

[John Snow]

This patch is meant to ensure that console entry point scripts will
always generate on Python 3.7 installations where we may not have access
to importlib.metadata. By running it from a separate process *inside*
the venv, we can be assured to have access to setuptools and by
extension pkg_resources as a fallback.

It isn't strictly necessary to do this for Python 3.8+, which will
always have importlib.metadata available.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 99 ++++++++++++++++++++++++++++++++++------
 1 file changed, 85 insertions(+), 14 deletions(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 2172774403..4daa652f12 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -11,6 +11,8 @@
 Commands:
   command     Description
     create    create a venv
+    post_init
+              post-venv initialization
 
 --------------------------------------------------
 
@@ -23,6 +25,15 @@
   -h, --help  show this help message and exit
   --gen GEN   Regenerate console_scripts for given packages, if found.
 
+--------------------------------------------------
+
+usage: mkvenv post_init [-h] [--gen GEN] [--binpath BINPATH]
+
+options:
+  -h, --help         show this help message and exit
+  --gen GEN          Regenerate console_scripts for given packages, if found.
+  --binpath BINPATH  Path where console script shims should be generated
+
 """
 
 # Copyright (C) 2022-2023 Red Hat, Inc.
@@ -59,6 +70,7 @@
 # Do not add any mandatory dependencies from outside the stdlib:
 # This script *must* be usable standalone!
 
+DirType = Union[str, bytes, "os.PathLike[str]", "os.PathLike[bytes]"]
 logger = logging.getLogger("mkvenv")
 
 
@@ -89,23 +101,42 @@ def __init__(self, *args: Any, **kwargs: Any) -> None:
         self.script_packages = kwargs.pop("script_packages", ())
         super().__init__(*args, **kwargs)
 
-        # The EnvBuilder class is cute and toggles this setting off
-        # before post_setup, but we need it to decide if we want to
-        # generate shims or not.
-        self._system_site_packages = self.system_site_packages
+        # Make the context available post-creation:
+        self._context: Optional[SimpleNamespace] = None
+
+    def ensure_directories(self, env_dir: DirType) -> SimpleNamespace:
+        logger.debug("ensure_directories(env_dir=%s)", env_dir)
+        self._context = super().ensure_directories(env_dir)
+        return self._context
+
+    def create(self, env_dir: DirType) -> None:
+        logger.debug("create(env_dir=%s)", env_dir)
+        super().create(env_dir)
+        assert self._context is not None
+        self.post_post_setup(self._context)
 
     def post_setup(self, context: SimpleNamespace) -> None:
         logger.debug("post_setup(...)")
-
-        # Generate console_script entry points for system packages:
-        if self._system_site_packages:
-            generate_console_scripts(
-                context.env_exe, context.bin_path, self.script_packages
-            )
-
         # print the python executable to stdout for configure.
         print(context.env_exe)
 
+    def post_post_setup(self, context: SimpleNamespace) -> None:
+        """
+        The final, final hook. Enter the venv and run commands inside of it.
+        """
+        args = [
+            context.env_exe,
+            __file__,
+            "post_init",
+            "--binpath",
+            context.bin_path,
+        ]
+        if self.system_site_packages:
+            scripts = ",".join(self.script_packages)
+            if scripts:
+                args += ["--gen", scripts]
+        subprocess.run(args, check=True)
+
 
 def need_ensurepip() -> bool:
     """
@@ -359,6 +390,13 @@ def generate_console_scripts(
     """
     Generate script shims for console_script entry points in @packages.
     """
+    logger.debug(
+        "generate_console_scripts(python_path=%s, bin_path=%s, packages=%s)",
+        python_path,
+        bin_path,
+        packages,
+    )
+
     if not packages:
         return
 
@@ -392,6 +430,17 @@ def _get_entry_points() -> Iterator[Dict[str, str]]:
         logger.debug("wrote '%s'", script_path)
 
 
+def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
+    """
+    This is intended to be run *inside the venv* after it is created.
+    """
+    python_path = sys.executable
+    logger.debug(
+        "post_venv_setup(bin_path=%s, packages=%s)", bin_path, packages
+    )
+    generate_console_scripts(python_path, bin_path, packages)
+
+
 def _add_create_subcommand(subparsers: Any) -> None:
     subparser = subparsers.add_parser("create", help="create a venv")
     subparser.add_argument(
@@ -408,6 +457,24 @@ def _add_create_subcommand(subparsers: Any) -> None:
     )
 
 
+def _add_post_init_subcommand(subparsers: Any) -> None:
+    subparser = subparsers.add_parser(
+        "post_init", help="post-venv initialization"
+    )
+    subparser.add_argument(
+        "--gen",
+        type=str,
+        action="append",
+        help="Regenerate console_scripts for given packages, if found.",
+    )
+    subparser.add_argument(
+        "--binpath",
+        type=str,
+        action="store",
+        help="Path where console script shims should be generated",
+    )
+
+
 def main() -> int:
     """CLI interface to make_qemu_venv. See module docstring."""
     if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"):
@@ -429,19 +496,23 @@ def main() -> int:
     )
 
     _add_create_subcommand(subparsers)
+    _add_post_init_subcommand(subparsers)
 
     args = parser.parse_args()
+    script_packages = []
+    for element in args.gen or ():
+        script_packages.extend(element.split(","))
+
     try:
         if args.command == "create":
-            script_packages = []
-            for element in args.gen or ():
-                script_packages.extend(element.split(","))
             make_venv(
                 args.target,
                 system_site_packages=True,
                 clear=True,
                 script_packages=script_packages,
             )
+        if args.command == "post_init":
+            post_venv_setup(args.binpath, script_packages)
         logger.debug("mkvenv.py %s: exiting", args.command)
     except Ouch as exc:
         print("\n*** Ouch! ***\n", file=sys.stderr)
-- 
2.39.2

[RFC PATCH v3 06/20] mkvenv: work around broken pip installations on Debian 10

[John Snow]

This is a workaround intended for Debian 10, where the debian-patched
pip does not function correctly if accessed from within a virtual
environment.

We don't support Debian 10 any longer, but it's possible that this bug
might appear on other derivative platforms and this workaround may prove
useful.

RFC, a note from Paolo:

"BTW, another way to repair Debian 10's pip is to create a symbolic link
to sys.base_prefix + '/share/python-wheels' in sys.prefix +
'/share/python-wheels'. Since this is much faster, perhaps it can be
done unconditionally [...] ?"

I was slightly apprehensive about this as it felt "hackier", but it is
indeed a lot less code and much faster. It's probably low-risk. Should
we do that instead, or should we just scrap any fix at all under the
premise that Debian 10 support is dropped anyway?

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 67 +++++++++++++++++++++++++++++++---------
 1 file changed, 52 insertions(+), 15 deletions(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 4daa652f12..445f4eb092 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -151,26 +151,26 @@ def need_ensurepip() -> bool:
     return True
 
 
-def check_ensurepip(with_pip: bool) -> None:
+def check_ensurepip(prefix: str = "", suggest_remedy: bool = False) -> None:
     """
     Check that we have ensurepip.
 
     Raise a fatal exception with a helpful hint if it isn't available.
     """
-    if not with_pip:
-        return
-
     if not find_spec("ensurepip"):
         msg = (
             "Python's ensurepip module is not found.\n"
             "It's normally part of the Python standard library, "
             "maybe your distribution packages it separately?\n"
-            "Either install ensurepip, or alleviate the need for it in the "
-            "first place by installing pip and setuptools for "
-            f"'{sys.executable}'.\n"
-            "(Hint: Debian puts ensurepip in its python3-venv package.)"
+            "(Debian puts ensurepip in its python3-venv package.)\n"
         )
-        raise Ouch(msg)
+        if suggest_remedy:
+            msg += (
+                "Either install ensurepip, or alleviate the need for it in the"
+                " first place by installing pip and setuptools for "
+                f"'{sys.executable}'.\n"
+            )
+        raise Ouch(prefix + msg)
 
     # ensurepip uses pyexpat, which can also go missing on us:
     if not find_spec("pyexpat"):
@@ -178,12 +178,15 @@ def check_ensurepip(with_pip: bool) -> None:
             "Python's pyexpat module is not found.\n"
             "It's normally part of the Python standard library, "
             "maybe your distribution packages it separately?\n"
-            "Either install pyexpat, or alleviate the need for it in the "
-            "first place by installing pip and setuptools for "
-            f"'{sys.executable}'.\n\n"
-            "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)"
+            "(NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)\n"
         )
-        raise Ouch(msg)
+        if suggest_remedy:
+            msg += (
+                "Either install pyexpat, or alleviate the need for it in the "
+                "first place by installing pip and setuptools for "
+                f"'{sys.executable}'.\n"
+            )
+        raise Ouch(prefix + msg)
 
 
 def make_venv(  # pylint: disable=too-many-arguments
@@ -238,7 +241,8 @@ def make_venv(  # pylint: disable=too-many-arguments
         with_pip = True if not system_site_packages else need_ensurepip()
         logger.debug("with_pip unset, choosing %s", with_pip)
 
-    check_ensurepip(with_pip)
+    if with_pip:
+        check_ensurepip(suggest_remedy=True)
 
     if symlinks is None:
         # Default behavior of standard venv CLI
@@ -430,6 +434,36 @@ def _get_entry_points() -> Iterator[Dict[str, str]]:
         logger.debug("wrote '%s'", script_path)
 
 
+def checkpip() -> None:
+    """
+    Debian10 has a pip that's broken when used inside of a virtual environment.
+
+    We try to detect and correct that case here.
+    """
+    try:
+        # pylint: disable=import-outside-toplevel, unused-import
+        import pip._internal  # noqa: F401
+
+        logger.debug("pip appears to be working correctly.")
+        return
+    except ModuleNotFoundError as exc:
+        if exc.name == "pip._internal":
+            # Uh, fair enough. They did say "internal".
+            # Let's just assume it's fine.
+            return
+        logger.warning("pip appears to be malfunctioning: %s", str(exc))
+
+    check_ensurepip("pip appears to be non-functional, and ")
+
+    logging.debug("Attempting to repair pip ...")
+    subprocess.run(
+        (sys.executable, "-m", "ensurepip"),
+        stdout=subprocess.DEVNULL,
+        check=True,
+    )
+    logging.debug("Pip is now (hopefully) repaired!")
+
+
 def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
     """
     This is intended to be run *inside the venv* after it is created.
@@ -440,6 +474,9 @@ def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
     )
     generate_console_scripts(python_path, bin_path, packages)
 
+    # Test for a broken pip (Debian 10 or derivative?) and fix it if needed
+    checkpip()
+
 
 def _add_create_subcommand(subparsers: Any) -> None:
     subparser = subparsers.add_parser("create", help="create a venv")
-- 
2.39.2

[RFC PATCH v3 07/20] mkvenv: add nested venv workaround

[John Snow]

Python virtual environments do not typically nest; they may inherit from
the top-level system packages or not at all.

For our purposes, it would be convenient to emulate "nested" virtual
environments to allow callers of the configure script to install
specific versions of python utilities in order to test build system
features, utility version compatibility, etc.

While it is possible to install packages into the system environment
(say, by using the --user flag), it's nicer to install test packages
into a totally isolated environment instead.

As detailed in https://www.qemu.org/2023/03/24/python/, Emulate a nested
venv environment by using .pth files installed into the site-packages
folder that points to the parent environment when appropriate.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 72 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 69 insertions(+), 3 deletions(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 445f4eb092..45d1b772e5 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -51,9 +51,11 @@
 import os
 from pathlib import Path
 import re
+import site
 import stat
 import subprocess
 import sys
+import sysconfig
 import traceback
 from types import SimpleNamespace
 from typing import (
@@ -74,6 +76,11 @@
 logger = logging.getLogger("mkvenv")
 
 
+def inside_a_venv() -> bool:
+    """Returns True if it is executed inside of a virtual environment."""
+    return sys.prefix != sys.base_prefix
+
+
 class Ouch(RuntimeError):
     """An Exception class we can't confuse with a builtin."""
 
@@ -82,9 +89,15 @@ class QemuEnvBuilder(venv.EnvBuilder):
     """
     An extension of venv.EnvBuilder for building QEMU's configure-time venv.
 
-    The only functional change is that it adds the ability to regenerate
-    console_script shims for packages available via system_site
-    packages.
+    The primary differences are:
+
+    (1) It adds the ability to regenerate console_script shims for
+    packages available via system_site_packages for any packages
+    specified by the 'script_packages' argument
+
+    (2) It emulates a "nested" virtual environment when invoked from
+    inside of an existing virtual environment by including packages from
+    the parent.
 
     Parameters for base class init:
       - system_site_packages: bool = False
@@ -99,11 +112,51 @@ class QemuEnvBuilder(venv.EnvBuilder):
     def __init__(self, *args: Any, **kwargs: Any) -> None:
         logger.debug("QemuEnvBuilder.__init__(...)")
         self.script_packages = kwargs.pop("script_packages", ())
+
+        # For nested venv emulation:
+        self.use_parent_packages = False
+        if inside_a_venv():
+            # Include parent packages only if we're in a venv and
+            # system_site_packages was True.
+            self.use_parent_packages = kwargs.pop(
+                "system_site_packages", False
+            )
+            # Include system_site_packages only when the parent,
+            # The venv we are currently in, also does so.
+            kwargs["system_site_packages"] = sys.base_prefix in site.PREFIXES
+
         super().__init__(*args, **kwargs)
 
         # Make the context available post-creation:
         self._context: Optional[SimpleNamespace] = None
 
+    def compute_venv_libpath(self, context: SimpleNamespace) -> str:
+        """
+        Compatibility wrapper for context.lib_path for Python < 3.12
+        """
+        # Python 3.12+, not strictly necessary because it's documented
+        # to be the same as 3.10 code below:
+        if sys.version_info >= (3, 12):
+            return context.lib_path
+
+        # Python 3.10+
+        if "venv" in sysconfig.get_scheme_names():
+            return sysconfig.get_path(
+                "purelib", scheme="venv", vars={"base": context.env_dir}
+            )
+
+        # For Python <= 3.9 we need to hardcode this. Fortunately the
+        # code below was the same in Python 3.6-3.10, so there is only
+        # one case.
+        if sys.platform == "win32":
+            return os.path.join(context.env_dir, "Lib", "site-packages")
+        return os.path.join(
+            context.env_dir,
+            "lib",
+            "python%d.%d" % sys.version_info[:2],
+            "site-packages",
+        )
+
     def ensure_directories(self, env_dir: DirType) -> SimpleNamespace:
         logger.debug("ensure_directories(env_dir=%s)", env_dir)
         self._context = super().ensure_directories(env_dir)
@@ -124,6 +177,19 @@ def post_post_setup(self, context: SimpleNamespace) -> None:
         """
         The final, final hook. Enter the venv and run commands inside of it.
         """
+        if self.use_parent_packages:
+            # We're inside of a venv and we want to include the parent
+            # venv's packages.
+            parent_libpath = sysconfig.get_path("purelib")
+            logger.debug("parent_libpath: %s", parent_libpath)
+
+            our_libpath = self.compute_venv_libpath(context)
+            logger.debug("our_libpath: %s", our_libpath)
+
+            pth_file = os.path.join(our_libpath, "nested.pth")
+            with open(pth_file, "w", encoding="UTF-8") as file:
+                file.write(parent_libpath + os.linesep)
+
         args = [
             context.env_exe,
             __file__,
-- 
2.39.2

[RFC PATCH v3 08/20] mkvenv: add ensure subcommand

[John Snow]

This command is to be used to add various packages (or ensure they're
already present) into the configure-provided venv in a modular fashion.

Examples:

mkvenv ensure --online --dir "${source_dir}/python/wheels/" "meson>=0.61.5"
mkvenv ensure --online "sphinx>=1.6.0"
mkvenv ensure "qemu.qmp==0.0.2"

It's designed to look for packages in three places, in order:

(1) In system packages, if the version installed is already good
enough. This way your distribution-provided meson, sphinx, etc are
always used as first preference.

(2) In a vendored packages directory. Here I am suggesting
qemu.git/python/wheels/ as that directory. This is intended to serve as
a replacement for vendoring the meson source for QEMU tarballs. It is
also highly likely to be extremely useful for packaging the "qemu.qmp"
package in source distributions for platforms that do not yet package
qemu.qmp separately.

(3) Online, via PyPI, ***only when "--online" is passed***. This is only
ever used as a fallback if the first two sources do not have an
appropriate package that meets the requirement. The ability to build
QEMU and run tests *completely offline* is not impinged.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 116 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 114 insertions(+), 2 deletions(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 45d1b772e5..937664ea9c 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -13,6 +13,7 @@
     create    create a venv
     post_init
               post-venv initialization
+    ensure    Ensure that the specified package is installed.
 
 --------------------------------------------------
 
@@ -34,6 +35,18 @@
   --gen GEN          Regenerate console_scripts for given packages, if found.
   --binpath BINPATH  Path where console script shims should be generated
 
+--------------------------------------------------
+
+usage: mkvenv ensure [-h] [--online] [--dir DIR] dep_spec
+
+positional arguments:
+  dep_spec    PEP 508 Dependency specification, e.g. 'meson>=0.61.5'
+
+options:
+  -h, --help  show this help message and exit
+  --online    Install packages from PyPI, if necessary.
+  --dir DIR   Path to vendored packages where we may install from.
+
 """
 
 # Copyright (C) 2022-2023 Red Hat, Inc.
@@ -530,6 +543,71 @@ def checkpip() -> None:
     logging.debug("Pip is now (hopefully) repaired!")
 
 
+def pip_install(
+    args: Sequence[str],
+    online: bool = False,
+    wheels_dir: Optional[Union[str, Path]] = None,
+) -> None:
+    """
+    Use pip to install a package or package(s) as specified in @args.
+    """
+    loud = bool(
+        os.environ.get("DEBUG")
+        or os.environ.get("GITLAB_CI")
+        or os.environ.get("V")
+    )
+
+    full_args = [
+        sys.executable,
+        "-m",
+        "pip",
+        "install",
+        "--disable-pip-version-check",
+        "-v" if loud else "-q",
+    ]
+    if not online:
+        full_args += ["--no-index"]
+    if wheels_dir:
+        full_args += ["--find-links", f"file://{str(wheels_dir)}"]
+    full_args += list(args)
+    subprocess.run(full_args, check=True)
+
+
+def ensure(
+    dep_spec: str,
+    online: bool = False,
+    wheels_dir: Optional[Union[str, Path]] = None,
+) -> None:
+    """
+    Use pip to ensure we have the package specified by @dep_spec.
+
+    If the package is already installed, do nothing. If online and
+    wheels_dir are both provided, prefer packages found in wheels_dir
+    first before connecting to PyPI.
+
+    :param dep_spec:
+        A PEP 508 dependency specification. e.g. 'meson>=0.61.5'.
+    :param online: If True, fall back to PyPI.
+    :param wheels_dir: If specified, search this path for packages.
+    """
+    # This first install command will:
+    # (A) Do nothing, if we already have a suitable package.
+    # (B) Install the package from vendored source, if possible.
+    # (C) Fail if neither A nor B.
+    try:
+        pip_install([dep_spec], online=False, wheels_dir=wheels_dir)
+        # (A) or (B) happened. Success.
+        return
+    except subprocess.CalledProcessError:
+        # (C) Happened.
+        # The package is missing or isn't a suitable version,
+        # and we weren't able to install a suitable vendored package.
+        if online:
+            pip_install([dep_spec], online=True)
+        else:
+            raise
+
+
 def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
     """
     This is intended to be run *inside the venv* after it is created.
@@ -578,6 +656,29 @@ def _add_post_init_subcommand(subparsers: Any) -> None:
     )
 
 
+def _add_ensure_subcommand(subparsers: Any) -> None:
+    subparser = subparsers.add_parser(
+        "ensure", help="Ensure that the specified package is installed."
+    )
+    subparser.add_argument(
+        "--online",
+        action="store_true",
+        help="Install packages from PyPI, if necessary.",
+    )
+    subparser.add_argument(
+        "--dir",
+        type=str,
+        action="store",
+        help="Path to vendored packages where we may install from.",
+    )
+    subparser.add_argument(
+        "dep_spec",
+        type=str,
+        action="store",
+        help="PEP 508 Dependency specification, e.g. 'meson>=0.61.5'",
+    )
+
+
 def main() -> int:
     """CLI interface to make_qemu_venv. See module docstring."""
     if os.environ.get("DEBUG") or os.environ.get("GITLAB_CI"):
@@ -600,14 +701,18 @@ def main() -> int:
 
     _add_create_subcommand(subparsers)
     _add_post_init_subcommand(subparsers)
+    _add_ensure_subcommand(subparsers)
 
     args = parser.parse_args()
     script_packages = []
-    for element in args.gen or ():
-        script_packages.extend(element.split(","))
+
+    def _normalize_gen() -> None:
+        for element in args.gen or ():
+            script_packages.extend(element.split(","))
 
     try:
         if args.command == "create":
+            _normalize_gen()
             make_venv(
                 args.target,
                 system_site_packages=True,
@@ -615,7 +720,14 @@ def main() -> int:
                 script_packages=script_packages,
             )
         if args.command == "post_init":
+            _normalize_gen()
             post_venv_setup(args.binpath, script_packages)
+        if args.command == "ensure":
+            ensure(
+                dep_spec=args.dep_spec,
+                online=args.online,
+                wheels_dir=args.dir,
+            )
         logger.debug("mkvenv.py %s: exiting", args.command)
     except Ouch as exc:
         print("\n*** Ouch! ***\n", file=sys.stderr)
-- 
2.39.2

[RFC PATCH v3 09/20] tests/docker: add python3-venv dependency

[John Snow]

Several debian-based tests need the python3-venv dependency as a
consequence of Debian debundling the "ensurepip" module normally
included with Python.

As mkvenv.py stands as of this commit, Debian requires EITHER:

(A) setuptools and pip, or
(B) ensurepip

mkvenv is a few seconds faster if you have setuptools and pip, so
developers should prefer the first requirement. For the purposes of CI,
the time-save is a wash; it's only a matter of who is responsible for
installing pip and when; the timing is about the same.

Arbitrarily, I chose adding ensurepip to the test configuration because
it is normally part of the Python stdlib, and always having it allows us
a more consistent cross-platform environment.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++-
 tests/docker/dockerfiles/debian-hexagon-cross.docker  | 3 ++-
 tests/docker/dockerfiles/debian-riscv64-cross.docker  | 3 ++-
 tests/docker/dockerfiles/debian-tricore-cross.docker  | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-all-test-cross.docker b/tests/docker/dockerfiles/debian-all-test-cross.docker
index 981e9bdc7b..f9f401544a 100644
--- a/tests/docker/dockerfiles/debian-all-test-cross.docker
+++ b/tests/docker/dockerfiles/debian-all-test-cross.docker
@@ -57,7 +57,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         gcc-sh4-linux-gnu \
         libc6-dev-sh4-cross \
         gcc-sparc64-linux-gnu \
-        libc6-dev-sparc64-cross
+        libc6-dev-sparc64-cross \
+        python3-venv
 
 ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
 ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc64-linux-user
diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker
index b99d99f943..c2cfb6a5d0 100644
--- a/tests/docker/dockerfiles/debian-hexagon-cross.docker
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
@@ -20,7 +20,8 @@ RUN apt-get update && \
         bison \
         flex \
         git \
-        ninja-build && \
+        ninja-build \
+        python3-venv && \
 # Install QEMU build deps for use in CI
     DEBIAN_FRONTEND=noninteractive eatmydata \
     apt build-dep -yy --arch-only qemu
diff --git a/tests/docker/dockerfiles/debian-riscv64-cross.docker b/tests/docker/dockerfiles/debian-riscv64-cross.docker
index 803afb9573..081404e014 100644
--- a/tests/docker/dockerfiles/debian-riscv64-cross.docker
+++ b/tests/docker/dockerfiles/debian-riscv64-cross.docker
@@ -28,7 +28,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -yy \
     libglib2.0-dev \
     ninja-build \
     pkg-config \
-    python3
+    python3 \
+    python3-venv
 
 # Add ports and riscv64 architecture
 RUN echo "deb http://ftp.ports.debian.org/debian-ports/ sid main" >> /etc/apt/sources.list
diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index cfd2faf9a8..269bfa8d42 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -33,7 +33,8 @@ RUN apt update && \
        pkgconf \
        python3-pip \
        python3-setuptools \
-       python3-wheel
+       python3-wheel \
+       python3-venv
 
 RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \
     | tar -xzC /usr/local/
-- 
2.39.2

[RFC PATCH v3 10/20] tests/vm: Configure netbsd to use Python 3.10

[John Snow]

NetBSD removes some packages from the Python stdlib, but only
re-packages them for Python 3.10. Switch to using Python 3.10.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/vm/netbsd | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index 0b9536ca17..13eae109c0 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -30,6 +30,7 @@ class NetBSDVM(basevm.BaseVM):
         "git-base",
         "pkgconf",
         "xz",
+        "python310",
         "ninja-build",
 
         # gnu tools
-- 
2.39.2

[RFC PATCH v3 11/20] tests/vm: add py310-expat to NetBSD

[John Snow]

NetBSD cannot successfully run "ensurepip" without access to the pyexpat
module, which NetBSD debundles. Like the Debian patch, it would be
strictly faster long term to install pip/setuptools, and I recommend
developers at their workstations take that approach instead.

For the purposes of a throwaway VM, there's not really a speed
difference for who is responsible for installing pip; us (needs
py310-pip) or Python (needs py310-expat).

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/vm/netbsd | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/vm/netbsd b/tests/vm/netbsd
index 13eae109c0..c7e3f1e735 100755
--- a/tests/vm/netbsd
+++ b/tests/vm/netbsd
@@ -31,6 +31,7 @@ class NetBSDVM(basevm.BaseVM):
         "pkgconf",
         "xz",
         "python310",
+        "py310-expat",
         "ninja-build",
 
         # gnu tools
-- 
2.39.2

[RFC PATCH v3 12/20] scripts/make-release: download meson==0.61.5 .whl

[John Snow]

In preference to vendoring meson source, vendor a built
distributable. This has two benefits:

(1) We can get rid of a git submodule,
(2) Installing built meson into a venv doesn't require any extra dependencies.

RFC:

The alternative approach is to just check in the .whl file into the git
tree directly, and have it available for both git and tarball
installations. That approach slightly changes the necessity of some
subsequent patches, but otherwise either way will work.

Owing to how "mkvenv ensure" will prefer vendored files prior to
connecting to PyPI, checking in a vendored meson file in this manner
means we will generally never use PyPI to install meson ever.

("Vote now on your phones.")

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/make-release | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/scripts/make-release b/scripts/make-release
index 44a9d86a04..a59bad11f9 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -41,6 +41,17 @@ git submodule update --init --single-branch
         BaseTools/Source/C/BrotliCompress/brotli \
         CryptoPkg/Library/OpensslLib/openssl \
         MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
+
+# Handle vendoring Python dependencies:
+mkdir python/wheels
+pushd python/wheels
+pip download meson==0.61.5
+sha256sum -c <<EOF
+58c2ddb5f885da0e929f15d89f38d8a7d97f981f56815bcba008414f8511f59a meson-0.61.5-py3-none-any.whl
+EOF
 popd
+
+popd
+
 tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
-- 
2.39.2

[RFC PATCH v3 13/20] configure: create a python venv unconditionally

[John Snow]

This patch changes the configure script so that it always creates and
uses a python virtual environment unconditionally.

Meson bootstrapping is temporarily altered to force the use of meson
from git or vendored source. The next commit restores the use of
distribution-vendored Meson.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 36 +++++++++++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 5 deletions(-)

diff --git a/configure b/configure
index 77c03315f8..462fe604d6 100755
--- a/configure
+++ b/configure
@@ -625,7 +625,6 @@ check_py_version() {
 python=
 first_python=
 if test -z "${PYTHON}"; then
-    explicit_python=no
     # A bare 'python' is traditionally python 2.x, but some distros
     # have it as python 3.x, so check in both places.
     for binary in python3 python python3.11 python3.10 python3.9 python3.8 python3.7 python3.6; do
@@ -644,7 +643,6 @@ else
     # Same as above, but only check the environment variable.
     has "${PYTHON}" || error_exit "The PYTHON environment variable does not point to an executable"
     python=$(command -v "$PYTHON")
-    explicit_python=yes
     if check_py_version "$python"; then
         # This one is good.
         first_python=
@@ -729,7 +727,7 @@ for opt do
   ;;
   --install=*)
   ;;
-  --python=*) python="$optarg" ; explicit_python=yes
+  --python=*) python="$optarg"
   ;;
   --skip-meson) skip_meson=yes
   ;;
@@ -1089,8 +1087,34 @@ if ! check_py_version "$python"; then
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
-# Resolve PATH + suppress writing compiled files
-python="$(command -v "$python") -B"
+# Resolve PATH
+python="$(command -v "$python")"
+explicit_python=yes
+
+# Create a Python virtual environment using our configured python.
+# The stdout of this script will be the location of a symlink that
+# points to the configured Python.
+# Entry point scripts for pip, meson, and sphinx are generated if those
+# packages are present.
+
+# Defaults assumed for now:
+# - venv is cleared if it exists already;
+# - venv is allowed to use system packages;
+# - all setup is performed **offline**;
+# - No packages are installed by default;
+# - pip is not installed into the venv when possible,
+#   but ensurepip is called as a fallback when necessary.
+
+echo "python determined to be '$python'"
+echo "python version: $($python --version)"
+
+python="$($python -B "${source_path}/python/scripts/mkvenv.py" create --gen pip,meson,sphinx pyvenv)"
+if test "$?" -ne 0 ; then
+    error_exit "python venv creation failed"
+fi
+
+# Suppress writing compiled files
+python="$python -B"
 
 has_meson() {
   local python_dir=$(dirname "$python")
@@ -1145,6 +1169,8 @@ case "$meson" in
     *) meson=$(command -v "$meson") ;;
 esac
 
+echo "MKVENV ok!"
+
 # Probe for ninja
 
 if test -z "$ninja"; then
-- 
2.39.2

[RFC PATCH v3 14/20] configure: use 'mkvenv ensure meson' to bootstrap meson

[John Snow]

This commit changes how we detect and install meson. It notably removes
'--meson='.

The previous patch creates a lightweight Python virtual environment
unconditionally using the user's configured $python that inherits system
packages. If Meson is installed there and meets our minimum version
requirements, we will use that Meson.

In the event that Meson is installed but *not for the chosen Python
interpreter*, not found, or of insufficient version, we will attempt to
install Meson from vendored source into the newly created Python virtual
environment. This vendored installation is considered to replace the
mechanism from prior tarball distributions.

This commit restores the ability to use a system meson, but in turn
temporarily removes the ability to use a meson as obtained from the
internet at configure-time (git submodules, as it stood prior to this
patch); that ability will be restored in the next commit.

As a result of this patch, the Python interpreter we use for both our
own build scripts *and* Meson extensions are always known to be the
exact same Python. As a further benefit, there will also be a symlink
available in the build directory that points to the correct, configured
python and can be used by e.g. manual tests to invoke the correct,
configured Python unambiguously.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure                           | 72 ++++++++---------------------
 .gitlab-ci.d/buildtest-template.yml |  4 +-
 2 files changed, 21 insertions(+), 55 deletions(-)

diff --git a/configure b/configure
index 462fe604d6..e9947369b2 100755
--- a/configure
+++ b/configure
@@ -731,8 +731,6 @@ for opt do
   ;;
   --skip-meson) skip_meson=yes
   ;;
-  --meson=*) meson="$optarg"
-  ;;
   --ninja=*) ninja="$optarg"
   ;;
   --smbd=*) smbd="$optarg"
@@ -1016,7 +1014,6 @@ Advanced options (experts only):
   --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases
   --make=MAKE              use specified make [$make]
   --python=PYTHON          use specified python [$python]
-  --meson=MESON            use specified meson [$meson]
   --ninja=NINJA            use specified ninja [$ninja]
   --smbd=SMBD              use specified smbd [$smbd]
   --with-git=GIT           use specified git [$git]
@@ -1089,7 +1086,8 @@ fi
 
 # Resolve PATH
 python="$(command -v "$python")"
-explicit_python=yes
+# This variable is intended to be used only for error messages:
+target_python=$python
 
 # Create a Python virtual environment using our configured python.
 # The stdout of this script will be the location of a symlink that
@@ -1101,7 +1099,6 @@ explicit_python=yes
 # - venv is cleared if it exists already;
 # - venv is allowed to use system packages;
 # - all setup is performed **offline**;
-# - No packages are installed by default;
 # - pip is not installed into the venv when possible,
 #   but ensurepip is called as a fallback when necessary.
 
@@ -1116,58 +1113,27 @@ fi
 # Suppress writing compiled files
 python="$python -B"
 
-has_meson() {
-  local python_dir=$(dirname "$python")
-  # PEP405: pyvenv.cfg is either adjacent to the Python executable
-  # or one directory above
-  if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then
-    # Ensure that Meson and Python come from the same virtual environment
-    test -x "$python_dir/meson" &&
-      test "$(command -v meson)" -ef "$python_dir/meson"
-  else
-    has meson
-  fi
-}
 
-if test -z "$meson"; then
-    if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then
-        meson=meson
-    elif test "$git_submodules_action" != 'ignore' ; then
-        meson=git
-    elif test -e "${source_path}/meson/meson.py" ; then
-        meson=internal
-    else
-        if test "$explicit_python" = yes; then
-            error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found."
-        else
-            error_exit "Meson not found.  Use --meson=/path/to/meson"
-        fi
+if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \
+     --dir "${source_path}/python/wheels" \
+     "meson>=0.61.5" ;
+then
+    # We're very out of luck. Try to give a good diagnostic.
+    if test -e pyvenv/bin/meson; then
+        echo "Meson is too old: $(pyvenv/bin/meson --version)"
+    elif has meson ; then
+        echo "Meson was found installed on your system," \
+             "but not for the configured Python interpreter ($target_python)."
+        echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)"
     fi
-else
-    # Meson uses its own Python interpreter to invoke other Python scripts,
-    # but the user wants to use the one they specified with --python.
-    #
-    # We do not want to override the distro Python interpreter (and sometimes
-    # cannot: for example in Homebrew /usr/bin/meson is a bash script), so
-    # just require --meson=git|internal together with --python.
-    if test "$explicit_python" = yes; then
-        case "$meson" in
-            git | internal) ;;
-            *) error_exit "--python requires using QEMU's embedded Meson distribution." ;;
-        esac
-    fi
-fi
-
-if test "$meson" = git; then
-    git_submodules="${git_submodules} meson"
+    exit 1
 fi
 
-case "$meson" in
-    git | internal)
-        meson="$python ${source_path}/meson/meson.py"
-        ;;
-    *) meson=$(command -v "$meson") ;;
-esac
+# At this point, we expect Meson to be installed and available.
+# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
+# We ignore PATH completely here: we want to use the venv's Meson
+# *exclusively*.
+meson="$(cd pyvenv/bin; pwd)/meson"
 
 echo "MKVENV ok!"
 
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index a6cfe9be97..7edb50b760 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -12,12 +12,12 @@
     - mkdir build
     - cd build
     - ../configure --enable-werror --disable-docs --enable-fdt=system
-          ${LD_JOBS:+--meson=git} ${TARGETS:+--target-list="$TARGETS"}
+          ${TARGETS:+--target-list="$TARGETS"}
           $CONFIGURE_ARGS ||
       { cat config.log meson-logs/meson-log.txt && exit 1; }
     - if test -n "$LD_JOBS";
       then
-        ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
+        pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
     - make -j"$JOBS"
     - if test -n "$MAKE_CHECK_ARGS";
-- 
2.39.2

[RFC PATCH v3 15/20] configure: add --enable-pypi and --disable-pypi

[John Snow]

In the event that there's no vendored source present and no sufficient
version of meson is found, we will attempt to connect to PyPI to install
the package ... only if '--disable-pypi' was not passed. This mechanism
is intended to replace the online functionality of the meson git submodule.

While --enable-pypi is the default, vendored source will always be
preferred when found, making PyPI a fallback. This should ensure that
configure-time venv building "just works" for almost everyone in almost
every circumstance.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index e9947369b2..7421bb8364 100755
--- a/configure
+++ b/configure
@@ -623,6 +623,7 @@ check_py_version() {
 }
 
 python=
+pypi="enabled"
 first_python=
 if test -z "${PYTHON}"; then
     # A bare 'python' is traditionally python 2.x, but some distros
@@ -883,6 +884,10 @@ for opt do
   --with-git-submodules=*)
       git_submodules_action="$optarg"
   ;;
+  --disable-pypi) pypi="disabled"
+  ;;
+  --enable-pypi) pypi="enabled"
+  ;;
   --enable-plugins) if test "$mingw32" = "yes"; then
                         error_exit "TCG plugins not currently supported on Windows platforms"
                     else
@@ -1098,7 +1103,9 @@ target_python=$python
 # Defaults assumed for now:
 # - venv is cleared if it exists already;
 # - venv is allowed to use system packages;
-# - all setup is performed **offline**;
+# - all setup can be performed offline;
+# - missing packages may be fetched from PyPI,
+#   unless --disable-pypi is passed.
 # - pip is not installed into the venv when possible,
 #   but ensurepip is called as a fallback when necessary.
 
@@ -1114,7 +1121,13 @@ fi
 python="$python -B"
 
 
+mkvenv_flags=""
+if test "$pypi" = "enabled" ; then
+    mkvenv_flags="--online"
+fi
+
 if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \
+     $mkvenv_flags \
      --dir "${source_path}/python/wheels" \
      "meson>=0.61.5" ;
 then
-- 
2.39.2

[RFC PATCH v3 16/20] tests: Use configure-provided pyvenv for tests

[John Snow]

This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use the
configure-provided 'pyvenv' instead, and install optional packages into
that.

Note 1: This doesn't respect the "disable-pypi" setting; the avocado
tests remain an unconditional PyPI hit when you run "make
check-avocado". Future series may endeavor to offer tighter integration,
for now, this is a cheap win that doesn't regress anything.

Note 2: At the time of writing, avocado tests require
avocado-framework<90 whereas the qemu.qmp self-tests rely on
avocado-framework>=90. This collision is avoided for now because the
qemu.git/python/qemu/ code does not need avocado at *runtime*; it does
not install avocado-framework as a necessary dependency and is skipped
in this circumstance.

Nevertheless, we do want to address that discrepancy in the future so
that it will be possible to re-use the same venv for
qemu.git/python/qemu self-tests to introduce them to make check as "make
check-python".

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/devel/acpi-bits.rst                           |  6 +++---
 docs/devel/testing.rst                             | 14 +++++++-------
 .gitlab-ci.d/buildtest.yml                         |  6 +++---
 scripts/ci/org.centos/stream/8/x86_64/test-avocado |  4 ++--
 scripts/device-crash-test                          |  2 +-
 tests/Makefile.include                             | 10 +++++-----
 tests/requirements.txt                             |  7 +++++--
 7 files changed, 26 insertions(+), 23 deletions(-)

diff --git a/docs/devel/acpi-bits.rst b/docs/devel/acpi-bits.rst
index 9eb4b9e666..0c40359109 100644
--- a/docs/devel/acpi-bits.rst
+++ b/docs/devel/acpi-bits.rst
@@ -61,19 +61,19 @@ Under ``tests/avocado/`` as the root we have:
    ::
 
      $ make check-venv (needed only the first time to create the venv)
-     $ ./tests/venv/bin/avocado run -t acpi tests/avocado
+     $ ./pyvenv/bin/avocado run -t acpi tests/avocado
 
    The above will run all acpi avocado tests including this one.
    In order to run the individual tests, perform the following:
    ::
 
-     $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap -
+     $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py --tap -
 
    The above will produce output in tap format. You can omit "--tap -" in the
    end and it will produce output like the following:
    ::
 
-      $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
+      $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py
       Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
       JOB ID     : eab225724da7b64c012c65705dc2fa14ab1defef
       JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4071e72710..50664d9eb9 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -882,9 +882,9 @@ You can run the avocado tests simply by executing:
 
   make check-avocado
 
-This involves the automatic creation of Python virtual environment
-within the build tree (at ``tests/venv``) which will have all the
-right dependencies, and will save tests results also within the
+This involves the automatic installation, from PyPI, of all the
+necessary avocado-framework dependencies into the QEMU venv within the
+build tree (at ``./pyvenv``). Test results are also saved within the
 build tree (at ``tests/results``).
 
 Note: the build environment must be using a Python 3 stack, and have
@@ -941,7 +941,7 @@ may be invoked by running:
 
  .. code::
 
-  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
+  pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
 
 Note that if ``make check-avocado`` was not executed before, it is
 possible to create the Python virtual environment with the dependencies
@@ -956,20 +956,20 @@ a test file. To run tests from a single file within the build tree, use:
 
  .. code::
 
-  tests/venv/bin/avocado run tests/avocado/$TESTFILE
+  pyvenv/bin/avocado run tests/avocado/$TESTFILE
 
 To run a single test within a test file, use:
 
  .. code::
 
-  tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
+  pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
 
 Valid test names are visible in the output from any previous execution
 of Avocado or ``make check-avocado``, and can also be queried using:
 
  .. code::
 
-  tests/venv/bin/avocado list tests/avocado
+  pyvenv/bin/avocado list tests/avocado
 
 Manual Installation
 ~~~~~~~~~~~~~~~~~~~
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index ba6f551752..53de9f23c4 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -103,7 +103,7 @@ crash-test-debian:
   script:
     - cd build
     - make check-venv
-    - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386
+    - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386
 
 build-system-fedora:
   extends:
@@ -146,8 +146,8 @@ crash-test-fedora:
   script:
     - cd build
     - make check-venv
-    - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
-    - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
+    - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
+    - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
 
 build-system-centos:
   extends:
diff --git a/scripts/ci/org.centos/stream/8/x86_64/test-avocado b/scripts/ci/org.centos/stream/8/x86_64/test-avocado
index d2c0e5fb4c..7bb5b317b6 100755
--- a/scripts/ci/org.centos/stream/8/x86_64/test-avocado
+++ b/scripts/ci/org.centos/stream/8/x86_64/test-avocado
@@ -4,7 +4,7 @@
 # KVM and x86_64, or tests that are generic enough to be valid for all
 # targets. Such a test list can be generated with:
 #
-# ./tests/venv/bin/avocado list --filter-by-tags-include-empty \
+# ./pyvenv/bin/avocado list --filter-by-tags-include-empty \
 #   --filter-by-tags-include-empty-key -t accel:kvm,arch:x86_64 \
 #   tests/avocado/
 #
@@ -22,7 +22,7 @@
 #   - tests/avocado/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types
 #
 make get-vm-images
-./tests/venv/bin/avocado run \
+./pyvenv/bin/avocado run \
     --job-results-dir=tests/results/ \
     tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_kvm \
     tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm \
diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 73bcb98693..ef6ac262fa 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -43,7 +43,7 @@ except ModuleNotFoundError as exc:
     print(f"Module '{exc.name}' not found.")
     print("  Try 'make check-venv' from your build directory,")
     print("  and then one way to run this script is like so:")
-    print(f'  > $builddir/tests/venv/bin/python3 "{path}"')
+    print(f'  > $builddir/pyvenv/bin/python3 "{path}"')
     sys.exit(1)
 
 logger = logging.getLogger('device-crash-test')
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 9422ddaece..3f39c607cc 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -89,7 +89,8 @@ distclean-tcg: $(DISTCLEAN_TCG_TARGET_RULES)
 # Build up our target list from the filtered list of ninja targets
 TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets)))
 
-TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
+TESTS_VENV_DIR=$(BUILD_DIR)/pyvenv
+TESTS_VENV_TOKEN=$(BUILD_DIR)/pyvenv/tests.group
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
 TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 TESTS_PYTHON=$(TESTS_VENV_DIR)/bin/python3
@@ -111,8 +112,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
     $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \
     "VENVPIP","$1")
 
-$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
-	$(call quiet-command, $(PYTHON) -m venv $@, VENV, $@)
+$(TESTS_VENV_TOKEN): $(TESTS_VENV_REQ)
 	$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
 	$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
 	$(call quiet-command, touch $@)
@@ -121,7 +121,7 @@ $(TESTS_RESULTS_DIR):
 	$(call quiet-command, mkdir -p $@, \
             MKDIR, $@)
 
-check-venv: $(TESTS_VENV_DIR)
+check-venv: $(TESTS_VENV_TOKEN)
 
 FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS)))
 FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS))
@@ -163,7 +163,7 @@ check:
 check-build: run-ninja
 
 check-clean:
-	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
+	rm -rf $(TESTS_RESULTS_DIR)
 
 clean: check-clean clean-tcg
 distclean: distclean-tcg
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 0ba561b6bd..07e713ef5a 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,6 +1,9 @@
 # Add Python module requirements, one per line, to be installed
-# in the tests/venv Python virtual environment. For more info,
+# in the qemu build_dir/pyvenv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-# Note that qemu.git/python/ is always implicitly installed.
+#
+# Note that qemu.git/python/ is implicitly installed to this venv when
+# 'make check-venv' is run, and will persist until configure is run
+# again.
 avocado-framework==88.1
 pycdlib==1.11.0
-- 
2.39.2

[RFC PATCH v3 17/20] configure: move --enable-docs and --disable-docs back to configure

[John Snow]

Move this option back from meson into configure for the purposes of
using the configuration value to bootstrap Sphinx in different ways
based on this value.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/configure b/configure
index 7421bb8364..ae55de1408 100755
--- a/configure
+++ b/configure
@@ -279,6 +279,7 @@ debug_tcg="no"
 sanitizers="no"
 tsan="no"
 fortify_source="yes"
+docs="auto"
 EXESUF=""
 modules="no"
 prefix="/usr/local"
@@ -752,6 +753,10 @@ for opt do
   ;;
   --disable-debug-info) meson_option_add -Ddebug=false
   ;;
+  --enable-docs) docs=enabled
+  ;;
+  --disable-docs) docs=disabled
+  ;;
   --enable-modules)
       modules="yes"
   ;;
@@ -2638,6 +2643,7 @@ if test "$skip_meson" = no; then
 
   # QEMU options
   test "$cfi" != false && meson_option_add "-Dcfi=$cfi"
+  test "$docs" != auto && meson_option_add "-Ddocs=$docs"
   test "$fdt" != auto && meson_option_add "-Dfdt=$fdt"
   test -n "${LIB_FUZZING_ENGINE+xxx}" && meson_option_add "-Dfuzzing_engine=$LIB_FUZZING_ENGINE"
   test "$qemu_suffix" != qemu && meson_option_add "-Dqemu_suffix=$qemu_suffix"
-- 
2.39.2

[RFC PATCH v3 18/20] mkvenv: add diagnose() method for ensure() failures

[John Snow]

This is a routine that is designed to print some usable info for human
beings back out to the terminal if/when "mkvenv ensure" fails to locate
or install a package during configure time, such as meson or sphinx.

Since we are requiring that "meson" and "sphinx" are installed to the
same Python environment as QEMU is configured to build with, this can
produce some surprising failures when things are mismatched. This method
is here to try and ease that sting by offering some actionable
diagnosis.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 153 +++++++++++++++++++++++++++++++++++----
 1 file changed, 140 insertions(+), 13 deletions(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 937664ea9c..1bc4bc01b1 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -64,6 +64,7 @@
 import os
 from pathlib import Path
 import re
+import shutil
 import site
 import stat
 import subprocess
@@ -543,6 +544,103 @@ def checkpip() -> None:
     logging.debug("Pip is now (hopefully) repaired!")
 
 
+def diagnose(
+    dep_spec: str,
+    online: bool,
+    wheels_dir: Optional[Union[str, Path]],
+    prog: Optional[str],
+) -> str:
+    """
+    Offer a summary to the user as to why a package failed to be installed.
+
+    :param dep_spec: The package we tried to ensure, e.g. 'meson>=0.61.5'
+    :param online: Did we allow PyPI access?
+    :param prog:
+        Optionally, a shell program name that can be used as a
+        bellwether to detect if this program is installed elsewhere on
+        the system. This is used to offer advice when a program is
+        detected for a different python version.
+    :param wheels_dir:
+        Optionally, a directory that was searched for vendored packages.
+    """
+    # pylint: disable=too-many-branches
+
+    # Parse name out of PEP-508 depspec.
+    # See https://peps.python.org/pep-0508/#names
+    match = re.match(
+        r"^([A-Z0-9]([A-Z0-9._-]*[A-Z0-9])?)", dep_spec, re.IGNORECASE
+    )
+    if not match:
+        raise ValueError(
+            f"dep_spec '{dep_spec}'"
+            " does not appear to contain a valid package name"
+        )
+    pkg_name = match.group(0)
+    pkg_version = None
+
+    has_importlib = False
+    try:
+        # Python 3.8+ stdlib
+        # pylint: disable=import-outside-toplevel
+        from importlib.metadata import PackageNotFoundError, version
+
+        has_importlib = True
+        try:
+            pkg_version = version(pkg_name)
+        except PackageNotFoundError:
+            pass
+    except ModuleNotFoundError:
+        pass
+
+    lines = []
+
+    if pkg_version:
+        lines.append(
+            f"Python package '{pkg_name}' version '{pkg_version}' was found,"
+            " but isn't suitable."
+        )
+    elif has_importlib:
+        lines.append(
+            f"Python package '{pkg_name}' was not found nor installed."
+        )
+    else:
+        lines.append(
+            f"Python package '{pkg_name}' is either not found or"
+            " not a suitable version."
+        )
+
+    if wheels_dir:
+        lines.append(
+            "No suitable version found in, or failed to install from"
+            f" '{wheels_dir}'."
+        )
+    else:
+        lines.append("No local package directory was searched.")
+
+    if online:
+        lines.append("A suitable version could not be obtained from PyPI.")
+    else:
+        lines.append(
+            "mkvenv was configured to operate offline and did not check PyPI."
+        )
+
+    if prog and not pkg_version:
+        which = shutil.which(prog)
+        if which:
+            pypath = Path(sys.executable).resolve()
+            lines.append(
+                f"'{prog}' was detected on your system at '{which}', "
+                f"but the Python package '{pkg_name}' was not found by this "
+                f"Python interpreter ('{pypath}'). "
+                f"Typically this means that '{prog}' has been installed "
+                "against a different Python interpreter on your system."
+            )
+
+    lines = [f" • {line}" for line in lines]
+    lines.insert(0, f"Could not ensure availability of '{dep_spec}':")
+    return os.linesep.join(lines)
+
+
 def pip_install(
     args: Sequence[str],
     online: bool = False,
@@ -573,23 +671,11 @@ def pip_install(
     subprocess.run(full_args, check=True)
 
 
-def ensure(
+def _do_ensure(
     dep_spec: str,
     online: bool = False,
     wheels_dir: Optional[Union[str, Path]] = None,
 ) -> None:
-    """
-    Use pip to ensure we have the package specified by @dep_spec.
-
-    If the package is already installed, do nothing. If online and
-    wheels_dir are both provided, prefer packages found in wheels_dir
-    first before connecting to PyPI.
-
-    :param dep_spec:
-        A PEP 508 dependency specification. e.g. 'meson>=0.61.5'.
-    :param online: If True, fall back to PyPI.
-    :param wheels_dir: If specified, search this path for packages.
-    """
     # This first install command will:
     # (A) Do nothing, if we already have a suitable package.
     # (B) Install the package from vendored source, if possible.
@@ -603,11 +689,42 @@ def ensure(
         # The package is missing or isn't a suitable version,
         # and we weren't able to install a suitable vendored package.
         if online:
+            logger.info("offline ensure failed, trying PyPI ...")
             pip_install([dep_spec], online=True)
         else:
             raise
 
 
+def ensure(
+    dep_spec: str,
+    online: bool = False,
+    wheels_dir: Optional[Union[str, Path]] = None,
+    prog: Optional[str] = None,
+) -> None:
+    """
+    Use pip to ensure we have the package specified by @dep_spec.
+
+    If the package is already installed, do nothing. If online and
+    wheels_dir are both provided, prefer packages found in wheels_dir
+    first before connecting to PyPI.
+
+    :param dep_spec:
+        A PEP 508 dependency specification. e.g. 'meson>=0.61.5'.
+    :param online: If True, fall back to PyPI.
+    :param wheels_dir: If specified, search this path for packages.
+    :param prog:
+        If specified, use this program name for error diagnostics that will
+        be presented to the user. e.g., 'sphinx-build' can be used as a
+        bellwether for the presence of 'sphinx'.
+    """
+    print(f"MKVENV ensure {dep_spec}", file=sys.stderr)
+    try:
+        _do_ensure(dep_spec, online, wheels_dir)
+    except subprocess.CalledProcessError as exc:
+        # Well, that's not good.
+        raise Ouch(diagnose(dep_spec, online, wheels_dir, prog)) from exc
+
+
 def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
     """
     This is intended to be run *inside the venv* after it is created.
@@ -671,6 +788,15 @@ def _add_ensure_subcommand(subparsers: Any) -> None:
         action="store",
         help="Path to vendored packages where we may install from.",
     )
+    subparser.add_argument(
+        "--diagnose",
+        type=str,
+        action="store",
+        help=(
+            "Name of a shell utility to use for "
+            "diagnostics if this command fails."
+        ),
+    )
     subparser.add_argument(
         "dep_spec",
         type=str,
@@ -727,6 +853,7 @@ def _normalize_gen() -> None:
                 dep_spec=args.dep_spec,
                 online=args.online,
                 wheels_dir=args.dir,
+                prog=args.diagnose,
             )
         logger.debug("mkvenv.py %s: exiting", args.command)
     except Ouch as exc:
-- 
2.39.2

[RFC PATCH v3 19/20] configure: use --diagnose option with meson ensure

[John Snow]

Rely on the diagnosis information from mkvenv so it can be removed from
the configure script.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 11 +----------
 1 file changed, 1 insertion(+), 10 deletions(-)

diff --git a/configure b/configure
index ae55de1408..0881fffc14 100755
--- a/configure
+++ b/configure
@@ -1096,8 +1096,6 @@ fi
 
 # Resolve PATH
 python="$(command -v "$python")"
-# This variable is intended to be used only for error messages:
-target_python=$python
 
 # Create a Python virtual environment using our configured python.
 # The stdout of this script will be the location of a symlink that
@@ -1134,16 +1132,9 @@ fi
 if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \
      $mkvenv_flags \
      --dir "${source_path}/python/wheels" \
+     --diagnose "meson" \
      "meson>=0.61.5" ;
 then
-    # We're very out of luck. Try to give a good diagnostic.
-    if test -e pyvenv/bin/meson; then
-        echo "Meson is too old: $(pyvenv/bin/meson --version)"
-    elif has meson ; then
-        echo "Meson was found installed on your system," \
-             "but not for the configured Python interpreter ($target_python)."
-        echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)"
-    fi
     exit 1
 fi
 
-- 
2.39.2

[RFC PATCH v3 20/20] configure: bootstrap sphinx with mkvenv

[John Snow]

When docs are explicitly requested, require Sphinx>=1.6.0. When docs are
explicitly disabled, don't bother to check for Sphinx at all. If docs
are set to "auto", attempt to locate Sphinx, but continue onward if it
wasn't located.

For the case that --enable-pypi is set to 'enabled' (the default) but
docs are set to 'auto' (also the default), do not actually consult PyPI
to install Sphinx. Only perform this action when docs are requested
explicitly.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index 0881fffc14..a247b9491c 100755
--- a/configure
+++ b/configure
@@ -1122,14 +1122,14 @@ fi
 
 # Suppress writing compiled files
 python="$python -B"
-
+mkvenv="$python ${source_path}/python/scripts/mkvenv.py"
 
 mkvenv_flags=""
 if test "$pypi" = "enabled" ; then
     mkvenv_flags="--online"
 fi
 
-if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \
+if ! $mkvenv ensure \
      $mkvenv_flags \
      --dir "${source_path}/python/wheels" \
      --diagnose "meson" \
@@ -1144,6 +1144,29 @@ fi
 # *exclusively*.
 meson="$(cd pyvenv/bin; pwd)/meson"
 
+# Conditionally ensure Sphinx is installed.
+
+mkvenv_flags=""
+if test "$pypi" = "enabled" -a "$docs" = "enabled" ; then
+    mkvenv_flags="--online"
+fi
+
+if test "$docs" != "disabled" ; then
+    if ! $mkvenv ensure \
+         $mkvenv_flags \
+         --diagnose "sphinx-build" \
+         "sphinx>=1.6.0" ;
+    then
+        if test "$docs" = "enabled" ; then
+            exit 1
+        fi
+        echo "Sphinx not found/usable, disabling docs."
+        docs=disabled
+    else
+        docs=enabled
+    fi
+fi
+
 echo "MKVENV ok!"
 
 # Probe for ninja
-- 
2.39.2

Re: [RFC PATCH v3 14/20] configure: use 'mkvenv ensure meson' to bootstrap meson

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 7061 bytes --]

On Mon, Apr 24, 2023, 2:03 PM John Snow <jsnow@redhat.com> wrote:

> This commit changes how we detect and install meson. It notably removes
> '--meson='.
>
> The previous patch creates a lightweight Python virtual environment
> unconditionally using the user's configured $python that inherits system
> packages. If Meson is installed there and meets our minimum version
> requirements, we will use that Meson.
>
> In the event that Meson is installed but *not for the chosen Python
> interpreter*, not found, or of insufficient version, we will attempt to
> install Meson from vendored source into the newly created Python virtual
> environment. This vendored installation is considered to replace the
> mechanism from prior tarball distributions.
>
> This commit restores the ability to use a system meson, but in turn
> temporarily removes the ability to use a meson as obtained from the
> internet at configure-time (git submodules, as it stood prior to this
> patch); that ability will be restored in the next commit.
>
> As a result of this patch, the Python interpreter we use for both our
> own build scripts *and* Meson extensions are always known to be the
> exact same Python. As a further benefit, there will also be a symlink
> available in the build directory that points to the correct, configured
> python and can be used by e.g. manual tests to invoke the correct,
> configured Python unambiguously.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  configure                           | 72 ++++++++---------------------
>  .gitlab-ci.d/buildtest-template.yml |  4 +-
>  2 files changed, 21 insertions(+), 55 deletions(-)
>
> diff --git a/configure b/configure
> index 462fe604d6..e9947369b2 100755
> --- a/configure
> +++ b/configure
> @@ -731,8 +731,6 @@ for opt do
>    ;;
>    --skip-meson) skip_meson=yes
>    ;;
> -  --meson=*) meson="$optarg"
> -  ;;
>    --ninja=*) ninja="$optarg"
>    ;;
>    --smbd=*) smbd="$optarg"
> @@ -1016,7 +1014,6 @@ Advanced options (experts only):
>    --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH
> guest test cases
>    --make=MAKE              use specified make [$make]
>    --python=PYTHON          use specified python [$python]
> -  --meson=MESON            use specified meson [$meson]
>    --ninja=NINJA            use specified ninja [$ninja]
>    --smbd=SMBD              use specified smbd [$smbd]
>    --with-git=GIT           use specified git [$git]
> @@ -1089,7 +1086,8 @@ fi
>
>  # Resolve PATH
>  python="$(command -v "$python")"
> -explicit_python=yes
> +# This variable is intended to be used only for error messages:
> +target_python=$python
>
>  # Create a Python virtual environment using our configured python.
>  # The stdout of this script will be the location of a symlink that
> @@ -1101,7 +1099,6 @@ explicit_python=yes
>  # - venv is cleared if it exists already;
>  # - venv is allowed to use system packages;
>  # - all setup is performed **offline**;
> -# - No packages are installed by default;
>  # - pip is not installed into the venv when possible,
>  #   but ensurepip is called as a fallback when necessary.
>
> @@ -1116,58 +1113,27 @@ fi
>  # Suppress writing compiled files
>  python="$python -B"
>
> -has_meson() {
> -  local python_dir=$(dirname "$python")
> -  # PEP405: pyvenv.cfg is either adjacent to the Python executable
> -  # or one directory above
> -  if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg;
> then
> -    # Ensure that Meson and Python come from the same virtual environment
> -    test -x "$python_dir/meson" &&
> -      test "$(command -v meson)" -ef "$python_dir/meson"
> -  else
> -    has meson
> -  fi
> -}
>
> -if test -z "$meson"; then
> -    if test "$explicit_python" = no && has_meson && version_ge "$(meson
> --version)" 0.61.5; then
> -        meson=meson
> -    elif test "$git_submodules_action" != 'ignore' ; then
> -        meson=git
> -    elif test -e "${source_path}/meson/meson.py" ; then
> -        meson=internal
> -    else
> -        if test "$explicit_python" = yes; then
> -            error_exit "--python requires using QEMU's embedded Meson
> distribution, but it was not found."
> -        else
> -            error_exit "Meson not found.  Use --meson=/path/to/meson"
> -        fi
> +if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \
> +     --dir "${source_path}/python/wheels" \
> +     "meson>=0.61.5" ;
> +then
> +    # We're very out of luck. Try to give a good diagnostic.
> +    if test -e pyvenv/bin/meson; then
> +        echo "Meson is too old:


Does a minimum version still get printed? I've needed to know that in the
past when I got the error...

Warner
 $(pyvenv/bin/meson --version)"

> +    elif has meson ; then
> +        echo "Meson was found installed on your system," \
> +             "but not for the configured Python interpreter
> ($target_python)."
> +        echo "(Hint: check '$(which meson)' to see which interpreter its
> shebang uses.)"
>      fi
> -else
> -    # Meson uses its own Python interpreter to invoke other Python
> scripts,
> -    # but the user wants to use the one they specified with --python.
> -    #
> -    # We do not want to override the distro Python interpreter (and
> sometimes
> -    # cannot: for example in Homebrew /usr/bin/meson is a bash script), so
> -    # just require --meson=git|internal together with --python.
> -    if test "$explicit_python" = yes; then
> -        case "$meson" in
> -            git | internal) ;;
> -            *) error_exit "--python requires using QEMU's embedded Meson
> distribution." ;;
> -        esac
> -    fi
> -fi
> -
> -if test "$meson" = git; then
> -    git_submodules="${git_submodules} meson"
> +    exit 1
>  fi
>
> -case "$meson" in
> -    git | internal)
> -        meson="$python ${source_path}/meson/meson.py"
> -        ;;
> -    *) meson=$(command -v "$meson") ;;
> -esac
> +# At this point, we expect Meson to be installed and available.
> +# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
> +# We ignore PATH completely here: we want to use the venv's Meson
> +# *exclusively*.
> +meson="$(cd pyvenv/bin; pwd)/meson"
>
>  echo "MKVENV ok!"
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml
> b/.gitlab-ci.d/buildtest-template.yml
> index a6cfe9be97..7edb50b760 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -12,12 +12,12 @@
>      - mkdir build
>      - cd build
>      - ../configure --enable-werror --disable-docs --enable-fdt=system
> -          ${LD_JOBS:+--meson=git} ${TARGETS:+--target-list="$TARGETS"}
> +          ${TARGETS:+--target-list="$TARGETS"}
>            $CONFIGURE_ARGS ||
>        { cat config.log meson-logs/meson-log.txt && exit 1; }
>      - if test -n "$LD_JOBS";
>        then
> -        ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
> +        pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ;
>        fi || exit 1;
>      - make -j"$JOBS"
>      - if test -n "$MAKE_CHECK_ARGS";
> --
> 2.39.2
>
>

[-- Attachment #2: Type: text/html, Size: 8990 bytes --]

Re: [RFC PATCH v3 14/20] configure: use 'mkvenv ensure meson' to bootstrap meson

[John Snow]

On Mon, Apr 24, 2023 at 4:36 PM Warner Losh <imp@bsdimp.com> wrote:
>
>
>
> On Mon, Apr 24, 2023, 2:03 PM John Snow <jsnow@redhat.com> wrote:
>>
>> This commit changes how we detect and install meson. It notably removes
>> '--meson='.
>>
>> The previous patch creates a lightweight Python virtual environment
>> unconditionally using the user's configured $python that inherits system
>> packages. If Meson is installed there and meets our minimum version
>> requirements, we will use that Meson.
>>
>> In the event that Meson is installed but *not for the chosen Python
>> interpreter*, not found, or of insufficient version, we will attempt to
>> install Meson from vendored source into the newly created Python virtual
>> environment. This vendored installation is considered to replace the
>> mechanism from prior tarball distributions.
>>
>> This commit restores the ability to use a system meson, but in turn
>> temporarily removes the ability to use a meson as obtained from the
>> internet at configure-time (git submodules, as it stood prior to this
>> patch); that ability will be restored in the next commit.
>>
>> As a result of this patch, the Python interpreter we use for both our
>> own build scripts *and* Meson extensions are always known to be the
>> exact same Python. As a further benefit, there will also be a symlink
>> available in the build directory that points to the correct, configured
>> python and can be used by e.g. manual tests to invoke the correct,
>> configured Python unambiguously.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>  configure                           | 72 ++++++++---------------------
>>  .gitlab-ci.d/buildtest-template.yml |  4 +-
>>  2 files changed, 21 insertions(+), 55 deletions(-)
>>
>> diff --git a/configure b/configure
>> index 462fe604d6..e9947369b2 100755
>> --- a/configure
>> +++ b/configure
>> @@ -731,8 +731,6 @@ for opt do
>>    ;;
>>    --skip-meson) skip_meson=yes
>>    ;;
>> -  --meson=*) meson="$optarg"
>> -  ;;
>>    --ninja=*) ninja="$optarg"
>>    ;;
>>    --smbd=*) smbd="$optarg"
>> @@ -1016,7 +1014,6 @@ Advanced options (experts only):
>>    --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases
>>    --make=MAKE              use specified make [$make]
>>    --python=PYTHON          use specified python [$python]
>> -  --meson=MESON            use specified meson [$meson]
>>    --ninja=NINJA            use specified ninja [$ninja]
>>    --smbd=SMBD              use specified smbd [$smbd]
>>    --with-git=GIT           use specified git [$git]
>> @@ -1089,7 +1086,8 @@ fi
>>
>>  # Resolve PATH
>>  python="$(command -v "$python")"
>> -explicit_python=yes
>> +# This variable is intended to be used only for error messages:
>> +target_python=$python
>>
>>  # Create a Python virtual environment using our configured python.
>>  # The stdout of this script will be the location of a symlink that
>> @@ -1101,7 +1099,6 @@ explicit_python=yes
>>  # - venv is cleared if it exists already;
>>  # - venv is allowed to use system packages;
>>  # - all setup is performed **offline**;
>> -# - No packages are installed by default;
>>  # - pip is not installed into the venv when possible,
>>  #   but ensurepip is called as a fallback when necessary.
>>
>> @@ -1116,58 +1113,27 @@ fi
>>  # Suppress writing compiled files
>>  python="$python -B"
>>
>> -has_meson() {
>> -  local python_dir=$(dirname "$python")
>> -  # PEP405: pyvenv.cfg is either adjacent to the Python executable
>> -  # or one directory above
>> -  if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then
>> -    # Ensure that Meson and Python come from the same virtual environment
>> -    test -x "$python_dir/meson" &&
>> -      test "$(command -v meson)" -ef "$python_dir/meson"
>> -  else
>> -    has meson
>> -  fi
>> -}
>>
>> -if test -z "$meson"; then
>> -    if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then
>> -        meson=meson
>> -    elif test "$git_submodules_action" != 'ignore' ; then
>> -        meson=git
>> -    elif test -e "${source_path}/meson/meson.py" ; then
>> -        meson=internal
>> -    else
>> -        if test "$explicit_python" = yes; then
>> -            error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found."
>> -        else
>> -            error_exit "Meson not found.  Use --meson=/path/to/meson"
>> -        fi
>> +if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \
>> +     --dir "${source_path}/python/wheels" \
>> +     "meson>=0.61.5" ;
>> +then
>> +    # We're very out of luck. Try to give a good diagnostic.
>> +    if test -e pyvenv/bin/meson; then
>> +        echo "Meson is too old:
>
>
> Does a minimum version still get printed? I've needed to know that in the past when I got the error...
>
> Warner
>  $(pyvenv/bin/meson --version)"

At the end of the series, here's what happens if i change the meson
requirement to a fictionally too-high version that would be impossible
to fulfill:

--------------------------------------------------------------------------------
jsnow@scv ~/s/q/b/git (python-configure-venv)> ../../configure
--enable-debug --disable-pypi;
python determined to be '/usr/bin/python3'
python version: Python 3.11.2
MKVENV pyvenv
MKVENV ensure meson>=2.61.5
ERROR: Could not find a version that satisfies the requirement
meson>=2.61.5 (from versions: none)
ERROR: No matching distribution found for meson>=2.61.5

*** Ouch! ***

Could not ensure availability of 'meson>=2.61.5':
 • Python package 'meson' version '1.0.1' was found, but isn't suitable.
 • No suitable version found in, or failed to install from
'/home/jsnow/src/qemu/python/wheels'.
 • mkvenv was configured to operate offline and did not check PyPI.
--------------------------------------------------------------------------------

Re: [RFC PATCH v3 14/20] configure: use 'mkvenv ensure meson' to bootstrap meson

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 6501 bytes --]

On Mon, Apr 24, 2023 at 2:41 PM John Snow <jsnow@redhat.com> wrote:

> On Mon, Apr 24, 2023 at 4:36 PM Warner Losh <imp@bsdimp.com> wrote:
> >
> >
> >
> > On Mon, Apr 24, 2023, 2:03 PM John Snow <jsnow@redhat.com> wrote:
> >>
> >> This commit changes how we detect and install meson. It notably removes
> >> '--meson='.
> >>
> >> The previous patch creates a lightweight Python virtual environment
> >> unconditionally using the user's configured $python that inherits system
> >> packages. If Meson is installed there and meets our minimum version
> >> requirements, we will use that Meson.
> >>
> >> In the event that Meson is installed but *not for the chosen Python
> >> interpreter*, not found, or of insufficient version, we will attempt to
> >> install Meson from vendored source into the newly created Python virtual
> >> environment. This vendored installation is considered to replace the
> >> mechanism from prior tarball distributions.
> >>
> >> This commit restores the ability to use a system meson, but in turn
> >> temporarily removes the ability to use a meson as obtained from the
> >> internet at configure-time (git submodules, as it stood prior to this
> >> patch); that ability will be restored in the next commit.
> >>
> >> As a result of this patch, the Python interpreter we use for both our
> >> own build scripts *and* Meson extensions are always known to be the
> >> exact same Python. As a further benefit, there will also be a symlink
> >> available in the build directory that points to the correct, configured
> >> python and can be used by e.g. manual tests to invoke the correct,
> >> configured Python unambiguously.
> >>
> >> Signed-off-by: John Snow <jsnow@redhat.com>
> >> ---
> >>  configure                           | 72 ++++++++---------------------
> >>  .gitlab-ci.d/buildtest-template.yml |  4 +-
> >>  2 files changed, 21 insertions(+), 55 deletions(-)
> >>
> >> diff --git a/configure b/configure
> >> index 462fe604d6..e9947369b2 100755
> >> --- a/configure
> >> +++ b/configure
> >> @@ -731,8 +731,6 @@ for opt do
> >>    ;;
> >>    --skip-meson) skip_meson=yes
> >>    ;;
> >> -  --meson=*) meson="$optarg"
> >> -  ;;
> >>    --ninja=*) ninja="$optarg"
> >>    ;;
> >>    --smbd=*) smbd="$optarg"
> >> @@ -1016,7 +1014,6 @@ Advanced options (experts only):
> >>    --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH
> guest test cases
> >>    --make=MAKE              use specified make [$make]
> >>    --python=PYTHON          use specified python [$python]
> >> -  --meson=MESON            use specified meson [$meson]
> >>    --ninja=NINJA            use specified ninja [$ninja]
> >>    --smbd=SMBD              use specified smbd [$smbd]
> >>    --with-git=GIT           use specified git [$git]
> >> @@ -1089,7 +1086,8 @@ fi
> >>
> >>  # Resolve PATH
> >>  python="$(command -v "$python")"
> >> -explicit_python=yes
> >> +# This variable is intended to be used only for error messages:
> >> +target_python=$python
> >>
> >>  # Create a Python virtual environment using our configured python.
> >>  # The stdout of this script will be the location of a symlink that
> >> @@ -1101,7 +1099,6 @@ explicit_python=yes
> >>  # - venv is cleared if it exists already;
> >>  # - venv is allowed to use system packages;
> >>  # - all setup is performed **offline**;
> >> -# - No packages are installed by default;
> >>  # - pip is not installed into the venv when possible,
> >>  #   but ensurepip is called as a fallback when necessary.
> >>
> >> @@ -1116,58 +1113,27 @@ fi
> >>  # Suppress writing compiled files
> >>  python="$python -B"
> >>
> >> -has_meson() {
> >> -  local python_dir=$(dirname "$python")
> >> -  # PEP405: pyvenv.cfg is either adjacent to the Python executable
> >> -  # or one directory above
> >> -  if test -f $python_dir/pyvenv.cfg || test -f
> $python_dir/../pyvenv.cfg; then
> >> -    # Ensure that Meson and Python come from the same virtual
> environment
> >> -    test -x "$python_dir/meson" &&
> >> -      test "$(command -v meson)" -ef "$python_dir/meson"
> >> -  else
> >> -    has meson
> >> -  fi
> >> -}
> >>
> >> -if test -z "$meson"; then
> >> -    if test "$explicit_python" = no && has_meson && version_ge
> "$(meson --version)" 0.61.5; then
> >> -        meson=meson
> >> -    elif test "$git_submodules_action" != 'ignore' ; then
> >> -        meson=git
> >> -    elif test -e "${source_path}/meson/meson.py" ; then
> >> -        meson=internal
> >> -    else
> >> -        if test "$explicit_python" = yes; then
> >> -            error_exit "--python requires using QEMU's embedded Meson
> distribution, but it was not found."
> >> -        else
> >> -            error_exit "Meson not found.  Use --meson=/path/to/meson"
> >> -        fi
> >> +if ! $python "${source_path}/python/scripts/mkvenv.py" ensure \
> >> +     --dir "${source_path}/python/wheels" \
> >> +     "meson>=0.61.5" ;
> >> +then
> >> +    # We're very out of luck. Try to give a good diagnostic.
> >> +    if test -e pyvenv/bin/meson; then
> >> +        echo "Meson is too old:
> >
> >
> > Does a minimum version still get printed? I've needed to know that in
> the past when I got the error...
> >
> > Warner
> >  $(pyvenv/bin/meson --version)"
>
> At the end of the series, here's what happens if i change the meson
> requirement to a fictionally too-high version that would be impossible
> to fulfill:
>
>
> --------------------------------------------------------------------------------
> jsnow@scv ~/s/q/b/git (python-configure-venv)> ../../configure
> --enable-debug --disable-pypi;
> python determined to be '/usr/bin/python3'
> python version: Python 3.11.2
> MKVENV pyvenv
> MKVENV ensure meson>=2.61.5
> ERROR: Could not find a version that satisfies the requirement
> meson>=2.61.5 (from versions: none)
> ERROR: No matching distribution found for meson>=2.61.5
>
> *** Ouch! ***
>

Oh, that's enough :) :) thanks! I'd be happy with less...

Warner


> Could not ensure availability of 'meson>=2.61.5':
>  • Python package 'meson' version '1.0.1' was found, but isn't suitable.
>  • No suitable version found in, or failed to install from
> '/home/jsnow/src/qemu/python/wheels'.
>  • mkvenv was configured to operate offline and did not check PyPI.
>
> --------------------------------------------------------------------------------
>
>

[-- Attachment #2: Type: text/html, Size: 8636 bytes --]

Re: [RFC PATCH v3 01/20] python: update pylint configuration

[Daniel P. Berrangé]

On Mon, Apr 24, 2023 at 04:02:29PM -0400, John Snow wrote:
> Pylint 2.17.x decided that SocketAddrT was a bad name for a Type Alias for some
> reason. Sure, fine, whatever.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/setup.cfg | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 09/20] tests/docker: add python3-venv dependency

[Daniel P. Berrangé]

On Mon, Apr 24, 2023 at 04:02:37PM -0400, John Snow wrote:
> Several debian-based tests need the python3-venv dependency as a
> consequence of Debian debundling the "ensurepip" module normally
> included with Python.
> 
> As mkvenv.py stands as of this commit, Debian requires EITHER:
> 
> (A) setuptools and pip, or
> (B) ensurepip
> 
> mkvenv is a few seconds faster if you have setuptools and pip, so
> developers should prefer the first requirement. For the purposes of CI,
> the time-save is a wash; it's only a matter of who is responsible for
> installing pip and when; the timing is about the same.
> 
> Arbitrarily, I chose adding ensurepip to the test configuration because
> it is normally part of the Python stdlib, and always having it allows us
> a more consistent cross-platform environment.

python3-venv also matches the package that we already list in
the lcitool auto generated dockerfiles, so that consistency
is good.

> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++-
>  tests/docker/dockerfiles/debian-hexagon-cross.docker  | 3 ++-
>  tests/docker/dockerfiles/debian-riscv64-cross.docker  | 3 ++-
>  tests/docker/dockerfiles/debian-tricore-cross.docker  | 3 ++-
>  4 files changed, 8 insertions(+), 4 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 10/20] tests/vm: Configure netbsd to use Python 3.10

[Daniel P. Berrangé]

On Mon, Apr 24, 2023 at 04:02:38PM -0400, John Snow wrote:
> NetBSD removes some packages from the Python stdlib, but only
> re-packages them for Python 3.10. Switch to using Python 3.10.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/vm/netbsd | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 11/20] tests/vm: add py310-expat to NetBSD

[Daniel P. Berrangé]

On Mon, Apr 24, 2023 at 04:02:39PM -0400, John Snow wrote:
> NetBSD cannot successfully run "ensurepip" without access to the pyexpat
> module, which NetBSD debundles. Like the Debian patch, it would be
> strictly faster long term to install pip/setuptools, and I recommend
> developers at their workstations take that approach instead.

This sounds a bit like a NetBSD packaging bug surely ?  If we've got
ensurepip installed and it doesn't work due to missing expat I would
have though NetBSD should add a dependancy.

None the less, we'll need a fix in QEMU until that's addressed.

> 
> For the purposes of a throwaway VM, there's not really a speed
> difference for who is responsible for installing pip; us (needs
> py310-pip) or Python (needs py310-expat).
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/vm/netbsd | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 11/20] tests/vm: add py310-expat to NetBSD

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]

On Tue, Apr 25, 2023, 12:45 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Mon, Apr 24, 2023 at 04:02:39PM -0400, John Snow wrote:
> > NetBSD cannot successfully run "ensurepip" without access to the pyexpat
> > module, which NetBSD debundles. Like the Debian patch, it would be
> > strictly faster long term to install pip/setuptools, and I recommend
> > developers at their workstations take that approach instead.
>
> This sounds a bit like a NetBSD packaging bug surely ?  If we've got
> ensurepip installed and it doesn't work due to missing expat I would
> have though NetBSD should add a dependancy.
>

Yes, I agree - though I think NetBSD doesn't unbundle ensurepip, so there
might be nothing to add a dependency against ... and I assume they split
out pyexpat for a reason.

I can report it, anyway. It should at least fail more gracefully than it
does.


> None the less, we'll need a fix in QEMU until that's addressed.
>
> >
> > For the purposes of a throwaway VM, there's not really a speed
> > difference for who is responsible for installing pip; us (needs
> > py310-pip) or Python (needs py310-expat).
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  tests/vm/netbsd | 1 +
> >  1 file changed, 1 insertion(+)
>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

[-- Attachment #2: Type: text/html, Size: 3025 bytes --]

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Daniel P. Berrangé]

On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote:
> GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
>            (All green, except Python self-tests, see below)
> 
> This patch series creates a mandatory python virtual environment
> ("venv") during configure time and uses it to ensure the availability of
> meson and sphinx.
> 
> See https://www.qemu.org/2023/03/24/python/ for details. The summary is
> that the goal of this series is to ensure that the `python` used to run
> meson is the same `python` used to run Sphinx, tests, and any build-time
> python scripting we have. As it stands, meson and sphinx (and their
> extensions) *may* run in a different python environment than the one
> configured and chosen by the user at configure/build time.
> 
> The effective change of this series is that QEMU will now
> unconditionally create a venv at configure-time and will ensure that
> meson (and sphinx, if docs are enabled) are available through that venv.
> 
> Some important points as a pre-emptive "FAQ":
> 
> - This venv is unconditionally created and lives at {build_dir}/pyvenv.
> 
> - The python interpreter used by this venv is always the one identified
>   by configure. (Which in turn is always the one specified by --python
>   or $PYTHON)
> 
> - *almost* all python scripts in qemu.git executed as part of the build
>   system, meson, sphinx, avocado tests, vm tests or CI are always
>   executed within this venv.
> 
>   (iotests are not yet integrated; I plan to tackle this separately as a
>   follow-up in order to have a more tightly focused scope on that
>   series.)
> 
> - It remains possible to build and test fully offline.
>   (In most cases, you just need meson and sphinx from your distro's repo.)
> 
> - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
>   possible as the highest preference.
> 
> - Vendored versions of e.g. 'meson' are always preferred to PyPI
>   versions for speed, repeatability and ensuring tarball builds work
>   as-is offline.
> 
>   (Sphinx will not be vendored, just like it already isn't.)
> 
> - Missing dependencies, when possible, are fetched and installed
>   on-demand automatically to make developer environments "just work".
> 
> - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
>   Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
> 
> - No new dependencies (...for most platforms. Debian and NetBSD get an
>   asterisk.)
> 
> - The meson git submodule is unused after this series and can be removed.
> 
> For reviewers, here's how the series is broken up:
> 
> Patch 1 is a testing pre-req. Note that even with this patch,
> 'check-python-minreqs' and 'check-python-tox' CI jobs will both still
> fail on origin/master because this series requires 3.7+, but
> origin/master is currently still 3.6+.
> 
> - python: update pylint configuration
> 
> Patches 2-8 add the mkvenv script. The first patch checks in the barest
> essentials, and each subsequent patch adds a workaround or feature one
> at a time.
> 
> - python: add mkvenv.py
> - mkvenv: add console script entry point generation
> - mkvenv: Add better error message for missing pyexapt module
> - mkvenv: generate console entry shims from inside the venv
> - mkvenv: work around broken pip installations on Debian 10
> - mkvenv: add nested venv workaround
> - mkvenv: add ensure subcommand
> 
> Patches 9-11 modify our testing configuration to add new dependencies as
> needed.
> 
> - tests/docker: add python3-venv dependency
> - tests/vm: Configure netbsd to use Python 3.10
> - tests/vm: add py310-expat to NetBSD
> 
> Patch 12 changes how we package release tarballs.
> 
> - scripts/make-release: download meson==0.61.5 .whl
> 
> Patches 13-16 wire mkvenv into configure and tests.
> 
> - configure: create a python venv unconditionally
> - configure: use 'mkvenv ensure meson' to bootstrap meson
> - configure: add --enable-pypi and --disable-pypi
> - tests: Use configure-provided pyvenv for tests
> 
> Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
> changes could be folded earlier in the series (like the diagnose()
> patch), but I'm keeping it separate for review for now.
> 
> - configure: move --enable-docs and --disable-docs back to configure
> - mkvenv: add diagnose() method for ensure() failures
> - configure: use --diagnose option with meson ensure
> - configure: bootstrap sphinx with mkvenv

I'm not sure this last bit is working.

I uninstalled meson and python3-sphinx from my F38 host and ran
configure --target-list=x86_64-softmmu and got this:

$ ./configure --target-list=x86_64-softmmu
Using './build' as the directory for build output
python determined to be '/usr/bin/python3'
python version: Python 3.11.3
MKVENV pyvenv
Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B'
MKVENV ensure meson>=0.61.5
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory.
ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none)
ERROR: No matching distribution found for meson>=0.61.5
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
MKVENV ensure sphinx>=1.6.0
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none)
ERROR: No matching distribution found for sphinx>=1.6.0

*** Ouch! ***

Could not ensure availability of 'sphinx>=1.6.0':
 • Python package 'sphinx' was not found nor installed.
 • No local package directory was searched.
 • mkvenv was configured to operate offline and did not check PyPI. 


Sphinx not found/usable, disabling docs.
MKVENV ok!



It says mkvenv was configured to run offline, but I didn't do
anything. I thought the intention was for developers it would
live download from PyPI ?


On a system where i already have meson/sphinx installed, it
all just worked fine AFAICT.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 7999 bytes --]

On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote:
> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
> >            (All green, except Python self-tests, see below)
> >
> > This patch series creates a mandatory python virtual environment
> > ("venv") during configure time and uses it to ensure the availability of
> > meson and sphinx.
> >
> > See https://www.qemu.org/2023/03/24/python/ for details. The summary is
> > that the goal of this series is to ensure that the `python` used to run
> > meson is the same `python` used to run Sphinx, tests, and any build-time
> > python scripting we have. As it stands, meson and sphinx (and their
> > extensions) *may* run in a different python environment than the one
> > configured and chosen by the user at configure/build time.
> >
> > The effective change of this series is that QEMU will now
> > unconditionally create a venv at configure-time and will ensure that
> > meson (and sphinx, if docs are enabled) are available through that venv.
> >
> > Some important points as a pre-emptive "FAQ":
> >
> > - This venv is unconditionally created and lives at {build_dir}/pyvenv.
> >
> > - The python interpreter used by this venv is always the one identified
> >   by configure. (Which in turn is always the one specified by --python
> >   or $PYTHON)
> >
> > - *almost* all python scripts in qemu.git executed as part of the build
> >   system, meson, sphinx, avocado tests, vm tests or CI are always
> >   executed within this venv.
> >
> >   (iotests are not yet integrated; I plan to tackle this separately as a
> >   follow-up in order to have a more tightly focused scope on that
> >   series.)
> >
> > - It remains possible to build and test fully offline.
> >   (In most cases, you just need meson and sphinx from your distro's
> repo.)
> >
> > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
> >   possible as the highest preference.
> >
> > - Vendored versions of e.g. 'meson' are always preferred to PyPI
> >   versions for speed, repeatability and ensuring tarball builds work
> >   as-is offline.
> >
> >   (Sphinx will not be vendored, just like it already isn't.)
> >
> > - Missing dependencies, when possible, are fetched and installed
> >   on-demand automatically to make developer environments "just work".
> >
> > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
> >   Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
> >
> > - No new dependencies (...for most platforms. Debian and NetBSD get an
> >   asterisk.)
> >
> > - The meson git submodule is unused after this series and can be removed.
> >
> > For reviewers, here's how the series is broken up:
> >
> > Patch 1 is a testing pre-req. Note that even with this patch,
> > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still
> > fail on origin/master because this series requires 3.7+, but
> > origin/master is currently still 3.6+.
> >
> > - python: update pylint configuration
> >
> > Patches 2-8 add the mkvenv script. The first patch checks in the barest
> > essentials, and each subsequent patch adds a workaround or feature one
> > at a time.
> >
> > - python: add mkvenv.py
> > - mkvenv: add console script entry point generation
> > - mkvenv: Add better error message for missing pyexapt module
> > - mkvenv: generate console entry shims from inside the venv
> > - mkvenv: work around broken pip installations on Debian 10
> > - mkvenv: add nested venv workaround
> > - mkvenv: add ensure subcommand
> >
> > Patches 9-11 modify our testing configuration to add new dependencies as
> > needed.
> >
> > - tests/docker: add python3-venv dependency
> > - tests/vm: Configure netbsd to use Python 3.10
> > - tests/vm: add py310-expat to NetBSD
> >
> > Patch 12 changes how we package release tarballs.
> >
> > - scripts/make-release: download meson==0.61.5 .whl
> >
> > Patches 13-16 wire mkvenv into configure and tests.
> >
> > - configure: create a python venv unconditionally
> > - configure: use 'mkvenv ensure meson' to bootstrap meson
> > - configure: add --enable-pypi and --disable-pypi
> > - tests: Use configure-provided pyvenv for tests
> >
> > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
> > changes could be folded earlier in the series (like the diagnose()
> > patch), but I'm keeping it separate for review for now.
> >
> > - configure: move --enable-docs and --disable-docs back to configure
> > - mkvenv: add diagnose() method for ensure() failures
> > - configure: use --diagnose option with meson ensure
> > - configure: bootstrap sphinx with mkvenv
>
> I'm not sure this last bit is working.
>
> I uninstalled meson and python3-sphinx from my F38 host and ran
> configure --target-list=x86_64-softmmu and got this:
>
> $ ./configure --target-list=x86_64-softmmu
> Using './build' as the directory for build output
> python determined to be '/usr/bin/python3'
> python version: Python 3.11.3
> MKVENV pyvenv
> Configured python as
> '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B'
> MKVENV ensure meson>=0.61.5
> WARNING: Skipping
> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> invalid metadata entry 'name'
> WARNING: Skipping
> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> invalid metadata entry 'name'
> WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is
> ignored: it is neither a file nor a directory.
> ERROR: Could not find a version that satisfies the requirement
> meson>=0.61.5 (from versions: none)
> ERROR: No matching distribution found for meson>=0.61.5
> WARNING: Skipping
> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> invalid metadata entry 'name'
> WARNING: Skipping
> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> invalid metadata entry 'name'
> WARNING: Skipping
> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> invalid metadata entry 'name'
> MKVENV ensure sphinx>=1.6.0
> WARNING: Skipping
> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> invalid metadata entry 'name'
> WARNING: Skipping
> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> invalid metadata entry 'name'
> ERROR: Could not find a version that satisfies the requirement
> sphinx>=1.6.0 (from versions: none)
> ERROR: No matching distribution found for sphinx>=1.6.0
>
> *** Ouch! ***
>
> Could not ensure availability of 'sphinx>=1.6.0':
>  • Python package 'sphinx' was not found nor installed.
>  • No local package directory was searched.
>  • mkvenv was configured to operate offline and did not check PyPI.
>
>
> Sphinx not found/usable, disabling docs.
> MKVENV ok!
>
>
>
> It says mkvenv was configured to run offline, but I didn't do
> anything. I thought the intention was for developers it would
> live download from PyPI ?
>

Ah. So... with enable pypi being the default and with docs set to "auto", I
actually fall back to not installing sphinx from pypi *by default*. It
tries to locate it on your system and will enable docs if it can, but it
doesn't try too hard and doesn't get upset if it fails.

Try passing --enable-docs to convince the build system you'd really
definitely like docs, and it'll force the pypi access.


>
> On a system where i already have meson/sphinx installed, it
> all just worked fine AFAICT.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

[-- Attachment #2: Type: text/html, Size: 10215 bytes --]

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 9224 bytes --]

On Tue, Apr 25, 2023, 1:22 PM John Snow <jsnow@redhat.com> wrote:

>
>
> On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
>
>> On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote:
>> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
>> >            (All green, except Python self-tests, see below)
>> >
>> > This patch series creates a mandatory python virtual environment
>> > ("venv") during configure time and uses it to ensure the availability of
>> > meson and sphinx.
>> >
>> > See https://www.qemu.org/2023/03/24/python/ for details. The summary is
>> > that the goal of this series is to ensure that the `python` used to run
>> > meson is the same `python` used to run Sphinx, tests, and any build-time
>> > python scripting we have. As it stands, meson and sphinx (and their
>> > extensions) *may* run in a different python environment than the one
>> > configured and chosen by the user at configure/build time.
>> >
>> > The effective change of this series is that QEMU will now
>> > unconditionally create a venv at configure-time and will ensure that
>> > meson (and sphinx, if docs are enabled) are available through that venv.
>> >
>> > Some important points as a pre-emptive "FAQ":
>> >
>> > - This venv is unconditionally created and lives at {build_dir}/pyvenv.
>> >
>> > - The python interpreter used by this venv is always the one identified
>> >   by configure. (Which in turn is always the one specified by --python
>> >   or $PYTHON)
>> >
>> > - *almost* all python scripts in qemu.git executed as part of the build
>> >   system, meson, sphinx, avocado tests, vm tests or CI are always
>> >   executed within this venv.
>> >
>> >   (iotests are not yet integrated; I plan to tackle this separately as a
>> >   follow-up in order to have a more tightly focused scope on that
>> >   series.)
>> >
>> > - It remains possible to build and test fully offline.
>> >   (In most cases, you just need meson and sphinx from your distro's
>> repo.)
>> >
>> > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
>> >   possible as the highest preference.
>> >
>> > - Vendored versions of e.g. 'meson' are always preferred to PyPI
>> >   versions for speed, repeatability and ensuring tarball builds work
>> >   as-is offline.
>> >
>> >   (Sphinx will not be vendored, just like it already isn't.)
>> >
>> > - Missing dependencies, when possible, are fetched and installed
>> >   on-demand automatically to make developer environments "just work".
>> >
>> > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
>> >   Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
>> >
>> > - No new dependencies (...for most platforms. Debian and NetBSD get an
>> >   asterisk.)
>> >
>> > - The meson git submodule is unused after this series and can be
>> removed.
>> >
>> > For reviewers, here's how the series is broken up:
>> >
>> > Patch 1 is a testing pre-req. Note that even with this patch,
>> > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still
>> > fail on origin/master because this series requires 3.7+, but
>> > origin/master is currently still 3.6+.
>> >
>> > - python: update pylint configuration
>> >
>> > Patches 2-8 add the mkvenv script. The first patch checks in the barest
>> > essentials, and each subsequent patch adds a workaround or feature one
>> > at a time.
>> >
>> > - python: add mkvenv.py
>> > - mkvenv: add console script entry point generation
>> > - mkvenv: Add better error message for missing pyexapt module
>> > - mkvenv: generate console entry shims from inside the venv
>> > - mkvenv: work around broken pip installations on Debian 10
>> > - mkvenv: add nested venv workaround
>> > - mkvenv: add ensure subcommand
>> >
>> > Patches 9-11 modify our testing configuration to add new dependencies as
>> > needed.
>> >
>> > - tests/docker: add python3-venv dependency
>> > - tests/vm: Configure netbsd to use Python 3.10
>> > - tests/vm: add py310-expat to NetBSD
>> >
>> > Patch 12 changes how we package release tarballs.
>> >
>> > - scripts/make-release: download meson==0.61.5 .whl
>> >
>> > Patches 13-16 wire mkvenv into configure and tests.
>> >
>> > - configure: create a python venv unconditionally
>> > - configure: use 'mkvenv ensure meson' to bootstrap meson
>> > - configure: add --enable-pypi and --disable-pypi
>> > - tests: Use configure-provided pyvenv for tests
>> >
>> > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
>> > changes could be folded earlier in the series (like the diagnose()
>> > patch), but I'm keeping it separate for review for now.
>> >
>> > - configure: move --enable-docs and --disable-docs back to configure
>> > - mkvenv: add diagnose() method for ensure() failures
>> > - configure: use --diagnose option with meson ensure
>> > - configure: bootstrap sphinx with mkvenv
>>
>> I'm not sure this last bit is working.
>>
>> I uninstalled meson and python3-sphinx from my F38 host and ran
>> configure --target-list=x86_64-softmmu and got this:
>>
>> $ ./configure --target-list=x86_64-softmmu
>> Using './build' as the directory for build output
>> python determined to be '/usr/bin/python3'
>> python version: Python 3.11.3
>> MKVENV pyvenv
>> Configured python as
>> '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B'
>> MKVENV ensure meson>=0.61.5
>> WARNING: Skipping
>> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
>> invalid metadata entry 'name'
>> WARNING: Skipping
>> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
>> invalid metadata entry 'name'
>> WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is
>> ignored: it is neither a file nor a directory.
>> ERROR: Could not find a version that satisfies the requirement
>> meson>=0.61.5 (from versions: none)
>> ERROR: No matching distribution found for meson>=0.61.5
>> WARNING: Skipping
>> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
>> invalid metadata entry 'name'
>> WARNING: Skipping
>> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
>> invalid metadata entry 'name'
>> WARNING: Skipping
>> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
>> invalid metadata entry 'name'
>> MKVENV ensure sphinx>=1.6.0
>> WARNING: Skipping
>> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
>> invalid metadata entry 'name'
>> WARNING: Skipping
>> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
>> invalid metadata entry 'name'
>> ERROR: Could not find a version that satisfies the requirement
>> sphinx>=1.6.0 (from versions: none)
>> ERROR: No matching distribution found for sphinx>=1.6.0
>>
>> *** Ouch! ***
>>
>> Could not ensure availability of 'sphinx>=1.6.0':
>>  • Python package 'sphinx' was not found nor installed.
>>  • No local package directory was searched.
>>  • mkvenv was configured to operate offline and did not check PyPI.
>>
>>
>> Sphinx not found/usable, disabling docs.
>> MKVENV ok!
>>
>>
>>
>> It says mkvenv was configured to run offline, but I didn't do
>> anything. I thought the intention was for developers it would
>> live download from PyPI ?
>>
>
> Ah. So... with enable pypi being the default and with docs set to "auto",
> I actually fall back to not installing sphinx from pypi *by default*. It
> tries to locate it on your system and will enable docs if it can, but it
> doesn't try too hard and doesn't get upset if it fails.
>

(Though all of those errors and warnings sure are noisy for meaning "we
couldn't find an optional package". mkvenv just doesn't distinguish between
optional and required right now so it just leaves it to the caller to
interpret. Any suggestions for improving this?)


> Try passing --enable-docs to convince the build system you'd really
> definitely like docs, and it'll force the pypi access.
>

I can make it try PyPI in this double-default case too, I was just being
very conservative about when we tried PyPI - this solution is fairly
reluctant by design to do it.

If we all agree that it shouldn't be so reluctant, and anyone who doesnt
want it at all should just pass --disable-pypi, I can make that change
easily.

I wonder if I should make an "auto" setting for the pypi access which
represents this behavior ("only if I have to"), and make "enable" more
aggressive (consult pypi even for optional features.)

I was just being conservative on the first pass. Trying to strike a balance
between convenience, speed, and least surprise.


>
>>
>> On a system where i already have meson/sphinx installed, it
>> all just worked fine AFAICT.
>>
>> With regards,
>> Daniel
>> --
>> |: https://berrange.com      -o-
>> https://www.flickr.com/photos/dberrange :|
>> |: https://libvirt.org         -o-
>> https://fstop138.berrange.com :|
>> |: https://entangle-photo.org    -o-
>> https://www.instagram.com/dberrange :|
>>
>>

[-- Attachment #2: Type: text/html, Size: 12306 bytes --]

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Daniel P. Berrangé]

On Tue, Apr 25, 2023 at 01:22:38PM -0400, John Snow wrote:
> On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote:
> > > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
> > >            (All green, except Python self-tests, see below)
> > >
> > > This patch series creates a mandatory python virtual environment
> > > ("venv") during configure time and uses it to ensure the availability of
> > > meson and sphinx.
> > >
> > > See https://www.qemu.org/2023/03/24/python/ for details. The summary is
> > > that the goal of this series is to ensure that the `python` used to run
> > > meson is the same `python` used to run Sphinx, tests, and any build-time
> > > python scripting we have. As it stands, meson and sphinx (and their
> > > extensions) *may* run in a different python environment than the one
> > > configured and chosen by the user at configure/build time.
> > >
> > > The effective change of this series is that QEMU will now
> > > unconditionally create a venv at configure-time and will ensure that
> > > meson (and sphinx, if docs are enabled) are available through that venv.
> > >
> > > Some important points as a pre-emptive "FAQ":
> > >
> > > - This venv is unconditionally created and lives at {build_dir}/pyvenv.
> > >
> > > - The python interpreter used by this venv is always the one identified
> > >   by configure. (Which in turn is always the one specified by --python
> > >   or $PYTHON)
> > >
> > > - *almost* all python scripts in qemu.git executed as part of the build
> > >   system, meson, sphinx, avocado tests, vm tests or CI are always
> > >   executed within this venv.
> > >
> > >   (iotests are not yet integrated; I plan to tackle this separately as a
> > >   follow-up in order to have a more tightly focused scope on that
> > >   series.)
> > >
> > > - It remains possible to build and test fully offline.
> > >   (In most cases, you just need meson and sphinx from your distro's
> > repo.)
> > >
> > > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
> > >   possible as the highest preference.
> > >
> > > - Vendored versions of e.g. 'meson' are always preferred to PyPI
> > >   versions for speed, repeatability and ensuring tarball builds work
> > >   as-is offline.
> > >
> > >   (Sphinx will not be vendored, just like it already isn't.)
> > >
> > > - Missing dependencies, when possible, are fetched and installed
> > >   on-demand automatically to make developer environments "just work".
> > >
> > > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
> > >   Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
> > >
> > > - No new dependencies (...for most platforms. Debian and NetBSD get an
> > >   asterisk.)
> > >
> > > - The meson git submodule is unused after this series and can be removed.
> > >
> > > For reviewers, here's how the series is broken up:
> > >
> > > Patch 1 is a testing pre-req. Note that even with this patch,
> > > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still
> > > fail on origin/master because this series requires 3.7+, but
> > > origin/master is currently still 3.6+.
> > >
> > > - python: update pylint configuration
> > >
> > > Patches 2-8 add the mkvenv script. The first patch checks in the barest
> > > essentials, and each subsequent patch adds a workaround or feature one
> > > at a time.
> > >
> > > - python: add mkvenv.py
> > > - mkvenv: add console script entry point generation
> > > - mkvenv: Add better error message for missing pyexapt module
> > > - mkvenv: generate console entry shims from inside the venv
> > > - mkvenv: work around broken pip installations on Debian 10
> > > - mkvenv: add nested venv workaround
> > > - mkvenv: add ensure subcommand
> > >
> > > Patches 9-11 modify our testing configuration to add new dependencies as
> > > needed.
> > >
> > > - tests/docker: add python3-venv dependency
> > > - tests/vm: Configure netbsd to use Python 3.10
> > > - tests/vm: add py310-expat to NetBSD
> > >
> > > Patch 12 changes how we package release tarballs.
> > >
> > > - scripts/make-release: download meson==0.61.5 .whl
> > >
> > > Patches 13-16 wire mkvenv into configure and tests.
> > >
> > > - configure: create a python venv unconditionally
> > > - configure: use 'mkvenv ensure meson' to bootstrap meson
> > > - configure: add --enable-pypi and --disable-pypi
> > > - tests: Use configure-provided pyvenv for tests
> > >
> > > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
> > > changes could be folded earlier in the series (like the diagnose()
> > > patch), but I'm keeping it separate for review for now.
> > >
> > > - configure: move --enable-docs and --disable-docs back to configure
> > > - mkvenv: add diagnose() method for ensure() failures
> > > - configure: use --diagnose option with meson ensure
> > > - configure: bootstrap sphinx with mkvenv
> >
> > I'm not sure this last bit is working.
> >
> > I uninstalled meson and python3-sphinx from my F38 host and ran
> > configure --target-list=x86_64-softmmu and got this:
> >
> > $ ./configure --target-list=x86_64-softmmu
> > Using './build' as the directory for build output
> > python determined to be '/usr/bin/python3'
> > python version: Python 3.11.3
> > MKVENV pyvenv
> > Configured python as
> > '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B'
> > MKVENV ensure meson>=0.61.5
> > WARNING: Skipping
> > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > invalid metadata entry 'name'
> > WARNING: Skipping
> > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > invalid metadata entry 'name'
> > WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is
> > ignored: it is neither a file nor a directory.
> > ERROR: Could not find a version that satisfies the requirement
> > meson>=0.61.5 (from versions: none)
> > ERROR: No matching distribution found for meson>=0.61.5
> > WARNING: Skipping
> > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > invalid metadata entry 'name'
> > WARNING: Skipping
> > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > invalid metadata entry 'name'
> > WARNING: Skipping
> > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > invalid metadata entry 'name'
> > MKVENV ensure sphinx>=1.6.0
> > WARNING: Skipping
> > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > invalid metadata entry 'name'
> > WARNING: Skipping
> > /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > invalid metadata entry 'name'
> > ERROR: Could not find a version that satisfies the requirement
> > sphinx>=1.6.0 (from versions: none)
> > ERROR: No matching distribution found for sphinx>=1.6.0
> >
> > *** Ouch! ***
> >
> > Could not ensure availability of 'sphinx>=1.6.0':
> >  • Python package 'sphinx' was not found nor installed.
> >  • No local package directory was searched.
> >  • mkvenv was configured to operate offline and did not check PyPI.
> >
> >
> > Sphinx not found/usable, disabling docs.
> > MKVENV ok!
> >
> >
> >
> > It says mkvenv was configured to run offline, but I didn't do
> > anything. I thought the intention was for developers it would
> > live download from PyPI ?
> >
> 
> Ah. So... with enable pypi being the default and with docs set to "auto", I
> actually fall back to not installing sphinx from pypi *by default*. It
> tries to locate it on your system and will enable docs if it can, but it
> doesn't try too hard and doesn't get upset if it fails.

Ah right, yes, I think that behaviour makes sense.

> Try passing --enable-docs to convince the build system you'd really
> definitely like docs, and it'll force the pypi access.

Ok will try.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Daniel P. Berrangé]

On Tue, Apr 25, 2023 at 01:34:52PM -0400, John Snow wrote:
> On Tue, Apr 25, 2023, 1:22 PM John Snow <jsnow@redhat.com> wrote:
> 
> >
> >
> > On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com>
> > wrote:
> >
> >> On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote:
> >> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
> >> >            (All green, except Python self-tests, see below)
> >> >
> >> > This patch series creates a mandatory python virtual environment
> >> > ("venv") during configure time and uses it to ensure the availability of
> >> > meson and sphinx.
> >> >
> >> > See https://www.qemu.org/2023/03/24/python/ for details. The summary is
> >> > that the goal of this series is to ensure that the `python` used to run
> >> > meson is the same `python` used to run Sphinx, tests, and any build-time
> >> > python scripting we have. As it stands, meson and sphinx (and their
> >> > extensions) *may* run in a different python environment than the one
> >> > configured and chosen by the user at configure/build time.
> >> >
> >> > The effective change of this series is that QEMU will now
> >> > unconditionally create a venv at configure-time and will ensure that
> >> > meson (and sphinx, if docs are enabled) are available through that venv.
> >> >
> >> > Some important points as a pre-emptive "FAQ":
> >> >
> >> > - This venv is unconditionally created and lives at {build_dir}/pyvenv.
> >> >
> >> > - The python interpreter used by this venv is always the one identified
> >> >   by configure. (Which in turn is always the one specified by --python
> >> >   or $PYTHON)
> >> >
> >> > - *almost* all python scripts in qemu.git executed as part of the build
> >> >   system, meson, sphinx, avocado tests, vm tests or CI are always
> >> >   executed within this venv.
> >> >
> >> >   (iotests are not yet integrated; I plan to tackle this separately as a
> >> >   follow-up in order to have a more tightly focused scope on that
> >> >   series.)
> >> >
> >> > - It remains possible to build and test fully offline.
> >> >   (In most cases, you just need meson and sphinx from your distro's
> >> repo.)
> >> >
> >> > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
> >> >   possible as the highest preference.
> >> >
> >> > - Vendored versions of e.g. 'meson' are always preferred to PyPI
> >> >   versions for speed, repeatability and ensuring tarball builds work
> >> >   as-is offline.
> >> >
> >> >   (Sphinx will not be vendored, just like it already isn't.)
> >> >
> >> > - Missing dependencies, when possible, are fetched and installed
> >> >   on-demand automatically to make developer environments "just work".
> >> >
> >> > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
> >> >   Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
> >> >
> >> > - No new dependencies (...for most platforms. Debian and NetBSD get an
> >> >   asterisk.)
> >> >
> >> > - The meson git submodule is unused after this series and can be
> >> removed.
> >> >
> >> > For reviewers, here's how the series is broken up:
> >> >
> >> > Patch 1 is a testing pre-req. Note that even with this patch,
> >> > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still
> >> > fail on origin/master because this series requires 3.7+, but
> >> > origin/master is currently still 3.6+.
> >> >
> >> > - python: update pylint configuration
> >> >
> >> > Patches 2-8 add the mkvenv script. The first patch checks in the barest
> >> > essentials, and each subsequent patch adds a workaround or feature one
> >> > at a time.
> >> >
> >> > - python: add mkvenv.py
> >> > - mkvenv: add console script entry point generation
> >> > - mkvenv: Add better error message for missing pyexapt module
> >> > - mkvenv: generate console entry shims from inside the venv
> >> > - mkvenv: work around broken pip installations on Debian 10
> >> > - mkvenv: add nested venv workaround
> >> > - mkvenv: add ensure subcommand
> >> >
> >> > Patches 9-11 modify our testing configuration to add new dependencies as
> >> > needed.
> >> >
> >> > - tests/docker: add python3-venv dependency
> >> > - tests/vm: Configure netbsd to use Python 3.10
> >> > - tests/vm: add py310-expat to NetBSD
> >> >
> >> > Patch 12 changes how we package release tarballs.
> >> >
> >> > - scripts/make-release: download meson==0.61.5 .whl
> >> >
> >> > Patches 13-16 wire mkvenv into configure and tests.
> >> >
> >> > - configure: create a python venv unconditionally
> >> > - configure: use 'mkvenv ensure meson' to bootstrap meson
> >> > - configure: add --enable-pypi and --disable-pypi
> >> > - tests: Use configure-provided pyvenv for tests
> >> >
> >> > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
> >> > changes could be folded earlier in the series (like the diagnose()
> >> > patch), but I'm keeping it separate for review for now.
> >> >
> >> > - configure: move --enable-docs and --disable-docs back to configure
> >> > - mkvenv: add diagnose() method for ensure() failures
> >> > - configure: use --diagnose option with meson ensure
> >> > - configure: bootstrap sphinx with mkvenv
> >>
> >> I'm not sure this last bit is working.
> >>
> >> I uninstalled meson and python3-sphinx from my F38 host and ran
> >> configure --target-list=x86_64-softmmu and got this:
> >>
> >> $ ./configure --target-list=x86_64-softmmu
> >> Using './build' as the directory for build output
> >> python determined to be '/usr/bin/python3'
> >> python version: Python 3.11.3
> >> MKVENV pyvenv
> >> Configured python as
> >> '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B'
> >> MKVENV ensure meson>=0.61.5
> >> WARNING: Skipping
> >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> >> invalid metadata entry 'name'
> >> WARNING: Skipping
> >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> >> invalid metadata entry 'name'
> >> WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is
> >> ignored: it is neither a file nor a directory.
> >> ERROR: Could not find a version that satisfies the requirement
> >> meson>=0.61.5 (from versions: none)
> >> ERROR: No matching distribution found for meson>=0.61.5
> >> WARNING: Skipping
> >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> >> invalid metadata entry 'name'
> >> WARNING: Skipping
> >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> >> invalid metadata entry 'name'
> >> WARNING: Skipping
> >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> >> invalid metadata entry 'name'
> >> MKVENV ensure sphinx>=1.6.0
> >> WARNING: Skipping
> >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> >> invalid metadata entry 'name'
> >> WARNING: Skipping
> >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> >> invalid metadata entry 'name'
> >> ERROR: Could not find a version that satisfies the requirement
> >> sphinx>=1.6.0 (from versions: none)
> >> ERROR: No matching distribution found for sphinx>=1.6.0
> >>
> >> *** Ouch! ***
> >>
> >> Could not ensure availability of 'sphinx>=1.6.0':
> >>  • Python package 'sphinx' was not found nor installed.
> >>  • No local package directory was searched.
> >>  • mkvenv was configured to operate offline and did not check PyPI.
> >>
> >>
> >> Sphinx not found/usable, disabling docs.
> >> MKVENV ok!
> >>
> >>
> >>
> >> It says mkvenv was configured to run offline, but I didn't do
> >> anything. I thought the intention was for developers it would
> >> live download from PyPI ?
> >>
> >
> > Ah. So... with enable pypi being the default and with docs set to "auto",
> > I actually fall back to not installing sphinx from pypi *by default*. It
> > tries to locate it on your system and will enable docs if it can, but it
> > doesn't try too hard and doesn't get upset if it fails.
> >
> 
> (Though all of those errors and warnings sure are noisy for meaning "we
> couldn't find an optional package". mkvenv just doesn't distinguish between
> optional and required right now so it just leaves it to the caller to
> interpret. Any suggestions for improving this?)
> 
> 
> > Try passing --enable-docs to convince the build system you'd really
> > definitely like docs, and it'll force the pypi access.
> >
> 
> I can make it try PyPI in this double-default case too, I was just being
> very conservative about when we tried PyPI - this solution is fairly
> reluctant by design to do it.
> 
> If we all agree that it shouldn't be so reluctant, and anyone who doesnt
> want it at all should just pass --disable-pypi, I can make that change
> easily.
> 
> I wonder if I should make an "auto" setting for the pypi access which
> represents this behavior ("only if I have to"), and make "enable" more
> aggressive (consult pypi even for optional features.)
> 
> I was just being conservative on the first pass. Trying to strike a balance
> between convenience, speed, and least surprise.

How about having --enable-pypi never|auto|force with the following
semantics for --enable-docs + --enable-pypi


  * docs=no   - pypi never used

  * docs=auto + pypi=never  => docs only enable if sphinx is already
                               installed locally, otherwise disabled

  * docs=auto + pypi=auto  => docs enable if sphinx is already
                              installed locally, or can download from
			      pypi as fallback

  * docs=auto + pypi=force  => always download sphinx from pypi


  * docs=yes + pypi=never  => ERROR if sphinx is not already
                              installed locally

  * docs=yes + pypi=auto  => docs enable if sphinx is already
                             installed locally, or can download from
			     pypi as fallback

  * docs=yes + pypi=force  => always download sphinx from pypi



So eg distros could use pypi=never, devs would use pypi=auto mostly,
while CI might use pypi=force to test specific versions indepenant
of distros ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[John Snow]

On Tue, Apr 25, 2023 at 2:10 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Apr 25, 2023 at 01:34:52PM -0400, John Snow wrote:
> > On Tue, Apr 25, 2023, 1:22 PM John Snow <jsnow@redhat.com> wrote:
> >
> > >
> > >
> > > On Tue, Apr 25, 2023, 1:17 PM Daniel P. Berrangé <berrange@redhat.com>
> > > wrote:
> > >
> > >> On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote:
> > >> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
> > >> >            (All green, except Python self-tests, see below)
> > >> >
> > >> > This patch series creates a mandatory python virtual environment
> > >> > ("venv") during configure time and uses it to ensure the availability of
> > >> > meson and sphinx.
> > >> >
> > >> > See https://www.qemu.org/2023/03/24/python/ for details. The summary is
> > >> > that the goal of this series is to ensure that the `python` used to run
> > >> > meson is the same `python` used to run Sphinx, tests, and any build-time
> > >> > python scripting we have. As it stands, meson and sphinx (and their
> > >> > extensions) *may* run in a different python environment than the one
> > >> > configured and chosen by the user at configure/build time.
> > >> >
> > >> > The effective change of this series is that QEMU will now
> > >> > unconditionally create a venv at configure-time and will ensure that
> > >> > meson (and sphinx, if docs are enabled) are available through that venv.
> > >> >
> > >> > Some important points as a pre-emptive "FAQ":
> > >> >
> > >> > - This venv is unconditionally created and lives at {build_dir}/pyvenv.
> > >> >
> > >> > - The python interpreter used by this venv is always the one identified
> > >> >   by configure. (Which in turn is always the one specified by --python
> > >> >   or $PYTHON)
> > >> >
> > >> > - *almost* all python scripts in qemu.git executed as part of the build
> > >> >   system, meson, sphinx, avocado tests, vm tests or CI are always
> > >> >   executed within this venv.
> > >> >
> > >> >   (iotests are not yet integrated; I plan to tackle this separately as a
> > >> >   follow-up in order to have a more tightly focused scope on that
> > >> >   series.)
> > >> >
> > >> > - It remains possible to build and test fully offline.
> > >> >   (In most cases, you just need meson and sphinx from your distro's
> > >> repo.)
> > >> >
> > >> > - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
> > >> >   possible as the highest preference.
> > >> >
> > >> > - Vendored versions of e.g. 'meson' are always preferred to PyPI
> > >> >   versions for speed, repeatability and ensuring tarball builds work
> > >> >   as-is offline.
> > >> >
> > >> >   (Sphinx will not be vendored, just like it already isn't.)
> > >> >
> > >> > - Missing dependencies, when possible, are fetched and installed
> > >> >   on-demand automatically to make developer environments "just work".
> > >> >
> > >> > - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
> > >> >   Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
> > >> >
> > >> > - No new dependencies (...for most platforms. Debian and NetBSD get an
> > >> >   asterisk.)
> > >> >
> > >> > - The meson git submodule is unused after this series and can be
> > >> removed.
> > >> >
> > >> > For reviewers, here's how the series is broken up:
> > >> >
> > >> > Patch 1 is a testing pre-req. Note that even with this patch,
> > >> > 'check-python-minreqs' and 'check-python-tox' CI jobs will both still
> > >> > fail on origin/master because this series requires 3.7+, but
> > >> > origin/master is currently still 3.6+.
> > >> >
> > >> > - python: update pylint configuration
> > >> >
> > >> > Patches 2-8 add the mkvenv script. The first patch checks in the barest
> > >> > essentials, and each subsequent patch adds a workaround or feature one
> > >> > at a time.
> > >> >
> > >> > - python: add mkvenv.py
> > >> > - mkvenv: add console script entry point generation
> > >> > - mkvenv: Add better error message for missing pyexapt module
> > >> > - mkvenv: generate console entry shims from inside the venv
> > >> > - mkvenv: work around broken pip installations on Debian 10
> > >> > - mkvenv: add nested venv workaround
> > >> > - mkvenv: add ensure subcommand
> > >> >
> > >> > Patches 9-11 modify our testing configuration to add new dependencies as
> > >> > needed.
> > >> >
> > >> > - tests/docker: add python3-venv dependency
> > >> > - tests/vm: Configure netbsd to use Python 3.10
> > >> > - tests/vm: add py310-expat to NetBSD
> > >> >
> > >> > Patch 12 changes how we package release tarballs.
> > >> >
> > >> > - scripts/make-release: download meson==0.61.5 .whl
> > >> >
> > >> > Patches 13-16 wire mkvenv into configure and tests.
> > >> >
> > >> > - configure: create a python venv unconditionally
> > >> > - configure: use 'mkvenv ensure meson' to bootstrap meson
> > >> > - configure: add --enable-pypi and --disable-pypi
> > >> > - tests: Use configure-provided pyvenv for tests
> > >> >
> > >> > Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
> > >> > changes could be folded earlier in the series (like the diagnose()
> > >> > patch), but I'm keeping it separate for review for now.
> > >> >
> > >> > - configure: move --enable-docs and --disable-docs back to configure
> > >> > - mkvenv: add diagnose() method for ensure() failures
> > >> > - configure: use --diagnose option with meson ensure
> > >> > - configure: bootstrap sphinx with mkvenv
> > >>
> > >> I'm not sure this last bit is working.
> > >>
> > >> I uninstalled meson and python3-sphinx from my F38 host and ran
> > >> configure --target-list=x86_64-softmmu and got this:
> > >>
> > >> $ ./configure --target-list=x86_64-softmmu
> > >> Using './build' as the directory for build output
> > >> python determined to be '/usr/bin/python3'
> > >> python version: Python 3.11.3
> > >> MKVENV pyvenv
> > >> Configured python as
> > >> '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B'
> > >> MKVENV ensure meson>=0.61.5
> > >> WARNING: Skipping
> > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > >> invalid metadata entry 'name'
> > >> WARNING: Skipping
> > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > >> invalid metadata entry 'name'
> > >> WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is
> > >> ignored: it is neither a file nor a directory.
> > >> ERROR: Could not find a version that satisfies the requirement
> > >> meson>=0.61.5 (from versions: none)
> > >> ERROR: No matching distribution found for meson>=0.61.5
> > >> WARNING: Skipping
> > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > >> invalid metadata entry 'name'
> > >> WARNING: Skipping
> > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > >> invalid metadata entry 'name'
> > >> WARNING: Skipping
> > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > >> invalid metadata entry 'name'
> > >> MKVENV ensure sphinx>=1.6.0
> > >> WARNING: Skipping
> > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > >> invalid metadata entry 'name'
> > >> WARNING: Skipping
> > >> /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to
> > >> invalid metadata entry 'name'
> > >> ERROR: Could not find a version that satisfies the requirement
> > >> sphinx>=1.6.0 (from versions: none)
> > >> ERROR: No matching distribution found for sphinx>=1.6.0
> > >>
> > >> *** Ouch! ***
> > >>
> > >> Could not ensure availability of 'sphinx>=1.6.0':
> > >>  • Python package 'sphinx' was not found nor installed.
> > >>  • No local package directory was searched.
> > >>  • mkvenv was configured to operate offline and did not check PyPI.
> > >>
> > >>
> > >> Sphinx not found/usable, disabling docs.
> > >> MKVENV ok!
> > >>
> > >>
> > >>
> > >> It says mkvenv was configured to run offline, but I didn't do
> > >> anything. I thought the intention was for developers it would
> > >> live download from PyPI ?
> > >>
> > >
> > > Ah. So... with enable pypi being the default and with docs set to "auto",
> > > I actually fall back to not installing sphinx from pypi *by default*. It
> > > tries to locate it on your system and will enable docs if it can, but it
> > > doesn't try too hard and doesn't get upset if it fails.
> > >
> >
> > (Though all of those errors and warnings sure are noisy for meaning "we
> > couldn't find an optional package". mkvenv just doesn't distinguish between
> > optional and required right now so it just leaves it to the caller to
> > interpret. Any suggestions for improving this?)
> >
> >
> > > Try passing --enable-docs to convince the build system you'd really
> > > definitely like docs, and it'll force the pypi access.
> > >
> >
> > I can make it try PyPI in this double-default case too, I was just being
> > very conservative about when we tried PyPI - this solution is fairly
> > reluctant by design to do it.
> >
> > If we all agree that it shouldn't be so reluctant, and anyone who doesnt
> > want it at all should just pass --disable-pypi, I can make that change
> > easily.
> >
> > I wonder if I should make an "auto" setting for the pypi access which
> > represents this behavior ("only if I have to"), and make "enable" more
> > aggressive (consult pypi even for optional features.)
> >
> > I was just being conservative on the first pass. Trying to strike a balance
> > between convenience, speed, and least surprise.
>
> How about having --enable-pypi never|auto|force with the following
> semantics for --enable-docs + --enable-pypi
>
>
>   * docs=no   - pypi never used
>
>   * docs=auto + pypi=never  => docs only enable if sphinx is already
>                                installed locally, otherwise disabled
>
>   * docs=auto + pypi=auto  => docs enable if sphinx is already
>                               installed locally, or can download from
>                               pypi as fallback
>

As long as this doesn't cause needless trouble for existing configure
invocations baked into scripts and the like. It's a bit more
aggressive about enabling docs than we have been in the past, but
maybe that's not a bad thing.

>   * docs=auto + pypi=force  => always download sphinx from pypi
>

So if you already have Sphinx, this should perform an upgrade to the
latest version?

>
>   * docs=yes + pypi=never  => ERROR if sphinx is not already
>                               installed locally
>
>   * docs=yes + pypi=auto  => docs enable if sphinx is already
>                              installed locally, or can download from
>                              pypi as fallback
>
>   * docs=yes + pypi=force  => always download sphinx from pypi

Yeah, there's room for settings like these, and the above mostly makes
sense to me.

Paolo and I had written up a more elaborate set of flags at one point
under the premise of integrating testing, too -- for instance,
pre-loading the testing dependencies for iotests (local qemu package,
qemu.qmp, etc) or for avocado tests (avocado-framework and other deps
needed for those tests). I wanted to try and check in a "minimum
viable" version of this patch set first close to the 8.1 tree opening
to iron out any bugs in the core feature and then I'd work to expand
the flags and capability of the system as differential patches.

So, I think changing --enable-pypi too much is beyond the scope of the
current patchset, but it's on my mind for what will follow almost
immediately after it. With that in mind, we can brainstorm a little
here and throw some darts at the wall:

The version of flags we originally came up with was this:

--python=... # runtime used to create venv
--enable-pip-groups=testing,devel,avocado,meson,sphinx
--enable-pip=now  # install all python deps now
--enable-pip=on-demand  # install qemu.git/meson/sphinx, delay the rest
--enable-pip=no    # offline
--{enable,disable}-isolated-venv # let venv use system/distro if disable

I may have pulled us in a slightly different direction with the
version of the patches I actually sent here today, but the key ideas
here were:

- The ability to opt into or out of different package groups at
configure-time. meson is always required, docs/sphinx is optional, and
the other three groups are testing related. (testing: iotests and
qemu.qmp, devel: python self-tests, avocado: avocado tests.)
- The ability to decide when packages get installed to the venv;
either at configure time ("now") or as-needed ("on-demand") or
functionally never ("no")
- The ability to enable or disable venv isolation (turning on or off
system packages).

I think I abandoned the idea of configuring the venv isolation and
have it set to "always off". We may revisit this later, but I think
for this series it's fine to have glossed over it.
I also skipped over the idea of having package installation being
"now", "on-demand" or "no" -- this patch-set more or less is using a
hybrid of "now" and "on-demand" where meson and sphinx are "now" but
avocado is "on-demand". Unifying that discrepancy can occur in
subsequent patches as it closely resembles what we had in 8.0 and
earlier.
I also didn't implement configurable package groups at all yet. meson
and sphinx are just implicitly selected; everything else is not yet
thoroughly integrated.

The testing portions of this -- avocado, qemu.qmp, and python
self-tests -- are not as integrated as the build portions, sphinx and
meson. That will likely be the focus of my next patchset when I move
to integrate iotests and actually set about the work of finally
dropping the qemu.qmp package from the qemu.git source tree.

I guess before I spend *too* much time writing that series (for the
fourth time...!) I wanted to try and solidify the basics here to give
me a nice solid foundation.

--js

>
>
>
> So eg distros could use pypi=never, devs would use pypi=auto mostly,
> while CI might use pypi=force to test specific versions indepenant
> of distros ?
>

Sounds reasonable in general, I think, but I have some questions about
what 'force' does, exactly.

> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Paolo Bonzini]

On 4/24/23 22:02, John Snow wrote:
> Some important points as a pre-emptive "FAQ":
> 
> - This venv is unconditionally created and lives at {build_dir}/pyvenv.
> 
> - The python interpreter used by this venv is always the one identified
>    by configure. (Which in turn is always the one specified by --python
>    or $PYTHON)
> 
> -*almost*  all python scripts in qemu.git executed as part of the build
>    system, meson, sphinx, avocado tests, vm tests or CI are always
>    executed within this venv.
> 
>    (iotests are not yet integrated; I plan to tackle this separately as a
>    follow-up in order to have a more tightly focused scope on that
>    series.)
> 
> - It remains possible to build and test fully offline.
>    (In most cases, you just need meson and sphinx from your distro's repo.)
> 
> - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
>    possible as the highest preference.
> 
> - Vendored versions of e.g. 'meson' are always preferred to PyPI
>    versions for speed, repeatability and ensuring tarball builds work
>    as-is offline.
> 
>    (Sphinx will not be vendored, just like it already isn't.)
> 
> - Missing dependencies, when possible, are fetched and installed
>    on-demand automatically to make developer environments "just work".
> 
> - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
>    Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
> 
> - No new dependencies (...for most platforms. Debian and NetBSD get an
>    asterisk.)
> 
> - The meson git submodule is unused after this series and can be removed.

Thanks, this looks pretty good.  Some changes I'd make for the non-RFC 
version:

- I think we should just check in the meson wheel (which also removes 
the need for patch 12, so it can be done in its stead) and remove the 
submodule

- The verbosity of mkvenv.py can be tuned down and most prints replaced 
with logger.info() or logger.debug()

- While I agree with keeping patch 18 separate, I would move it earlier 
so that patch 19 can be squashed into patch 14

- I am ambivalent about keeping --enable/--disable-pypi in the first 
committed patchset, but in any case I would move patches 16 and 20 
before patch 15

Paolo

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Daniel P. Berrangé]

On Tue, Apr 25, 2023 at 02:58:53PM -0400, John Snow wrote:
> On Tue, Apr 25, 2023 at 2:10 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > How about having --enable-pypi never|auto|force with the following
> > semantics for --enable-docs + --enable-pypi
> >
> >
> >   * docs=no   - pypi never used
> >
> >   * docs=auto + pypi=never  => docs only enable if sphinx is already
> >                                installed locally, otherwise disabled
> >
> >   * docs=auto + pypi=auto  => docs enable if sphinx is already
> >                               installed locally, or can download from
> >                               pypi as fallback
> >
> 
> As long as this doesn't cause needless trouble for existing configure
> invocations baked into scripts and the like. It's a bit more
> aggressive about enabling docs than we have been in the past, but
> maybe that's not a bad thing.
> 
> >   * docs=auto + pypi=force  => always download sphinx from pypi
> >
> 
> So if you already have Sphinx, this should perform an upgrade to the
> latest version?

Essentially I meant 'force' to mean *never* use the host python
installation packages. Always install all the deps in the venv,
even if they exist in the host with sufficient version met.

> >   * docs=yes + pypi=never  => ERROR if sphinx is not already
> >                               installed locally
> >
> >   * docs=yes + pypi=auto  => docs enable if sphinx is already
> >                              installed locally, or can download from
> >                              pypi as fallback
> >
> >   * docs=yes + pypi=force  => always download sphinx from pypi
> 
> Yeah, there's room for settings like these, and the above mostly makes
> sense to me.
> 
> Paolo and I had written up a more elaborate set of flags at one point
> under the premise of integrating testing, too -- for instance,
> pre-loading the testing dependencies for iotests (local qemu package,
> qemu.qmp, etc) or for avocado tests (avocado-framework and other deps
> needed for those tests). I wanted to try and check in a "minimum
> viable" version of this patch set first close to the 8.1 tree opening
> to iron out any bugs in the core feature and then I'd work to expand
> the flags and capability of the system as differential patches.
> 
> So, I think changing --enable-pypi too much is beyond the scope of the
> current patchset, but it's on my mind for what will follow almost
> immediately after it. With that in mind, we can brainstorm a little
> here and throw some darts at the wall:

Yep, I don't consider it a pre-requisite for this series.

> 
> The version of flags we originally came up with was this:
> 
> --python=... # runtime used to create venv
> --enable-pip-groups=testing,devel,avocado,meson,sphinx
> --enable-pip=now  # install all python deps now
> --enable-pip=on-demand  # install qemu.git/meson/sphinx, delay the rest
> --enable-pip=no    # offline
> --{enable,disable}-isolated-venv # let venv use system/distro if disable

This feels like a bit of overkill to me, and would create a hell
of a lot of combinations to test if you expand the matrix of
options.

> I may have pulled us in a slightly different direction with the
> version of the patches I actually sent here today, but the key ideas
> here were:
> 
> - The ability to opt into or out of different package groups at
> configure-time. meson is always required, docs/sphinx is optional, and
> the other three groups are testing related. (testing: iotests and
> qemu.qmp, devel: python self-tests, avocado: avocado tests.)

I think this is especially overkill. While you can probably come up
with some scenarios where you could use this fine grained selection,
I don't think it would be very compelling.

> - The ability to decide when packages get installed to the venv;
> either at configure time ("now") or as-needed ("on-demand") or
> functionally never ("no")

The distinction between now vs on-demand is really just about
avoiding the overhead of downloading bits that you might not
end up using. eg not downloading avocado unless you will be
running 'make check-avocado'. That's saving a few 10s or 100s
of KB of download, but doesn't feel like it'd make a noticable
difference in the overall QEMU build time.

THe 'now' and 'no' options look sufficient (corresponding to
the 'auto' and 'never' options I suggested above)

> - The ability to enable or disable venv isolation (turning on or off
> system packages).

Corresponds to the 'force' option I suggested above.

> 
> I think I abandoned the idea of configuring the venv isolation and
> have it set to "always off". We may revisit this later, but I think
> for this series it's fine to have glossed over it.
> I also skipped over the idea of having package installation being
> "now", "on-demand" or "no" -- this patch-set more or less is using a
> hybrid of "now" and "on-demand" where meson and sphinx are "now" but
> avocado is "on-demand". Unifying that discrepancy can occur in
> subsequent patches as it closely resembles what we had in 8.0 and
> earlier.

I guess the hybrid of 'now' and 'on-demand' could equally well
map to what I suggested by 'auto'. 


> > So eg distros could use pypi=never, devs would use pypi=auto mostly,
> > while CI might use pypi=force to test specific versions indepenant
> > of distros ?
> >
> 
> Sounds reasonable in general, I think, but I have some questions about
> what 'force' does, exactly.

Turning of system packages in the venv, and always installing everything
from pip.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Paolo Bonzini]

On 4/26/23 10:21, Daniel P. Berrangé wrote:
>> So if you already have Sphinx, this should perform an upgrade to the
>> latest version?
>
> Essentially I meant 'force' to mean*never*  use the host python
> installation packages. Always install all the deps in the venv,
> even if they exist in the host with sufficient version met.

I think this is essentially --enable-isolated-venv.  I don't think there 
is a usecase for "let the venv use system packages, but override them 
with pip right away".

>>
>> --python=... # runtime used to create venv
>> --enable-pip-groups=testing,devel,avocado,meson,sphinx
>> --enable-pip=now  # install all python deps now
>> --enable-pip=on-demand  # install qemu.git/meson/sphinx, delay the rest
>> --enable-pip=no    # offline
>> --{enable,disable}-isolated-venv # let venv use system/distro if disable
> 
> This feels like a bit of overkill to me, and would create a hell
> of a lot of combinations to test if you expand the matrix of
> options.

Yeah, this is a bit overkill.  I think we can reduce it to three cases, 
corresponding to:

- --enable-pypi --enable-isolated-venv - use pip to install everything, 
including for options in "auto" state (e.g. would install sphinx without 
--enable-docs)

- --enable-pypi --disable-isolated-venv - use pip to install missing 
packages.  TBD whether to do so for options in "auto" state or only for 
"enabled" (i.e., TBD whether to install sphinx without --enable-docs).

- --disable-pypi (only meaningful for --disable-isolated-venv) - apart 
from vendored wheels, just use system site packages (same as QEMU <= 8.0)

I think we want to hash out this detail first, and thus we should leave 
online mode out of the non-RFC version.  It can be implemented together 
with isolated mode.

Paolo

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Paolo Bonzini]

On 4/26/23 10:05, Paolo Bonzini wrote:
>>
> 
> Thanks, this looks pretty good.  Some changes I'd make for the non-RFC 
> version:
> 
> - I think we should just check in the meson wheel (which also removes 
> the need for patch 12, so it can be done in its stead) and remove the 
> submodule
> 
> - The verbosity of mkvenv.py can be tuned down and most prints replaced 
> with logger.info() or logger.debug()
> 
> - While I agree with keeping patch 18 separate, I would move it earlier 
> so that patch 19 can be squashed into patch 14
> 
> - I am ambivalent about keeping --enable/--disable-pypi in the first 
> committed patchset, but in any case I would move patches 16 and 20 
> before patch 15

Just one extra thing, since we're changing so much of Python handling 
and since the code is written, I would keep the Debian 10 workarounds 
for now, and only drop them after we drop support for 3.6.

Paolo

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Daniel P. Berrangé]

On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote:
> GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
>            (All green, except Python self-tests, see below)
> 
> This patch series creates a mandatory python virtual environment
> ("venv") during configure time and uses it to ensure the availability of
> meson and sphinx.
> 
> See https://www.qemu.org/2023/03/24/python/ for details. The summary is
> that the goal of this series is to ensure that the `python` used to run
> meson is the same `python` used to run Sphinx, tests, and any build-time
> python scripting we have. As it stands, meson and sphinx (and their
> extensions) *may* run in a different python environment than the one
> configured and chosen by the user at configure/build time.

I mentioned this when we were chatting on IRC, but to repeat for
the general audience..

I think it'd be useful for support purposes if the configure
script summary printed details of how we setup the venv.
eg perhaps a summary for each python module of whether we
added it to the venv, or relied on te site packages:

   python venv: meson (site), sphinx (venv), avocado (venv)


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Paolo Bonzini]

On 4/26/23 10:53, Daniel P. Berrangé wrote:
> I think it'd be useful for support purposes if the configure
> script summary printed details of how we setup the venv.
> eg perhaps a summary for each python module of whether we
> added it to the venv, or relied on te site packages:
> 
>     python venv: meson (site), sphinx (venv), avocado (venv)

Yes, I agree John did a great job with the errors but a little more work 
is needed to cleanup of mkvenv.py's logging of the "good" case.  The 
installation case is already covered by "pip install"'s output, but we 
need to have something like:

mkvenv: Creating {isolated|non-isolated} virtual environment [based on 
/home/pbonzini/myvenv]

and when creating the console-scripts shims:

mkvenv: Found avocado v90.0

Paolo

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]

On Wed, Apr 26, 2023, 4:49 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 4/26/23 10:05, Paolo Bonzini wrote:
> >>
> >
> > Thanks, this looks pretty good.  Some changes I'd make for the non-RFC
> > version:
> >
> > - I think we should just check in the meson wheel (which also removes
> > the need for patch 12, so it can be done in its stead) and remove the
> > submodule
>

OK, if there's no objections.

As a consequence, meson will likely never be downloaded from PyPI with the
patch set written as-is.

No real problem with that, just a difference.

>
> > - The verbosity of mkvenv.py can be tuned down and most prints replaced
> > with logger.info() or logger.debug()
>

😅

>
> > - While I agree with keeping patch 18 separate, I would move it earlier
> > so that patch 19 can be squashed into patch 14
> >


Agree. It was newer / more experimental so I kept it unsquash'd. If you
like it enough, asquashing I shall go.


> > - I am ambivalent about keeping --enable/--disable-pypi in the first
> > committed patchset, but in any case I would move patches 16 and 20
> > before patch 15
>

I might be stubborn but I think I want to keep it in for now. If it needs
redesigned to fit with the other flags you want to add, I think that's OK.

if we vendor the whl directly in qemu.git we won't need PyPI for meson, but
it's still useful for Sphinx so I think I'm still leaning towards keeping
it.

I'll try to refactor to keep it at the tail end of the series.


> Just one extra thing, since we're changing so much of Python handling
> and since the code is written, I would keep the Debian 10 workarounds
> for now, and only drop them after we drop support for 3.6.
>

This series was written assuming we get to drop 3.6 as a prereq. Is that
not the case?

Or did you mean to write 3.7 there?


> Paolo
>
>

[-- Attachment #2: Type: text/html, Size: 3780 bytes --]

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2094 bytes --]

On Wed, Apr 26, 2023, 5:08 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 4/26/23 10:53, Daniel P. Berrangé wrote:
> > I think it'd be useful for support purposes if the configure
> > script summary printed details of how we setup the venv.
> > eg perhaps a summary for each python module of whether we
> > added it to the venv, or relied on te site packages:
> >
> >     python venv: meson (site), sphinx (venv), avocado (venv)
>

OK, I'll try to work this out.

At the moment "already had the package" or "installed it from vendored
source" is the same pip call, so it's hard to differentiate post-hoc
without some further analysis.

One reason this is hard is because I don't have a good method for quickly
determining "Do we already have $package that fulfills $depspec" except by
invoking pip, and that's costly, 500ms at the absolute fastest I have yet
managed. 1.2s in more typical cases. Owowowow.

So, for speed, I combine "do we already have it" and "can we install it
from vendored source" into a single call to amortize pip startup cost.

With Python 3.8+ I might be able to inspect the package post-hoc to find
where it's installed to and determine if it came from system packages or
not.

It's easy to tell the user if we used PyPI or not, though.

I'll experiment.


> Yes, I agree John did a great job with the errors but a little more work
> is needed to cleanup of mkvenv.py's logging of the "good" case.


In the good case, it worked! you should be happy! :D

(kidding.)

  The
> installation case is already covered by "pip install"'s output, but we
> need to have something like:
>
> mkvenv: Creating {isolated|non-isolated} virtual environment [based on
> /home/pbonzini/myvenv]
>

"based on ..." for nested venv case only?


> and when creating the console-scripts shims:
>
> mkvenv: Found avocado v90.0
>

Sure. Up to the user at that point to figure out if we used that package or
not.

e.g. "found meson 0.5.0" might occur even if we require >=1.0.

Simple to implement, though.


> Paolo
>
>

[-- Attachment #2: Type: text/html, Size: 3898 bytes --]

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Paolo Bonzini]

On 4/26/23 18:16, John Snow wrote:
> 
>      > - I am ambivalent about keeping --enable/--disable-pypi in the first
>      > committed patchset, but in any case I would move patches 16 and 20
>      > before patch 15
> 
> I might be stubborn but I think I want to keep it in for now. If it 
> needs redesigned to fit with the other flags you want to add, I think 
> that's OK.
> 
> if we vendor the whl directly in qemu.git we won't need PyPI for meson, 
> but it's still useful for Sphinx so I think I'm still leaning towards 
> keeping it.

Yes, it's definitely useful.  It's just that unifying $python with 
sphinx-build's interpreter is as a separate (and earlier) step than 
introducing PyPI.

> I'll try to refactor to keep it at the tail end of the series.

Cool, thanks.

>     Just one extra thing, since we're changing so much of Python handling
>     and since the code is written, I would keep the Debian 10 workarounds
>     for now, and only drop them after we drop support for 3.6.
> 
> 
> This series was written assuming we get to drop 3.6 as a prereq. Is that 
> not the case?
> 
> Or did you mean to write 3.7 there?

Yes, 3.7.

Paolo

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Paolo Bonzini]

On 4/26/23 18:32, John Snow wrote:
> 
>     mkvenv: Creating {isolated|non-isolated} virtual environment [based on
>     /home/pbonzini/myvenv]
> 
> "based on ..." for nested venv case only?

Yes.

> 
>     and when creating the console-scripts shims:
> 
>     mkvenv: Found avocado v90.0
> 
> Sure. Up to the user at that point to figure out if we used that package 
> or not.
> 
> e.g. "found meson 0.5.0" might occur even if we require >=1.0.

John and I discussed offlist and he'll try to remove --gen so that all 
system packages handling is done at "mkvenv ensure" time.  The "Found" 
message can then be printed at "mkvenv ensure" time too.

Thanks,

Paolo

> Simple to implement, though.
> 

Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 7740 bytes --]

On Mon, Apr 24, 2023, 4:02 PM John Snow <jsnow@redhat.com> wrote:

> GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
>            (All green, except Python self-tests, see below)
>
> This patch series creates a mandatory python virtual environment
> ("venv") during configure time and uses it to ensure the availability of
> meson and sphinx.
>
> See https://www.qemu.org/2023/03/24/python/ for details. The summary is
> that the goal of this series is to ensure that the `python` used to run
> meson is the same `python` used to run Sphinx, tests, and any build-time
> python scripting we have. As it stands, meson and sphinx (and their
> extensions) *may* run in a different python environment than the one
> configured and chosen by the user at configure/build time.
>
> The effective change of this series is that QEMU will now
> unconditionally create a venv at configure-time and will ensure that
> meson (and sphinx, if docs are enabled) are available through that venv.
>
> Some important points as a pre-emptive "FAQ":
>
> - This venv is unconditionally created and lives at {build_dir}/pyvenv.
>
> - The python interpreter used by this venv is always the one identified
>   by configure. (Which in turn is always the one specified by --python
>   or $PYTHON)
>
> - *almost* all python scripts in qemu.git executed as part of the build
>   system, meson, sphinx, avocado tests, vm tests or CI are always
>   executed within this venv.
>
>   (iotests are not yet integrated; I plan to tackle this separately as a
>   follow-up in order to have a more tightly focused scope on that
>   series.)
>
> - It remains possible to build and test fully offline.
>   (In most cases, you just need meson and sphinx from your distro's repo.)
>
> - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
>   possible as the highest preference.
>
> - Vendored versions of e.g. 'meson' are always preferred to PyPI
>   versions for speed, repeatability and ensuring tarball builds work
>   as-is offline.
>
>   (Sphinx will not be vendored, just like it already isn't.)
>
> - Missing dependencies, when possible, are fetched and installed
>   on-demand automatically to make developer environments "just work".
>
> - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
>   Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
>
> - No new dependencies (...for most platforms. Debian and NetBSD get an
>   asterisk.)
>
> - The meson git submodule is unused after this series and can be removed.
>
> For reviewers, here's how the series is broken up:
>
> Patch 1 is a testing pre-req. Note that even with this patch,
> 'check-python-minreqs' and 'check-python-tox' CI jobs will both still
> fail on origin/master because this series requires 3.7+, but
> origin/master is currently still 3.6+.
>
> - python: update pylint configuration
>
> Patches 2-8 add the mkvenv script. The first patch checks in the barest
> essentials, and each subsequent patch adds a workaround or feature one
> at a time.
>
> - python: add mkvenv.py
> - mkvenv: add console script entry point generation
> - mkvenv: Add better error message for missing pyexapt module
> - mkvenv: generate console entry shims from inside the venv
> - mkvenv: work around broken pip installations on Debian 10
> - mkvenv: add nested venv workaround
> - mkvenv: add ensure subcommand
>
> Patches 9-11 modify our testing configuration to add new dependencies as
> needed.
>
> - tests/docker: add python3-venv dependency
> - tests/vm: Configure netbsd to use Python 3.10
> - tests/vm: add py310-expat to NetBSD
>
> Patch 12 changes how we package release tarballs.
>
> - scripts/make-release: download meson==0.61.5 .whl
>
> Patches 13-16 wire mkvenv into configure and tests.
>
> - configure: create a python venv unconditionally
> - configure: use 'mkvenv ensure meson' to bootstrap meson
> - configure: add --enable-pypi and --disable-pypi
> - tests: Use configure-provided pyvenv for tests
>
> Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
> changes could be folded earlier in the series (like the diagnose()
> patch), but I'm keeping it separate for review for now.
>
> - configure: move --enable-docs and --disable-docs back to configure
> - mkvenv: add diagnose() method for ensure() failures
> - configure: use --diagnose option with meson ensure
> - configure: bootstrap sphinx with mkvenv
>
> That's all for now, seeya!
> --js
>
> John Snow (20):
>   python: update pylint configuration
>   python: add mkvenv.py
>   mkvenv: add console script entry point generation
>   mkvenv: Add better error message for missing pyexpat module
>   mkvenv: generate console entry shims from inside the venv
>   mkvenv: work around broken pip installations on Debian 10
>   mkvenv: add nested venv workaround
>   mkvenv: add ensure subcommand
>   tests/docker: add python3-venv dependency
>   tests/vm: Configure netbsd to use Python 3.10
>   tests/vm: add py310-expat to NetBSD
>   scripts/make-release: download meson==0.61.5 .whl
>   configure: create a python venv unconditionally
>   configure: use 'mkvenv ensure meson' to bootstrap meson
>   configure: add --enable-pypi and --disable-pypi
>   tests: Use configure-provided pyvenv for tests
>   configure: move --enable-docs and --disable-docs back to configure
>   mkvenv: add diagnose() method for ensure() failures
>   configure: use --diagnose option with meson ensure
>   configure: bootstrap sphinx with mkvenv
>
>  docs/devel/acpi-bits.rst                      |   6 +-
>  docs/devel/testing.rst                        |  14 +-
>  configure                                     | 139 +--
>  .gitlab-ci.d/buildtest-template.yml           |   4 +-
>  .gitlab-ci.d/buildtest.yml                    |   6 +-
>  python/scripts/mkvenv.py                      | 871 ++++++++++++++++++
>  python/setup.cfg                              |  10 +
>  python/tests/flake8.sh                        |   1 +
>  python/tests/isort.sh                         |   1 +
>  python/tests/mypy.sh                          |   1 +
>  python/tests/pylint.sh                        |   1 +
>  .../org.centos/stream/8/x86_64/test-avocado   |   4 +-
>  scripts/device-crash-test                     |   2 +-
>  scripts/make-release                          |  11 +
>  tests/Makefile.include                        |  10 +-
>  .../dockerfiles/debian-all-test-cross.docker  |   3 +-
>  .../dockerfiles/debian-hexagon-cross.docker   |   3 +-
>  .../dockerfiles/debian-riscv64-cross.docker   |   3 +-
>  .../dockerfiles/debian-tricore-cross.docker   |   3 +-
>  tests/requirements.txt                        |   7 +-
>  tests/vm/netbsd                               |   2 +
>  21 files changed, 1016 insertions(+), 86 deletions(-)
>  create mode 100644 python/scripts/mkvenv.py
>
> --
> 2.39.2
>

Just a note to say I'm working on the respin here - Paolo suggested I
remove the "--gen" arguments to mkvenv and generate the entry point scripts
conditionally at ensure time instead.

Good idea!

I ran into some problems with msys2, though, where the entry point scripts
native to the system are binary .exe files I don't know how to recreate.

I have some research to do on windows perhaps to understand what's expected
on that platform and how to achieve it.

I'm hoping the fix is as easy as "You can still use console scripts, but
you need to use the .py or .pyw suffix". I'll know shortly!

If I don't smooth over all the issues before Wednesday, though, I'm going
to be away from my desk for the rest of the week and may need to ask Paolo
for help finishing up the polish so we can land this in a timely fashion so
we have a lot of runway to fix build issues before 8.1 rc0.

--js

>

[-- Attachment #2: Type: text/html, Size: 9667 bytes --]