[RFC PATCH v2 09/10] configure: remove --meson=; install meson to the pyvenv

From: John Snow <jsnow@redhat.com>
To: qemu-devel@nongnu.org
Cc: "Cleber Rosa" <crosa@redhat.com>, "Ani Sinha" <ani@anisinha.ca>,
	"Warner Losh" <imp@bsdimp.com>,
	"Alex Bennée" <alex.bennee@linaro.org>,
	"Reinoud Zandijk" <reinoud@netbsd.org>,
	"Thomas Huth" <thuth@redhat.com>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"John Snow" <jsnow@redhat.com>, "Ryo ONODERA" <ryoon@netbsd.org>,
	"Kyle Evans" <kevans@freebsd.org>,
	"Beraldo Leal" <bleal@redhat.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
	"Philippe Mathieu-Daudé" <philmd@linaro.org>
Subject: [RFC PATCH v2 09/10] configure: remove --meson=; install meson to the pyvenv
Date: Fri, 14 Apr 2023 01:54:48 -0400	[thread overview]
Message-ID: <20230414055449.4028284-10-jsnow@redhat.com> (raw)
In-Reply-To: <20230414055449.4028284-1-jsnow@redhat.com>

This patch changes how we detect and install meson.

The previous patch creates a lightweight Python virtual environment
unconditionally using the user's configured $python that inherits system
packages. If Meson is installed there and meets our minimum version
requirements, we will use that Meson.

In the event that Meson is installed but *not for the chosen Python
interpreter*, not found, or of insufficient version, we will attempt to
install Meson from source into the newly created Python virtual
environment.

At present, the source is obtained in the same manner as it has been;
preferring git submodules first and relying on vendored source as a
backup.

This patch (as of now) does *not* connect to PyPI and will work offline
for all supported build platforms.

As a result of this patch, the Python interpreter we use for both our
own build scripts *and* Meson extensions are always known to be the
exact same Python. As a further benefit, there will also be a symlink
available in the build directory that points to the correct, configured
python and can be used by e.g. manual tests to invoke the correct,
configured Python unambiguously.

Notes:

- The meson git submodule can be removed in favor of just loading meson
  from PyPI; but we probably want a configure flag to toggle
  online/offline behavior. (What do we want the default to be? Online is
  my guess.)

- Installing meson from the source tree for vendored cases (tarball
  releases) can be replaced by vendoring a .whl file instead. This will
  remove some of the `--no-use-pep517` hackiness and alleviates the
  requirement that users install the python3 'wheel' dependency.

- Most of this logic can be moved into mkvenv.py, *especially* if the
  meson git submodule is removed.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 configure                           | 104 +++++++++++++++-------------
 .gitlab-ci.d/buildtest-template.yml |   4 +-
 2 files changed, 59 insertions(+), 49 deletions(-)

diff --git a/configure b/configure
index 03278fd891..c8f73d4ff7 100755
--- a/configure
+++ b/configure
@@ -721,8 +721,6 @@ for opt do
   ;;
   --skip-meson) skip_meson=yes
   ;;
-  --meson=*) meson="$optarg"
-  ;;
   --ninja=*) ninja="$optarg"
   ;;
   --smbd=*) smbd="$optarg"
@@ -1006,7 +1004,6 @@ Advanced options (experts only):
   --cross-prefix-ARCH=PREFIX cross compiler prefix when building ARCH guest test cases
   --make=MAKE              use specified make [$make]
   --python=PYTHON          use specified python [$python]
-  --meson=MESON            use specified meson [$meson]
   --ninja=NINJA            use specified ninja [$ninja]
   --smbd=SMBD              use specified smbd [$smbd]
   --with-git=GIT           use specified git [$git]
@@ -1079,7 +1076,8 @@ fi
 
 # Resolve PATH
 python="$(command -v "$python")"
-explicit_python=yes
+# This variable is intended to be used only for error messages:
+target_python=$python
 
 # Create a Python virtual environment using our configured python.
 # The stdout of this script will be the location of a symlink that
@@ -1106,58 +1104,70 @@ fi
 # Suppress writing compiled files
 python="$python -B"
 
-has_meson() {
-  local python_dir=$(dirname "$python")
-  # PEP405: pyvenv.cfg is either adjacent to the Python executable
-  # or one directory above
-  if test -f $python_dir/pyvenv.cfg || test -f $python_dir/../pyvenv.cfg; then
-    # Ensure that Meson and Python come from the same virtual environment
-    test -x "$python_dir/meson" &&
-      test "$(command -v meson)" -ef "$python_dir/meson"
-  else
-    has meson
-  fi
+pip_install() {
+    $python -m pip install -v \
+            --disable-pip-version-check \
+            --no-index \
+            "$@"
 }
 
-if test -z "$meson"; then
-    if test "$explicit_python" = no && has_meson && version_ge "$(meson --version)" 0.61.5; then
-        meson=meson
-    elif test "$git_submodules_action" != 'ignore' ; then
-        meson=git
-    elif test -e "${source_path}/meson/meson.py" ; then
-        meson=internal
+# OK, let's have some fun!
+
+# This install command is intended to either fail or be a NOP;
+# because we're offline, it's just a convenient version check.
+if ! pip_install 'meson>=0.61.5'; then
+    # Either we don't have Meson, or our Meson is too old.
+    # (Future revisions of this patchset can be less chatty.)
+    if test -e pyvenv/bin/meson; then
+        echo "Meson in pyvenv is too old: $(pyvenv/bin/meson --version)"
+    elif has meson ; then
+        echo "Meson was found installed on your system," \
+             "but not for the configured Python interpreter ($target_python)."
+        echo "(Hint: check '$(which meson)' to see which interpreter its shebang uses.)"
     else
-        if test "$explicit_python" = yes; then
-            error_exit "--python requires using QEMU's embedded Meson distribution, but it was not found."
-        else
-            error_exit "Meson not found.  Use --meson=/path/to/meson"
+        echo "Meson was not found."
+    fi
+
+    # OK, but can we fix it, though? :~)
+    if test "$git_submodules_action" != 'ignore' ; then
+        git_submodules="${git_submodules} meson"
+        echo "Attempting to install meson from git submodule ..."
+        # Stolen from later in the configure file.
+        # Is it a problem if we front-load this now and run it again later?
+        if ! (GIT="$git" "$source_path/scripts/git-submodule.sh" "$git_submodules_action" "$git_submodules"); then
+            exit 1
         fi
+    elif test -e "${source_path}/meson/setup.cfg" ; then
+        echo "Attempting to install meson from vendored source ..."
+    else
+        # In the future, we could use PyPI as a source if the user allows it.
+        # For now, you're outta luck!
+        error_exit "A suitable version of Meson was not found."
     fi
-else
-    # Meson uses its own Python interpreter to invoke other Python scripts,
-    # but the user wants to use the one they specified with --python.
+
+    # If we're here, we have the meson source and we can attempt to
+    # install it into our venv.
+
+    # We want to install meson with --no-use-pep517 if possible,
+    # because it avoids needing a 'wheel' dependency. Old versions
+    # of pip do this by default, so test for the behavior.
     #
-    # We do not want to override the distro Python interpreter (and sometimes
-    # cannot: for example in Homebrew /usr/bin/meson is a bash script), so
-    # just require --meson=git|internal together with --python.
-    if test "$explicit_python" = yes; then
-        case "$meson" in
-            git | internal) ;;
-            *) error_exit "--python requires using QEMU's embedded Meson distribution." ;;
-        esac
+    # --no-build-isolation was added to pip 10.0.
+    # --no-use-pep517 was added ... sometime after 18.1?
+    pip_flags='--no-build-isolation'
+    if $python -m pip install --help | grep 'no-use-pep517' > /dev/null 2>&1 ; then
+        pip_flags="${pip_flags} --no-use-pep517"
+    fi
+    if ! pip_install $pip_flags "${source_path}/meson" ; then
+        exit 1
     fi
 fi
 
-if test "$meson" = git; then
-    git_submodules="${git_submodules} meson"
-fi
-
-case "$meson" in
-    git | internal)
-        meson="$python ${source_path}/meson/meson.py"
-        ;;
-    *) meson=$(command -v "$meson") ;;
-esac
+# At this point, we expect Meson to be installed and available.
+# We expect mkvenv or pip to have created pyvenv/bin/meson for us.
+# We ignore PATH completely here: we want to use the venv's Meson
+# *exclusively*.
+meson="$(cd pyvenv/bin; pwd)/meson"
 
 # Probe for ninja
 
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index a6cfe9be97..7edb50b760 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -12,12 +12,12 @@
     - mkdir build
     - cd build
     - ../configure --enable-werror --disable-docs --enable-fdt=system
-          ${LD_JOBS:+--meson=git} ${TARGETS:+--target-list="$TARGETS"}
+          ${TARGETS:+--target-list="$TARGETS"}
           $CONFIGURE_ARGS ||
       { cat config.log meson-logs/meson-log.txt && exit 1; }
     - if test -n "$LD_JOBS";
       then
-        ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
+        pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
     - make -j"$JOBS"
     - if test -n "$MAKE_CHECK_ARGS";
-- 
2.39.2


