[RFC PATCH 0/3] Automatically convert configure options to meson build options

[RFC PATCH 0/3] Automatically convert configure options to meson build options

[Paolo Bonzini]

Right now meson_options.txt lists less than a dozen options, but about
40 more could come as configure tests are converted and moved to
meson.build.  Each option needs code in configure to parse it and pass
the option down to Meson as a -D command-line argument; in addition the
default must be duplicated between configure and meson_options.txt.

This series tries to remove the code duplication by passing unknown
--enable and --disable options to a Python program, and using
Meson's introspection support to match those to meson_options.txt

The disadvantages are:

- because we parse command-line options before meson is available,
the introspection output is stored in the source tree.  The output
is slightly modified using the "jq" tool in order to ensure that it's
stable and that modifications to meson_buildoptions.txt do not cause
horrible conflicts.  This is the main reason for the unattractive
diffstat (the number of JSON lines added is higher than the number
of configure lines removed, though of course the latter are code
that must be maintained manually and the former is not).

- we now need Python to generate the full help, so if Python is
missing we can only print a partial message and direct the user
to specify the interpreter with --python.  It would be possible to fix
this by rewriting the script in Perl (at least on Fedora, JSON::PP is
always installed if Perl is, because it's a dependency for CPAN; I'd
have to check Ubuntu and the BSDs), or if we want to write it as a
Bourne shell script, to further massage the introspection output into
for example TAB-separated values.

Opinions are welcome on whether this is worthwhile and how to solve
the above doubts.

Paolo

Paolo Bonzini (3):
  configure: quote command line arguments in config.status
  configure: early test for Python
  configure: automatically parse command line for meson -D options

 Makefile                                |   6 ++
 configure                               | 107 ++++++++-----------
 docs/devel/build-system.rst             |  35 +------
 meson-buildoptions.json                 | 130 ++++++++++++++++++++++++
 scripts/configure-parse-buildoptions.py |  94 +++++++++++++++++
 5 files changed, 280 insertions(+), 92 deletions(-)
 create mode 100644 meson-buildoptions.json
 create mode 100644 scripts/configure-parse-buildoptions.py

-- 
2.26.2

[PATCH 1/3] configure: quote command line arguments in config.status

[Paolo Bonzini]

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 configure | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index 53723ace57..beae010e39 100755
--- a/configure
+++ b/configure
@@ -89,6 +89,10 @@ printf " '%s'" "$0" "$@" >> config.log
 echo >> config.log
 echo "#" >> config.log
 
+quote_sh() {
+    printf "'%s'" "$(echo "$1" | sed "s,','\\',")"
+}
+
 print_error() {
     (echo
     echo "ERROR: $1"
@@ -8061,7 +8065,7 @@ preserve_env WINDRES
 
 printf "exec" >>config.status
 for i in "$0" "$@"; do
-  test "$i" = --skip-meson || printf " '%s'" "$i" >>config.status
+  test "$i" = --skip-meson || printf " %s" "$(quote_sh $i)" >>config.status
 done
 echo ' "$@"' >>config.status
 chmod +x config.status
-- 
2.26.2

[PATCH 2/3] configure: early test for Python

[Paolo Bonzini]

---
 configure | 30 +++++++++++++++++-------------
 1 file changed, 17 insertions(+), 13 deletions(-)

diff --git a/configure b/configure
index beae010e39..b4f5d2e272 100755
--- a/configure
+++ b/configure
@@ -1631,6 +1631,23 @@ for opt do
   esac
 done
 
+if test -n "$python"
+then
+    # Preserve python version since some functionality is dependent on it
+    python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0], sys.version_info[1], sys.version_info[2]))' 2>/dev/null)
+
+    # Suppress writing compiled files
+    python="$python -B"
+
+    # Note that if the Python conditional here evaluates True we will exit
+    # with status 1 which is a shell 'false' value.
+    if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
+        print_error "Cannot use '$python', Python >= 3.5 is required." \
+            "Use --python=/path/to/python to specify a supported Python."
+        python=
+    fi
+fi
+
 firmwarepath="${firmwarepath:-$prefix/share/qemu-firmware}"
 libdir="${libdir:-$prefix/lib}"
 libexecdir="${libexecdir:-$prefix/libexec}"
@@ -1962,19 +1979,6 @@ then
     error_exit "Python not found. Use --python=/path/to/python"
 fi
 
-# Note that if the Python conditional here evaluates True we will exit
-# with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
-  error_exit "Cannot use '$python', Python >= 3.5 is required." \
-      "Use --python=/path/to/python to specify a supported Python."
-fi
-
-# Preserve python version since some functionality is dependent on it
-python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0], sys.version_info[1], sys.version_info[2]))' 2>/dev/null)
-
-# Suppress writing compiled files
-python="$python -B"
-
 if test -z "$meson"; then
     if test "$explicit_python" = no && has meson && version_ge "$(meson --version)" 0.55.1; then
         meson=meson
-- 
2.26.2

[PATCH 3/3] configure: automatically parse command line for meson -D options

[Paolo Bonzini]

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Makefile                                |   6 ++
 configure                               |  71 +++++--------
 docs/devel/build-system.rst             |  35 +------
 meson-buildoptions.json                 | 130 ++++++++++++++++++++++++
 scripts/configure-parse-buildoptions.py |  94 +++++++++++++++++
 5 files changed, 258 insertions(+), 78 deletions(-)
 create mode 100644 meson-buildoptions.json
 create mode 100644 scripts/configure-parse-buildoptions.py

diff --git a/Makefile b/Makefile
index d6c5c9fdef..b22e5b272e 100644
--- a/Makefile
+++ b/Makefile
@@ -80,6 +80,12 @@ ifneq ($(MESON),)
 Makefile.mtest: build.ninja scripts/mtest2make.py
 	$(MESON) introspect --tests --benchmarks | $(PYTHON) scripts/mtest2make.py > $@
 -include Makefile.mtest
+
+# jq is used to keep the result stable
+meson-buildoptions.json: meson_options.txt
+	$(MESON) introspect --buildoptions . | \
+	    jq 'map(select(.section == "user")) | sort_by(.name)' \
+	    > $(SRC_PATH)/meson-buildoptions.json
 endif
 
 Makefile: .git-submodule-status
diff --git a/configure b/configure
index b4f5d2e272..54958bfc17 100755
--- a/configure
+++ b/configure
@@ -396,16 +396,10 @@ docs=""
 fdt=""
 netmap="no"
 sdl="auto"
-sdl_image="auto"
 virtfs=""
 mpath=""
-vnc="enabled"
 sparse="no"
 vde=""
-vnc_sasl="auto"
-vnc_jpeg="auto"
-vnc_png="auto"
-xkbcommon="auto"
 xen=""
 xen_ctrl_version=""
 xen_pci_passthrough=""
@@ -466,7 +460,6 @@ trace_file="trace"
 spice=""
 rbd=""
 smartcard=""
-u2f="auto"
 libusb=""
 usb_redir=""
 opengl=""
@@ -539,6 +532,7 @@ rng_none="no"
 secret_keyring=""
 libdaxctl=""
 meson=""
+meson_args=""
 ninja=""
 skip_meson=no
 gettext=""
@@ -1100,10 +1094,6 @@ for opt do
   ;;
   --enable-sdl) sdl="enabled"
   ;;
-  --disable-sdl-image) sdl_image="disabled"
-  ;;
-  --enable-sdl-image) sdl_image="enabled"
-  ;;
   --disable-qom-cast-debug) qom_cast_debug="no"
   ;;
   --enable-qom-cast-debug) qom_cast_debug="yes"
@@ -1116,10 +1106,6 @@ for opt do
   ;;
   --enable-mpath) mpath="yes"
   ;;
-  --disable-vnc) vnc="disabled"
-  ;;
-  --enable-vnc) vnc="enabled"
-  ;;
   --disable-gettext) gettext="false"
   ;;
   --enable-gettext) gettext="true"
@@ -1158,18 +1144,6 @@ for opt do
   ;;
   --disable-strip) strip_opt="no"
   ;;
-  --disable-vnc-sasl) vnc_sasl="disabled"
-  ;;
-  --enable-vnc-sasl) vnc_sasl="enabled"
-  ;;
-  --disable-vnc-jpeg) vnc_jpeg="disabled"
-  ;;
-  --enable-vnc-jpeg) vnc_jpeg="enabled"
-  ;;
-  --disable-vnc-png) vnc_png="disabled"
-  ;;
-  --enable-vnc-png) vnc_png="enabled"
-  ;;
   --disable-slirp) slirp="no"
   ;;
   --enable-slirp=git) slirp="git"
@@ -1369,10 +1343,6 @@ for opt do
   ;;
   --enable-smartcard) smartcard="yes"
   ;;
-  --disable-u2f) u2f="disabled"
-  ;;
-  --enable-u2f) u2f="enabled"
-  ;;
   --disable-libusb) libusb="no"
   ;;
   --enable-libusb) libusb="yes"
@@ -1593,10 +1563,6 @@ for opt do
   ;;
   --disable-libpmem) libpmem=no
   ;;
-  --enable-xkbcommon) xkbcommon="enabled"
-  ;;
-  --disable-xkbcommon) xkbcommon="disabled"
-  ;;
   --enable-plugins) plugins="yes"
   ;;
   --disable-plugins) plugins="no"
@@ -1623,6 +1589,9 @@ for opt do
   ;;
   --disable-libdaxctl) libdaxctl=no
   ;;
+  --enable-* | --disable-*)
+      meson_args="$meson_args $(quote_sh "$opt")"
+  ;;
   *)
       echo "ERROR: unknown option $opt"
       echo "Try '$0 --help' for more information"
@@ -1648,6 +1617,16 @@ then
     fi
 fi
 
+# Convert unknown --enable/--disable arguments to Meson build options.
+meson_options=$(eval \
+    \$python \
+    \$source_path/scripts/configure-parse-buildoptions.py \
+    \< \$source_path/meson-buildoptions.json $meson_args)
+if test $? = 1; then
+    echo "Try '$0 --help' for more information"
+    exit 1
+fi
+
 firmwarepath="${firmwarepath:-$prefix/share/qemu-firmware}"
 libdir="${libdir:-$prefix/lib}"
 libexecdir="${libexecdir:-$prefix/libexec}"
@@ -1853,10 +1831,14 @@ Advanced options (experts only):
                            enable plugins via shared library loading
   --disable-containers     don't use containers for cross-building
   --gdb=GDB-path           gdb to use for gdbstub tests [$gdb_bin]
-
-Optional features, enabled with --enable-FEATURE and
-disabled with --disable-FEATURE, default is enabled if available:
-
+EOF
+  if test -z "$python"; then
+    echo
+    echo "Python not found. Use --python=/path/to/python to see complete help"
+    exit 0
+  fi
+  $python $source_path/scripts/configure-parse-buildoptions.py --print-help < $source_path/meson-buildoptions.json
+cat << EOF
   system          all system emulation targets
   user            supported user emulation targets
   linux-user      all linux usermode emulation targets
@@ -1883,10 +1865,6 @@ disabled with --disable-FEATURE, default is enabled if available:
   vte             vte support for the gtk UI
   curses          curses UI
   iconv           font glyph conversion support
-  vnc             VNC UI support
-  vnc-sasl        SASL encryption for VNC server
-  vnc-jpeg        JPEG lossy compression for VNC server
-  vnc-png         PNG compression for VNC server
   cocoa           Cocoa UI (Mac OS X only)
   virtfs          VirtFS
   mpath           Multipath persistent reservation passthrough
@@ -1920,7 +1898,6 @@ disabled with --disable-FEATURE, default is enabled if available:
   libiscsi        iscsi support
   libnfs          nfs support
   smartcard       smartcard support (libcacard)
-  u2f             U2F support (u2f-emu)
   libusb          libusb (for usb passthrough)
   live-block-migration   Block migration in the main migration stream
   usb-redir       usb network redirection support
@@ -1962,7 +1939,6 @@ disabled with --disable-FEATURE, default is enabled if available:
   capstone        capstone disassembler support
   debug-mutex     mutex debugging support
   libpmem         libpmem support
-  xkbcommon       xkbcommon support
   rng-none        dummy RNG, avoid using /dev/(u)random and getrandom()
   libdaxctl       libdaxctl support
 
@@ -8008,9 +7984,8 @@ NINJA=${ninja:-$PWD/ninjatool} $meson setup \
         -Dstrip=$(if test "$strip_opt" = yes; then echo true; else echo false; fi) \
         -Db_pie=$(if test "$pie" = yes; then echo true; else echo false; fi) \
         -Db_coverage=$(if test "$gcov" = yes; then echo true; else echo false; fi) \
-	-Dsdl=$sdl -Dsdl_image=$sdl_image \
-	-Dvnc=$vnc -Dvnc_sasl=$vnc_sasl -Dvnc_jpeg=$vnc_jpeg -Dvnc_png=$vnc_png \
-	-Dgettext=$gettext -Dxkbcommon=$xkbcommon -Du2f=$u2f\
+	$meson_args \
+	-Dsdl=$sdl -Dgettext=$gettext \
         $cross_arg \
         "$PWD" "$source_path"
 
diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
index 08e85c69e1..9fd321f227 100644
--- a/docs/devel/build-system.rst
+++ b/docs/devel/build-system.rst
@@ -52,12 +52,6 @@ following tasks:
 
  - Add a Meson build option to meson_options.txt.
 
- - Add support to the command line arg parser to handle any new
-   `--enable-XXX`/`--disable-XXX` flags required by the feature.
-
- - Add information to the help output message to report on the new
-   feature flag.
-
  - Add code to perform the actual feature check.
 
  - Add code to include the feature status in `config-host.h`
@@ -66,30 +60,6 @@ following tasks:
    upon completion.
 
 
-Taking the probe for SDL2_Image as an example, we have the following pieces
-in configure::
-
-  # Initial variable state
-  sdl_image=auto
-
-  ..snip..
-
-  # Configure flag processing
-  --disable-sdl-image) sdl_image=disabled
-  ;;
-  --enable-sdl-image) sdl_image=enabled
-  ;;
-
-  ..snip..
-
-  # Help output feature message
-  sdl-image         SDL Image support for icons
-
-  ..snip..
-
-  # Meson invocation
-  -Dsdl_image=$sdl_image
-
 In meson_options.txt::
 
   option('sdl', type : 'feature', value : 'auto',
@@ -108,6 +78,11 @@ In meson.build::
   # Summary
   summary_info += {'SDL image support': sdl_image.found()}
 
+The configure script automatically converts command line options to
+Meson options as long as the `meson-buildoptions.json` file in the
+root source directory is up-to-date.  You can simply type
+`make meson-buildoptions.json` to update it; the `jq` program
+is needed.
 
 
 Helper functions
diff --git a/meson-buildoptions.json b/meson-buildoptions.json
new file mode 100644
index 0000000000..dbaf5bc71e
--- /dev/null
+++ b/meson-buildoptions.json
@@ -0,0 +1,130 @@
+[
+  {
+    "name": "docdir",
+    "value": "doc",
+    "section": "user",
+    "machine": "any",
+    "type": "string",
+    "description": "Base directory for documentation installation (can be empty)"
+  },
+  {
+    "name": "gettext",
+    "value": true,
+    "section": "user",
+    "machine": "any",
+    "type": "boolean",
+    "description": "Localization of the GTK+ user interface"
+  },
+  {
+    "name": "qemu_suffix",
+    "value": "qemu",
+    "section": "user",
+    "machine": "any",
+    "type": "string",
+    "description": "Suffix for QEMU data/modules/config directories (can be empty)"
+  },
+  {
+    "name": "sdl",
+    "value": "auto",
+    "section": "user",
+    "machine": "any",
+    "choices": [
+      "enabled",
+      "disabled",
+      "auto"
+    ],
+    "type": "combo",
+    "description": "SDL user interface"
+  },
+  {
+    "name": "sdl_image",
+    "value": "auto",
+    "section": "user",
+    "machine": "any",
+    "choices": [
+      "enabled",
+      "disabled",
+      "auto"
+    ],
+    "type": "combo",
+    "description": "SDL Image support for icons"
+  },
+  {
+    "name": "u2f",
+    "value": "auto",
+    "section": "user",
+    "machine": "any",
+    "choices": [
+      "enabled",
+      "disabled",
+      "auto"
+    ],
+    "type": "combo",
+    "description": "U2F emulation support"
+  },
+  {
+    "name": "vnc",
+    "value": "enabled",
+    "section": "user",
+    "machine": "any",
+    "choices": [
+      "enabled",
+      "disabled",
+      "auto"
+    ],
+    "type": "combo",
+    "description": "VNC server"
+  },
+  {
+    "name": "vnc_jpeg",
+    "value": "auto",
+    "section": "user",
+    "machine": "any",
+    "choices": [
+      "enabled",
+      "disabled",
+      "auto"
+    ],
+    "type": "combo",
+    "description": "JPEG lossy compression for VNC server"
+  },
+  {
+    "name": "vnc_png",
+    "value": "auto",
+    "section": "user",
+    "machine": "any",
+    "choices": [
+      "enabled",
+      "disabled",
+      "auto"
+    ],
+    "type": "combo",
+    "description": "PNG compression for VNC server"
+  },
+  {
+    "name": "vnc_sasl",
+    "value": "auto",
+    "section": "user",
+    "machine": "any",
+    "choices": [
+      "enabled",
+      "disabled",
+      "auto"
+    ],
+    "type": "combo",
+    "description": "SASL authentication for VNC server"
+  },
+  {
+    "name": "xkbcommon",
+    "value": "auto",
+    "section": "user",
+    "machine": "any",
+    "choices": [
+      "enabled",
+      "disabled",
+      "auto"
+    ],
+    "type": "combo",
+    "description": "xkbcommon support"
+  }
+]
diff --git a/scripts/configure-parse-buildoptions.py b/scripts/configure-parse-buildoptions.py
new file mode 100644
index 0000000000..4171312d75
--- /dev/null
+++ b/scripts/configure-parse-buildoptions.py
@@ -0,0 +1,94 @@
+#! /usr/bin/env python3
+
+# Parse configure command line options based on Meson's user build options
+# introspection data (passed on stdin).
+#
+# Copyright (C) 2020 Red Hat, Inc.
+#
+# Author: Paolo Bonzini <pbonzini@redhat.com>
+
+import json
+import re
+import shlex
+import sys
+import textwrap
+
+SKIP_OPTIONS = [ 'docdir', 'qemu_suffix' ]
+FEATURE_CHOICES = [ 'auto', 'disabled', 'enabled' ]
+
+options = { x['name']: x
+            for x in json.load(sys.stdin)
+            if x['section'] == 'user' and x['name'] not in SKIP_OPTIONS }
+
+for x in options.values():
+    if x['type'] == 'combo' and sorted(x['choices']) == FEATURE_CHOICES:
+        x['type'] = 'feature'
+
+def value_to_help(x):
+    if x == True:
+        return 'enabled'
+    if x == False:
+        return 'disabled'
+    return str(x)
+
+def print_help_line(key, opt, indent):
+    key = '  ' + key
+    value = '%s [%s]' % (opt['description'], value_to_help(opt['value']))
+    if len(key) >= indent:
+        print(key)
+        key = ''
+    spaces = ' ' * indent
+    key = (key + spaces)[0:indent]
+    print(textwrap.fill(value, initial_indent=key, subsequent_indent=spaces))
+
+def print_help():
+    for o, x in options.items():
+        if x['type'] not in ('boolean', 'feature'):
+            print_help_line('--enable-' + o, x, 24)
+
+    print()
+    print('Optional features, enabled with --enable-FEATURE and')
+    print('disabled with --disable-FEATURE:')
+    for o, x in options.items():
+        if x['type'] in ('boolean', 'feature'):
+            print_help_line(o, x, 18)
+
+def error(s, *args):
+    print('ERROR:', s % args, file=sys.stderr)
+    sys.exit(1)
+
+def main(argv):
+    if not argv:
+        return
+
+    if argv[0] == '--print-help':
+        print_help()
+        return
+
+    args = []
+    for arg in sys.argv[1:]:
+        m = re.search('--(enable|disable)-([^=]*)(?:=(.*))?', arg)
+        if not m:
+            error('internal error parsing command line')
+        opt = m.group(2).replace('-', '_')
+        if opt not in options:
+            error('Unknown option --%s-%s', m.group(1), m.group(2))
+        opt_type = options[opt]['type']
+        if opt_type in ('boolean', 'feature'):
+            if m.group(3) is not None:
+                error('option --%s-%s does not take an argument', m.group(1), m.group(2))
+            if opt_type == 'feature':
+                value = m.group(1) + 'd'
+            else:
+                value = 'true' if m.group(1) == 'enable' else 'false'
+        else:
+            if m.group(1) == 'disable':
+                error('Unknown option --disable-%s', m.group(2))
+            if m.group(3) is None:
+                error('option --enable-%s takes an argument', m.group(2))
+
+        args.append(shlex.quote('-D%s=%s' % (opt, value)))
+    print(' '.join(args))
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
-- 
2.26.2

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[罗勇刚(Yonggang Luo)]

[-- Attachment #1: Type: text/plain, Size: 3060 bytes --]

On Sun, Sep 13, 2020 at 6:07 PM Paolo Bonzini <pbonzini@redhat.com> wrote:

> Right now meson_options.txt lists less than a dozen options, but about
> 40 more could come as configure tests are converted and moved to
> meson.build.  Each option needs code in configure to parse it and pass
> the option down to Meson as a -D command-line argument; in addition the
> default must be duplicated between configure and meson_options.txt.
>
> This series tries to remove the code duplication by passing unknown
> --enable and --disable options to a Python program, and using
> Meson's introspection support to match those to meson_options.txt
>
> The disadvantages are:
>
> - because we parse command-line options before meson is available,
> the introspection output is stored in the source tree.  The output
> is slightly modified using the "jq" tool in order to ensure that it's
> stable and that modifications to meson_buildoptions.txt do not cause
> horrible conflicts.  This is the main reason for the unattractive
> diffstat (the number of JSON lines added is higher than the number
> of configure lines removed, though of course the latter are code
> that must be maintained manually and the former is not).
>
> - we now need Python to generate the full help, so if Python is
> missing we can only print a partial message and direct the user
> to specify the interpreter with --python.  It would be possible to fix
> this by rewriting the script in Perl (at least on Fedora, JSON::PP is
> always installed if Perl is, because it's a dependency for CPAN; I'd
> have to check Ubuntu and the BSDs), or if we want to write it as a
> Bourne shell script, to further massage the introspection output into
> for example TAB-separated values.
>
I am confusing about this? we were converting configure to meson, and
python is a
force dependencies, why we need rewrite the script in Perl? If we wanna
build qemu, the first thing
we need to install is python+meson, so there is no need convert to Perl or
bash script.
And perl/bash will incur msys2/mingw user, the ideal way is we only depends
on python+meson to building qemu

>
> Opinions are welcome on whether this is worthwhile and how to solve
> the above doubts.
>
> Paolo
>
> Paolo Bonzini (3):
>   configure: quote command line arguments in config.status
>   configure: early test for Python
>   configure: automatically parse command line for meson -D options
>
>  Makefile                                |   6 ++
>  configure                               | 107 ++++++++-----------
>  docs/devel/build-system.rst             |  35 +------
>  meson-buildoptions.json                 | 130 ++++++++++++++++++++++++
>  scripts/configure-parse-buildoptions.py |  94 +++++++++++++++++
>  5 files changed, 280 insertions(+), 92 deletions(-)
>  create mode 100644 meson-buildoptions.json
>  create mode 100644 scripts/configure-parse-buildoptions.py
>
> --
> 2.26.2
>
>
>

-- 
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

[-- Attachment #2: Type: text/html, Size: 3830 bytes --]

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[Paolo Bonzini]

On 13/09/20 12:15, 罗勇刚(Yonggang Luo) wrote:
> 
> I am confusing about this? we were converting configure to meson,
> and python is a force dependencies, why we need rewrite the script in
> Perl? If we wanna build qemu, the first thing we need to install is
> python+meson, so there is no need convert to Perl or bash script. And
> perl/bash will incur msys2/mingw user, the ideal way is we only 
> depends on python+meson to building qemu

The main issue is that the Python interpreter is any of python, python3,
python3.5.  Furthermore Meson's shebang python is not necessarily the
first "python3" in the path.

Perl is pretty much always "/usr/bin/env perl".  Note that Perl is
already needed to run "make check".  When/if it will be possible to only
build QEMU with python+meson there will be no configure script anymore,
so Perl is a reasonable choice for parsing the configure command line
options.  It wasn't my first choice, in fact this series includes a
Python implementation, but it does have some advantages.

Paolo

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[Daniel P. Berrangé]

On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote:
> Right now meson_options.txt lists less than a dozen options, but about
> 40 more could come as configure tests are converted and moved to
> meson.build.  Each option needs code in configure to parse it and pass
> the option down to Meson as a -D command-line argument; in addition the
> default must be duplicated between configure and meson_options.txt.
> 
> This series tries to remove the code duplication by passing unknown
> --enable and --disable options to a Python program, and using
> Meson's introspection support to match those to meson_options.txt
> 
> The disadvantages are:
> 
> - because we parse command-line options before meson is available,
> the introspection output is stored in the source tree.  The output
> is slightly modified using the "jq" tool in order to ensure that it's
> stable and that modifications to meson_buildoptions.txt do not cause
> horrible conflicts.  This is the main reason for the unattractive
> diffstat (the number of JSON lines added is higher than the number
> of configure lines removed, though of course the latter are code
> that must be maintained manually and the former is not).
> 
> - we now need Python to generate the full help, so if Python is
> missing we can only print a partial message and direct the user
> to specify the interpreter with --python.  It would be possible to fix
> this by rewriting the script in Perl (at least on Fedora, JSON::PP is
> always installed if Perl is, because it's a dependency for CPAN; I'd
> have to check Ubuntu and the BSDs), or if we want to write it as a
> Bourne shell script, to further massage the introspection output into
> for example TAB-separated values.

IMHO we should stay as far away from Perl as possible, and I say this as
someone who enjoys writing Perl scripts !  Perl is a significant barrier
to entry for potential contributors, because it just doesn't have the
wide knowledge base that it did in the past. You can expect most people
to have some familiarity with Python, or be able to pick it up fairly
easily in a way that just isn't possible in Perl.

As for shell, we have a never ending stream of bugs due to the wide
range of different shell impls which make portable coding very hard.

In general I think our overall goal should be to focus on getting down to
use of a single scripting language in QEMU, and that language clearly has
to be python3. We shouldn't introduce any new usage of Perl or Shell
in QEMU IMHO. Even if we think the usage will only be short term temporary
workaround, short term hacks in QEMU have a habit of living for years longer
than we expect.

> Opinions are welcome on whether this is worthwhile and how to solve
> the above doubts.

IIUC, the motivation of this series is just to remove some duplication
of defaults / args, nothing more than that ?

If correct, then I'd say it probably isn't worth the hassle. I'd say we
should focus effort on just converting more of configure to meson, as
quickly as practical, and not try to add much more magic that's only
relevant for the transition time.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote:
> - because we parse command-line options before meson is available,
> the introspection output is stored in the source tree.  The output
> is slightly modified using the "jq" tool in order to ensure that it's
> stable and that modifications to meson_buildoptions.txt do not cause
> horrible conflicts.  This is the main reason for the unattractive
> diffstat (the number of JSON lines added is higher than the number
> of configure lines removed, though of course the latter are code
> that must be maintained manually and the former is not).

Does this add a build dependency on jq? Is it possible to use a Python
script or even a command-line one-liner instead?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[罗勇刚(Yonggang Luo)]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Mon, Sep 14, 2020 at 5:58 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote:
> > - because we parse command-line options before meson is available,
> > the introspection output is stored in the source tree.  The output
> > is slightly modified using the "jq" tool in order to ensure that it's
> > stable and that modifications to meson_buildoptions.txt do not cause
> > horrible conflicts.  This is the main reason for the unattractive
> > diffstat (the number of JSON lines added is higher than the number
> > of configure lines removed, though of course the latter are code
> > that must be maintained manually and the former is not).
>
> Does this add a build dependency on jq? Is it possible to use a Python
> script or even a command-line one-liner instead?
What's jq stands for, is that a perl script?



--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

[-- Attachment #2: Type: text/html, Size: 1189 bytes --]

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[Paolo Bonzini]

On 14/09/20 11:57, Stefan Hajnoczi wrote:
> On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote:
>> - because we parse command-line options before meson is available,
>> the introspection output is stored in the source tree.  The output
>> is slightly modified using the "jq" tool in order to ensure that it's
>> stable and that modifications to meson_buildoptions.txt do not cause
>> horrible conflicts.  This is the main reason for the unattractive
>> diffstat (the number of JSON lines added is higher than the number
>> of configure lines removed, though of course the latter are code
>> that must be maintained manually and the former is not).
> 
> Does this add a build dependency on jq? Is it possible to use a Python
> script or even a command-line one-liner instead?

No, only developers need jq and only if they add configure options.
Using Python would certainly be possible, though probably it wouldn't be
a one-liner as for jq.

Paolo

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[Paolo Bonzini]

On 14/09/20 11:12, Daniel P. Berrangé wrote:
> IMHO we should stay as far away from Perl as possible, and I say this as
> someone who enjoys writing Perl scripts !  Perl is a significant barrier
> to entry for potential contributors, because it just doesn't have the
> wide knowledge base that it did in the past. You can expect most people
> to have some familiarity with Python, or be able to pick it up fairly
> easily in a way that just isn't possible in Perl.
> 
> As for shell, we have a never ending stream of bugs due to the wide
> range of different shell impls which make portable coding very hard.
> 
> In general I think our overall goal should be to focus on getting down to
> use of a single scripting language in QEMU, and that language clearly has
> to be python3. We shouldn't introduce any new usage of Perl or Shell
> in QEMU IMHO. Even if we think the usage will only be short term temporary
> workaround, short term hacks in QEMU have a habit of living for years longer
> than we expect.

I totally agree.  This would not be a short term workaround, I expect it
to stay for years; I don't expect the transition of the configure script
to Meson to be very short.

This can be both a pro and a con.  On one hand it means this is "more
magic" that is here to stay.  On the other hand it makes transition
patches smaller and easier to review.

I agree that in principle all scripting should be Python.  On the other
hand Perl is quite suitable for "configure tasks that are too
complicated for shell" and are not going to be rewritten in Meson down
the road.  And I say that as someone who has written exactly 3 serious
Perl programs in his life, two of which are part of QEMU. :)

In this case shell scripting could also be a plausible alternative; I
didn't try only because I first tried the obvious choice of Python.

Or we can just decide that an incomplete help below a "please use
--python" error is good enough.

>> Opinions are welcome on whether this is worthwhile and how to solve
>> the above doubts.
> 
> IIUC, the motivation of this series is just to remove some duplication
> of defaults / args, nothing more than that ?

Yes, though it's quite some duplication: right now an option needs

- a default in configure

- command line parsing in configure

- a Meson -D argument in configure

- an help message in configure

- a declaration (including default and help) in meson_options.txt

so this patch reduces an option from 8-9 lines to 2.

There are a couple secondary goals too.  First, making sure all the
policy is in meson_options.txt and not configure; this avoids surprises
when/if we can start invoking meson directly.  Second, removing
variables completely from the configure script gives a better idea of
interdependencies between the configure tests.

Thanks,

Paolo

> I'd say we should focus effort on just converting more of configure
> to meson, as quickly as practical, and not try to add much more magic
> that's only relevant for the transition time.

Re: [PATCH 1/3] configure: quote command line arguments in config.status

[Eric Blake]

On 9/13/20 5:05 AM, Paolo Bonzini wrote:
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>   configure | 6 +++++-
>   1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/configure b/configure
> index 53723ace57..beae010e39 100755
> --- a/configure
> +++ b/configure
> @@ -89,6 +89,10 @@ printf " '%s'" "$0" "$@" >> config.log
>   echo >> config.log
>   echo "#" >> config.log
>   
> +quote_sh() {
> +    printf "'%s'" "$(echo "$1" | sed "s,','\\',")"

This is unsafe if $1 starts with - or contains \.  Better is using 
printf.  It also eats any trailing newlines in $1, although that may be 
less of a concern.

> +}
> +
>   print_error() {
>       (echo
>       echo "ERROR: $1"
> @@ -8061,7 +8065,7 @@ preserve_env WINDRES
>   
>   printf "exec" >>config.status
>   for i in "$0" "$@"; do
> -  test "$i" = --skip-meson || printf " '%s'" "$i" >>config.status
> +  test "$i" = --skip-meson || printf " %s" "$(quote_sh $i)" >>config.status

And this unquoted use of $i is wrong.

>   done
>   echo ' "$@"' >>config.status
>   chmod +x config.status
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH 3/3] configure: automatically parse command line for meson -D options

[Eric Blake]

On 9/13/20 5:05 AM, Paolo Bonzini wrote:
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>   Makefile                                |   6 ++
>   configure                               |  71 +++++--------
>   docs/devel/build-system.rst             |  35 +------
>   meson-buildoptions.json                 | 130 ++++++++++++++++++++++++
>   scripts/configure-parse-buildoptions.py |  94 +++++++++++++++++
>   5 files changed, 258 insertions(+), 78 deletions(-)
>   create mode 100644 meson-buildoptions.json
>   create mode 100644 scripts/configure-parse-buildoptions.py
> 
> diff --git a/Makefile b/Makefile
> index d6c5c9fdef..b22e5b272e 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -80,6 +80,12 @@ ifneq ($(MESON),)
>   Makefile.mtest: build.ninja scripts/mtest2make.py
>   	$(MESON) introspect --tests --benchmarks | $(PYTHON) scripts/mtest2make.py > $@
>   -include Makefile.mtest
> +
> +# jq is used to keep the result stable
> +meson-buildoptions.json: meson_options.txt
> +	$(MESON) introspect --buildoptions . | \
> +	    jq 'map(select(.section == "user")) | sort_by(.name)' \
> +	    > $(SRC_PATH)/meson-buildoptions.json

Are we guaranteed to have jq on the system?  This appears to be the 
first use of it in qemu.git.

> @@ -108,6 +78,11 @@ In meson.build::
>     # Summary
>     summary_info += {'SDL image support': sdl_image.found()}
>   
> +The configure script automatically converts command line options to
> +Meson options as long as the `meson-buildoptions.json` file in the
> +root source directory is up-to-date.  You can simply type
> +`make meson-buildoptions.json` to update it; the `jq` program
> +is needed.

Oh, you're using it for an optional step, that most people don't have to 
run, but where a developer needing to regenerate things can be likely to 
understand how to get it.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On Mon, Sep 14, 2020 at 06:00:11PM +0800, 罗勇刚(Yonggang Luo) wrote:
> On Mon, Sep 14, 2020 at 5:58 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >
> > On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote:
> > > - because we parse command-line options before meson is available,
> > > the introspection output is stored in the source tree.  The output
> > > is slightly modified using the "jq" tool in order to ensure that it's
> > > stable and that modifications to meson_buildoptions.txt do not cause
> > > horrible conflicts.  This is the main reason for the unattractive
> > > diffstat (the number of JSON lines added is higher than the number
> > > of configure lines removed, though of course the latter are code
> > > that must be maintained manually and the former is not).
> >
> > Does this add a build dependency on jq? Is it possible to use a Python
> > script or even a command-line one-liner instead?
> What's jq stands for, is that a perl script?

https://stedolan.github.io/jq/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On Mon, Sep 14, 2020 at 12:27:59PM +0200, Paolo Bonzini wrote:
> On 14/09/20 11:57, Stefan Hajnoczi wrote:
> > On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote:
> >> - because we parse command-line options before meson is available,
> >> the introspection output is stored in the source tree.  The output
> >> is slightly modified using the "jq" tool in order to ensure that it's
> >> stable and that modifications to meson_buildoptions.txt do not cause
> >> horrible conflicts.  This is the main reason for the unattractive
> >> diffstat (the number of JSON lines added is higher than the number
> >> of configure lines removed, though of course the latter are code
> >> that must be maintained manually and the former is not).
> > 
> > Does this add a build dependency on jq? Is it possible to use a Python
> > script or even a command-line one-liner instead?
> 
> No, only developers need jq and only if they add configure options.
> Using Python would certainly be possible, though probably it wouldn't be
> a one-liner as for jq.

I see. Avoiding the dependency would be nice but only if the alternative
is reasonable.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options

[Paolo Bonzini]

On 16/09/20 13:25, Stefan Hajnoczi wrote:
>> No, only developers need jq and only if they add configure options.
>> Using Python would certainly be possible, though probably it wouldn't be
>> a one-liner as for jq.
> I see. Avoiding the dependency would be nice but only if the alternative
> is reasonable.

I guess this counts as reasonable:
 
python3 -c 'import json, sys; print(
  json.dumps(sorted([x for x in json.loads(sys.stdin.read()) 
                    if x["section"] == "user"], key=lambda x: x["name"]),
             indent=2))'

Paolo