* [RFC PATCH 0/3] Automatically convert configure options to meson build options
@ 2020-09-13 10:05 Paolo Bonzini
2020-09-13 10:05 ` [PATCH 1/3] configure: quote command line arguments in config.status Paolo Bonzini
` (5 more replies)
0 siblings, 6 replies; 16+ messages in thread
From: Paolo Bonzini @ 2020-09-13 10:05 UTC (permalink / raw)
To: qemu-devel; +Cc: marcandre.lureau, berrange, peter.maydell
Right now meson_options.txt lists less than a dozen options, but about
40 more could come as configure tests are converted and moved to
meson.build. Each option needs code in configure to parse it and pass
the option down to Meson as a -D command-line argument; in addition the
default must be duplicated between configure and meson_options.txt.
This series tries to remove the code duplication by passing unknown
--enable and --disable options to a Python program, and using
Meson's introspection support to match those to meson_options.txt
The disadvantages are:
- because we parse command-line options before meson is available,
the introspection output is stored in the source tree. The output
is slightly modified using the "jq" tool in order to ensure that it's
stable and that modifications to meson_buildoptions.txt do not cause
horrible conflicts. This is the main reason for the unattractive
diffstat (the number of JSON lines added is higher than the number
of configure lines removed, though of course the latter are code
that must be maintained manually and the former is not).
- we now need Python to generate the full help, so if Python is
missing we can only print a partial message and direct the user
to specify the interpreter with --python. It would be possible to fix
this by rewriting the script in Perl (at least on Fedora, JSON::PP is
always installed if Perl is, because it's a dependency for CPAN; I'd
have to check Ubuntu and the BSDs), or if we want to write it as a
Bourne shell script, to further massage the introspection output into
for example TAB-separated values.
Opinions are welcome on whether this is worthwhile and how to solve
the above doubts.
Paolo
Paolo Bonzini (3):
configure: quote command line arguments in config.status
configure: early test for Python
configure: automatically parse command line for meson -D options
Makefile | 6 ++
configure | 107 ++++++++-----------
docs/devel/build-system.rst | 35 +------
meson-buildoptions.json | 130 ++++++++++++++++++++++++
scripts/configure-parse-buildoptions.py | 94 +++++++++++++++++
5 files changed, 280 insertions(+), 92 deletions(-)
create mode 100644 meson-buildoptions.json
create mode 100644 scripts/configure-parse-buildoptions.py
--
2.26.2
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH 1/3] configure: quote command line arguments in config.status 2020-09-13 10:05 [RFC PATCH 0/3] Automatically convert configure options to meson build options Paolo Bonzini @ 2020-09-13 10:05 ` Paolo Bonzini 2020-09-14 19:17 ` Eric Blake 2020-09-13 10:05 ` [PATCH 2/3] configure: early test for Python Paolo Bonzini ` (4 subsequent siblings) 5 siblings, 1 reply; 16+ messages in thread From: Paolo Bonzini @ 2020-09-13 10:05 UTC (permalink / raw) To: qemu-devel; +Cc: marcandre.lureau, berrange, peter.maydell Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> --- configure | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/configure b/configure index 53723ace57..beae010e39 100755 --- a/configure +++ b/configure @@ -89,6 +89,10 @@ printf " '%s'" "$0" "$@" >> config.log echo >> config.log echo "#" >> config.log +quote_sh() { + printf "'%s'" "$(echo "$1" | sed "s,','\\',")" +} + print_error() { (echo echo "ERROR: $1" @@ -8061,7 +8065,7 @@ preserve_env WINDRES printf "exec" >>config.status for i in "$0" "$@"; do - test "$i" = --skip-meson || printf " '%s'" "$i" >>config.status + test "$i" = --skip-meson || printf " %s" "$(quote_sh $i)" >>config.status done echo ' "$@"' >>config.status chmod +x config.status -- 2.26.2 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 1/3] configure: quote command line arguments in config.status 2020-09-13 10:05 ` [PATCH 1/3] configure: quote command line arguments in config.status Paolo Bonzini @ 2020-09-14 19:17 ` Eric Blake 0 siblings, 0 replies; 16+ messages in thread From: Eric Blake @ 2020-09-14 19:17 UTC (permalink / raw) To: Paolo Bonzini, qemu-devel; +Cc: marcandre.lureau, berrange, peter.maydell On 9/13/20 5:05 AM, Paolo Bonzini wrote: > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> > --- > configure | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/configure b/configure > index 53723ace57..beae010e39 100755 > --- a/configure > +++ b/configure > @@ -89,6 +89,10 @@ printf " '%s'" "$0" "$@" >> config.log > echo >> config.log > echo "#" >> config.log > > +quote_sh() { > + printf "'%s'" "$(echo "$1" | sed "s,','\\',")" This is unsafe if $1 starts with - or contains \. Better is using printf. It also eats any trailing newlines in $1, although that may be less of a concern. > +} > + > print_error() { > (echo > echo "ERROR: $1" > @@ -8061,7 +8065,7 @@ preserve_env WINDRES > > printf "exec" >>config.status > for i in "$0" "$@"; do > - test "$i" = --skip-meson || printf " '%s'" "$i" >>config.status > + test "$i" = --skip-meson || printf " %s" "$(quote_sh $i)" >>config.status And this unquoted use of $i is wrong. > done > echo ' "$@"' >>config.status > chmod +x config.status > -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 2/3] configure: early test for Python 2020-09-13 10:05 [RFC PATCH 0/3] Automatically convert configure options to meson build options Paolo Bonzini 2020-09-13 10:05 ` [PATCH 1/3] configure: quote command line arguments in config.status Paolo Bonzini @ 2020-09-13 10:05 ` Paolo Bonzini 2020-09-13 10:05 ` [PATCH 3/3] configure: automatically parse command line for meson -D options Paolo Bonzini ` (3 subsequent siblings) 5 siblings, 0 replies; 16+ messages in thread From: Paolo Bonzini @ 2020-09-13 10:05 UTC (permalink / raw) To: qemu-devel; +Cc: marcandre.lureau, berrange, peter.maydell --- configure | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/configure b/configure index beae010e39..b4f5d2e272 100755 --- a/configure +++ b/configure @@ -1631,6 +1631,23 @@ for opt do esac done +if test -n "$python" +then + # Preserve python version since some functionality is dependent on it + python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0], sys.version_info[1], sys.version_info[2]))' 2>/dev/null) + + # Suppress writing compiled files + python="$python -B" + + # Note that if the Python conditional here evaluates True we will exit + # with status 1 which is a shell 'false' value. + if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then + print_error "Cannot use '$python', Python >= 3.5 is required." \ + "Use --python=/path/to/python to specify a supported Python." + python= + fi +fi + firmwarepath="${firmwarepath:-$prefix/share/qemu-firmware}" libdir="${libdir:-$prefix/lib}" libexecdir="${libexecdir:-$prefix/libexec}" @@ -1962,19 +1979,6 @@ then error_exit "Python not found. Use --python=/path/to/python" fi -# Note that if the Python conditional here evaluates True we will exit -# with status 1 which is a shell 'false' value. -if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then - error_exit "Cannot use '$python', Python >= 3.5 is required." \ - "Use --python=/path/to/python to specify a supported Python." -fi - -# Preserve python version since some functionality is dependent on it -python_version=$($python -c 'import sys; print("%d.%d.%d" % (sys.version_info[0], sys.version_info[1], sys.version_info[2]))' 2>/dev/null) - -# Suppress writing compiled files -python="$python -B" - if test -z "$meson"; then if test "$explicit_python" = no && has meson && version_ge "$(meson --version)" 0.55.1; then meson=meson -- 2.26.2 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 3/3] configure: automatically parse command line for meson -D options 2020-09-13 10:05 [RFC PATCH 0/3] Automatically convert configure options to meson build options Paolo Bonzini 2020-09-13 10:05 ` [PATCH 1/3] configure: quote command line arguments in config.status Paolo Bonzini 2020-09-13 10:05 ` [PATCH 2/3] configure: early test for Python Paolo Bonzini @ 2020-09-13 10:05 ` Paolo Bonzini 2020-09-14 19:20 ` Eric Blake 2020-09-13 10:15 ` [RFC PATCH 0/3] Automatically convert configure options to meson build options 罗勇刚(Yonggang Luo) ` (2 subsequent siblings) 5 siblings, 1 reply; 16+ messages in thread From: Paolo Bonzini @ 2020-09-13 10:05 UTC (permalink / raw) To: qemu-devel; +Cc: marcandre.lureau, berrange, peter.maydell Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> --- Makefile | 6 ++ configure | 71 +++++-------- docs/devel/build-system.rst | 35 +------ meson-buildoptions.json | 130 ++++++++++++++++++++++++ scripts/configure-parse-buildoptions.py | 94 +++++++++++++++++ 5 files changed, 258 insertions(+), 78 deletions(-) create mode 100644 meson-buildoptions.json create mode 100644 scripts/configure-parse-buildoptions.py diff --git a/Makefile b/Makefile index d6c5c9fdef..b22e5b272e 100644 --- a/Makefile +++ b/Makefile @@ -80,6 +80,12 @@ ifneq ($(MESON),) Makefile.mtest: build.ninja scripts/mtest2make.py $(MESON) introspect --tests --benchmarks | $(PYTHON) scripts/mtest2make.py > $@ -include Makefile.mtest + +# jq is used to keep the result stable +meson-buildoptions.json: meson_options.txt + $(MESON) introspect --buildoptions . | \ + jq 'map(select(.section == "user")) | sort_by(.name)' \ + > $(SRC_PATH)/meson-buildoptions.json endif Makefile: .git-submodule-status diff --git a/configure b/configure index b4f5d2e272..54958bfc17 100755 --- a/configure +++ b/configure @@ -396,16 +396,10 @@ docs="" fdt="" netmap="no" sdl="auto" -sdl_image="auto" virtfs="" mpath="" -vnc="enabled" sparse="no" vde="" -vnc_sasl="auto" -vnc_jpeg="auto" -vnc_png="auto" -xkbcommon="auto" xen="" xen_ctrl_version="" xen_pci_passthrough="" @@ -466,7 +460,6 @@ trace_file="trace" spice="" rbd="" smartcard="" -u2f="auto" libusb="" usb_redir="" opengl="" @@ -539,6 +532,7 @@ rng_none="no" secret_keyring="" libdaxctl="" meson="" +meson_args="" ninja="" skip_meson=no gettext="" @@ -1100,10 +1094,6 @@ for opt do ;; --enable-sdl) sdl="enabled" ;; - --disable-sdl-image) sdl_image="disabled" - ;; - --enable-sdl-image) sdl_image="enabled" - ;; --disable-qom-cast-debug) qom_cast_debug="no" ;; --enable-qom-cast-debug) qom_cast_debug="yes" @@ -1116,10 +1106,6 @@ for opt do ;; --enable-mpath) mpath="yes" ;; - --disable-vnc) vnc="disabled" - ;; - --enable-vnc) vnc="enabled" - ;; --disable-gettext) gettext="false" ;; --enable-gettext) gettext="true" @@ -1158,18 +1144,6 @@ for opt do ;; --disable-strip) strip_opt="no" ;; - --disable-vnc-sasl) vnc_sasl="disabled" - ;; - --enable-vnc-sasl) vnc_sasl="enabled" - ;; - --disable-vnc-jpeg) vnc_jpeg="disabled" - ;; - --enable-vnc-jpeg) vnc_jpeg="enabled" - ;; - --disable-vnc-png) vnc_png="disabled" - ;; - --enable-vnc-png) vnc_png="enabled" - ;; --disable-slirp) slirp="no" ;; --enable-slirp=git) slirp="git" @@ -1369,10 +1343,6 @@ for opt do ;; --enable-smartcard) smartcard="yes" ;; - --disable-u2f) u2f="disabled" - ;; - --enable-u2f) u2f="enabled" - ;; --disable-libusb) libusb="no" ;; --enable-libusb) libusb="yes" @@ -1593,10 +1563,6 @@ for opt do ;; --disable-libpmem) libpmem=no ;; - --enable-xkbcommon) xkbcommon="enabled" - ;; - --disable-xkbcommon) xkbcommon="disabled" - ;; --enable-plugins) plugins="yes" ;; --disable-plugins) plugins="no" @@ -1623,6 +1589,9 @@ for opt do ;; --disable-libdaxctl) libdaxctl=no ;; + --enable-* | --disable-*) + meson_args="$meson_args $(quote_sh "$opt")" + ;; *) echo "ERROR: unknown option $opt" echo "Try '$0 --help' for more information" @@ -1648,6 +1617,16 @@ then fi fi +# Convert unknown --enable/--disable arguments to Meson build options. +meson_options=$(eval \ + \$python \ + \$source_path/scripts/configure-parse-buildoptions.py \ + \< \$source_path/meson-buildoptions.json $meson_args) +if test $? = 1; then + echo "Try '$0 --help' for more information" + exit 1 +fi + firmwarepath="${firmwarepath:-$prefix/share/qemu-firmware}" libdir="${libdir:-$prefix/lib}" libexecdir="${libexecdir:-$prefix/libexec}" @@ -1853,10 +1831,14 @@ Advanced options (experts only): enable plugins via shared library loading --disable-containers don't use containers for cross-building --gdb=GDB-path gdb to use for gdbstub tests [$gdb_bin] - -Optional features, enabled with --enable-FEATURE and -disabled with --disable-FEATURE, default is enabled if available: - +EOF + if test -z "$python"; then + echo + echo "Python not found. Use --python=/path/to/python to see complete help" + exit 0 + fi + $python $source_path/scripts/configure-parse-buildoptions.py --print-help < $source_path/meson-buildoptions.json +cat << EOF system all system emulation targets user supported user emulation targets linux-user all linux usermode emulation targets @@ -1883,10 +1865,6 @@ disabled with --disable-FEATURE, default is enabled if available: vte vte support for the gtk UI curses curses UI iconv font glyph conversion support - vnc VNC UI support - vnc-sasl SASL encryption for VNC server - vnc-jpeg JPEG lossy compression for VNC server - vnc-png PNG compression for VNC server cocoa Cocoa UI (Mac OS X only) virtfs VirtFS mpath Multipath persistent reservation passthrough @@ -1920,7 +1898,6 @@ disabled with --disable-FEATURE, default is enabled if available: libiscsi iscsi support libnfs nfs support smartcard smartcard support (libcacard) - u2f U2F support (u2f-emu) libusb libusb (for usb passthrough) live-block-migration Block migration in the main migration stream usb-redir usb network redirection support @@ -1962,7 +1939,6 @@ disabled with --disable-FEATURE, default is enabled if available: capstone capstone disassembler support debug-mutex mutex debugging support libpmem libpmem support - xkbcommon xkbcommon support rng-none dummy RNG, avoid using /dev/(u)random and getrandom() libdaxctl libdaxctl support @@ -8008,9 +7984,8 @@ NINJA=${ninja:-$PWD/ninjatool} $meson setup \ -Dstrip=$(if test "$strip_opt" = yes; then echo true; else echo false; fi) \ -Db_pie=$(if test "$pie" = yes; then echo true; else echo false; fi) \ -Db_coverage=$(if test "$gcov" = yes; then echo true; else echo false; fi) \ - -Dsdl=$sdl -Dsdl_image=$sdl_image \ - -Dvnc=$vnc -Dvnc_sasl=$vnc_sasl -Dvnc_jpeg=$vnc_jpeg -Dvnc_png=$vnc_png \ - -Dgettext=$gettext -Dxkbcommon=$xkbcommon -Du2f=$u2f\ + $meson_args \ + -Dsdl=$sdl -Dgettext=$gettext \ $cross_arg \ "$PWD" "$source_path" diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst index 08e85c69e1..9fd321f227 100644 --- a/docs/devel/build-system.rst +++ b/docs/devel/build-system.rst @@ -52,12 +52,6 @@ following tasks: - Add a Meson build option to meson_options.txt. - - Add support to the command line arg parser to handle any new - `--enable-XXX`/`--disable-XXX` flags required by the feature. - - - Add information to the help output message to report on the new - feature flag. - - Add code to perform the actual feature check. - Add code to include the feature status in `config-host.h` @@ -66,30 +60,6 @@ following tasks: upon completion. -Taking the probe for SDL2_Image as an example, we have the following pieces -in configure:: - - # Initial variable state - sdl_image=auto - - ..snip.. - - # Configure flag processing - --disable-sdl-image) sdl_image=disabled - ;; - --enable-sdl-image) sdl_image=enabled - ;; - - ..snip.. - - # Help output feature message - sdl-image SDL Image support for icons - - ..snip.. - - # Meson invocation - -Dsdl_image=$sdl_image - In meson_options.txt:: option('sdl', type : 'feature', value : 'auto', @@ -108,6 +78,11 @@ In meson.build:: # Summary summary_info += {'SDL image support': sdl_image.found()} +The configure script automatically converts command line options to +Meson options as long as the `meson-buildoptions.json` file in the +root source directory is up-to-date. You can simply type +`make meson-buildoptions.json` to update it; the `jq` program +is needed. Helper functions diff --git a/meson-buildoptions.json b/meson-buildoptions.json new file mode 100644 index 0000000000..dbaf5bc71e --- /dev/null +++ b/meson-buildoptions.json @@ -0,0 +1,130 @@ +[ + { + "name": "docdir", + "value": "doc", + "section": "user", + "machine": "any", + "type": "string", + "description": "Base directory for documentation installation (can be empty)" + }, + { + "name": "gettext", + "value": true, + "section": "user", + "machine": "any", + "type": "boolean", + "description": "Localization of the GTK+ user interface" + }, + { + "name": "qemu_suffix", + "value": "qemu", + "section": "user", + "machine": "any", + "type": "string", + "description": "Suffix for QEMU data/modules/config directories (can be empty)" + }, + { + "name": "sdl", + "value": "auto", + "section": "user", + "machine": "any", + "choices": [ + "enabled", + "disabled", + "auto" + ], + "type": "combo", + "description": "SDL user interface" + }, + { + "name": "sdl_image", + "value": "auto", + "section": "user", + "machine": "any", + "choices": [ + "enabled", + "disabled", + "auto" + ], + "type": "combo", + "description": "SDL Image support for icons" + }, + { + "name": "u2f", + "value": "auto", + "section": "user", + "machine": "any", + "choices": [ + "enabled", + "disabled", + "auto" + ], + "type": "combo", + "description": "U2F emulation support" + }, + { + "name": "vnc", + "value": "enabled", + "section": "user", + "machine": "any", + "choices": [ + "enabled", + "disabled", + "auto" + ], + "type": "combo", + "description": "VNC server" + }, + { + "name": "vnc_jpeg", + "value": "auto", + "section": "user", + "machine": "any", + "choices": [ + "enabled", + "disabled", + "auto" + ], + "type": "combo", + "description": "JPEG lossy compression for VNC server" + }, + { + "name": "vnc_png", + "value": "auto", + "section": "user", + "machine": "any", + "choices": [ + "enabled", + "disabled", + "auto" + ], + "type": "combo", + "description": "PNG compression for VNC server" + }, + { + "name": "vnc_sasl", + "value": "auto", + "section": "user", + "machine": "any", + "choices": [ + "enabled", + "disabled", + "auto" + ], + "type": "combo", + "description": "SASL authentication for VNC server" + }, + { + "name": "xkbcommon", + "value": "auto", + "section": "user", + "machine": "any", + "choices": [ + "enabled", + "disabled", + "auto" + ], + "type": "combo", + "description": "xkbcommon support" + } +] diff --git a/scripts/configure-parse-buildoptions.py b/scripts/configure-parse-buildoptions.py new file mode 100644 index 0000000000..4171312d75 --- /dev/null +++ b/scripts/configure-parse-buildoptions.py @@ -0,0 +1,94 @@ +#! /usr/bin/env python3 + +# Parse configure command line options based on Meson's user build options +# introspection data (passed on stdin). +# +# Copyright (C) 2020 Red Hat, Inc. +# +# Author: Paolo Bonzini <pbonzini@redhat.com> + +import json +import re +import shlex +import sys +import textwrap + +SKIP_OPTIONS = [ 'docdir', 'qemu_suffix' ] +FEATURE_CHOICES = [ 'auto', 'disabled', 'enabled' ] + +options = { x['name']: x + for x in json.load(sys.stdin) + if x['section'] == 'user' and x['name'] not in SKIP_OPTIONS } + +for x in options.values(): + if x['type'] == 'combo' and sorted(x['choices']) == FEATURE_CHOICES: + x['type'] = 'feature' + +def value_to_help(x): + if x == True: + return 'enabled' + if x == False: + return 'disabled' + return str(x) + +def print_help_line(key, opt, indent): + key = ' ' + key + value = '%s [%s]' % (opt['description'], value_to_help(opt['value'])) + if len(key) >= indent: + print(key) + key = '' + spaces = ' ' * indent + key = (key + spaces)[0:indent] + print(textwrap.fill(value, initial_indent=key, subsequent_indent=spaces)) + +def print_help(): + for o, x in options.items(): + if x['type'] not in ('boolean', 'feature'): + print_help_line('--enable-' + o, x, 24) + + print() + print('Optional features, enabled with --enable-FEATURE and') + print('disabled with --disable-FEATURE:') + for o, x in options.items(): + if x['type'] in ('boolean', 'feature'): + print_help_line(o, x, 18) + +def error(s, *args): + print('ERROR:', s % args, file=sys.stderr) + sys.exit(1) + +def main(argv): + if not argv: + return + + if argv[0] == '--print-help': + print_help() + return + + args = [] + for arg in sys.argv[1:]: + m = re.search('--(enable|disable)-([^=]*)(?:=(.*))?', arg) + if not m: + error('internal error parsing command line') + opt = m.group(2).replace('-', '_') + if opt not in options: + error('Unknown option --%s-%s', m.group(1), m.group(2)) + opt_type = options[opt]['type'] + if opt_type in ('boolean', 'feature'): + if m.group(3) is not None: + error('option --%s-%s does not take an argument', m.group(1), m.group(2)) + if opt_type == 'feature': + value = m.group(1) + 'd' + else: + value = 'true' if m.group(1) == 'enable' else 'false' + else: + if m.group(1) == 'disable': + error('Unknown option --disable-%s', m.group(2)) + if m.group(3) is None: + error('option --enable-%s takes an argument', m.group(2)) + + args.append(shlex.quote('-D%s=%s' % (opt, value))) + print(' '.join(args)) + +if __name__ == "__main__": + main(sys.argv[1:]) -- 2.26.2 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 3/3] configure: automatically parse command line for meson -D options 2020-09-13 10:05 ` [PATCH 3/3] configure: automatically parse command line for meson -D options Paolo Bonzini @ 2020-09-14 19:20 ` Eric Blake 0 siblings, 0 replies; 16+ messages in thread From: Eric Blake @ 2020-09-14 19:20 UTC (permalink / raw) To: Paolo Bonzini, qemu-devel; +Cc: marcandre.lureau, berrange, peter.maydell On 9/13/20 5:05 AM, Paolo Bonzini wrote: > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> > --- > Makefile | 6 ++ > configure | 71 +++++-------- > docs/devel/build-system.rst | 35 +------ > meson-buildoptions.json | 130 ++++++++++++++++++++++++ > scripts/configure-parse-buildoptions.py | 94 +++++++++++++++++ > 5 files changed, 258 insertions(+), 78 deletions(-) > create mode 100644 meson-buildoptions.json > create mode 100644 scripts/configure-parse-buildoptions.py > > diff --git a/Makefile b/Makefile > index d6c5c9fdef..b22e5b272e 100644 > --- a/Makefile > +++ b/Makefile > @@ -80,6 +80,12 @@ ifneq ($(MESON),) > Makefile.mtest: build.ninja scripts/mtest2make.py > $(MESON) introspect --tests --benchmarks | $(PYTHON) scripts/mtest2make.py > $@ > -include Makefile.mtest > + > +# jq is used to keep the result stable > +meson-buildoptions.json: meson_options.txt > + $(MESON) introspect --buildoptions . | \ > + jq 'map(select(.section == "user")) | sort_by(.name)' \ > + > $(SRC_PATH)/meson-buildoptions.json Are we guaranteed to have jq on the system? This appears to be the first use of it in qemu.git. > @@ -108,6 +78,11 @@ In meson.build:: > # Summary > summary_info += {'SDL image support': sdl_image.found()} > > +The configure script automatically converts command line options to > +Meson options as long as the `meson-buildoptions.json` file in the > +root source directory is up-to-date. You can simply type > +`make meson-buildoptions.json` to update it; the `jq` program > +is needed. Oh, you're using it for an optional step, that most people don't have to run, but where a developer needing to regenerate things can be likely to understand how to get it. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-13 10:05 [RFC PATCH 0/3] Automatically convert configure options to meson build options Paolo Bonzini ` (2 preceding siblings ...) 2020-09-13 10:05 ` [PATCH 3/3] configure: automatically parse command line for meson -D options Paolo Bonzini @ 2020-09-13 10:15 ` 罗勇刚(Yonggang Luo) 2020-09-13 13:57 ` Paolo Bonzini 2020-09-14 9:12 ` Daniel P. Berrangé 2020-09-14 9:57 ` Stefan Hajnoczi 5 siblings, 1 reply; 16+ messages in thread From: 罗勇刚(Yonggang Luo) @ 2020-09-13 10:15 UTC (permalink / raw) To: Paolo Bonzini Cc: Marc-André Lureau, Daniel P. Berrangé, qemu-level, Peter Maydell [-- Attachment #1: Type: text/plain, Size: 3060 bytes --] On Sun, Sep 13, 2020 at 6:07 PM Paolo Bonzini <pbonzini@redhat.com> wrote: > Right now meson_options.txt lists less than a dozen options, but about > 40 more could come as configure tests are converted and moved to > meson.build. Each option needs code in configure to parse it and pass > the option down to Meson as a -D command-line argument; in addition the > default must be duplicated between configure and meson_options.txt. > > This series tries to remove the code duplication by passing unknown > --enable and --disable options to a Python program, and using > Meson's introspection support to match those to meson_options.txt > > The disadvantages are: > > - because we parse command-line options before meson is available, > the introspection output is stored in the source tree. The output > is slightly modified using the "jq" tool in order to ensure that it's > stable and that modifications to meson_buildoptions.txt do not cause > horrible conflicts. This is the main reason for the unattractive > diffstat (the number of JSON lines added is higher than the number > of configure lines removed, though of course the latter are code > that must be maintained manually and the former is not). > > - we now need Python to generate the full help, so if Python is > missing we can only print a partial message and direct the user > to specify the interpreter with --python. It would be possible to fix > this by rewriting the script in Perl (at least on Fedora, JSON::PP is > always installed if Perl is, because it's a dependency for CPAN; I'd > have to check Ubuntu and the BSDs), or if we want to write it as a > Bourne shell script, to further massage the introspection output into > for example TAB-separated values. > I am confusing about this? we were converting configure to meson, and python is a force dependencies, why we need rewrite the script in Perl? If we wanna build qemu, the first thing we need to install is python+meson, so there is no need convert to Perl or bash script. And perl/bash will incur msys2/mingw user, the ideal way is we only depends on python+meson to building qemu > > Opinions are welcome on whether this is worthwhile and how to solve > the above doubts. > > Paolo > > Paolo Bonzini (3): > configure: quote command line arguments in config.status > configure: early test for Python > configure: automatically parse command line for meson -D options > > Makefile | 6 ++ > configure | 107 ++++++++----------- > docs/devel/build-system.rst | 35 +------ > meson-buildoptions.json | 130 ++++++++++++++++++++++++ > scripts/configure-parse-buildoptions.py | 94 +++++++++++++++++ > 5 files changed, 280 insertions(+), 92 deletions(-) > create mode 100644 meson-buildoptions.json > create mode 100644 scripts/configure-parse-buildoptions.py > > -- > 2.26.2 > > > -- 此致 礼 罗勇刚 Yours sincerely, Yonggang Luo [-- Attachment #2: Type: text/html, Size: 3830 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-13 10:15 ` [RFC PATCH 0/3] Automatically convert configure options to meson build options 罗勇刚(Yonggang Luo) @ 2020-09-13 13:57 ` Paolo Bonzini 0 siblings, 0 replies; 16+ messages in thread From: Paolo Bonzini @ 2020-09-13 13:57 UTC (permalink / raw) To: luoyonggang Cc: Marc-André Lureau, Daniel P. Berrangé, qemu-level, Peter Maydell On 13/09/20 12:15, 罗勇刚(Yonggang Luo) wrote: > > I am confusing about this? we were converting configure to meson, > and python is a force dependencies, why we need rewrite the script in > Perl? If we wanna build qemu, the first thing we need to install is > python+meson, so there is no need convert to Perl or bash script. And > perl/bash will incur msys2/mingw user, the ideal way is we only > depends on python+meson to building qemu The main issue is that the Python interpreter is any of python, python3, python3.5. Furthermore Meson's shebang python is not necessarily the first "python3" in the path. Perl is pretty much always "/usr/bin/env perl". Note that Perl is already needed to run "make check". When/if it will be possible to only build QEMU with python+meson there will be no configure script anymore, so Perl is a reasonable choice for parsing the configure command line options. It wasn't my first choice, in fact this series includes a Python implementation, but it does have some advantages. Paolo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-13 10:05 [RFC PATCH 0/3] Automatically convert configure options to meson build options Paolo Bonzini ` (3 preceding siblings ...) 2020-09-13 10:15 ` [RFC PATCH 0/3] Automatically convert configure options to meson build options 罗勇刚(Yonggang Luo) @ 2020-09-14 9:12 ` Daniel P. Berrangé 2020-09-14 10:49 ` Paolo Bonzini 2020-09-14 9:57 ` Stefan Hajnoczi 5 siblings, 1 reply; 16+ messages in thread From: Daniel P. Berrangé @ 2020-09-14 9:12 UTC (permalink / raw) To: Paolo Bonzini; +Cc: marcandre.lureau, qemu-devel, peter.maydell On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote: > Right now meson_options.txt lists less than a dozen options, but about > 40 more could come as configure tests are converted and moved to > meson.build. Each option needs code in configure to parse it and pass > the option down to Meson as a -D command-line argument; in addition the > default must be duplicated between configure and meson_options.txt. > > This series tries to remove the code duplication by passing unknown > --enable and --disable options to a Python program, and using > Meson's introspection support to match those to meson_options.txt > > The disadvantages are: > > - because we parse command-line options before meson is available, > the introspection output is stored in the source tree. The output > is slightly modified using the "jq" tool in order to ensure that it's > stable and that modifications to meson_buildoptions.txt do not cause > horrible conflicts. This is the main reason for the unattractive > diffstat (the number of JSON lines added is higher than the number > of configure lines removed, though of course the latter are code > that must be maintained manually and the former is not). > > - we now need Python to generate the full help, so if Python is > missing we can only print a partial message and direct the user > to specify the interpreter with --python. It would be possible to fix > this by rewriting the script in Perl (at least on Fedora, JSON::PP is > always installed if Perl is, because it's a dependency for CPAN; I'd > have to check Ubuntu and the BSDs), or if we want to write it as a > Bourne shell script, to further massage the introspection output into > for example TAB-separated values. IMHO we should stay as far away from Perl as possible, and I say this as someone who enjoys writing Perl scripts ! Perl is a significant barrier to entry for potential contributors, because it just doesn't have the wide knowledge base that it did in the past. You can expect most people to have some familiarity with Python, or be able to pick it up fairly easily in a way that just isn't possible in Perl. As for shell, we have a never ending stream of bugs due to the wide range of different shell impls which make portable coding very hard. In general I think our overall goal should be to focus on getting down to use of a single scripting language in QEMU, and that language clearly has to be python3. We shouldn't introduce any new usage of Perl or Shell in QEMU IMHO. Even if we think the usage will only be short term temporary workaround, short term hacks in QEMU have a habit of living for years longer than we expect. > Opinions are welcome on whether this is worthwhile and how to solve > the above doubts. IIUC, the motivation of this series is just to remove some duplication of defaults / args, nothing more than that ? If correct, then I'd say it probably isn't worth the hassle. I'd say we should focus effort on just converting more of configure to meson, as quickly as practical, and not try to add much more magic that's only relevant for the transition time. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-14 9:12 ` Daniel P. Berrangé @ 2020-09-14 10:49 ` Paolo Bonzini 0 siblings, 0 replies; 16+ messages in thread From: Paolo Bonzini @ 2020-09-14 10:49 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: marcandre.lureau, qemu-devel, peter.maydell On 14/09/20 11:12, Daniel P. Berrangé wrote: > IMHO we should stay as far away from Perl as possible, and I say this as > someone who enjoys writing Perl scripts ! Perl is a significant barrier > to entry for potential contributors, because it just doesn't have the > wide knowledge base that it did in the past. You can expect most people > to have some familiarity with Python, or be able to pick it up fairly > easily in a way that just isn't possible in Perl. > > As for shell, we have a never ending stream of bugs due to the wide > range of different shell impls which make portable coding very hard. > > In general I think our overall goal should be to focus on getting down to > use of a single scripting language in QEMU, and that language clearly has > to be python3. We shouldn't introduce any new usage of Perl or Shell > in QEMU IMHO. Even if we think the usage will only be short term temporary > workaround, short term hacks in QEMU have a habit of living for years longer > than we expect. I totally agree. This would not be a short term workaround, I expect it to stay for years; I don't expect the transition of the configure script to Meson to be very short. This can be both a pro and a con. On one hand it means this is "more magic" that is here to stay. On the other hand it makes transition patches smaller and easier to review. I agree that in principle all scripting should be Python. On the other hand Perl is quite suitable for "configure tasks that are too complicated for shell" and are not going to be rewritten in Meson down the road. And I say that as someone who has written exactly 3 serious Perl programs in his life, two of which are part of QEMU. :) In this case shell scripting could also be a plausible alternative; I didn't try only because I first tried the obvious choice of Python. Or we can just decide that an incomplete help below a "please use --python" error is good enough. >> Opinions are welcome on whether this is worthwhile and how to solve >> the above doubts. > > IIUC, the motivation of this series is just to remove some duplication > of defaults / args, nothing more than that ? Yes, though it's quite some duplication: right now an option needs - a default in configure - command line parsing in configure - a Meson -D argument in configure - an help message in configure - a declaration (including default and help) in meson_options.txt so this patch reduces an option from 8-9 lines to 2. There are a couple secondary goals too. First, making sure all the policy is in meson_options.txt and not configure; this avoids surprises when/if we can start invoking meson directly. Second, removing variables completely from the configure script gives a better idea of interdependencies between the configure tests. Thanks, Paolo > I'd say we should focus effort on just converting more of configure > to meson, as quickly as practical, and not try to add much more magic > that's only relevant for the transition time. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-13 10:05 [RFC PATCH 0/3] Automatically convert configure options to meson build options Paolo Bonzini ` (4 preceding siblings ...) 2020-09-14 9:12 ` Daniel P. Berrangé @ 2020-09-14 9:57 ` Stefan Hajnoczi 2020-09-14 10:00 ` 罗勇刚(Yonggang Luo) 2020-09-14 10:27 ` Paolo Bonzini 5 siblings, 2 replies; 16+ messages in thread From: Stefan Hajnoczi @ 2020-09-14 9:57 UTC (permalink / raw) To: Paolo Bonzini; +Cc: marcandre.lureau, berrange, qemu-devel, peter.maydell [-- Attachment #1: Type: text/plain, Size: 728 bytes --] On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote: > - because we parse command-line options before meson is available, > the introspection output is stored in the source tree. The output > is slightly modified using the "jq" tool in order to ensure that it's > stable and that modifications to meson_buildoptions.txt do not cause > horrible conflicts. This is the main reason for the unattractive > diffstat (the number of JSON lines added is higher than the number > of configure lines removed, though of course the latter are code > that must be maintained manually and the former is not). Does this add a build dependency on jq? Is it possible to use a Python script or even a command-line one-liner instead? [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-14 9:57 ` Stefan Hajnoczi @ 2020-09-14 10:00 ` 罗勇刚(Yonggang Luo) 2020-09-16 11:24 ` Stefan Hajnoczi 2020-09-14 10:27 ` Paolo Bonzini 1 sibling, 1 reply; 16+ messages in thread From: 罗勇刚(Yonggang Luo) @ 2020-09-14 10:00 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Peter Maydell, Paolo Bonzini, Daniel P. Berrangé, qemu-level, Marc-André Lureau [-- Attachment #1: Type: text/plain, Size: 969 bytes --] On Mon, Sep 14, 2020 at 5:58 PM Stefan Hajnoczi <stefanha@gmail.com> wrote: > > On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote: > > - because we parse command-line options before meson is available, > > the introspection output is stored in the source tree. The output > > is slightly modified using the "jq" tool in order to ensure that it's > > stable and that modifications to meson_buildoptions.txt do not cause > > horrible conflicts. This is the main reason for the unattractive > > diffstat (the number of JSON lines added is higher than the number > > of configure lines removed, though of course the latter are code > > that must be maintained manually and the former is not). > > Does this add a build dependency on jq? Is it possible to use a Python > script or even a command-line one-liner instead? What's jq stands for, is that a perl script? -- 此致 礼 罗勇刚 Yours sincerely, Yonggang Luo [-- Attachment #2: Type: text/html, Size: 1189 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-14 10:00 ` 罗勇刚(Yonggang Luo) @ 2020-09-16 11:24 ` Stefan Hajnoczi 0 siblings, 0 replies; 16+ messages in thread From: Stefan Hajnoczi @ 2020-09-16 11:24 UTC (permalink / raw) To: 罗勇刚(Yonggang Luo) Cc: Peter Maydell, Paolo Bonzini, Daniel P. Berrangé, qemu-level, Marc-André Lureau [-- Attachment #1: Type: text/plain, Size: 1027 bytes --] On Mon, Sep 14, 2020 at 06:00:11PM +0800, 罗勇刚(Yonggang Luo) wrote: > On Mon, Sep 14, 2020 at 5:58 PM Stefan Hajnoczi <stefanha@gmail.com> wrote: > > > > On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote: > > > - because we parse command-line options before meson is available, > > > the introspection output is stored in the source tree. The output > > > is slightly modified using the "jq" tool in order to ensure that it's > > > stable and that modifications to meson_buildoptions.txt do not cause > > > horrible conflicts. This is the main reason for the unattractive > > > diffstat (the number of JSON lines added is higher than the number > > > of configure lines removed, though of course the latter are code > > > that must be maintained manually and the former is not). > > > > Does this add a build dependency on jq? Is it possible to use a Python > > script or even a command-line one-liner instead? > What's jq stands for, is that a perl script? https://stedolan.github.io/jq/ [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-14 9:57 ` Stefan Hajnoczi 2020-09-14 10:00 ` 罗勇刚(Yonggang Luo) @ 2020-09-14 10:27 ` Paolo Bonzini 2020-09-16 11:25 ` Stefan Hajnoczi 1 sibling, 1 reply; 16+ messages in thread From: Paolo Bonzini @ 2020-09-14 10:27 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: marcandre.lureau, berrange, qemu-devel, peter.maydell On 14/09/20 11:57, Stefan Hajnoczi wrote: > On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote: >> - because we parse command-line options before meson is available, >> the introspection output is stored in the source tree. The output >> is slightly modified using the "jq" tool in order to ensure that it's >> stable and that modifications to meson_buildoptions.txt do not cause >> horrible conflicts. This is the main reason for the unattractive >> diffstat (the number of JSON lines added is higher than the number >> of configure lines removed, though of course the latter are code >> that must be maintained manually and the former is not). > > Does this add a build dependency on jq? Is it possible to use a Python > script or even a command-line one-liner instead? No, only developers need jq and only if they add configure options. Using Python would certainly be possible, though probably it wouldn't be a one-liner as for jq. Paolo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-14 10:27 ` Paolo Bonzini @ 2020-09-16 11:25 ` Stefan Hajnoczi 2020-09-16 13:15 ` Paolo Bonzini 0 siblings, 1 reply; 16+ messages in thread From: Stefan Hajnoczi @ 2020-09-16 11:25 UTC (permalink / raw) To: Paolo Bonzini; +Cc: marcandre.lureau, berrange, qemu-devel, peter.maydell [-- Attachment #1: Type: text/plain, Size: 1168 bytes --] On Mon, Sep 14, 2020 at 12:27:59PM +0200, Paolo Bonzini wrote: > On 14/09/20 11:57, Stefan Hajnoczi wrote: > > On Sun, Sep 13, 2020 at 12:05:31PM +0200, Paolo Bonzini wrote: > >> - because we parse command-line options before meson is available, > >> the introspection output is stored in the source tree. The output > >> is slightly modified using the "jq" tool in order to ensure that it's > >> stable and that modifications to meson_buildoptions.txt do not cause > >> horrible conflicts. This is the main reason for the unattractive > >> diffstat (the number of JSON lines added is higher than the number > >> of configure lines removed, though of course the latter are code > >> that must be maintained manually and the former is not). > > > > Does this add a build dependency on jq? Is it possible to use a Python > > script or even a command-line one-liner instead? > > No, only developers need jq and only if they add configure options. > Using Python would certainly be possible, though probably it wouldn't be > a one-liner as for jq. I see. Avoiding the dependency would be nice but only if the alternative is reasonable. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH 0/3] Automatically convert configure options to meson build options 2020-09-16 11:25 ` Stefan Hajnoczi @ 2020-09-16 13:15 ` Paolo Bonzini 0 siblings, 0 replies; 16+ messages in thread From: Paolo Bonzini @ 2020-09-16 13:15 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: marcandre.lureau, berrange, qemu-devel, peter.maydell On 16/09/20 13:25, Stefan Hajnoczi wrote: >> No, only developers need jq and only if they add configure options. >> Using Python would certainly be possible, though probably it wouldn't be >> a one-liner as for jq. > I see. Avoiding the dependency would be nice but only if the alternative > is reasonable. I guess this counts as reasonable: python3 -c 'import json, sys; print( json.dumps(sorted([x for x in json.loads(sys.stdin.read()) if x["section"] == "user"], key=lambda x: x["name"]), indent=2))' Paolo ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2020-09-16 13:16 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-09-13 10:05 [RFC PATCH 0/3] Automatically convert configure options to meson build options Paolo Bonzini 2020-09-13 10:05 ` [PATCH 1/3] configure: quote command line arguments in config.status Paolo Bonzini 2020-09-14 19:17 ` Eric Blake 2020-09-13 10:05 ` [PATCH 2/3] configure: early test for Python Paolo Bonzini 2020-09-13 10:05 ` [PATCH 3/3] configure: automatically parse command line for meson -D options Paolo Bonzini 2020-09-14 19:20 ` Eric Blake 2020-09-13 10:15 ` [RFC PATCH 0/3] Automatically convert configure options to meson build options 罗勇刚(Yonggang Luo) 2020-09-13 13:57 ` Paolo Bonzini 2020-09-14 9:12 ` Daniel P. Berrangé 2020-09-14 10:49 ` Paolo Bonzini 2020-09-14 9:57 ` Stefan Hajnoczi 2020-09-14 10:00 ` 罗勇刚(Yonggang Luo) 2020-09-16 11:24 ` Stefan Hajnoczi 2020-09-14 10:27 ` Paolo Bonzini 2020-09-16 11:25 ` Stefan Hajnoczi 2020-09-16 13:15 ` Paolo Bonzini
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.