* [PATCH v4 00/46] qapi: static typing conversion, pt1
@ 2020-09-30 4:31 John Snow
2020-09-30 4:31 ` [PATCH v4 01/46] [DO-NOT-MERGE] docs: replace single backtick (`) with double-backtick (``) John Snow
` (45 more replies)
0 siblings, 46 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Based-on: <20200925162316.21205-1-peter.maydell@linaro.org>
Hi, this series adds static type hints to the QAPI module.
This is part one!
Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6
- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)
In general, this series tackles the cleanup of one individual QAPI
module at a time. Once it passes pylint or mypy checks, those checks are
enabled for that file.
Type hints are added in patches that add *only* type hints and change no
other behavior. Any necessary changes to behavior to accommodate typing
are split out into their own tiny patches.
Notes:
- `make sphinxdocs` should work on every commit. It begins to include
docstring content after the DO-NOT-MERGE patch 5.
- After patch 11, `isort -c` should pass 100% on this and every
future commit.
- After patch 12, `flake8 qapi/` should pass 100% on this and every
future commit.
- After patch 13, `pylint --rcfile=3Dqapi/pylintrc qapi/` should pass 100%
on this and every future commit.
- After patch 22, `mypy --config-file=3Dqapi/mypy.ini qapi/` should pass
100% on this and every future commit.
V4:
- Rebase on Peter Maydell's work;
- 05: Context differences from Peter Maydell's work
- 06: Remove qapi.doc
- 07: Remove qapi.doc, remove superfluous "if match"
- 09: Remove qapi.doc
- 13: remove qapi.doc
- 18: remove qapi.doc
- 22: remove qapi.doc
- 31: remove QAPIGenDoc changes
- Minor adjustments from list feedback;
- 01: More backticks for QAPI json files, now that they are in Sphinx-exe
- 02: Use :ref: role for the `docker-ref` cross-reference
- 04: Remove doc.py work changes; add references for start_if/end_if
- 10: Changes to accommodate isort
- 11: Added lines_after_imports=3D2
- 16: isort whitespace changes
- 24: Take Markus's docstring phrasing
- 34: add comment explaining os.open()
- 37: isort differences
Status:
(This is my stgit summary with reviewer tags visible.)
[01] do-not-merge-docs-replace #
[02] docs-repair-broken-references #
[03] docs-sphinx-change-default #
[04] qapi-modify-docstrings-to-be #
[05] do-not-merge-docs-enable #
[06] qapi-gen-separate-arg-parsing # [SOB] JS [RB] CR,EH [TB] CR
[07] qapi-move-generator-entrypoint # [SOB] JS [RB] CR,EH [TB] CR
[08] do-not-merge-more-apidoc #
[09] qapi-prefer-explicit-relative # [SOB] JS [RB] CR,EH
[10] qapi-remove-wildcard-includes # [SOB] JS [RB] CR,EH
[11] qapi-enforce-import-order # [SOB] JS [RB] CR [TB] CR
[12] qapi-delint-using-flake8 # [SOB] JS [RB] CR,EH
[13] qapi-add-pylintrc # [SOB] JS [TB] CR,EH [RB] CR
[14] qapi-common-py-remove-python # [SOB] JS [RB] CR,EH
[15] qapi-common-add-indent-manager # [SOB] JS [RB] CR,EH
[16] qapi-common-py-delint-with # [SOB] JS [RB] CR,EH
[17] replace-c-by-char # [SOB] JS [RB] CR,EH
[18] qapi-common-py-check-with # [SOB] JS [RB] CR [TB] CR,EH
[19] qapi-common-py-add-notational # [SOB] JS [RB] CR,EH
[20] qapi-common-move-comments-into # [SOB] JS [RB] CR,EH
[21] qapi-split-build_params-into # [SOB] JS [RB] CR,EH
[22] qapi-establish-mypy-type # [SOB] JS [TB] CR,EH [RB] CR
[23] qapi-events-py-add-notational # [SOB] JS [RB] CR,EH
[24] qapi-events-move-comments-into # [SOB] JS [RB] CR,EH
[25] qapi-commands-py-don-t-re-bind # [SOB] JS [RB] CR,EH
[26] qapi-commands-py-add # [SOB] JS [RB] CR,EH
[27] qapi-commands-py-enable # [SOB] JS [RB] CR,EH
[28] qapi-source-py-add-notational # [SOB] JS [RB] CR,EH [TB] CR
[29] qapi-source-py-delint-with # [SOB] JS [RB] CR,EH [TB] CR
[30] qapi-gen-py-fix-edge-case-of # [SOB] JS [RB] CR
[31] qapi-gen-py-add-notational # [SOB] JS [RB] CR,EH
[32] qapi-gen-py-enable-checking # [SOB] JS [RB] CR,EH [TB] CR
[33] qapi-gen-py-remove-unused # [SOB] JS [RB] CR,EH
[34] qapi-gen-py-update-write-to-be # [SOB] JS [RB] CR,EH
[35] qapi-gen-py-delint-with-pylint # [SOB] JS [RB] CR,EH
[36] qapi-introspect-py-assert-obj #
[37] qapi-introspect-py-create-a # [SOB] EH,JS
[38] qapi-introspect-py-add #
[39] qapi-introspect-py-unify #
[40] qapi-introspect-py-replace #
[41] qapi-introspect-py-create-a-0 #
[42] qapi-types-py-add-type-hint # [SOB] JS [RB] CR,EH
[43] qapi-types-py-remove-one # [SOB] JS [RB] CR,EH
[44] qapi-visit-py-assert # [SOB] JS [RB] CR,EH
[45] qapi-visit-py-remove-unused # [SOB] JS [RB] CR,EH [TB] CR
[46] qapi-visit-py-add-notational # [SOB] JS [RB] CR,EH [TB] CR
Changelog:
(Patches without changes elided)
001/46:[0004] [FC] '[DO-NOT-MERGE] docs: replace single backtick (`) with dou=
ble-backtick (``)'
002/46:[0002] [FC] 'docs: repair broken references'
004/46:[0014] [FC] 'qapi: modify docstrings to be sphinx-compatible'
005/46:[0015] [FC] '[DO-NOT-MERGE] docs: enable sphinx-autodoc for scripts/qa=
pi'
006/46:[0004] [FC] 'qapi-gen: Separate arg-parsing from generation'
007/46:[0008] [FC] 'qapi: move generator entrypoint into module'
009/46:[0004] [FC] 'qapi: Prefer explicit relative imports'
010/46:[0006] [FC] 'qapi: Remove wildcard includes'
011/46:[0023] [FC] 'qapi: enforce import order/styling with isort'
013/46:[0001] [FC] 'qapi: add pylintrc'
016/46:[0001] [FC] 'qapi/common.py: delint with pylint'
018/46:[0004] [FC] 'qapi/common.py: check with pylint'
022/46:[0008] [FC] 'qapi: establish mypy type-checking baseline'
024/46:[0004] [FC] 'qapi/events.py: Move comments into docstrings'
031/46:[0003] [FC] 'qapi/gen.py: add type hint annotations'
034/46:[0001] [FC] 'qapi/gen.py: update write() to be more idiomatic'
037/46:[0009] [FC] 'qapi/instrospect.py: add preliminary type hint annotation=
s'
V3:
- Use isort to enforce import consistency
- Use sphinx apidoc to check docstring format
V2:
- Removed Python3.6 patch in favor of Thomas Huth's
- Addressed (most) feedback from Markus
- Renamed type hint annotation commits
Eduardo Habkost (1):
qapi/instrospect.py: add preliminary type hint annotations
John Snow (45):
[DO-NOT-MERGE] docs: replace single backtick (`) with double-backtick
(``)
docs: repair broken references
[DO-NOT-MERGE] docs/sphinx: change default role to "any"
qapi: modify docstrings to be sphinx-compatible
[DO-NOT-MERGE] docs: enable sphinx-autodoc for scripts/qapi
qapi-gen: Separate arg-parsing from generation
qapi: move generator entrypoint into module
[DO-NOT-MERGE] docs: add scripts/qapi/main to python manual
qapi: Prefer explicit relative imports
qapi: Remove wildcard includes
qapi: enforce import order/styling with isort
qapi: delint using flake8
qapi: add pylintrc
qapi/common.py: Remove python compatibility workaround
qapi/common.py: Add indent manager
qapi/common.py: delint with pylint
qapi/common.py: Replace one-letter 'c' variable
qapi/common.py: check with pylint
qapi/common.py: add type hint annotations
qapi/common.py: Convert comments into docstrings, and elaborate
qapi/common.py: move build_params into gen.py
qapi: establish mypy type-checking baseline
qapi/events.py: add type hint annotations
qapi/events.py: Move comments into docstrings
qapi/commands.py: Don't re-bind to variable of different type
qapi/commands.py: add type hint annotations
qapi/commands.py: enable checking with mypy
qapi/source.py: add type hint annotations
qapi/source.py: delint with pylint
qapi/gen.py: Fix edge-case of _is_user_module
qapi/gen.py: add type hint annotations
qapi/gen.py: Enable checking with mypy
qapi/gen.py: Remove unused parameter
qapi/gen.py: update write() to be more idiomatic
qapi/gen.py: delint with pylint
qapi/introspect.py: assert obj is a dict when features are given
qapi/introspect.py: add _gen_features helper
qapi/introspect.py: Unify return type of _make_tree()
qapi/introspect.py: replace 'extra' dict with 'comment' argument
qapi/introspect.py: create a typed 'Node' data structure
qapi/types.py: add type hint annotations
qapi/types.py: remove one-letter variables
qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
qapi/visit.py: remove unused parameters from gen_visit_object
qapi/visit.py: add type hint annotations
docs/conf.py | 6 +-
docs/devel/build-system.rst | 118 +++++++-------
docs/devel/index.rst | 1 +
docs/devel/migration.rst | 59 +++----
docs/devel/multi-thread-tcg.rst | 2 +-
docs/devel/python/index.rst | 7 +
docs/devel/python/qapi.commands.rst | 7 +
docs/devel/python/qapi.common.rst | 7 +
docs/devel/python/qapi.error.rst | 7 +
docs/devel/python/qapi.events.rst | 7 +
docs/devel/python/qapi.expr.rst | 7 +
docs/devel/python/qapi.gen.rst | 7 +
docs/devel/python/qapi.introspect.rst | 7 +
docs/devel/python/qapi.main.rst | 7 +
docs/devel/python/qapi.parser.rst | 8 +
docs/devel/python/qapi.rst | 26 +++
docs/devel/python/qapi.schema.rst | 7 +
docs/devel/python/qapi.source.rst | 7 +
docs/devel/python/qapi.types.rst | 7 +
docs/devel/python/qapi.visit.rst | 7 +
docs/devel/tcg-plugins.rst | 14 +-
docs/devel/testing.rst | 4 +-
docs/interop/live-block-operations.rst | 4 +-
docs/system/arm/cpu-features.rst | 110 ++++++-------
docs/system/arm/nuvoton.rst | 2 +-
docs/system/s390x/protvirt.rst | 10 +-
qapi/block-core.json | 4 +-
scripts/qapi-gen.py | 57 ++-----
scripts/qapi/.flake8 | 2 +
scripts/qapi/.isort.cfg | 7 +
scripts/qapi/commands.py | 87 +++++++---
scripts/qapi/common.py | 163 ++++++++++---------
scripts/qapi/events.py | 58 +++++--
scripts/qapi/expr.py | 7 +-
scripts/qapi/gen.py | 182 +++++++++++++--------
scripts/qapi/introspect.py | 210 +++++++++++++++++--------
scripts/qapi/main.py | 89 +++++++++++
scripts/qapi/mypy.ini | 25 +++
scripts/qapi/parser.py | 15 +-
scripts/qapi/pylintrc | 70 +++++++++
scripts/qapi/schema.py | 35 +++--
scripts/qapi/source.py | 34 ++--
scripts/qapi/types.py | 125 ++++++++++-----
scripts/qapi/visit.py | 116 ++++++++++----
44 files changed, 1175 insertions(+), 566 deletions(-)
create mode 100644 docs/devel/python/index.rst
create mode 100644 docs/devel/python/qapi.commands.rst
create mode 100644 docs/devel/python/qapi.common.rst
create mode 100644 docs/devel/python/qapi.error.rst
create mode 100644 docs/devel/python/qapi.events.rst
create mode 100644 docs/devel/python/qapi.expr.rst
create mode 100644 docs/devel/python/qapi.gen.rst
create mode 100644 docs/devel/python/qapi.introspect.rst
create mode 100644 docs/devel/python/qapi.main.rst
create mode 100644 docs/devel/python/qapi.parser.rst
create mode 100644 docs/devel/python/qapi.rst
create mode 100644 docs/devel/python/qapi.schema.rst
create mode 100644 docs/devel/python/qapi.source.rst
create mode 100644 docs/devel/python/qapi.types.rst
create mode 100644 docs/devel/python/qapi.visit.rst
create mode 100644 scripts/qapi/.flake8
create mode 100644 scripts/qapi/.isort.cfg
create mode 100644 scripts/qapi/main.py
create mode 100644 scripts/qapi/mypy.ini
create mode 100644 scripts/qapi/pylintrc
--=20
2.26.2
^ permalink raw reply [flat|nested] 69+ messages in thread
* [PATCH v4 01/46] [DO-NOT-MERGE] docs: replace single backtick (`) with double-backtick (``)
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 02/46] docs: repair broken references John Snow
` (44 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
The single backtick in ReST is the "default role". Currently, Sphinx's
default role is called "content". Sphinx suggests you can use the "Any"
role instead to turn any single-backtick enclosed item into a
cross-reference.
Before we do that, though, we'll need to turn all existing usages of the
"content" role to inline verbatim markup by using double backticks
instead.
Signed-off-by: John Snow <jsnow@redhat.com>
---
docs/devel/build-system.rst | 118 ++++++++++++-------------
docs/devel/migration.rst | 59 +++++++------
docs/devel/tcg-plugins.rst | 14 +--
docs/devel/testing.rst | 2 +-
docs/interop/live-block-operations.rst | 4 +-
docs/system/arm/cpu-features.rst | 110 +++++++++++------------
docs/system/arm/nuvoton.rst | 2 +-
docs/system/s390x/protvirt.rst | 10 +--
qapi/block-core.json | 4 +-
9 files changed, 163 insertions(+), 160 deletions(-)
diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
index 08e85c69e1f..a4711cd4e1a 100644
--- a/docs/devel/build-system.rst
+++ b/docs/devel/build-system.rst
@@ -53,14 +53,14 @@ following tasks:
- Add a Meson build option to meson_options.txt.
- Add support to the command line arg parser to handle any new
- `--enable-XXX`/`--disable-XXX` flags required by the feature.
+ ``--enable-XXX``/``--disable-XXX`` flags required by the feature.
- Add information to the help output message to report on the new
feature flag.
- Add code to perform the actual feature check.
- - Add code to include the feature status in `config-host.h`
+ - Add code to include the feature status in ``config-host.h``
- Add code to print out the feature status in the configure summary
upon completion.
@@ -116,51 +116,51 @@ Helper functions
The configure script provides a variety of helper functions to assist
developers in checking for system features:
-`do_cc $ARGS...`
+``do_cc $ARGS...``
Attempt to run the system C compiler passing it $ARGS...
-`do_cxx $ARGS...`
+``do_cxx $ARGS...``
Attempt to run the system C++ compiler passing it $ARGS...
-`compile_object $CFLAGS`
+``compile_object $CFLAGS``
Attempt to compile a test program with the system C compiler using
$CFLAGS. The test program must have been previously written to a file
- called $TMPC. The replacement in Meson is the compiler object `cc`,
- which has methods such as `cc.compiles()`,
- `cc.check_header()`, `cc.has_function()`.
+ called $TMPC. The replacement in Meson is the compiler object ``cc``,
+ which has methods such as ``cc.compiles()``,
+ ``cc.check_header()``, ``cc.has_function()``.
-`compile_prog $CFLAGS $LDFLAGS`
+``compile_prog $CFLAGS $LDFLAGS``
Attempt to compile a test program with the system C compiler using
$CFLAGS and link it with the system linker using $LDFLAGS. The test
program must have been previously written to a file called $TMPC.
- The replacement in Meson is `cc.find_library()` and `cc.links()`.
+ The replacement in Meson is ``cc.find_library()`` and ``cc.links()``.
-`has $COMMAND`
+``has $COMMAND``
Determine if $COMMAND exists in the current environment, either as a
shell builtin, or executable binary, returning 0 on success. The
- replacement in Meson is `find_program()`.
+ replacement in Meson is ``find_program()``.
-`check_define $NAME`
+``check_define $NAME``
Determine if the macro $NAME is defined by the system C compiler
-`check_include $NAME`
+``check_include $NAME``
Determine if the include $NAME file is available to the system C
- compiler. The replacement in Meson is `cc.has_header()`.
+ compiler. The replacement in Meson is ``cc.has_header()``.
-`write_c_skeleton`
+``write_c_skeleton``
Write a minimal C program main() function to the temporary file
indicated by $TMPC
-`feature_not_found $NAME $REMEDY`
+``feature_not_found $NAME $REMEDY``
Print a message to stderr that the feature $NAME was not available
on the system, suggesting the user try $REMEDY to address the
problem.
-`error_exit $MESSAGE $MORE...`
+``error_exit $MESSAGE $MORE...``
Print $MESSAGE to stderr, followed by $MORE... and then exit from the
configure script with non-zero status
-`query_pkg_config $ARGS...`
+``query_pkg_config $ARGS...``
Run pkg-config passing it $ARGS. If QEMU is doing a static build,
then --static will be automatically added to $ARGS
@@ -194,8 +194,8 @@ to list the files and their dependency on various configuration
symbols.
Various subsystems that are common to both tools and emulators have
-their own sourceset, for example `block_ss` for the block device subsystem,
-`chardev_ss` for the character device subsystem, etc. These sourcesets
+their own sourceset, for example ``block_ss`` for the block device subsystem,
+``chardev_ss`` for the character device subsystem, etc. These sourcesets
are then turned into static libraries as follows::
libchardev = static_library('chardev', chardev_ss.sources(),
@@ -204,8 +204,8 @@ are then turned into static libraries as follows::
chardev = declare_dependency(link_whole: libchardev)
-As of Meson 0.55.1, the special `.fa` suffix should be used for everything
-that is used with `link_whole`, to ensure that the link flags are placed
+As of Meson 0.55.1, the special ``.fa`` suffix should be used for everything
+that is used with ``link_whole``, to ensure that the link flags are placed
correctly in the command line.
Files linked into emulator targets there can be split into two distinct groups
@@ -216,25 +216,25 @@ In the target-independent set lives various general purpose helper code,
such as error handling infrastructure, standard data structures,
platform portability wrapper functions, etc. This code can be compiled
once only and the .o files linked into all output binaries.
-Target-independent code lives in the `common_ss`, `softmmu_ss` and
-`user_ss` sourcesets. `common_ss` is linked into all emulators, `softmmu_ss`
-only in system emulators, `user_ss` only in user-mode emulators.
+Target-independent code lives in the ``common_ss``, ``softmmu_ss`` and
+``user_ss`` sourcesets. ``common_ss`` is linked into all emulators, ``softmmu_ss``
+only in system emulators, ``user_ss`` only in user-mode emulators.
In the target-dependent set lives CPU emulation, device emulation and
much glue code. This sometimes also has to be compiled multiple times,
once for each target being built. Target-dependent files are included
-in the `specific_ss` sourceset.
+in the ``specific_ss`` sourceset.
-All binaries link with a static library `libqemuutil.a`, which is then
-linked to all the binaries. `libqemuutil.a` is built from several
+All binaries link with a static library ``libqemuutil.a``, which is then
+linked to all the binaries. ``libqemuutil.a`` is built from several
sourcesets; most of them however host generated code, and the only two
-of general interest are `util_ss` and `stub_ss`.
+of general interest are ``util_ss`` and ``stub_ss``.
The separation between these two is purely for documentation purposes.
-`util_ss` contains generic utility files. Even though this code is only
+``util_ss`` contains generic utility files. Even though this code is only
linked in some binaries, sometimes it requires hooks only in some of
these and depend on other functions that are not fully implemented by
-all QEMU binaries. `stub_ss` links dummy stubs that will only be linked
+all QEMU binaries. ``stub_ss`` links dummy stubs that will only be linked
into the binary if the real implementation is not present. In a way,
the stubs can be thought of as a portable implementation of the weak
symbols concept.
@@ -242,7 +242,7 @@ symbols concept.
The following files concur in the definition of which files are linked
into each emulator:
-`default-configs/*.mak`
+``default-configs/*.mak``
The files under default-configs/ control what emulated hardware is built
into each QEMU system and userspace emulator targets. They merely contain
a list of config variable definitions like the machines that should be
@@ -252,8 +252,8 @@ into each emulator:
CONFIG_XLNX_ZYNQMP_ARM=y
CONFIG_XLNX_VERSAL=y
-`*/Kconfig`
- These files are processed together with `default-configs/*.mak` and
+``*/Kconfig``
+ These files are processed together with ``default-configs/*.mak`` and
describe the dependencies between various features, subsystems and
device models. They are described in kconfig.rst.
@@ -265,19 +265,19 @@ Support scripts
---------------
Meson has a special convention for invoking Python scripts: if their
-first line is `#! /usr/bin/env python3` and the file is *not* executable,
+first line is ``#! /usr/bin/env python3`` and the file is *not* executable,
find_program() arranges to invoke the script under the same Python
interpreter that was used to invoke Meson. This is the most common
and preferred way to invoke support scripts from Meson build files,
because it automatically uses the value of configure's --python= option.
-In case the script is not written in Python, use a `#! /usr/bin/env ...`
+In case the script is not written in Python, use a ``#! /usr/bin/env ...``
line and make the script executable.
Scripts written in Python, where it is desirable to make the script
executable (for example for test scripts that developers may want to
invoke from the command line, such as tests/qapi-schema/test-qapi.py),
-should be invoked through the `python` variable in meson.build. For
+should be invoked through the ``python`` variable in meson.build. For
example::
test('QAPI schema regression tests', python,
@@ -301,10 +301,10 @@ rules and wraps them so that e.g. submodules are built before QEMU.
The resulting build system is largely non-recursive in nature, in
contrast to common practices seen with automake.
-Tests are also ran by the Makefile with the traditional `make check`
-phony target, while benchmarks are run with `make bench`. Meson test
-suites such as `unit` can be ran with `make check-unit` too. It is also
-possible to run tests defined in meson.build with `meson test`.
+Tests are also ran by the Makefile with the traditional ``make check``
+phony target, while benchmarks are run with ``make bench``. Meson test
+suites such as ``unit`` can be ran with ``make check-unit`` too. It is also
+possible to run tests defined in meson.build with ``meson test``.
Important files for the build system
====================================
@@ -316,28 +316,28 @@ The following key files are statically defined in the source tree, with
the rules needed to build QEMU. Their behaviour is influenced by a
number of dynamically created files listed later.
-`Makefile`
+``Makefile``
The main entry point used when invoking make to build all the components
of QEMU. The default 'all' target will naturally result in the build of
every component. Makefile takes care of recursively building submodules
directly via a non-recursive set of rules.
-`*/meson.build`
+``*/meson.build``
The meson.build file in the root directory is the main entry point for the
Meson build system, and it coordinates the configuration and build of all
executables. Build rules for various subdirectories are included in
other meson.build files spread throughout the QEMU source tree.
-`tests/Makefile.include`
+``tests/Makefile.include``
Rules for external test harnesses. These include the TCG tests,
- `qemu-iotests` and the Avocado-based acceptance tests.
+ ``qemu-iotests`` and the Avocado-based acceptance tests.
-`tests/docker/Makefile.include`
+``tests/docker/Makefile.include``
Rules for Docker tests. Like tests/Makefile, this file is included
directly by the top level Makefile, anything defined in this file will
influence the entire build system.
-`tests/vm/Makefile.include`
+``tests/vm/Makefile.include``
Rules for VM-based tests. Like tests/Makefile, this file is included
directly by the top level Makefile, anything defined in this file will
influence the entire build system.
@@ -353,11 +353,11 @@ Makefile.
Built by configure:
-`config-host.mak`
+``config-host.mak``
When configure has determined the characteristics of the build host it
will write a long list of variables to config-host.mak file. This
provides the various install directories, compiler / linker flags and a
- variety of `CONFIG_*` variables related to optionally enabled features.
+ variety of ``CONFIG_*`` variables related to optionally enabled features.
This is imported by the top level Makefile and meson.build in order to
tailor the build output.
@@ -369,7 +369,7 @@ Built by configure:
build outputs. Variables which are potentially different for each
emulator target are defined by the next file...
-`$TARGET-NAME/config-target.mak`
+``$TARGET-NAME/config-target.mak``
TARGET-NAME is the name of a system or userspace emulator, for example,
x86_64-softmmu denotes the system emulator for the x86_64 architecture.
This file contains the variables which need to vary on a per-target
@@ -380,31 +380,31 @@ Built by configure:
Built by Meson:
-`${TARGET-NAME}-config-devices.mak`
+``${TARGET-NAME}-config-devices.mak``
TARGET-NAME is again the name of a system or userspace emulator. The
config-devices.mak file is automatically generated by make using the
scripts/make_device_config.sh program, feeding it the
default-configs/$TARGET-NAME file as input.
-`config-host.h`, `$TARGET-NAME/config-target.h`, `$TARGET-NAME/config-devices.h`
+``config-host.h``, ``$TARGET-NAME/config-target.h``, ``$TARGET-NAME/config-devices.h``
These files are used by source code to determine what features
are enabled. They are generated from the contents of the corresponding
- `*.h` files using the scripts/create_config program. This extracts
+ ``*.h`` files using the scripts/create_config program. This extracts
relevant variables and formats them as C preprocessor macros.
-`build.ninja`
+``build.ninja``
The build rules.
Built by Makefile:
-`Makefile.ninja`
+``Makefile.ninja``
A Makefile conversion of the build rules in build.ninja. The conversion
is straightforward and, were it necessary to debug the rules produced
by Meson, it should be enough to look at build.ninja. The conversion
is performed by scripts/ninjatool.py.
-`Makefile.mtest`
+``Makefile.mtest``
The Makefile definitions that let "make check" run tests defined in
meson.build. The rules are produced from Meson's JSON description of
tests (obtained with "meson introspect --tests") through the script
@@ -414,9 +414,9 @@ Built by Makefile:
Useful make targets
-------------------
-`help`
+``help``
Print a help message for the most common build targets.
-`print-VAR`
+``print-VAR``
Print the value of the variable VAR. Useful for debugging the build
system.
diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst
index 49112bb27aa..f50e1250359 100644
--- a/docs/devel/migration.rst
+++ b/docs/devel/migration.rst
@@ -53,7 +53,7 @@ savevm/loadvm functionality.
Debugging
=========
-The migration stream can be analyzed thanks to `scripts/analyze_migration.py`.
+The migration stream can be analyzed thanks to ``scripts/analyze_migration.py``.
Example usage:
@@ -74,8 +74,8 @@ Common infrastructure
=====================
The files, sockets or fd's that carry the migration stream are abstracted by
-the ``QEMUFile`` type (see `migration/qemu-file.h`). In most cases this
-is connected to a subtype of ``QIOChannel`` (see `io/`).
+the ``QEMUFile`` type (see ``migration/qemu-file.h``). In most cases this
+is connected to a subtype of ``QIOChannel`` (see ``io/``).
Saving the state of one device
@@ -165,14 +165,14 @@ An example (from hw/input/pckbd.c)
};
We are declaring the state with name "pckbd".
-The `version_id` is 3, and the fields are 4 uint8_t in a KBDState structure.
+The ``version_id`` is 3, and the fields are 4 uint8_t in a KBDState structure.
We registered this with:
.. code:: c
vmstate_register(NULL, 0, &vmstate_kbd, s);
-For devices that are `qdev` based, we can register the device in the class
+For devices that are ``qdev`` based, we can register the device in the class
init function:
.. code:: c
@@ -209,9 +209,9 @@ another to load the state back.
SaveVMHandlers *ops,
void *opaque);
-Two functions in the ``ops`` structure are the `save_state`
-and `load_state` functions. Notice that `load_state` receives a version_id
-parameter to know what state format is receiving. `save_state` doesn't
+Two functions in the ``ops`` structure are the ``save_state``
+and ``load_state`` functions. Notice that ``load_state`` receives a version_id
+parameter to know what state format is receiving. ``save_state`` doesn't
have a version_id parameter because it always uses the latest version.
Note that because the VMState macros still save the data in a raw
@@ -384,26 +384,28 @@ migration of a device, and using them breaks backward-migration
compatibility; in general most changes can be made by adding Subsections
(see above) or _TEST macros (see above) which won't break compatibility.
-Each version is associated with a series of fields saved. The `save_state` always saves
-the state as the newer version. But `load_state` sometimes is able to
-load state from an older version.
+Each version is associated with a series of fields saved. The
+``save_state`` always saves the state as the newer version. But
+``load_state`` sometimes is able to load state from an older version.
You can see that there are several version fields:
-- `version_id`: the maximum version_id supported by VMState for that device.
-- `minimum_version_id`: the minimum version_id that VMState is able to understand
- for that device.
-- `minimum_version_id_old`: For devices that were not able to port to vmstate, we can
- assign a function that knows how to read this old state. This field is
- ignored if there is no `load_state_old` handler.
+- ``version_id``: the maximum version_id supported by VMState for that
+ device.
+- ``minimum_version_id``: the minimum version_id that VMState is able to
+ understand for that device.
+- ``minimum_version_id_old``: For devices that were not able to port to
+ vmstate, we can assign a function that knows how to read this old
+ state. This field is ignored if there is no ``load_state_old``
+ handler.
VMState is able to read versions from minimum_version_id to
version_id. And the function ``load_state_old()`` (if present) is able to
load state from minimum_version_id_old to minimum_version_id. This
function is deprecated and will be removed when no more users are left.
-There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields,
-e.g.
+There are *_V* forms of many ``VMSTATE_`` macros to load fields for
+version dependent fields, e.g.
.. code:: c
@@ -453,7 +455,7 @@ data and then transferred to the main structure.
If you use memory API functions that update memory layout outside
initialization (i.e., in response to a guest action), this is a strong
-indication that you need to call these functions in a `post_load` callback.
+indication that you need to call these functions in a ``post_load`` callback.
Examples of such memory API functions are:
- memory_region_add_subregion()
@@ -818,17 +820,18 @@ Postcopy now works with hugetlbfs backed memory:
Postcopy with shared memory
---------------------------
-Postcopy migration with shared memory needs explicit support from the other
-processes that share memory and from QEMU. There are restrictions on the type of
-memory that userfault can support shared.
+Postcopy migration with shared memory needs explicit support from the
+other processes that share memory and from QEMU. There are restrictions
+on the type of memory that userfault can support shared.
-The Linux kernel userfault support works on `/dev/shm` memory and on `hugetlbfs`
-(although the kernel doesn't provide an equivalent to `madvise(MADV_DONTNEED)`
-for hugetlbfs which may be a problem in some configurations).
+The Linux kernel userfault support works on ``/dev/shm`` memory and on
+``hugetlbfs`` (although the kernel doesn't provide an equivalent to
+``madvise(MADV_DONTNEED)`` for hugetlbfs which may be a problem in some
+configurations).
The vhost-user code in QEMU supports clients that have Postcopy support,
-and the `vhost-user-bridge` (in `tests/`) and the DPDK package have changes
-to support postcopy.
+and the ``vhost-user-bridge`` (in ``tests/``) and the DPDK package have
+changes to support postcopy.
The client needs to open a userfaultfd and register the areas
of memory that it maps with userfault. The client must then pass the
diff --git a/docs/devel/tcg-plugins.rst b/docs/devel/tcg-plugins.rst
index 0568dfa6a49..5057b6e1b2b 100644
--- a/docs/devel/tcg-plugins.rst
+++ b/docs/devel/tcg-plugins.rst
@@ -34,11 +34,11 @@ version they were built against. This can be done simply by::
QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION;
The core code will refuse to load a plugin that doesn't export a
-`qemu_plugin_version` symbol or if plugin version is outside of QEMU's
+``qemu_plugin_version`` symbol or if plugin version is outside of QEMU's
supported range of API versions.
-Additionally the `qemu_info_t` structure which is passed to the
-`qemu_plugin_install` method of a plugin will detail the minimum and
+Additionally the ``qemu_info_t`` structure which is passed to the
+``qemu_plugin_install`` method of a plugin will detail the minimum and
current API versions supported by QEMU. The API version will be
incremented if new APIs are added. The minimum API version will be
incremented if existing APIs are changed or removed.
@@ -140,12 +140,12 @@ Example Plugins
There are a number of plugins included with QEMU and you are
encouraged to contribute your own plugins plugins upstream. There is a
-`contrib/plugins` directory where they can go.
+``contrib/plugins`` directory where they can go.
- tests/plugins
These are some basic plugins that are used to test and exercise the
-API during the `make check-tcg` target.
+API during the ``make check-tcg`` target.
- contrib/plugins/hotblocks.c
@@ -157,7 +157,7 @@ with linux-user execution as system emulation tends to generate
re-translations as blocks from different programs get swapped in and
out of system memory.
-If your program is single-threaded you can use the `inline` option for
+If your program is single-threaded you can use the ``inline`` option for
slightly faster (but not thread safe) counters.
Example::
@@ -245,7 +245,7 @@ which will lead to a sorted list after the class breakdown::
...
To find the argument shorthand for the class you need to examine the
-source code of the plugin at the moment, specifically the `*opt`
+source code of the plugin at the moment, specifically the ``*opt``
argument in the InsnClassExecCount tables.
- contrib/plugins/lockstep.c
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index bd64c1bdcdd..666c4d72408 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -749,7 +749,7 @@ The base test class has also support for tests with more than one
QEMUMachine. The way to get machines is through the ``self.get_vm()``
method which will return a QEMUMachine instance. The ``self.get_vm()``
method accepts arguments that will be passed to the QEMUMachine creation
-and also an optional `name` attribute so you can identify a specific
+and also an optional ``name`` attribute so you can identify a specific
machine and get it more than once through the tests methods. A simple
and hypothetical example follows:
diff --git a/docs/interop/live-block-operations.rst b/docs/interop/live-block-operations.rst
index e13f5a21f8d..5a3f0458275 100644
--- a/docs/interop/live-block-operations.rst
+++ b/docs/interop/live-block-operations.rst
@@ -638,7 +638,7 @@ at this point:
(QEMU) block-job-complete device=job0
In either of the above cases, if you once again run the
-`query-block-jobs` command, there should not be any active block
+``query-block-jobs`` command, there should not be any active block
operation.
Comparing 'commit' and 'mirror': In both then cases, the overlay images
@@ -777,7 +777,7 @@ the content of image [D].
}
(6) [On *destination* QEMU] Finally, resume the guest vCPUs by issuing the
- QMP command `cont`::
+ QMP command ``cont``::
(QEMU) cont
{
diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst
index 2d5c06cd016..640ee09f88b 100644
--- a/docs/system/arm/cpu-features.rst
+++ b/docs/system/arm/cpu-features.rst
@@ -10,22 +10,22 @@ is the Performance Monitoring Unit (PMU). CPU types such as the
Cortex-A15 and the Cortex-A57, which respectively implement Arm
architecture reference manuals ARMv7-A and ARMv8-A, may both optionally
implement PMUs. For example, if a user wants to use a Cortex-A15 without
-a PMU, then the `-cpu` parameter should contain `pmu=off` on the QEMU
-command line, i.e. `-cpu cortex-a15,pmu=off`.
+a PMU, then the ``-cpu`` parameter should contain ``pmu=off`` on the QEMU
+command line, i.e. ``-cpu cortex-a15,pmu=off``.
As not all CPU types support all optional CPU features, then whether or
not a CPU property exists depends on the CPU type. For example, CPUs
that implement the ARMv8-A architecture reference manual may optionally
support the AArch32 CPU feature, which may be enabled by disabling the
-`aarch64` CPU property. A CPU type such as the Cortex-A15, which does
-not implement ARMv8-A, will not have the `aarch64` CPU property.
+``aarch64`` CPU property. A CPU type such as the Cortex-A15, which does
+not implement ARMv8-A, will not have the ``aarch64`` CPU property.
QEMU's support may be limited for some CPU features, only partially
supporting the feature or only supporting the feature under certain
-configurations. For example, the `aarch64` CPU feature, which, when
+configurations. For example, the ``aarch64`` CPU feature, which, when
disabled, enables the optional AArch32 CPU feature, is only supported
when using the KVM accelerator and when running on a host CPU type that
-supports the feature. While `aarch64` currently only works with KVM,
+supports the feature. While ``aarch64`` currently only works with KVM,
it could work with TCG. CPU features that are specific to KVM are
prefixed with "kvm-" and are described in "KVM VCPU Features".
@@ -33,12 +33,12 @@ CPU Feature Probing
===================
Determining which CPU features are available and functional for a given
-CPU type is possible with the `query-cpu-model-expansion` QMP command.
-Below are some examples where `scripts/qmp/qmp-shell` (see the top comment
+CPU type is possible with the ``query-cpu-model-expansion`` QMP command.
+Below are some examples where ``scripts/qmp/qmp-shell`` (see the top comment
block in the script for usage) is used to issue the QMP commands.
-1. Determine which CPU features are available for the `max` CPU type
- (Note, we started QEMU with qemu-system-aarch64, so `max` is
+1. Determine which CPU features are available for the ``max`` CPU type
+ (Note, we started QEMU with qemu-system-aarch64, so ``max`` is
implementing the ARMv8-A reference manual in this case)::
(QEMU) query-cpu-model-expansion type=full model={"name":"max"}
@@ -51,9 +51,9 @@ block in the script for usage) is used to issue the QMP commands.
"sve896": true, "sve1280": true, "sve2048": true
}}}}
-We see that the `max` CPU type has the `pmu`, `aarch64`, `sve`, and many
-`sve<N>` CPU features. We also see that all the CPU features are
-enabled, as they are all `true`. (The `sve<N>` CPU features are all
+We see that the ``max`` CPU type has the ``pmu``, ``aarch64``, ``sve``, and many
+``sve<N>`` CPU features. We also see that all the CPU features are
+enabled, as they are all ``true``. (The ``sve<N>`` CPU features are all
optional SVE vector lengths (see "SVE CPU Properties"). While with TCG
all SVE vector lengths can be supported, when KVM is in use it's more
likely that only a few lengths will be supported, if SVE is supported at
@@ -71,9 +71,9 @@ all.)
"sve896": true, "sve1280": true, "sve2048": true
}}}}
-We see it worked, as `pmu` is now `false`.
+We see it worked, as ``pmu`` is now ``false``.
-(3) Let's try to disable `aarch64`, which enables the AArch32 CPU feature::
+(3) Let's try to disable ``aarch64``, which enables the AArch32 CPU feature::
(QEMU) query-cpu-model-expansion type=full model={"name":"max","props":{"aarch64":false}}
{"error": {
@@ -84,7 +84,7 @@ We see it worked, as `pmu` is now `false`.
It looks like this feature is limited to a configuration we do not
currently have.
-(4) Let's disable `sve` and see what happens to all the optional SVE
+(4) Let's disable ``sve`` and see what happens to all the optional SVE
vector lengths::
(QEMU) query-cpu-model-expansion type=full model={"name":"max","props":{"sve":false}}
@@ -97,14 +97,14 @@ currently have.
"sve896": false, "sve1280": false, "sve2048": false
}}}}
-As expected they are now all `false`.
+As expected they are now all ``false``.
(5) Let's try probing CPU features for the Cortex-A15 CPU type::
(QEMU) query-cpu-model-expansion type=full model={"name":"cortex-a15"}
{"return": {"model": {"name": "cortex-a15", "props": {"pmu": true}}}}
-Only the `pmu` CPU feature is available.
+Only the ``pmu`` CPU feature is available.
A note about CPU feature dependencies
-------------------------------------
@@ -123,29 +123,29 @@ A note about CPU models and KVM
-------------------------------
Named CPU models generally do not work with KVM. There are a few cases
-that do work, e.g. using the named CPU model `cortex-a57` with KVM on a
-seattle host, but mostly if KVM is enabled the `host` CPU type must be
+that do work, e.g. using the named CPU model ``cortex-a57`` with KVM on a
+seattle host, but mostly if KVM is enabled the ``host`` CPU type must be
used. This means the guest is provided all the same CPU features as the
-host CPU type has. And, for this reason, the `host` CPU type should
+host CPU type has. And, for this reason, the ``host`` CPU type should
enable all CPU features that the host has by default. Indeed it's even
a bit strange to allow disabling CPU features that the host has when using
-the `host` CPU type, but in the absence of CPU models it's the best we can
+the ``host`` CPU type, but in the absence of CPU models it's the best we can
do if we want to launch guests without all the host's CPU features enabled.
-Enabling KVM also affects the `query-cpu-model-expansion` QMP command. The
+Enabling KVM also affects the ``query-cpu-model-expansion`` QMP command. The
affect is not only limited to specific features, as pointed out in example
(3) of "CPU Feature Probing", but also to which CPU types may be expanded.
-When KVM is enabled, only the `max`, `host`, and current CPU type may be
+When KVM is enabled, only the ``max``, ``host``, and current CPU type may be
expanded. This restriction is necessary as it's not possible to know all
CPU types that may work with KVM, but it does impose a small risk of users
experiencing unexpected errors. For example on a seattle, as mentioned
-above, the `cortex-a57` CPU type is also valid when KVM is enabled.
-Therefore a user could use the `host` CPU type for the current type, but
-then attempt to query `cortex-a57`, however that query will fail with our
+above, the ``cortex-a57`` CPU type is also valid when KVM is enabled.
+Therefore a user could use the ``host`` CPU type for the current type, but
+then attempt to query ``cortex-a57``, however that query will fail with our
restrictions. This shouldn't be an issue though as management layers and
-users have been preferring the `host` CPU type for use with KVM for quite
+users have been preferring the ``host`` CPU type for use with KVM for quite
some time. Additionally, if the KVM-enabled QEMU instance running on a
-seattle host is using the `cortex-a57` CPU type, then querying `cortex-a57`
+seattle host is using the ``cortex-a57`` CPU type, then querying ``cortex-a57``
will work.
Using CPU Features
@@ -158,12 +158,12 @@ QEMU command line with that CPU type::
$ qemu-system-aarch64 -M virt -cpu max,pmu=off,sve=on,sve128=on,sve256=on
The example above disables the PMU and enables the first two SVE vector
-lengths for the `max` CPU type. Note, the `sve=on` isn't actually
-necessary, because, as we observed above with our probe of the `max` CPU
-type, `sve` is already on by default. Also, based on our probe of
+lengths for the ``max`` CPU type. Note, the ``sve=on`` isn't actually
+necessary, because, as we observed above with our probe of the ``max`` CPU
+type, ``sve`` is already on by default. Also, based on our probe of
defaults, it would seem we need to disable many SVE vector lengths, rather
than only enabling the two we want. This isn't the case, because, as
-disabling many SVE vector lengths would be quite verbose, the `sve<N>` CPU
+disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU
properties have special semantics (see "SVE CPU Property Parsing
Semantics").
@@ -203,49 +203,49 @@ the list of KVM VCPU features and their descriptions.
SVE CPU Properties
==================
-There are two types of SVE CPU properties: `sve` and `sve<N>`. The first
-is used to enable or disable the entire SVE feature, just as the `pmu`
+There are two types of SVE CPU properties: ``sve`` and ``sve<N>``. The first
+is used to enable or disable the entire SVE feature, just as the ``pmu``
CPU property completely enables or disables the PMU. The second type
-is used to enable or disable specific vector lengths, where `N` is the
-number of bits of the length. The `sve<N>` CPU properties have special
+is used to enable or disable specific vector lengths, where ``N`` is the
+number of bits of the length. The ``sve<N>`` CPU properties have special
dependencies and constraints, see "SVE CPU Property Dependencies and
Constraints" below. Additionally, as we want all supported vector lengths
to be enabled by default, then, in order to avoid overly verbose command
-lines (command lines full of `sve<N>=off`, for all `N` not wanted), we
+lines (command lines full of ``sve<N>=off``, for all ``N`` not wanted), we
provide the parsing semantics listed in "SVE CPU Property Parsing
Semantics".
SVE CPU Property Dependencies and Constraints
---------------------------------------------
- 1) At least one vector length must be enabled when `sve` is enabled.
+ 1) At least one vector length must be enabled when ``sve`` is enabled.
- 2) If a vector length `N` is enabled, then, when KVM is enabled, all
+ 2) If a vector length ``N`` is enabled, then, when KVM is enabled, all
smaller, host supported vector lengths must also be enabled. If
KVM is not enabled, then only all the smaller, power-of-two vector
lengths must be enabled. E.g. with KVM if the host supports all
- vector lengths up to 512-bits (128, 256, 384, 512), then if `sve512`
+ vector lengths up to 512-bits (128, 256, 384, 512), then if ``sve512``
is enabled, the 128-bit vector length, 256-bit vector length, and
384-bit vector length must also be enabled. Without KVM, the 384-bit
vector length would not be required.
3) If KVM is enabled then only vector lengths that the host CPU type
support may be enabled. If SVE is not supported by the host, then
- no `sve*` properties may be enabled.
+ no ``sve*`` properties may be enabled.
SVE CPU Property Parsing Semantics
----------------------------------
- 1) If SVE is disabled (`sve=off`), then which SVE vector lengths
+ 1) If SVE is disabled (``sve=off``), then which SVE vector lengths
are enabled or disabled is irrelevant to the guest, as the entire
SVE feature is disabled and that disables all vector lengths for
- the guest. However QEMU will still track any `sve<N>` CPU
- properties provided by the user. If later an `sve=on` is provided,
- then the guest will get only the enabled lengths. If no `sve=on`
+ the guest. However QEMU will still track any ``sve<N>`` CPU
+ properties provided by the user. If later an ``sve=on`` is provided,
+ then the guest will get only the enabled lengths. If no ``sve=on``
is provided and there are explicitly enabled vector lengths, then
an error is generated.
- 2) If SVE is enabled (`sve=on`), but no `sve<N>` CPU properties are
+ 2) If SVE is enabled (``sve=on``), but no ``sve<N>`` CPU properties are
provided, then all supported vector lengths are enabled, which when
KVM is not in use means including the non-power-of-two lengths, and,
when KVM is in use, it means all vector lengths supported by the host
@@ -261,7 +261,7 @@ SVE CPU Property Parsing Semantics
constraint (2) of "SVE CPU Property Dependencies and Constraints").
5) When KVM is enabled, if the host does not support SVE, then an error
- is generated when attempting to enable any `sve*` properties (see
+ is generated when attempting to enable any ``sve*`` properties (see
constraint (3) of "SVE CPU Property Dependencies and Constraints").
6) When KVM is enabled, if the host does support SVE, then an error is
@@ -269,8 +269,8 @@ SVE CPU Property Parsing Semantics
by the host (see constraint (3) of "SVE CPU Property Dependencies and
Constraints").
- 7) If one or more `sve<N>` CPU properties are set `off`, but no `sve<N>`,
- CPU properties are set `on`, then the specified vector lengths are
+ 7) If one or more ``sve<N>`` CPU properties are set ``off``, but no ``sve<N>``,
+ CPU properties are set ``on``, then the specified vector lengths are
disabled but the default for any unspecified lengths remains enabled.
When KVM is not enabled, disabling a power-of-two vector length also
disables all vector lengths larger than the power-of-two length.
@@ -278,15 +278,15 @@ SVE CPU Property Parsing Semantics
disables all larger vector lengths (see constraint (2) of "SVE CPU
Property Dependencies and Constraints").
- 8) If one or more `sve<N>` CPU properties are set to `on`, then they
+ 8) If one or more ``sve<N>`` CPU properties are set to ``on``, then they
are enabled and all unspecified lengths default to disabled, except
for the required lengths per constraint (2) of "SVE CPU Property
Dependencies and Constraints", which will even be auto-enabled if
they were not explicitly enabled.
- 9) If SVE was disabled (`sve=off`), allowing all vector lengths to be
+ 9) If SVE was disabled (``sve=off``), allowing all vector lengths to be
explicitly disabled (i.e. avoiding the error specified in (3) of
- "SVE CPU Property Parsing Semantics"), then if later an `sve=on` is
+ "SVE CPU Property Parsing Semantics"), then if later an ``sve=on`` is
provided an error will be generated. To avoid this error, one must
enable at least one vector length prior to enabling SVE.
@@ -297,12 +297,12 @@ SVE CPU Property Examples
$ qemu-system-aarch64 -M virt -cpu max,sve=off
- 2) Implicitly enable all vector lengths for the `max` CPU type::
+ 2) Implicitly enable all vector lengths for the ``max`` CPU type::
$ qemu-system-aarch64 -M virt -cpu max
3) When KVM is enabled, implicitly enable all host CPU supported vector
- lengths with the `host` CPU type::
+ lengths with the ``host`` CPU type::
$ qemu-system-aarch64 -M virt,accel=kvm -cpu host
diff --git a/docs/system/arm/nuvoton.rst b/docs/system/arm/nuvoton.rst
index e3e1a3a3a73..b0a35b54bd4 100644
--- a/docs/system/arm/nuvoton.rst
+++ b/docs/system/arm/nuvoton.rst
@@ -77,7 +77,7 @@ Boot options
------------
The Nuvoton machines can boot from an OpenBMC firmware image, or directly into
-a kernel using the ``-kernel`` option. OpenBMC images for `quanta-gsj` and
+a kernel using the ``-kernel`` option. OpenBMC images for ``quanta-gsj`` and
possibly others can be downloaded from the OpenPOWER jenkins :
https://openpower.xyz/
diff --git a/docs/system/s390x/protvirt.rst b/docs/system/s390x/protvirt.rst
index 712974ad87b..d208c12a962 100644
--- a/docs/system/s390x/protvirt.rst
+++ b/docs/system/s390x/protvirt.rst
@@ -14,11 +14,11 @@ Prerequisites
To run PVMs, a machine with the Protected Virtualization feature, as
indicated by the Ultravisor Call facility (stfle bit 158), is
required. The Ultravisor needs to be initialized at boot by setting
-`prot_virt=1` on the host's kernel command line.
+``prot_virt=1`` on the host's kernel command line.
Running PVMs requires using the KVM hypervisor.
-If those requirements are met, the capability `KVM_CAP_S390_PROTECTED`
+If those requirements are met, the capability ``KVM_CAP_S390_PROTECTED``
will indicate that KVM can support PVMs on that LPAR.
@@ -26,8 +26,8 @@ QEMU Settings
-------------
To indicate to the VM that it can transition into protected mode, the
-`Unpack facility` (stfle bit 161 represented by the feature
-`unpack`/`S390_FEAT_UNPACK`) needs to be part of the cpu model of
+``Unpack facility`` (stfle bit 161 represented by the feature
+``unpack``/``S390_FEAT_UNPACK``) needs to be part of the cpu model of
the VM.
All I/O devices need to use the IOMMU.
@@ -56,5 +56,5 @@ from the disk boot. This memory layout includes the encrypted
components (kernel, initrd, cmdline), the stage3a loader and
metadata. In case this boot method is used, the command line
options -initrd and -cmdline are ineffective. The preparation of a PVM
-image is done via the `genprotimg` tool from the s390-tools
+image is done via the ``genprotimg`` tool from the s390-tools
collection.
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 86ed72ef9fa..13254d3c3ba 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -492,11 +492,11 @@
# @status: current status of the dirty bitmap (since 2.4)
#
# @recording: true if the bitmap is recording new writes from the guest.
-# Replaces `active` and `disabled` statuses. (since 4.0)
+# Replaces ``active`` and ``disabled`` statuses. (since 4.0)
#
# @busy: true if the bitmap is in-use by some operation (NBD or jobs)
# and cannot be modified via QMP or used by another operation.
-# Replaces `locked` and `frozen` statuses. (since 4.0)
+# Replaces ``locked`` and ``frozen`` statuses. (since 4.0)
#
# @persistent: true if the bitmap was stored on disk, is scheduled to be stored
# on disk, or both. (since 4.0)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 02/46] docs: repair broken references
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
2020-09-30 4:31 ` [PATCH v4 01/46] [DO-NOT-MERGE] docs: replace single backtick (`) with double-backtick (``) John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 03/46] [DO-NOT-MERGE] docs/sphinx: change default role to "any" John Snow
` (43 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
In two different places, we are not making a cross-reference to some
resource correctly.
Signed-off-by: John Snow <jsnow@redhat.com>
---
docs/devel/multi-thread-tcg.rst | 2 +-
docs/devel/testing.rst | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst
index 21483870dbc..92a9eba13c9 100644
--- a/docs/devel/multi-thread-tcg.rst
+++ b/docs/devel/multi-thread-tcg.rst
@@ -267,7 +267,7 @@ of view of external observers (e.g. another processor core). They can
apply to any memory operations as well as just loads or stores.
The Linux kernel has an excellent `write-up
-<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`
+<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`_
on the various forms of memory barrier and the guarantees they can
provide.
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 666c4d72408..dfeda820482 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -953,7 +953,7 @@ compiler flags are needed to build for a given target.
If you have the ability to run containers as the user you can also
take advantage of the build systems "Docker" support. It will then use
containers to build any test case for an enabled guest where there is
-no system compiler available. See :ref: `_docker-ref` for details.
+no system compiler available. See :ref:`docker-ref` for details.
Running subset of tests
-----------------------
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 03/46] [DO-NOT-MERGE] docs/sphinx: change default role to "any"
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
2020-09-30 4:31 ` [PATCH v4 01/46] [DO-NOT-MERGE] docs: replace single backtick (`) with double-backtick (``) John Snow
2020-09-30 4:31 ` [PATCH v4 02/46] docs: repair broken references John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible John Snow
` (42 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Signed-off-by: John Snow <jsnow@redhat.com>
---
docs/conf.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/docs/conf.py b/docs/conf.py
index 606f623211d..49599e38314 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -84,6 +84,9 @@
# The master toctree document.
master_doc = 'index'
+# Interpret `this` to be a cross-reference to "anything".
+default_role = 'any'
+
# General information about the project.
project = u'QEMU'
copyright = u'2020, The QEMU Project Developers'
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (2 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 03/46] [DO-NOT-MERGE] docs/sphinx: change default role to "any" John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 8:47 ` Markus Armbruster
2020-09-30 4:31 ` [PATCH v4 05/46] [DO-NOT-MERGE] docs: enable sphinx-autodoc for scripts/qapi John Snow
` (41 subsequent siblings)
45 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
I did not say "sphinx beautiful", just "sphinx compatible". They will
not throw errors when parsed and interpreted as ReST.
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/gen.py | 6 ++++--
scripts/qapi/parser.py | 9 +++++----
2 files changed, 9 insertions(+), 6 deletions(-)
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index ca66c82b5b8..fc19b2aeb9b 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -154,9 +154,11 @@ def _bottom(self):
@contextmanager
def ifcontext(ifcond, *args):
- """A 'with' statement context manager to wrap with start_if()/end_if()
+ """
+ A 'with' statement context manager that wraps with `start_if` and `end_if`.
- *args: any number of QAPIGenCCode
+ :param ifcond: List of conditionals
+ :param args: any number of `QAPIGenCCode`.
Example::
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 9d1a3e2eea9..02983979965 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -381,10 +381,11 @@ def append(self, line):
The way that the line is dealt with depends on which part of
the documentation we're parsing right now:
- * The body section: ._append_line is ._append_body_line
- * An argument section: ._append_line is ._append_args_line
- * A features section: ._append_line is ._append_features_line
- * An additional section: ._append_line is ._append_various_line
+
+ * The body section: ._append_line is ._append_body_line
+ * An argument section: ._append_line is ._append_args_line
+ * A features section: ._append_line is ._append_features_line
+ * An additional section: ._append_line is ._append_various_line
"""
line = line[1:]
if not line:
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 05/46] [DO-NOT-MERGE] docs: enable sphinx-autodoc for scripts/qapi
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (3 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 06/46] qapi-gen: Separate arg-parsing from generation John Snow
` (40 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Signed-off-by: John Snow <jsnow@redhat.com>
---
docs/conf.py | 3 ++-
docs/devel/index.rst | 1 +
docs/devel/python/index.rst | 7 +++++++
docs/devel/python/qapi.commands.rst | 7 +++++++
docs/devel/python/qapi.common.rst | 7 +++++++
docs/devel/python/qapi.error.rst | 7 +++++++
docs/devel/python/qapi.events.rst | 7 +++++++
docs/devel/python/qapi.expr.rst | 7 +++++++
docs/devel/python/qapi.gen.rst | 7 +++++++
docs/devel/python/qapi.introspect.rst | 7 +++++++
docs/devel/python/qapi.parser.rst | 8 ++++++++
docs/devel/python/qapi.rst | 25 +++++++++++++++++++++++++
docs/devel/python/qapi.schema.rst | 7 +++++++
docs/devel/python/qapi.source.rst | 7 +++++++
docs/devel/python/qapi.types.rst | 7 +++++++
docs/devel/python/qapi.visit.rst | 7 +++++++
16 files changed, 120 insertions(+), 1 deletion(-)
create mode 100644 docs/devel/python/index.rst
create mode 100644 docs/devel/python/qapi.commands.rst
create mode 100644 docs/devel/python/qapi.common.rst
create mode 100644 docs/devel/python/qapi.error.rst
create mode 100644 docs/devel/python/qapi.events.rst
create mode 100644 docs/devel/python/qapi.expr.rst
create mode 100644 docs/devel/python/qapi.gen.rst
create mode 100644 docs/devel/python/qapi.introspect.rst
create mode 100644 docs/devel/python/qapi.parser.rst
create mode 100644 docs/devel/python/qapi.rst
create mode 100644 docs/devel/python/qapi.schema.rst
create mode 100644 docs/devel/python/qapi.source.rst
create mode 100644 docs/devel/python/qapi.types.rst
create mode 100644 docs/devel/python/qapi.visit.rst
diff --git a/docs/conf.py b/docs/conf.py
index 49599e38314..1e428a4bda8 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -70,7 +70,8 @@
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
-extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile', 'qapidoc']
+extensions = ['kerneldoc', 'qmp_lexer', 'hxtool',
+ 'depfile', 'qapidoc', 'sphinx.ext.autodoc']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 04773ce076b..04726ca7870 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -31,3 +31,4 @@ Contents:
reset
s390-dasd-ipl
clocks
+ python/index
diff --git a/docs/devel/python/index.rst b/docs/devel/python/index.rst
new file mode 100644
index 00000000000..31c470154b3
--- /dev/null
+++ b/docs/devel/python/index.rst
@@ -0,0 +1,7 @@
+qapi
+====
+
+.. toctree::
+ :maxdepth: 4
+
+ qapi
diff --git a/docs/devel/python/qapi.commands.rst b/docs/devel/python/qapi.commands.rst
new file mode 100644
index 00000000000..018f7b08a9c
--- /dev/null
+++ b/docs/devel/python/qapi.commands.rst
@@ -0,0 +1,7 @@
+qapi.commands module
+====================
+
+.. automodule:: qapi.commands
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.common.rst b/docs/devel/python/qapi.common.rst
new file mode 100644
index 00000000000..128a90d74be
--- /dev/null
+++ b/docs/devel/python/qapi.common.rst
@@ -0,0 +1,7 @@
+qapi.common module
+==================
+
+.. automodule:: qapi.common
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.error.rst b/docs/devel/python/qapi.error.rst
new file mode 100644
index 00000000000..980e32b63de
--- /dev/null
+++ b/docs/devel/python/qapi.error.rst
@@ -0,0 +1,7 @@
+qapi.error module
+=================
+
+.. automodule:: qapi.error
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.events.rst b/docs/devel/python/qapi.events.rst
new file mode 100644
index 00000000000..1fce85b044e
--- /dev/null
+++ b/docs/devel/python/qapi.events.rst
@@ -0,0 +1,7 @@
+qapi.events module
+==================
+
+.. automodule:: qapi.events
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.expr.rst b/docs/devel/python/qapi.expr.rst
new file mode 100644
index 00000000000..0660270629c
--- /dev/null
+++ b/docs/devel/python/qapi.expr.rst
@@ -0,0 +1,7 @@
+qapi.expr module
+================
+
+.. automodule:: qapi.expr
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.gen.rst b/docs/devel/python/qapi.gen.rst
new file mode 100644
index 00000000000..7b495fd4bf2
--- /dev/null
+++ b/docs/devel/python/qapi.gen.rst
@@ -0,0 +1,7 @@
+qapi.gen module
+===============
+
+.. automodule:: qapi.gen
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.introspect.rst b/docs/devel/python/qapi.introspect.rst
new file mode 100644
index 00000000000..f65ebfccd1b
--- /dev/null
+++ b/docs/devel/python/qapi.introspect.rst
@@ -0,0 +1,7 @@
+qapi.introspect module
+======================
+
+.. automodule:: qapi.introspect
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.parser.rst b/docs/devel/python/qapi.parser.rst
new file mode 100644
index 00000000000..1a8f7b347eb
--- /dev/null
+++ b/docs/devel/python/qapi.parser.rst
@@ -0,0 +1,8 @@
+qapi.parser module
+==================
+
+.. automodule:: qapi.parser
+ :members:
+ :undoc-members:
+ :show-inheritance:
+ :private-members:
diff --git a/docs/devel/python/qapi.rst b/docs/devel/python/qapi.rst
new file mode 100644
index 00000000000..cfeb759d763
--- /dev/null
+++ b/docs/devel/python/qapi.rst
@@ -0,0 +1,25 @@
+qapi package
+============
+
+.. automodule:: qapi
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+Submodules
+----------
+
+.. toctree::
+
+ qapi.commands
+ qapi.common
+ qapi.error
+ qapi.events
+ qapi.expr
+ qapi.gen
+ qapi.introspect
+ qapi.parser
+ qapi.schema
+ qapi.source
+ qapi.types
+ qapi.visit
diff --git a/docs/devel/python/qapi.schema.rst b/docs/devel/python/qapi.schema.rst
new file mode 100644
index 00000000000..a08f75ed720
--- /dev/null
+++ b/docs/devel/python/qapi.schema.rst
@@ -0,0 +1,7 @@
+qapi.schema module
+==================
+
+.. automodule:: qapi.schema
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.source.rst b/docs/devel/python/qapi.source.rst
new file mode 100644
index 00000000000..e61e9f60212
--- /dev/null
+++ b/docs/devel/python/qapi.source.rst
@@ -0,0 +1,7 @@
+qapi.source module
+==================
+
+.. automodule:: qapi.source
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.types.rst b/docs/devel/python/qapi.types.rst
new file mode 100644
index 00000000000..6eea827557d
--- /dev/null
+++ b/docs/devel/python/qapi.types.rst
@@ -0,0 +1,7 @@
+qapi.types module
+=================
+
+.. automodule:: qapi.types
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.visit.rst b/docs/devel/python/qapi.visit.rst
new file mode 100644
index 00000000000..84307cbc236
--- /dev/null
+++ b/docs/devel/python/qapi.visit.rst
@@ -0,0 +1,7 @@
+qapi.visit module
+=================
+
+.. automodule:: qapi.visit
+ :members:
+ :undoc-members:
+ :show-inheritance:
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 06/46] qapi-gen: Separate arg-parsing from generation
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (4 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 05/46] [DO-NOT-MERGE] docs: enable sphinx-autodoc for scripts/qapi John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 07/46] qapi: move generator entrypoint into module John Snow
` (39 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
This is a minor re-work of the entrypoint script. It isolates a
generate() method from the actual command-line mechanism.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
1 file changed, 62 insertions(+), 23 deletions(-)
diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 541e8c1f55d..117b396a595 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -1,30 +1,77 @@
#!/usr/bin/env python3
-# QAPI generator
-#
+
# This work is licensed under the terms of the GNU GPL, version 2 or later.
# See the COPYING file in the top-level directory.
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
import argparse
import re
import sys
from qapi.commands import gen_commands
+from qapi.error import QAPIError
from qapi.events import gen_events
from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
from qapi.types import gen_types
from qapi.visit import gen_visit
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+ output_dir: str,
+ prefix: str,
+ unmask: bool = False,
+ builtins: bool = False) -> None:
+ """
+ generate uses a given schema to produce C code in the target directory.
+
+ :param schema_file: The primary QAPI schema file.
+ :param output_dir: The output directory to store generated code.
+ :param prefix: Optional C-code prefix for symbol names.
+ :param unmask: Expose non-ABI names through introspection?
+ :param builtins: Generate code for built-in types?
+
+ :raise QAPIError: On failures.
+ """
+ match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+ if match.end() != len(prefix):
+ msg = "funny character '{:s}' in prefix '{:s}'".format(
+ prefix[match.end()], prefix)
+ raise QAPIError('', None, msg)
+
+ schema = QAPISchema(schema_file)
+ gen_types(schema, output_dir, prefix, builtins)
+ gen_visit(schema, output_dir, prefix, builtins)
+ gen_commands(schema, output_dir, prefix)
+ gen_events(schema, output_dir, prefix)
+ gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+ """
+ gapi-gen shell script entrypoint.
+ Expects arguments via sys.argv, see --help for details.
+
+ :return: int, 0 on success, 1 on failure.
+ """
parser = argparse.ArgumentParser(
description='Generate code from a QAPI schema')
parser.add_argument('-b', '--builtins', action='store_true',
help="generate code for built-in types")
- parser.add_argument('-o', '--output-dir', action='store', default='',
+ parser.add_argument('-o', '--output-dir', action='store',
+ default=DEFAULT_OUTPUT_DIR,
help="write output to directory OUTPUT_DIR")
- parser.add_argument('-p', '--prefix', action='store', default='',
+ parser.add_argument('-p', '--prefix', action='store',
+ default=DEFAULT_PREFIX,
help="prefix for symbols")
parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
parser.add_argument('schema', action='store')
args = parser.parse_args()
- match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
- if match.end() != len(args.prefix):
- print("%s: 'funny character '%s' in argument of --prefix"
- % (sys.argv[0], args.prefix[match.end()]),
- file=sys.stderr)
- sys.exit(1)
-
try:
- schema = QAPISchema(args.schema)
+ generate(args.schema,
+ output_dir=args.output_dir,
+ prefix=args.prefix,
+ unmask=args.unmask,
+ builtins=args.builtins)
except QAPIError as err:
- print(err, file=sys.stderr)
- exit(1)
-
- gen_types(schema, args.output_dir, args.prefix, args.builtins)
- gen_visit(schema, args.output_dir, args.prefix, args.builtins)
- gen_commands(schema, args.output_dir, args.prefix)
- gen_events(schema, args.output_dir, args.prefix)
- gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+ print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+ return 1
+ return 0
if __name__ == '__main__':
- main(sys.argv)
+ sys.exit(main())
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 07/46] qapi: move generator entrypoint into module
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (5 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 06/46] qapi-gen: Separate arg-parsing from generation John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 08/46] [DO-NOT-MERGE] docs: add scripts/qapi/main to python manual John Snow
` (38 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
As part of delinting and adding type hints to the QAPI generator, it's
helpful for the entrypoint to be part of the package, only leaving a
very tiny entrypoint shim outside of the module.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi-gen.py | 88 +++----------------------------------------
scripts/qapi/main.py | 89 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 95 insertions(+), 82 deletions(-)
create mode 100644 scripts/qapi/main.py
diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 117b396a595..f3518d29a54 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -4,92 +4,16 @@
# See the COPYING file in the top-level directory.
"""
-QAPI Generator
+QAPI code generation execution shim.
-This script is the main entry point for generating C code from the QAPI schema.
+This standalone script exists primarily to facilitate the running of the QAPI
+code generator without needing to install the python module to the current
+execution environment.
"""
-import argparse
-import re
import sys
-from qapi.commands import gen_commands
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
-
-
-DEFAULT_OUTPUT_DIR = ''
-DEFAULT_PREFIX = ''
-
-
-def generate(schema_file: str,
- output_dir: str,
- prefix: str,
- unmask: bool = False,
- builtins: bool = False) -> None:
- """
- generate uses a given schema to produce C code in the target directory.
-
- :param schema_file: The primary QAPI schema file.
- :param output_dir: The output directory to store generated code.
- :param prefix: Optional C-code prefix for symbol names.
- :param unmask: Expose non-ABI names through introspection?
- :param builtins: Generate code for built-in types?
-
- :raise QAPIError: On failures.
- """
- match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
- if match.end() != len(prefix):
- msg = "funny character '{:s}' in prefix '{:s}'".format(
- prefix[match.end()], prefix)
- raise QAPIError('', None, msg)
-
- schema = QAPISchema(schema_file)
- gen_types(schema, output_dir, prefix, builtins)
- gen_visit(schema, output_dir, prefix, builtins)
- gen_commands(schema, output_dir, prefix)
- gen_events(schema, output_dir, prefix)
- gen_introspect(schema, output_dir, prefix, unmask)
-
-
-def main() -> int:
- """
- gapi-gen shell script entrypoint.
- Expects arguments via sys.argv, see --help for details.
-
- :return: int, 0 on success, 1 on failure.
- """
- parser = argparse.ArgumentParser(
- description='Generate code from a QAPI schema')
- parser.add_argument('-b', '--builtins', action='store_true',
- help="generate code for built-in types")
- parser.add_argument('-o', '--output-dir', action='store',
- default=DEFAULT_OUTPUT_DIR,
- help="write output to directory OUTPUT_DIR")
- parser.add_argument('-p', '--prefix', action='store',
- default=DEFAULT_PREFIX,
- help="prefix for symbols")
- parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
- dest='unmask',
- help="expose non-ABI names in introspection")
- parser.add_argument('schema', action='store')
- args = parser.parse_args()
-
- try:
- generate(args.schema,
- output_dir=args.output_dir,
- prefix=args.prefix,
- unmask=args.unmask,
- builtins=args.builtins)
- except QAPIError as err:
- print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
- return 1
- return 0
-
+from qapi import main
if __name__ == '__main__':
- sys.exit(main())
+ sys.exit(main.main())
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
new file mode 100644
index 00000000000..9210a0e1a80
--- /dev/null
+++ b/scripts/qapi/main.py
@@ -0,0 +1,89 @@
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+"""
+QAPI Generator
+
+This is the main entry point for generating C code from the QAPI schema.
+"""
+
+import argparse
+import re
+import sys
+
+from qapi.commands import gen_commands
+from qapi.error import QAPIError
+from qapi.events import gen_events
+from qapi.introspect import gen_introspect
+from qapi.schema import QAPISchema
+from qapi.types import gen_types
+from qapi.visit import gen_visit
+
+
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+ output_dir: str,
+ prefix: str,
+ unmask: bool = False,
+ builtins: bool = False) -> None:
+ """
+ generate uses a given schema to produce C code in the target directory.
+
+ :param schema_file: The primary QAPI schema file.
+ :param output_dir: The output directory to store generated code.
+ :param prefix: Optional C-code prefix for symbol names.
+ :param unmask: Expose non-ABI names through introspection?
+ :param builtins: Generate code for built-in types?
+
+ :raise QAPIError: On failures.
+ """
+ match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+ if match.end() != len(prefix):
+ msg = "funny character '{:s}' in prefix '{:s}'".format(
+ prefix[match.end()], prefix)
+ raise QAPIError('', None, msg)
+
+ schema = QAPISchema(schema_file)
+ gen_types(schema, output_dir, prefix, builtins)
+ gen_visit(schema, output_dir, prefix, builtins)
+ gen_commands(schema, output_dir, prefix)
+ gen_events(schema, output_dir, prefix)
+ gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+ """
+ gapi-gen shell script entrypoint.
+ Expects arguments via sys.argv, see --help for details.
+
+ :return: int, 0 on success, 1 on failure.
+ """
+ parser = argparse.ArgumentParser(
+ description='Generate code from a QAPI schema')
+ parser.add_argument('-b', '--builtins', action='store_true',
+ help="generate code for built-in types")
+ parser.add_argument('-o', '--output-dir', action='store',
+ default=DEFAULT_OUTPUT_DIR,
+ help="write output to directory OUTPUT_DIR")
+ parser.add_argument('-p', '--prefix', action='store',
+ default=DEFAULT_PREFIX,
+ help="prefix for symbols")
+ parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
+ dest='unmask',
+ help="expose non-ABI names in introspection")
+ parser.add_argument('schema', action='store')
+ args = parser.parse_args()
+
+ try:
+ generate(args.schema,
+ output_dir=args.output_dir,
+ prefix=args.prefix,
+ unmask=args.unmask,
+ builtins=args.builtins)
+ except QAPIError as err:
+ print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+ return 1
+ return 0
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 08/46] [DO-NOT-MERGE] docs: add scripts/qapi/main to python manual
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (6 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 07/46] qapi: move generator entrypoint into module John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 09/46] qapi: Prefer explicit relative imports John Snow
` (37 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Signed-off-by: John Snow <jsnow@redhat.com>
---
docs/devel/python/qapi.main.rst | 7 +++++++
docs/devel/python/qapi.rst | 1 +
2 files changed, 8 insertions(+)
create mode 100644 docs/devel/python/qapi.main.rst
diff --git a/docs/devel/python/qapi.main.rst b/docs/devel/python/qapi.main.rst
new file mode 100644
index 00000000000..1255fcda633
--- /dev/null
+++ b/docs/devel/python/qapi.main.rst
@@ -0,0 +1,7 @@
+qapi.main module
+================
+
+.. automodule:: qapi.main
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/devel/python/qapi.rst b/docs/devel/python/qapi.rst
index cfeb759d763..c762019aad3 100644
--- a/docs/devel/python/qapi.rst
+++ b/docs/devel/python/qapi.rst
@@ -18,6 +18,7 @@ Submodules
qapi.expr
qapi.gen
qapi.introspect
+ qapi.main
qapi.parser
qapi.schema
qapi.source
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 09/46] qapi: Prefer explicit relative imports
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (7 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 08/46] [DO-NOT-MERGE] docs: add scripts/qapi/main to python manual John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 10/46] qapi: Remove wildcard includes John Snow
` (36 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
All of the QAPI include statements are changed to be package-aware, as
explicit relative imports.
A quirk of Python packages is that the name of the package exists only
*outside* of the package. This means that to a module inside of the qapi
folder, there is inherently no such thing as the "qapi" package. The
reason these imports work is because the "qapi" package exists in the
context of the caller -- the execution shim, where sys.path includes a
directory that has a 'qapi' folder in it.
When we write "from qapi import sibling", we are NOT referencing the folder
'qapi', but rather "any package named qapi in sys.path". If you should
so happen to have a 'qapi' package in your path, it will use *that*
package.
When we write "from .sibling import foo", we always reference explicitly
our sibling module; guaranteeing consistency in *where* we are importing
these modules from.
This can be useful when working with virtual environments and packages
in development mode. In development mode, a package is installed as a
series of symlinks that forwards to your same source files. The problem
arises because code quality checkers will follow "import qapi.x" to the
"installed" version instead of the sibling file and -- even though they
are the same file -- they have different module paths, and this causes
cyclic import problems, false positive type mismatch errors, and more.
It can also be useful when dealing with hierarchical packages, e.g. if
we allow qemu.core.qmp, qemu.qapi.parser, etc.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/commands.py | 4 ++--
scripts/qapi/events.py | 8 ++++----
scripts/qapi/expr.py | 4 ++--
scripts/qapi/gen.py | 4 ++--
scripts/qapi/introspect.py | 8 ++++----
scripts/qapi/main.py | 14 +++++++-------
scripts/qapi/parser.py | 4 ++--
scripts/qapi/schema.py | 8 ++++----
scripts/qapi/types.py | 6 +++---
scripts/qapi/visit.py | 6 +++---
10 files changed, 33 insertions(+), 33 deletions(-)
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 3cf9e1110b2..ce5926146a4 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,8 +13,8 @@
See the COPYING file in the top-level directory.
"""
-from qapi.common import *
-from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import *
+from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
def gen_command_decl(name, arg_type, boxed, ret_type):
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index b544af5a1ce..04672724388 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,10 +12,10 @@
See the COPYING file in the top-level directory.
"""
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember
-from qapi.types import gen_enum, gen_enum_lookup
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember
+from .types import gen_enum, gen_enum_lookup
def build_event_send_proto(name, arg_type, boxed):
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 2942520399a..03b31ecfc19 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -16,8 +16,8 @@
import re
from collections import OrderedDict
-from qapi.common import c_name
-from qapi.error import QAPISemError
+from .common import c_name
+from .error import QAPISemError
# Names must be letters, numbers, -, and _. They must start with letter,
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index fc19b2aeb9b..14d584680dc 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -17,8 +17,8 @@
import re
from contextlib import contextmanager
-from qapi.common import *
-from qapi.schema import QAPISchemaVisitor
+from .common import *
+from .schema import QAPISchemaVisitor
class QAPIGen:
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 23652be8102..2a34cd1e8ea 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,10 +10,10 @@
See the COPYING file in the top-level directory.
"""
-from qapi.common import *
-from qapi.gen import QAPISchemaMonolithicCVisitor
-from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
- QAPISchemaType)
+from .common import *
+from .gen import QAPISchemaMonolithicCVisitor
+from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
+ QAPISchemaType)
def _make_tree(obj, ifcond, features, extra=None):
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
index 9210a0e1a80..1b168a9771a 100644
--- a/scripts/qapi/main.py
+++ b/scripts/qapi/main.py
@@ -11,13 +11,13 @@
import re
import sys
-from qapi.commands import gen_commands
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
+from .commands import gen_commands
+from .error import QAPIError
+from .events import gen_events
+from .introspect import gen_introspect
+from .schema import QAPISchema
+from .types import gen_types
+from .visit import gen_visit
DEFAULT_OUTPUT_DIR = ''
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 02983979965..a9388eaf765 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -18,8 +18,8 @@
import re
from collections import OrderedDict
-from qapi.error import QAPIParseError, QAPISemError
-from qapi.source import QAPISourceInfo
+from .error import QAPIParseError, QAPISemError
+from .source import QAPISourceInfo
class QAPISchemaParser:
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 78309a00f0a..a835ee6fde3 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,10 +18,10 @@
import re
from collections import OrderedDict
-from qapi.common import c_name, pointer_suffix
-from qapi.error import QAPIError, QAPISemError
-from qapi.expr import check_exprs
-from qapi.parser import QAPISchemaParser
+from .common import c_name, pointer_suffix
+from .error import QAPIError, QAPISemError
+from .expr import check_exprs
+from .parser import QAPISchemaParser
class QAPISchemaEntity:
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 3640f17cd67..ca9a5aacb39 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,9 +13,9 @@
# See the COPYING file in the top-level directory.
"""
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
# variants must be emitted before their container; track what has already
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index cdabc5fa283..7850f6e8480 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,9 +13,9 @@
See the COPYING file in the top-level directory.
"""
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaObjectType
def gen_visit_decl(name, scalar=False):
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 10/46] qapi: Remove wildcard includes
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (8 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 09/46] qapi: Prefer explicit relative imports John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 11/46] qapi: enforce import order/styling with isort John Snow
` (35 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Wildcard includes become hard to manage when refactoring and dealing
with circular dependencies with strictly typed mypy.
flake8 also flags each one as a warning, as it is not smart enough to
know which names exist in the imported file.
Remove them and include things explicitly by name instead.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/commands.py | 2 +-
scripts/qapi/events.py | 7 ++++++-
scripts/qapi/gen.py | 12 +++++++++---
scripts/qapi/introspect.py | 7 ++++++-
scripts/qapi/types.py | 8 +++++++-
scripts/qapi/visit.py | 10 +++++++++-
6 files changed, 38 insertions(+), 8 deletions(-)
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index ce5926146a4..64ed5278f93 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,7 +13,7 @@
See the COPYING file in the top-level directory.
"""
-from .common import *
+from .common import build_params, c_name, mcgen
from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 04672724388..6b3afa14d72 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,7 +12,12 @@
See the COPYING file in the top-level directory.
"""
-from .common import *
+from .common import (
+ build_params,
+ c_enum_const,
+ c_name,
+ mcgen,
+)
from .gen import QAPISchemaModularCVisitor, ifcontext
from .schema import QAPISchemaEnumMember
from .types import gen_enum, gen_enum_lookup
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 14d584680dc..61b3c53b180 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -11,13 +11,19 @@
# This work is licensed under the terms of the GNU GPL, version 2.
# See the COPYING file in the top-level directory.
-
+from contextlib import contextmanager
import errno
import os
import re
-from contextlib import contextmanager
-from .common import *
+from .common import (
+ c_fname,
+ gen_endif,
+ gen_if,
+ guardend,
+ guardstart,
+ mcgen,
+)
from .schema import QAPISchemaVisitor
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 2a34cd1e8ea..b036fcf9ce7 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,12 @@
See the COPYING file in the top-level directory.
"""
-from .common import *
+from .common import (
+ c_name,
+ gen_endif,
+ gen_if,
+ mcgen,
+)
from .gen import QAPISchemaMonolithicCVisitor
from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
QAPISchemaType)
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index ca9a5aacb39..53b47f9e58a 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,7 +13,13 @@
# See the COPYING file in the top-level directory.
"""
-from .common import *
+from .common import (
+ c_enum_const,
+ c_name,
+ gen_endif,
+ gen_if,
+ mcgen,
+)
from .gen import QAPISchemaModularCVisitor, ifcontext
from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 7850f6e8480..ea277e7704b 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,7 +13,15 @@
See the COPYING file in the top-level directory.
"""
-from .common import *
+from .common import (
+ c_enum_const,
+ c_name,
+ gen_endif,
+ gen_if,
+ mcgen,
+ pop_indent,
+ push_indent,
+)
from .gen import QAPISchemaModularCVisitor, ifcontext
from .schema import QAPISchemaObjectType
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 11/46] qapi: enforce import order/styling with isort
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (9 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 10/46] qapi: Remove wildcard includes John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 12/46] qapi: delint using flake8 John Snow
` (34 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
While we're mucking around with imports, we might as well formalize the
style we use. Let's use isort to do it for us.
lines_after_imports=2: Use two lines after imports, to match PEP8's
desire to have "two lines before and after" class definitions, which are
likely to start immediately after imports.
force_sort_within_sections: Intermingles "from x" and "import x" style
statements, such that sorting is always performed strictly on the module
name itself.
force_grid_wrap=4: Four or more imports from a single module will force
the one-per-line style that's more git-friendly. This will generally
happen for 'typing' imports.
multi_line_output=3: Uses the one-per-line indented style for long
imports.
include_trailing_comma: Adds a comma to the last import in a group,
which makes git conflicts nicer to deal with, generally.
line_length: 72 is chosen to match PEP8's "docstrings and comments" line
length limit. If you have a single line import that exceeds 72
characters, your names are too long!
Suggested-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/.isort.cfg | 7 +++++++
scripts/qapi/expr.py | 3 ++-
scripts/qapi/introspect.py | 7 +++++--
scripts/qapi/parser.py | 2 +-
scripts/qapi/schema.py | 2 +-
5 files changed, 16 insertions(+), 5 deletions(-)
create mode 100644 scripts/qapi/.isort.cfg
diff --git a/scripts/qapi/.isort.cfg b/scripts/qapi/.isort.cfg
new file mode 100644
index 00000000000..6d0fd6cc0d3
--- /dev/null
+++ b/scripts/qapi/.isort.cfg
@@ -0,0 +1,7 @@
+[settings]
+force_grid_wrap=4
+force_sort_within_sections=True
+include_trailing_comma=True
+line_length=72
+lines_after_imports=2
+multi_line_output=3
\ No newline at end of file
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 03b31ecfc19..e73b65b6a7e 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -14,8 +14,9 @@
# This work is licensed under the terms of the GNU GPL, version 2.
# See the COPYING file in the top-level directory.
-import re
from collections import OrderedDict
+import re
+
from .common import c_name
from .error import QAPISemError
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index b036fcf9ce7..31acd2f230a 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -17,8 +17,11 @@
mcgen,
)
from .gen import QAPISchemaMonolithicCVisitor
-from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
- QAPISchemaType)
+from .schema import (
+ QAPISchemaArrayType,
+ QAPISchemaBuiltinType,
+ QAPISchemaType,
+)
def _make_tree(obj, ifcond, features, extra=None):
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index a9388eaf765..a6081a0c5d4 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -14,9 +14,9 @@
# This work is licensed under the terms of the GNU GPL, version 2.
# See the COPYING file in the top-level directory.
+from collections import OrderedDict
import os
import re
-from collections import OrderedDict
from .error import QAPIParseError, QAPISemError
from .source import QAPISourceInfo
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index a835ee6fde3..093f7a38d88 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -14,9 +14,9 @@
# TODO catching name collisions in generated code would be nice
+from collections import OrderedDict
import os
import re
-from collections import OrderedDict
from .common import c_name, pointer_suffix
from .error import QAPIError, QAPISemError
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 12/46] qapi: delint using flake8
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (10 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 11/46] qapi: enforce import order/styling with isort John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 13/46] qapi: add pylintrc John Snow
` (33 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Petty style guide fixes and line length enforcement. Not a big win, not
a big loss, but flake8 passes 100% on the qapi module, which gives us an
easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/.flake8 | 2 ++
scripts/qapi/commands.py | 3 ++-
scripts/qapi/schema.py | 8 +++++---
scripts/qapi/visit.py | 16 +++++++++++-----
4 files changed, 20 insertions(+), 9 deletions(-)
create mode 100644 scripts/qapi/.flake8
diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
new file mode 100644
index 00000000000..6b158c68b84
--- /dev/null
+++ b/scripts/qapi/.flake8
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722 # Prefer pylint's bare-except checks to flake8's
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 64ed5278f93..5dc2f5a9fa8 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type):
def gen_marshal_output(ret_type):
return mcgen('''
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
+ QObject **ret_out, Error **errp)
{
Visitor *v;
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 093f7a38d88..cfc52e1ae44 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -536,7 +536,7 @@ def set_defined_in(self, name):
v.set_defined_in(name)
def check(self, schema, seen):
- if not self.tag_member: # flat union
+ if not self.tag_member: # flat union
self.tag_member = seen.get(c_name(self._tag_name))
base = "'base'"
# Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname):
self._entity_dict = {}
self._module_dict = OrderedDict()
self._schema_dir = os.path.dirname(fname)
- self._make_module(None) # built-ins
+ self._make_module(None) # built-ins
self._make_module(fname)
self._predefining = True
self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
# But it's not tight: the disjunction need not imply it. We
# may end up compiling useless wrapper types.
# TODO kill simple unions or implement the disjunction
- assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
+
+ # pylint: disable=protected-access
+ assert (ifcond or []) == typ._ifcond
else:
self._def_entity(QAPISchemaObjectType(
name, info, None, ifcond, None, None, members, None))
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index ea277e7704b..808410d6f1b 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False):
if not scalar:
c_type += '*'
return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
+
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+ %(c_type)sobj, Error **errp);
''',
c_name=c_name(name), c_type=c_type)
@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants):
def gen_visit_list(name, element_type):
return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+ %(c_name)s **obj, Error **errp)
{
bool ok = false;
%(c_name)s *tail;
@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type):
def gen_visit_enum(name):
return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+ %(c_name)s *obj, Error **errp)
{
int value = *obj;
bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +176,8 @@ def gen_visit_enum(name):
def gen_visit_alternate(name, variants):
ret = mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+ (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
{
bool ok = false;
@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants):
def gen_visit_object(name, base, members, variants):
return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+ (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
{
bool ok = false;
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 13/46] qapi: add pylintrc
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (11 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 12/46] qapi: delint using flake8 John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 14/46] qapi/common.py: Remove python compatibility workaround John Snow
` (32 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Using `pylint --generate-rcfile > pylintrc`, generate a skeleton
pylintrc file. Sections that are not presently relevant (by the end of
this series) are removed leaving just the empty section as a search
engine / documentation hint to future authors.
I am targeting pylint 2.6.0. In the future (and hopefully before 5.2 is
released), I aim to have gitlab CI running the specific targeted
versions of pylint, mypy, flake8, etc in a job.
2.5.x will work if you additionally pass --disable=bad-whitespace.
This warning was removed from 2.6.x, for lack of consistent support.
Right now, quite a few modules are ignored as they are known to fail as
of this commit. modules will be removed from the known-bad list
throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as
that folder is shared by many things, it is going inside the module
folder (for now). Due to a bug in pylint 2.5+, pylint does not
correctly recognize when it is being run from "inside" a package, and
must be run *outside* of the package.
Therefore, to run it, you must:
> pylint scripts/qapi/ --rcfile=scripts/qapi/pylintrc
Signed-off-by: John Snow <jsnow@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/pylintrc | 73 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 73 insertions(+)
create mode 100644 scripts/qapi/pylintrc
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
new file mode 100644
index 00000000000..76d54c30f85
--- /dev/null
+++ b/scripts/qapi/pylintrc
@@ -0,0 +1,73 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the ignore list.
+# The regex matches against base names, not paths.
+ignore-patterns=common.py,
+ error.py,
+ expr.py,
+ gen.py,
+ parser.py,
+ schema.py,
+ source.py,
+ types.py,
+ visit.py,
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=fixme,
+ missing-docstring,
+ too-many-arguments,
+ too-many-branches,
+ too-many-statements,
+ too-many-instance-attributes,
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+ j,
+ k,
+ ex,
+ Run,
+ _
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore import statements themselves when computing similarities.
+ignore-imports=yes
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 14/46] qapi/common.py: Remove python compatibility workaround
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (12 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 13/46] qapi: add pylintrc John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 15/46] qapi/common.py: Add indent manager John Snow
` (31 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/common.py | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index ba35abea478..cee63eb95c7 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
raw = code % kwds
if indent_level:
indent = genindent(indent_level)
- # re.subn() lacks flags support before Python 2.7, use re.compile()
- raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
- indent, raw)
- raw = raw[0]
+ raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
return re.sub(re.escape(eatspace) + r' *', '', raw)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 15/46] qapi/common.py: Add indent manager
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (13 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 14/46] qapi/common.py: Remove python compatibility workaround John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 16/46] qapi/common.py: delint with pylint John Snow
` (30 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Code style tools really dislike the use of global keywords, because it
generally involves re-binding the name at runtime which can have strange
effects depending on when and how that global name is referenced in
other modules.
Make a little indent level manager instead.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++--------------
scripts/qapi/visit.py | 7 +++---
2 files changed, 36 insertions(+), 20 deletions(-)
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index cee63eb95c7..b35318b72cf 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -93,33 +93,50 @@ def c_name(name, protect=True):
pointer_suffix = ' *' + eatspace
-def genindent(count):
- ret = ''
- for _ in range(count):
- ret += ' '
- return ret
+class Indentation:
+ """
+ Indentation level management.
+ :param initial: Initial number of spaces, default 0.
+ """
+ def __init__(self, initial: int = 0) -> None:
+ self._level = initial
-indent_level = 0
+ def __int__(self) -> int:
+ return self._level
+ def __repr__(self) -> str:
+ return "{}({:d})".format(type(self).__name__, self._level)
-def push_indent(indent_amount=4):
- global indent_level
- indent_level += indent_amount
+ def __str__(self) -> str:
+ """Return the current indentation as a string of spaces."""
+ return ' ' * self._level
+ def __bool__(self) -> bool:
+ """True when there is a non-zero indentation."""
+ return bool(self._level)
-def pop_indent(indent_amount=4):
- global indent_level
- indent_level -= indent_amount
+ def increase(self, amount: int = 4) -> None:
+ """Increase the indentation level by ``amount``, default 4."""
+ self._level += amount
+
+ def decrease(self, amount: int = 4) -> None:
+ """Decrease the indentation level by ``amount``, default 4."""
+ if self._level < amount:
+ raise ArithmeticError(
+ f"Can't remove {amount:d} spaces from {self!r}")
+ self._level -= amount
+
+
+indent = Indentation()
# Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey indent, and strip eatspace.
def cgen(code, **kwds):
raw = code % kwds
- if indent_level:
- indent = genindent(indent_level)
- raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+ if indent:
+ raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
return re.sub(re.escape(eatspace) + r' *', '', raw)
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 808410d6f1b..14f30c228b7 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -18,9 +18,8 @@
c_name,
gen_endif,
gen_if,
+ indent,
mcgen,
- pop_indent,
- push_indent,
)
from .gen import QAPISchemaModularCVisitor, ifcontext
from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants):
if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
''',
name=memb.name, c_name=c_name(memb.name))
- push_indent()
+ indent.increase()
ret += mcgen('''
if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
return false;
@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants):
c_type=memb.type.c_name(), name=memb.name,
c_name=c_name(memb.name))
if memb.optional:
- pop_indent()
+ indent.decrease()
ret += mcgen('''
}
''')
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 16/46] qapi/common.py: delint with pylint
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (14 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 15/46] qapi/common.py: Add indent manager John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 17/46] qapi/common.py: Replace one-letter 'c' variable John Snow
` (29 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
At this point, that just means using a consistent strategy for constant names.
constants get UPPER_CASE and names not used externally get a leading underscore.
As a preference, while renaming constants to be UPPERCASE, move them to
the head of the file. Generally, it's nice to be able to audit the code
that runs on import in one central place.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/common.py | 18 ++++++++----------
scripts/qapi/schema.py | 14 +++++++-------
2 files changed, 15 insertions(+), 17 deletions(-)
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index b35318b72cf..a417b6029c8 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -14,6 +14,11 @@
import re
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+_C_NAME_TRANS = str.maketrans('.-', '__')
+
+
# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
# ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
-c_name_trans = str.maketrans('.-', '__')
-
-
# Map @name to a valid C identifier.
# If @protect, avoid returning certain ticklish identifiers (like
# C keywords) by prepending 'q_'.
@@ -82,17 +84,13 @@ def c_name(name, protect=True):
'not_eq', 'or', 'or_eq', 'xor', 'xor_eq'])
# namespace pollution:
polluted_words = set(['unix', 'errno', 'mips', 'sparc', 'i386'])
- name = name.translate(c_name_trans)
+ name = name.translate(_C_NAME_TRANS)
if protect and (name in c89_words | c99_words | c11_words | gcc_words
| cpp_words | polluted_words):
return 'q_' + name
return name
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
class Indentation:
"""
Indentation level management.
@@ -132,12 +130,12 @@ def decrease(self, amount: int = 4) -> None:
# Generate @code with @kwds interpolated.
-# Obey indent, and strip eatspace.
+# Obey indent, and strip EATSPACE.
def cgen(code, **kwds):
raw = code % kwds
if indent:
raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
- return re.sub(re.escape(eatspace) + r' *', '', raw)
+ return re.sub(re.escape(EATSPACE) + r' *', '', raw)
def mcgen(code, **kwds):
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index cfc52e1ae44..74c6b96d391 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,7 +18,7 @@
import os
import re
-from .common import c_name, pointer_suffix
+from .common import POINTER_SUFFIX, c_name
from .error import QAPIError, QAPISemError
from .expr import check_exprs
from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self):
return True
def c_type(self):
- return c_name(self.name) + pointer_suffix
+ return c_name(self.name) + POINTER_SUFFIX
def json_type(self):
return 'array'
@@ -430,7 +430,7 @@ def c_name(self):
def c_type(self):
assert not self.is_implicit()
- return c_name(self.name) + pointer_suffix
+ return c_name(self.name) + POINTER_SUFFIX
def c_unboxed_type(self):
return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None):
v.connect_doc(doc)
def c_type(self):
- return c_name(self.name) + pointer_suffix
+ return c_name(self.name) + POINTER_SUFFIX
def json_type(self):
return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type):
self._make_array_type(name, None)
def _def_predefineds(self):
- for t in [('str', 'string', 'char' + pointer_suffix),
+ for t in [('str', 'string', 'char' + POINTER_SUFFIX),
('number', 'number', 'double'),
('int', 'int', 'int64_t'),
('int8', 'int', 'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self):
('uint64', 'int', 'uint64_t'),
('size', 'int', 'uint64_t'),
('bool', 'boolean', 'bool'),
- ('any', 'value', 'QObject' + pointer_suffix),
- ('null', 'null', 'QNull' + pointer_suffix)]:
+ ('any', 'value', 'QObject' + POINTER_SUFFIX),
+ ('null', 'null', 'QNull' + POINTER_SUFFIX)]:
self._def_builtin_type(*t)
self.the_empty_object_type = QAPISchemaObjectType(
'q_empty', None, None, None, None, None, [], None)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 17/46] qapi/common.py: Replace one-letter 'c' variable
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (15 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 16/46] qapi/common.py: delint with pylint John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 18/46] qapi/common.py: check with pylint John Snow
` (28 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
---
scripts/qapi/common.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index a417b6029c8..338adedef4f 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -30,14 +30,14 @@ def camel_to_upper(value):
new_name = ''
length = len(c_fun_str)
for i in range(length):
- c = c_fun_str[i]
- # When c is upper and no '_' appears before, do more checks
- if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+ char = c_fun_str[i]
+ # When char is upper case and no '_' appears before, do more checks
+ if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
if i < length - 1 and c_fun_str[i + 1].islower():
new_name += '_'
elif c_fun_str[i - 1].isdigit():
new_name += '_'
- new_name += c
+ new_name += char
return new_name.lstrip('_').upper()
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 18/46] qapi/common.py: check with pylint
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (16 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 17/46] qapi/common.py: Replace one-letter 'c' variable John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 19/46] qapi/common.py: add type hint annotations John Snow
` (27 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Remove qapi/common.py from the pylintrc ignore list.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
---
scripts/qapi/pylintrc | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 76d54c30f85..507f15537ab 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -2,8 +2,7 @@
# Add files or directories matching the regex patterns to the ignore list.
# The regex matches against base names, not paths.
-ignore-patterns=common.py,
- error.py,
+ignore-patterns=error.py,
expr.py,
gen.py,
parser.py,
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 19/46] qapi/common.py: add type hint annotations
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (17 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 18/46] qapi/common.py: check with pylint John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 20/46] qapi/common.py: Convert comments into docstrings, and elaborate John Snow
` (26 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Annotations do not change runtime behavior.
This commit *only* adds annotations.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/common.py | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 338adedef4f..74a2c001ed9 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -12,6 +12,7 @@
# See the COPYING file in the top-level directory.
import re
+from typing import Optional, Sequence
EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
# ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
c_fun_str = c_name(value, False)
if value.isupper():
return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
return new_name.lstrip('_').upper()
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+ const_name: str,
+ prefix: Optional[str] = None) -> str:
if prefix is not None:
type_name = prefix
return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
# into substrings of a generated C function name.
# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
# ANSI X3J11/88-090, 3.1.1
c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None:
# Generate @code with @kwds interpolated.
# Obey indent, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: object) -> str:
raw = code % kwds
if indent:
raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
return re.sub(re.escape(EATSPACE) + r' *', '', raw)
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: object) -> str:
if code[0] == '\n':
code = code[1:]
return cgen(code, **kwds)
-def c_fname(filename):
+def c_fname(filename: str) -> str:
return re.sub(r'[^A-Za-z0-9_]', '_', filename)
-def guardstart(name):
+def guardstart(name: str) -> str:
return mcgen('''
#ifndef %(name)s
#define %(name)s
@@ -157,7 +160,7 @@ def guardstart(name):
name=c_fname(name).upper())
-def guardend(name):
+def guardend(name: str) -> str:
return mcgen('''
#endif /* %(name)s */
@@ -165,7 +168,7 @@ def guardend(name):
name=c_fname(name).upper())
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
ret = ''
for ifc in ifcond:
ret += mcgen('''
@@ -174,7 +177,7 @@ def gen_if(ifcond):
return ret
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
ret = ''
for ifc in reversed(ifcond):
ret += mcgen('''
@@ -183,7 +186,9 @@ def gen_endif(ifcond):
return ret
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+ boxed: bool,
+ extra: Optional[str] = None) -> str:
ret = ''
sep = ''
if boxed:
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 20/46] qapi/common.py: Convert comments into docstrings, and elaborate
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (18 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 19/46] qapi/common.py: add type hint annotations John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 21/46] qapi/common.py: move build_params into gen.py John Snow
` (25 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
As docstrings, they'll show up in documentation and IDE help.
The docstring style being targeted is the Sphinx documentation
style. Sphinx uses an extension of ReST with "domains". We use the
(implicit) Python domain, which supports a number of custom "info
fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Primarily, we use `:param X: descr`, `:return[s]: descr`, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.
(No, nothing checks or enforces this style that I am aware of. Sphinx
either chokes or succeeds, but does not enforce a standard of what is
otherwise inside the docstring. Pycharm does highlight when your param
fields are not aligned with the actual fields present. It does not
highlight missing return or exception statements. There is no existing
style guide I am aware of that covers a standard for a minimally
acceptable docstring. I am debating writing one.)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
1 file changed, 39 insertions(+), 14 deletions(-)
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 74a2c001ed9..0ef38ea5fe0 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -15,15 +15,24 @@
from typing import Optional, Sequence
+#: Sentinel value that causes all space to its right to be removed.
EATSPACE = '\033EATSPACE.'
POINTER_SUFFIX = ' *' + EATSPACE
_C_NAME_TRANS = str.maketrans('.-', '__')
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
def camel_to_upper(value: str) -> str:
+ """
+ Converts CamelCase to CAMEL_CASE.
+
+ Examples:
+ ENUMName -> ENUM_NAME
+ EnumName1 -> ENUM_NAME1
+ ENUM_NAME -> ENUM_NAME
+ ENUM_NAME1 -> ENUM_NAME1
+ ENUM_Name2 -> ENUM_NAME2
+ ENUM24_Name -> ENUM24_NAME
+ """
c_fun_str = c_name(value, False)
if value.isupper():
return c_fun_str
@@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str:
def c_enum_const(type_name: str,
const_name: str,
prefix: Optional[str] = None) -> str:
+ """
+ Generate a C enumeration constant name.
+
+ :param type_name: The name of the enumeration.
+ :param const_name: The name of this constant.
+ :param prefix: Optional, prefix that overrides the type_name.
+ """
if prefix is not None:
type_name = prefix
return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
def c_name(name: str, protect: bool = True) -> str:
+ """
+ Map ``name`` to a valid C identifier.
+
+ Used for converting 'name' from a 'name':'type' qapi definition
+ into a generated struct member, as well as converting type names
+ into substrings of a generated C function name.
+
+ '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+ protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+ :param name: The name to map.
+ :param protect: If true, avoid returning certain ticklish identifiers
+ (like C keywords) by prepending ``q_``.
+ """
# ANSI X3J11/88-090, 3.1.1
c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None:
self._level -= amount
+#: Global, current indent level for code generation.
indent = Indentation()
-# Generate @code with @kwds interpolated.
-# Obey indent, and strip EATSPACE.
def cgen(code: str, **kwds: object) -> str:
+ """
+ Generate ``code`` with ``kwds`` interpolated.
+
+ Obey `indent`, and strip `EATSPACE`.
+ """
raw = code % kwds
if indent:
raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 21/46] qapi/common.py: move build_params into gen.py
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (19 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 20/46] qapi/common.py: Convert comments into docstrings, and elaborate John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 22/46] qapi: establish mypy type-checking baseline John Snow
` (24 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Including it in common.py creates a circular import dependency; schema
relies on common, but common.build_params requires a type annotation
from schema. To type this properly, it needs to be moved outside the
cycle.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/commands.py | 9 +++++++--
scripts/qapi/common.py | 23 -----------------------
scripts/qapi/events.py | 9 ++-------
scripts/qapi/gen.py | 31 +++++++++++++++++++++++++++++--
4 files changed, 38 insertions(+), 34 deletions(-)
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 5dc2f5a9fa8..f67393f8713 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,8 +13,13 @@
See the COPYING file in the top-level directory.
"""
-from .common import build_params, c_name, mcgen
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import c_name, mcgen
+from .gen import (
+ QAPIGenCCode,
+ QAPISchemaModularCVisitor,
+ build_params,
+ ifcontext,
+)
def gen_command_decl(name, arg_type, boxed, ret_type):
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 0ef38ea5fe0..9ab0685cc51 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
#endif /* %(cond)s */
''', cond=ifc)
return ret
-
-
-def build_params(arg_type,
- boxed: bool,
- extra: Optional[str] = None) -> str:
- ret = ''
- sep = ''
- if boxed:
- assert arg_type
- ret += '%s arg' % arg_type.c_param_type()
- sep = ', '
- elif arg_type:
- assert not arg_type.variants
- for memb in arg_type.members:
- ret += sep
- sep = ', '
- if memb.optional:
- ret += 'bool has_%s, ' % c_name(memb.name)
- ret += '%s %s' % (memb.type.c_param_type(),
- c_name(memb.name))
- if extra:
- ret += sep + extra
- return ret if ret else 'void'
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 6b3afa14d72..f840a62ed92 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,13 +12,8 @@
See the COPYING file in the top-level directory.
"""
-from .common import (
- build_params,
- c_enum_const,
- c_name,
- mcgen,
-)
-from .gen import QAPISchemaModularCVisitor, ifcontext
+from .common import c_enum_const, c_name, mcgen
+from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
from .schema import QAPISchemaEnumMember
from .types import gen_enum, gen_enum_lookup
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 61b3c53b180..b8a6fe8a5c3 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -2,9 +2,11 @@
#
# QAPI code generation
#
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2019 Red Hat Inc.
#
# Authors:
+# Anthony Liguori <aliguori@us.ibm.com>
# Markus Armbruster <armbru@redhat.com>
# Marc-André Lureau <marcandre.lureau@redhat.com>
#
@@ -15,16 +17,18 @@
import errno
import os
import re
+from typing import Optional
from .common import (
c_fname,
+ c_name,
gen_endif,
gen_if,
guardend,
guardstart,
mcgen,
)
-from .schema import QAPISchemaVisitor
+from .schema import QAPISchemaObjectType, QAPISchemaVisitor
class QAPIGen:
@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
return out
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ extra: Optional[str] = None) -> str:
+ ret = ''
+ sep = ''
+ if boxed:
+ assert arg_type
+ ret += '%s arg' % arg_type.c_param_type()
+ sep = ', '
+ elif arg_type:
+ assert not arg_type.variants
+ for memb in arg_type.members:
+ ret += sep
+ sep = ', '
+ if memb.optional:
+ ret += 'bool has_%s, ' % c_name(memb.name)
+ ret += '%s %s' % (memb.type.c_param_type(),
+ c_name(memb.name))
+ if extra:
+ ret += sep + extra
+ return ret if ret else 'void'
+
+
class QAPIGenCCode(QAPIGen):
def __init__(self, fname):
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 22/46] qapi: establish mypy type-checking baseline
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (20 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 21/46] qapi/common.py: move build_params into gen.py John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 23/46] qapi/events.py: add type hint annotations John Snow
` (23 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Fix two very minor issues, and then establish a mypy type-checking
baseline.
Like pylint, this should be run from the folder above:
> mypy --config-file=qapi/mypy.ini qapi/
This is designed and tested for mypy 0.770 or greater.
Signed-off-by: John Snow <jsnow@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/mypy.ini | 60 ++++++++++++++++++++++++++++++++++++++++++
scripts/qapi/schema.py | 3 ++-
2 files changed, 62 insertions(+), 1 deletion(-)
create mode 100644 scripts/qapi/mypy.ini
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
new file mode 100644
index 00000000000..00fac15dc6e
--- /dev/null
+++ b/scripts/qapi/mypy.ini
@@ -0,0 +1,60 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 74c6b96d391..483b4b68dff 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -17,6 +17,7 @@
from collections import OrderedDict
import os
import re
+from typing import Optional
from .common import POINTER_SUFFIX, c_name
from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
class QAPISchemaEntity:
- meta = None
+ meta: Optional[str] = None
def __init__(self, name, info, doc, ifcond=None, features=None):
assert name is None or isinstance(name, str)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 23/46] qapi/events.py: add type hint annotations
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (21 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 22/46] qapi: establish mypy type-checking baseline John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 24/46] qapi/events.py: Move comments into docstrings John Snow
` (22 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Annotations do not change runtime behavior.
This commit *only* adds annotations.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
scripts/qapi/mypy.ini | 5 -----
2 files changed, 35 insertions(+), 16 deletions(-)
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index f840a62ed92..57e0939e963 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,19 +12,31 @@
See the COPYING file in the top-level directory.
"""
+from typing import List
+
from .common import c_enum_const, c_name, mcgen
from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
-from .schema import QAPISchemaEnumMember
+from .schema import (
+ QAPISchema,
+ QAPISchemaEnumMember,
+ QAPISchemaFeature,
+ QAPISchemaObjectType,
+)
+from .source import QAPISourceInfo
from .types import gen_enum, gen_enum_lookup
-def build_event_send_proto(name, arg_type, boxed):
+def build_event_send_proto(name: str,
+ arg_type: QAPISchemaObjectType,
+ boxed: bool) -> str:
return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
'c_name': c_name(name.lower()),
'param': build_params(arg_type, boxed)}
-def gen_event_send_decl(name, arg_type, boxed):
+def gen_event_send_decl(name: str,
+ arg_type: QAPISchemaObjectType,
+ boxed: bool) -> str:
return mcgen('''
%(proto)s;
@@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed):
# Declare and initialize an object 'qapi' using parameters from build_params()
-def gen_param_var(typ):
+def gen_param_var(typ: QAPISchemaObjectType) -> str:
assert not typ.variants
ret = mcgen('''
%(c_name)s param = {
@@ -61,7 +73,11 @@ def gen_param_var(typ):
return ret
-def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
+def gen_event_send(name: str,
+ arg_type: QAPISchemaObjectType,
+ boxed: bool,
+ event_enum_name: str,
+ event_emit: str) -> str:
# FIXME: Our declaration of local variables (and of 'errp' in the
# parameter list) can collide with exploded members of the event's
# data type passed in as parameters. If this collision ever hits in
@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
- def __init__(self, prefix):
+ def __init__(self, prefix: str):
super().__init__(
prefix, 'qapi-events',
' * Schema-defined QAPI/QMP events', None, __doc__)
self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
- self._event_enum_members = []
+ self._event_enum_members: List[QAPISchemaEnumMember] = []
self._event_emit_name = c_name(prefix + 'qapi_event_emit')
- def _begin_user_module(self, name):
+ def _begin_user_module(self, name: str) -> None:
events = self._module_basename('qapi-events', name)
types = self._module_basename('qapi-types', name)
visit = self._module_basename('qapi-visit', name)
@@ -168,7 +184,7 @@ def _begin_user_module(self, name):
''',
types=types))
- def visit_end(self):
+ def visit_end(self) -> None:
self._add_system_module('emit', ' * QAPI Events emission')
self._genc.preamble_add(mcgen('''
#include "qemu/osdep.h"
@@ -189,7 +205,13 @@ def visit_end(self):
event_emit=self._event_emit_name,
event_enum=self._event_enum_name))
- def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+ def visit_event(self,
+ name: str,
+ info: QAPISourceInfo,
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ arg_type: QAPISchemaObjectType,
+ boxed: bool) -> None:
with ifcontext(ifcond, self._genh, self._genc):
self._genh.add(gen_event_send_decl(name, arg_type, boxed))
self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -200,7 +222,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed):
self._event_enum_members.append(QAPISchemaEnumMember(name, None))
-def gen_events(schema, output_dir, prefix):
+def gen_events(schema: QAPISchema,
+ output_dir: str,
+ prefix: str) -> None:
vis = QAPISchemaGenEventVisitor(prefix)
schema.visit(vis)
vis.write(output_dir)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 00fac15dc6e..5df11e53fd1 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -14,11 +14,6 @@ disallow_untyped_defs = False
disallow_incomplete_defs = False
check_untyped_defs = False
-[mypy-qapi.events]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
[mypy-qapi.expr]
disallow_untyped_defs = False
disallow_incomplete_defs = False
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 24/46] qapi/events.py: Move comments into docstrings
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (22 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 23/46] qapi/events.py: add type hint annotations John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 25/46] qapi/commands.py: Don't re-bind to variable of different type John Snow
` (21 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Clarify them while we're here.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/events.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 57e0939e963..599f3d1f564 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -44,8 +44,12 @@ def gen_event_send_decl(name: str,
proto=build_event_send_proto(name, arg_type, boxed))
-# Declare and initialize an object 'qapi' using parameters from build_params()
def gen_param_var(typ: QAPISchemaObjectType) -> str:
+ """
+ Generate a struct variable holding the event parameters.
+
+ Initialize it with the function arguments defined in `gen_event_send`.
+ """
assert not typ.variants
ret = mcgen('''
%(c_name)s param = {
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 25/46] qapi/commands.py: Don't re-bind to variable of different type
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (23 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 24/46] qapi/events.py: Move comments into docstrings John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 26/46] qapi/commands.py: add type hint annotations John Snow
` (20 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Mypy isn't a fan of rebinding a variable with a new data type.
It's easy enough to avoid.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/commands.py | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index f67393f8713..586774a23c7 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -195,14 +195,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
if not options:
options = ['QCO_NO_OPTIONS']
- options = " | ".join(options)
-
ret = mcgen('''
qmp_register_command(cmds, "%(name)s",
qmp_marshal_%(c_name)s, %(opts)s);
''',
name=name, c_name=c_name(name),
- opts=options)
+ opts=" | ".join(options))
return ret
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 26/46] qapi/commands.py: add type hint annotations
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (24 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 25/46] qapi/commands.py: Don't re-bind to variable of different type John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 27/46] qapi/commands.py: enable checking with mypy John Snow
` (19 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Annotations do not change runtime behavior.
This commit *only* adds annotations.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/commands.py | 71 ++++++++++++++++++++++++++++++----------
1 file changed, 54 insertions(+), 17 deletions(-)
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 586774a23c7..24f347a3f12 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,16 +13,34 @@
See the COPYING file in the top-level directory.
"""
+from typing import (
+ Dict,
+ List,
+ Optional,
+ Set,
+)
+
from .common import c_name, mcgen
from .gen import (
+ QAPIGenC,
QAPIGenCCode,
QAPISchemaModularCVisitor,
build_params,
ifcontext,
)
+from .schema import (
+ QAPISchema,
+ QAPISchemaFeature,
+ QAPISchemaObjectType,
+ QAPISchemaType,
+)
+from .source import QAPISourceInfo
-def gen_command_decl(name, arg_type, boxed, ret_type):
+def gen_command_decl(name: str,
+ arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ ret_type: Optional[QAPISchemaType]) -> str:
return mcgen('''
%(c_type)s qmp_%(c_name)s(%(params)s);
''',
@@ -31,7 +49,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type):
params=build_params(arg_type, boxed, 'Error **errp'))
-def gen_call(name, arg_type, boxed, ret_type):
+def gen_call(name: str,
+ arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ ret_type: Optional[QAPISchemaType]) -> str:
ret = ''
argstr = ''
@@ -67,7 +88,7 @@ def gen_call(name, arg_type, boxed, ret_type):
return ret
-def gen_marshal_output(ret_type):
+def gen_marshal_output(ret_type: QAPISchemaType) -> str:
return mcgen('''
static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
@@ -88,19 +109,22 @@ def gen_marshal_output(ret_type):
c_type=ret_type.c_type(), c_name=ret_type.c_name())
-def build_marshal_proto(name):
+def build_marshal_proto(name: str) -> str:
return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)'
% c_name(name))
-def gen_marshal_decl(name):
+def gen_marshal_decl(name: str) -> str:
return mcgen('''
%(proto)s;
''',
proto=build_marshal_proto(name))
-def gen_marshal(name, arg_type, boxed, ret_type):
+def gen_marshal(name: str,
+ arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ ret_type: Optional[QAPISchemaType]) -> str:
have_args = boxed or (arg_type and not arg_type.is_empty())
ret = mcgen('''
@@ -182,7 +206,10 @@ def gen_marshal(name, arg_type, boxed, ret_type):
return ret
-def gen_register_command(name, success_response, allow_oob, allow_preconfig):
+def gen_register_command(name: str,
+ success_response: bool,
+ allow_oob: bool,
+ allow_preconfig: bool) -> str:
options = []
if not success_response:
@@ -204,7 +231,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
return ret
-def gen_registry(registry, prefix):
+def gen_registry(registry: str, prefix: str) -> str:
ret = mcgen('''
void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -221,15 +248,14 @@ def gen_registry(registry, prefix):
class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
-
- def __init__(self, prefix):
+ def __init__(self, prefix: str):
super().__init__(
prefix, 'qapi-commands',
' * Schema-defined QAPI/QMP commands', None, __doc__)
self._regy = QAPIGenCCode(None)
- self._visited_ret_types = {}
+ self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {}
- def _begin_user_module(self, name):
+ def _begin_user_module(self, name: str) -> None:
self._visited_ret_types[self._genc] = set()
commands = self._module_basename('qapi-commands', name)
types = self._module_basename('qapi-types', name)
@@ -253,7 +279,7 @@ def _begin_user_module(self, name):
''',
types=types))
- def visit_end(self):
+ def visit_end(self) -> None:
self._add_system_module('init', ' * QAPI Commands initialization')
self._genh.add(mcgen('''
#include "qapi/qmp/dispatch.h"
@@ -269,9 +295,18 @@ def visit_end(self):
prefix=self._prefix))
self._genc.add(gen_registry(self._regy.get_content(), self._prefix))
- def visit_command(self, name, info, ifcond, features,
- arg_type, ret_type, gen, success_response, boxed,
- allow_oob, allow_preconfig):
+ def visit_command(self,
+ name: str,
+ info: QAPISourceInfo,
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ arg_type: Optional[QAPISchemaObjectType],
+ ret_type: Optional[QAPISchemaType],
+ gen: bool,
+ success_response: bool,
+ boxed: bool,
+ allow_oob: bool,
+ allow_preconfig: bool) -> None:
if not gen:
return
# FIXME: If T is a user-defined type, the user is responsible
@@ -292,7 +327,9 @@ def visit_command(self, name, info, ifcond, features,
allow_oob, allow_preconfig))
-def gen_commands(schema, output_dir, prefix):
+def gen_commands(schema: QAPISchema,
+ output_dir: str,
+ prefix: str) -> None:
vis = QAPISchemaGenCommandVisitor(prefix)
schema.visit(vis)
vis.write(output_dir)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 27/46] qapi/commands.py: enable checking with mypy
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (25 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 26/46] qapi/commands.py: add type hint annotations John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 28/46] qapi/source.py: add type hint annotations John Snow
` (18 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/mypy.ini | 5 -----
1 file changed, 5 deletions(-)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 5df11e53fd1..8ab9ac52cc4 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -4,11 +4,6 @@ strict_optional = False
disallow_untyped_calls = False
python_version = 3.6
-[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
[mypy-qapi.error]
disallow_untyped_defs = False
disallow_incomplete_defs = False
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 28/46] qapi/source.py: add type hint annotations
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (26 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 27/46] qapi/commands.py: enable checking with mypy John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 29/46] qapi/source.py: delint with pylint John Snow
` (17 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Annotations do not change runtime behavior.
This commit *only* adds annotations.
A note on typing of __init__: mypy requires init functions with no
parameters to document a return type of None to be considered fully
typed. In the case when there are input parameters, None may be omitted.
Since __init__ may never return any value, it is preferred to omit the
return annotation whenever possible.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/mypy.ini | 5 -----
scripts/qapi/source.py | 31 ++++++++++++++++++-------------
2 files changed, 18 insertions(+), 18 deletions(-)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 8ab9ac52cc4..1b8555dfa39 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -34,11 +34,6 @@ disallow_untyped_defs = False
disallow_incomplete_defs = False
check_untyped_defs = False
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
[mypy-qapi.types]
disallow_untyped_defs = False
disallow_incomplete_defs = False
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index e97b9a8e15e..1cc6a5b82dc 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -11,37 +11,42 @@
import copy
import sys
+from typing import List, Optional, TypeVar
class QAPISchemaPragma:
- def __init__(self):
+ def __init__(self) -> None:
# Are documentation comments required?
self.doc_required = False
# Whitelist of commands allowed to return a non-dictionary
- self.returns_whitelist = []
+ self.returns_whitelist: List[str] = []
# Whitelist of entities allowed to violate case conventions
- self.name_case_whitelist = []
+ self.name_case_whitelist: List[str] = []
class QAPISourceInfo:
- def __init__(self, fname, line, parent):
+ T = TypeVar('T', bound='QAPISourceInfo')
+
+ def __init__(self: T, fname: str, line: int, parent: Optional[T]):
self.fname = fname
self.line = line
self.parent = parent
- self.pragma = parent.pragma if parent else QAPISchemaPragma()
- self.defn_meta = None
- self.defn_name = None
+ self.pragma: QAPISchemaPragma = (
+ parent.pragma if parent else QAPISchemaPragma()
+ )
+ self.defn_meta: Optional[str] = None
+ self.defn_name: Optional[str] = None
- def set_defn(self, meta, name):
+ def set_defn(self, meta: str, name: str) -> None:
self.defn_meta = meta
self.defn_name = name
- def next_line(self):
+ def next_line(self: T) -> T:
info = copy.copy(self)
info.line += 1
return info
- def loc(self):
+ def loc(self) -> str:
if self.fname is None:
return sys.argv[0]
ret = self.fname
@@ -49,13 +54,13 @@ def loc(self):
ret += ':%d' % self.line
return ret
- def in_defn(self):
+ def in_defn(self) -> str:
if self.defn_name:
return "%s: In %s '%s':\n" % (self.fname,
self.defn_meta, self.defn_name)
return ''
- def include_path(self):
+ def include_path(self) -> str:
ret = ''
parent = self.parent
while parent:
@@ -63,5 +68,5 @@ def include_path(self):
parent = parent.parent
return ret
- def __str__(self):
+ def __str__(self) -> str:
return self.include_path() + self.in_defn() + self.loc()
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 29/46] qapi/source.py: delint with pylint
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (27 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 28/46] qapi/source.py: add type hint annotations John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 30/46] qapi/gen.py: Fix edge-case of _is_user_module John Snow
` (16 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Shush an error and leave a hint for future cleanups when we're allowed
to use Python 3.7+.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/pylintrc | 1 -
scripts/qapi/source.py | 3 +++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 507f15537ab..d840b150313 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -7,7 +7,6 @@ ignore-patterns=error.py,
gen.py,
parser.py,
schema.py,
- source.py,
types.py,
visit.py,
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index 1cc6a5b82dc..ba991d798fe 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -15,6 +15,9 @@
class QAPISchemaPragma:
+ # Replace with @dataclass in Python 3.7+
+ # pylint: disable=too-few-public-methods
+
def __init__(self) -> None:
# Are documentation comments required?
self.doc_required = False
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 30/46] qapi/gen.py: Fix edge-case of _is_user_module
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (28 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 29/46] qapi/source.py: delint with pylint John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 11:03 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 31/46] qapi/gen.py: add type hint annotations John Snow
` (15 subsequent siblings)
45 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
The edge case is that if the name is '', this expression returns a
string instead of a bool, which violates our declared type.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/gen.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index b8a6fe8a5c3..58cc1d8e734 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
@staticmethod
def _is_user_module(name):
- return name and not name.startswith('./')
+ return bool(name and not name.startswith('./'))
@staticmethod
def _is_builtin_module(name):
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 31/46] qapi/gen.py: add type hint annotations
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (29 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 30/46] qapi/gen.py: Fix edge-case of _is_user_module John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 32/46] qapi/gen.py: Enable checking with mypy John Snow
` (14 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Annotations do not change runtime behavior.
This commit *only* adds annotations.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
1 file changed, 57 insertions(+), 47 deletions(-)
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 58cc1d8e734..4c6bba940ed 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -17,7 +17,13 @@
import errno
import os
import re
-from typing import Optional
+from typing import (
+ Dict,
+ Iterator,
+ List,
+ Optional,
+ Tuple,
+)
from .common import (
c_fname,
@@ -29,31 +35,31 @@
mcgen,
)
from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo
class QAPIGen:
-
- def __init__(self, fname):
+ def __init__(self, fname: Optional[str]):
self.fname = fname
self._preamble = ''
self._body = ''
- def preamble_add(self, text):
+ def preamble_add(self, text: str) -> None:
self._preamble += text
- def add(self, text):
+ def add(self, text: str) -> None:
self._body += text
- def get_content(self):
+ def get_content(self) -> str:
return self._top() + self._preamble + self._body + self._bottom()
- def _top(self):
+ def _top(self) -> str:
return ''
- def _bottom(self):
+ def _bottom(self) -> str:
return ''
- def write(self, output_dir):
+ def write(self, output_dir: str) -> None:
# Include paths starting with ../ are used to reuse modules of the main
# schema in specialised schemas. Don't overwrite the files that are
# already generated for the main schema.
@@ -78,7 +84,7 @@ def write(self, output_dir):
f.close()
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
if before == after:
return after # suppress empty #if ... #endif
@@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
class QAPIGenCCode(QAPIGen):
-
- def __init__(self, fname):
+ def __init__(self, fname: Optional[str]):
super().__init__(fname)
- self._start_if = None
+ self._start_if: Optional[Tuple[List[str], str, str]] = None
- def start_if(self, ifcond):
+ def start_if(self, ifcond: List[str]) -> None:
assert self._start_if is None
self._start_if = (ifcond, self._body, self._preamble)
- def end_if(self):
+ def end_if(self) -> None:
assert self._start_if
self._wrap_ifcond()
self._start_if = None
- def _wrap_ifcond(self):
+ def _wrap_ifcond(self) -> None:
self._body = _wrap_ifcond(self._start_if[0],
self._start_if[1], self._body)
self._preamble = _wrap_ifcond(self._start_if[0],
self._start_if[2], self._preamble)
- def get_content(self):
+ def get_content(self) -> str:
assert self._start_if is None
return super().get_content()
class QAPIGenC(QAPIGenCCode):
-
- def __init__(self, fname, blurb, pydoc):
+ def __init__(self, fname: str, blurb: str, pydoc: str):
super().__init__(fname)
self._blurb = blurb
self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
re.MULTILINE))
- def _top(self):
+ def _top(self) -> str:
return mcgen('''
/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
@@ -167,7 +171,7 @@ def _top(self):
''',
blurb=self._blurb, copyright=self._copyright)
- def _bottom(self):
+ def _bottom(self) -> str:
return mcgen('''
/* Dummy declaration to prevent empty .o file */
@@ -177,16 +181,15 @@ def _bottom(self):
class QAPIGenH(QAPIGenC):
-
- def _top(self):
+ def _top(self) -> str:
return super()._top() + guardstart(self.fname)
- def _bottom(self):
+ def _bottom(self) -> str:
return guardend(self.fname)
@contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:
"""
A 'with' statement context manager that wraps with `start_if` and `end_if`.
@@ -214,8 +217,11 @@ def ifcontext(ifcond, *args):
class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
- def __init__(self, prefix, what, blurb, pydoc):
+ def __init__(self,
+ prefix: str,
+ what: str,
+ blurb: str,
+ pydoc: str):
self._prefix = prefix
self._what = what
self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc):
self._genh = QAPIGenH(self._prefix + self._what + '.h',
blurb, pydoc)
- def write(self, output_dir):
+ def write(self, output_dir: str) -> None:
self._genc.write(output_dir)
self._genh.write(output_dir)
class QAPISchemaModularCVisitor(QAPISchemaVisitor):
-
- def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
+ def __init__(self,
+ prefix: str,
+ what: str,
+ user_blurb: str,
+ builtin_blurb: Optional[str],
+ pydoc: str):
self._prefix = prefix
self._what = what
self._user_blurb = user_blurb
self._builtin_blurb = builtin_blurb
self._pydoc = pydoc
- self._genc = None
- self._genh = None
- self._module = {}
- self._main_module = None
+ self._genc: Optional[QAPIGenC] = None
+ self._genh: Optional[QAPIGenH] = None
+ self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
+ self._main_module: Optional[str] = None
@staticmethod
- def _is_user_module(name):
+ def _is_user_module(name: Optional[str]) -> bool:
return bool(name and not name.startswith('./'))
@staticmethod
- def _is_builtin_module(name):
+ def _is_builtin_module(name: Optional[str]) -> bool:
return not name
- def _module_dirname(self, what, name):
+ def _module_dirname(self, what: str, name: Optional[str]) -> str:
if self._is_user_module(name):
return os.path.dirname(name)
return ''
- def _module_basename(self, what, name):
+ def _module_basename(self, what: str, name: Optional[str]) -> str:
ret = '' if self._is_builtin_module(name) else self._prefix
if self._is_user_module(name):
basename = os.path.basename(name)
@@ -266,27 +276,27 @@ def _module_basename(self, what, name):
ret += re.sub(r'-', '-' + name + '-', what)
return ret
- def _module_filename(self, what, name):
+ def _module_filename(self, what: str, name: Optional[str]) -> str:
return os.path.join(self._module_dirname(what, name),
self._module_basename(what, name))
- def _add_module(self, name, blurb):
+ def _add_module(self, name: Optional[str], blurb: str) -> None:
basename = self._module_filename(self._what, name)
genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
self._module[name] = (genc, genh)
self._genc, self._genh = self._module[name]
- def _add_user_module(self, name, blurb):
+ def _add_user_module(self, name: str, blurb: str) -> None:
assert self._is_user_module(name)
if self._main_module is None:
self._main_module = name
self._add_module(name, blurb)
- def _add_system_module(self, name, blurb):
+ def _add_system_module(self, name: Optional[str], blurb: str) -> None:
self._add_module(name and './' + name, blurb)
- def write(self, output_dir, opt_builtins=False):
+ def write(self, output_dir: str, opt_builtins: bool = False) -> None:
for name in self._module:
if self._is_builtin_module(name) and not opt_builtins:
continue
@@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False):
genc.write(output_dir)
genh.write(output_dir)
- def _begin_system_module(self, name):
+ def _begin_system_module(self, name: None) -> None:
pass
- def _begin_user_module(self, name):
+ def _begin_user_module(self, name: str) -> None:
pass
- def visit_module(self, name):
+ def visit_module(self, name: Optional[str]) -> None:
if name is None:
if self._builtin_blurb:
self._add_system_module(None, self._builtin_blurb)
@@ -314,7 +324,7 @@ def visit_module(self, name):
self._add_user_module(name, self._user_blurb)
self._begin_user_module(name)
- def visit_include(self, name, info):
+ def visit_include(self, name: str, info: QAPISourceInfo) -> None:
relname = os.path.relpath(self._module_filename(self._what, name),
os.path.dirname(self._genh.fname))
self._genh.preamble_add(mcgen('''
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 32/46] qapi/gen.py: Enable checking with mypy
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (30 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 31/46] qapi/gen.py: add type hint annotations John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 33/46] qapi/gen.py: Remove unused parameter John Snow
` (13 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/mypy.ini | 5 -----
1 file changed, 5 deletions(-)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 1b8555dfa39..c6960ff2dbd 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -14,11 +14,6 @@ disallow_untyped_defs = False
disallow_incomplete_defs = False
check_untyped_defs = False
-[mypy-qapi.gen]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
[mypy-qapi.introspect]
disallow_untyped_defs = False
disallow_incomplete_defs = False
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 33/46] qapi/gen.py: Remove unused parameter
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (31 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 32/46] qapi/gen.py: Enable checking with mypy John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 34/46] qapi/gen.py: update write() to be more idiomatic John Snow
` (12 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
module_basename doesn't use the 'what' argument, so remove it.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/gen.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 4c6bba940ed..5694a00062c 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool:
def _is_builtin_module(name: Optional[str]) -> bool:
return not name
- def _module_dirname(self, what: str, name: Optional[str]) -> str:
+ def _module_dirname(self, name: Optional[str]) -> str:
if self._is_user_module(name):
return os.path.dirname(name)
return ''
@@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
return ret
def _module_filename(self, what: str, name: Optional[str]) -> str:
- return os.path.join(self._module_dirname(what, name),
+ return os.path.join(self._module_dirname(name),
self._module_basename(what, name))
def _add_module(self, name: Optional[str], blurb: str) -> None:
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 34/46] qapi/gen.py: update write() to be more idiomatic
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (32 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 33/46] qapi/gen.py: Remove unused parameter John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 35/46] qapi/gen.py: delint with pylint John Snow
` (11 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Make the file handling here just a tiny bit more idiomatic.
(I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception,
use fdopen() to wrap the file descriptor in a File-like object,
and use a context manager for managing the file pointer.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/gen.py | 25 +++++++++++--------------
1 file changed, 11 insertions(+), 14 deletions(-)
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 5694a00062c..2ad96e396e1 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -14,7 +14,6 @@
# See the COPYING file in the top-level directory.
from contextlib import contextmanager
-import errno
import os
import re
from typing import (
@@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None:
return
pathname = os.path.join(output_dir, self.fname)
odir = os.path.dirname(pathname)
+
if odir:
- try:
- os.makedirs(odir)
- except os.error as e:
- if e.errno != errno.EEXIST:
- raise
+ os.makedirs(odir, exist_ok=True)
+
+ # use os.open for O_CREAT to create and read a non-existant file
fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
- f = open(fd, 'r+', encoding='utf-8')
- text = self.get_content()
- oldtext = f.read(len(text) + 1)
- if text != oldtext:
- f.seek(0)
- f.truncate(0)
- f.write(text)
- f.close()
+ with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+ text = self.get_content()
+ oldtext = fp.read(len(text) + 1)
+ if text != oldtext:
+ fp.seek(0)
+ fp.truncate(0)
+ fp.write(text)
def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 35/46] qapi/gen.py: delint with pylint
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (33 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 34/46] qapi/gen.py: update write() to be more idiomatic John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 36/46] qapi/introspect.py: assert obj is a dict when features are given John Snow
` (10 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
'fp' and 'fd' are self-evident in context, add them to the list of OK
names.
_top and _bottom also need to stay standard methods because some users
override the method and need to use `self`. Tell pylint to shush.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/gen.py | 2 ++
scripts/qapi/pylintrc | 5 +++--
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 2ad96e396e1..fc793552036 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -53,9 +53,11 @@ def get_content(self) -> str:
return self._top() + self._preamble + self._body + self._bottom()
def _top(self) -> str:
+ # pylint: disable=no-self-use
return ''
def _bottom(self) -> str:
+ # pylint: disable=no-self-use
return ''
def write(self, output_dir: str) -> None:
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index d840b150313..8badcb11cda 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -4,7 +4,6 @@
# The regex matches against base names, not paths.
ignore-patterns=error.py,
expr.py,
- gen.py,
parser.py,
schema.py,
types.py,
@@ -45,7 +44,9 @@ good-names=i,
k,
ex,
Run,
- _
+ _,
+ fp, # fp = open(...)
+ fd, # fd = os.open(...)
[VARIABLES]
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 36/46] qapi/introspect.py: assert obj is a dict when features are given
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (34 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 35/46] qapi/gen.py: delint with pylint John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 11:04 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 37/46] qapi/instrospect.py: add preliminary type hint annotations John Snow
` (9 subsequent siblings)
45 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
This is necessary to keep mypy passing in the next patch when we add
preliminary type hints. It will be removed shortly.
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/introspect.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 31acd2f230a..83140f2c564 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -30,6 +30,7 @@ def _make_tree(obj, ifcond, features, extra=None):
if ifcond:
extra['if'] = ifcond
if features:
+ assert isinstance(obj, dict)
obj['features'] = [(f.name, {'if': f.ifcond}) for f in features]
if extra:
return (obj, extra)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 37/46] qapi/instrospect.py: add preliminary type hint annotations
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (35 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 36/46] qapi/introspect.py: assert obj is a dict when features are given John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 18:31 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 38/46] qapi/introspect.py: add _gen_features helper John Snow
` (8 subsequent siblings)
45 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
From: Eduardo Habkost <ehabkost@redhat.com>
The typing of _make_tree and friends is a bit involved, but it can be
done with some stubbed out types and a bit of elbow grease. The
forthcoming patches attempt to make some simplifications, but having the
type hints in advance can aid in review.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/introspect.py | 133 +++++++++++++++++++++++++++----------
scripts/qapi/mypy.ini | 5 --
scripts/qapi/schema.py | 2 +-
3 files changed, 98 insertions(+), 42 deletions(-)
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 83140f2c564..f7de3953391 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,6 +10,15 @@
See the COPYING file in the top-level directory.
"""
+from typing import (
+ Dict,
+ List,
+ Optional,
+ Sequence,
+ Tuple,
+ Union,
+)
+
from .common import (
c_name,
gen_endif,
@@ -18,13 +27,34 @@
)
from .gen import QAPISchemaMonolithicCVisitor
from .schema import (
+ QAPISchema,
QAPISchemaArrayType,
QAPISchemaBuiltinType,
+ QAPISchemaEntity,
+ QAPISchemaEnumMember,
+ QAPISchemaFeature,
+ QAPISchemaObjectType,
+ QAPISchemaObjectTypeMember,
QAPISchemaType,
+ QAPISchemaVariant,
+ QAPISchemaVariants,
)
+from .source import QAPISourceInfo
-def _make_tree(obj, ifcond, features, extra=None):
+# The correct type for TreeNode is actually:
+# Union[AnnotatedNode, List[TreeNode], Dict[str, TreeNode], str, bool]
+# but mypy does not support recursive types yet.
+TreeNode = object
+_DObject = Dict[str, object]
+Extra = Dict[str, object]
+AnnotatedNode = Tuple[TreeNode, Extra]
+
+
+def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ extra: Optional[Extra] = None
+ ) -> Union[TreeNode, AnnotatedNode]:
if extra is None:
extra = {}
if ifcond:
@@ -37,9 +67,11 @@ def _make_tree(obj, ifcond, features, extra=None):
return obj
-def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
+def _tree_to_qlit(obj: TreeNode,
+ level: int = 0,
+ suppress_first_indent: bool = False) -> str:
- def indent(level):
+ def indent(level: int) -> str:
return level * 4 * ' '
if isinstance(obj, tuple):
@@ -89,21 +121,20 @@ def indent(level):
return ret
-def to_c_string(string):
+def to_c_string(string: str) -> str:
return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"'
class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor):
-
- def __init__(self, prefix, unmask):
+ def __init__(self, prefix: str, unmask: bool):
super().__init__(
prefix, 'qapi-introspect',
' * QAPI/QMP schema introspection', __doc__)
self._unmask = unmask
- self._schema = None
- self._trees = []
- self._used_types = []
- self._name_map = {}
+ self._schema: Optional[QAPISchema] = None
+ self._trees: List[TreeNode] = []
+ self._used_types: List[QAPISchemaType] = []
+ self._name_map: Dict[str, str] = {}
self._genc.add(mcgen('''
#include "qemu/osdep.h"
#include "%(prefix)sqapi-introspect.h"
@@ -111,10 +142,10 @@ def __init__(self, prefix, unmask):
''',
prefix=prefix))
- def visit_begin(self, schema):
+ def visit_begin(self, schema: QAPISchema) -> None:
self._schema = schema
- def visit_end(self):
+ def visit_end(self) -> None:
# visit the types that are actually used
for typ in self._used_types:
typ.visit(self)
@@ -136,18 +167,18 @@ def visit_end(self):
self._used_types = []
self._name_map = {}
- def visit_needed(self, entity):
+ def visit_needed(self, entity: QAPISchemaEntity) -> bool:
# Ignore types on first pass; visit_end() will pick up used types
return not isinstance(entity, QAPISchemaType)
- def _name(self, name):
+ def _name(self, name: str) -> str:
if self._unmask:
return name
if name not in self._name_map:
self._name_map[name] = '%d' % len(self._name_map)
return self._name_map[name]
- def _use_type(self, typ):
+ def _use_type(self, typ: QAPISchemaType) -> str:
# Map the various integer types to plain int
if typ.json_type() == 'int':
typ = self._schema.lookup_type('int')
@@ -166,8 +197,10 @@ def _use_type(self, typ):
return '[' + self._use_type(typ.element_type) + ']'
return self._name(typ.name)
- def _gen_tree(self, name, mtype, obj, ifcond, features):
- extra = None
+ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
+ ifcond: List[str],
+ features: Optional[List[QAPISchemaFeature]]) -> None:
+ extra: Extra = None
if mtype not in ('command', 'event', 'builtin', 'array'):
if not self._unmask:
# Output a comment to make it easy to map masked names
@@ -178,44 +211,64 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
obj['meta-type'] = mtype
self._trees.append(_make_tree(obj, ifcond, features, extra))
- def _gen_member(self, member):
- obj = {'name': member.name, 'type': self._use_type(member.type)}
+ def _gen_member(self,
+ member: QAPISchemaObjectTypeMember) -> TreeNode:
+ obj: _DObject = {
+ 'name': member.name,
+ 'type': self._use_type(member.type)
+ }
if member.optional:
obj['default'] = None
return _make_tree(obj, member.ifcond, member.features)
- def _gen_variants(self, tag_name, variants):
+ def _gen_variants(self, tag_name: str,
+ variants: List[QAPISchemaVariant]) -> _DObject:
return {'tag': tag_name,
'variants': [self._gen_variant(v) for v in variants]}
- def _gen_variant(self, variant):
- obj = {'case': variant.name, 'type': self._use_type(variant.type)}
+ def _gen_variant(self, variant: QAPISchemaVariant) -> TreeNode:
+ obj: _DObject = {
+ 'case': variant.name,
+ 'type': self._use_type(variant.type)
+ }
return _make_tree(obj, variant.ifcond, None)
- def visit_builtin_type(self, name, info, json_type):
+ def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
+ json_type: str) -> None:
self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
- def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+ def visit_enum_type(self, name: str, info: QAPISourceInfo,
+ ifcond: List[str], features: List[QAPISchemaFeature],
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str]) -> None:
self._gen_tree(name, 'enum',
{'values': [_make_tree(m.name, m.ifcond, None)
for m in members]},
ifcond, features)
- def visit_array_type(self, name, info, ifcond, element_type):
+ def visit_array_type(self, name: str, info: Optional[QAPISourceInfo],
+ ifcond: List[str],
+ element_type: QAPISchemaType) -> None:
element = self._use_type(element_type)
self._gen_tree('[' + element + ']', 'array', {'element-type': element},
ifcond, None)
- def visit_object_type_flat(self, name, info, ifcond, features,
- members, variants):
- obj = {'members': [self._gen_member(m) for m in members]}
+ def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo],
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ members: Sequence[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]) -> None:
+ obj: _DObject = {'members': [self._gen_member(m) for m in members]}
if variants:
obj.update(self._gen_variants(variants.tag_member.name,
variants.variants))
self._gen_tree(name, 'object', obj, ifcond, features)
- def visit_alternate_type(self, name, info, ifcond, features, variants):
+ def visit_alternate_type(self, name: str, info: QAPISourceInfo,
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ variants: QAPISchemaVariants) -> None:
self._gen_tree(name, 'alternate',
{'members': [
_make_tree({'type': self._use_type(m.type)},
@@ -223,24 +276,32 @@ def visit_alternate_type(self, name, info, ifcond, features, variants):
for m in variants.variants]},
ifcond, features)
- def visit_command(self, name, info, ifcond, features,
- arg_type, ret_type, gen, success_response, boxed,
- allow_oob, allow_preconfig):
+ def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ arg_type: QAPISchemaObjectType,
+ ret_type: Optional[QAPISchemaType], gen: bool,
+ success_response: bool, boxed: bool, allow_oob: bool,
+ allow_preconfig: bool) -> None:
arg_type = arg_type or self._schema.the_empty_object_type
ret_type = ret_type or self._schema.the_empty_object_type
- obj = {'arg-type': self._use_type(arg_type),
- 'ret-type': self._use_type(ret_type)}
+ obj: _DObject = {
+ 'arg-type': self._use_type(arg_type),
+ 'ret-type': self._use_type(ret_type)
+ }
if allow_oob:
obj['allow-oob'] = allow_oob
self._gen_tree(name, 'command', obj, ifcond, features)
- def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+ def visit_event(self, name: str, info: QAPISourceInfo,
+ ifcond: List[str], features: List[QAPISchemaFeature],
+ arg_type: QAPISchemaObjectType, boxed: bool) -> None:
arg_type = arg_type or self._schema.the_empty_object_type
self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)},
ifcond, features)
-def gen_introspect(schema, output_dir, prefix, opt_unmask):
+def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str,
+ opt_unmask: bool) -> None:
vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
schema.visit(vis)
vis.write(output_dir)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index c6960ff2dbd..5e5c305062e 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -14,11 +14,6 @@ disallow_untyped_defs = False
disallow_incomplete_defs = False
check_untyped_defs = False
-[mypy-qapi.introspect]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
[mypy-qapi.parser]
disallow_untyped_defs = False
disallow_incomplete_defs = False
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 483b4b68dff..a26b25c5e99 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -28,7 +28,7 @@
class QAPISchemaEntity:
meta: Optional[str] = None
- def __init__(self, name, info, doc, ifcond=None, features=None):
+ def __init__(self, name: str, info, doc, ifcond=None, features=None):
assert name is None or isinstance(name, str)
for f in features or []:
assert isinstance(f, QAPISchemaFeature)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 38/46] qapi/introspect.py: add _gen_features helper
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (36 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 37/46] qapi/instrospect.py: add preliminary type hint annotations John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 17:21 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree() John Snow
` (7 subsequent siblings)
45 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
_make_tree might receive a dict or some other type. Adding features
information should arguably be performed by the caller at such a time
when we know the type of the object and don't have to re-interrogate it.
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/introspect.py | 19 ++++++++++++-------
1 file changed, 12 insertions(+), 7 deletions(-)
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index f7de3953391..5cbdc7414bd 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -52,16 +52,12 @@
def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
- features: List[QAPISchemaFeature],
extra: Optional[Extra] = None
) -> Union[TreeNode, AnnotatedNode]:
if extra is None:
extra = {}
if ifcond:
extra['if'] = ifcond
- if features:
- assert isinstance(obj, dict)
- obj['features'] = [(f.name, {'if': f.ifcond}) for f in features]
if extra:
return (obj, extra)
return obj
@@ -197,6 +193,11 @@ def _use_type(self, typ: QAPISchemaType) -> str:
return '[' + self._use_type(typ.element_type) + ']'
return self._name(typ.name)
+ @classmethod
+ def _gen_features(cls,
+ features: List[QAPISchemaFeature]) -> List[TreeNode]:
+ return [_make_tree(f.name, f.ifcond) for f in features]
+
def _gen_tree(self, name: str, mtype: str, obj: _DObject,
ifcond: List[str],
features: Optional[List[QAPISchemaFeature]]) -> None:
@@ -209,7 +210,9 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
name = self._name(name)
obj['name'] = name
obj['meta-type'] = mtype
- self._trees.append(_make_tree(obj, ifcond, features, extra))
+ if features:
+ obj['features'] = self._gen_features(features)
+ self._trees.append(_make_tree(obj, ifcond, extra))
def _gen_member(self,
member: QAPISchemaObjectTypeMember) -> TreeNode:
@@ -219,7 +222,9 @@ def _gen_member(self,
}
if member.optional:
obj['default'] = None
- return _make_tree(obj, member.ifcond, member.features)
+ if member.features:
+ obj['features'] = self._gen_features(member.features)
+ return _make_tree(obj, member.ifcond)
def _gen_variants(self, tag_name: str,
variants: List[QAPISchemaVariant]) -> _DObject:
@@ -231,7 +236,7 @@ def _gen_variant(self, variant: QAPISchemaVariant) -> TreeNode:
'case': variant.name,
'type': self._use_type(variant.type)
}
- return _make_tree(obj, variant.ifcond, None)
+ return _make_tree(obj, variant.ifcond)
def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
json_type: str) -> None:
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree()
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (37 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 38/46] qapi/introspect.py: add _gen_features helper John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 18:24 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 40/46] qapi/introspect.py: replace 'extra' dict with 'comment' argument John Snow
` (6 subsequent siblings)
45 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Returning a *something* or a Tuple of *something* is hard to accurately
type. Let's just always return a tuple for structural consistency.
Instances of the 'TreeNode' type can be replaced with the slightly more
specific 'AnnotatedNode' type.
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/introspect.py | 15 +++++++--------
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 5cbdc7414bd..1c3ba41f1dc 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -53,14 +53,12 @@
def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
extra: Optional[Extra] = None
- ) -> Union[TreeNode, AnnotatedNode]:
+ ) -> AnnotatedNode:
if extra is None:
extra = {}
if ifcond:
extra['if'] = ifcond
- if extra:
- return (obj, extra)
- return obj
+ return (obj, extra)
def _tree_to_qlit(obj: TreeNode,
@@ -128,7 +126,7 @@ def __init__(self, prefix: str, unmask: bool):
' * QAPI/QMP schema introspection', __doc__)
self._unmask = unmask
self._schema: Optional[QAPISchema] = None
- self._trees: List[TreeNode] = []
+ self._trees: List[AnnotatedNode] = []
self._used_types: List[QAPISchemaType] = []
self._name_map: Dict[str, str] = {}
self._genc.add(mcgen('''
@@ -195,7 +193,8 @@ def _use_type(self, typ: QAPISchemaType) -> str:
@classmethod
def _gen_features(cls,
- features: List[QAPISchemaFeature]) -> List[TreeNode]:
+ features: List[QAPISchemaFeature]
+ ) -> List[AnnotatedNode]:
return [_make_tree(f.name, f.ifcond) for f in features]
def _gen_tree(self, name: str, mtype: str, obj: _DObject,
@@ -215,7 +214,7 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
self._trees.append(_make_tree(obj, ifcond, extra))
def _gen_member(self,
- member: QAPISchemaObjectTypeMember) -> TreeNode:
+ member: QAPISchemaObjectTypeMember) -> AnnotatedNode:
obj: _DObject = {
'name': member.name,
'type': self._use_type(member.type)
@@ -231,7 +230,7 @@ def _gen_variants(self, tag_name: str,
return {'tag': tag_name,
'variants': [self._gen_variant(v) for v in variants]}
- def _gen_variant(self, variant: QAPISchemaVariant) -> TreeNode:
+ def _gen_variant(self, variant: QAPISchemaVariant) -> AnnotatedNode:
obj: _DObject = {
'case': variant.name,
'type': self._use_type(variant.type)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 40/46] qapi/introspect.py: replace 'extra' dict with 'comment' argument
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (38 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree() John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 18:24 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure John Snow
` (5 subsequent siblings)
45 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
This is only used to pass in a dictionary with a comment already set, so
skip the runaround and just accept the comment.
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/introspect.py | 17 ++++++++---------
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 1c3ba41f1dc..43b6ba5df1f 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -52,12 +52,11 @@
def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
- extra: Optional[Extra] = None
- ) -> AnnotatedNode:
- if extra is None:
- extra = {}
- if ifcond:
- extra['if'] = ifcond
+ comment: Optional[str] = None) -> AnnotatedNode:
+ extra: Extra = {
+ 'if': ifcond,
+ 'comment': comment,
+ }
return (obj, extra)
@@ -200,18 +199,18 @@ def _gen_features(cls,
def _gen_tree(self, name: str, mtype: str, obj: _DObject,
ifcond: List[str],
features: Optional[List[QAPISchemaFeature]]) -> None:
- extra: Extra = None
+ comment: Optional[str] = None
if mtype not in ('command', 'event', 'builtin', 'array'):
if not self._unmask:
# Output a comment to make it easy to map masked names
# back to the source when reading the generated output.
- extra = {'comment': '"%s" = %s' % (self._name(name), name)}
+ comment = f'"{self._name(name)}" = {name}'
name = self._name(name)
obj['name'] = name
obj['meta-type'] = mtype
if features:
obj['features'] = self._gen_features(features)
- self._trees.append(_make_tree(obj, ifcond, extra))
+ self._trees.append(_make_tree(obj, ifcond, comment))
def _gen_member(self,
member: QAPISchemaObjectTypeMember) -> AnnotatedNode:
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (39 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 40/46] qapi/introspect.py: replace 'extra' dict with 'comment' argument John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 18:39 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 42/46] qapi/types.py: add type hint annotations John Snow
` (4 subsequent siblings)
45 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
This replaces _make_tree with Node.__init__(), effectively. By creating
it as a generic container, we can more accurately describe the exact
nature of this particular Node.
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/introspect.py | 77 +++++++++++++++++++-------------------
1 file changed, 38 insertions(+), 39 deletions(-)
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 43b6ba5df1f..86286e755ca 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -12,11 +12,12 @@
from typing import (
Dict,
+ Generic,
+ Iterable,
List,
Optional,
Sequence,
- Tuple,
- Union,
+ TypeVar,
)
from .common import (
@@ -43,42 +44,42 @@
# The correct type for TreeNode is actually:
-# Union[AnnotatedNode, List[TreeNode], Dict[str, TreeNode], str, bool]
+# Union[Node[TreeNode], List[TreeNode], Dict[str, TreeNode], str, bool]
# but mypy does not support recursive types yet.
TreeNode = object
+_NodeType = TypeVar('_NodeType', bound=TreeNode)
_DObject = Dict[str, object]
-Extra = Dict[str, object]
-AnnotatedNode = Tuple[TreeNode, Extra]
-def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
- comment: Optional[str] = None) -> AnnotatedNode:
- extra: Extra = {
- 'if': ifcond,
- 'comment': comment,
- }
- return (obj, extra)
+class Node(Generic[_NodeType]):
+ """
+ Node generally contains a SchemaInfo-like type (as a dict),
+ But it also used to wrap comments/ifconds around leaf value types.
+ """
+ # Remove after 3.7 adds @dataclass:
+ # pylint: disable=too-few-public-methods
+ def __init__(self, data: _NodeType, ifcond: Iterable[str],
+ comment: Optional[str] = None):
+ self.data = data
+ self.comment: Optional[str] = comment
+ self.ifcond: Sequence[str] = tuple(ifcond)
-def _tree_to_qlit(obj: TreeNode,
- level: int = 0,
+def _tree_to_qlit(obj: TreeNode, level: int = 0,
suppress_first_indent: bool = False) -> str:
def indent(level: int) -> str:
return level * 4 * ' '
- if isinstance(obj, tuple):
- ifobj, extra = obj
- ifcond = extra.get('if')
- comment = extra.get('comment')
+ if isinstance(obj, Node):
ret = ''
- if comment:
- ret += indent(level) + '/* %s */\n' % comment
- if ifcond:
- ret += gen_if(ifcond)
- ret += _tree_to_qlit(ifobj, level)
- if ifcond:
- ret += '\n' + gen_endif(ifcond)
+ if obj.comment:
+ ret += indent(level) + '/* %s */\n' % obj.comment
+ if obj.ifcond:
+ ret += gen_if(obj.ifcond)
+ ret += _tree_to_qlit(obj.data, level)
+ if obj.ifcond:
+ ret += '\n' + gen_endif(obj.ifcond)
return ret
ret = ''
@@ -125,7 +126,7 @@ def __init__(self, prefix: str, unmask: bool):
' * QAPI/QMP schema introspection', __doc__)
self._unmask = unmask
self._schema: Optional[QAPISchema] = None
- self._trees: List[AnnotatedNode] = []
+ self._trees: List[Node[_DObject]] = []
self._used_types: List[QAPISchemaType] = []
self._name_map: Dict[str, str] = {}
self._genc.add(mcgen('''
@@ -192,9 +193,8 @@ def _use_type(self, typ: QAPISchemaType) -> str:
@classmethod
def _gen_features(cls,
- features: List[QAPISchemaFeature]
- ) -> List[AnnotatedNode]:
- return [_make_tree(f.name, f.ifcond) for f in features]
+ features: List[QAPISchemaFeature]) -> List[Node[str]]:
+ return [Node(f.name, f.ifcond) for f in features]
def _gen_tree(self, name: str, mtype: str, obj: _DObject,
ifcond: List[str],
@@ -210,10 +210,10 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
obj['meta-type'] = mtype
if features:
obj['features'] = self._gen_features(features)
- self._trees.append(_make_tree(obj, ifcond, comment))
+ self._trees.append(Node(obj, ifcond, comment))
def _gen_member(self,
- member: QAPISchemaObjectTypeMember) -> AnnotatedNode:
+ member: QAPISchemaObjectTypeMember) -> Node[_DObject]:
obj: _DObject = {
'name': member.name,
'type': self._use_type(member.type)
@@ -222,19 +222,19 @@ def _gen_member(self,
obj['default'] = None
if member.features:
obj['features'] = self._gen_features(member.features)
- return _make_tree(obj, member.ifcond)
+ return Node(obj, member.ifcond)
def _gen_variants(self, tag_name: str,
variants: List[QAPISchemaVariant]) -> _DObject:
return {'tag': tag_name,
'variants': [self._gen_variant(v) for v in variants]}
- def _gen_variant(self, variant: QAPISchemaVariant) -> AnnotatedNode:
+ def _gen_variant(self, variant: QAPISchemaVariant) -> Node[_DObject]:
obj: _DObject = {
'case': variant.name,
'type': self._use_type(variant.type)
}
- return _make_tree(obj, variant.ifcond)
+ return Node(obj, variant.ifcond)
def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
json_type: str) -> None:
@@ -245,8 +245,7 @@ def visit_enum_type(self, name: str, info: QAPISourceInfo,
members: List[QAPISchemaEnumMember],
prefix: Optional[str]) -> None:
self._gen_tree(name, 'enum',
- {'values': [_make_tree(m.name, m.ifcond, None)
- for m in members]},
+ {'values': [Node(m.name, m.ifcond) for m in members]},
ifcond, features)
def visit_array_type(self, name: str, info: Optional[QAPISourceInfo],
@@ -274,9 +273,9 @@ def visit_alternate_type(self, name: str, info: QAPISourceInfo,
variants: QAPISchemaVariants) -> None:
self._gen_tree(name, 'alternate',
{'members': [
- _make_tree({'type': self._use_type(m.type)},
- m.ifcond, None)
- for m in variants.variants]},
+ Node({'type': self._use_type(m.type)}, m.ifcond)
+ for m in variants.variants
+ ]},
ifcond, features)
def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str],
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 42/46] qapi/types.py: add type hint annotations
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (40 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 43/46] qapi/types.py: remove one-letter variables John Snow
` (3 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Annotations do not change runtime behavior.
This commit *only* adds annotations.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/mypy.ini | 5 ---
scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++-----------
2 files changed, 64 insertions(+), 27 deletions(-)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 5e5c305062e..eeb697c487c 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -24,11 +24,6 @@ disallow_untyped_defs = False
disallow_incomplete_defs = False
check_untyped_defs = False
-[mypy-qapi.types]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
[mypy-qapi.visit]
disallow_untyped_defs = False
disallow_incomplete_defs = False
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 53b47f9e58a..766822feb3a 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,6 +13,8 @@
# See the COPYING file in the top-level directory.
"""
+from typing import List, Optional
+
from .common import (
c_enum_const,
c_name,
@@ -21,7 +23,16 @@
mcgen,
)
from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .schema import (
+ QAPISchema,
+ QAPISchemaEnumMember,
+ QAPISchemaFeature,
+ QAPISchemaObjectType,
+ QAPISchemaObjectTypeMember,
+ QAPISchemaType,
+ QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
# variants must be emitted before their container; track what has already
@@ -29,7 +40,9 @@
objects_seen = set()
-def gen_enum_lookup(name, members, prefix=None):
+def gen_enum_lookup(name: str,
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str] = None) -> str:
ret = mcgen('''
const QEnumLookup %(c_name)s_lookup = {
@@ -54,7 +67,9 @@ def gen_enum_lookup(name, members, prefix=None):
return ret
-def gen_enum(name, members, prefix=None):
+def gen_enum(name: str,
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str] = None) -> str:
# append automatically generated _MAX value
enum_members = members + [QAPISchemaEnumMember('_MAX', None)]
@@ -88,7 +103,7 @@ def gen_enum(name, members, prefix=None):
return ret
-def gen_fwd_object_or_array(name):
+def gen_fwd_object_or_array(name: str) -> str:
return mcgen('''
typedef struct %(c_name)s %(c_name)s;
@@ -96,7 +111,7 @@ def gen_fwd_object_or_array(name):
c_name=c_name(name))
-def gen_array(name, element_type):
+def gen_array(name: str, element_type: QAPISchemaType) -> str:
return mcgen('''
struct %(c_name)s {
@@ -107,7 +122,7 @@ def gen_array(name, element_type):
c_name=c_name(name), c_type=element_type.c_type())
-def gen_struct_members(members):
+def gen_struct_members(members: List[QAPISchemaObjectTypeMember]) -> str:
ret = ''
for memb in members:
ret += gen_if(memb.ifcond)
@@ -124,7 +139,10 @@ def gen_struct_members(members):
return ret
-def gen_object(name, ifcond, base, members, variants):
+def gen_object(name: str, ifcond: List[str],
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]) -> str:
if name in objects_seen:
return ''
objects_seen.add(name)
@@ -178,7 +196,7 @@ def gen_object(name, ifcond, base, members, variants):
return ret
-def gen_upcast(name, base):
+def gen_upcast(name: str, base: QAPISchemaObjectType) -> str:
# C makes const-correctness ugly. We have to cast away const to let
# this function work for both const and non-const obj.
return mcgen('''
@@ -191,7 +209,7 @@ def gen_upcast(name, base):
c_name=c_name(name), base=base.c_name())
-def gen_variants(variants):
+def gen_variants(variants: QAPISchemaVariants) -> str:
ret = mcgen('''
union { /* union tag is @%(c_name)s */
''',
@@ -215,7 +233,7 @@ def gen_variants(variants):
return ret
-def gen_type_cleanup_decl(name):
+def gen_type_cleanup_decl(name: str) -> str:
ret = mcgen('''
void qapi_free_%(c_name)s(%(c_name)s *obj);
@@ -225,7 +243,7 @@ def gen_type_cleanup_decl(name):
return ret
-def gen_type_cleanup(name):
+def gen_type_cleanup(name: str) -> str:
ret = mcgen('''
void qapi_free_%(c_name)s(%(c_name)s *obj)
@@ -247,12 +265,12 @@ def gen_type_cleanup(name):
class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor):
- def __init__(self, prefix):
+ def __init__(self, prefix: str):
super().__init__(
prefix, 'qapi-types', ' * Schema-defined QAPI types',
' * Built-in QAPI types', __doc__)
- def _begin_system_module(self, name):
+ def _begin_system_module(self, name: None) -> None:
self._genc.preamble_add(mcgen('''
#include "qemu/osdep.h"
#include "qapi/dealloc-visitor.h"
@@ -263,7 +281,7 @@ def _begin_system_module(self, name):
#include "qapi/util.h"
'''))
- def _begin_user_module(self, name):
+ def _begin_user_module(self, name: str) -> None:
types = self._module_basename('qapi-types', name)
visit = self._module_basename('qapi-visit', name)
self._genc.preamble_add(mcgen('''
@@ -277,27 +295,43 @@ def _begin_user_module(self, name):
#include "qapi/qapi-builtin-types.h"
'''))
- def visit_begin(self, schema):
+ def visit_begin(self, schema: QAPISchema) -> None:
# gen_object() is recursive, ensure it doesn't visit the empty type
objects_seen.add(schema.the_empty_object_type.name)
- def _gen_type_cleanup(self, name):
+ def _gen_type_cleanup(self, name: str) -> None:
self._genh.add(gen_type_cleanup_decl(name))
self._genc.add(gen_type_cleanup(name))
- def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+ def visit_enum_type(self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str]) -> None:
with ifcontext(ifcond, self._genh, self._genc):
self._genh.preamble_add(gen_enum(name, members, prefix))
self._genc.add(gen_enum_lookup(name, members, prefix))
- def visit_array_type(self, name, info, ifcond, element_type):
+ def visit_array_type(self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: List[str],
+ element_type: QAPISchemaType) -> None:
with ifcontext(ifcond, self._genh, self._genc):
self._genh.preamble_add(gen_fwd_object_or_array(name))
self._genh.add(gen_array(name, element_type))
self._gen_type_cleanup(name)
- def visit_object_type(self, name, info, ifcond, features,
- base, members, variants):
+ def visit_object_type(self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]) -> None:
# Nothing to do for the special empty builtin
if name == 'q_empty':
return
@@ -313,7 +347,12 @@ def visit_object_type(self, name, info, ifcond, features,
# implicit types won't be directly allocated/freed
self._gen_type_cleanup(name)
- def visit_alternate_type(self, name, info, ifcond, features, variants):
+ def visit_alternate_type(self,
+ name: str,
+ info: QAPISourceInfo,
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ variants: QAPISchemaVariants) -> None:
with ifcontext(ifcond, self._genh):
self._genh.preamble_add(gen_fwd_object_or_array(name))
self._genh.add(gen_object(name, ifcond, None,
@@ -322,7 +361,10 @@ def visit_alternate_type(self, name, info, ifcond, features, variants):
self._gen_type_cleanup(name)
-def gen_types(schema, output_dir, prefix, opt_builtins):
+def gen_types(schema: QAPISchema,
+ output_dir: str,
+ prefix: str,
+ opt_builtins: bool) -> None:
vis = QAPISchemaGenTypeVisitor(prefix)
schema.visit(vis)
vis.write(output_dir, opt_builtins)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 43/46] qapi/types.py: remove one-letter variables
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (41 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 42/46] qapi/types.py: add type hint annotations John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 44/46] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType John Snow
` (2 subsequent siblings)
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
"John, if pylint told you to jump off a bridge, would you?"
Hey, if it looked like fun, I might.
Now that this file is clean, enable pylint checks on this file.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/pylintrc | 1 -
scripts/qapi/types.py | 29 +++++++++++++++--------------
2 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 8badcb11cda..b3c4cf46dbf 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
expr.py,
parser.py,
schema.py,
- types.py,
visit.py,
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 766822feb3a..9d1e79d503d 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
.array = (const char *const[]) {
''',
c_name=c_name(name))
- for m in members:
- ret += gen_if(m.ifcond)
- index = c_enum_const(name, m.name, prefix)
+ for member in members:
+ ret += gen_if(member.ifcond)
+ index = c_enum_const(name, member.name, prefix)
ret += mcgen('''
[%(index)s] = "%(name)s",
''',
- index=index, name=m.name)
- ret += gen_endif(m.ifcond)
+ index=index, name=member.name)
+ ret += gen_endif(member.ifcond)
ret += mcgen('''
},
@@ -79,13 +79,13 @@ def gen_enum(name: str,
''',
c_name=c_name(name))
- for m in enum_members:
- ret += gen_if(m.ifcond)
+ for member in enum_members:
+ ret += gen_if(member.ifcond)
ret += mcgen('''
%(c_enum)s,
''',
- c_enum=c_enum_const(name, m.name, prefix))
- ret += gen_endif(m.ifcond)
+ c_enum=c_enum_const(name, member.name, prefix))
+ ret += gen_endif(member.ifcond)
ret += mcgen('''
} %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
objects_seen.add(name)
ret = ''
- if variants:
- for v in variants.variants:
- if isinstance(v.type, QAPISchemaObjectType):
- ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
- v.type.local_members, v.type.variants)
+ for variant in variants.variants if variants else ():
+ obj = variant.type
+ if not isinstance(obj, QAPISchemaObjectType):
+ continue
+ ret += gen_object(obj.name, obj.ifcond, obj.base,
+ obj.local_members, obj.variants)
ret += mcgen('''
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 44/46] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (42 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 43/46] qapi/types.py: remove one-letter variables John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 45/46] qapi/visit.py: remove unused parameters from gen_visit_object John Snow
2020-09-30 4:31 ` [PATCH v4 46/46] qapi/visit.py: add type hint annotations John Snow
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
This is true by design, but not presently able to be expressed in the
type system. An assertion helps mypy understand our constraints.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/visit.py | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 14f30c228b7..4f11fd325b8 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -22,7 +22,7 @@
mcgen,
)
from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import QAPISchemaEnumType, QAPISchemaObjectType
def gen_visit_decl(name, scalar=False):
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
ret += gen_endif(memb.ifcond)
if variants:
+ tag_member = variants.tag_member
+ assert isinstance(tag_member.type, QAPISchemaEnumType)
+
ret += mcgen('''
switch (obj->%(c_name)s) {
''',
- c_name=c_name(variants.tag_member.name))
+ c_name=c_name(tag_member.name))
for var in variants.variants:
- case_str = c_enum_const(variants.tag_member.type.name,
- var.name,
- variants.tag_member.type.prefix)
+ case_str = c_enum_const(tag_member.type.name, var.name,
+ tag_member.type.prefix)
ret += gen_if(var.ifcond)
if var.type.name == 'q_empty':
# valid variant and nothing to do
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 45/46] qapi/visit.py: remove unused parameters from gen_visit_object
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (43 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 44/46] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType John Snow
@ 2020-09-30 4:31 ` John Snow
2020-09-30 4:31 ` [PATCH v4 46/46] qapi/visit.py: add type hint annotations John Snow
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
And this fixes the pylint report for this file, so make sure we check
this in the future, too.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/pylintrc | 1 -
scripts/qapi/visit.py | 4 ++--
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index b3c4cf46dbf..b9e077a1642 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
expr.py,
parser.py,
schema.py,
- visit.py,
[MESSAGES CONTROL]
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 4f11fd325b8..e54694e23db 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -250,7 +250,7 @@ def gen_visit_alternate(name, variants):
return ret
-def gen_visit_object(name, base, members, variants):
+def gen_visit_object(name):
return mcgen('''
bool visit_type_%(c_name)s
@@ -343,7 +343,7 @@ def visit_object_type(self, name, info, ifcond, features,
if not name.startswith('q_'):
# only explicit types need an allocating visit
self._genh.add(gen_visit_decl(name))
- self._genc.add(gen_visit_object(name, base, members, variants))
+ self._genc.add(gen_visit_object(name))
def visit_alternate_type(self, name, info, ifcond, features, variants):
with ifcontext(ifcond, self._genh, self._genc):
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* [PATCH v4 46/46] qapi/visit.py: add type hint annotations
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
` (44 preceding siblings ...)
2020-09-30 4:31 ` [PATCH v4 45/46] qapi/visit.py: remove unused parameters from gen_visit_object John Snow
@ 2020-09-30 4:31 ` John Snow
45 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 4:31 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, Markus Armbruster,
John Snow, Cleber Rosa, Alex Bennée
Annotations do not change runtime behavior.
This commit *only* adds annotations.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
scripts/qapi/mypy.ini | 5 ---
scripts/qapi/visit.py | 73 +++++++++++++++++++++++++++++++++----------
2 files changed, 56 insertions(+), 22 deletions(-)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index eeb697c487c..c0f2a58306d 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -23,8 +23,3 @@ check_untyped_defs = False
disallow_untyped_defs = False
disallow_incomplete_defs = False
check_untyped_defs = False
-
-[mypy-qapi.visit]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index e54694e23db..14d4f0b261f 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,6 +13,8 @@
See the COPYING file in the top-level directory.
"""
+from typing import List, Optional
+
from .common import (
c_enum_const,
c_name,
@@ -22,10 +24,20 @@
mcgen,
)
from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumType, QAPISchemaObjectType
+from .schema import (
+ QAPISchema,
+ QAPISchemaEnumMember,
+ QAPISchemaEnumType,
+ QAPISchemaFeature,
+ QAPISchemaObjectType,
+ QAPISchemaObjectTypeMember,
+ QAPISchemaType,
+ QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
-def gen_visit_decl(name, scalar=False):
+def gen_visit_decl(name: str, scalar: bool = False) -> str:
c_type = c_name(name) + ' *'
if not scalar:
c_type += '*'
@@ -37,7 +49,7 @@ def gen_visit_decl(name, scalar=False):
c_name=c_name(name), c_type=c_type)
-def gen_visit_members_decl(name):
+def gen_visit_members_decl(name: str) -> str:
return mcgen('''
bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp);
@@ -45,7 +57,10 @@ def gen_visit_members_decl(name):
c_name=c_name(name))
-def gen_visit_object_members(name, base, members, variants):
+def gen_visit_object_members(name: str,
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]) -> str:
ret = mcgen('''
bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
@@ -125,7 +140,7 @@ def gen_visit_object_members(name, base, members, variants):
return ret
-def gen_visit_list(name, element_type):
+def gen_visit_list(name: str, element_type: QAPISchemaType) -> str:
return mcgen('''
bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -159,7 +174,7 @@ def gen_visit_list(name, element_type):
c_name=c_name(name), c_elt_type=element_type.c_name())
-def gen_visit_enum(name):
+def gen_visit_enum(name: str) -> str:
return mcgen('''
bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -174,7 +189,7 @@ def gen_visit_enum(name):
c_name=c_name(name))
-def gen_visit_alternate(name, variants):
+def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
ret = mcgen('''
bool visit_type_%(c_name)s
@@ -250,7 +265,7 @@ def gen_visit_alternate(name, variants):
return ret
-def gen_visit_object(name):
+def gen_visit_object(name: str) -> str:
return mcgen('''
bool visit_type_%(c_name)s
@@ -285,12 +300,12 @@ def gen_visit_object(name):
class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
- def __init__(self, prefix):
+ def __init__(self, prefix: str):
super().__init__(
prefix, 'qapi-visit', ' * Schema-defined QAPI visitors',
' * Built-in QAPI visitors', __doc__)
- def _begin_system_module(self, name):
+ def _begin_system_module(self, name: None) -> None:
self._genc.preamble_add(mcgen('''
#include "qemu/osdep.h"
#include "qapi/error.h"
@@ -302,7 +317,7 @@ def _begin_system_module(self, name):
'''))
- def _begin_user_module(self, name):
+ def _begin_user_module(self, name: str) -> None:
types = self._module_basename('qapi-types', name)
visit = self._module_basename('qapi-visit', name)
self._genc.preamble_add(mcgen('''
@@ -319,18 +334,34 @@ def _begin_user_module(self, name):
''',
types=types))
- def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+ def visit_enum_type(self,
+ name: str,
+ info: QAPISourceInfo,
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str]) -> None:
with ifcontext(ifcond, self._genh, self._genc):
self._genh.add(gen_visit_decl(name, scalar=True))
self._genc.add(gen_visit_enum(name))
- def visit_array_type(self, name, info, ifcond, element_type):
+ def visit_array_type(self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: List[str],
+ element_type: QAPISchemaType) -> None:
with ifcontext(ifcond, self._genh, self._genc):
self._genh.add(gen_visit_decl(name))
self._genc.add(gen_visit_list(name, element_type))
- def visit_object_type(self, name, info, ifcond, features,
- base, members, variants):
+ def visit_object_type(self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]) -> None:
# Nothing to do for the special empty builtin
if name == 'q_empty':
return
@@ -345,13 +376,21 @@ def visit_object_type(self, name, info, ifcond, features,
self._genh.add(gen_visit_decl(name))
self._genc.add(gen_visit_object(name))
- def visit_alternate_type(self, name, info, ifcond, features, variants):
+ def visit_alternate_type(self,
+ name: str,
+ info: QAPISourceInfo,
+ ifcond: List[str],
+ features: List[QAPISchemaFeature],
+ variants: QAPISchemaVariants) -> None:
with ifcontext(ifcond, self._genh, self._genc):
self._genh.add(gen_visit_decl(name))
self._genc.add(gen_visit_alternate(name, variants))
-def gen_visit(schema, output_dir, prefix, opt_builtins):
+def gen_visit(schema: QAPISchema,
+ output_dir: str,
+ prefix: str,
+ opt_builtins: bool) -> None:
vis = QAPISchemaGenVisitVisitor(prefix)
schema.visit(vis)
vis.write(output_dir, opt_builtins)
--
2.26.2
^ permalink raw reply related [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-09-30 4:31 ` [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible John Snow
@ 2020-09-30 8:47 ` Markus Armbruster
2020-09-30 17:22 ` John Snow
2020-09-30 17:38 ` John Snow
0 siblings, 2 replies; 69+ messages in thread
From: Markus Armbruster @ 2020-09-30 8:47 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Cleber Rosa, Alex Bennée
John Snow <jsnow@redhat.com> writes:
> I did not say "sphinx beautiful", just "sphinx compatible". They will
> not throw errors when parsed and interpreted as ReST.
"Bang on the keyboard until Sphinx doesn't throw errors anymore" might
be good enough for a certain kind of mathematician, but a constructive
solution needs a bit more direction. Is there a specification to
follow? Other useful resources?
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
> scripts/qapi/gen.py | 6 ++++--
> scripts/qapi/parser.py | 9 +++++----
> 2 files changed, 9 insertions(+), 6 deletions(-)
>
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index ca66c82b5b8..fc19b2aeb9b 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -154,9 +154,11 @@ def _bottom(self):
>
> @contextmanager
> def ifcontext(ifcond, *args):
> - """A 'with' statement context manager to wrap with start_if()/end_if()
> + """
> + A 'with' statement context manager that wraps with `start_if` and `end_if`.
Sadly, the fact that start_if() and end_if() are functions isn't
immediately obvious anymore.
I've seen :func:`start_if` elsewhere. Is this something we should or
want to use?
>
> - *args: any number of QAPIGenCCode
> + :param ifcond: List of conditionals
> + :param args: any number of `QAPIGenCCode`.
>
> Example::
>
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index 9d1a3e2eea9..02983979965 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -381,10 +381,11 @@ def append(self, line):
>
> The way that the line is dealt with depends on which part of
> the documentation we're parsing right now:
> - * The body section: ._append_line is ._append_body_line
> - * An argument section: ._append_line is ._append_args_line
> - * A features section: ._append_line is ._append_features_line
> - * An additional section: ._append_line is ._append_various_line
> +
> + * The body section: ._append_line is ._append_body_line
> + * An argument section: ._append_line is ._append_args_line
> + * A features section: ._append_line is ._append_features_line
> + * An additional section: ._append_line is ._append_various_line
> """
> line = line[1:]
> if not line:
I understand why you insert a blank line (reST wants blank lines around
lists), I don't understand why you indent. Can you explain?
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 30/46] qapi/gen.py: Fix edge-case of _is_user_module
2020-09-30 4:31 ` [PATCH v4 30/46] qapi/gen.py: Fix edge-case of _is_user_module John Snow
@ 2020-09-30 11:03 ` Eduardo Habkost
0 siblings, 0 replies; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 11:03 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 12:31:34AM -0400, John Snow wrote:
> The edge case is that if the name is '', this expression returns a
> string instead of a bool, which violates our declared type.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 36/46] qapi/introspect.py: assert obj is a dict when features are given
2020-09-30 4:31 ` [PATCH v4 36/46] qapi/introspect.py: assert obj is a dict when features are given John Snow
@ 2020-09-30 11:04 ` Eduardo Habkost
0 siblings, 0 replies; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 11:04 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 12:31:40AM -0400, John Snow wrote:
> This is necessary to keep mypy passing in the next patch when we add
> preliminary type hints. It will be removed shortly.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 38/46] qapi/introspect.py: add _gen_features helper
2020-09-30 4:31 ` [PATCH v4 38/46] qapi/introspect.py: add _gen_features helper John Snow
@ 2020-09-30 17:21 ` Eduardo Habkost
0 siblings, 0 replies; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 17:21 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 12:31:42AM -0400, John Snow wrote:
> _make_tree might receive a dict or some other type. Adding features
> information should arguably be performed by the caller at such a time
> when we know the type of the object and don't have to re-interrogate it.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
> scripts/qapi/introspect.py | 19 ++++++++++++-------
> 1 file changed, 12 insertions(+), 7 deletions(-)
>
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index f7de3953391..5cbdc7414bd 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -52,16 +52,12 @@
>
>
> def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
> - features: List[QAPISchemaFeature],
> extra: Optional[Extra] = None
> ) -> Union[TreeNode, AnnotatedNode]:
> if extra is None:
> extra = {}
> if ifcond:
> extra['if'] = ifcond
> - if features:
> - assert isinstance(obj, dict)
> - obj['features'] = [(f.name, {'if': f.ifcond}) for f in features]
Now the reason for moving this code outside _make_tree() is more
obvious due to the type annotations.
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> if extra:
> return (obj, extra)
> return obj
> @@ -197,6 +193,11 @@ def _use_type(self, typ: QAPISchemaType) -> str:
> return '[' + self._use_type(typ.element_type) + ']'
> return self._name(typ.name)
>
> + @classmethod
> + def _gen_features(cls,
> + features: List[QAPISchemaFeature]) -> List[TreeNode]:
> + return [_make_tree(f.name, f.ifcond) for f in features]
> +
> def _gen_tree(self, name: str, mtype: str, obj: _DObject,
> ifcond: List[str],
> features: Optional[List[QAPISchemaFeature]]) -> None:
> @@ -209,7 +210,9 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
> name = self._name(name)
> obj['name'] = name
> obj['meta-type'] = mtype
> - self._trees.append(_make_tree(obj, ifcond, features, extra))
> + if features:
> + obj['features'] = self._gen_features(features)
> + self._trees.append(_make_tree(obj, ifcond, extra))
>
> def _gen_member(self,
> member: QAPISchemaObjectTypeMember) -> TreeNode:
> @@ -219,7 +222,9 @@ def _gen_member(self,
> }
> if member.optional:
> obj['default'] = None
> - return _make_tree(obj, member.ifcond, member.features)
> + if member.features:
> + obj['features'] = self._gen_features(member.features)
> + return _make_tree(obj, member.ifcond)
>
> def _gen_variants(self, tag_name: str,
> variants: List[QAPISchemaVariant]) -> _DObject:
> @@ -231,7 +236,7 @@ def _gen_variant(self, variant: QAPISchemaVariant) -> TreeNode:
> 'case': variant.name,
> 'type': self._use_type(variant.type)
> }
> - return _make_tree(obj, variant.ifcond, None)
> + return _make_tree(obj, variant.ifcond)
>
> def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
> json_type: str) -> None:
> --
> 2.26.2
>
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-09-30 8:47 ` Markus Armbruster
@ 2020-09-30 17:22 ` John Snow
2020-10-01 8:52 ` Markus Armbruster
2020-09-30 17:38 ` John Snow
1 sibling, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 17:22 UTC (permalink / raw)
To: Markus Armbruster
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Cleber Rosa, Alex Bennée
On 9/30/20 4:47 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
>
>> I did not say "sphinx beautiful", just "sphinx compatible". They will
>> not throw errors when parsed and interpreted as ReST.
>
> "Bang on the keyboard until Sphinx doesn't throw errors anymore" might
> be good enough for a certain kind of mathematician, but a constructive
> solution needs a bit more direction. Is there a specification to
> follow? Other useful resources?
>
I don't know if you are asking this question rhetorically, or in good faith.
Let me preface this by saying: This series, and these 119 patches, are
not about finding a style guide for our docstring utilization or about
proposing one. It is also not about rigorously adding such documentation
or about finding ways to meaningfully publish it with e.g. Sphinx, or
the styling of such pages.
Why bother to add docstrings at all, then? Because I needed them for my
own sake when learning this code and I felt it would be a waste to just
delete them, and I am of sound mind and able body and believe that some
documentation was better than none. They are useful even just as plaintext.
Having said that, let's explore the actual style I tend to use.
I mentioned before in response to a review comment that there isn't
really any standard for docstrings. There are a few competing "styles",
but none of which are able to be programmatically checked and validated.
The primary guide for docstrings is PEP 257, of which I follow some but
not all of the recommendations.
https://www.python.org/dev/peps/pep-0257/
In general,
- Always use triple-double quotes (unenforced)
- Modules, classes, and functions should have docstrings (pylint)
- No empty lines before or after the docstring (unenforced)
- Multi-line docstrings should take the form (unenforced):
"""
single-line summary of function.
Additional prose if needed describing the function.
:param x: Input such that blah blah.
:raises y: When input ``x`` is unsuitable because blah blah.
:returns: A value that blah blah.
"""
PEP257 suggests a form where the single-line summary appears on the same
line as the opening triple quotes. I don't like this, and prefer
symmetry. PEP257 *also* suggests that writing it my way is equivalent to
their way, because any docstring processor should trim the first line. I
take this as tacit admission that my way is acceptable and has merit.
What about the syntax or markup inside the docstring itself? there is
*absolutely no standard*, but Sphinx autodoc recognizes a few field
lists as significant in its parsing, so I prefer using them:
:param x: Denotes the parameter X. Do not use type information in the
string, we rely on mypy for that now.
:raises y: explains a case in which an Exception type y may be raised
either directly by this code or anticipated to be allowed to be raised
by a helper call. (There's no standard here that I am aware of. I use my
judgment. Always document direct raise calls, but use your judgment for
sub-calls.)
:returns: explains the semantics of the return value.
That said, literally any sphinx/ReST markup is OK as long as it passes
make sphinxdocs. Some sphinx markup is prohibited, like adding new
full-throated sections. You can use arbitrary field lists, definition
lists, pre-formatted text, examples, code blocks, whatever.
In general, you are not going to find the kind of semantic validation
you want to ensure that the parameter names are correct, or that you
spelled :param: right, or that you didn't miss a parameter or an
exception. None of that tooling exists for Python.
Thus, it's all rather subjective. No right answers, no validation tools.
Just whatever seems reasonable to a human eye until such time we
actually decide to pursue publishing the API docs in the development
manual, if indeed we ever do so at all.
That series sounds like a great opportunity to hash this all out. That
is when I would like to remove --missing-docstring, too. There will
absolutely be a "docstring series" in the future, but I am insisting
stubbornly it happen after strict typing.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>> scripts/qapi/gen.py | 6 ++++--
>> scripts/qapi/parser.py | 9 +++++----
>> 2 files changed, 9 insertions(+), 6 deletions(-)
>>
>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> index ca66c82b5b8..fc19b2aeb9b 100644
>> --- a/scripts/qapi/gen.py
>> +++ b/scripts/qapi/gen.py
>> @@ -154,9 +154,11 @@ def _bottom(self):
>>
>> @contextmanager
>> def ifcontext(ifcond, *args):
>> - """A 'with' statement context manager to wrap with start_if()/end_if()
>> + """
>> + A 'with' statement context manager that wraps with `start_if` and `end_if`.
>
> Sadly, the fact that start_if() and end_if() are functions isn't
> immediately obvious anymore.
>
> I've seen :func:`start_if` elsewhere. Is this something we should or
> want to use?
>
We *could*.
`start_if` relies on the default role, which I have provisionally set to
"any" here, so this is shorthand for :any:`start_if`.
The :any: role means: "cross-reference any type of thing." If there is
not exactly one thing that matches, it results in an error during the
documentation build.
I like this, because it's nice short-hand syntax that I think
communicates effectively to the reader that this is a symbol of some
kind without needing a premium of ReST-ese.
CONSTANTS are capitalized, Classes are title cased, and functions are
lower_case. `lower_case` references can be assumed to be functions, but
I will admit that this is not enforced or necessarily true as we add
more cross reference types in the future.
(I am trying to add QMP cross-reference syntax!)
I still prefer `start_if` to :func:`start_if` simply because it's less
markup and is easier to read in plaintext contexts. You're right, it
doesn't look like a function anymore.
I'm not sure if another annotations would work -- `start_if`() or
`start_if()`. Both seem kind of clunky to me, to be honest. Personal
feeling is "not really worth the hassle."
>>
>> - *args: any number of QAPIGenCCode
>> + :param ifcond: List of conditionals
>> + :param args: any number of `QAPIGenCCode`.
>>
>> Example::
>>
>> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>> index 9d1a3e2eea9..02983979965 100644
>> --- a/scripts/qapi/parser.py
>> +++ b/scripts/qapi/parser.py
>> @@ -381,10 +381,11 @@ def append(self, line):
>>
>> The way that the line is dealt with depends on which part of
>> the documentation we're parsing right now:
>> - * The body section: ._append_line is ._append_body_line
>> - * An argument section: ._append_line is ._append_args_line
>> - * A features section: ._append_line is ._append_features_line
>> - * An additional section: ._append_line is ._append_various_line
>> +
>> + * The body section: ._append_line is ._append_body_line
>> + * An argument section: ._append_line is ._append_args_line
>> + * A features section: ._append_line is ._append_features_line
>> + * An additional section: ._append_line is ._append_various_line
>> """
>> line = line[1:]
>> if not line:
>
> I understand why you insert a blank line (reST wants blank lines around
> lists), I don't understand why you indent. Can you explain?
I was mistaken about it needing the indent!
--js
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-09-30 8:47 ` Markus Armbruster
2020-09-30 17:22 ` John Snow
@ 2020-09-30 17:38 ` John Snow
2020-10-01 8:54 ` Markus Armbruster
1 sibling, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 17:38 UTC (permalink / raw)
To: Markus Armbruster
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Cleber Rosa, Alex Bennée
On 9/30/20 4:47 AM, Markus Armbruster wrote:
> Sadly, the fact that start_if() and end_if() are functions isn't
> immediately obvious anymore.
>
> I've seen :func:`start_if` elsewhere. Is this something we should or
> want to use?
Looks like `start_if()` works just fine too. If you are hard-set in
wanting to see those parentheses I can use this style, but admit I am
not likely to use them myself in newer docstrings, and there's no way to
enforce their presence OR absence that I am aware of.
I'll bake that change in for now until I see another reply.
--js
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree()
2020-09-30 4:31 ` [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree() John Snow
@ 2020-09-30 18:24 ` Eduardo Habkost
2020-09-30 18:32 ` John Snow
0 siblings, 1 reply; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 18:24 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 12:31:43AM -0400, John Snow wrote:
> Returning a *something* or a Tuple of *something* is hard to accurately
> type. Let's just always return a tuple for structural consistency.
>
> Instances of the 'TreeNode' type can be replaced with the slightly more
> specific 'AnnotatedNode' type.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
So, the only place where this seems to make a difference is
_tree_to_qlit().
We just need to prove that
_tree_to_qlit(o, ...)
will have exactly the same result as
_tree_to_qlit((o, None), ...).
For reference, this is the beginning of _tree_to_qlit():
| def _tree_to_qlit(obj: TreeNode,
| level: int = 0,
| suppress_first_indent: bool = False) -> str:
|
| def indent(level: int) -> str:
| return level * 4 * ' '
|
| if isinstance(obj, tuple):
| ifobj, extra = obj
`obj` is the return value of _make_tree()
`ifobj` is the original `obj` argument to _make_tree().
| ifcond = extra.get('if')
ifcond will be None.
| comment = extra.get('comment')
comment will be None
| ret = ''
| if comment:
| ret += indent(level) + '/* %s */\n' % comment
nop
| if ifcond:
| ret += gen_if(ifcond)
nop
| ret += _tree_to_qlit(ifobj, level)
ret will be '', so this is equivalent to:
ret = _tree_to_qlit(ifobj, level)
which is almost good.
The only difference seems to that suppress_first_indent=True will
be ignored. We should pass suppress_first_indent as argument in
the recursive call above, just in case.
The existing code will behave weirdly if there are comments or
conditions and suppress_first_indent=True, but I suggest we try
to address this issue later.
| if ifcond:
| ret += '\n' + gen_endif(ifcond)
nop
| return ret
> ---
> scripts/qapi/introspect.py | 15 +++++++--------
> 1 file changed, 7 insertions(+), 8 deletions(-)
>
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index 5cbdc7414bd..1c3ba41f1dc 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -53,14 +53,12 @@
>
> def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
> extra: Optional[Extra] = None
> - ) -> Union[TreeNode, AnnotatedNode]:
> + ) -> AnnotatedNode:
> if extra is None:
> extra = {}
> if ifcond:
> extra['if'] = ifcond
> - if extra:
> - return (obj, extra)
> - return obj
> + return (obj, extra)
>
>
> def _tree_to_qlit(obj: TreeNode,
> @@ -128,7 +126,7 @@ def __init__(self, prefix: str, unmask: bool):
> ' * QAPI/QMP schema introspection', __doc__)
> self._unmask = unmask
> self._schema: Optional[QAPISchema] = None
> - self._trees: List[TreeNode] = []
> + self._trees: List[AnnotatedNode] = []
> self._used_types: List[QAPISchemaType] = []
> self._name_map: Dict[str, str] = {}
> self._genc.add(mcgen('''
> @@ -195,7 +193,8 @@ def _use_type(self, typ: QAPISchemaType) -> str:
>
> @classmethod
> def _gen_features(cls,
> - features: List[QAPISchemaFeature]) -> List[TreeNode]:
> + features: List[QAPISchemaFeature]
> + ) -> List[AnnotatedNode]:
> return [_make_tree(f.name, f.ifcond) for f in features]
>
> def _gen_tree(self, name: str, mtype: str, obj: _DObject,
> @@ -215,7 +214,7 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
> self._trees.append(_make_tree(obj, ifcond, extra))
>
> def _gen_member(self,
> - member: QAPISchemaObjectTypeMember) -> TreeNode:
> + member: QAPISchemaObjectTypeMember) -> AnnotatedNode:
> obj: _DObject = {
> 'name': member.name,
> 'type': self._use_type(member.type)
> @@ -231,7 +230,7 @@ def _gen_variants(self, tag_name: str,
> return {'tag': tag_name,
> 'variants': [self._gen_variant(v) for v in variants]}
>
> - def _gen_variant(self, variant: QAPISchemaVariant) -> TreeNode:
> + def _gen_variant(self, variant: QAPISchemaVariant) -> AnnotatedNode:
> obj: _DObject = {
> 'case': variant.name,
> 'type': self._use_type(variant.type)
> --
> 2.26.2
>
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 40/46] qapi/introspect.py: replace 'extra' dict with 'comment' argument
2020-09-30 4:31 ` [PATCH v4 40/46] qapi/introspect.py: replace 'extra' dict with 'comment' argument John Snow
@ 2020-09-30 18:24 ` Eduardo Habkost
0 siblings, 0 replies; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 18:24 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 12:31:44AM -0400, John Snow wrote:
> This is only used to pass in a dictionary with a comment already set, so
> skip the runaround and just accept the comment.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 37/46] qapi/instrospect.py: add preliminary type hint annotations
2020-09-30 4:31 ` [PATCH v4 37/46] qapi/instrospect.py: add preliminary type hint annotations John Snow
@ 2020-09-30 18:31 ` Eduardo Habkost
0 siblings, 0 replies; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 18:31 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 12:31:41AM -0400, John Snow wrote:
> From: Eduardo Habkost <ehabkost@redhat.com>
>
> The typing of _make_tree and friends is a bit involved, but it can be
> done with some stubbed out types and a bit of elbow grease. The
> forthcoming patches attempt to make some simplifications, but having the
> type hints in advance can aid in review.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree()
2020-09-30 18:24 ` Eduardo Habkost
@ 2020-09-30 18:32 ` John Snow
2020-09-30 18:57 ` Eduardo Habkost
0 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 18:32 UTC (permalink / raw)
To: Eduardo Habkost
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On 9/30/20 2:24 PM, Eduardo Habkost wrote:
> On Wed, Sep 30, 2020 at 12:31:43AM -0400, John Snow wrote:
>> Returning a *something* or a Tuple of *something* is hard to accurately
>> type. Let's just always return a tuple for structural consistency.
>>
>> Instances of the 'TreeNode' type can be replaced with the slightly more
>> specific 'AnnotatedNode' type.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>
> So, the only place where this seems to make a difference is
> _tree_to_qlit().
>
> We just need to prove that
> _tree_to_qlit(o, ...)
> will have exactly the same result as
> _tree_to_qlit((o, None), ...).
>
> For reference, this is the beginning of _tree_to_qlit():
>
> | def _tree_to_qlit(obj: TreeNode,
> | level: int = 0,
> | suppress_first_indent: bool = False) -> str:
> |
> | def indent(level: int) -> str:
> | return level * 4 * ' '
> |
> | if isinstance(obj, tuple):
> | ifobj, extra = obj
>
> `obj` is the return value of _make_tree()
>
> `ifobj` is the original `obj` argument to _make_tree().
>
> | ifcond = extra.get('if')
>
> ifcond will be None.
>
> | comment = extra.get('comment')
>
> comment will be None
>
> | ret = ''
> | if comment:
> | ret += indent(level) + '/* %s */\n' % comment
>
> nop
>
> | if ifcond:
> | ret += gen_if(ifcond)
>
> nop
>
> | ret += _tree_to_qlit(ifobj, level)
>
> ret will be '', so this is equivalent to:
>
> ret = _tree_to_qlit(ifobj, level)
>
> which is almost good.
>
> The only difference seems to that suppress_first_indent=True will
> be ignored. We should pass suppress_first_indent as argument in
> the recursive call above, just in case.
>
This is a really good spot, and I indeed hadn't considered it at all
when I did this.
(I simply made the change and observed it worked just fine!)
> The existing code will behave weirdly if there are comments or
> conditions and suppress_first_indent=True, but I suggest we try
> to address this issue later.
>
> | if ifcond:
> | ret += '\n' + gen_endif(ifcond)
>
> nop
>
> | return ret
>
Hm, yes, it's a hypothetical case, but perhaps we can use an assertion
to help guard against it if development creates that case later by accident.
That ought to be good enough for now to not waste time accommodating a
(presently) fictional circumstance?
Thanks for the good sleuthing here.
--js
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure
2020-09-30 4:31 ` [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure John Snow
@ 2020-09-30 18:39 ` Eduardo Habkost
2020-09-30 18:58 ` John Snow
0 siblings, 1 reply; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 18:39 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 12:31:45AM -0400, John Snow wrote:
> This replaces _make_tree with Node.__init__(), effectively. By creating
> it as a generic container, we can more accurately describe the exact
> nature of this particular Node.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
> scripts/qapi/introspect.py | 77 +++++++++++++++++++-------------------
> 1 file changed, 38 insertions(+), 39 deletions(-)
>
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index 43b6ba5df1f..86286e755ca 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -12,11 +12,12 @@
>
> from typing import (
> Dict,
> + Generic,
> + Iterable,
> List,
> Optional,
> Sequence,
> - Tuple,
> - Union,
> + TypeVar,
> )
>
> from .common import (
> @@ -43,42 +44,42 @@
>
>
> # The correct type for TreeNode is actually:
> -# Union[AnnotatedNode, List[TreeNode], Dict[str, TreeNode], str, bool]
> +# Union[Node[TreeNode], List[TreeNode], Dict[str, TreeNode], str, bool]
> # but mypy does not support recursive types yet.
> TreeNode = object
> +_NodeType = TypeVar('_NodeType', bound=TreeNode)
> _DObject = Dict[str, object]
> -Extra = Dict[str, object]
> -AnnotatedNode = Tuple[TreeNode, Extra]
Do you have plans to make Node replace TreeNode completely?
I'd understand this patch as a means to reach that goal, but I'm
not sure the intermediate state of having both Node and TreeNode
types (that can be confused with each other) is desirable.
>
>
> -def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
> - comment: Optional[str] = None) -> AnnotatedNode:
> - extra: Extra = {
> - 'if': ifcond,
> - 'comment': comment,
> - }
> - return (obj, extra)
> +class Node(Generic[_NodeType]):
> + """
> + Node generally contains a SchemaInfo-like type (as a dict),
> + But it also used to wrap comments/ifconds around leaf value types.
> + """
> + # Remove after 3.7 adds @dataclass:
> + # pylint: disable=too-few-public-methods
> + def __init__(self, data: _NodeType, ifcond: Iterable[str],
> + comment: Optional[str] = None):
> + self.data = data
> + self.comment: Optional[str] = comment
> + self.ifcond: Sequence[str] = tuple(ifcond)
>
>
> -def _tree_to_qlit(obj: TreeNode,
> - level: int = 0,
> +def _tree_to_qlit(obj: TreeNode, level: int = 0,
> suppress_first_indent: bool = False) -> str:
>
> def indent(level: int) -> str:
> return level * 4 * ' '
>
> - if isinstance(obj, tuple):
> - ifobj, extra = obj
> - ifcond = extra.get('if')
> - comment = extra.get('comment')
> + if isinstance(obj, Node):
> ret = ''
> - if comment:
> - ret += indent(level) + '/* %s */\n' % comment
> - if ifcond:
> - ret += gen_if(ifcond)
> - ret += _tree_to_qlit(ifobj, level)
> - if ifcond:
> - ret += '\n' + gen_endif(ifcond)
> + if obj.comment:
> + ret += indent(level) + '/* %s */\n' % obj.comment
> + if obj.ifcond:
> + ret += gen_if(obj.ifcond)
> + ret += _tree_to_qlit(obj.data, level)
> + if obj.ifcond:
> + ret += '\n' + gen_endif(obj.ifcond)
> return ret
>
> ret = ''
> @@ -125,7 +126,7 @@ def __init__(self, prefix: str, unmask: bool):
> ' * QAPI/QMP schema introspection', __doc__)
> self._unmask = unmask
> self._schema: Optional[QAPISchema] = None
> - self._trees: List[AnnotatedNode] = []
> + self._trees: List[Node[_DObject]] = []
> self._used_types: List[QAPISchemaType] = []
> self._name_map: Dict[str, str] = {}
> self._genc.add(mcgen('''
> @@ -192,9 +193,8 @@ def _use_type(self, typ: QAPISchemaType) -> str:
>
> @classmethod
> def _gen_features(cls,
> - features: List[QAPISchemaFeature]
> - ) -> List[AnnotatedNode]:
> - return [_make_tree(f.name, f.ifcond) for f in features]
> + features: List[QAPISchemaFeature]) -> List[Node[str]]:
> + return [Node(f.name, f.ifcond) for f in features]
>
> def _gen_tree(self, name: str, mtype: str, obj: _DObject,
> ifcond: List[str],
> @@ -210,10 +210,10 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
> obj['meta-type'] = mtype
> if features:
> obj['features'] = self._gen_features(features)
> - self._trees.append(_make_tree(obj, ifcond, comment))
> + self._trees.append(Node(obj, ifcond, comment))
>
> def _gen_member(self,
> - member: QAPISchemaObjectTypeMember) -> AnnotatedNode:
> + member: QAPISchemaObjectTypeMember) -> Node[_DObject]:
> obj: _DObject = {
> 'name': member.name,
> 'type': self._use_type(member.type)
> @@ -222,19 +222,19 @@ def _gen_member(self,
> obj['default'] = None
> if member.features:
> obj['features'] = self._gen_features(member.features)
> - return _make_tree(obj, member.ifcond)
> + return Node(obj, member.ifcond)
>
> def _gen_variants(self, tag_name: str,
> variants: List[QAPISchemaVariant]) -> _DObject:
> return {'tag': tag_name,
> 'variants': [self._gen_variant(v) for v in variants]}
>
> - def _gen_variant(self, variant: QAPISchemaVariant) -> AnnotatedNode:
> + def _gen_variant(self, variant: QAPISchemaVariant) -> Node[_DObject]:
> obj: _DObject = {
> 'case': variant.name,
> 'type': self._use_type(variant.type)
> }
> - return _make_tree(obj, variant.ifcond)
> + return Node(obj, variant.ifcond)
>
> def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
> json_type: str) -> None:
> @@ -245,8 +245,7 @@ def visit_enum_type(self, name: str, info: QAPISourceInfo,
> members: List[QAPISchemaEnumMember],
> prefix: Optional[str]) -> None:
> self._gen_tree(name, 'enum',
> - {'values': [_make_tree(m.name, m.ifcond, None)
> - for m in members]},
> + {'values': [Node(m.name, m.ifcond) for m in members]},
> ifcond, features)
>
> def visit_array_type(self, name: str, info: Optional[QAPISourceInfo],
> @@ -274,9 +273,9 @@ def visit_alternate_type(self, name: str, info: QAPISourceInfo,
> variants: QAPISchemaVariants) -> None:
> self._gen_tree(name, 'alternate',
> {'members': [
> - _make_tree({'type': self._use_type(m.type)},
> - m.ifcond, None)
> - for m in variants.variants]},
> + Node({'type': self._use_type(m.type)}, m.ifcond)
> + for m in variants.variants
> + ]},
> ifcond, features)
>
> def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str],
> --
> 2.26.2
>
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree()
2020-09-30 18:32 ` John Snow
@ 2020-09-30 18:57 ` Eduardo Habkost
2020-09-30 19:02 ` John Snow
0 siblings, 1 reply; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 18:57 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 02:32:49PM -0400, John Snow wrote:
> On 9/30/20 2:24 PM, Eduardo Habkost wrote:
> > On Wed, Sep 30, 2020 at 12:31:43AM -0400, John Snow wrote:
> > > Returning a *something* or a Tuple of *something* is hard to accurately
> > > type. Let's just always return a tuple for structural consistency.
> > >
> > > Instances of the 'TreeNode' type can be replaced with the slightly more
> > > specific 'AnnotatedNode' type.
> > >
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> >
> > So, the only place where this seems to make a difference is
> > _tree_to_qlit().
> >
> > We just need to prove that
> > _tree_to_qlit(o, ...)
> > will have exactly the same result as
> > _tree_to_qlit((o, None), ...).
> >
> > For reference, this is the beginning of _tree_to_qlit():
> >
> > | def _tree_to_qlit(obj: TreeNode,
> > | level: int = 0,
> > | suppress_first_indent: bool = False) -> str:
> > |
> > | def indent(level: int) -> str:
> > | return level * 4 * ' '
> > |
> > | if isinstance(obj, tuple):
> > | ifobj, extra = obj
> >
> > `obj` is the return value of _make_tree()
> >
> > `ifobj` is the original `obj` argument to _make_tree().
> >
> > | ifcond = extra.get('if')
> >
> > ifcond will be None.
> >
> > | comment = extra.get('comment')
> >
> > comment will be None
> >
> > | ret = ''
> > | if comment:
> > | ret += indent(level) + '/* %s */\n' % comment
> >
> > nop
> >
> > | if ifcond:
> > | ret += gen_if(ifcond)
> >
> > nop
> >
> > | ret += _tree_to_qlit(ifobj, level)
> >
> > ret will be '', so this is equivalent to:
> >
> > ret = _tree_to_qlit(ifobj, level)
> >
> > which is almost good.
> >
> > The only difference seems to that suppress_first_indent=True will
> > be ignored. We should pass suppress_first_indent as argument in
> > the recursive call above, just in case.
> >
>
> This is a really good spot, and I indeed hadn't considered it at all when I
> did this.
>
> (I simply made the change and observed it worked just fine!)
>
> > The existing code will behave weirdly if there are comments or
> > conditions and suppress_first_indent=True, but I suggest we try
> > to address this issue later.
> >
> > | if ifcond:
> > | ret += '\n' + gen_endif(ifcond)
> >
> > nop
> >
> > | return ret
> >
>
> Hm, yes, it's a hypothetical case, but perhaps we can use an assertion to
> help guard against it if development creates that case later by accident.
>
> That ought to be good enough for now to not waste time accommodating a
> (presently) fictional circumstance?
>
> Thanks for the good sleuthing here.
With the current code, both
ret += _tree_to_qlit(ifobj, level)
and
ret += _tree_to_qlit(ifobj, level, suppress_first_indent)
will behave exactly the same.
The former will behave weirdly if we wrap a dictionary value using
_tree_node(). We don't do that today.
The latter will behave weirdly if there's a comment or ifcond
attached in a dictionary value. We don't do that today.
I believe the latter is less likely to be triggered by accident.
But I'd be happy with either:
# _make_tree() shouldn't be use to wrap nodes that
# may be printed using suppress_first_indent=True
# (in other words, dictionary values shouldn't be wrapped using _make_tree())
assert(not suppress_first_indent)
ret += _tree_to_qlit(ifobj, level)
or
# we can't add ifcond or comments to nodes that may be
# printed using suppress_first_indent=True
# (in other words, dictionary values can't have ifcond or comments)
assert(not suppress_first_indent or (not comment and not ifcond))
ret += _tree_to_qlit(ifobj, level, suppress_first_indent)
If we have time to spare later, we could do this:
def _value_to_qlit(obj: Union[None, str, Dict[str, object], List[object], bool],
level: int = 0,
suppress_first_indent: bool = False) -> str:
...
if obj is None:
...
elif isinstance(obj, str):
...
elif isinstance(obj, list):
...
...
def _tree_to_qlit(obj: TreeNode, level: int = 0) -> str:
if isinstance(obj, AnnotatedNode):
...
else:
return _value_to_qlit(obj, level)
This way, it will be impossible to set suppress_first_indent=True
on an annotated node.
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure
2020-09-30 18:39 ` Eduardo Habkost
@ 2020-09-30 18:58 ` John Snow
2020-09-30 19:52 ` Eduardo Habkost
0 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-09-30 18:58 UTC (permalink / raw)
To: Eduardo Habkost
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On 9/30/20 2:39 PM, Eduardo Habkost wrote:
> On Wed, Sep 30, 2020 at 12:31:45AM -0400, John Snow wrote:
>> This replaces _make_tree with Node.__init__(), effectively. By creating
>> it as a generic container, we can more accurately describe the exact
>> nature of this particular Node.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>> scripts/qapi/introspect.py | 77 +++++++++++++++++++-------------------
>> 1 file changed, 38 insertions(+), 39 deletions(-)
>>
>> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
>> index 43b6ba5df1f..86286e755ca 100644
>> --- a/scripts/qapi/introspect.py
>> +++ b/scripts/qapi/introspect.py
>> @@ -12,11 +12,12 @@
>>
>> from typing import (
>> Dict,
>> + Generic,
>> + Iterable,
>> List,
>> Optional,
>> Sequence,
>> - Tuple,
>> - Union,
>> + TypeVar,
>> )
>>
>> from .common import (
>> @@ -43,42 +44,42 @@
>>
>>
>> # The correct type for TreeNode is actually:
>> -# Union[AnnotatedNode, List[TreeNode], Dict[str, TreeNode], str, bool]
>> +# Union[Node[TreeNode], List[TreeNode], Dict[str, TreeNode], str, bool]
>> # but mypy does not support recursive types yet.
>> TreeNode = object
>> +_NodeType = TypeVar('_NodeType', bound=TreeNode)
>> _DObject = Dict[str, object]
>> -Extra = Dict[str, object]
>> -AnnotatedNode = Tuple[TreeNode, Extra]
>
> Do you have plans to make Node replace TreeNode completely?
>
> I'd understand this patch as a means to reach that goal, but I'm
> not sure the intermediate state of having both Node and TreeNode
> types (that can be confused with each other) is desirable.
>
The problem is that _tree_to_qlit still accepts a broad array of types.
The "TreeNode" comment above explains that those types are:
Node[TreeNode], List[TreeNode], Dict[str, TreeNode], str, bool
Three are containers, two are leaf values.
of the containers, the Node container is special in that it houses
explicitly one of the four other types (but never itself.)
Even if I somehow always enforced Node[T] heading into _tree_to_qlit, I
would still need to describe what 'T' is, which is another recursive
type that I cannot exactly describe with mypy's current descriptive power:
INNER_TYPE = List[Node[INNER_TYPE]], Dict[str, Node[INNER_TYPE]], str, bool
And that's not really a huge win, or indeed any different to the
existing TreeNode being an "object".
So ... no, I felt like I was going to stop here, where we have
fundamentally:
1. Undecorated nodes (list, dict, str, bool) ("TreeNode")
2. Decorated nodes (Node[T]) ("Node")
which leads to the question: Why bother swapping Tuple for Node at that
point?
My answer is simply that having a strong type name allows us to
distinguish this from garden-variety Tuples that might sneak in for
other reasons in other data types.
Maybe we want a different nomenclature though, like Node vs AnnotatedNode?
--js
(Also: 'TreeNode' is just an alias for object, it doesn't mean anything
grammatically. I could just as soon erase it entirely if you felt it
provided no value. It doesn't enforce that it only takes objects we
declared were 'TreeNode' types, for instance. It's just a preprocessor
name, basically.)
>>
>>
>> -def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
>> - comment: Optional[str] = None) -> AnnotatedNode:
>> - extra: Extra = {
>> - 'if': ifcond,
>> - 'comment': comment,
>> - }
>> - return (obj, extra)
>> +class Node(Generic[_NodeType]):
>> + """
>> + Node generally contains a SchemaInfo-like type (as a dict),
>> + But it also used to wrap comments/ifconds around leaf value types.
>> + """
>> + # Remove after 3.7 adds @dataclass:
>> + # pylint: disable=too-few-public-methods
>> + def __init__(self, data: _NodeType, ifcond: Iterable[str],
>> + comment: Optional[str] = None):
>> + self.data = data
>> + self.comment: Optional[str] = comment
>> + self.ifcond: Sequence[str] = tuple(ifcond)
>>
>>
>> -def _tree_to_qlit(obj: TreeNode,
>> - level: int = 0,
>> +def _tree_to_qlit(obj: TreeNode, level: int = 0,
>> suppress_first_indent: bool = False) -> str:
>>
>> def indent(level: int) -> str:
>> return level * 4 * ' '
>>
>> - if isinstance(obj, tuple):
>> - ifobj, extra = obj
>> - ifcond = extra.get('if')
>> - comment = extra.get('comment')
>> + if isinstance(obj, Node):
>> ret = ''
>> - if comment:
>> - ret += indent(level) + '/* %s */\n' % comment
>> - if ifcond:
>> - ret += gen_if(ifcond)
>> - ret += _tree_to_qlit(ifobj, level)
>> - if ifcond:
>> - ret += '\n' + gen_endif(ifcond)
>> + if obj.comment:
>> + ret += indent(level) + '/* %s */\n' % obj.comment
>> + if obj.ifcond:
>> + ret += gen_if(obj.ifcond)
>> + ret += _tree_to_qlit(obj.data, level)
>> + if obj.ifcond:
>> + ret += '\n' + gen_endif(obj.ifcond)
>> return ret
>>
>> ret = ''
>> @@ -125,7 +126,7 @@ def __init__(self, prefix: str, unmask: bool):
>> ' * QAPI/QMP schema introspection', __doc__)
>> self._unmask = unmask
>> self._schema: Optional[QAPISchema] = None
>> - self._trees: List[AnnotatedNode] = []
>> + self._trees: List[Node[_DObject]] = []
>> self._used_types: List[QAPISchemaType] = []
>> self._name_map: Dict[str, str] = {}
>> self._genc.add(mcgen('''
>> @@ -192,9 +193,8 @@ def _use_type(self, typ: QAPISchemaType) -> str:
>>
>> @classmethod
>> def _gen_features(cls,
>> - features: List[QAPISchemaFeature]
>> - ) -> List[AnnotatedNode]:
>> - return [_make_tree(f.name, f.ifcond) for f in features]
>> + features: List[QAPISchemaFeature]) -> List[Node[str]]:
>> + return [Node(f.name, f.ifcond) for f in features]
>>
>> def _gen_tree(self, name: str, mtype: str, obj: _DObject,
>> ifcond: List[str],
>> @@ -210,10 +210,10 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
>> obj['meta-type'] = mtype
>> if features:
>> obj['features'] = self._gen_features(features)
>> - self._trees.append(_make_tree(obj, ifcond, comment))
>> + self._trees.append(Node(obj, ifcond, comment))
>>
>> def _gen_member(self,
>> - member: QAPISchemaObjectTypeMember) -> AnnotatedNode:
>> + member: QAPISchemaObjectTypeMember) -> Node[_DObject]:
>> obj: _DObject = {
>> 'name': member.name,
>> 'type': self._use_type(member.type)
>> @@ -222,19 +222,19 @@ def _gen_member(self,
>> obj['default'] = None
>> if member.features:
>> obj['features'] = self._gen_features(member.features)
>> - return _make_tree(obj, member.ifcond)
>> + return Node(obj, member.ifcond)
>>
>> def _gen_variants(self, tag_name: str,
>> variants: List[QAPISchemaVariant]) -> _DObject:
>> return {'tag': tag_name,
>> 'variants': [self._gen_variant(v) for v in variants]}
>>
>> - def _gen_variant(self, variant: QAPISchemaVariant) -> AnnotatedNode:
>> + def _gen_variant(self, variant: QAPISchemaVariant) -> Node[_DObject]:
>> obj: _DObject = {
>> 'case': variant.name,
>> 'type': self._use_type(variant.type)
>> }
>> - return _make_tree(obj, variant.ifcond)
>> + return Node(obj, variant.ifcond)
>>
>> def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
>> json_type: str) -> None:
>> @@ -245,8 +245,7 @@ def visit_enum_type(self, name: str, info: QAPISourceInfo,
>> members: List[QAPISchemaEnumMember],
>> prefix: Optional[str]) -> None:
>> self._gen_tree(name, 'enum',
>> - {'values': [_make_tree(m.name, m.ifcond, None)
>> - for m in members]},
>> + {'values': [Node(m.name, m.ifcond) for m in members]},
>> ifcond, features)
>>
>> def visit_array_type(self, name: str, info: Optional[QAPISourceInfo],
>> @@ -274,9 +273,9 @@ def visit_alternate_type(self, name: str, info: QAPISourceInfo,
>> variants: QAPISchemaVariants) -> None:
>> self._gen_tree(name, 'alternate',
>> {'members': [
>> - _make_tree({'type': self._use_type(m.type)},
>> - m.ifcond, None)
>> - for m in variants.variants]},
>> + Node({'type': self._use_type(m.type)}, m.ifcond)
>> + for m in variants.variants
>> + ]},
>> ifcond, features)
>>
>> def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str],
>> --
>> 2.26.2
>>
>
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree()
2020-09-30 18:57 ` Eduardo Habkost
@ 2020-09-30 19:02 ` John Snow
0 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-09-30 19:02 UTC (permalink / raw)
To: Eduardo Habkost
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On 9/30/20 2:57 PM, Eduardo Habkost wrote:
> On Wed, Sep 30, 2020 at 02:32:49PM -0400, John Snow wrote:
>> On 9/30/20 2:24 PM, Eduardo Habkost wrote:
>>> On Wed, Sep 30, 2020 at 12:31:43AM -0400, John Snow wrote:
>>>> Returning a *something* or a Tuple of *something* is hard to accurately
>>>> type. Let's just always return a tuple for structural consistency.
>>>>
>>>> Instances of the 'TreeNode' type can be replaced with the slightly more
>>>> specific 'AnnotatedNode' type.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>
>>> So, the only place where this seems to make a difference is
>>> _tree_to_qlit().
>>>
>>> We just need to prove that
>>> _tree_to_qlit(o, ...)
>>> will have exactly the same result as
>>> _tree_to_qlit((o, None), ...).
>>>
>>> For reference, this is the beginning of _tree_to_qlit():
>>>
>>> | def _tree_to_qlit(obj: TreeNode,
>>> | level: int = 0,
>>> | suppress_first_indent: bool = False) -> str:
>>> |
>>> | def indent(level: int) -> str:
>>> | return level * 4 * ' '
>>> |
>>> | if isinstance(obj, tuple):
>>> | ifobj, extra = obj
>>>
>>> `obj` is the return value of _make_tree()
>>>
>>> `ifobj` is the original `obj` argument to _make_tree().
>>>
>>> | ifcond = extra.get('if')
>>>
>>> ifcond will be None.
>>>
>>> | comment = extra.get('comment')
>>>
>>> comment will be None
>>>
>>> | ret = ''
>>> | if comment:
>>> | ret += indent(level) + '/* %s */\n' % comment
>>>
>>> nop
>>>
>>> | if ifcond:
>>> | ret += gen_if(ifcond)
>>>
>>> nop
>>>
>>> | ret += _tree_to_qlit(ifobj, level)
>>>
>>> ret will be '', so this is equivalent to:
>>>
>>> ret = _tree_to_qlit(ifobj, level)
>>>
>>> which is almost good.
>>>
>>> The only difference seems to that suppress_first_indent=True will
>>> be ignored. We should pass suppress_first_indent as argument in
>>> the recursive call above, just in case.
>>>
>>
>> This is a really good spot, and I indeed hadn't considered it at all when I
>> did this.
>>
>> (I simply made the change and observed it worked just fine!)
>>
>>> The existing code will behave weirdly if there are comments or
>>> conditions and suppress_first_indent=True, but I suggest we try
>>> to address this issue later.
>>>
>>> | if ifcond:
>>> | ret += '\n' + gen_endif(ifcond)
>>>
>>> nop
>>>
>>> | return ret
>>>
>>
>> Hm, yes, it's a hypothetical case, but perhaps we can use an assertion to
>> help guard against it if development creates that case later by accident.
>>
>> That ought to be good enough for now to not waste time accommodating a
>> (presently) fictional circumstance?
>>
>> Thanks for the good sleuthing here.
>
> With the current code, both
> ret += _tree_to_qlit(ifobj, level)
> and
> ret += _tree_to_qlit(ifobj, level, suppress_first_indent)
> will behave exactly the same.
>
> The former will behave weirdly if we wrap a dictionary value using
> _tree_node(). We don't do that today.
>
> The latter will behave weirdly if there's a comment or ifcond
> attached in a dictionary value. We don't do that today.
>
> I believe the latter is less likely to be triggered by accident.
>
> But I'd be happy with either:
>
> # _make_tree() shouldn't be use to wrap nodes that
> # may be printed using suppress_first_indent=True
> # (in other words, dictionary values shouldn't be wrapped using _make_tree())
> assert(not suppress_first_indent)
> ret += _tree_to_qlit(ifobj, level)
>
> or
>
> # we can't add ifcond or comments to nodes that may be
> # printed using suppress_first_indent=True
> # (in other words, dictionary values can't have ifcond or comments)
> assert(not suppress_first_indent or (not comment and not ifcond))
> ret += _tree_to_qlit(ifobj, level, suppress_first_indent)
>
>
> If we have time to spare later, we could do this:
>
> def _value_to_qlit(obj: Union[None, str, Dict[str, object], List[object], bool],
> level: int = 0,
> suppress_first_indent: bool = False) -> str:
> ...
> if obj is None:
> ...
> elif isinstance(obj, str):
> ...
> elif isinstance(obj, list):
> ...
> ...
>
> def _tree_to_qlit(obj: TreeNode, level: int = 0) -> str:
> if isinstance(obj, AnnotatedNode):
> ...
> else:
> return _value_to_qlit(obj, level)
>
> This way, it will be impossible to set suppress_first_indent=True
> on an annotated node.
>
Maybe it's the right thing to separate out container types from leaf
types and make this mutually recursive.
I debating doing that earlier, but the patches were already so strangled
and messy I was afraid of plunging deeper into refactors.
Maybe I'll go take a nap and do it when I wake up. :)
--js
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure
2020-09-30 18:58 ` John Snow
@ 2020-09-30 19:52 ` Eduardo Habkost
2020-10-01 17:59 ` John Snow
0 siblings, 1 reply; 69+ messages in thread
From: Eduardo Habkost @ 2020-09-30 19:52 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On Wed, Sep 30, 2020 at 02:58:04PM -0400, John Snow wrote:
> On 9/30/20 2:39 PM, Eduardo Habkost wrote:
> > On Wed, Sep 30, 2020 at 12:31:45AM -0400, John Snow wrote:
> > > This replaces _make_tree with Node.__init__(), effectively. By creating
> > > it as a generic container, we can more accurately describe the exact
> > > nature of this particular Node.
> > >
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > > scripts/qapi/introspect.py | 77 +++++++++++++++++++-------------------
> > > 1 file changed, 38 insertions(+), 39 deletions(-)
> > >
> > > diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> > > index 43b6ba5df1f..86286e755ca 100644
> > > --- a/scripts/qapi/introspect.py
> > > +++ b/scripts/qapi/introspect.py
> > > @@ -12,11 +12,12 @@
> > > from typing import (
> > > Dict,
> > > + Generic,
> > > + Iterable,
> > > List,
> > > Optional,
> > > Sequence,
> > > - Tuple,
> > > - Union,
> > > + TypeVar,
> > > )
> > > from .common import (
> > > @@ -43,42 +44,42 @@
> > > # The correct type for TreeNode is actually:
> > > -# Union[AnnotatedNode, List[TreeNode], Dict[str, TreeNode], str, bool]
> > > +# Union[Node[TreeNode], List[TreeNode], Dict[str, TreeNode], str, bool]
> > > # but mypy does not support recursive types yet.
> > > TreeNode = object
> > > +_NodeType = TypeVar('_NodeType', bound=TreeNode)
> > > _DObject = Dict[str, object]
> > > -Extra = Dict[str, object]
> > > -AnnotatedNode = Tuple[TreeNode, Extra]
> >
> > Do you have plans to make Node replace TreeNode completely?
> >
> > I'd understand this patch as a means to reach that goal, but I'm
> > not sure the intermediate state of having both Node and TreeNode
> > types (that can be confused with each other) is desirable.
> >
>
> The problem is that _tree_to_qlit still accepts a broad array of types. The
> "TreeNode" comment above explains that those types are:
>
> Node[TreeNode], List[TreeNode], Dict[str, TreeNode], str, bool
>
> Three are containers, two are leaf values.
> of the containers, the Node container is special in that it houses
> explicitly one of the four other types (but never itself.)
>
> Even if I somehow always enforced Node[T] heading into _tree_to_qlit, I
> would still need to describe what 'T' is, which is another recursive type
> that I cannot exactly describe with mypy's current descriptive power:
>
> INNER_TYPE = List[Node[INNER_TYPE]], Dict[str, Node[INNER_TYPE]], str, bool
>
> And that's not really a huge win, or indeed any different to the existing
> TreeNode being an "object".
>
> So ... no, I felt like I was going to stop here, where we have
> fundamentally:
>
> 1. Undecorated nodes (list, dict, str, bool) ("TreeNode")
> 2. Decorated nodes (Node[T]) ("Node")
>
> which leads to the question: Why bother swapping Tuple for Node at that
> point?
>
> My answer is simply that having a strong type name allows us to distinguish
> this from garden-variety Tuples that might sneak in for other reasons in
> other data types.
Would:
AnnotatedNode = NewType('AnnotatedNode', Tuple[TreeNode, Extra])
be enough, then?
>
> Maybe we want a different nomenclature though, like Node vs AnnotatedNode?
Yes, I believe having a more explicit name would be better.
>
> --js
>
> (Also: 'TreeNode' is just an alias for object, it doesn't mean anything
> grammatically. I could just as soon erase it entirely if you felt it
> provided no value. It doesn't enforce that it only takes objects we declared
> were 'TreeNode' types, for instance. It's just a preprocessor name,
> basically.)
>
> > > -def _make_tree(obj: Union[_DObject, str], ifcond: List[str],
> > > - comment: Optional[str] = None) -> AnnotatedNode:
> > > - extra: Extra = {
> > > - 'if': ifcond,
> > > - 'comment': comment,
> > > - }
> > > - return (obj, extra)
> > > +class Node(Generic[_NodeType]):
> > > + """
> > > + Node generally contains a SchemaInfo-like type (as a dict),
> > > + But it also used to wrap comments/ifconds around leaf value types.
> > > + """
> > > + # Remove after 3.7 adds @dataclass:
> > > + # pylint: disable=too-few-public-methods
> > > + def __init__(self, data: _NodeType, ifcond: Iterable[str],
> > > + comment: Optional[str] = None):
> > > + self.data = data
> > > + self.comment: Optional[str] = comment
> > > + self.ifcond: Sequence[str] = tuple(ifcond)
> > > -def _tree_to_qlit(obj: TreeNode,
> > > - level: int = 0,
> > > +def _tree_to_qlit(obj: TreeNode, level: int = 0,
> > > suppress_first_indent: bool = False) -> str:
> > > def indent(level: int) -> str:
> > > return level * 4 * ' '
> > > - if isinstance(obj, tuple):
> > > - ifobj, extra = obj
> > > - ifcond = extra.get('if')
> > > - comment = extra.get('comment')
> > > + if isinstance(obj, Node):
> > > ret = ''
> > > - if comment:
> > > - ret += indent(level) + '/* %s */\n' % comment
> > > - if ifcond:
> > > - ret += gen_if(ifcond)
> > > - ret += _tree_to_qlit(ifobj, level)
> > > - if ifcond:
> > > - ret += '\n' + gen_endif(ifcond)
> > > + if obj.comment:
> > > + ret += indent(level) + '/* %s */\n' % obj.comment
> > > + if obj.ifcond:
> > > + ret += gen_if(obj.ifcond)
> > > + ret += _tree_to_qlit(obj.data, level)
> > > + if obj.ifcond:
> > > + ret += '\n' + gen_endif(obj.ifcond)
> > > return ret
> > > ret = ''
> > > @@ -125,7 +126,7 @@ def __init__(self, prefix: str, unmask: bool):
> > > ' * QAPI/QMP schema introspection', __doc__)
> > > self._unmask = unmask
> > > self._schema: Optional[QAPISchema] = None
> > > - self._trees: List[AnnotatedNode] = []
> > > + self._trees: List[Node[_DObject]] = []
> > > self._used_types: List[QAPISchemaType] = []
> > > self._name_map: Dict[str, str] = {}
> > > self._genc.add(mcgen('''
> > > @@ -192,9 +193,8 @@ def _use_type(self, typ: QAPISchemaType) -> str:
> > > @classmethod
> > > def _gen_features(cls,
> > > - features: List[QAPISchemaFeature]
> > > - ) -> List[AnnotatedNode]:
> > > - return [_make_tree(f.name, f.ifcond) for f in features]
> > > + features: List[QAPISchemaFeature]) -> List[Node[str]]:
> > > + return [Node(f.name, f.ifcond) for f in features]
> > > def _gen_tree(self, name: str, mtype: str, obj: _DObject,
> > > ifcond: List[str],
> > > @@ -210,10 +210,10 @@ def _gen_tree(self, name: str, mtype: str, obj: _DObject,
> > > obj['meta-type'] = mtype
> > > if features:
> > > obj['features'] = self._gen_features(features)
> > > - self._trees.append(_make_tree(obj, ifcond, comment))
> > > + self._trees.append(Node(obj, ifcond, comment))
> > > def _gen_member(self,
> > > - member: QAPISchemaObjectTypeMember) -> AnnotatedNode:
> > > + member: QAPISchemaObjectTypeMember) -> Node[_DObject]:
> > > obj: _DObject = {
> > > 'name': member.name,
> > > 'type': self._use_type(member.type)
> > > @@ -222,19 +222,19 @@ def _gen_member(self,
> > > obj['default'] = None
> > > if member.features:
> > > obj['features'] = self._gen_features(member.features)
> > > - return _make_tree(obj, member.ifcond)
> > > + return Node(obj, member.ifcond)
> > > def _gen_variants(self, tag_name: str,
> > > variants: List[QAPISchemaVariant]) -> _DObject:
> > > return {'tag': tag_name,
> > > 'variants': [self._gen_variant(v) for v in variants]}
> > > - def _gen_variant(self, variant: QAPISchemaVariant) -> AnnotatedNode:
> > > + def _gen_variant(self, variant: QAPISchemaVariant) -> Node[_DObject]:
> > > obj: _DObject = {
> > > 'case': variant.name,
> > > 'type': self._use_type(variant.type)
> > > }
> > > - return _make_tree(obj, variant.ifcond)
> > > + return Node(obj, variant.ifcond)
> > > def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
> > > json_type: str) -> None:
> > > @@ -245,8 +245,7 @@ def visit_enum_type(self, name: str, info: QAPISourceInfo,
> > > members: List[QAPISchemaEnumMember],
> > > prefix: Optional[str]) -> None:
> > > self._gen_tree(name, 'enum',
> > > - {'values': [_make_tree(m.name, m.ifcond, None)
> > > - for m in members]},
> > > + {'values': [Node(m.name, m.ifcond) for m in members]},
> > > ifcond, features)
> > > def visit_array_type(self, name: str, info: Optional[QAPISourceInfo],
> > > @@ -274,9 +273,9 @@ def visit_alternate_type(self, name: str, info: QAPISourceInfo,
> > > variants: QAPISchemaVariants) -> None:
> > > self._gen_tree(name, 'alternate',
> > > {'members': [
> > > - _make_tree({'type': self._use_type(m.type)},
> > > - m.ifcond, None)
> > > - for m in variants.variants]},
> > > + Node({'type': self._use_type(m.type)}, m.ifcond)
> > > + for m in variants.variants
> > > + ]},
> > > ifcond, features)
> > > def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str],
> > > --
> > > 2.26.2
> > >
> >
>
--
Eduardo
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-09-30 17:22 ` John Snow
@ 2020-10-01 8:52 ` Markus Armbruster
2020-10-01 14:48 ` John Snow
0 siblings, 1 reply; 69+ messages in thread
From: Markus Armbruster @ 2020-10-01 8:52 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Cleber Rosa, Alex Bennée
John Snow <jsnow@redhat.com> writes:
> On 9/30/20 4:47 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>>
>>> I did not say "sphinx beautiful", just "sphinx compatible". They will
>>> not throw errors when parsed and interpreted as ReST.
>> "Bang on the keyboard until Sphinx doesn't throw errors anymore"
>> might
>> be good enough for a certain kind of mathematician, but a constructive
>> solution needs a bit more direction. Is there a specification to
>> follow? Other useful resources?
>>
>
> I don't know if you are asking this question rhetorically, or in good faith.
I ask to make sure I understand goals and limitations of your doc string
work in this series.
Also, even a passing to Sphinx becomes more useful when accompanied by a
link to relevant documentation.
> Let me preface this by saying: This series, and these 119 patches, are
> not about finding a style guide for our docstring utilization or about
> proposing one. It is also not about rigorously adding such
> documentation or about finding ways to meaningfully publish it with
> e.g. Sphinx, or the styling of such pages.
>
> Why bother to add docstrings at all, then? Because I needed them for
> my own sake when learning this code and I felt it would be a waste to
> just delete them, and I am of sound mind and able body and believe
> that some documentation was better than none. They are useful even
> just as plaintext.
>
> Having said that, let's explore the actual style I tend to use.
>
> I mentioned before in response to a review comment that there isn't
> really any standard for docstrings. There are a few competing
> "styles", but none of which are able to be programmatically checked
> and validated.
>
> The primary guide for docstrings is PEP 257, of which I follow some
> but not all of the recommendations.
>
> https://www.python.org/dev/peps/pep-0257/
I find PEP 257 frustrating. It leaves me with more questions than
answers.
> In general,
>
> - Always use triple-double quotes (unenforced)
> - Modules, classes, and functions should have docstrings (pylint)
> - No empty lines before or after the docstring (unenforced)
> - Multi-line docstrings should take the form (unenforced):
>
> """
> single-line summary of function.
>
> Additional prose if needed describing the function.
>
> :param x: Input such that blah blah.
> :raises y: When input ``x`` is unsuitable because blah blah.
> :returns: A value that blah blah.
This paragraph is already not PEP 257.
> """
>
> PEP257 suggests a form where the single-line summary appears on the
> same line as the opening triple quotes. I don't like this, and prefer
> symmetry. PEP257 *also* suggests that writing it my way is equivalent
> to their way, because any docstring processor should trim the first
> line. I take this as tacit admission that my way is acceptable and has
> merit.
I prefer the symmetric form myself.
> What about the syntax or markup inside the docstring itself? there is
> *absolutely no standard*, but Sphinx autodoc recognizes a few field
> lists as significant in its parsing, so I prefer using them:
Doc link?
> :param x: Denotes the parameter X. Do not use type information in the
> string, we rely on mypy for that now.
>
> :raises y: explains a case in which an Exception type y may be raised
> either directly by this code or anticipated to be allowed to be raised
> by a helper call. (There's no standard here that I am aware of. I use
> my judgment. Always document direct raise calls, but use your judgment
> for sub-calls.)
>
> :returns: explains the semantics of the return value.
>
> That said, literally any sphinx/ReST markup is OK as long as it passes
> make sphinxdocs. Some sphinx markup is prohibited, like adding new
> full-throated sections. You can use arbitrary field lists, definition
> lists, pre-formatted text, examples, code blocks, whatever.
>
> In general, you are not going to find the kind of semantic validation
> you want to ensure that the parameter names are correct, or that you
> spelled :param: right, or that you didn't miss a parameter or an
> exception. None of that tooling exists for Python.
>
> Thus, it's all rather subjective. No right answers, no validation
> tools. Just whatever seems reasonable to a human eye until such time
> we actually decide to pursue publishing the API docs in the
> development manual, if indeed we ever do so at all.
>
> That series sounds like a great opportunity to hash this all out. That
> is when I would like to remove --missing-docstring, too. There will
> absolutely be a "docstring series" in the future, but I am insisting
> stubbornly it happen after strict typing.
Okay. Nevertheless, I'd prefer a bit more information in the commit
message. Here's my try:
qapi: Modify docstrings to be sphinx-compatible
I did not say "sphinx beautiful", just "sphinx compatible". They
will not throw errors when parsed and interpreted as ReST. Finding
a comprehensive style guide for our docstring utilization is left
for another day.
For now, use field lists recognized by Sphinx autodoc.
FIXME link to their documentation
>
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>> scripts/qapi/gen.py | 6 ++++--
>>> scripts/qapi/parser.py | 9 +++++----
>>> 2 files changed, 9 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>>> index ca66c82b5b8..fc19b2aeb9b 100644
>>> --- a/scripts/qapi/gen.py
>>> +++ b/scripts/qapi/gen.py
>>> @@ -154,9 +154,11 @@ def _bottom(self):
>>> @contextmanager
>>> def ifcontext(ifcond, *args):
>>> - """A 'with' statement context manager to wrap with start_if()/end_if()
>>> + """
>>> + A 'with' statement context manager that wraps with `start_if` and `end_if`.
>> Sadly, the fact that start_if() and end_if() are functions isn't
>> immediately obvious anymore.
>> I've seen :func:`start_if` elsewhere. Is this something we should
>> or
>> want to use?
>>
>
> We *could*.
>
> `start_if` relies on the default role, which I have provisionally set
> to "any" here, so this is shorthand for :any:`start_if`.
>
> The :any: role means: "cross-reference any type of thing." If there is
> not exactly one thing that matches, it results in an error during the
> documentation build.
>
> I like this, because it's nice short-hand syntax that I think
> communicates effectively to the reader that this is a symbol of some
> kind without needing a premium of ReST-ese.
>
> CONSTANTS are capitalized, Classes are title cased, and functions are
> lower_case. `lower_case` references can be assumed to be functions,
`lower_case` could also refer to an attribute, variable, or parameter.
> but I will admit that this is not enforced or necessarily true as we
> add more cross reference types in the future.
>
> (I am trying to add QMP cross-reference syntax!)
>
> I still prefer `start_if` to :func:`start_if` simply because it's less
> markup and is easier to read in plaintext contexts. You're right, it
> doesn't look like a function anymore.
Yes, :func:`start_if` is rather heavy. I asked because I wanted to
understand what :func: buys us. Not meant as endorsement.
GDK-Doc seems smart enough to recognize start_if(). Sphinx isn't,
because it's built around reST syntax. We put our money on the Sphinx
horse, so...
> I'm not sure if another annotations would work -- `start_if`() or
> `start_if()`. Both seem kind of clunky to me, to be honest. Personal
> feeling is "not really worth the hassle."
You later reported the latter works.
I prefer `start_if()` to `start_if`. Matter of taste.
>
>>> - *args: any number of QAPIGenCCode
>>> + :param ifcond: List of conditionals
>>> + :param args: any number of `QAPIGenCCode`.
>>> Example::
>>> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>>> index 9d1a3e2eea9..02983979965 100644
>>> --- a/scripts/qapi/parser.py
>>> +++ b/scripts/qapi/parser.py
>>> @@ -381,10 +381,11 @@ def append(self, line):
>>> The way that the line is dealt with depends on which
>>> part of
>>> the documentation we're parsing right now:
>>> - * The body section: ._append_line is ._append_body_line
>>> - * An argument section: ._append_line is ._append_args_line
>>> - * A features section: ._append_line is ._append_features_line
>>> - * An additional section: ._append_line is ._append_various_line
>>> +
>>> + * The body section: ._append_line is ._append_body_line
>>> + * An argument section: ._append_line is ._append_args_line
>>> + * A features section: ._append_line is ._append_features_line
>>> + * An additional section: ._append_line is ._append_various_line
>>> """
>>> line = line[1:]
>>> if not line:
>> I understand why you insert a blank line (reST wants blank lines
>> around
>> lists), I don't understand why you indent. Can you explain?
>
> I was mistaken about it needing the indent!
Easy enough to tidy up :)
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-09-30 17:38 ` John Snow
@ 2020-10-01 8:54 ` Markus Armbruster
2020-10-01 14:28 ` John Snow
0 siblings, 1 reply; 69+ messages in thread
From: Markus Armbruster @ 2020-10-01 8:54 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Cleber Rosa, Alex Bennée
John Snow <jsnow@redhat.com> writes:
> On 9/30/20 4:47 AM, Markus Armbruster wrote:
>> Sadly, the fact that start_if() and end_if() are functions isn't
>> immediately obvious anymore.
>> I've seen :func:`start_if` elsewhere. Is this something we should
>> or
>> want to use?
>
> Looks like `start_if()` works just fine too. If you are hard-set in
> wanting to see those parentheses I can use this style, but admit I am
> not likely to use them myself in newer docstrings, and there's no way
> to enforce their presence OR absence that I am aware of.
Well, there's no way to enforce presence or absence of ` either, right?
Aside: checking the enclosed text resolves as a cross-reference could
provide good-enough enforcement of absence, but enclosing stuff in `
when you shouldn't is probably the lesser issue.
> I'll bake that change in for now until I see another reply.
Since this is a side show right now, consider the arguments raised so
far, use your judgement, and move on to the main show.
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-10-01 8:54 ` Markus Armbruster
@ 2020-10-01 14:28 ` John Snow
0 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-10-01 14:28 UTC (permalink / raw)
To: Markus Armbruster
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Cleber Rosa, Alex Bennée
On 10/1/20 4:54 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
>
>> On 9/30/20 4:47 AM, Markus Armbruster wrote:
>>> Sadly, the fact that start_if() and end_if() are functions isn't
>>> immediately obvious anymore.
>>> I've seen :func:`start_if` elsewhere. Is this something we should
>>> or
>>> want to use?
>>
>> Looks like `start_if()` works just fine too. If you are hard-set in
>> wanting to see those parentheses I can use this style, but admit I am
>> not likely to use them myself in newer docstrings, and there's no way
>> to enforce their presence OR absence that I am aware of.
>
> Well, there's no way to enforce presence or absence of ` either, right?
>
Yeah, just a warning that enforcing mechanical consistency along these
lines is not something we can do at the moment.
(We maybe could by creating a sphinx plugin, but that's a bridge to
cross much, much later. Anything is possible with time, engineers, and
money, right?)
> Aside: checking the enclosed text resolves as a cross-reference could
> provide good-enough enforcement of absence, but enclosing stuff in `
> when you shouldn't is probably the lesser issue.
>
>> I'll bake that change in for now until I see another reply.
>
> Since this is a side show right now, consider the arguments raised so
> far, use your judgement, and move on to the main show.
>
Keeping emails out of my inbox is presently the nicest thing :)
Thanks for looking.
--js
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-10-01 8:52 ` Markus Armbruster
@ 2020-10-01 14:48 ` John Snow
2020-10-02 9:19 ` Markus Armbruster
0 siblings, 1 reply; 69+ messages in thread
From: John Snow @ 2020-10-01 14:48 UTC (permalink / raw)
To: Markus Armbruster
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Cleber Rosa, Alex Bennée
On 10/1/20 4:52 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
>
>> On 9/30/20 4:47 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> I did not say "sphinx beautiful", just "sphinx compatible". They will
>>>> not throw errors when parsed and interpreted as ReST.
>>> "Bang on the keyboard until Sphinx doesn't throw errors anymore"
>>> might
>>> be good enough for a certain kind of mathematician, but a constructive
>>> solution needs a bit more direction. Is there a specification to
>>> follow? Other useful resources?
>>>
>>
>> I don't know if you are asking this question rhetorically, or in good faith.
>
> I ask to make sure I understand goals and limitations of your doc string
> work in this series.
>
> Also, even a passing to Sphinx becomes more useful when accompanied by a
> link to relevant documentation.
>
>> Let me preface this by saying: This series, and these 119 patches, are
>> not about finding a style guide for our docstring utilization or about
>> proposing one. It is also not about rigorously adding such
>> documentation or about finding ways to meaningfully publish it with
>> e.g. Sphinx, or the styling of such pages.
>>
>> Why bother to add docstrings at all, then? Because I needed them for
>> my own sake when learning this code and I felt it would be a waste to
>> just delete them, and I am of sound mind and able body and believe
>> that some documentation was better than none. They are useful even
>> just as plaintext.
>>
>> Having said that, let's explore the actual style I tend to use.
>>
>> I mentioned before in response to a review comment that there isn't
>> really any standard for docstrings. There are a few competing
>> "styles", but none of which are able to be programmatically checked
>> and validated.
>>
>> The primary guide for docstrings is PEP 257, of which I follow some
>> but not all of the recommendations.
>>
>> https://www.python.org/dev/peps/pep-0257/
>
> I find PEP 257 frustrating. It leaves me with more questions than
> answers.
>
Yeah, sorry. That's what we're dealing with. It's very open-ended.
>> In general,
>>
>> - Always use triple-double quotes (unenforced)
>> - Modules, classes, and functions should have docstrings (pylint)
>> - No empty lines before or after the docstring (unenforced)
>> - Multi-line docstrings should take the form (unenforced):
>>
>> """
>> single-line summary of function.
>>
>> Additional prose if needed describing the function.
>>
>> :param x: Input such that blah blah.
>> :raises y: When input ``x`` is unsuitable because blah blah.
>> :returns: A value that blah blah.
>
> This paragraph is already not PEP 257.
>
Right -- well, it isn't NOT PEP 257 either. It just suggests you have to
describe these features, but it doesn't say HOW.
>> """
>>
>> PEP257 suggests a form where the single-line summary appears on the
>> same line as the opening triple quotes. I don't like this, and prefer
>> symmetry. PEP257 *also* suggests that writing it my way is equivalent
>> to their way, because any docstring processor should trim the first
>> line. I take this as tacit admission that my way is acceptable and has
>> merit.
>
> I prefer the symmetric form myself.
>
>> What about the syntax or markup inside the docstring itself? there is
>> *absolutely no standard*, but Sphinx autodoc recognizes a few field
>> lists as significant in its parsing, so I prefer using them:
>
> Doc link?
>
Hard to search for in my opinion; you want to search for "sphinx python
domain", and then click on "field lists" on the sidebar.
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
It has a special understanding for:
param/parameter/arg/argument/key/keyword: I prefer "param" here.
Possibly key/keyword if we use a **kwargs form with a key that we
specially recognize, but I've not tested that yet. I know pycharm
understands "param" in a semantic way, and that's been good enough for me.
type: Defines the type of a parameter. In my opinion, do not use this.
Let native type hints do the lifting.
raises/raise/except/exception: I prefer "raises". "raises ErrorType
when...." is a good sentence.
var/ivar/cvar: Describes a variable, presumably in the body of the
function below. I've never used this, I always describe it in prose instead.
vartype: Defines a type for a variable; I would again defer to the
native type system instead now.
returns/return: I prefer "returns" for grammatical reasons again.
("Returns such-and-such when...")
rtype: again, type information. Don't use.
meta: For directives to sphinx, e.g. :meta private: or :meta public: to
toggle the visibility class from its default. I don't use this.
None of these are validated or checked in any meaningful way; you can
use arbitrarily field lists (and I do in a few places!) to define your
own terms and so on.
(I would like to improve autodoc in the future to validate your
docstrings such that you can enforce :param:, :raises: and :return: and
it uses the type hints and introspection information to raise an error
when you make an obvious mistake. I am not there yet, but I am using
Peter Maydell's work to help inform how I might write such an extension
to autodoc. This work is not critical, but it will likely occur
upstream, outside of the QEMU context because I believe this is a good
thing to do for the ecosystem in general, to allow autodoc to function
slightly more like e.g. Doxygen does.)
>> :param x: Denotes the parameter X. Do not use type information in the
>> string, we rely on mypy for that now.
>>
>> :raises y: explains a case in which an Exception type y may be raised
>> either directly by this code or anticipated to be allowed to be raised
>> by a helper call. (There's no standard here that I am aware of. I use
>> my judgment. Always document direct raise calls, but use your judgment
>> for sub-calls.)
>>
>> :returns: explains the semantics of the return value.
>>
>> That said, literally any sphinx/ReST markup is OK as long as it passes
>> make sphinxdocs. Some sphinx markup is prohibited, like adding new
>> full-throated sections. You can use arbitrary field lists, definition
>> lists, pre-formatted text, examples, code blocks, whatever.
>>
>> In general, you are not going to find the kind of semantic validation
>> you want to ensure that the parameter names are correct, or that you
>> spelled :param: right, or that you didn't miss a parameter or an
>> exception. None of that tooling exists for Python.
>>
>> Thus, it's all rather subjective. No right answers, no validation
>> tools. Just whatever seems reasonable to a human eye until such time
>> we actually decide to pursue publishing the API docs in the
>> development manual, if indeed we ever do so at all.
>>
>> That series sounds like a great opportunity to hash this all out. That
>> is when I would like to remove --missing-docstring, too. There will
>> absolutely be a "docstring series" in the future, but I am insisting
>> stubbornly it happen after strict typing.
>
> Okay. Nevertheless, I'd prefer a bit more information in the commit
> message. Here's my try:
>
> qapi: Modify docstrings to be sphinx-compatible
>
> I did not say "sphinx beautiful", just "sphinx compatible". They
> will not throw errors when parsed and interpreted as ReST. Finding
> a comprehensive style guide for our docstring utilization is left
> for another day.
>
> For now, use field lists recognized by Sphinx autodoc.
> FIXME link to their documentation
>
That I can do -- and I will double down on my IOU for a more formal
style guide: https://gitlab.com/jsnow/qemu/-/issues/7
I didn't bother writing it in any of the commits because I felt like
it'd get lost there and would be mostly useless; but a .rst doc inside
the package folder would be hard to miss.
I plan to check in something like ./python/README.rst or
./python/CODING_STYLE.rst to try and formalize a lot of what I am doing
here, where it's going to be harder to miss.
>>
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>> scripts/qapi/gen.py | 6 ++++--
>>>> scripts/qapi/parser.py | 9 +++++----
>>>> 2 files changed, 9 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>>>> index ca66c82b5b8..fc19b2aeb9b 100644
>>>> --- a/scripts/qapi/gen.py
>>>> +++ b/scripts/qapi/gen.py
>>>> @@ -154,9 +154,11 @@ def _bottom(self):
>>>> @contextmanager
>>>> def ifcontext(ifcond, *args):
>>>> - """A 'with' statement context manager to wrap with start_if()/end_if()
>>>> + """
>>>> + A 'with' statement context manager that wraps with `start_if` and `end_if`.
>>> Sadly, the fact that start_if() and end_if() are functions isn't
>>> immediately obvious anymore.
>>> I've seen :func:`start_if` elsewhere. Is this something we should
>>> or
>>> want to use?
>>>
>>
>> We *could*.
>>
>> `start_if` relies on the default role, which I have provisionally set
>> to "any" here, so this is shorthand for :any:`start_if`.
>>
>> The :any: role means: "cross-reference any type of thing." If there is
>> not exactly one thing that matches, it results in an error during the
>> documentation build.
>>
>> I like this, because it's nice short-hand syntax that I think
>> communicates effectively to the reader that this is a symbol of some
>> kind without needing a premium of ReST-ese.
>>
>> CONSTANTS are capitalized, Classes are title cased, and functions are
>> lower_case. `lower_case` references can be assumed to be functions,
>
> `lower_case` could also refer to an attribute, variable, or parameter.
>
Hm. Attribute yes, actually. variable and parameter no -- sphinx does
not presently provide syntax or roles for creating anchors to parameter
names or variables, so they are not able to be cross-referenced.
Attributes CAN be cross-referenced, but only when they are documented.
Another style guide thing:
#: x is a number that represents "The Answer". See `Douglas Adams`_.
self.x = 42
You can use the special comment form "#:" to add a one-line description
of an attribute that Sphinx will pick up. Sphinx skips these attributes
otherwise. If you consider them part of the interface of the module,
it's maybe a good idea to do this.
You can also use docstrings, but the ordering changes:
self.x = 42
"""x is a number that represents "The Answer". See `Douglas Adams`_.
I kind of like the #: form because it announces what follows, but I
admit it's a bit of special sphinx magic.
>> but I will admit that this is not enforced or necessarily true as we
>> add more cross reference types in the future.
>>
>> (I am trying to add QMP cross-reference syntax!)
>>
>> I still prefer `start_if` to :func:`start_if` simply because it's less
>> markup and is easier to read in plaintext contexts. You're right, it
>> doesn't look like a function anymore.
>
> Yes, :func:`start_if` is rather heavy. I asked because I wanted to
> understand what :func: buys us. Not meant as endorsement.
>
It specifically targets only cross-references of that exact type. In the
case that the :any: reference is ambiguous, :func: is the disambiguation.
> GDK-Doc seems smart enough to recognize start_if(). Sphinx isn't,
> because it's built around reST syntax. We put our money on the Sphinx
> horse, so...
>
>> I'm not sure if another annotations would work -- `start_if`() or
>> `start_if()`. Both seem kind of clunky to me, to be honest. Personal
>> feeling is "not really worth the hassle."
>
> You later reported the latter works.
>
> I prefer `start_if()` to `start_if`. Matter of taste.
>
Change made.
>>
>>>> - *args: any number of QAPIGenCCode
>>>> + :param ifcond: List of conditionals
>>>> + :param args: any number of `QAPIGenCCode`.
>>>> Example::
>>>> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>>>> index 9d1a3e2eea9..02983979965 100644
>>>> --- a/scripts/qapi/parser.py
>>>> +++ b/scripts/qapi/parser.py
>>>> @@ -381,10 +381,11 @@ def append(self, line):
>>>> The way that the line is dealt with depends on which
>>>> part of
>>>> the documentation we're parsing right now:
>>>> - * The body section: ._append_line is ._append_body_line
>>>> - * An argument section: ._append_line is ._append_args_line
>>>> - * A features section: ._append_line is ._append_features_line
>>>> - * An additional section: ._append_line is ._append_various_line
>>>> +
>>>> + * The body section: ._append_line is ._append_body_line
>>>> + * An argument section: ._append_line is ._append_args_line
>>>> + * A features section: ._append_line is ._append_features_line
>>>> + * An additional section: ._append_line is ._append_various_line
>>>> """
>>>> line = line[1:]
>>>> if not line:
>>> I understand why you insert a blank line (reST wants blank lines
>>> around
>>> lists), I don't understand why you indent. Can you explain?
>>
>> I was mistaken about it needing the indent!
>
> Easy enough to tidy up :)
>
Already done!
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure
2020-09-30 19:52 ` Eduardo Habkost
@ 2020-10-01 17:59 ` John Snow
0 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-10-01 17:59 UTC (permalink / raw)
To: Eduardo Habkost
Cc: Peter Maydell, Thomas Huth, qemu-devel, Markus Armbruster,
Cleber Rosa, Alex Bennée
On 9/30/20 3:52 PM, Eduardo Habkost wrote:
> On Wed, Sep 30, 2020 at 02:58:04PM -0400, John Snow wrote:
>> On 9/30/20 2:39 PM, Eduardo Habkost wrote:
>>> On Wed, Sep 30, 2020 at 12:31:45AM -0400, John Snow wrote:
>>>> This replaces _make_tree with Node.__init__(), effectively. By creating
>>>> it as a generic container, we can more accurately describe the exact
>>>> nature of this particular Node.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>> scripts/qapi/introspect.py | 77 +++++++++++++++++++-------------------
>>>> 1 file changed, 38 insertions(+), 39 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
>>>> index 43b6ba5df1f..86286e755ca 100644
>>>> --- a/scripts/qapi/introspect.py
>>>> +++ b/scripts/qapi/introspect.py
>>>> @@ -12,11 +12,12 @@
>>>> from typing import (
>>>> Dict,
>>>> + Generic,
>>>> + Iterable,
>>>> List,
>>>> Optional,
>>>> Sequence,
>>>> - Tuple,
>>>> - Union,
>>>> + TypeVar,
>>>> )
>>>> from .common import (
>>>> @@ -43,42 +44,42 @@
>>>> # The correct type for TreeNode is actually:
>>>> -# Union[AnnotatedNode, List[TreeNode], Dict[str, TreeNode], str, bool]
>>>> +# Union[Node[TreeNode], List[TreeNode], Dict[str, TreeNode], str, bool]
>>>> # but mypy does not support recursive types yet.
>>>> TreeNode = object
>>>> +_NodeType = TypeVar('_NodeType', bound=TreeNode)
>>>> _DObject = Dict[str, object]
>>>> -Extra = Dict[str, object]
>>>> -AnnotatedNode = Tuple[TreeNode, Extra]
>>>
>>> Do you have plans to make Node replace TreeNode completely?
>>>
>>> I'd understand this patch as a means to reach that goal, but I'm
>>> not sure the intermediate state of having both Node and TreeNode
>>> types (that can be confused with each other) is desirable.
>>>
>>
>> The problem is that _tree_to_qlit still accepts a broad array of types. The
>> "TreeNode" comment above explains that those types are:
>>
>> Node[TreeNode], List[TreeNode], Dict[str, TreeNode], str, bool
>>
>> Three are containers, two are leaf values.
>> of the containers, the Node container is special in that it houses
>> explicitly one of the four other types (but never itself.)
>>
>> Even if I somehow always enforced Node[T] heading into _tree_to_qlit, I
>> would still need to describe what 'T' is, which is another recursive type
>> that I cannot exactly describe with mypy's current descriptive power:
>>
>> INNER_TYPE = List[Node[INNER_TYPE]], Dict[str, Node[INNER_TYPE]], str, bool
>>
>> And that's not really a huge win, or indeed any different to the existing
>> TreeNode being an "object".
>>
>> So ... no, I felt like I was going to stop here, where we have
>> fundamentally:
>>
>> 1. Undecorated nodes (list, dict, str, bool) ("TreeNode")
>> 2. Decorated nodes (Node[T]) ("Node")
>>
>> which leads to the question: Why bother swapping Tuple for Node at that
>> point?
>>
>> My answer is simply that having a strong type name allows us to distinguish
>> this from garden-variety Tuples that might sneak in for other reasons in
>> other data types.
>
> Would:
> AnnotatedNode = NewType('AnnotatedNode', Tuple[TreeNode, Extra])
> be enough, then?
>
I don't think so, because the runtime check still checks for tuple. I
like the consistency and simplicity of a named type.
>>
>> Maybe we want a different nomenclature though, like Node vs AnnotatedNode?
>
> Yes, I believe having a more explicit name would be better.
>
I give up on introspect.py; I'm dropping it from my series. I cannot
possibly justify another single second spent here.
--js
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-10-01 14:48 ` John Snow
@ 2020-10-02 9:19 ` Markus Armbruster
2020-10-02 15:14 ` John Snow
0 siblings, 1 reply; 69+ messages in thread
From: Markus Armbruster @ 2020-10-02 9:19 UTC (permalink / raw)
To: John Snow
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Markus Armbruster, Cleber Rosa, Alex Bennée
John Snow <jsnow@redhat.com> writes:
> On 10/1/20 4:52 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>>
>>> On 9/30/20 4:47 AM, Markus Armbruster wrote:
>>>> John Snow <jsnow@redhat.com> writes:
>>>>
>>>>> I did not say "sphinx beautiful", just "sphinx compatible". They will
>>>>> not throw errors when parsed and interpreted as ReST.
>>>> "Bang on the keyboard until Sphinx doesn't throw errors anymore"
>>>> might
>>>> be good enough for a certain kind of mathematician, but a constructive
>>>> solution needs a bit more direction. Is there a specification to
>>>> follow? Other useful resources?
>>>>
>>>
>>> I don't know if you are asking this question rhetorically, or in good faith.
>> I ask to make sure I understand goals and limitations of your doc
>> string
>> work in this series.
>> Also, even a passing to Sphinx becomes more useful when accompanied
>> by a
>> link to relevant documentation.
>>
>>> Let me preface this by saying: This series, and these 119 patches, are
>>> not about finding a style guide for our docstring utilization or about
>>> proposing one. It is also not about rigorously adding such
>>> documentation or about finding ways to meaningfully publish it with
>>> e.g. Sphinx, or the styling of such pages.
>>>
>>> Why bother to add docstrings at all, then? Because I needed them for
>>> my own sake when learning this code and I felt it would be a waste to
>>> just delete them, and I am of sound mind and able body and believe
>>> that some documentation was better than none. They are useful even
>>> just as plaintext.
>>>
>>> Having said that, let's explore the actual style I tend to use.
>>>
>>> I mentioned before in response to a review comment that there isn't
>>> really any standard for docstrings. There are a few competing
>>> "styles", but none of which are able to be programmatically checked
>>> and validated.
>>>
>>> The primary guide for docstrings is PEP 257, of which I follow some
>>> but not all of the recommendations.
>>>
>>> https://www.python.org/dev/peps/pep-0257/
>>
>> I find PEP 257 frustrating. It leaves me with more questions than
>> answers.
>
> Yeah, sorry. That's what we're dealing with. It's very open-ended.
>
>>> In general,
>>>
>>> - Always use triple-double quotes (unenforced)
>>> - Modules, classes, and functions should have docstrings (pylint)
>>> - No empty lines before or after the docstring (unenforced)
>>> - Multi-line docstrings should take the form (unenforced):
>>>
>>> """
>>> single-line summary of function.
>>>
>>> Additional prose if needed describing the function.
>>>
>>> :param x: Input such that blah blah.
>>> :raises y: When input ``x`` is unsuitable because blah blah.
>>> :returns: A value that blah blah.
>> This paragraph is already not PEP 257.
>>
>
> Right -- well, it isn't NOT PEP 257 either. It just suggests you have
> to describe these features, but it doesn't say HOW.
Yep. Frustrating.
>>> """
>>>
>>> PEP257 suggests a form where the single-line summary appears on the
>>> same line as the opening triple quotes. I don't like this, and prefer
>>> symmetry. PEP257 *also* suggests that writing it my way is equivalent
>>> to their way, because any docstring processor should trim the first
>>> line. I take this as tacit admission that my way is acceptable and has
>>> merit.
>> I prefer the symmetric form myself.
>>
>>> What about the syntax or markup inside the docstring itself? there is
>>> *absolutely no standard*, but Sphinx autodoc recognizes a few field
>>> lists as significant in its parsing, so I prefer using them:
>>
>> Doc link?
>
> Hard to search for in my opinion;
More reason to provide a link!
> you want to search for "sphinx
> python domain", and then click on "field lists" on the sidebar.
>
> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
>
> It has a special understanding for:
>
> param/parameter/arg/argument/key/keyword: I prefer "param"
> here. Possibly key/keyword if we use a **kwargs form with a key that
> we specially recognize, but I've not tested that yet. I know pycharm
> understands "param" in a semantic way, and that's been good enough for me.
>
> type: Defines the type of a parameter. In my opinion, do not use
> this. Let native type hints do the lifting.
Agree.
> raises/raise/except/exception: I prefer "raises". "raises ErrorType
> when...." is a good sentence.
>
> var/ivar/cvar: Describes a variable, presumably in the body of the
> function below. I've never used this, I always describe it in prose
> instead.
>
> vartype: Defines a type for a variable; I would again defer to the
> native type system instead now.
>
> returns/return: I prefer "returns" for grammatical reasons
> again. ("Returns such-and-such when...")
"Return such-and-such when..." is just as correct: imperative mood. I
prefer imperative mood for function contracts.
> rtype: again, type information. Don't use.
>
> meta: For directives to sphinx, e.g. :meta private: or :meta public:
> to toggle the visibility class from its default. I don't use this.
>
>
> None of these are validated or checked in any meaningful way; you can
> use arbitrarily field lists (and I do in a few places!) to define your
> own terms and so on.
>
>
> (I would like to improve autodoc in the future to validate your
> docstrings such that you can enforce :param:, :raises: and :return:
> and it uses the type hints and introspection information to raise an
> error when you make an obvious mistake. I am not there yet, but I am
> using Peter Maydell's work to help inform how I might write such an
> extension to autodoc. This work is not critical, but it will likely
> occur upstream, outside of the QEMU context because I believe this is
> a good thing to do for the ecosystem in general, to allow autodoc to
> function slightly more like e.g. Doxygen does.)
Sounds useful, but yes, it's clearly outside QEMU context.
>>> :param x: Denotes the parameter X. Do not use type information in the
>>> string, we rely on mypy for that now.
>>>
>>> :raises y: explains a case in which an Exception type y may be raised
>>> either directly by this code or anticipated to be allowed to be raised
>>> by a helper call. (There's no standard here that I am aware of. I use
>>> my judgment. Always document direct raise calls, but use your judgment
>>> for sub-calls.)
>>>
>>> :returns: explains the semantics of the return value.
>>>
>>> That said, literally any sphinx/ReST markup is OK as long as it passes
>>> make sphinxdocs. Some sphinx markup is prohibited, like adding new
>>> full-throated sections. You can use arbitrary field lists, definition
>>> lists, pre-formatted text, examples, code blocks, whatever.
>>>
>>> In general, you are not going to find the kind of semantic validation
>>> you want to ensure that the parameter names are correct, or that you
>>> spelled :param: right, or that you didn't miss a parameter or an
>>> exception. None of that tooling exists for Python.
>>>
>>> Thus, it's all rather subjective. No right answers, no validation
>>> tools. Just whatever seems reasonable to a human eye until such time
>>> we actually decide to pursue publishing the API docs in the
>>> development manual, if indeed we ever do so at all.
>>>
>>> That series sounds like a great opportunity to hash this all out. That
>>> is when I would like to remove --missing-docstring, too. There will
>>> absolutely be a "docstring series" in the future, but I am insisting
>>> stubbornly it happen after strict typing.
>>
>> Okay. Nevertheless, I'd prefer a bit more information in the commit
>> message. Here's my try:
>>
>> qapi: Modify docstrings to be sphinx-compatible
>>
>> I did not say "sphinx beautiful", just "sphinx compatible". They
>> will not throw errors when parsed and interpreted as ReST. Finding
>> a comprehensive style guide for our docstring utilization is left
>> for another day.
>>
>> For now, use field lists recognized by Sphinx autodoc.
>> FIXME link to their documentation
>
> That I can do -- and I will double down on my IOU for a more formal
> style guide: https://gitlab.com/jsnow/qemu/-/issues/7
>
> I didn't bother writing it in any of the commits because I felt like
> it'd get lost there and would be mostly useless; but a .rst doc inside
> the package folder would be hard to miss.
>
> I plan to check in something like ./python/README.rst or
> ./python/CODING_STYLE.rst to try and formalize a lot of what I am
> doing here, where it's going to be harder to miss.
Makes sense.
>>>>>
>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>>> ---
>>>>> scripts/qapi/gen.py | 6 ++++--
>>>>> scripts/qapi/parser.py | 9 +++++----
>>>>> 2 files changed, 9 insertions(+), 6 deletions(-)
>>>>>
>>>>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>>>>> index ca66c82b5b8..fc19b2aeb9b 100644
>>>>> --- a/scripts/qapi/gen.py
>>>>> +++ b/scripts/qapi/gen.py
>>>>> @@ -154,9 +154,11 @@ def _bottom(self):
>>>>> @contextmanager
>>>>> def ifcontext(ifcond, *args):
>>>>> - """A 'with' statement context manager to wrap with start_if()/end_if()
>>>>> + """
>>>>> + A 'with' statement context manager that wraps with `start_if` and `end_if`.
>>>> Sadly, the fact that start_if() and end_if() are functions isn't
>>>> immediately obvious anymore.
>>>> I've seen :func:`start_if` elsewhere. Is this something we should
>>>> or
>>>> want to use?
>>>>
>>>
>>> We *could*.
>>>
>>> `start_if` relies on the default role, which I have provisionally set
>>> to "any" here, so this is shorthand for :any:`start_if`.
>>>
>>> The :any: role means: "cross-reference any type of thing." If there is
>>> not exactly one thing that matches, it results in an error during the
>>> documentation build.
>>>
>>> I like this, because it's nice short-hand syntax that I think
>>> communicates effectively to the reader that this is a symbol of some
>>> kind without needing a premium of ReST-ese.
>>>
>>> CONSTANTS are capitalized, Classes are title cased, and functions are
>>> lower_case. `lower_case` references can be assumed to be functions,
>>
>> `lower_case` could also refer to an attribute, variable, or
>> parameter.
>
> Hm. Attribute yes, actually. variable and parameter no -- sphinx does
> not presently provide syntax or roles for creating anchors to
> parameter names or variables, so they are not able to be
> cross-referenced.
How would you mark up variable names in doc strings? Often, their
"variableness" is obvious from context, but not always. In C comments,
we tend to use @var [*].
> Attributes CAN be cross-referenced, but only when they are documented.
>
> Another style guide thing:
>
> #: x is a number that represents "The Answer". See `Douglas Adams`_.
> self.x = 42
>
> You can use the special comment form "#:" to add a one-line
> description of an attribute that Sphinx will pick up. Sphinx skips
> these attributes otherwise. If you consider them part of the interface
> of the module, it's maybe a good idea to do this.
>
> You can also use docstrings, but the ordering changes:
>
> self.x = 42
> """x is a number that represents "The Answer". See `Douglas Adams`_.
>
> I kind of like the #: form because it announces what follows, but I
> admit it's a bit of special sphinx magic.
Are both equally available in Python IDEs and in interactive Python?
>>> but I will admit that this is not enforced or necessarily true as we
>>> add more cross reference types in the future.
>>>
>>> (I am trying to add QMP cross-reference syntax!)
>>>
>>> I still prefer `start_if` to :func:`start_if` simply because it's less
>>> markup and is easier to read in plaintext contexts. You're right, it
>>> doesn't look like a function anymore.
>> Yes, :func:`start_if` is rather heavy. I asked because I wanted to
>> understand what :func: buys us. Not meant as endorsement.
>>
>
> It specifically targets only cross-references of that exact type. In
> the case that the :any: reference is ambiguous, :func: is the
> disambiguation.
>
>> GDK-Doc seems smart enough to recognize start_if(). Sphinx isn't,
>> because it's built around reST syntax. We put our money on the Sphinx
>> horse, so...
>>
>>> I'm not sure if another annotations would work -- `start_if`() or
>>> `start_if()`. Both seem kind of clunky to me, to be honest. Personal
>>> feeling is "not really worth the hassle."
>>
>> You later reported the latter works.
>> I prefer `start_if()` to `start_if`. Matter of taste.
>
> Change made.
Thanks!
>>>>> - *args: any number of QAPIGenCCode
>>>>> + :param ifcond: List of conditionals
>>>>> + :param args: any number of `QAPIGenCCode`.
>>>>> Example::
>>>>> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>>>>> index 9d1a3e2eea9..02983979965 100644
>>>>> --- a/scripts/qapi/parser.py
>>>>> +++ b/scripts/qapi/parser.py
>>>>> @@ -381,10 +381,11 @@ def append(self, line):
>>>>> The way that the line is dealt with depends on which
>>>>> part of
>>>>> the documentation we're parsing right now:
>>>>> - * The body section: ._append_line is ._append_body_line
>>>>> - * An argument section: ._append_line is ._append_args_line
>>>>> - * A features section: ._append_line is ._append_features_line
>>>>> - * An additional section: ._append_line is ._append_various_line
>>>>> +
>>>>> + * The body section: ._append_line is ._append_body_line
>>>>> + * An argument section: ._append_line is ._append_args_line
>>>>> + * A features section: ._append_line is ._append_features_line
>>>>> + * An additional section: ._append_line is ._append_various_line
>>>>> """
>>>>> line = line[1:]
>>>>> if not line:
>>>> I understand why you insert a blank line (reST wants blank lines
>>>> around
>>>> lists), I don't understand why you indent. Can you explain?
>>>
>>> I was mistaken about it needing the indent!
>> Easy enough to tidy up :)
>>
>
> Already done!
Thanks again!
[*] GTK-Doc says @var is just for parameters, but since it offers
nothing for variables, we sometimes use it for variables as well.
^ permalink raw reply [flat|nested] 69+ messages in thread
* Re: [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible
2020-10-02 9:19 ` Markus Armbruster
@ 2020-10-02 15:14 ` John Snow
0 siblings, 0 replies; 69+ messages in thread
From: John Snow @ 2020-10-02 15:14 UTC (permalink / raw)
To: Markus Armbruster
Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel,
Cleber Rosa, Alex Bennée
On 10/2/20 5:19 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
>
>> On 10/1/20 4:52 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> On 9/30/20 4:47 AM, Markus Armbruster wrote:
>>>>> John Snow <jsnow@redhat.com> writes:
>>>>>
>>>>>> I did not say "sphinx beautiful", just "sphinx compatible". They will
>>>>>> not throw errors when parsed and interpreted as ReST.
>>>>> "Bang on the keyboard until Sphinx doesn't throw errors anymore"
>>>>> might
>>>>> be good enough for a certain kind of mathematician, but a constructive
>>>>> solution needs a bit more direction. Is there a specification to
>>>>> follow? Other useful resources?
>>>>>
>>>>
>>>> I don't know if you are asking this question rhetorically, or in good faith.
>>> I ask to make sure I understand goals and limitations of your doc
>>> string
>>> work in this series.
>>> Also, even a passing to Sphinx becomes more useful when accompanied
>>> by a
>>> link to relevant documentation.
>>>
>>>> Let me preface this by saying: This series, and these 119 patches, are
>>>> not about finding a style guide for our docstring utilization or about
>>>> proposing one. It is also not about rigorously adding such
>>>> documentation or about finding ways to meaningfully publish it with
>>>> e.g. Sphinx, or the styling of such pages.
>>>>
>>>> Why bother to add docstrings at all, then? Because I needed them for
>>>> my own sake when learning this code and I felt it would be a waste to
>>>> just delete them, and I am of sound mind and able body and believe
>>>> that some documentation was better than none. They are useful even
>>>> just as plaintext.
>>>>
>>>> Having said that, let's explore the actual style I tend to use.
>>>>
>>>> I mentioned before in response to a review comment that there isn't
>>>> really any standard for docstrings. There are a few competing
>>>> "styles", but none of which are able to be programmatically checked
>>>> and validated.
>>>>
>>>> The primary guide for docstrings is PEP 257, of which I follow some
>>>> but not all of the recommendations.
>>>>
>>>> https://www.python.org/dev/peps/pep-0257/
>>>
>>> I find PEP 257 frustrating. It leaves me with more questions than
>>> answers.
>>
>> Yeah, sorry. That's what we're dealing with. It's very open-ended.
>>
>>>> In general,
>>>>
>>>> - Always use triple-double quotes (unenforced)
>>>> - Modules, classes, and functions should have docstrings (pylint)
>>>> - No empty lines before or after the docstring (unenforced)
>>>> - Multi-line docstrings should take the form (unenforced):
>>>>
>>>> """
>>>> single-line summary of function.
>>>>
>>>> Additional prose if needed describing the function.
>>>>
>>>> :param x: Input such that blah blah.
>>>> :raises y: When input ``x`` is unsuitable because blah blah.
>>>> :returns: A value that blah blah.
>>> This paragraph is already not PEP 257.
>>>
>>
>> Right -- well, it isn't NOT PEP 257 either. It just suggests you have
>> to describe these features, but it doesn't say HOW.
>
> Yep. Frustrating.
>
>>>> """
>>>>
>>>> PEP257 suggests a form where the single-line summary appears on the
>>>> same line as the opening triple quotes. I don't like this, and prefer
>>>> symmetry. PEP257 *also* suggests that writing it my way is equivalent
>>>> to their way, because any docstring processor should trim the first
>>>> line. I take this as tacit admission that my way is acceptable and has
>>>> merit.
>>> I prefer the symmetric form myself.
>>>
>>>> What about the syntax or markup inside the docstring itself? there is
>>>> *absolutely no standard*, but Sphinx autodoc recognizes a few field
>>>> lists as significant in its parsing, so I prefer using them:
>>>
>>> Doc link?
>>
>> Hard to search for in my opinion;
>
> More reason to provide a link!
>
Yup. I took a stab at it in a new commit message.
>> you want to search for "sphinx
>> python domain", and then click on "field lists" on the sidebar.
>>
>> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
>>
>> It has a special understanding for:
>>
>> param/parameter/arg/argument/key/keyword: I prefer "param"
>> here. Possibly key/keyword if we use a **kwargs form with a key that
>> we specially recognize, but I've not tested that yet. I know pycharm
>> understands "param" in a semantic way, and that's been good enough for me.
>>
>> type: Defines the type of a parameter. In my opinion, do not use
>> this. Let native type hints do the lifting.
>
> Agree.
>
>> raises/raise/except/exception: I prefer "raises". "raises ErrorType
>> when...." is a good sentence.
>>
>> var/ivar/cvar: Describes a variable, presumably in the body of the
>> function below. I've never used this, I always describe it in prose
>> instead.
>>
>> vartype: Defines a type for a variable; I would again defer to the
>> native type system instead now.
>>
>> returns/return: I prefer "returns" for grammatical reasons
>> again. ("Returns such-and-such when...")
>
> "Return such-and-such when..." is just as correct: imperative mood. I
> prefer imperative mood for function contracts.
>
I think there's some other style guide I saw that prefers this too. I
think I casually prefer "raises" and "returns", but "raise X when" and
"return Y such that" both make sense, too.
Not something I'll go to fight and die for, but it's probably the case
that I have already been using the other form.
I can fix these alongside a style guide if you'd like, but I'd ask for
your lenience to allow me do that as a follow-up in a series that I
won't mind spending more time in review to perfect our style.
>> rtype: again, type information. Don't use.
>>
>> meta: For directives to sphinx, e.g. :meta private: or :meta public:
>> to toggle the visibility class from its default. I don't use this.
>>
>>
>> None of these are validated or checked in any meaningful way; you can
>> use arbitrarily field lists (and I do in a few places!) to define your
>> own terms and so on.
>>
>>
>> (I would like to improve autodoc in the future to validate your
>> docstrings such that you can enforce :param:, :raises: and :return:
>> and it uses the type hints and introspection information to raise an
>> error when you make an obvious mistake. I am not there yet, but I am
>> using Peter Maydell's work to help inform how I might write such an
>> extension to autodoc. This work is not critical, but it will likely
>> occur upstream, outside of the QEMU context because I believe this is
>> a good thing to do for the ecosystem in general, to allow autodoc to
>> function slightly more like e.g. Doxygen does.)
>
> Sounds useful, but yes, it's clearly outside QEMU context.
>
>>>> :param x: Denotes the parameter X. Do not use type information in the
>>>> string, we rely on mypy for that now.
>>>>
>>>> :raises y: explains a case in which an Exception type y may be raised
>>>> either directly by this code or anticipated to be allowed to be raised
>>>> by a helper call. (There's no standard here that I am aware of. I use
>>>> my judgment. Always document direct raise calls, but use your judgment
>>>> for sub-calls.)
>>>>
>>>> :returns: explains the semantics of the return value.
>>>>
>>>> That said, literally any sphinx/ReST markup is OK as long as it passes
>>>> make sphinxdocs. Some sphinx markup is prohibited, like adding new
>>>> full-throated sections. You can use arbitrary field lists, definition
>>>> lists, pre-formatted text, examples, code blocks, whatever.
>>>>
>>>> In general, you are not going to find the kind of semantic validation
>>>> you want to ensure that the parameter names are correct, or that you
>>>> spelled :param: right, or that you didn't miss a parameter or an
>>>> exception. None of that tooling exists for Python.
>>>>
>>>> Thus, it's all rather subjective. No right answers, no validation
>>>> tools. Just whatever seems reasonable to a human eye until such time
>>>> we actually decide to pursue publishing the API docs in the
>>>> development manual, if indeed we ever do so at all.
>>>>
>>>> That series sounds like a great opportunity to hash this all out. That
>>>> is when I would like to remove --missing-docstring, too. There will
>>>> absolutely be a "docstring series" in the future, but I am insisting
>>>> stubbornly it happen after strict typing.
>>>
>>> Okay. Nevertheless, I'd prefer a bit more information in the commit
>>> message. Here's my try:
>>>
>>> qapi: Modify docstrings to be sphinx-compatible
>>>
>>> I did not say "sphinx beautiful", just "sphinx compatible". They
>>> will not throw errors when parsed and interpreted as ReST. Finding
>>> a comprehensive style guide for our docstring utilization is left
>>> for another day.
>>>
>>> For now, use field lists recognized by Sphinx autodoc.
>>> FIXME link to their documentation
>>
>> That I can do -- and I will double down on my IOU for a more formal
>> style guide: https://gitlab.com/jsnow/qemu/-/issues/7
>>
>> I didn't bother writing it in any of the commits because I felt like
>> it'd get lost there and would be mostly useless; but a .rst doc inside
>> the package folder would be hard to miss.
>>
>> I plan to check in something like ./python/README.rst or
>> ./python/CODING_STYLE.rst to try and formalize a lot of what I am
>> doing here, where it's going to be harder to miss.
>
> Makes sense.
>
>>>>>>
>>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>>>> ---
>>>>>> scripts/qapi/gen.py | 6 ++++--
>>>>>> scripts/qapi/parser.py | 9 +++++----
>>>>>> 2 files changed, 9 insertions(+), 6 deletions(-)
>>>>>>
>>>>>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>>>>>> index ca66c82b5b8..fc19b2aeb9b 100644
>>>>>> --- a/scripts/qapi/gen.py
>>>>>> +++ b/scripts/qapi/gen.py
>>>>>> @@ -154,9 +154,11 @@ def _bottom(self):
>>>>>> @contextmanager
>>>>>> def ifcontext(ifcond, *args):
>>>>>> - """A 'with' statement context manager to wrap with start_if()/end_if()
>>>>>> + """
>>>>>> + A 'with' statement context manager that wraps with `start_if` and `end_if`.
>>>>> Sadly, the fact that start_if() and end_if() are functions isn't
>>>>> immediately obvious anymore.
>>>>> I've seen :func:`start_if` elsewhere. Is this something we should
>>>>> or
>>>>> want to use?
>>>>>
>>>>
>>>> We *could*.
>>>>
>>>> `start_if` relies on the default role, which I have provisionally set
>>>> to "any" here, so this is shorthand for :any:`start_if`.
>>>>
>>>> The :any: role means: "cross-reference any type of thing." If there is
>>>> not exactly one thing that matches, it results in an error during the
>>>> documentation build.
>>>>
>>>> I like this, because it's nice short-hand syntax that I think
>>>> communicates effectively to the reader that this is a symbol of some
>>>> kind without needing a premium of ReST-ese.
>>>>
>>>> CONSTANTS are capitalized, Classes are title cased, and functions are
>>>> lower_case. `lower_case` references can be assumed to be functions,
>>>
>>> `lower_case` could also refer to an attribute, variable, or
>>> parameter.
>>
>> Hm. Attribute yes, actually. variable and parameter no -- sphinx does
>> not presently provide syntax or roles for creating anchors to
>> parameter names or variables, so they are not able to be
>> cross-referenced.
>
> How would you mark up variable names in doc strings? Often, their
> "variableness" is obvious from context, but not always. In C comments,
> we tend to use @var [*].
>
I have been using ``var`` so far, to render it in a preformatted
monowidth style. It doesn't cross-reference anything and it is no
different from other pre-formatted literal blocks. I use the same markup
for Python types; ``str`` and ``List[str]`` and so on -- other literal
values like ``True``, ``False``, and ``"string_literal"``.
We could look into creating markup for it, like :v:`var` or something,
to which we apply rendering logic to distinguish them, but that's beyond
my means to do quickly at the moment. I can add it to my list to
investigate ("someday").
Also possible that we might look at autodoc extensions to generate
cross-references for variable names (by modifying handling for
:param:?), but I also don't know if rst/sphinx references can be
"scoped"? Variables and parameters will collide far more often than
function/class/module names.
If you could do, say, `funcname.varname` in cases where the variable was
ambiguous and it jumped to the sphinx docs where that variable/parameter
was described, that'd be hot.
But it's not something that's tractable to me right now, sorry.
>> Attributes CAN be cross-referenced, but only when they are documented.
>>
>> Another style guide thing:
>>
>> #: x is a number that represents "The Answer". See `Douglas Adams`_.
>> self.x = 42
>>
>> You can use the special comment form "#:" to add a one-line
>> description of an attribute that Sphinx will pick up. Sphinx skips
>> these attributes otherwise. If you consider them part of the interface
>> of the module, it's maybe a good idea to do this.
>>
>> You can also use docstrings, but the ordering changes:
>>
>> self.x = 42
>> """x is a number that represents "The Answer". See `Douglas Adams`_.
>>
>> I kind of like the #: form because it announces what follows, but I
>> admit it's a bit of special sphinx magic.
>
> Are both equally available in Python IDEs and in interactive Python?
>
Docstrings are only recognized as special for modules, classes and
functions. I do not think Python recognizes docstrings for attributes at
all, actually.
>>> help(qapi.common)
is going to show you:
- the description (docstring, comments) for the module
- the classes defined in it (Indentation),
- functions and their docstrings (c_enum_const, c_fname, c_name, ...)
- "Data": (EATSPACE, POINTER_SUFFIX, indent)
I commented EATSPACE in my patches because I reference it from somewhere
else. Either style does not show up in help(qapi.common).
qapi.common.EATSPACE does have a __doc__ attribute, but it's actually
the doc attribute for `str()`! Not what we wanted at all.
So either style is invisible at runtime. Sad.
We can also prefer to use @property style functions which can receive
docstrings -- but this only works for classes! I don't think there's a
standard way to define "properties" for modules exactly as such.
That means that module-level data (constants, globals, etc.) don't
really have a great way to be documented. The best place is likely
actually in the module docstring itself, but you cannot cross-reference
such a location with Sphinx, I think.
Needs more research, but again I plead that this is good fodder for a
later series when we attempt to calcify a standard.
>>>> but I will admit that this is not enforced or necessarily true as we
>>>> add more cross reference types in the future.
>>>>
>>>> (I am trying to add QMP cross-reference syntax!)
>>>>
>>>> I still prefer `start_if` to :func:`start_if` simply because it's less
>>>> markup and is easier to read in plaintext contexts. You're right, it
>>>> doesn't look like a function anymore.
>>> Yes, :func:`start_if` is rather heavy. I asked because I wanted to
>>> understand what :func: buys us. Not meant as endorsement.
>>>
>>
>> It specifically targets only cross-references of that exact type. In
>> the case that the :any: reference is ambiguous, :func: is the
>> disambiguation.
>>
>>> GDK-Doc seems smart enough to recognize start_if(). Sphinx isn't,
>>> because it's built around reST syntax. We put our money on the Sphinx
>>> horse, so...
>>>
>>>> I'm not sure if another annotations would work -- `start_if`() or
>>>> `start_if()`. Both seem kind of clunky to me, to be honest. Personal
>>>> feeling is "not really worth the hassle."
>>>
>>> You later reported the latter works.
>>> I prefer `start_if()` to `start_if`. Matter of taste.
>>
>> Change made.
>
> Thanks!
>
>>>>>> - *args: any number of QAPIGenCCode
>>>>>> + :param ifcond: List of conditionals
>>>>>> + :param args: any number of `QAPIGenCCode`.
>>>>>> Example::
>>>>>> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>>>>>> index 9d1a3e2eea9..02983979965 100644
>>>>>> --- a/scripts/qapi/parser.py
>>>>>> +++ b/scripts/qapi/parser.py
>>>>>> @@ -381,10 +381,11 @@ def append(self, line):
>>>>>> The way that the line is dealt with depends on which
>>>>>> part of
>>>>>> the documentation we're parsing right now:
>>>>>> - * The body section: ._append_line is ._append_body_line
>>>>>> - * An argument section: ._append_line is ._append_args_line
>>>>>> - * A features section: ._append_line is ._append_features_line
>>>>>> - * An additional section: ._append_line is ._append_various_line
>>>>>> +
>>>>>> + * The body section: ._append_line is ._append_body_line
>>>>>> + * An argument section: ._append_line is ._append_args_line
>>>>>> + * A features section: ._append_line is ._append_features_line
>>>>>> + * An additional section: ._append_line is ._append_various_line
>>>>>> """
>>>>>> line = line[1:]
>>>>>> if not line:
>>>>> I understand why you insert a blank line (reST wants blank lines
>>>>> around
>>>>> lists), I don't understand why you indent. Can you explain?
>>>>
>>>> I was mistaken about it needing the indent!
>>> Easy enough to tidy up :)
>>>
>>
>> Already done!
>
> Thanks again!
>
>
> [*] GTK-Doc says @var is just for parameters, but since it offers
> nothing for variables, we sometimes use it for variables as well.
>
^ permalink raw reply [flat|nested] 69+ messages in thread
end of thread, other threads:[~2020-10-02 15:25 UTC | newest]
Thread overview: 69+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-30 4:31 [PATCH v4 00/46] qapi: static typing conversion, pt1 John Snow
2020-09-30 4:31 ` [PATCH v4 01/46] [DO-NOT-MERGE] docs: replace single backtick (`) with double-backtick (``) John Snow
2020-09-30 4:31 ` [PATCH v4 02/46] docs: repair broken references John Snow
2020-09-30 4:31 ` [PATCH v4 03/46] [DO-NOT-MERGE] docs/sphinx: change default role to "any" John Snow
2020-09-30 4:31 ` [PATCH v4 04/46] qapi: modify docstrings to be sphinx-compatible John Snow
2020-09-30 8:47 ` Markus Armbruster
2020-09-30 17:22 ` John Snow
2020-10-01 8:52 ` Markus Armbruster
2020-10-01 14:48 ` John Snow
2020-10-02 9:19 ` Markus Armbruster
2020-10-02 15:14 ` John Snow
2020-09-30 17:38 ` John Snow
2020-10-01 8:54 ` Markus Armbruster
2020-10-01 14:28 ` John Snow
2020-09-30 4:31 ` [PATCH v4 05/46] [DO-NOT-MERGE] docs: enable sphinx-autodoc for scripts/qapi John Snow
2020-09-30 4:31 ` [PATCH v4 06/46] qapi-gen: Separate arg-parsing from generation John Snow
2020-09-30 4:31 ` [PATCH v4 07/46] qapi: move generator entrypoint into module John Snow
2020-09-30 4:31 ` [PATCH v4 08/46] [DO-NOT-MERGE] docs: add scripts/qapi/main to python manual John Snow
2020-09-30 4:31 ` [PATCH v4 09/46] qapi: Prefer explicit relative imports John Snow
2020-09-30 4:31 ` [PATCH v4 10/46] qapi: Remove wildcard includes John Snow
2020-09-30 4:31 ` [PATCH v4 11/46] qapi: enforce import order/styling with isort John Snow
2020-09-30 4:31 ` [PATCH v4 12/46] qapi: delint using flake8 John Snow
2020-09-30 4:31 ` [PATCH v4 13/46] qapi: add pylintrc John Snow
2020-09-30 4:31 ` [PATCH v4 14/46] qapi/common.py: Remove python compatibility workaround John Snow
2020-09-30 4:31 ` [PATCH v4 15/46] qapi/common.py: Add indent manager John Snow
2020-09-30 4:31 ` [PATCH v4 16/46] qapi/common.py: delint with pylint John Snow
2020-09-30 4:31 ` [PATCH v4 17/46] qapi/common.py: Replace one-letter 'c' variable John Snow
2020-09-30 4:31 ` [PATCH v4 18/46] qapi/common.py: check with pylint John Snow
2020-09-30 4:31 ` [PATCH v4 19/46] qapi/common.py: add type hint annotations John Snow
2020-09-30 4:31 ` [PATCH v4 20/46] qapi/common.py: Convert comments into docstrings, and elaborate John Snow
2020-09-30 4:31 ` [PATCH v4 21/46] qapi/common.py: move build_params into gen.py John Snow
2020-09-30 4:31 ` [PATCH v4 22/46] qapi: establish mypy type-checking baseline John Snow
2020-09-30 4:31 ` [PATCH v4 23/46] qapi/events.py: add type hint annotations John Snow
2020-09-30 4:31 ` [PATCH v4 24/46] qapi/events.py: Move comments into docstrings John Snow
2020-09-30 4:31 ` [PATCH v4 25/46] qapi/commands.py: Don't re-bind to variable of different type John Snow
2020-09-30 4:31 ` [PATCH v4 26/46] qapi/commands.py: add type hint annotations John Snow
2020-09-30 4:31 ` [PATCH v4 27/46] qapi/commands.py: enable checking with mypy John Snow
2020-09-30 4:31 ` [PATCH v4 28/46] qapi/source.py: add type hint annotations John Snow
2020-09-30 4:31 ` [PATCH v4 29/46] qapi/source.py: delint with pylint John Snow
2020-09-30 4:31 ` [PATCH v4 30/46] qapi/gen.py: Fix edge-case of _is_user_module John Snow
2020-09-30 11:03 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 31/46] qapi/gen.py: add type hint annotations John Snow
2020-09-30 4:31 ` [PATCH v4 32/46] qapi/gen.py: Enable checking with mypy John Snow
2020-09-30 4:31 ` [PATCH v4 33/46] qapi/gen.py: Remove unused parameter John Snow
2020-09-30 4:31 ` [PATCH v4 34/46] qapi/gen.py: update write() to be more idiomatic John Snow
2020-09-30 4:31 ` [PATCH v4 35/46] qapi/gen.py: delint with pylint John Snow
2020-09-30 4:31 ` [PATCH v4 36/46] qapi/introspect.py: assert obj is a dict when features are given John Snow
2020-09-30 11:04 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 37/46] qapi/instrospect.py: add preliminary type hint annotations John Snow
2020-09-30 18:31 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 38/46] qapi/introspect.py: add _gen_features helper John Snow
2020-09-30 17:21 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 39/46] qapi/introspect.py: Unify return type of _make_tree() John Snow
2020-09-30 18:24 ` Eduardo Habkost
2020-09-30 18:32 ` John Snow
2020-09-30 18:57 ` Eduardo Habkost
2020-09-30 19:02 ` John Snow
2020-09-30 4:31 ` [PATCH v4 40/46] qapi/introspect.py: replace 'extra' dict with 'comment' argument John Snow
2020-09-30 18:24 ` Eduardo Habkost
2020-09-30 4:31 ` [PATCH v4 41/46] qapi/introspect.py: create a typed 'Node' data structure John Snow
2020-09-30 18:39 ` Eduardo Habkost
2020-09-30 18:58 ` John Snow
2020-09-30 19:52 ` Eduardo Habkost
2020-10-01 17:59 ` John Snow
2020-09-30 4:31 ` [PATCH v4 42/46] qapi/types.py: add type hint annotations John Snow
2020-09-30 4:31 ` [PATCH v4 43/46] qapi/types.py: remove one-letter variables John Snow
2020-09-30 4:31 ` [PATCH v4 44/46] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType John Snow
2020-09-30 4:31 ` [PATCH v4 45/46] qapi/visit.py: remove unused parameters from gen_visit_object John Snow
2020-09-30 4:31 ` [PATCH v4 46/46] qapi/visit.py: add type hint annotations John Snow
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.