[PATCH v5 00/36] qapi: static typing conversion, pt1

[PATCH v5 00/36] qapi: static typing conversion, pt1

[John Snow]

Hi, this series adds static type hints to the QAPI module.
This is part one!

Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6

- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)

In general, this series tackles the cleanup of one individual QAPI
module at a time. Once it passes pylint or mypy checks, those checks are
enabled for that file.

Type hints are added in patches that add *only* type hints and change no
other behavior. Any necessary changes to behavior to accommodate typing
are split out into their own tiny patches.

Notes:

- After patch 07, `isort -c` should pass 100% on this and every
  future commit.

- After patch 08, `flake8 qapi/` should pass 100% on this and every
  future commit.

- After patch 09, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
  on this and every future commit.

- After patch 18, `mypy --config-file=qapi/mypy.ini qapi/` should pass
  100% on this and every future commit.

Review Status:

[01] docs-repair-broken-references  #
[02] qapi-modify-docstrings-to-be   #
[03] qapi-gen-separate-arg-parsing  # [RB] CR,EH [TB] CR [SOB] JS
[04] qapi-move-generator-entrypoint # [RB] CR,EH [TB] CR [SOB] JS
[05] qapi-prefer-explicit-relative  # [RB] CR,EH [SOB] JS
[06] qapi-remove-wildcard-includes  # [RB] CR,EH [SOB] JS
[07] qapi-enforce-import-order      # [RB] CR [TB] CR [SOB] JS
[08] qapi-delint-using-flake8       # [RB] CR,EH [SOB] JS
[09] qapi-add-pylintrc              # [RB] CR [TB] CR,EH [SOB] JS
[10] qapi-common-py-remove-python   # [RB] CR,EH [SOB] JS
[11] qapi-common-add-indent-manager # [RB] CR,EH [SOB] JS
[12] qapi-common-py-delint-with     # [RB] CR,EH [SOB] JS
[13] replace-c-by-char              # [RB] CR,EH [SOB] JS
[14] qapi-common-py-check-with      # [RB] CR [TB] CR,EH [SOB] JS
[15] qapi-common-py-add-notational  # [RB] CR,EH [SOB] JS
[16] qapi-common-move-comments-into # [RB] CR,EH [SOB] JS
[17] qapi-split-build_params-into   # [RB] CR,EH [SOB] JS
[18] qapi-establish-mypy-type       # [RB] CR [TB] CR,EH [SOB] JS
[19] qapi-events-py-add-notational  # [RB] CR,EH [SOB] JS
[20] qapi-events-move-comments-into # [RB] CR,EH [SOB] JS
[21] qapi-commands-py-don-t-re-bind # [RB] CR,EH [SOB] JS
[22] qapi-commands-py-add           # [RB] CR,EH [SOB] JS
[23] qapi-commands-py-enable        # [RB] CR,EH [SOB] JS
[24] qapi-source-py-add-notational  # [RB] CR,EH [TB] CR [SOB] JS
[25] qapi-source-py-delint-with     # [RB] CR,EH [TB] CR [SOB] JS
[26] qapi-gen-py-fix-edge-case-of   # [RB] CR,EH [SOB] JS
[27] qapi-gen-py-add-notational     # [RB] CR,EH [SOB] JS
[28] qapi-gen-py-enable-checking    # [RB] CR,EH [TB] CR [SOB] JS
[29] qapi-gen-py-remove-unused      # [RB] CR,EH [SOB] JS
[30] qapi-gen-py-update-write-to-be # [RB] CR,EH [SOB] JS
[31] qapi-gen-py-delint-with-pylint # [RB] CR,EH [SOB] JS
[32] qapi-types-py-add-type-hint    # [RB] CR,EH [SOB] JS
[33] qapi-types-py-remove-one       # [RB] CR,EH [SOB] JS
[34] qapi-visit-py-assert           # [RB] CR,EH [SOB] JS
[35] qapi-visit-py-remove-unused    # [RB] CR,EH [TB] CR [SOB] JS
[36] qapi-visit-py-add-notational   # [RB] CR,EH [TB] CR [SOB] JS

Changelog:

002/36:[0012] [FC] 'qapi: modify docstrings to be sphinx-compatible'
027/36:[----] [-C] 'qapi/gen.py: add type hint annotations'

V5:
 - Remove DO-NOT-MERGE patches (Now in Part2)
 - Remove introspect.py patches (Now in Part2)
 - 02: Docstring formatting, more commit message (Markus)

V4:
 - Rebase on Peter Maydell's work;
  - 05: Context differences from Peter Maydell's work
  - 06: Remove qapi.doc
  - 07: Remove qapi.doc, remove superfluous "if match"
  - 09: Remove qapi.doc
  - 13: remove qapi.doc
  - 18: remove qapi.doc
  - 22: remove qapi.doc
  - 31: remove QAPIGenDoc changes

 - Minor adjustments from list feedback;
  - 01: More backticks for QAPI json files, now that they are in Sphinx-exe
  - 02: Use :ref: role for the `docker-ref` cross-reference
  - 04: Remove doc.py work changes; add references for start_if/end_if
  - 10: Changes to accommodate isort
  - 11: Added lines_after_imports=2
  - 16: isort whitespace changes
  - 24: Take Markus's docstring phrasing
  - 34: add comment explaining os.open()
  - 37: isort differences

V3:
 - Use isort to enforce import consistency
 - Use sphinx apidoc to check docstring format

V2:
 - Removed Python3.6 patch in favor of Thomas Huth's
 - Addressed (most) feedback from Markus
 - Renamed type hint annotation commits

John Snow (36):
  docs: repair broken references
  qapi: modify docstrings to be sphinx-compatible
  qapi-gen: Separate arg-parsing from generation
  qapi: move generator entrypoint into module
  qapi: Prefer explicit relative imports
  qapi: Remove wildcard includes
  qapi: enforce import order/styling with isort
  qapi: delint using flake8
  qapi: add pylintrc
  qapi/common.py: Remove python compatibility workaround
  qapi/common.py: Add indent manager
  qapi/common.py: delint with pylint
  qapi/common.py: Replace one-letter 'c' variable
  qapi/common.py: check with pylint
  qapi/common.py: add type hint annotations
  qapi/common.py: Convert comments into docstrings, and elaborate
  qapi/common.py: move build_params into gen.py
  qapi: establish mypy type-checking baseline
  qapi/events.py: add type hint annotations
  qapi/events.py: Move comments into docstrings
  qapi/commands.py: Don't re-bind to variable of different type
  qapi/commands.py: add type hint annotations
  qapi/commands.py: enable checking with mypy
  qapi/source.py: add type hint annotations
  qapi/source.py: delint with pylint
  qapi/gen.py: Fix edge-case of _is_user_module
  qapi/gen.py: add type hint annotations
  qapi/gen.py: Enable checking with mypy
  qapi/gen.py: Remove unused parameter
  qapi/gen.py: update write() to be more idiomatic
  qapi/gen.py: delint with pylint
  qapi/types.py: add type hint annotations
  qapi/types.py: remove one-letter variables
  qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
  qapi/visit.py: remove unused parameters from gen_visit_object
  qapi/visit.py: add type hint annotations

 docs/devel/multi-thread-tcg.rst |   2 +-
 docs/devel/testing.rst          |   2 +-
 scripts/qapi-gen.py             |  57 ++--------
 scripts/qapi/.flake8            |   2 +
 scripts/qapi/.isort.cfg         |   7 ++
 scripts/qapi/commands.py        |  87 +++++++++++----
 scripts/qapi/common.py          | 163 +++++++++++++++-------------
 scripts/qapi/events.py          |  58 +++++++---
 scripts/qapi/expr.py            |   7 +-
 scripts/qapi/gen.py             | 182 ++++++++++++++++++++------------
 scripts/qapi/introspect.py      |  16 ++-
 scripts/qapi/main.py            |  89 ++++++++++++++++
 scripts/qapi/mypy.ini           |  30 ++++++
 scripts/qapi/parser.py          |   7 +-
 scripts/qapi/pylintrc           |  70 ++++++++++++
 scripts/qapi/schema.py          |  33 +++---
 scripts/qapi/source.py          |  34 +++---
 scripts/qapi/types.py           | 125 +++++++++++++++-------
 scripts/qapi/visit.py           | 116 ++++++++++++++------
 19 files changed, 752 insertions(+), 335 deletions(-)
 create mode 100644 scripts/qapi/.flake8
 create mode 100644 scripts/qapi/.isort.cfg
 create mode 100644 scripts/qapi/main.py
 create mode 100644 scripts/qapi/mypy.ini
 create mode 100644 scripts/qapi/pylintrc

-- 
2.26.2

[PATCH v5 01/36] docs: repair broken references

[John Snow]

In two different places, we are not making a cross-reference to some
resource correctly.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/devel/multi-thread-tcg.rst | 2 +-
 docs/devel/testing.rst          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst
index 21483870dbc..92a9eba13c9 100644
--- a/docs/devel/multi-thread-tcg.rst
+++ b/docs/devel/multi-thread-tcg.rst
@@ -267,7 +267,7 @@ of view of external observers (e.g. another processor core). They can
 apply to any memory operations as well as just loads or stores.
 
 The Linux kernel has an excellent `write-up
-<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`
+<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`_
 on the various forms of memory barrier and the guarantees they can
 provide.
 
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index bd64c1bdcdd..8875a40a2b6 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -953,7 +953,7 @@ compiler flags are needed to build for a given target.
 If you have the ability to run containers as the user you can also
 take advantage of the build systems "Docker" support. It will then use
 containers to build any test case for an enabled guest where there is
-no system compiler available. See :ref: `_docker-ref` for details.
+no system compiler available. See :ref:`docker-ref` for details.
 
 Running subset of tests
 -----------------------
-- 
2.26.2

[PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

[John Snow]

A precise style guide and a package-wide overhaul is forthcoming pending
further discussion and consensus. At present, we are avoiding obvious
errors that cause sphinx documentation build problems.

A preliminary style guide is loosely based on PEP 257 and Sphinx
Autodoc. It is chosen for interoperability with our existing Sphinx
framework, and because it has loose recognition in the Pycharm IDE.

- Use Triple-double quotes (""").
- Opening and closing quotes appear on their own lines for multi-line docs.
- A single-sentence summary should be the first line of the docstring.
- A blank line follows.
- Further prose, if needed, is written next and can be multiple paragraphs,
  contain RST markup, etc.
- The :param x: desc, :returns: desc, and :raises z: desc info fields follow.
- Additional examples, diagrams, or other metadata follows below.

See also:

https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/gen.py    | 6 ++++--
 scripts/qapi/parser.py | 1 +
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index ca66c82b5b8..dc7b94aa115 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -154,9 +154,11 @@ def _bottom(self):
 
 @contextmanager
 def ifcontext(ifcond, *args):
-    """A 'with' statement context manager to wrap with start_if()/end_if()
+    """
+    A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
-    *args: any number of QAPIGenCCode
+    :param ifcond: A list of conditionals, passed to `start_if()`.
+    :param args: any number of `QAPIGenCCode`.
 
     Example::
 
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 9d1a3e2eea9..31bc2e6dca9 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -381,6 +381,7 @@ def append(self, line):
 
         The way that the line is dealt with depends on which part of
         the documentation we're parsing right now:
+
         * The body section: ._append_line is ._append_body_line
         * An argument section: ._append_line is ._append_args_line
         * A features section: ._append_line is ._append_features_line
-- 
2.26.2

[PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

This is a minor re-work of the entrypoint script. It isolates a
generate() method from the actual command-line mechanism.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
 1 file changed, 62 insertions(+), 23 deletions(-)

diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 541e8c1f55d..117b396a595 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -1,30 +1,77 @@
 #!/usr/bin/env python3
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
 import argparse
 import re
 import sys
 
 from qapi.commands import gen_commands
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
     parser = argparse.ArgumentParser(
         description='Generate code from a QAPI schema')
     parser.add_argument('-b', '--builtins', action='store_true',
                         help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                         help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                         help="prefix for symbols")
     parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                         dest='unmask',
@@ -32,25 +79,17 @@ def main(argv):
     parser.add_argument('schema', action='store')
     args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
     try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
     except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
 
 
 if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())
-- 
2.26.2

[PATCH v5 04/36] qapi: move generator entrypoint into module

[John Snow]

As part of delinting and adding type hints to the QAPI generator, it's
helpful for the entrypoint to be part of the package, only leaving a
very tiny entrypoint shim outside of the module.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi-gen.py  | 88 +++----------------------------------------
 scripts/qapi/main.py | 89 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+), 82 deletions(-)
 create mode 100644 scripts/qapi/main.py

diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 117b396a595..f3518d29a54 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -4,92 +4,16 @@
 # See the COPYING file in the top-level directory.
 
 """
-QAPI Generator
+QAPI code generation execution shim.
 
-This script is the main entry point for generating C code from the QAPI schema.
+This standalone script exists primarily to facilitate the running of the QAPI
+code generator without needing to install the python module to the current
+execution environment.
 """
 
-import argparse
-import re
 import sys
 
-from qapi.commands import gen_commands
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
-
-
-DEFAULT_OUTPUT_DIR = ''
-DEFAULT_PREFIX = ''
-
-
-def generate(schema_file: str,
-             output_dir: str,
-             prefix: str,
-             unmask: bool = False,
-             builtins: bool = False) -> None:
-    """
-    generate uses a given schema to produce C code in the target directory.
-
-    :param schema_file: The primary QAPI schema file.
-    :param output_dir: The output directory to store generated code.
-    :param prefix: Optional C-code prefix for symbol names.
-    :param unmask: Expose non-ABI names through introspection?
-    :param builtins: Generate code for built-in types?
-
-    :raise QAPIError: On failures.
-    """
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
-    if match.end() != len(prefix):
-        msg = "funny character '{:s}' in prefix '{:s}'".format(
-            prefix[match.end()], prefix)
-        raise QAPIError('', None, msg)
-
-    schema = QAPISchema(schema_file)
-    gen_types(schema, output_dir, prefix, builtins)
-    gen_visit(schema, output_dir, prefix, builtins)
-    gen_commands(schema, output_dir, prefix)
-    gen_events(schema, output_dir, prefix)
-    gen_introspect(schema, output_dir, prefix, unmask)
-
-
-def main() -> int:
-    """
-    gapi-gen shell script entrypoint.
-    Expects arguments via sys.argv, see --help for details.
-
-    :return: int, 0 on success, 1 on failure.
-    """
-    parser = argparse.ArgumentParser(
-        description='Generate code from a QAPI schema')
-    parser.add_argument('-b', '--builtins', action='store_true',
-                        help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store',
-                        default=DEFAULT_OUTPUT_DIR,
-                        help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store',
-                        default=DEFAULT_PREFIX,
-                        help="prefix for symbols")
-    parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
-                        dest='unmask',
-                        help="expose non-ABI names in introspection")
-    parser.add_argument('schema', action='store')
-    args = parser.parse_args()
-
-    try:
-        generate(args.schema,
-                 output_dir=args.output_dir,
-                 prefix=args.prefix,
-                 unmask=args.unmask,
-                 builtins=args.builtins)
-    except QAPIError as err:
-        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
-        return 1
-    return 0
-
+from qapi import main
 
 if __name__ == '__main__':
-    sys.exit(main())
+    sys.exit(main.main())
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
new file mode 100644
index 00000000000..9210a0e1a80
--- /dev/null
+++ b/scripts/qapi/main.py
@@ -0,0 +1,89 @@
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+"""
+QAPI Generator
+
+This is the main entry point for generating C code from the QAPI schema.
+"""
+
+import argparse
+import re
+import sys
+
+from qapi.commands import gen_commands
+from qapi.error import QAPIError
+from qapi.events import gen_events
+from qapi.introspect import gen_introspect
+from qapi.schema import QAPISchema
+from qapi.types import gen_types
+from qapi.visit import gen_visit
+
+
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
+    parser = argparse.ArgumentParser(
+        description='Generate code from a QAPI schema')
+    parser.add_argument('-b', '--builtins', action='store_true',
+                        help="generate code for built-in types")
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
+                        help="write output to directory OUTPUT_DIR")
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
+                        help="prefix for symbols")
+    parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
+                        dest='unmask',
+                        help="expose non-ABI names in introspection")
+    parser.add_argument('schema', action='store')
+    args = parser.parse_args()
+
+    try:
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
+    except QAPIError as err:
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
-- 
2.26.2

[PATCH v5 05/36] qapi: Prefer explicit relative imports

[John Snow]

All of the QAPI include statements are changed to be package-aware, as
explicit relative imports.

A quirk of Python packages is that the name of the package exists only
*outside* of the package. This means that to a module inside of the qapi
folder, there is inherently no such thing as the "qapi" package. The
reason these imports work is because the "qapi" package exists in the
context of the caller -- the execution shim, where sys.path includes a
directory that has a 'qapi' folder in it.

When we write "from qapi import sibling", we are NOT referencing the folder
'qapi', but rather "any package named qapi in sys.path". If you should
so happen to have a 'qapi' package in your path, it will use *that*
package.

When we write "from .sibling import foo", we always reference explicitly
our sibling module; guaranteeing consistency in *where* we are importing
these modules from.

This can be useful when working with virtual environments and packages
in development mode. In development mode, a package is installed as a
series of symlinks that forwards to your same source files. The problem
arises because code quality checkers will follow "import qapi.x" to the
"installed" version instead of the sibling file and -- even though they
are the same file -- they have different module paths, and this causes
cyclic import problems, false positive type mismatch errors, and more.

It can also be useful when dealing with hierarchical packages, e.g. if
we allow qemu.core.qmp, qemu.qapi.parser, etc.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/commands.py   |  4 ++--
 scripts/qapi/events.py     |  8 ++++----
 scripts/qapi/expr.py       |  4 ++--
 scripts/qapi/gen.py        |  4 ++--
 scripts/qapi/introspect.py |  8 ++++----
 scripts/qapi/main.py       | 14 +++++++-------
 scripts/qapi/parser.py     |  4 ++--
 scripts/qapi/schema.py     |  8 ++++----
 scripts/qapi/types.py      |  6 +++---
 scripts/qapi/visit.py      |  6 +++---
 10 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 3cf9e1110b2..ce5926146a4 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,8 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import *
+from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index b544af5a1ce..04672724388 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,10 +12,10 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember
-from qapi.types import gen_enum, gen_enum_lookup
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember
+from .types import gen_enum, gen_enum_lookup
 
 
 def build_event_send_proto(name, arg_type, boxed):
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 2942520399a..03b31ecfc19 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -16,8 +16,8 @@
 
 import re
 from collections import OrderedDict
-from qapi.common import c_name
-from qapi.error import QAPISemError
+from .common import c_name
+from .error import QAPISemError
 
 
 # Names must be letters, numbers, -, and _.  They must start with letter,
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index dc7b94aa115..fc57fdca5b9 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -17,8 +17,8 @@
 import re
 from contextlib import contextmanager
 
-from qapi.common import *
-from qapi.schema import QAPISchemaVisitor
+from .common import *
+from .schema import QAPISchemaVisitor
 
 
 class QAPIGen:
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 23652be8102..2a34cd1e8ea 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,10 +10,10 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaMonolithicCVisitor
-from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                         QAPISchemaType)
+from .common import *
+from .gen import QAPISchemaMonolithicCVisitor
+from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
+                     QAPISchemaType)
 
 
 def _make_tree(obj, ifcond, features, extra=None):
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
index 9210a0e1a80..1b168a9771a 100644
--- a/scripts/qapi/main.py
+++ b/scripts/qapi/main.py
@@ -11,13 +11,13 @@
 import re
 import sys
 
-from qapi.commands import gen_commands
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
+from .commands import gen_commands
+from .error import QAPIError
+from .events import gen_events
+from .introspect import gen_introspect
+from .schema import QAPISchema
+from .types import gen_types
+from .visit import gen_visit
 
 
 DEFAULT_OUTPUT_DIR = ''
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 31bc2e6dca9..68d8a1ce1cf 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -18,8 +18,8 @@
 import re
 from collections import OrderedDict
 
-from qapi.error import QAPIParseError, QAPISemError
-from qapi.source import QAPISourceInfo
+from .error import QAPIParseError, QAPISemError
+from .source import QAPISourceInfo
 
 
 class QAPISchemaParser:
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 78309a00f0a..a835ee6fde3 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,10 +18,10 @@
 import re
 from collections import OrderedDict
 
-from qapi.common import c_name, pointer_suffix
-from qapi.error import QAPIError, QAPISemError
-from qapi.expr import check_exprs
-from qapi.parser import QAPISchemaParser
+from .common import c_name, pointer_suffix
+from .error import QAPIError, QAPISemError
+from .expr import check_exprs
+from .parser import QAPISchemaParser
 
 
 class QAPISchemaEntity:
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 3640f17cd67..ca9a5aacb39 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,9 +13,9 @@
 # See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
 
 # variants must be emitted before their container; track what has already
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index cdabc5fa283..7850f6e8480 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,9 +13,9 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaObjectType
 
 
 def gen_visit_decl(name, scalar=False):
-- 
2.26.2

[PATCH v5 06/36] qapi: Remove wildcard includes

[John Snow]

Wildcard includes become hard to manage when refactoring and dealing
with circular dependencies with strictly typed mypy.

flake8 also flags each one as a warning, as it is not smart enough to
know which names exist in the imported file.

Remove them and include things explicitly by name instead.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/commands.py   |  2 +-
 scripts/qapi/events.py     |  7 ++++++-
 scripts/qapi/gen.py        | 12 +++++++++---
 scripts/qapi/introspect.py |  7 ++++++-
 scripts/qapi/types.py      |  8 +++++++-
 scripts/qapi/visit.py      | 10 +++++++++-
 6 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index ce5926146a4..64ed5278f93 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,7 +13,7 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import build_params, c_name, mcgen
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 04672724388..6b3afa14d72 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,7 +12,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    build_params,
+    c_enum_const,
+    c_name,
+    mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index fc57fdca5b9..1fed712b43b 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -11,13 +11,19 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-
+from contextlib import contextmanager
 import errno
 import os
 import re
-from contextlib import contextmanager
 
-from .common import *
+from .common import (
+    c_fname,
+    gen_endif,
+    gen_if,
+    guardend,
+    guardstart,
+    mcgen,
+)
 from .schema import QAPISchemaVisitor
 
 
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 2a34cd1e8ea..b036fcf9ce7 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+)
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
                      QAPISchemaType)
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index ca9a5aacb39..53b47f9e58a 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,7 +13,13 @@
 # See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_enum_const,
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 7850f6e8480..ea277e7704b 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,7 +13,15 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_enum_const,
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+    pop_indent,
+    push_indent,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
 
-- 
2.26.2

[PATCH v5 07/36] qapi: enforce import order/styling with isort

[John Snow]

While we're mucking around with imports, we might as well formalize the
style we use. Let's use isort to do it for us.

lines_after_imports=2: Use two lines after imports, to match PEP8's
desire to have "two lines before and after" class definitions, which are
likely to start immediately after imports.

force_sort_within_sections: Intermingles "from x" and "import x" style
statements, such that sorting is always performed strictly on the module
name itself.

force_grid_wrap=4: Four or more imports from a single module will force
the one-per-line style that's more git-friendly. This will generally
happen for 'typing' imports.

multi_line_output=3: Uses the one-per-line indented style for long
imports.

include_trailing_comma: Adds a comma to the last import in a group,
which makes git conflicts nicer to deal with, generally.

line_length: 72 is chosen to match PEP8's "docstrings and comments" line
length limit. If you have a single line import that exceeds 72
characters, your names are too long!

Suggested-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/.isort.cfg    | 7 +++++++
 scripts/qapi/expr.py       | 3 ++-
 scripts/qapi/introspect.py | 7 +++++--
 scripts/qapi/parser.py     | 2 +-
 scripts/qapi/schema.py     | 2 +-
 5 files changed, 16 insertions(+), 5 deletions(-)
 create mode 100644 scripts/qapi/.isort.cfg

diff --git a/scripts/qapi/.isort.cfg b/scripts/qapi/.isort.cfg
new file mode 100644
index 00000000000..6d0fd6cc0d3
--- /dev/null
+++ b/scripts/qapi/.isort.cfg
@@ -0,0 +1,7 @@
+[settings]
+force_grid_wrap=4
+force_sort_within_sections=True
+include_trailing_comma=True
+line_length=72
+lines_after_imports=2
+multi_line_output=3
\ No newline at end of file
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 03b31ecfc19..e73b65b6a7e 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -14,8 +14,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-import re
 from collections import OrderedDict
+import re
+
 from .common import c_name
 from .error import QAPISemError
 
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index b036fcf9ce7..31acd2f230a 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -17,8 +17,11 @@
     mcgen,
 )
 from .gen import QAPISchemaMonolithicCVisitor
-from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                     QAPISchemaType)
+from .schema import (
+    QAPISchemaArrayType,
+    QAPISchemaBuiltinType,
+    QAPISchemaType,
+)
 
 
 def _make_tree(obj, ifcond, features, extra=None):
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 68d8a1ce1cf..477ef25ccdf 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -14,9 +14,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+from collections import OrderedDict
 import os
 import re
-from collections import OrderedDict
 
 from .error import QAPIParseError, QAPISemError
 from .source import QAPISourceInfo
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index a835ee6fde3..093f7a38d88 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -14,9 +14,9 @@
 
 # TODO catching name collisions in generated code would be nice
 
+from collections import OrderedDict
 import os
 import re
-from collections import OrderedDict
 
 from .common import c_name, pointer_suffix
 from .error import QAPIError, QAPISemError
-- 
2.26.2

[PATCH v5 08/36] qapi: delint using flake8

[John Snow]

Petty style guide fixes and line length enforcement.  Not a big win, not
a big loss, but flake8 passes 100% on the qapi module, which gives us an
easy baseline to enforce hereafter.

A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/.flake8     |  2 ++
 scripts/qapi/commands.py |  3 ++-
 scripts/qapi/schema.py   |  8 +++++---
 scripts/qapi/visit.py    | 16 +++++++++++-----
 4 files changed, 20 insertions(+), 9 deletions(-)
 create mode 100644 scripts/qapi/.flake8

diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
new file mode 100644
index 00000000000..6b158c68b84
--- /dev/null
+++ b/scripts/qapi/.flake8
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 64ed5278f93..5dc2f5a9fa8 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type):
 def gen_marshal_output(ret_type):
     return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
+                                QObject **ret_out, Error **errp)
 {
     Visitor *v;
 
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 093f7a38d88..cfc52e1ae44 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -536,7 +536,7 @@ def set_defined_in(self, name):
             v.set_defined_in(name)
 
     def check(self, schema, seen):
-        if not self.tag_member: # flat union
+        if not self.tag_member:  # flat union
             self.tag_member = seen.get(c_name(self._tag_name))
             base = "'base'"
             # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname):
         self._entity_dict = {}
         self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
-        self._make_module(None) # built-ins
+        self._make_module(None)  # built-ins
         self._make_module(fname)
         self._predefining = True
         self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
             # But it's not tight: the disjunction need not imply it.  We
             # may end up compiling useless wrapper types.
             # TODO kill simple unions or implement the disjunction
-            assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
+
+            # pylint: disable=protected-access
+            assert (ifcond or []) == typ._ifcond
         else:
             self._def_entity(QAPISchemaObjectType(
                 name, info, None, ifcond, None, None, members, None))
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index ea277e7704b..808410d6f1b 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False):
     if not scalar:
         c_type += '*'
     return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
+
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_type)sobj, Error **errp);
 ''',
                  c_name=c_name(name), c_type=c_type)
 
@@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants):
 def gen_visit_list(name, element_type):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_name)s **obj, Error **errp)
 {
     bool ok = false;
     %(c_name)s *tail;
@@ -158,7 +161,8 @@ def gen_visit_list(name, element_type):
 def gen_visit_enum(name):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+                 %(c_name)s *obj, Error **errp)
 {
     int value = *obj;
     bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +176,8 @@ def gen_visit_enum(name):
 def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
 {
     bool ok = false;
 
@@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants):
 def gen_visit_object(name, base, members, variants):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s
+    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
 {
     bool ok = false;
 
-- 
2.26.2

[PATCH v5 09/36] qapi: add pylintrc

[John Snow]

Using `pylint --generate-rcfile > pylintrc`, generate a skeleton
pylintrc file. Sections that are not presently relevant (by the end of
this series) are removed leaving just the empty section as a search
engine / documentation hint to future authors.

I am targeting pylint 2.6.0. In the future (and hopefully before 5.2 is
released), I aim to have gitlab CI running the specific targeted
versions of pylint, mypy, flake8, etc in a job.

2.5.x will work if you additionally pass --disable=bad-whitespace.
This warning was removed from 2.6.x, for lack of consistent support.

Right now, quite a few modules are ignored as they are known to fail as
of this commit. modules will be removed from the known-bad list
throughout this and following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as
that folder is shared by many things, it is going inside the module
folder (for now). Due to a bug in pylint 2.5+, pylint does not
correctly recognize when it is being run from "inside" a package, and
must be run *outside* of the package.

Therefore, to run it, you must:

 > pylint scripts/qapi/ --rcfile=scripts/qapi/pylintrc

Signed-off-by: John Snow <jsnow@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/pylintrc | 73 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
 create mode 100644 scripts/qapi/pylintrc

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
new file mode 100644
index 00000000000..76d54c30f85
--- /dev/null
+++ b/scripts/qapi/pylintrc
@@ -0,0 +1,73 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the ignore list.
+# The regex matches against base names, not paths.
+ignore-patterns=common.py,
+                error.py,
+                expr.py,
+                gen.py,
+                parser.py,
+                schema.py,
+                source.py,
+                types.py,
+                visit.py,
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=fixme,
+        missing-docstring,
+        too-many-arguments,
+        too-many-branches,
+        too-many-statements,
+        too-many-instance-attributes,
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore import statements themselves when computing similarities.
+ignore-imports=yes
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]
-- 
2.26.2

[PATCH v5 10/36] qapi/common.py: Remove python compatibility workaround

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/common.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index ba35abea478..cee63eb95c7 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
     raw = code % kwds
     if indent_level:
         indent = genindent(indent_level)
-        # re.subn() lacks flags support before Python 2.7, use re.compile()
-        raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
-                      indent, raw)
-        raw = raw[0]
+        raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
-- 
2.26.2

[PATCH v5 11/36] qapi/common.py: Add indent manager

[John Snow]

Code style tools really dislike the use of global keywords, because it
generally involves re-binding the name at runtime which can have strange
effects depending on when and how that global name is referenced in
other modules.

Make a little indent level manager instead.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++--------------
 scripts/qapi/visit.py  |  7 +++---
 2 files changed, 36 insertions(+), 20 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index cee63eb95c7..b35318b72cf 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -93,33 +93,50 @@ def c_name(name, protect=True):
 pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indentation:
+    """
+    Indentation level management.
 
+    :param initial: Initial number of spaces, default 0.
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
 
-indent_level = 0
+    def __int__(self) -> int:
+        return self._level
 
+    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)
 
-def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the current indentation as a string of spaces."""
+        return ' ' * self._level
 
+    def __bool__(self) -> bool:
+        """True when there is a non-zero indentation."""
+        return bool(self._level)
 
-def pop_indent(indent_amount=4):
-    global indent_level
-    indent_level -= indent_amount
+    def increase(self, amount: int = 4) -> None:
+        """Increase the indentation level by ``amount``, default 4."""
+        self._level += amount
+
+    def decrease(self, amount: int = 4) -> None:
+        """Decrease the indentation level by ``amount``, default 4."""
+        if self._level < amount:
+            raise ArithmeticError(
+                f"Can't remove {amount:d} spaces from {self!r}")
+        self._level -= amount
+
+
+indent = Indentation()
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey indent, and strip eatspace.
 def cgen(code, **kwds):
     raw = code % kwds
-    if indent_level:
-        indent = genindent(indent_level)
-        raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+    if indent:
+        raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 808410d6f1b..14f30c228b7 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -18,9 +18,8 @@
     c_name,
     gen_endif,
     gen_if,
+    indent,
     mcgen,
-    pop_indent,
-    push_indent,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants):
     if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
 ''',
                          name=memb.name, c_name=c_name(memb.name))
-            push_indent()
+            indent.increase()
         ret += mcgen('''
     if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
         return false;
@@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants):
                      c_type=memb.type.c_name(), name=memb.name,
                      c_name=c_name(memb.name))
         if memb.optional:
-            pop_indent()
+            indent.decrease()
             ret += mcgen('''
     }
 ''')
-- 
2.26.2

[PATCH v5 12/36] qapi/common.py: delint with pylint

[John Snow]

At this point, that just means using a consistent strategy for constant names.
constants get UPPER_CASE and names not used externally get a leading underscore.

As a preference, while renaming constants to be UPPERCASE, move them to
the head of the file. Generally, it's nice to be able to audit the code
that runs on import in one central place.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/common.py | 18 ++++++++----------
 scripts/qapi/schema.py | 14 +++++++-------
 2 files changed, 15 insertions(+), 17 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index b35318b72cf..a417b6029c8 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -14,6 +14,11 @@
 import re
 
 
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+_C_NAME_TRANS = str.maketrans('.-', '__')
+
+
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-c_name_trans = str.maketrans('.-', '__')
-
-
 # Map @name to a valid C identifier.
 # If @protect, avoid returning certain ticklish identifiers (like
 # C keywords) by prepending 'q_'.
@@ -82,17 +84,13 @@ def c_name(name, protect=True):
                      'not_eq', 'or', 'or_eq', 'xor', 'xor_eq'])
     # namespace pollution:
     polluted_words = set(['unix', 'errno', 'mips', 'sparc', 'i386'])
-    name = name.translate(c_name_trans)
+    name = name.translate(_C_NAME_TRANS)
     if protect and (name in c89_words | c99_words | c11_words | gcc_words
                     | cpp_words | polluted_words):
         return 'q_' + name
     return name
 
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
 class Indentation:
     """
     Indentation level management.
@@ -132,12 +130,12 @@ def decrease(self, amount: int = 4) -> None:
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent, and strip eatspace.
+# Obey indent, and strip EATSPACE.
 def cgen(code, **kwds):
     raw = code % kwds
     if indent:
         raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
-    return re.sub(re.escape(eatspace) + r' *', '', raw)
+    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
 def mcgen(code, **kwds):
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index cfc52e1ae44..74c6b96d391 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,7 +18,7 @@
 import os
 import re
 
-from .common import c_name, pointer_suffix
+from .common import POINTER_SUFFIX, c_name
 from .error import QAPIError, QAPISemError
 from .expr import check_exprs
 from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self):
         return True
 
     def c_type(self):
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def json_type(self):
         return 'array'
@@ -430,7 +430,7 @@ def c_name(self):
 
     def c_type(self):
         assert not self.is_implicit()
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def c_unboxed_type(self):
         return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None):
             v.connect_doc(doc)
 
     def c_type(self):
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def json_type(self):
         return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type):
         self._make_array_type(name, None)
 
     def _def_predefineds(self):
-        for t in [('str',    'string',  'char' + pointer_suffix),
+        for t in [('str',    'string',  'char' + POINTER_SUFFIX),
                   ('number', 'number',  'double'),
                   ('int',    'int',     'int64_t'),
                   ('int8',   'int',     'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self):
                   ('uint64', 'int',     'uint64_t'),
                   ('size',   'int',     'uint64_t'),
                   ('bool',   'boolean', 'bool'),
-                  ('any',    'value',   'QObject' + pointer_suffix),
-                  ('null',   'null',    'QNull' + pointer_suffix)]:
+                  ('any',    'value',   'QObject' + POINTER_SUFFIX),
+                  ('null',   'null',    'QNull' + POINTER_SUFFIX)]:
             self._def_builtin_type(*t)
         self.the_empty_object_type = QAPISchemaObjectType(
             'q_empty', None, None, None, None, None, [], None)
-- 
2.26.2

[PATCH v5 13/36] qapi/common.py: Replace one-letter 'c' variable

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index a417b6029c8..338adedef4f 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -30,14 +30,14 @@ def camel_to_upper(value):
     new_name = ''
     length = len(c_fun_str)
     for i in range(length):
-        c = c_fun_str[i]
-        # When c is upper and no '_' appears before, do more checks
-        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+        char = c_fun_str[i]
+        # When char is upper case and no '_' appears before, do more checks
+        if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
             if i < length - 1 and c_fun_str[i + 1].islower():
                 new_name += '_'
             elif c_fun_str[i - 1].isdigit():
                 new_name += '_'
-        new_name += c
+        new_name += char
     return new_name.lstrip('_').upper()
 
 
-- 
2.26.2

[PATCH v5 14/36] qapi/common.py: check with pylint

[John Snow]

Remove qapi/common.py from the pylintrc ignore list.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qapi/pylintrc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 76d54c30f85..507f15537ab 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -2,8 +2,7 @@
 
 # Add files or directories matching the regex patterns to the ignore list.
 # The regex matches against base names, not paths.
-ignore-patterns=common.py,
-                error.py,
+ignore-patterns=error.py,
                 expr.py,
                 gen.py,
                 parser.py,
-- 
2.26.2

[PATCH v5 15/36] qapi/common.py: add type hint annotations

[John Snow]

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/common.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 338adedef4f..74a2c001ed9 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -12,6 +12,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Optional, Sequence
 
 
 EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
     return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
 # into substrings of a generated C function name.
 # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
 # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None:
 
 # Generate @code with @kwds interpolated.
 # Obey indent, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: object) -> str:
     raw = code % kwds
     if indent:
         raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
     return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: object) -> str:
     if code[0] == '\n':
         code = code[1:]
     return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
     return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
     return mcgen('''
 #ifndef %(name)s
 #define %(name)s
@@ -157,7 +160,7 @@ def guardstart(name):
                  name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
     return mcgen('''
 
 #endif /* %(name)s */
@@ -165,7 +168,7 @@ def guardend(name):
                  name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in ifcond:
         ret += mcgen('''
@@ -174,7 +177,7 @@ def gen_if(ifcond):
     return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in reversed(ifcond):
         ret += mcgen('''
@@ -183,7 +186,9 @@ def gen_endif(ifcond):
     return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
     ret = ''
     sep = ''
     if boxed:
-- 
2.26.2

[PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate

[John Snow]

As docstrings, they'll show up in documentation and IDE help.

The docstring style being targeted is the Sphinx documentation
style. Sphinx uses an extension of ReST with "domains". We use the
(implicit) Python domain, which supports a number of custom "info
fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Primarily, we use `:param X: descr`, `:return[s]: descr`, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.

(No, nothing checks or enforces this style that I am aware of. Sphinx
either chokes or succeeds, but does not enforce a standard of what is
otherwise inside the docstring. Pycharm does highlight when your param
fields are not aligned with the actual fields present. It does not
highlight missing return or exception statements. There is no existing
style guide I am aware of that covers a standard for a minimally
acceptable docstring. I am debating writing one.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
 1 file changed, 39 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 74a2c001ed9..0ef38ea5fe0 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -15,15 +15,24 @@
 from typing import Optional, Sequence
 
 
+#: Sentinel value that causes all space to its right to be removed.
 EATSPACE = '\033EATSPACE.'
 POINTER_SUFFIX = ' *' + EATSPACE
 _C_NAME_TRANS = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
 def camel_to_upper(value: str) -> str:
+    """
+    Converts CamelCase to CAMEL_CASE.
+
+    Examples:
+      ENUMName -> ENUM_NAME
+      EnumName1 -> ENUM_NAME1
+      ENUM_NAME -> ENUM_NAME
+      ENUM_NAME1 -> ENUM_NAME1
+      ENUM_Name2 -> ENUM_NAME2
+      ENUM24_Name -> ENUM24_NAME
+    """
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str:
 def c_enum_const(type_name: str,
                  const_name: str,
                  prefix: Optional[str] = None) -> str:
+    """
+    Generate a C enumeration constant name.
+
+    :param type_name: The name of the enumeration.
+    :param const_name: The name of this constant.
+    :param prefix: Optional, prefix that overrides the type_name.
+    """
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 def c_name(name: str, protect: bool = True) -> str:
+    """
+    Map ``name`` to a valid C identifier.
+
+    Used for converting 'name' from a 'name':'type' qapi definition
+    into a generated struct member, as well as converting type names
+    into substrings of a generated C function name.
+
+    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+    :param name: The name to map.
+    :param protect: If true, avoid returning certain ticklish identifiers
+                    (like C keywords) by prepending ``q_``.
+    """
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None:
         self._level -= amount
 
 
+#: Global, current indent level for code generation.
 indent = Indentation()
 
 
-# Generate @code with @kwds interpolated.
-# Obey indent, and strip EATSPACE.
 def cgen(code: str, **kwds: object) -> str:
+    """
+    Generate ``code`` with ``kwds`` interpolated.
+
+    Obey `indent`, and strip `EATSPACE`.
+    """
     raw = code % kwds
     if indent:
         raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
-- 
2.26.2

[PATCH v5 17/36] qapi/common.py: move build_params into gen.py

[John Snow]

Including it in common.py creates a circular import dependency; schema
relies on common, but common.build_params requires a type annotation
from schema. To type this properly, it needs to be moved outside the
cycle.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/commands.py |  9 +++++++--
 scripts/qapi/common.py   | 23 -----------------------
 scripts/qapi/events.py   |  9 ++-------
 scripts/qapi/gen.py      | 31 +++++++++++++++++++++++++++++--
 4 files changed, 38 insertions(+), 34 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 5dc2f5a9fa8..f67393f8713 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,8 +13,13 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import build_params, c_name, mcgen
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import c_name, mcgen
+from .gen import (
+    QAPIGenCCode,
+    QAPISchemaModularCVisitor,
+    build_params,
+    ifcontext,
+)
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 0ef38ea5fe0..9ab0685cc51 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
 #endif /* %(cond)s */
 ''', cond=ifc)
     return ret
-
-
-def build_params(arg_type,
-                 boxed: bool,
-                 extra: Optional[str] = None) -> str:
-    ret = ''
-    sep = ''
-    if boxed:
-        assert arg_type
-        ret += '%s arg' % arg_type.c_param_type()
-        sep = ', '
-    elif arg_type:
-        assert not arg_type.variants
-        for memb in arg_type.members:
-            ret += sep
-            sep = ', '
-            if memb.optional:
-                ret += 'bool has_%s, ' % c_name(memb.name)
-            ret += '%s %s' % (memb.type.c_param_type(),
-                              c_name(memb.name))
-    if extra:
-        ret += sep + extra
-    return ret if ret else 'void'
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 6b3afa14d72..f840a62ed92 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,13 +12,8 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import (
-    build_params,
-    c_enum_const,
-    c_name,
-    mcgen,
-)
-from .gen import QAPISchemaModularCVisitor, ifcontext
+from .common import c_enum_const, c_name, mcgen
+from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
 
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 1fed712b43b..f2e2746fea5 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -2,9 +2,11 @@
 #
 # QAPI code generation
 #
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2019 Red Hat Inc.
 #
 # Authors:
+#  Anthony Liguori <aliguori@us.ibm.com>
 #  Markus Armbruster <armbru@redhat.com>
 #  Marc-André Lureau <marcandre.lureau@redhat.com>
 #
@@ -15,16 +17,18 @@
 import errno
 import os
 import re
+from typing import Optional
 
 from .common import (
     c_fname,
+    c_name,
     gen_endif,
     gen_if,
     guardend,
     guardstart,
     mcgen,
 )
-from .schema import QAPISchemaVisitor
+from .schema import QAPISchemaObjectType, QAPISchemaVisitor
 
 
 class QAPIGen:
@@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
     return out
 
 
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
+    ret = ''
+    sep = ''
+    if boxed:
+        assert arg_type
+        ret += '%s arg' % arg_type.c_param_type()
+        sep = ', '
+    elif arg_type:
+        assert not arg_type.variants
+        for memb in arg_type.members:
+            ret += sep
+            sep = ', '
+            if memb.optional:
+                ret += 'bool has_%s, ' % c_name(memb.name)
+            ret += '%s %s' % (memb.type.c_param_type(),
+                              c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret if ret else 'void'
+
+
 class QAPIGenCCode(QAPIGen):
 
     def __init__(self, fname):
-- 
2.26.2

[PATCH v5 18/36] qapi: establish mypy type-checking baseline

[John Snow]

Fix two very minor issues, and then establish a mypy type-checking
baseline.

Like pylint, this should be run from the folder above:

 > mypy --config-file=qapi/mypy.ini qapi/

This is designed and tested for mypy 0.770 or greater.

Signed-off-by: John Snow <jsnow@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/mypy.ini  | 60 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/schema.py |  3 ++-
 2 files changed, 62 insertions(+), 1 deletion(-)
 create mode 100644 scripts/qapi/mypy.ini

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
new file mode 100644
index 00000000000..00fac15dc6e
--- /dev/null
+++ b/scripts/qapi/mypy.ini
@@ -0,0 +1,60 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 74c6b96d391..483b4b68dff 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -17,6 +17,7 @@
 from collections import OrderedDict
 import os
 import re
+from typing import Optional
 
 from .common import POINTER_SUFFIX, c_name
 from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
 class QAPISchemaEntity:
-    meta = None
+    meta: Optional[str] = None
 
     def __init__(self, name, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
-- 
2.26.2

[PATCH v5 19/36] qapi/events.py: add type hint annotations

[John Snow]

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
 scripts/qapi/mypy.ini  |  5 -----
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index f840a62ed92..57e0939e963 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,19 +12,31 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List
+
 from .common import c_enum_const, c_name, mcgen
 from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
-from .schema import QAPISchemaEnumMember
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+)
+from .source import QAPISourceInfo
 from .types import gen_enum, gen_enum_lookup
 
 
-def build_event_send_proto(name, arg_type, boxed):
+def build_event_send_proto(name: str,
+                           arg_type: QAPISchemaObjectType,
+                           boxed: bool) -> str:
     return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
         'c_name': c_name(name.lower()),
         'param': build_params(arg_type, boxed)}
 
 
-def gen_event_send_decl(name, arg_type, boxed):
+def gen_event_send_decl(name: str,
+                        arg_type: QAPISchemaObjectType,
+                        boxed: bool) -> str:
     return mcgen('''
 
 %(proto)s;
@@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed):
 
 
 # Declare and initialize an object 'qapi' using parameters from build_params()
-def gen_param_var(typ):
+def gen_param_var(typ: QAPISchemaObjectType) -> str:
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
@@ -61,7 +73,11 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
+def gen_event_send(name: str,
+                   arg_type: QAPISchemaObjectType,
+                   boxed: bool,
+                   event_enum_name: str,
+                   event_emit: str) -> str:
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
 
 class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-events',
             ' * Schema-defined QAPI/QMP events', None, __doc__)
         self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-        self._event_enum_members = []
+        self._event_enum_members: List[QAPISchemaEnumMember] = []
         self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         events = self._module_basename('qapi-events', name)
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
@@ -168,7 +184,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         self._add_system_module('emit', ' * QAPI Events emission')
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
@@ -189,7 +205,13 @@ def visit_end(self):
                              event_emit=self._event_emit_name,
                              event_enum=self._event_enum_name))
 
-    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+    def visit_event(self,
+                    name: str,
+                    info: QAPISourceInfo,
+                    ifcond: List[str],
+                    features: List[QAPISchemaFeature],
+                    arg_type: QAPISchemaObjectType,
+                    boxed: bool) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_event_send_decl(name, arg_type, boxed))
             self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -200,7 +222,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed):
         self._event_enum_members.append(QAPISchemaEnumMember(name, None))
 
 
-def gen_events(schema, output_dir, prefix):
+def gen_events(schema: QAPISchema,
+               output_dir: str,
+               prefix: str) -> None:
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 00fac15dc6e..5df11e53fd1 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -14,11 +14,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.events]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.expr]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
-- 
2.26.2

[PATCH v5 20/36] qapi/events.py: Move comments into docstrings

[John Snow]

Clarify them while we're here.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/events.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 57e0939e963..599f3d1f564 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -44,8 +44,12 @@ def gen_event_send_decl(name: str,
                  proto=build_event_send_proto(name, arg_type, boxed))
 
 
-# Declare and initialize an object 'qapi' using parameters from build_params()
 def gen_param_var(typ: QAPISchemaObjectType) -> str:
+    """
+    Generate a struct variable holding the event parameters.
+
+    Initialize it with the function arguments defined in `gen_event_send`.
+    """
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
-- 
2.26.2

[PATCH v5 21/36] qapi/commands.py: Don't re-bind to variable of different type

[John Snow]

Mypy isn't a fan of rebinding a variable with a new data type.
It's easy enough to avoid.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/commands.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index f67393f8713..586774a23c7 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -195,14 +195,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     if not options:
         options = ['QCO_NO_OPTIONS']
 
-    options = " | ".join(options)
-
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
                          qmp_marshal_%(c_name)s, %(opts)s);
 ''',
                 name=name, c_name=c_name(name),
-                opts=options)
+                opts=" | ".join(options))
     return ret
 
 
-- 
2.26.2

[PATCH v5 22/36] qapi/commands.py: add type hint annotations

[John Snow]

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/commands.py | 71 ++++++++++++++++++++++++++++++----------
 1 file changed, 54 insertions(+), 17 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 586774a23c7..24f347a3f12 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,16 +13,34 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import (
+    Dict,
+    List,
+    Optional,
+    Set,
+)
+
 from .common import c_name, mcgen
 from .gen import (
+    QAPIGenC,
     QAPIGenCCode,
     QAPISchemaModularCVisitor,
     build_params,
     ifcontext,
 )
+from .schema import (
+    QAPISchema,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaType,
+)
+from .source import QAPISourceInfo
 
 
-def gen_command_decl(name, arg_type, boxed, ret_type):
+def gen_command_decl(name: str,
+                     arg_type: Optional[QAPISchemaObjectType],
+                     boxed: bool,
+                     ret_type: Optional[QAPISchemaType]) -> str:
     return mcgen('''
 %(c_type)s qmp_%(c_name)s(%(params)s);
 ''',
@@ -31,7 +49,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type):
                  params=build_params(arg_type, boxed, 'Error **errp'))
 
 
-def gen_call(name, arg_type, boxed, ret_type):
+def gen_call(name: str,
+             arg_type: Optional[QAPISchemaObjectType],
+             boxed: bool,
+             ret_type: Optional[QAPISchemaType]) -> str:
     ret = ''
 
     argstr = ''
@@ -67,7 +88,7 @@ def gen_call(name, arg_type, boxed, ret_type):
     return ret
 
 
-def gen_marshal_output(ret_type):
+def gen_marshal_output(ret_type: QAPISchemaType) -> str:
     return mcgen('''
 
 static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
@@ -88,19 +109,22 @@ def gen_marshal_output(ret_type):
                  c_type=ret_type.c_type(), c_name=ret_type.c_name())
 
 
-def build_marshal_proto(name):
+def build_marshal_proto(name: str) -> str:
     return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)'
             % c_name(name))
 
 
-def gen_marshal_decl(name):
+def gen_marshal_decl(name: str) -> str:
     return mcgen('''
 %(proto)s;
 ''',
                  proto=build_marshal_proto(name))
 
 
-def gen_marshal(name, arg_type, boxed, ret_type):
+def gen_marshal(name: str,
+                arg_type: Optional[QAPISchemaObjectType],
+                boxed: bool,
+                ret_type: Optional[QAPISchemaType]) -> str:
     have_args = boxed or (arg_type and not arg_type.is_empty())
 
     ret = mcgen('''
@@ -182,7 +206,10 @@ def gen_marshal(name, arg_type, boxed, ret_type):
     return ret
 
 
-def gen_register_command(name, success_response, allow_oob, allow_preconfig):
+def gen_register_command(name: str,
+                         success_response: bool,
+                         allow_oob: bool,
+                         allow_preconfig: bool) -> str:
     options = []
 
     if not success_response:
@@ -204,7 +231,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     return ret
 
 
-def gen_registry(registry, prefix):
+def gen_registry(registry: str, prefix: str) -> str:
     ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -221,15 +248,14 @@ def gen_registry(registry, prefix):
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
-
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-commands',
             ' * Schema-defined QAPI/QMP commands', None, __doc__)
         self._regy = QAPIGenCCode(None)
-        self._visited_ret_types = {}
+        self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {}
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         self._visited_ret_types[self._genc] = set()
         commands = self._module_basename('qapi-commands', name)
         types = self._module_basename('qapi-types', name)
@@ -253,7 +279,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         self._add_system_module('init', ' * QAPI Commands initialization')
         self._genh.add(mcgen('''
 #include "qapi/qmp/dispatch.h"
@@ -269,9 +295,18 @@ def visit_end(self):
                                       prefix=self._prefix))
         self._genc.add(gen_registry(self._regy.get_content(), self._prefix))
 
-    def visit_command(self, name, info, ifcond, features,
-                      arg_type, ret_type, gen, success_response, boxed,
-                      allow_oob, allow_preconfig):
+    def visit_command(self,
+                      name: str,
+                      info: QAPISourceInfo,
+                      ifcond: List[str],
+                      features: List[QAPISchemaFeature],
+                      arg_type: Optional[QAPISchemaObjectType],
+                      ret_type: Optional[QAPISchemaType],
+                      gen: bool,
+                      success_response: bool,
+                      boxed: bool,
+                      allow_oob: bool,
+                      allow_preconfig: bool) -> None:
         if not gen:
             return
         # FIXME: If T is a user-defined type, the user is responsible
@@ -292,7 +327,9 @@ def visit_command(self, name, info, ifcond, features,
                                                 allow_oob, allow_preconfig))
 
 
-def gen_commands(schema, output_dir, prefix):
+def gen_commands(schema: QAPISchema,
+                 output_dir: str,
+                 prefix: str) -> None:
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir)
-- 
2.26.2

[PATCH v5 23/36] qapi/commands.py: enable checking with mypy

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 5df11e53fd1..8ab9ac52cc4 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -4,11 +4,6 @@ strict_optional = False
 disallow_untyped_calls = False
 python_version = 3.6
 
-[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.error]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
-- 
2.26.2

[PATCH v5 24/36] qapi/source.py: add type hint annotations

[John Snow]

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no
parameters to document a return type of None to be considered fully
typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the
return annotation whenever possible.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/mypy.ini  |  5 -----
 scripts/qapi/source.py | 31 ++++++++++++++++++-------------
 2 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 8ab9ac52cc4..1b8555dfa39 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -34,11 +34,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.types]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index e97b9a8e15e..1cc6a5b82dc 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -11,37 +11,42 @@
 
 import copy
 import sys
+from typing import List, Optional, TypeVar
 
 
 class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
         # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist = []
+        self.returns_whitelist: List[str] = []
         # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = []
 
 
 class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent: Optional[T]):
         self.fname = fname
         self.line = line
         self.parent = parent
-        self.pragma = parent.pragma if parent else QAPISchemaPragma()
-        self.defn_meta = None
-        self.defn_name = None
+        self.pragma: QAPISchemaPragma = (
+            parent.pragma if parent else QAPISchemaPragma()
+        )
+        self.defn_meta: Optional[str] = None
+        self.defn_name: Optional[str] = None
 
-    def set_defn(self, meta, name):
+    def set_defn(self, meta: str, name: str) -> None:
         self.defn_meta = meta
         self.defn_name = name
 
-    def next_line(self):
+    def next_line(self: T) -> T:
         info = copy.copy(self)
         info.line += 1
         return info
 
-    def loc(self):
+    def loc(self) -> str:
         if self.fname is None:
             return sys.argv[0]
         ret = self.fname
@@ -49,13 +54,13 @@ def loc(self):
             ret += ':%d' % self.line
         return ret
 
-    def in_defn(self):
+    def in_defn(self) -> str:
         if self.defn_name:
             return "%s: In %s '%s':\n" % (self.fname,
                                           self.defn_meta, self.defn_name)
         return ''
 
-    def include_path(self):
+    def include_path(self) -> str:
         ret = ''
         parent = self.parent
         while parent:
@@ -63,5 +68,5 @@ def include_path(self):
             parent = parent.parent
         return ret
 
-    def __str__(self):
+    def __str__(self) -> str:
         return self.include_path() + self.in_defn() + self.loc()
-- 
2.26.2

[PATCH v5 25/36] qapi/source.py: delint with pylint

[John Snow]

Shush an error and leave a hint for future cleanups when we're allowed
to use Python 3.7+.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/pylintrc  | 1 -
 scripts/qapi/source.py | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 507f15537ab..d840b150313 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -7,7 +7,6 @@ ignore-patterns=error.py,
                 gen.py,
                 parser.py,
                 schema.py,
-                source.py,
                 types.py,
                 visit.py,
 
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index 1cc6a5b82dc..ba991d798fe 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -15,6 +15,9 @@
 
 
 class QAPISchemaPragma:
+    # Replace with @dataclass in Python 3.7+
+    # pylint: disable=too-few-public-methods
+
     def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
-- 
2.26.2

[PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module

[John Snow]

The edge case is that if the name is '', this expression returns a
string instead of a bool, which violates our declared type.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qapi/gen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index f2e2746fea5..1bad37fc06b 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
 
     @staticmethod
     def _is_user_module(name):
-        return name and not name.startswith('./')
+        return bool(name and not name.startswith('./'))
 
     @staticmethod
     def _is_builtin_module(name):
-- 
2.26.2

[PATCH v5 27/36] qapi/gen.py: add type hint annotations

[John Snow]

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
 1 file changed, 57 insertions(+), 47 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 1bad37fc06b..d0391cd8718 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -17,7 +17,13 @@
 import errno
 import os
 import re
-from typing import Optional
+from typing import (
+    Dict,
+    Iterator,
+    List,
+    Optional,
+    Tuple,
+)
 
 from .common import (
     c_fname,
@@ -29,31 +35,31 @@
     mcgen,
 )
 from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo
 
 
 class QAPIGen:
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):
         self.fname = fname
         self._preamble = ''
         self._body = ''
 
-    def preamble_add(self, text):
+    def preamble_add(self, text: str) -> None:
         self._preamble += text
 
-    def add(self, text):
+    def add(self, text: str) -> None:
         self._body += text
 
-    def get_content(self):
+    def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
-    def _top(self):
+    def _top(self) -> str:
         return ''
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return ''
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         # Include paths starting with ../ are used to reuse modules of the main
         # schema in specialised schemas. Don't overwrite the files that are
         # already generated for the main schema.
@@ -78,7 +84,7 @@ def write(self, output_dir):
         f.close()
 
 
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
     if before == after:
         return after   # suppress empty #if ... #endif
 
@@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
 
 
 class QAPIGenCCode(QAPIGen):
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):
         super().__init__(fname)
-        self._start_if = None
+        self._start_if: Optional[Tuple[List[str], str, str]] = None
 
-    def start_if(self, ifcond):
+    def start_if(self, ifcond: List[str]) -> None:
         assert self._start_if is None
         self._start_if = (ifcond, self._body, self._preamble)
 
-    def end_if(self):
+    def end_if(self) -> None:
         assert self._start_if
         self._wrap_ifcond()
         self._start_if = None
 
-    def _wrap_ifcond(self):
+    def _wrap_ifcond(self) -> None:
         self._body = _wrap_ifcond(self._start_if[0],
                                   self._start_if[1], self._body)
         self._preamble = _wrap_ifcond(self._start_if[0],
                                       self._start_if[2], self._preamble)
 
-    def get_content(self):
+    def get_content(self) -> str:
         assert self._start_if is None
         return super().get_content()
 
 
 class QAPIGenC(QAPIGenCCode):
-
-    def __init__(self, fname, blurb, pydoc):
+    def __init__(self, fname: str, blurb: str, pydoc: str):
         super().__init__(fname)
         self._blurb = blurb
         self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
                                                   re.MULTILINE))
 
-    def _top(self):
+    def _top(self) -> str:
         return mcgen('''
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
@@ -167,7 +171,7 @@ def _top(self):
 ''',
                      blurb=self._blurb, copyright=self._copyright)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return mcgen('''
 
 /* Dummy declaration to prevent empty .o file */
@@ -177,16 +181,15 @@ def _bottom(self):
 
 
 class QAPIGenH(QAPIGenC):
-
-    def _top(self):
+    def _top(self) -> str:
         return super()._top() + guardstart(self.fname)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return guardend(self.fname)
 
 
 @contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:
     """
     A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
@@ -214,8 +217,11 @@ def ifcontext(ifcond, *args):
 
 
 class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 blurb: str,
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc):
         self._genh = QAPIGenH(self._prefix + self._what + '.h',
                               blurb, pydoc)
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         self._genc.write(output_dir)
         self._genh.write(output_dir)
 
 
 class QAPISchemaModularCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 user_blurb: str,
+                 builtin_blurb: Optional[str],
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._user_blurb = user_blurb
         self._builtin_blurb = builtin_blurb
         self._pydoc = pydoc
-        self._genc = None
-        self._genh = None
-        self._module = {}
-        self._main_module = None
+        self._genc: Optional[QAPIGenC] = None
+        self._genh: Optional[QAPIGenH] = None
+        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
+        self._main_module: Optional[str] = None
 
     @staticmethod
-    def _is_user_module(name):
+    def _is_user_module(name: Optional[str]) -> bool:
         return bool(name and not name.startswith('./'))
 
     @staticmethod
-    def _is_builtin_module(name):
+    def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what, name):
+    def _module_dirname(self, what: str, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
 
-    def _module_basename(self, what, name):
+    def _module_basename(self, what: str, name: Optional[str]) -> str:
         ret = '' if self._is_builtin_module(name) else self._prefix
         if self._is_user_module(name):
             basename = os.path.basename(name)
@@ -266,27 +276,27 @@ def _module_basename(self, what, name):
             ret += re.sub(r'-', '-' + name + '-', what)
         return ret
 
-    def _module_filename(self, what, name):
+    def _module_filename(self, what: str, name: Optional[str]) -> str:
         return os.path.join(self._module_dirname(what, name),
                             self._module_basename(what, name))
 
-    def _add_module(self, name, blurb):
+    def _add_module(self, name: Optional[str], blurb: str) -> None:
         basename = self._module_filename(self._what, name)
         genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
         genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
         self._module[name] = (genc, genh)
         self._genc, self._genh = self._module[name]
 
-    def _add_user_module(self, name, blurb):
+    def _add_user_module(self, name: str, blurb: str) -> None:
         assert self._is_user_module(name)
         if self._main_module is None:
             self._main_module = name
         self._add_module(name, blurb)
 
-    def _add_system_module(self, name, blurb):
+    def _add_system_module(self, name: Optional[str], blurb: str) -> None:
         self._add_module(name and './' + name, blurb)
 
-    def write(self, output_dir, opt_builtins=False):
+    def write(self, output_dir: str, opt_builtins: bool = False) -> None:
         for name in self._module:
             if self._is_builtin_module(name) and not opt_builtins:
                 continue
@@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False):
             genc.write(output_dir)
             genh.write(output_dir)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:
         pass
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         pass
 
-    def visit_module(self, name):
+    def visit_module(self, name: Optional[str]) -> None:
         if name is None:
             if self._builtin_blurb:
                 self._add_system_module(None, self._builtin_blurb)
@@ -314,7 +324,7 @@ def visit_module(self, name):
             self._add_user_module(name, self._user_blurb)
             self._begin_user_module(name)
 
-    def visit_include(self, name, info):
+    def visit_include(self, name: str, info: QAPISourceInfo) -> None:
         relname = os.path.relpath(self._module_filename(self._what, name),
                                   os.path.dirname(self._genh.fname))
         self._genh.preamble_add(mcgen('''
-- 
2.26.2

[PATCH v5 28/36] qapi/gen.py: Enable checking with mypy

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 1b8555dfa39..c6960ff2dbd 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -14,11 +14,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.gen]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.introspect]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
-- 
2.26.2

[PATCH v5 29/36] qapi/gen.py: Remove unused parameter

[John Snow]

module_basename doesn't use the 'what' argument, so remove it.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/gen.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index d0391cd8718..3624162bb77 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool:
     def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what: str, name: Optional[str]) -> str:
+    def _module_dirname(self, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
@@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
         return ret
 
     def _module_filename(self, what: str, name: Optional[str]) -> str:
-        return os.path.join(self._module_dirname(what, name),
+        return os.path.join(self._module_dirname(name),
                             self._module_basename(what, name))
 
     def _add_module(self, name: Optional[str], blurb: str) -> None:
-- 
2.26.2

[PATCH v5 30/36] qapi/gen.py: update write() to be more idiomatic

[John Snow]

Make the file handling here just a tiny bit more idiomatic.
(I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception,
use fdopen() to wrap the file descriptor in a File-like object,
and use a context manager for managing the file pointer.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/gen.py | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 3624162bb77..579ee283297 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -14,7 +14,6 @@
 # See the COPYING file in the top-level directory.
 
 from contextlib import contextmanager
-import errno
 import os
 import re
 from typing import (
@@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None:
             return
         pathname = os.path.join(output_dir, self.fname)
         odir = os.path.dirname(pathname)
+
         if odir:
-            try:
-                os.makedirs(odir)
-            except os.error as e:
-                if e.errno != errno.EEXIST:
-                    raise
+            os.makedirs(odir, exist_ok=True)
+
+        # use os.open for O_CREAT to create and read a non-existant file
         fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-        f = open(fd, 'r+', encoding='utf-8')
-        text = self.get_content()
-        oldtext = f.read(len(text) + 1)
-        if text != oldtext:
-            f.seek(0)
-            f.truncate(0)
-            f.write(text)
-        f.close()
+        with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+            text = self.get_content()
+            oldtext = fp.read(len(text) + 1)
+            if text != oldtext:
+                fp.seek(0)
+                fp.truncate(0)
+                fp.write(text)
 
 
 def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
-- 
2.26.2

[PATCH v5 31/36] qapi/gen.py: delint with pylint

[John Snow]

'fp' and 'fd' are self-evident in context, add them to the list of OK
names.

_top and _bottom also need to stay standard methods because some users
override the method and need to use `self`. Tell pylint to shush.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/gen.py   | 2 ++
 scripts/qapi/pylintrc | 5 +++--
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 579ee283297..29bdd140587 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -53,9 +53,11 @@ def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
     def _top(self) -> str:
+        # pylint: disable=no-self-use
         return ''
 
     def _bottom(self) -> str:
+        # pylint: disable=no-self-use
         return ''
 
     def write(self, output_dir: str) -> None:
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index d840b150313..8badcb11cda 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -4,7 +4,6 @@
 # The regex matches against base names, not paths.
 ignore-patterns=error.py,
                 expr.py,
-                gen.py,
                 parser.py,
                 schema.py,
                 types.py,
@@ -45,7 +44,9 @@ good-names=i,
            k,
            ex,
            Run,
-           _
+           _,
+           fp,  # fp = open(...)
+           fd,  # fd = os.open(...)
 
 [VARIABLES]
 
-- 
2.26.2

[PATCH v5 32/36] qapi/types.py: add type hint annotations

[John Snow]

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index c6960ff2dbd..83f19813553 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -29,11 +29,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.types]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.visit]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 53b47f9e58a..766822feb3a 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,6 +13,8 @@
 # See the COPYING file in the top-level directory.
 """
 
+from typing import List, Optional
+
 from .common import (
     c_enum_const,
     c_name,
@@ -21,7 +23,16 @@
     mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaType,
+    QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
 
 
 # variants must be emitted before their container; track what has already
@@ -29,7 +40,9 @@
 objects_seen = set()
 
 
-def gen_enum_lookup(name, members, prefix=None):
+def gen_enum_lookup(name: str,
+                    members: List[QAPISchemaEnumMember],
+                    prefix: Optional[str] = None) -> str:
     ret = mcgen('''
 
 const QEnumLookup %(c_name)s_lookup = {
@@ -54,7 +67,9 @@ def gen_enum_lookup(name, members, prefix=None):
     return ret
 
 
-def gen_enum(name, members, prefix=None):
+def gen_enum(name: str,
+             members: List[QAPISchemaEnumMember],
+             prefix: Optional[str] = None) -> str:
     # append automatically generated _MAX value
     enum_members = members + [QAPISchemaEnumMember('_MAX', None)]
 
@@ -88,7 +103,7 @@ def gen_enum(name, members, prefix=None):
     return ret
 
 
-def gen_fwd_object_or_array(name):
+def gen_fwd_object_or_array(name: str) -> str:
     return mcgen('''
 
 typedef struct %(c_name)s %(c_name)s;
@@ -96,7 +111,7 @@ def gen_fwd_object_or_array(name):
                  c_name=c_name(name))
 
 
-def gen_array(name, element_type):
+def gen_array(name: str, element_type: QAPISchemaType) -> str:
     return mcgen('''
 
 struct %(c_name)s {
@@ -107,7 +122,7 @@ def gen_array(name, element_type):
                  c_name=c_name(name), c_type=element_type.c_type())
 
 
-def gen_struct_members(members):
+def gen_struct_members(members: List[QAPISchemaObjectTypeMember]) -> str:
     ret = ''
     for memb in members:
         ret += gen_if(memb.ifcond)
@@ -124,7 +139,10 @@ def gen_struct_members(members):
     return ret
 
 
-def gen_object(name, ifcond, base, members, variants):
+def gen_object(name: str, ifcond: List[str],
+               base: Optional[QAPISchemaObjectType],
+               members: List[QAPISchemaObjectTypeMember],
+               variants: Optional[QAPISchemaVariants]) -> str:
     if name in objects_seen:
         return ''
     objects_seen.add(name)
@@ -178,7 +196,7 @@ def gen_object(name, ifcond, base, members, variants):
     return ret
 
 
-def gen_upcast(name, base):
+def gen_upcast(name: str, base: QAPISchemaObjectType) -> str:
     # C makes const-correctness ugly.  We have to cast away const to let
     # this function work for both const and non-const obj.
     return mcgen('''
@@ -191,7 +209,7 @@ def gen_upcast(name, base):
                  c_name=c_name(name), base=base.c_name())
 
 
-def gen_variants(variants):
+def gen_variants(variants: QAPISchemaVariants) -> str:
     ret = mcgen('''
     union { /* union tag is @%(c_name)s */
 ''',
@@ -215,7 +233,7 @@ def gen_variants(variants):
     return ret
 
 
-def gen_type_cleanup_decl(name):
+def gen_type_cleanup_decl(name: str) -> str:
     ret = mcgen('''
 
 void qapi_free_%(c_name)s(%(c_name)s *obj);
@@ -225,7 +243,7 @@ def gen_type_cleanup_decl(name):
     return ret
 
 
-def gen_type_cleanup(name):
+def gen_type_cleanup(name: str) -> str:
     ret = mcgen('''
 
 void qapi_free_%(c_name)s(%(c_name)s *obj)
@@ -247,12 +265,12 @@ def gen_type_cleanup(name):
 
 class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-types', ' * Schema-defined QAPI types',
             ' * Built-in QAPI types', __doc__)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
@@ -263,7 +281,7 @@ def _begin_system_module(self, name):
 #include "qapi/util.h"
 '''))
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
         self._genc.preamble_add(mcgen('''
@@ -277,27 +295,43 @@ def _begin_user_module(self, name):
 #include "qapi/qapi-builtin-types.h"
 '''))
 
-    def visit_begin(self, schema):
+    def visit_begin(self, schema: QAPISchema) -> None:
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
 
-    def _gen_type_cleanup(self, name):
+    def _gen_type_cleanup(self, name: str) -> None:
         self._genh.add(gen_type_cleanup_decl(name))
         self._genc.add(gen_type_cleanup(name))
 
-    def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+    def visit_enum_type(self,
+                        name: str,
+                        info: Optional[QAPISourceInfo],
+                        ifcond: List[str],
+                        features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.preamble_add(gen_enum(name, members, prefix))
             self._genc.add(gen_enum_lookup(name, members, prefix))
 
-    def visit_array_type(self, name, info, ifcond, element_type):
+    def visit_array_type(self,
+                         name: str,
+                         info: Optional[QAPISourceInfo],
+                         ifcond: List[str],
+                         element_type: QAPISchemaType) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.preamble_add(gen_fwd_object_or_array(name))
             self._genh.add(gen_array(name, element_type))
             self._gen_type_cleanup(name)
 
-    def visit_object_type(self, name, info, ifcond, features,
-                          base, members, variants):
+    def visit_object_type(self,
+                          name: str,
+                          info: Optional[QAPISourceInfo],
+                          ifcond: List[str],
+                          features: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> None:
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
@@ -313,7 +347,12 @@ def visit_object_type(self, name, info, ifcond, features,
                 # implicit types won't be directly allocated/freed
                 self._gen_type_cleanup(name)
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self,
+                             name: str,
+                             info: QAPISourceInfo,
+                             ifcond: List[str],
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants) -> None:
         with ifcontext(ifcond, self._genh):
             self._genh.preamble_add(gen_fwd_object_or_array(name))
         self._genh.add(gen_object(name, ifcond, None,
@@ -322,7 +361,10 @@ def visit_alternate_type(self, name, info, ifcond, features, variants):
             self._gen_type_cleanup(name)
 
 
-def gen_types(schema, output_dir, prefix, opt_builtins):
+def gen_types(schema: QAPISchema,
+              output_dir: str,
+              prefix: str,
+              opt_builtins: bool) -> None:
     vis = QAPISchemaGenTypeVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir, opt_builtins)
-- 
2.26.2

[PATCH v5 33/36] qapi/types.py: remove one-letter variables

[John Snow]

"John, if pylint told you to jump off a bridge, would you?"
Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/pylintrc |  1 -
 scripts/qapi/types.py | 29 +++++++++++++++--------------
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 8badcb11cda..b3c4cf46dbf 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                types.py,
                 visit.py,
 
 
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 766822feb3a..9d1e79d503d 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
     .array = (const char *const[]) {
 ''',
                 c_name=c_name(name))
-    for m in members:
-        ret += gen_if(m.ifcond)
-        index = c_enum_const(name, m.name, prefix)
+    for member in members:
+        ret += gen_if(member.ifcond)
+        index = c_enum_const(name, member.name, prefix)
         ret += mcgen('''
         [%(index)s] = "%(name)s",
 ''',
-                     index=index, name=m.name)
-        ret += gen_endif(m.ifcond)
+                     index=index, name=member.name)
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
     },
@@ -79,13 +79,13 @@ def gen_enum(name: str,
 ''',
                 c_name=c_name(name))
 
-    for m in enum_members:
-        ret += gen_if(m.ifcond)
+    for member in enum_members:
+        ret += gen_if(member.ifcond)
         ret += mcgen('''
     %(c_enum)s,
 ''',
-                     c_enum=c_enum_const(name, m.name, prefix))
-        ret += gen_endif(m.ifcond)
+                     c_enum=c_enum_const(name, member.name, prefix))
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
 } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
     objects_seen.add(name)
 
     ret = ''
-    if variants:
-        for v in variants.variants:
-            if isinstance(v.type, QAPISchemaObjectType):
-                ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
-                                  v.type.local_members, v.type.variants)
+    for variant in variants.variants if variants else ():
+        obj = variant.type
+        if not isinstance(obj, QAPISchemaObjectType):
+            continue
+        ret += gen_object(obj.name, obj.ifcond, obj.base,
+                          obj.local_members, obj.variants)
 
     ret += mcgen('''
 
-- 
2.26.2

[PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

[John Snow]

This is true by design, but not presently able to be expressed in the
type system. An assertion helps mypy understand our constraints.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/visit.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 14f30c228b7..4f11fd325b8 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -22,7 +22,7 @@
     mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import QAPISchemaEnumType, QAPISchemaObjectType
 
 
 def gen_visit_decl(name, scalar=False):
@@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
         ret += gen_endif(memb.ifcond)
 
     if variants:
+        tag_member = variants.tag_member
+        assert isinstance(tag_member.type, QAPISchemaEnumType)
+
         ret += mcgen('''
     switch (obj->%(c_name)s) {
 ''',
-                     c_name=c_name(variants.tag_member.name))
+                     c_name=c_name(tag_member.name))
 
         for var in variants.variants:
-            case_str = c_enum_const(variants.tag_member.type.name,
-                                    var.name,
-                                    variants.tag_member.type.prefix)
+            case_str = c_enum_const(tag_member.type.name, var.name,
+                                    tag_member.type.prefix)
             ret += gen_if(var.ifcond)
             if var.type.name == 'q_empty':
                 # valid variant and nothing to do
-- 
2.26.2

[PATCH v5 35/36] qapi/visit.py: remove unused parameters from gen_visit_object

[John Snow]

And this fixes the pylint report for this file, so make sure we check
this in the future, too.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/pylintrc | 1 -
 scripts/qapi/visit.py | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index b3c4cf46dbf..b9e077a1642 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                visit.py,
 
 
 [MESSAGES CONTROL]
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 4f11fd325b8..e54694e23db 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -250,7 +250,7 @@ def gen_visit_alternate(name, variants):
     return ret
 
 
-def gen_visit_object(name, base, members, variants):
+def gen_visit_object(name):
     return mcgen('''
 
 bool visit_type_%(c_name)s
@@ -343,7 +343,7 @@ def visit_object_type(self, name, info, ifcond, features,
             if not name.startswith('q_'):
                 # only explicit types need an allocating visit
                 self._genh.add(gen_visit_decl(name))
-                self._genc.add(gen_visit_object(name, base, members, variants))
+                self._genc.add(gen_visit_object(name))
 
     def visit_alternate_type(self, name, info, ifcond, features, variants):
         with ifcontext(ifcond, self._genh, self._genc):
-- 
2.26.2

[PATCH v5 36/36] qapi/visit.py: add type hint annotations

[John Snow]

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/visit.py | 73 +++++++++++++++++++++++++++++++++----------
 2 files changed, 56 insertions(+), 22 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 83f19813553..74fc6c82153 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -28,8 +28,3 @@ check_untyped_defs = False
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
-
-[mypy-qapi.visit]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index e54694e23db..14d4f0b261f 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,6 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List, Optional
+
 from .common import (
     c_enum_const,
     c_name,
@@ -22,10 +24,20 @@
     mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumType, QAPISchemaObjectType
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaEnumType,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaType,
+    QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
 
 
-def gen_visit_decl(name, scalar=False):
+def gen_visit_decl(name: str, scalar: bool = False) -> str:
     c_type = c_name(name) + ' *'
     if not scalar:
         c_type += '*'
@@ -37,7 +49,7 @@ def gen_visit_decl(name, scalar=False):
                  c_name=c_name(name), c_type=c_type)
 
 
-def gen_visit_members_decl(name):
+def gen_visit_members_decl(name: str) -> str:
     return mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp);
@@ -45,7 +57,10 @@ def gen_visit_members_decl(name):
                  c_name=c_name(name))
 
 
-def gen_visit_object_members(name, base, members, variants):
+def gen_visit_object_members(name: str,
+                             base: Optional[QAPISchemaObjectType],
+                             members: List[QAPISchemaObjectTypeMember],
+                             variants: Optional[QAPISchemaVariants]) -> str:
     ret = mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
@@ -125,7 +140,7 @@ def gen_visit_object_members(name, base, members, variants):
     return ret
 
 
-def gen_visit_list(name, element_type):
+def gen_visit_list(name: str, element_type: QAPISchemaType) -> str:
     return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -159,7 +174,7 @@ def gen_visit_list(name, element_type):
                  c_name=c_name(name), c_elt_type=element_type.c_name())
 
 
-def gen_visit_enum(name):
+def gen_visit_enum(name: str) -> str:
     return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -174,7 +189,7 @@ def gen_visit_enum(name):
                  c_name=c_name(name))
 
 
-def gen_visit_alternate(name, variants):
+def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
     ret = mcgen('''
 
 bool visit_type_%(c_name)s
@@ -250,7 +265,7 @@ def gen_visit_alternate(name, variants):
     return ret
 
 
-def gen_visit_object(name):
+def gen_visit_object(name: str) -> str:
     return mcgen('''
 
 bool visit_type_%(c_name)s
@@ -285,12 +300,12 @@ def gen_visit_object(name):
 
 class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-visit', ' * Schema-defined QAPI visitors',
             ' * Built-in QAPI visitors', __doc__)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/error.h"
@@ -302,7 +317,7 @@ def _begin_system_module(self, name):
 
 '''))
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
         self._genc.preamble_add(mcgen('''
@@ -319,18 +334,34 @@ def _begin_user_module(self, name):
 ''',
                                       types=types))
 
-    def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+    def visit_enum_type(self,
+                        name: str,
+                        info: QAPISourceInfo,
+                        ifcond: List[str],
+                        features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name, scalar=True))
             self._genc.add(gen_visit_enum(name))
 
-    def visit_array_type(self, name, info, ifcond, element_type):
+    def visit_array_type(self,
+                         name: str,
+                         info: Optional[QAPISourceInfo],
+                         ifcond: List[str],
+                         element_type: QAPISchemaType) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name))
             self._genc.add(gen_visit_list(name, element_type))
 
-    def visit_object_type(self, name, info, ifcond, features,
-                          base, members, variants):
+    def visit_object_type(self,
+                          name: str,
+                          info: Optional[QAPISourceInfo],
+                          ifcond: List[str],
+                          features: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> None:
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
@@ -345,13 +376,21 @@ def visit_object_type(self, name, info, ifcond, features,
                 self._genh.add(gen_visit_decl(name))
                 self._genc.add(gen_visit_object(name))
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self,
+                             name: str,
+                             info: QAPISourceInfo,
+                             ifcond: List[str],
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name))
             self._genc.add(gen_visit_alternate(name, variants))
 
 
-def gen_visit(schema, output_dir, prefix, opt_builtins):
+def gen_visit(schema: QAPISchema,
+              output_dir: str,
+              prefix: str,
+              opt_builtins: bool) -> None:
     vis = QAPISchemaGenVisitVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir, opt_builtins)
-- 
2.26.2

Re: [PATCH v5 00/36] qapi: static typing conversion, pt1

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 1951 bytes --]

On Mon, Oct 05, 2020 at 03:51:22PM -0400, John Snow wrote:
> Hi, this series adds static type hints to the QAPI module.
> This is part one!
> 
> Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
> Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6
> 
> - Requires Python 3.6+
> - Requires mypy 0.770 or newer (for type analysis only)
> - Requires pylint 2.6.0 or newer (for lint checking only)
> 
> In general, this series tackles the cleanup of one individual QAPI
> module at a time. Once it passes pylint or mypy checks, those checks are
> enabled for that file.
> 
> Type hints are added in patches that add *only* type hints and change no
> other behavior. Any necessary changes to behavior to accommodate typing
> are split out into their own tiny patches.
> 
> Notes:
> 
> - After patch 07, `isort -c` should pass 100% on this and every
>   future commit.
>

FIY, I went on to replicate/validate your testing, and I ran on
patches 07 an later:

  isort --check-only --settings-path ./scripts/qapi/.isort.cfg --recursive ./scripts/qapi

With success.

> - After patch 08, `flake8 qapi/` should pass 100% on this and every
>   future commit.
>

Here on patches 08 an later, I've run:

  flake8 ./scripts/qapi

And starting on patch 24 ("qapi/source.py: add type hint annotations")
it complained about:

  /scripts/qapi/source.py:44:31: F821 undefined name 'T'

> - After patch 09, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
>   on this and every future commit.
>

Here I ran on patches 09 and later:

  pylint --rcfile=./scripts/qapi/pylintrc ./scripts/qapi/

And all succeeded.

> - After patch 18, `mypy --config-file=qapi/mypy.ini qapi/` should pass
>   100% on this and every future commit.
>

Here I ran on patches 18 and later:

  mypy --config-file=./scripts/qapi/mypy.ini ./scripts/qapi/

And all succeeded.

- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 00/36] qapi: static typing conversion, pt1

[John Snow]

On 10/5/20 7:05 PM, Cleber Rosa wrote:
> Here on patches 08 an later, I've run:
> 
>    flake8 ./scripts/qapi
> 
> And starting on patch 24 ("qapi/source.py: add type hint annotations")
> it complained about:
> 
>    /scripts/qapi/source.py:44:31: F821 undefined name 'T'

I think that must be flake8 < 3.8.0. Try using >= 3.8.0.

(Yes, still working on getting proper pipenv/requirements/something up 
for these, sorry.)

Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> A precise style guide and a package-wide overhaul is forthcoming pending
> further discussion and consensus. At present, we are avoiding obvious
> errors that cause sphinx documentation build problems.
>
> A preliminary style guide is loosely based on PEP 257 and Sphinx
> Autodoc. It is chosen for interoperability with our existing Sphinx
> framework, and because it has loose recognition in the Pycharm IDE.
>
> - Use Triple-double quotes (""").
> - Opening and closing quotes appear on their own lines for multi-line docs.
> - A single-sentence summary should be the first line of the docstring.
> - A blank line follows.
> - Further prose, if needed, is written next and can be multiple paragraphs,
>   contain RST markup, etc.
> - The :param x: desc, :returns: desc, and :raises z: desc info fields follow.

Mandatory when they apply?

> - Additional examples, diagrams, or other metadata follows below.
>
> See also:
>
> https://www.python.org/dev/peps/pep-0257/
> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/gen.py    | 6 ++++--
>  scripts/qapi/parser.py | 1 +
>  2 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index ca66c82b5b8..dc7b94aa115 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -154,9 +154,11 @@ def _bottom(self):
>  
>  @contextmanager
>  def ifcontext(ifcond, *args):
> -    """A 'with' statement context manager to wrap with start_if()/end_if()
> +    """
> +    A with-statement context manager that wraps with `start_if()` / `end_if()`.
>  
> -    *args: any number of QAPIGenCCode
> +    :param ifcond: A list of conditionals, passed to `start_if()`.
> +    :param args: any number of `QAPIGenCCode`.
>  
>      Example::
>  
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index 9d1a3e2eea9..31bc2e6dca9 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -381,6 +381,7 @@ def append(self, line):
>  
>          The way that the line is dealt with depends on which part of
>          the documentation we're parsing right now:
> +
>          * The body section: ._append_line is ._append_body_line
>          * An argument section: ._append_line is ._append_args_line
>          * A features section: ._append_line is ._append_features_line

I'm asking because you're not adding :param line: here.

Same for several other functions in this file.

In schema.py:

    class QAPISchemaMember:
        """ Represents object members, enum members and features """

Are the spaces next to """ okay?

Re: [PATCH v5 05/36] qapi: Prefer explicit relative imports

[Philippe Mathieu-Daudé]

On 10/5/20 9:51 PM, John Snow wrote:
> All of the QAPI include statements are changed to be package-aware, as
> explicit relative imports.
> 
> A quirk of Python packages is that the name of the package exists only
> *outside* of the package. This means that to a module inside of the qapi
> folder, there is inherently no such thing as the "qapi" package. The
> reason these imports work is because the "qapi" package exists in the
> context of the caller -- the execution shim, where sys.path includes a
> directory that has a 'qapi' folder in it.
> 
> When we write "from qapi import sibling", we are NOT referencing the folder
> 'qapi', but rather "any package named qapi in sys.path". If you should
> so happen to have a 'qapi' package in your path, it will use *that*
> package.
> 
> When we write "from .sibling import foo", we always reference explicitly
> our sibling module; guaranteeing consistency in *where* we are importing
> these modules from.
> 
> This can be useful when working with virtual environments and packages
> in development mode. In development mode, a package is installed as a
> series of symlinks that forwards to your same source files. The problem
> arises because code quality checkers will follow "import qapi.x" to the
> "installed" version instead of the sibling file and -- even though they
> are the same file -- they have different module paths, and this causes
> cyclic import problems, false positive type mismatch errors, and more.
> 
> It can also be useful when dealing with hierarchical packages, e.g. if
> we allow qemu.core.qmp, qemu.qapi.parser, etc.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/commands.py   |  4 ++--
>  scripts/qapi/events.py     |  8 ++++----
>  scripts/qapi/expr.py       |  4 ++--
>  scripts/qapi/gen.py        |  4 ++--
>  scripts/qapi/introspect.py |  8 ++++----
>  scripts/qapi/main.py       | 14 +++++++-------
>  scripts/qapi/parser.py     |  4 ++--
>  scripts/qapi/schema.py     |  8 ++++----
>  scripts/qapi/types.py      |  6 +++---
>  scripts/qapi/visit.py      |  6 +++---
>  10 files changed, 33 insertions(+), 33 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH v5 06/36] qapi: Remove wildcard includes

[Philippe Mathieu-Daudé]

On 10/5/20 9:51 PM, John Snow wrote:
> Wildcard includes become hard to manage when refactoring and dealing
> with circular dependencies with strictly typed mypy.
> 
> flake8 also flags each one as a warning, as it is not smart enough to
> know which names exist in the imported file.
> 
> Remove them and include things explicitly by name instead.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/commands.py   |  2 +-
>  scripts/qapi/events.py     |  7 ++++++-
>  scripts/qapi/gen.py        | 12 +++++++++---
>  scripts/qapi/introspect.py |  7 ++++++-
>  scripts/qapi/types.py      |  8 +++++++-
>  scripts/qapi/visit.py      | 10 +++++++++-
>  6 files changed, 38 insertions(+), 8 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH v5 13/36] qapi/common.py: Replace one-letter 'c' variable

[Philippe Mathieu-Daudé]

On 10/5/20 9:51 PM, John Snow wrote:
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  scripts/qapi/common.py | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index a417b6029c8..338adedef4f 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -30,14 +30,14 @@ def camel_to_upper(value):
>      new_name = ''
>      length = len(c_fun_str)
>      for i in range(length):
> -        c = c_fun_str[i]
> -        # When c is upper and no '_' appears before, do more checks
> -        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
> +        char = c_fun_str[i]
> +        # When char is upper case and no '_' appears before, do more checks
> +        if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
>              if i < length - 1 and c_fun_str[i + 1].islower():
>                  new_name += '_'
>              elif c_fun_str[i - 1].isdigit():
>                  new_name += '_'
> -        new_name += c
> +        new_name += char
>      return new_name.lstrip('_').upper()

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH v5 35/36] qapi/visit.py: remove unused parameters from gen_visit_object

[Philippe Mathieu-Daudé]

On 10/5/20 9:51 PM, John Snow wrote:
> And this fixes the pylint report for this file, so make sure we check
> this in the future, too.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/pylintrc | 1 -
>  scripts/qapi/visit.py | 4 ++--
>  2 files changed, 2 insertions(+), 3 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module

[Philippe Mathieu-Daudé]

On 10/5/20 9:51 PM, John Snow wrote:
> The edge case is that if the name is '', this expression returns a
> string instead of a bool, which violates our declared type.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  scripts/qapi/gen.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This is a minor re-work of the entrypoint script. It isolates a
> generate() method from the actual command-line mechanism.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>  1 file changed, 62 insertions(+), 23 deletions(-)
>
> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
> index 541e8c1f55d..117b396a595 100644
> --- a/scripts/qapi-gen.py
> +++ b/scripts/qapi-gen.py
> @@ -1,30 +1,77 @@
>  #!/usr/bin/env python3
> -# QAPI generator
> -#
> +
>  # This work is licensed under the terms of the GNU GPL, version 2 or later.
>  # See the COPYING file in the top-level directory.
>  
> +"""
> +QAPI Generator
> +
> +This script is the main entry point for generating C code from the QAPI schema.
> +"""
>  
>  import argparse
>  import re
>  import sys
>  
>  from qapi.commands import gen_commands
> +from qapi.error import QAPIError
>  from qapi.events import gen_events
>  from qapi.introspect import gen_introspect
> -from qapi.schema import QAPIError, QAPISchema
> +from qapi.schema import QAPISchema
>  from qapi.types import gen_types
>  from qapi.visit import gen_visit

Unrelated cleanup.  Okay.

>  
>  
> -def main(argv):
> +DEFAULT_OUTPUT_DIR = ''
> +DEFAULT_PREFIX = ''
> +
> +
> +def generate(schema_file: str,
> +             output_dir: str,
> +             prefix: str,
> +             unmask: bool = False,
> +             builtins: bool = False) -> None:
> +    """
> +    generate uses a given schema to produce C code in the target directory.
> +
> +    :param schema_file: The primary QAPI schema file.
> +    :param output_dir: The output directory to store generated code.
> +    :param prefix: Optional C-code prefix for symbol names.
> +    :param unmask: Expose non-ABI names through introspection?
> +    :param builtins: Generate code for built-in types?
> +
> +    :raise QAPIError: On failures.
> +    """
> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
> +    if match.end() != len(prefix):
> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
> +            prefix[match.end()], prefix)
> +        raise QAPIError('', None, msg)

Uh...

    $ python3 scripts/qapi-gen.py --prefix=@ x
    scripts/qapi-gen.py: : funny character '@' in prefix '@'

Unwanted " :".

This is due to a hack: you pass '' for info (*quack*).  Everything else
passes QAPISourceInfo (I believe).

Is it really a good idea to do this in generate?  It's not about
generating code, it's about validating a CLI option.

> +
> +    schema = QAPISchema(schema_file)
> +    gen_types(schema, output_dir, prefix, builtins)
> +    gen_visit(schema, output_dir, prefix, builtins)
> +    gen_commands(schema, output_dir, prefix)
> +    gen_events(schema, output_dir, prefix)
> +    gen_introspect(schema, output_dir, prefix, unmask)
> +
> +
> +def main() -> int:
> +    """
> +    gapi-gen shell script entrypoint.
> +    Expects arguments via sys.argv, see --help for details.
> +
> +    :return: int, 0 on success, 1 on failure.
> +    """
>      parser = argparse.ArgumentParser(
>          description='Generate code from a QAPI schema')
>      parser.add_argument('-b', '--builtins', action='store_true',
>                          help="generate code for built-in types")
> -    parser.add_argument('-o', '--output-dir', action='store', default='',
> +    parser.add_argument('-o', '--output-dir', action='store',
> +                        default=DEFAULT_OUTPUT_DIR,
>                          help="write output to directory OUTPUT_DIR")
> -    parser.add_argument('-p', '--prefix', action='store', default='',
> +    parser.add_argument('-p', '--prefix', action='store',
> +                        default=DEFAULT_PREFIX,
>                          help="prefix for symbols")

I don't like the changes to default=, because:

1. They are only losely related to the patch's purpose.

2. They split the definition of the CLI: most of it is here, except for
defaults, which are defined elsewhere.

3. The defaults will not change, and nothing else uses the constants.

>      parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>                          dest='unmask',
> @@ -32,25 +79,17 @@ def main(argv):
>      parser.add_argument('schema', action='store')
>      args = parser.parse_args()
>  
> -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
> -    if match.end() != len(args.prefix):
> -        print("%s: 'funny character '%s' in argument of --prefix"
> -              % (sys.argv[0], args.prefix[match.end()]),
> -              file=sys.stderr)
> -        sys.exit(1)
> -
>      try:
> -        schema = QAPISchema(args.schema)
> +        generate(args.schema,
> +                 output_dir=args.output_dir,
> +                 prefix=args.prefix,
> +                 unmask=args.unmask,
> +                 builtins=args.builtins)
>      except QAPIError as err:
> -        print(err, file=sys.stderr)
> -        exit(1)
> -
> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
> -    gen_commands(schema, args.output_dir, args.prefix)
> -    gen_events(schema, args.output_dir, args.prefix)
> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
> +        return 1
> +    return 0

Subtle change: you move the gen_FOO() into the try ... except.  Okay;
they don't raise QAPIError, but perhaps worth a mention in the commit
message.

>  
>  
>  if __name__ == '__main__':
> -    main(sys.argv)
> +    sys.exit(main())

"Python was designed to be easy to understand and fun to use."
Ha ha ha.

Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

[John Snow]

On 10/6/20 7:21 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> A precise style guide and a package-wide overhaul is forthcoming pending
>> further discussion and consensus. At present, we are avoiding obvious
>> errors that cause sphinx documentation build problems.
>>
>> A preliminary style guide is loosely based on PEP 257 and Sphinx
>> Autodoc. It is chosen for interoperability with our existing Sphinx
>> framework, and because it has loose recognition in the Pycharm IDE.
>>
>> - Use Triple-double quotes (""").
>> - Opening and closing quotes appear on their own lines for multi-line docs.
>> - A single-sentence summary should be the first line of the docstring.
>> - A blank line follows.
>> - Further prose, if needed, is written next and can be multiple paragraphs,
>>    contain RST markup, etc.
>> - The :param x: desc, :returns: desc, and :raises z: desc info fields follow.
> 
> Mandatory when they apply?
> 

Subject of debate...

- Some people really hate obvious docstring comments.
- Some people really like the consistency.

Which type of developer am I? Guess it depends on when you ask.

Figured we'd hash that out when I go to write a style guide document.

>> - Additional examples, diagrams, or other metadata follows below.
>>
>> See also:
>>
>> https://www.python.org/dev/peps/pep-0257/
>> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/gen.py    | 6 ++++--
>>   scripts/qapi/parser.py | 1 +
>>   2 files changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> index ca66c82b5b8..dc7b94aa115 100644
>> --- a/scripts/qapi/gen.py
>> +++ b/scripts/qapi/gen.py
>> @@ -154,9 +154,11 @@ def _bottom(self):
>>   
>>   @contextmanager
>>   def ifcontext(ifcond, *args):
>> -    """A 'with' statement context manager to wrap with start_if()/end_if()
>> +    """
>> +    A with-statement context manager that wraps with `start_if()` / `end_if()`.
>>   
>> -    *args: any number of QAPIGenCCode
>> +    :param ifcond: A list of conditionals, passed to `start_if()`.
>> +    :param args: any number of `QAPIGenCCode`.
>>   
>>       Example::
>>   
>> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>> index 9d1a3e2eea9..31bc2e6dca9 100644
>> --- a/scripts/qapi/parser.py
>> +++ b/scripts/qapi/parser.py
>> @@ -381,6 +381,7 @@ def append(self, line):
>>   
>>           The way that the line is dealt with depends on which part of
>>           the documentation we're parsing right now:
>> +
>>           * The body section: ._append_line is ._append_body_line
>>           * An argument section: ._append_line is ._append_args_line
>>           * A features section: ._append_line is ._append_features_line
> 
> I'm asking because you're not adding :param line: here.
> 

Yeah, it's not necessary to test the syntax of what else I've written 
with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't 
required at all and could be deleted. You can do so if you'd like; it 
will just show up later in some other patch or series more designed to 
fix formatting.

> Same for several other functions in this file.
> 
> In schema.py:
> 
>      class QAPISchemaMember:
>          """ Represents object members, enum members and features """
> 
> Are the spaces next to """ okay?
> 

Ideally cleaned up, but that's not a goal of this patch or series.

--js

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/6/20 7:51 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> This is a minor re-work of the entrypoint script. It isolates a
>> generate() method from the actual command-line mechanism.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> Tested-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>   1 file changed, 62 insertions(+), 23 deletions(-)
>>
>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>> index 541e8c1f55d..117b396a595 100644
>> --- a/scripts/qapi-gen.py
>> +++ b/scripts/qapi-gen.py
>> @@ -1,30 +1,77 @@
>>   #!/usr/bin/env python3
>> -# QAPI generator
>> -#
>> +
>>   # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>   # See the COPYING file in the top-level directory.
>>   
>> +"""
>> +QAPI Generator
>> +
>> +This script is the main entry point for generating C code from the QAPI schema.
>> +"""
>>   
>>   import argparse
>>   import re
>>   import sys
>>   
>>   from qapi.commands import gen_commands
>> +from qapi.error import QAPIError
>>   from qapi.events import gen_events
>>   from qapi.introspect import gen_introspect
>> -from qapi.schema import QAPIError, QAPISchema
>> +from qapi.schema import QAPISchema
>>   from qapi.types import gen_types
>>   from qapi.visit import gen_visit
> 
> Unrelated cleanup.  Okay.
> 
>>   
>>   
>> -def main(argv):
>> +DEFAULT_OUTPUT_DIR = ''
>> +DEFAULT_PREFIX = ''
>> +
>> +
>> +def generate(schema_file: str,
>> +             output_dir: str,
>> +             prefix: str,
>> +             unmask: bool = False,
>> +             builtins: bool = False) -> None:
>> +    """
>> +    generate uses a given schema to produce C code in the target directory.
>> +
>> +    :param schema_file: The primary QAPI schema file.
>> +    :param output_dir: The output directory to store generated code.
>> +    :param prefix: Optional C-code prefix for symbol names.
>> +    :param unmask: Expose non-ABI names through introspection?
>> +    :param builtins: Generate code for built-in types?
>> +
>> +    :raise QAPIError: On failures.
>> +    """
>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>> +    if match.end() != len(prefix):
>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>> +            prefix[match.end()], prefix)
>> +        raise QAPIError('', None, msg)
> 
> Uh...
> 
>      $ python3 scripts/qapi-gen.py --prefix=@ x
>      scripts/qapi-gen.py: : funny character '@' in prefix '@'
> 
> Unwanted " :".
> 
> This is due to a hack: you pass '' for info (*quack*).  Everything else
> passes QAPISourceInfo (I believe).
>

Quack indeed - why does our base error class require so much information 
from a specific part of the generation process?

Ah, someone changes this in part 4 so that we have a more generic error 
class to use as a base when we are missing such information.

You are witnessing some more future-bleed.
> Is it really a good idea to do this in generate?  It's not about
> generating code, it's about validating a CLI option.
> 

One might also ask: Is it a good idea to only validate this on a 
frontend, and not in the implementation?

The idea here was to create a function that could be used in a script 
(for tests, debugging interfaces, other python packages) to do all of 
the same things that the CLI tool did, just sans the actual CLI.

Wouldn't make sense to allow garbage to flow in from one interface but 
not the other; so the check is here.

>> +
>> +    schema = QAPISchema(schema_file)
>> +    gen_types(schema, output_dir, prefix, builtins)
>> +    gen_visit(schema, output_dir, prefix, builtins)
>> +    gen_commands(schema, output_dir, prefix)
>> +    gen_events(schema, output_dir, prefix)
>> +    gen_introspect(schema, output_dir, prefix, unmask)
>> +
>> +
>> +def main() -> int:
>> +    """
>> +    gapi-gen shell script entrypoint.
>> +    Expects arguments via sys.argv, see --help for details.
>> +
>> +    :return: int, 0 on success, 1 on failure.
>> +    """
>>       parser = argparse.ArgumentParser(
>>           description='Generate code from a QAPI schema')
>>       parser.add_argument('-b', '--builtins', action='store_true',
>>                           help="generate code for built-in types")
>> -    parser.add_argument('-o', '--output-dir', action='store', default='',
>> +    parser.add_argument('-o', '--output-dir', action='store',
>> +                        default=DEFAULT_OUTPUT_DIR,
>>                           help="write output to directory OUTPUT_DIR")
>> -    parser.add_argument('-p', '--prefix', action='store', default='',
>> +    parser.add_argument('-p', '--prefix', action='store',
>> +                        default=DEFAULT_PREFIX,
>>                           help="prefix for symbols")
> 
> I don't like the changes to default=, because:
> 
> 1. They are only losely related to the patch's purpose.
> 

Subjective, but OK.

> 2. They split the definition of the CLI: most of it is here, except for
> defaults, which are defined elsewhere.
> 

All of it is in main.py, though! If you were to, say, move generate() 
elsewhere, it'd look pretty compact as just the CLI frontend, no?

> 3. The defaults will not change, and nothing else uses the constants.
> 

But, fine. Cleber had the same comment but I wasn't fully on-board, but 
two folks saying the same thing ...

>>       parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>>                           dest='unmask',
>> @@ -32,25 +79,17 @@ def main(argv):
>>       parser.add_argument('schema', action='store')
>>       args = parser.parse_args()
>>   
>> -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
>> -    if match.end() != len(args.prefix):
>> -        print("%s: 'funny character '%s' in argument of --prefix"
>> -              % (sys.argv[0], args.prefix[match.end()]),
>> -              file=sys.stderr)
>> -        sys.exit(1)
>> -
>>       try:
>> -        schema = QAPISchema(args.schema)
>> +        generate(args.schema,
>> +                 output_dir=args.output_dir,
>> +                 prefix=args.prefix,
>> +                 unmask=args.unmask,
>> +                 builtins=args.builtins)
>>       except QAPIError as err:
>> -        print(err, file=sys.stderr)
>> -        exit(1)
>> -
>> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
>> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
>> -    gen_commands(schema, args.output_dir, args.prefix)
>> -    gen_events(schema, args.output_dir, args.prefix)
>> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
>> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
>> +        return 1
>> +    return 0
> 
> Subtle change: you move the gen_FOO() into the try ... except.  Okay;
> they don't raise QAPIError, but perhaps worth a mention in the commit
> message.
> 

Forbidden future knowledge; I intend them to.

>>   
>>   
>>   if __name__ == '__main__':
>> -    main(sys.argv)
>> +    sys.exit(main())
> 
> "Python was designed to be easy to understand and fun to use."
> Ha ha ha.
> 

I mean, I'm having fun, aren't you?

--js

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/6/20 7:51 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> This is a minor re-work of the entrypoint script. It isolates a
>> generate() method from the actual command-line mechanism.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> Tested-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>   1 file changed, 62 insertions(+), 23 deletions(-)
>>
>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>> index 541e8c1f55d..117b396a595 100644
>> --- a/scripts/qapi-gen.py
>> +++ b/scripts/qapi-gen.py
>> @@ -1,30 +1,77 @@
>>   #!/usr/bin/env python3
>> -# QAPI generator
>> -#
>> +
>>   # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>   # See the COPYING file in the top-level directory.
>>   
>> +"""
>> +QAPI Generator
>> +
>> +This script is the main entry point for generating C code from the QAPI schema.
>> +"""
>>   
>>   import argparse
>>   import re
>>   import sys
>>   
>>   from qapi.commands import gen_commands
>> +from qapi.error import QAPIError
>>   from qapi.events import gen_events
>>   from qapi.introspect import gen_introspect
>> -from qapi.schema import QAPIError, QAPISchema
>> +from qapi.schema import QAPISchema
>>   from qapi.types import gen_types
>>   from qapi.visit import gen_visit
> 
> Unrelated cleanup.  Okay.
> 
>>   
>>   
>> -def main(argv):
>> +DEFAULT_OUTPUT_DIR = ''
>> +DEFAULT_PREFIX = ''
>> +
>> +
>> +def generate(schema_file: str,
>> +             output_dir: str,
>> +             prefix: str,
>> +             unmask: bool = False,
>> +             builtins: bool = False) -> None:
>> +    """
>> +    generate uses a given schema to produce C code in the target directory.
>> +
>> +    :param schema_file: The primary QAPI schema file.
>> +    :param output_dir: The output directory to store generated code.
>> +    :param prefix: Optional C-code prefix for symbol names.
>> +    :param unmask: Expose non-ABI names through introspection?
>> +    :param builtins: Generate code for built-in types?
>> +
>> +    :raise QAPIError: On failures.
>> +    """
>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>> +    if match.end() != len(prefix):
>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>> +            prefix[match.end()], prefix)
>> +        raise QAPIError('', None, msg)
> 
> Uh...
> 
>      $ python3 scripts/qapi-gen.py --prefix=@ x
>      scripts/qapi-gen.py: : funny character '@' in prefix '@'
> 
> Unwanted " :".
> 
> This is due to a hack: you pass '' for info (*quack*).  Everything else
> passes QAPISourceInfo (I believe).
> 
> Is it really a good idea to do this in generate?  It's not about
> generating code, it's about validating a CLI option.
> 
>> +
>> +    schema = QAPISchema(schema_file)
>> +    gen_types(schema, output_dir, prefix, builtins)
>> +    gen_visit(schema, output_dir, prefix, builtins)
>> +    gen_commands(schema, output_dir, prefix)
>> +    gen_events(schema, output_dir, prefix)
>> +    gen_introspect(schema, output_dir, prefix, unmask)
>> +
>> +
>> +def main() -> int:
>> +    """
>> +    gapi-gen shell script entrypoint.
>> +    Expects arguments via sys.argv, see --help for details.
>> +
>> +    :return: int, 0 on success, 1 on failure.
>> +    """
>>       parser = argparse.ArgumentParser(
>>           description='Generate code from a QAPI schema')
>>       parser.add_argument('-b', '--builtins', action='store_true',
>>                           help="generate code for built-in types")
>> -    parser.add_argument('-o', '--output-dir', action='store', default='',
>> +    parser.add_argument('-o', '--output-dir', action='store',
>> +                        default=DEFAULT_OUTPUT_DIR,
>>                           help="write output to directory OUTPUT_DIR")
>> -    parser.add_argument('-p', '--prefix', action='store', default='',
>> +    parser.add_argument('-p', '--prefix', action='store',
>> +                        default=DEFAULT_PREFIX,
>>                           help="prefix for symbols")
> 
> I don't like the changes to default=, because:
> 
> 1. They are only losely related to the patch's purpose.
> 
> 2. They split the definition of the CLI: most of it is here, except for
> defaults, which are defined elsewhere.
> 
> 3. The defaults will not change, and nothing else uses the constants.
> 
>>       parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>>                           dest='unmask',
>> @@ -32,25 +79,17 @@ def main(argv):
>>       parser.add_argument('schema', action='store')
>>       args = parser.parse_args()
>>   
>> -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
>> -    if match.end() != len(args.prefix):
>> -        print("%s: 'funny character '%s' in argument of --prefix"
>> -              % (sys.argv[0], args.prefix[match.end()]),
>> -              file=sys.stderr)
>> -        sys.exit(1)
>> -
>>       try:
>> -        schema = QAPISchema(args.schema)
>> +        generate(args.schema,
>> +                 output_dir=args.output_dir,
>> +                 prefix=args.prefix,
>> +                 unmask=args.unmask,
>> +                 builtins=args.builtins)
>>       except QAPIError as err:
>> -        print(err, file=sys.stderr)
>> -        exit(1)
>> -
>> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
>> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
>> -    gen_commands(schema, args.output_dir, args.prefix)
>> -    gen_events(schema, args.output_dir, args.prefix)
>> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
>> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
>> +        return 1
>> +    return 0
> 
> Subtle change: you move the gen_FOO() into the try ... except.  Okay;
> they don't raise QAPIError, but perhaps worth a mention in the commit
> message.
> 
>>   
>>   
>>   if __name__ == '__main__':
>> -    main(sys.argv)
>> +    sys.exit(main())
> 
> "Python was designed to be easy to understand and fun to use."
> Ha ha ha.
> 
Here's a Fixup that requires more (but obvious) Fixups to later patches 
in this series. (The next patch needs the same changes re-applied to the 
now-moved file.)

A note: I add a FIXME, but when I respin my error series, it will take 
care of this piece. This is just the quickest, dumbest way to victory in 
the interim. It is not worth engineering a better solution, because 
there is one waiting in part 4 already.

(Future knowledge: I add a QAPIError that does not define its arguments 
that inherits Exception and is used as the basis for all error classes 
in QAPI; and the contextual error that requires "info" is changed to 
inherit from QAPIError. This is useful as a "package-level" error, one 
in which that *all* errors in this package inherit from. We already 
achieve that, but we needed a more general form to serve as the root of 
that inheritance tree.)


diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 117b396a595..c62c4e93323 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -22,10 +22,6 @@
  from qapi.visit import gen_visit


-DEFAULT_OUTPUT_DIR = ''
-DEFAULT_PREFIX = ''
-
-
  def generate(schema_file: str,
               output_dir: str,
               prefix: str,
@@ -68,10 +64,10 @@ def main() -> int:
      parser.add_argument('-b', '--builtins', action='store_true',
                          help="generate code for built-in types")
      parser.add_argument('-o', '--output-dir', action='store',
-                        default=DEFAULT_OUTPUT_DIR,
+                        default='',
                          help="write output to directory OUTPUT_DIR")
      parser.add_argument('-p', '--prefix', action='store',
-                        default=DEFAULT_PREFIX,
+                        default='',
                          help="prefix for symbols")
      parser.add_argument('-u', '--unmask-non-abi-names', 
action='store_true',
                          dest='unmask',
diff --git a/scripts/qapi/error.py b/scripts/qapi/error.py
index ae60d9e2fe8..f5a818ae35f 100644
--- a/scripts/qapi/error.py
+++ b/scripts/qapi/error.py
@@ -21,6 +21,9 @@ def __init__(self, info, col, msg):

      def __str__(self):
+        # FIXME: We need a general-purpose no-context error class.
+        if not self.info:
+            return self.msg
          loc = str(self.info)
          if self.col is not None:
              assert self.info.line is not None
              loc += ':%s' % self.col

Re: [PATCH v5 00/36] qapi: static typing conversion, pt1

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

On Mon, Oct 05, 2020 at 07:57:18PM -0400, John Snow wrote:
> On 10/5/20 7:05 PM, Cleber Rosa wrote:
> > Here on patches 08 an later, I've run:
> > 
> >    flake8 ./scripts/qapi
> > 
> > And starting on patch 24 ("qapi/source.py: add type hint annotations")
> > it complained about:
> > 
> >    /scripts/qapi/source.py:44:31: F821 undefined name 'T'
> 
> I think that must be flake8 < 3.8.0. Try using >= 3.8.0.
>

ACK, I was on 3.7.9.  They're all passing now with 3.8.4.

- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/6/20 7:21 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> A precise style guide and a package-wide overhaul is forthcoming pending
>>> further discussion and consensus. At present, we are avoiding obvious
>>> errors that cause sphinx documentation build problems.
>>>
>>> A preliminary style guide is loosely based on PEP 257 and Sphinx
>>> Autodoc. It is chosen for interoperability with our existing Sphinx
>>> framework, and because it has loose recognition in the Pycharm IDE.
>>>
>>> - Use Triple-double quotes (""").
>>> - Opening and closing quotes appear on their own lines for multi-line docs.
>>> - A single-sentence summary should be the first line of the docstring.
>>> - A blank line follows.
>>> - Further prose, if needed, is written next and can be multiple paragraphs,
>>>    contain RST markup, etc.
>>> - The :param x: desc, :returns: desc, and :raises z: desc info fields follow.
>> Mandatory when they apply?
>> 
>
> Subject of debate...
>
> - Some people really hate obvious docstring comments.
> - Some people really like the consistency.
>
> Which type of developer am I? Guess it depends on when you ask.
>
> Figured we'd hash that out when I go to write a style guide document.

Fair enough.

If I stop reading after the first paragraph, the patch matches
expectations built by the commit message.

If I speed-read, the first paragraph barely registers, but the second
makes me slow down, giving me the mistaken idea that this patch is about
converting to a preliminary style guide.  It's not, it's about getting
Sphinx errors out of the way.

I figure you didn't stop after the first paragraph because you felt a
need to explain why you resolve the "obvious errors" the way you do.

Perhaps:

    qapi: modify docstrings to be sphinx-compatible

    A precise style guide and a package-wide overhaul is forthcoming
    pending further discussion and consensus. For now, merely avoid
    obvious errors that cause Sphinx documentation build problems, using a
    style loosely based on PEP 257 and Sphinx Autodoc. It is chosen for
    interoperability with our existing Sphinx framework, and because it
    has loose recognition in the Pycharm IDE.

    [...]
   

>>> - Additional examples, diagrams, or other metadata follows below.
>>>
>>> See also:
>>>
>>> https://www.python.org/dev/peps/pep-0257/
>>> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Blank line here, by convention.

>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>   scripts/qapi/gen.py    | 6 ++++--
>>>   scripts/qapi/parser.py | 1 +
>>>   2 files changed, 5 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>>> index ca66c82b5b8..dc7b94aa115 100644
>>> --- a/scripts/qapi/gen.py
>>> +++ b/scripts/qapi/gen.py
>>> @@ -154,9 +154,11 @@ def _bottom(self):
>>>     @contextmanager
>>>   def ifcontext(ifcond, *args):
>>> -    """A 'with' statement context manager to wrap with start_if()/end_if()
>>> +    """
>>> +    A with-statement context manager that wraps with `start_if()` / `end_if()`.
>>>   -    *args: any number of QAPIGenCCode
>>> +    :param ifcond: A list of conditionals, passed to `start_if()`.
>>> +    :param args: any number of `QAPIGenCCode`.
>>>         Example::
>>>   diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>>> index 9d1a3e2eea9..31bc2e6dca9 100644
>>> --- a/scripts/qapi/parser.py
>>> +++ b/scripts/qapi/parser.py
>>> @@ -381,6 +381,7 @@ def append(self, line):
>>>             The way that the line is dealt with depends on which
>>> part of
>>>           the documentation we're parsing right now:
>>> +
>>>           * The body section: ._append_line is ._append_body_line
>>>           * An argument section: ._append_line is ._append_args_line
>>>           * A features section: ._append_line is ._append_features_line
>> I'm asking because you're not adding :param line: here.
>> 
>
> Yeah, it's not necessary to test the syntax of what else I've written
> with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't 
> required at all and could be deleted. You can do so if you'd like; it
> will just show up later in some other patch or series more designed to 
> fix formatting.

I recommend (but do not demand) to strictly limit this commit to
"avoiding obvious errors that cause sphinx documentation build
problems."

>> Same for several other functions in this file.
>> In schema.py:
>>      class QAPISchemaMember:
>>          """ Represents object members, enum members and features """
>> Are the spaces next to """ okay?
>> 
>
> Ideally cleaned up, but that's not a goal of this patch or series.

Got it.

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/6/20 7:51 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> This is a minor re-work of the entrypoint script. It isolates a
>>> generate() method from the actual command-line mechanism.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>   1 file changed, 62 insertions(+), 23 deletions(-)
>>>
>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>> index 541e8c1f55d..117b396a595 100644
>>> --- a/scripts/qapi-gen.py
>>> +++ b/scripts/qapi-gen.py
>>> @@ -1,30 +1,77 @@
>>>   #!/usr/bin/env python3
>>> -# QAPI generator
>>> -#
>>> +
>>>   # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>   # See the COPYING file in the top-level directory.
>>>   +"""
>>> +QAPI Generator
>>> +
>>> +This script is the main entry point for generating C code from the QAPI schema.
>>> +"""
>>>     import argparse
>>>   import re
>>>   import sys
>>>     from qapi.commands import gen_commands
>>> +from qapi.error import QAPIError
>>>   from qapi.events import gen_events
>>>   from qapi.introspect import gen_introspect
>>> -from qapi.schema import QAPIError, QAPISchema
>>> +from qapi.schema import QAPISchema
>>>   from qapi.types import gen_types
>>>   from qapi.visit import gen_visit
>> Unrelated cleanup.  Okay.
>> 
>>>     
>>> -def main(argv):
>>> +DEFAULT_OUTPUT_DIR = ''
>>> +DEFAULT_PREFIX = ''
>>> +
>>> +
>>> +def generate(schema_file: str,
>>> +             output_dir: str,
>>> +             prefix: str,
>>> +             unmask: bool = False,
>>> +             builtins: bool = False) -> None:
>>> +    """
>>> +    generate uses a given schema to produce C code in the target directory.
>>> +
>>> +    :param schema_file: The primary QAPI schema file.
>>> +    :param output_dir: The output directory to store generated code.
>>> +    :param prefix: Optional C-code prefix for symbol names.
>>> +    :param unmask: Expose non-ABI names through introspection?
>>> +    :param builtins: Generate code for built-in types?
>>> +
>>> +    :raise QAPIError: On failures.
>>> +    """
>>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>>> +    if match.end() != len(prefix):
>>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>>> +            prefix[match.end()], prefix)
>>> +        raise QAPIError('', None, msg)
>> Uh...
>>      $ python3 scripts/qapi-gen.py --prefix=@ x
>>      scripts/qapi-gen.py: : funny character '@' in prefix '@'
>> Unwanted " :".
>> This is due to a hack: you pass '' for info (*quack*).  Everything
>> else
>> passes QAPISourceInfo (I believe).
>>
>
> Quack indeed - why does our base error class require so much
> information from a specific part of the generation process?

Because it's not "a base error class", it's a base error class for the
QAPI schema compiler frontend.

> Ah, someone changes this in part 4 so that we have a more generic
> error class to use as a base when we are missing such information.

Evolving it to satisfy a need for a more widely usable error class is
okay.

> You are witnessing some more future-bleed.
>> Is it really a good idea to do this in generate?  It's not about
>> generating code, it's about validating a CLI option.
>> 
>
> One might also ask: Is it a good idea to only validate this on a
> frontend, and not in the implementation?

Yes, because that's where you can emit the better error message more
easily.

    $ python3 scripts/qapi-gen.py --prefix=@ x
    scripts/qapi-gen.py: 'funny character '@' in argument of --prefix

is better than

    $ python3 scripts/qapi-gen.py --prefix=@ x
    scripts/qapi-gen.py: funny character '@' in prefix '@'

In generate(), the knowledge where the offending prefix value comes from
is no longer available.

To emit this error message, you'd have to raise a sufficiently distinct
error in generate, catch it in main(), then put the error message
together somehow.  Bah.

Aside: there's a stray ' in the old error message.

> The idea here was to create a function that could be used in a script
> (for tests, debugging interfaces, other python packages) to do all of 
> the same things that the CLI tool did, just sans the actual CLI.

YAGNI.

> Wouldn't make sense to allow garbage to flow in from one interface but
> not the other; so the check is here.

"@prefix is sane" is a precondition of generate().

When there's a real risk of preconditions getting violated, or readers
getting confused about preconditions, check them with assert.

>>> +
>>> +    schema = QAPISchema(schema_file)
>>> +    gen_types(schema, output_dir, prefix, builtins)
>>> +    gen_visit(schema, output_dir, prefix, builtins)
>>> +    gen_commands(schema, output_dir, prefix)
>>> +    gen_events(schema, output_dir, prefix)
>>> +    gen_introspect(schema, output_dir, prefix, unmask)
>>> +
>>> +
>>> +def main() -> int:
>>> +    """
>>> +    gapi-gen shell script entrypoint.
>>> +    Expects arguments via sys.argv, see --help for details.
>>> +
>>> +    :return: int, 0 on success, 1 on failure.
>>> +    """
>>>       parser = argparse.ArgumentParser(
>>>           description='Generate code from a QAPI schema')
>>>       parser.add_argument('-b', '--builtins', action='store_true',
>>>                           help="generate code for built-in types")
>>> -    parser.add_argument('-o', '--output-dir', action='store', default='',
>>> +    parser.add_argument('-o', '--output-dir', action='store',
>>> +                        default=DEFAULT_OUTPUT_DIR,
>>>                           help="write output to directory OUTPUT_DIR")
>>> -    parser.add_argument('-p', '--prefix', action='store', default='',
>>> +    parser.add_argument('-p', '--prefix', action='store',
>>> +                        default=DEFAULT_PREFIX,
>>>                           help="prefix for symbols")
>> I don't like the changes to default=, because:
>> 1. They are only losely related to the patch's purpose.
>> 
>
> Subjective, but OK.
>
>> 2. They split the definition of the CLI: most of it is here, except for
>> defaults, which are defined elsewhere.
>> 
>
> All of it is in main.py, though! If you were to, say, move generate()
> elsewhere, it'd look pretty compact as just the CLI frontend, no?

Same statement is more compact than same screenful is more compact than
same file :)

>> 3. The defaults will not change, and nothing else uses the constants.
>> 
>
> But, fine. Cleber had the same comment but I wasn't fully on-board,
> but two folks saying the same thing ...
>
>>>       parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>>>                           dest='unmask',
>>> @@ -32,25 +79,17 @@ def main(argv):
>>>       parser.add_argument('schema', action='store')
>>>       args = parser.parse_args()
>>>   -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?',
>>> args.prefix)
>>> -    if match.end() != len(args.prefix):
>>> -        print("%s: 'funny character '%s' in argument of --prefix"
>>> -              % (sys.argv[0], args.prefix[match.end()]),
>>> -              file=sys.stderr)
>>> -        sys.exit(1)
>>> -
>>>       try:
>>> -        schema = QAPISchema(args.schema)
>>> +        generate(args.schema,
>>> +                 output_dir=args.output_dir,
>>> +                 prefix=args.prefix,
>>> +                 unmask=args.unmask,
>>> +                 builtins=args.builtins)
>>>       except QAPIError as err:
>>> -        print(err, file=sys.stderr)
>>> -        exit(1)
>>> -
>>> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
>>> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
>>> -    gen_commands(schema, args.output_dir, args.prefix)
>>> -    gen_events(schema, args.output_dir, args.prefix)
>>> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
>>> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
>>> +        return 1
>>> +    return 0
>> Subtle change: you move the gen_FOO() into the try ... except.
>> Okay;
>> they don't raise QAPIError, but perhaps worth a mention in the commit
>> message.
>> 
>
> Forbidden future knowledge; I intend them to.

I don't mind the move.

>>>     
>>>   if __name__ == '__main__':
>>> -    main(sys.argv)
>>> +    sys.exit(main())
>> "Python was designed to be easy to understand and fun to use."
>> Ha ha ha.
>> 
>
> I mean, I'm having fun, aren't you?

So many kinds of fun!  The fun I'm having with this patch hunk is
mocking "easy and fun" Python for requiring such an elaborate menuett
just to express "can run as program".

This emperor has no clothes, either.  And that's funny, isn't it?

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This is a minor re-work of the entrypoint script. It isolates a
> generate() method from the actual command-line mechanism.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>  1 file changed, 62 insertions(+), 23 deletions(-)
>
> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
> index 541e8c1f55d..117b396a595 100644
> --- a/scripts/qapi-gen.py
> +++ b/scripts/qapi-gen.py
> @@ -1,30 +1,77 @@
>  #!/usr/bin/env python3
> -# QAPI generator
> -#
> +
>  # This work is licensed under the terms of the GNU GPL, version 2 or later.
>  # See the COPYING file in the top-level directory.
>  
> +"""
> +QAPI Generator
> +
> +This script is the main entry point for generating C code from the QAPI schema.
> +"""
>  
>  import argparse
>  import re
>  import sys
>  
>  from qapi.commands import gen_commands
> +from qapi.error import QAPIError
>  from qapi.events import gen_events
>  from qapi.introspect import gen_introspect
> -from qapi.schema import QAPIError, QAPISchema
> +from qapi.schema import QAPISchema
>  from qapi.types import gen_types
>  from qapi.visit import gen_visit
>  
>  
> -def main(argv):
> +DEFAULT_OUTPUT_DIR = ''
> +DEFAULT_PREFIX = ''
> +
> +
> +def generate(schema_file: str,
> +             output_dir: str,
> +             prefix: str,
> +             unmask: bool = False,
> +             builtins: bool = False) -> None:
> +    """
> +    generate uses a given schema to produce C code in the target directory.
> +
> +    :param schema_file: The primary QAPI schema file.
> +    :param output_dir: The output directory to store generated code.
> +    :param prefix: Optional C-code prefix for symbol names.
> +    :param unmask: Expose non-ABI names through introspection?
> +    :param builtins: Generate code for built-in types?
> +
> +    :raise QAPIError: On failures.
> +    """
> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
> +    if match.end() != len(prefix):
> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
> +            prefix[match.end()], prefix)
> +        raise QAPIError('', None, msg)
> +
> +    schema = QAPISchema(schema_file)
> +    gen_types(schema, output_dir, prefix, builtins)
> +    gen_visit(schema, output_dir, prefix, builtins)
> +    gen_commands(schema, output_dir, prefix)
> +    gen_events(schema, output_dir, prefix)
> +    gen_introspect(schema, output_dir, prefix, unmask)
> +
> +
> +def main() -> int:
> +    """
> +    gapi-gen shell script entrypoint.

What's a "shell script entrypoint"?

Python docs talk about "when [...] run as a script":
https://docs.python.org/3/library/__main__.html

Similar:
https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts

> +    Expects arguments via sys.argv, see --help for details.
> +
> +    :return: int, 0 on success, 1 on failure.
> +    """
>      parser = argparse.ArgumentParser(
>          description='Generate code from a QAPI schema')
>      parser.add_argument('-b', '--builtins', action='store_true',
>                          help="generate code for built-in types")
> -    parser.add_argument('-o', '--output-dir', action='store', default='',
> +    parser.add_argument('-o', '--output-dir', action='store',
> +                        default=DEFAULT_OUTPUT_DIR,
>                          help="write output to directory OUTPUT_DIR")
> -    parser.add_argument('-p', '--prefix', action='store', default='',
> +    parser.add_argument('-p', '--prefix', action='store',
> +                        default=DEFAULT_PREFIX,
>                          help="prefix for symbols")
>      parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>                          dest='unmask',
> @@ -32,25 +79,17 @@ def main(argv):
>      parser.add_argument('schema', action='store')
>      args = parser.parse_args()
>  
> -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
> -    if match.end() != len(args.prefix):
> -        print("%s: 'funny character '%s' in argument of --prefix"
> -              % (sys.argv[0], args.prefix[match.end()]),
> -              file=sys.stderr)
> -        sys.exit(1)
> -
>      try:
> -        schema = QAPISchema(args.schema)
> +        generate(args.schema,
> +                 output_dir=args.output_dir,
> +                 prefix=args.prefix,
> +                 unmask=args.unmask,
> +                 builtins=args.builtins)
>      except QAPIError as err:
> -        print(err, file=sys.stderr)
> -        exit(1)
> -
> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
> -    gen_commands(schema, args.output_dir, args.prefix)
> -    gen_events(schema, args.output_dir, args.prefix)
> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
> +        return 1
> +    return 0
>  
>  
>  if __name__ == '__main__':
> -    main(sys.argv)
> +    sys.exit(main())

What does sys.exit() really buy us here?  I'm asking because both spots
in the Python docs I referenced above do without.

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

I keep stumbling over things in later patches that turn out to go back
to this one.

John Snow <jsnow@redhat.com> writes:

> This is a minor re-work of the entrypoint script. It isolates a
> generate() method from the actual command-line mechanism.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>  1 file changed, 62 insertions(+), 23 deletions(-)
>
> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
> index 541e8c1f55d..117b396a595 100644
> --- a/scripts/qapi-gen.py
> +++ b/scripts/qapi-gen.py
> @@ -1,30 +1,77 @@
>  #!/usr/bin/env python3
> -# QAPI generator
> -#
> +
>  # This work is licensed under the terms of the GNU GPL, version 2 or later.
>  # See the COPYING file in the top-level directory.
>  
> +"""
> +QAPI Generator
> +
> +This script is the main entry point for generating C code from the QAPI schema.

PEP 8: For flowing long blocks of text with fewer structural
restrictions (docstrings or comments), the line length should be limited
to 72 characters.

> +"""
>  
>  import argparse
>  import re
>  import sys
>  
>  from qapi.commands import gen_commands
> +from qapi.error import QAPIError
>  from qapi.events import gen_events
>  from qapi.introspect import gen_introspect
> -from qapi.schema import QAPIError, QAPISchema
> +from qapi.schema import QAPISchema
>  from qapi.types import gen_types
>  from qapi.visit import gen_visit
>  
>  
> -def main(argv):
> +DEFAULT_OUTPUT_DIR = ''
> +DEFAULT_PREFIX = ''
> +
> +
> +def generate(schema_file: str,
> +             output_dir: str,
> +             prefix: str,
> +             unmask: bool = False,
> +             builtins: bool = False) -> None:
> +    """
> +    generate uses a given schema to produce C code in the target directory.

PEP 257: The docstring is a phrase ending in a period.  It prescribes
the function or method's effect as a command ("Do this", "Return that"),
not as a description; e.g. don't write "Returns the pathname ...".

Suggest

       Generate C code for the given schema into the target directory.

> +
> +    :param schema_file: The primary QAPI schema file.
> +    :param output_dir: The output directory to store generated code.
> +    :param prefix: Optional C-code prefix for symbol names.
> +    :param unmask: Expose non-ABI names through introspection?
> +    :param builtins: Generate code for built-in types?
> +
> +    :raise QAPIError: On failures.
> +    """
[...]

Re: [PATCH v5 07/36] qapi: enforce import order/styling with isort

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> While we're mucking around with imports, we might as well formalize the
> style we use. Let's use isort to do it for us.
>
> lines_after_imports=2: Use two lines after imports, to match PEP8's
> desire to have "two lines before and after" class definitions, which are
> likely to start immediately after imports.
>
> force_sort_within_sections: Intermingles "from x" and "import x" style
> statements, such that sorting is always performed strictly on the module
> name itself.
>
> force_grid_wrap=4: Four or more imports from a single module will force
> the one-per-line style that's more git-friendly. This will generally
> happen for 'typing' imports.
>
> multi_line_output=3: Uses the one-per-line indented style for long
> imports.
>
> include_trailing_comma: Adds a comma to the last import in a group,
> which makes git conflicts nicer to deal with, generally.
>
> line_length: 72 is chosen to match PEP8's "docstrings and comments" line
> length limit. If you have a single line import that exceeds 72
> characters, your names are too long!
>
> Suggested-by: Cleber Rosa <crosa@redhat.com>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/.isort.cfg    | 7 +++++++
>  scripts/qapi/expr.py       | 3 ++-
>  scripts/qapi/introspect.py | 7 +++++--
>  scripts/qapi/parser.py     | 2 +-
>  scripts/qapi/schema.py     | 2 +-
>  5 files changed, 16 insertions(+), 5 deletions(-)
>  create mode 100644 scripts/qapi/.isort.cfg
>
> diff --git a/scripts/qapi/.isort.cfg b/scripts/qapi/.isort.cfg
> new file mode 100644
> index 00000000000..6d0fd6cc0d3
> --- /dev/null
> +++ b/scripts/qapi/.isort.cfg
> @@ -0,0 +1,7 @@
> +[settings]
> +force_grid_wrap=4
> +force_sort_within_sections=True
> +include_trailing_comma=True
> +line_length=72
> +lines_after_imports=2
> +multi_line_output=3
> \ No newline at end of file

Add one, please :)

> diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
> index 03b31ecfc19..e73b65b6a7e 100644
> --- a/scripts/qapi/expr.py
> +++ b/scripts/qapi/expr.py
> @@ -14,8 +14,9 @@
>  # This work is licensed under the terms of the GNU GPL, version 2.
>  # See the COPYING file in the top-level directory.
>  
> -import re
>  from collections import OrderedDict
> +import re
> +
>  from .common import c_name
>  from .error import QAPISemError
>  
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index b036fcf9ce7..31acd2f230a 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -17,8 +17,11 @@
>      mcgen,
>  )
>  from .gen import QAPISchemaMonolithicCVisitor
> -from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
> -                     QAPISchemaType)
> +from .schema import (
> +    QAPISchemaArrayType,
> +    QAPISchemaBuiltinType,
> +    QAPISchemaType,
> +)
>  
>  
>  def _make_tree(obj, ifcond, features, extra=None):
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index 68d8a1ce1cf..477ef25ccdf 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -14,9 +14,9 @@
>  # This work is licensed under the terms of the GNU GPL, version 2.
>  # See the COPYING file in the top-level directory.
>  
> +from collections import OrderedDict
>  import os
>  import re
> -from collections import OrderedDict
>  
>  from .error import QAPIParseError, QAPISemError
>  from .source import QAPISourceInfo
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index a835ee6fde3..093f7a38d88 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -14,9 +14,9 @@
>  
>  # TODO catching name collisions in generated code would be nice
>  
> +from collections import OrderedDict
>  import os
>  import re
> -from collections import OrderedDict
>  
>  from .common import c_name, pointer_suffix
>  from .error import QAPIError, QAPISemError

Reviewed-by: Markus Armbruster <armbru@redhat.com>

Re: [PATCH v5 08/36] qapi: delint using flake8

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Petty style guide fixes and line length enforcement.  Not a big win, not
> a big loss, but flake8 passes 100% on the qapi module, which gives us an
> easy baseline to enforce hereafter.
>
> A note on the flake8 exception: flake8 will warn on *any* bare except,
> but pylint's is context-aware and will suppress the warning if you
> re-raise the exception.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/.flake8     |  2 ++
>  scripts/qapi/commands.py |  3 ++-
>  scripts/qapi/schema.py   |  8 +++++---
>  scripts/qapi/visit.py    | 16 +++++++++++-----
>  4 files changed, 20 insertions(+), 9 deletions(-)
>  create mode 100644 scripts/qapi/.flake8
>
> diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
> new file mode 100644
> index 00000000000..6b158c68b84
> --- /dev/null
> +++ b/scripts/qapi/.flake8
> @@ -0,0 +1,2 @@
> +[flake8]
> +extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
> index 64ed5278f93..5dc2f5a9fa8 100644
> --- a/scripts/qapi/commands.py
> +++ b/scripts/qapi/commands.py
> @@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type):
>  def gen_marshal_output(ret_type):
>      return mcgen('''
>  
> -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
> +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
> +                                QObject **ret_out, Error **errp)
>  {
>      Visitor *v;
>  
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 093f7a38d88..cfc52e1ae44 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -536,7 +536,7 @@ def set_defined_in(self, name):
>              v.set_defined_in(name)
>  
>      def check(self, schema, seen):
> -        if not self.tag_member: # flat union
> +        if not self.tag_member:  # flat union
>              self.tag_member = seen.get(c_name(self._tag_name))
>              base = "'base'"
>              # Pointing to the base type when not implicit would be
> @@ -821,7 +821,7 @@ def __init__(self, fname):
>          self._entity_dict = {}
>          self._module_dict = OrderedDict()
>          self._schema_dir = os.path.dirname(fname)
> -        self._make_module(None) # built-ins
> +        self._make_module(None)  # built-ins
>          self._make_module(fname)
>          self._predefining = True
>          self._def_predefineds()
> @@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
>              # But it's not tight: the disjunction need not imply it.  We
>              # may end up compiling useless wrapper types.
>              # TODO kill simple unions or implement the disjunction
> -            assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
> +
> +            # pylint: disable=protected-access
> +            assert (ifcond or []) == typ._ifcond
>          else:
>              self._def_entity(QAPISchemaObjectType(
>                  name, info, None, ifcond, None, None, members, None))
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index ea277e7704b..808410d6f1b 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False):
>      if not scalar:
>          c_type += '*'
>      return mcgen('''
> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
> +
> +bool visit_type_%(c_name)s(Visitor *v, const char *name,
> +                 %(c_type)sobj, Error **errp);
>  ''',
>                   c_name=c_name(name), c_type=c_type)
>  
> @@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants):
>  def gen_visit_list(name, element_type):
>      return mcgen('''
>  
> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
> +bool visit_type_%(c_name)s(Visitor *v, const char *name,
> +                 %(c_name)s **obj, Error **errp)
>  {
>      bool ok = false;
>      %(c_name)s *tail;
> @@ -158,7 +161,8 @@ def gen_visit_list(name, element_type):
>  def gen_visit_enum(name):
>      return mcgen('''
>  
> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
> +bool visit_type_%(c_name)s(Visitor *v, const char *name,
> +                 %(c_name)s *obj, Error **errp)
>  {
>      int value = *obj;
>      bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
> @@ -172,7 +176,8 @@ def gen_visit_enum(name):
>  def gen_visit_alternate(name, variants):
>      ret = mcgen('''
>  
> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
> +bool visit_type_%(c_name)s
> +    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)

Please break this line the same way as the ones above:

   bool visit_type_%(c_name)s(Visitor *v, const char *name,
                    %(c_name)s **obj, Error **errp)

>  {
>      bool ok = false;
>  
> @@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants):
>  def gen_visit_object(name, base, members, variants):
>      return mcgen('''
>  
> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
> +bool visit_type_%(c_name)s
> +    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
>  {
>      bool ok = false;

Likewise.

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Code style tools really dislike the use of global keywords, because it
> generally involves re-binding the name at runtime which can have strange
> effects depending on when and how that global name is referenced in
> other modules.
>
> Make a little indent level manager instead.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>

Intentation is a job for QAPIGen (and its subtypes).  But if this patch
is easier to achieve this series' goal, I don't mind.

> ---
>  scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++--------------
>  scripts/qapi/visit.py  |  7 +++---
>  2 files changed, 36 insertions(+), 20 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index cee63eb95c7..b35318b72cf 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -93,33 +93,50 @@ def c_name(name, protect=True):
>  pointer_suffix = ' *' + eatspace
>  
>  
> -def genindent(count):
> -    ret = ''
> -    for _ in range(count):
> -        ret += ' '
> -    return ret
> +class Indentation:
> +    """
> +    Indentation level management.
>  
> +    :param initial: Initial number of spaces, default 0.
> +    """
> +    def __init__(self, initial: int = 0) -> None:
> +        self._level = initial
>  
> -indent_level = 0
> +    def __int__(self) -> int:
> +        return self._level
>  
> +    def __repr__(self) -> str:
> +        return "{}({:d})".format(type(self).__name__, self._level)
>  
> -def push_indent(indent_amount=4):
> -    global indent_level
> -    indent_level += indent_amount
> +    def __str__(self) -> str:
> +        """Return the current indentation as a string of spaces."""
> +        return ' ' * self._level
>  
> +    def __bool__(self) -> bool:
> +        """True when there is a non-zero indentation."""
> +        return bool(self._level)
>  
> -def pop_indent(indent_amount=4):
> -    global indent_level
> -    indent_level -= indent_amount
> +    def increase(self, amount: int = 4) -> None:
> +        """Increase the indentation level by ``amount``, default 4."""
> +        self._level += amount
> +
> +    def decrease(self, amount: int = 4) -> None:
> +        """Decrease the indentation level by ``amount``, default 4."""
> +        if self._level < amount:
> +            raise ArithmeticError(
> +                f"Can't remove {amount:d} spaces from {self!r}")

Raise a fancy error when there's an actual need for it.  You're not
coding a framework thousands of people you never heard of will put to
uses you cannot imagine.

> +        self._level -= amount
> +
> +
> +indent = Indentation()
>  
>  
>  # Generate @code with @kwds interpolated.
> -# Obey indent_level, and strip eatspace.
> +# Obey indent, and strip eatspace.
>  def cgen(code, **kwds):
>      raw = code % kwds
> -    if indent_level:
> -        indent = genindent(indent_level)
> -        raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
> +    if indent:
> +        raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
>      return re.sub(re.escape(eatspace) + r' *', '', raw)
>  
>  
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index 808410d6f1b..14f30c228b7 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -18,9 +18,8 @@
>      c_name,
>      gen_endif,
>      gen_if,
> +    indent,
>      mcgen,
> -    pop_indent,
> -    push_indent,
>  )
>  from .gen import QAPISchemaModularCVisitor, ifcontext
>  from .schema import QAPISchemaObjectType
> @@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants):
>      if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
>  ''',
>                           name=memb.name, c_name=c_name(memb.name))
> -            push_indent()
> +            indent.increase()
>          ret += mcgen('''
>      if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
>          return false;
> @@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants):
>                       c_type=memb.type.c_name(), name=memb.name,
>                       c_name=c_name(memb.name))
>          if memb.optional:
> -            pop_indent()
> +            indent.decrease()
>              ret += mcgen('''
>      }
>  ''')

Re: [PATCH v5 15/36] qapi/common.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Annotations do not change runtime behavior.
> This commit *only* adds annotations.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/common.py | 27 ++++++++++++++++-----------
>  1 file changed, 16 insertions(+), 11 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 338adedef4f..74a2c001ed9 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -12,6 +12,7 @@
>  # See the COPYING file in the top-level directory.
>  
>  import re
> +from typing import Optional, Sequence
>  
>  
>  EATSPACE = '\033EATSPACE.'
> @@ -22,7 +23,7 @@
>  # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>  # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>  # ENUM24_Name -> ENUM24_NAME
> -def camel_to_upper(value):
> +def camel_to_upper(value: str) -> str:
>      c_fun_str = c_name(value, False)
>      if value.isupper():
>          return c_fun_str
> @@ -41,7 +42,9 @@ def camel_to_upper(value):
>      return new_name.lstrip('_').upper()
>  
>  
> -def c_enum_const(type_name, const_name, prefix=None):
> +def c_enum_const(type_name: str,
> +                 const_name: str,
> +                 prefix: Optional[str] = None) -> str:
>      if prefix is not None:
>          type_name = prefix
>      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
> @@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
>  # into substrings of a generated C function name.
>  # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>  # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
> -def c_name(name, protect=True):
> +def c_name(name: str, protect: bool = True) -> str:
>      # ANSI X3J11/88-090, 3.1.1
>      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>                       'default', 'do', 'double', 'else', 'enum', 'extern',
> @@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None:
>  
>  # Generate @code with @kwds interpolated.
>  # Obey indent, and strip EATSPACE.
> -def cgen(code, **kwds):
> +def cgen(code: str, **kwds: object) -> str:
>      raw = code % kwds
>      if indent:
>          raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
>      return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>  
>  
> -def mcgen(code, **kwds):
> +def mcgen(code: str, **kwds: object) -> str:
>      if code[0] == '\n':
>          code = code[1:]
>      return cgen(code, **kwds)
>  
>  
> -def c_fname(filename):
> +def c_fname(filename: str) -> str:
>      return re.sub(r'[^A-Za-z0-9_]', '_', filename)
>  
>  
> -def guardstart(name):
> +def guardstart(name: str) -> str:
>      return mcgen('''
>  #ifndef %(name)s
>  #define %(name)s
> @@ -157,7 +160,7 @@ def guardstart(name):
>                   name=c_fname(name).upper())
>  
>  
> -def guardend(name):
> +def guardend(name: str) -> str:
>      return mcgen('''
>  
>  #endif /* %(name)s */
> @@ -165,7 +168,7 @@ def guardend(name):
>                   name=c_fname(name).upper())
>  
>  
> -def gen_if(ifcond):
> +def gen_if(ifcond: Sequence[str]) -> str:
>      ret = ''
>      for ifc in ifcond:
>          ret += mcgen('''
> @@ -174,7 +177,7 @@ def gen_if(ifcond):
>      return ret
>  
>  
> -def gen_endif(ifcond):
> +def gen_endif(ifcond: Sequence[str]) -> str:
>      ret = ''
>      for ifc in reversed(ifcond):
>          ret += mcgen('''
> @@ -183,7 +186,9 @@ def gen_endif(ifcond):
>      return ret
>  
>  
> -def build_params(arg_type, boxed, extra=None):
> +def build_params(arg_type,
> +                 boxed: bool,
> +                 extra: Optional[str] = None) -> str:
>      ret = ''
>      sep = ''
>      if boxed:

@arg_type is the only parameter left unannotated.  Scratching head...
aha:

    qapi/common.py: move build_params into gen.py
    
    Including it in common.py creates a circular import dependency; schema
    relies on common, but common.build_params requires a type annotation
    from schema. To type this properly, it needs to be moved outside the
    cycle.

Let's amend the commit message:

    Note that build_params() cannot be fully annotated due to import
    dependency issues.  The commit after next will take care of it.

If we swap the next two commits, the fix follows immediately.  Better,
but only worth it if swapping is pretty much effortless.

Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> As docstrings, they'll show up in documentation and IDE help.
>
> The docstring style being targeted is the Sphinx documentation
> style. Sphinx uses an extension of ReST with "domains". We use the
> (implicit) Python domain, which supports a number of custom "info
> fields". Those info fields are documented here:
> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
>
> Primarily, we use `:param X: descr`, `:return[s]: descr`, and `:raise[s]
> Z: when`. Everything else is the Sphinx dialect of ReST.
>
> (No, nothing checks or enforces this style that I am aware of. Sphinx
> either chokes or succeeds, but does not enforce a standard of what is
> otherwise inside the docstring. Pycharm does highlight when your param
> fields are not aligned with the actual fields present. It does not
> highlight missing return or exception statements. There is no existing
> style guide I am aware of that covers a standard for a minimally
> acceptable docstring. I am debating writing one.)
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
>  1 file changed, 39 insertions(+), 14 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 74a2c001ed9..0ef38ea5fe0 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -15,15 +15,24 @@
>  from typing import Optional, Sequence
>  
>  
> +#: Sentinel value that causes all space to its right to be removed.

What's the purpose of : after # ?

I'm not sure this is a "sentinel value".  Wikipedia:

    In computer programming, a sentinel value (also referred to as a
    flag value, trip value, rogue value, signal value, or dummy data)[1]
    is a special value in the context of an algorithm which uses its
    presence as a condition of termination, typically in a loop or
    recursive algorithm.

    https://en.wikipedia.org/wiki/Sentinel_value

Perhaps

   # Magic string value that gets removed along with all space to the
   # right.

>  EATSPACE = '\033EATSPACE.'
>  POINTER_SUFFIX = ' *' + EATSPACE
>  _C_NAME_TRANS = str.maketrans('.-', '__')
>  
>  
> -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
> -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
> -# ENUM24_Name -> ENUM24_NAME
>  def camel_to_upper(value: str) -> str:
> +    """
> +    Converts CamelCase to CAMEL_CASE.
> +
> +    Examples:
> +      ENUMName -> ENUM_NAME
> +      EnumName1 -> ENUM_NAME1
> +      ENUM_NAME -> ENUM_NAME
> +      ENUM_NAME1 -> ENUM_NAME1
> +      ENUM_Name2 -> ENUM_NAME2
> +      ENUM24_Name -> ENUM24_NAME
> +    """

I wonder whether these indented lines get wrapped into one
unintelligible parapgraph.

Have you eyeballed the output of Sphinx?

>      c_fun_str = c_name(value, False)
>      if value.isupper():
>          return c_fun_str
> @@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str:
>  def c_enum_const(type_name: str,
>                   const_name: str,
>                   prefix: Optional[str] = None) -> str:
> +    """
> +    Generate a C enumeration constant name.
> +
> +    :param type_name: The name of the enumeration.
> +    :param const_name: The name of this constant.
> +    :param prefix: Optional, prefix that overrides the type_name.
> +    """
>      if prefix is not None:
>          type_name = prefix
>      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>  
>  
> -# Map @name to a valid C identifier.
> -# If @protect, avoid returning certain ticklish identifiers (like
> -# C keywords) by prepending 'q_'.
> -#
> -# Used for converting 'name' from a 'name':'type' qapi definition
> -# into a generated struct member, as well as converting type names
> -# into substrings of a generated C function name.
> -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
> -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>  def c_name(name: str, protect: bool = True) -> str:
> +    """
> +    Map ``name`` to a valid C identifier.
> +
> +    Used for converting 'name' from a 'name':'type' qapi definition
> +    into a generated struct member, as well as converting type names
> +    into substrings of a generated C function name.
> +
> +    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
> +    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
> +
> +    :param name: The name to map.
> +    :param protect: If true, avoid returning certain ticklish identifiers
> +                    (like C keywords) by prepending ``q_``.
> +    """
>      # ANSI X3J11/88-090, 3.1.1
>      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>                       'default', 'do', 'double', 'else', 'enum', 'extern',
> @@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None:
>          self._level -= amount
>  
>  
> +#: Global, current indent level for code generation.
>  indent = Indentation()
>  
>  
> -# Generate @code with @kwds interpolated.
> -# Obey indent, and strip EATSPACE.
>  def cgen(code: str, **kwds: object) -> str:
> +    """
> +    Generate ``code`` with ``kwds`` interpolated.
> +
> +    Obey `indent`, and strip `EATSPACE`.
> +    """
>      raw = code % kwds
>      if indent:
>          raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)

Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Including it in common.py creates a circular import dependency; schema
> relies on common, but common.build_params requires a type annotation
> from schema. To type this properly, it needs to be moved outside the
> cycle.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/commands.py |  9 +++++++--
>  scripts/qapi/common.py   | 23 -----------------------
>  scripts/qapi/events.py   |  9 ++-------
>  scripts/qapi/gen.py      | 31 +++++++++++++++++++++++++++++--
>  4 files changed, 38 insertions(+), 34 deletions(-)
>
> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
> index 5dc2f5a9fa8..f67393f8713 100644
> --- a/scripts/qapi/commands.py
> +++ b/scripts/qapi/commands.py
> @@ -13,8 +13,13 @@
>  See the COPYING file in the top-level directory.
>  """
>  
> -from .common import build_params, c_name, mcgen
> -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
> +from .common import c_name, mcgen
> +from .gen import (
> +    QAPIGenCCode,
> +    QAPISchemaModularCVisitor,
> +    build_params,
> +    ifcontext,
> +)
>  
>  
>  def gen_command_decl(name, arg_type, boxed, ret_type):
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 0ef38ea5fe0..9ab0685cc51 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
>  #endif /* %(cond)s */
>  ''', cond=ifc)
>      return ret
> -
> -
> -def build_params(arg_type,
> -                 boxed: bool,
> -                 extra: Optional[str] = None) -> str:
> -    ret = ''
> -    sep = ''
> -    if boxed:
> -        assert arg_type
> -        ret += '%s arg' % arg_type.c_param_type()
> -        sep = ', '
> -    elif arg_type:
> -        assert not arg_type.variants
> -        for memb in arg_type.members:
> -            ret += sep
> -            sep = ', '
> -            if memb.optional:
> -                ret += 'bool has_%s, ' % c_name(memb.name)
> -            ret += '%s %s' % (memb.type.c_param_type(),
> -                              c_name(memb.name))
> -    if extra:
> -        ret += sep + extra
> -    return ret if ret else 'void'
> diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
> index 6b3afa14d72..f840a62ed92 100644
> --- a/scripts/qapi/events.py
> +++ b/scripts/qapi/events.py
> @@ -12,13 +12,8 @@
>  See the COPYING file in the top-level directory.
>  """
>  
> -from .common import (
> -    build_params,
> -    c_enum_const,
> -    c_name,
> -    mcgen,
> -)
> -from .gen import QAPISchemaModularCVisitor, ifcontext
> +from .common import c_enum_const, c_name, mcgen
> +from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
>  from .schema import QAPISchemaEnumMember
>  from .types import gen_enum, gen_enum_lookup
>  
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index 1fed712b43b..f2e2746fea5 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -2,9 +2,11 @@
>  #
>  # QAPI code generation
>  #
> -# Copyright (c) 2018-2019 Red Hat Inc.
> +# Copyright IBM, Corp. 2011
> +# Copyright (c) 2013-2019 Red Hat Inc.
>  #
>  # Authors:
> +#  Anthony Liguori <aliguori@us.ibm.com>
>  #  Markus Armbruster <armbru@redhat.com>
>  #  Marc-André Lureau <marcandre.lureau@redhat.com>
>  #

The code you move into this file is actually Red Hat's:

$ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date'
Author: Markus Armbruster <armbru@redhat.com>
Date:   Wed Aug 15 21:37:36 2018 +0800
Author: Marc-André Lureau <marcandre.lureau@redhat.com>
Date:   Thu Jun 1 16:41:41 2017 +0400
Author: Eric Blake <eblake@redhat.com>
Date:   Wed Jul 13 21:50:20 2016 -0600
Author: Eric Blake <eblake@redhat.com>
Date:   Wed Jul 13 21:50:19 2016 -0600
Author: Eric Blake <eblake@redhat.com>
Date:   Thu Mar 17 16:48:28 2016 -0600
Author: Markus Armbruster <armbru@redhat.com>
Date:   Wed Sep 16 13:06:20 2015 +0200

So the correct update is simply

   #
   # QAPI code generation
   #
  -# Copyright (c) 2018-2019 Red Hat Inc.
  +# Copyright (c) 2015-2019 Red Hat Inc.
   #
   # Authors:
   #  Markus Armbruster <armbru@redhat.com>
   #  Marc-André Lureau <marcandre.lureau@redhat.com>
   #

> @@ -15,16 +17,18 @@
>  import errno
>  import os
>  import re
> +from typing import Optional
>  
>  from .common import (
>      c_fname,
> +    c_name,
>      gen_endif,
>      gen_if,
>      guardend,
>      guardstart,
>      mcgen,
>  )
> -from .schema import QAPISchemaVisitor
> +from .schema import QAPISchemaObjectType, QAPISchemaVisitor
>  
>  
>  class QAPIGen:
> @@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
>      return out
>  
>  
> +def build_params(arg_type: Optional[QAPISchemaObjectType],
> +                 boxed: bool,
> +                 extra: Optional[str] = None) -> str:
> +    ret = ''
> +    sep = ''
> +    if boxed:
> +        assert arg_type
> +        ret += '%s arg' % arg_type.c_param_type()
> +        sep = ', '
> +    elif arg_type:
> +        assert not arg_type.variants
> +        for memb in arg_type.members:
> +            ret += sep
> +            sep = ', '
> +            if memb.optional:
> +                ret += 'bool has_%s, ' % c_name(memb.name)
> +            ret += '%s %s' % (memb.type.c_param_type(),
> +                              c_name(memb.name))
> +    if extra:
> +        ret += sep + extra
> +    return ret if ret else 'void'
> +
> +
>  class QAPIGenCCode(QAPIGen):
>  
>      def __init__(self, fname):

Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Fix two very minor issues,

What are the two issues?  I'm asking because I can see only one.

>                            and then establish a mypy type-checking
> baseline.

Any particular reason this can't go before PATCH 15 "qapi/common.py: add
type hint annotations".

> Like pylint, this should be run from the folder above:
>
>  > mypy --config-file=qapi/mypy.ini qapi/
>
> This is designed and tested for mypy 0.770 or greater.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Tested-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/mypy.ini  | 60 ++++++++++++++++++++++++++++++++++++++++++
>  scripts/qapi/schema.py |  3 ++-
>  2 files changed, 62 insertions(+), 1 deletion(-)
>  create mode 100644 scripts/qapi/mypy.ini
>
> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
> new file mode 100644
> index 00000000000..00fac15dc6e
> --- /dev/null
> +++ b/scripts/qapi/mypy.ini
> @@ -0,0 +1,60 @@
> +[mypy]
> +strict = True
> +strict_optional = False
> +disallow_untyped_calls = False
> +python_version = 3.6
> +
> +[mypy-qapi.commands]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.error]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.events]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.expr]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.gen]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.introspect]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.parser]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.schema]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.source]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.types]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.visit]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 74c6b96d391..483b4b68dff 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -17,6 +17,7 @@
>  from collections import OrderedDict
>  import os
>  import re
> +from typing import Optional
>  
>  from .common import POINTER_SUFFIX, c_name
>  from .error import QAPIError, QAPISemError
> @@ -25,7 +26,7 @@
>  
>  
>  class QAPISchemaEntity:
> -    meta = None
> +    meta: Optional[str] = None
>  
>      def __init__(self, name, info, doc, ifcond=None, features=None):
>          assert name is None or isinstance(name, str)

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Annotations do not change runtime behavior.
> This commit *only* adds annotations.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
>  scripts/qapi/mypy.ini  |  5 -----
>  2 files changed, 35 insertions(+), 16 deletions(-)
>
> diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
> index f840a62ed92..57e0939e963 100644
> --- a/scripts/qapi/events.py
> +++ b/scripts/qapi/events.py
> @@ -12,19 +12,31 @@
>  See the COPYING file in the top-level directory.
>  """
>  
> +from typing import List
> +
>  from .common import c_enum_const, c_name, mcgen
>  from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
> -from .schema import QAPISchemaEnumMember
> +from .schema import (
> +    QAPISchema,
> +    QAPISchemaEnumMember,
> +    QAPISchemaFeature,
> +    QAPISchemaObjectType,
> +)
> +from .source import QAPISourceInfo
>  from .types import gen_enum, gen_enum_lookup
>  
>  
> -def build_event_send_proto(name, arg_type, boxed):
> +def build_event_send_proto(name: str,
> +                           arg_type: QAPISchemaObjectType,
> +                           boxed: bool) -> str:
>      return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
>          'c_name': c_name(name.lower()),
>          'param': build_params(arg_type, boxed)}
>  
>  
> -def gen_event_send_decl(name, arg_type, boxed):
> +def gen_event_send_decl(name: str,
> +                        arg_type: QAPISchemaObjectType,
> +                        boxed: bool) -> str:
>      return mcgen('''
>  
>  %(proto)s;
> @@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed):
>  
>  
>  # Declare and initialize an object 'qapi' using parameters from build_params()
> -def gen_param_var(typ):
> +def gen_param_var(typ: QAPISchemaObjectType) -> str:
>      assert not typ.variants
>      ret = mcgen('''
>      %(c_name)s param = {
> @@ -61,7 +73,11 @@ def gen_param_var(typ):
>      return ret
>  
>  
> -def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
> +def gen_event_send(name: str,
> +                   arg_type: QAPISchemaObjectType,
> +                   boxed: bool,
> +                   event_enum_name: str,
> +                   event_emit: str) -> str:
>      # FIXME: Our declaration of local variables (and of 'errp' in the
>      # parameter list) can collide with exploded members of the event's
>      # data type passed in as parameters.  If this collision ever hits in
> @@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
>  
>  class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
>  
> -    def __init__(self, prefix):
> +    def __init__(self, prefix: str):
>          super().__init__(
>              prefix, 'qapi-events',
>              ' * Schema-defined QAPI/QMP events', None, __doc__)
>          self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
> -        self._event_enum_members = []
> +        self._event_enum_members: List[QAPISchemaEnumMember] = []
>          self._event_emit_name = c_name(prefix + 'qapi_event_emit')
>  
> -    def _begin_user_module(self, name):
> +    def _begin_user_module(self, name: str) -> None:
>          events = self._module_basename('qapi-events', name)
>          types = self._module_basename('qapi-types', name)
>          visit = self._module_basename('qapi-visit', name)
> @@ -168,7 +184,7 @@ def _begin_user_module(self, name):
>  ''',
>                               types=types))
>  
> -    def visit_end(self):
> +    def visit_end(self) -> None:

Ignorant question: what's the difference between -> None (like here) and
nothing (like __init__() above?

>          self._add_system_module('emit', ' * QAPI Events emission')
>          self._genc.preamble_add(mcgen('''
>  #include "qemu/osdep.h"
[...]

Re: [PATCH v5 21/36] qapi/commands.py: Don't re-bind to variable of different type

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Mypy isn't a fan of rebinding a variable with a new data type.
> It's easy enough to avoid.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/commands.py | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
> index f67393f8713..586774a23c7 100644
> --- a/scripts/qapi/commands.py
> +++ b/scripts/qapi/commands.py
> @@ -195,14 +195,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
>      if not options:
>          options = ['QCO_NO_OPTIONS']
>  
> -    options = " | ".join(options)
> -
>      ret = mcgen('''
>      qmp_register_command(cmds, "%(name)s",
>                           qmp_marshal_%(c_name)s, %(opts)s);
>  ''',
>                  name=name, c_name=c_name(name),
> -                opts=options)
> +                opts=" | ".join(options))
>      return ret

Definitely neater.

Reviewed-by: Markus Armbruster <armbru@redhat.com>

Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/mypy.ini | 5 -----
>  1 file changed, 5 deletions(-)
>
> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
> index 5df11e53fd1..8ab9ac52cc4 100644
> --- a/scripts/qapi/mypy.ini
> +++ b/scripts/qapi/mypy.ini
> @@ -4,11 +4,6 @@ strict_optional = False
>  disallow_untyped_calls = False
>  python_version = 3.6
>  
> -[mypy-qapi.commands]
> -disallow_untyped_defs = False
> -disallow_incomplete_defs = False
> -check_untyped_defs = False
> -
>  [mypy-qapi.error]
>  disallow_untyped_defs = False
>  disallow_incomplete_defs = False

The equivalent change for  events.py you squashed into the commit adding
type hints.  Any particular reason for not doing the same here?

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[Markus Armbruster]

Markus Armbruster <armbru@redhat.com> writes:

> John Snow <jsnow@redhat.com> writes:
>
>> Annotations do not change runtime behavior.
>> This commit *only* adds annotations.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
>>  scripts/qapi/mypy.ini  |  5 -----
>>  2 files changed, 35 insertions(+), 16 deletions(-)
>>
>> diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
>> index f840a62ed92..57e0939e963 100644
>> --- a/scripts/qapi/events.py
>> +++ b/scripts/qapi/events.py
>> @@ -12,19 +12,31 @@
[...]
>> @@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
>>  
>>  class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
>>  
>> -    def __init__(self, prefix):
>> +    def __init__(self, prefix: str):
>>          super().__init__(
>>              prefix, 'qapi-events',
>>              ' * Schema-defined QAPI/QMP events', None, __doc__)
>>          self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
>> -        self._event_enum_members = []
>> +        self._event_enum_members: List[QAPISchemaEnumMember] = []
>>          self._event_emit_name = c_name(prefix + 'qapi_event_emit')
>>  
>> -    def _begin_user_module(self, name):
>> +    def _begin_user_module(self, name: str) -> None:
>>          events = self._module_basename('qapi-events', name)
>>          types = self._module_basename('qapi-types', name)
>>          visit = self._module_basename('qapi-visit', name)
>> @@ -168,7 +184,7 @@ def _begin_user_module(self, name):
>>  ''',
>>                               types=types))
>>  
>> -    def visit_end(self):
>> +    def visit_end(self) -> None:
>
> Ignorant question: what's the difference between -> None (like here) and
> nothing (like __init__() above?

Looks like commit message of PATCH 24 has an answer.

Copy to all the commits that omit -> None similarly?

>>          self._add_system_module('emit', ' * QAPI Events emission')
>>          self._genc.preamble_add(mcgen('''
>>  #include "qemu/osdep.h"
> [...]

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Annotations do not change runtime behavior.
> This commit *only* adds annotations.
>
> A note on typing of __init__: mypy requires init functions with no
> parameters to document a return type of None to be considered fully
> typed. In the case when there are input parameters, None may be omitted.
>
> Since __init__ may never return any value, it is preferred to omit the
> return annotation whenever possible.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/mypy.ini  |  5 -----
>  scripts/qapi/source.py | 31 ++++++++++++++++++-------------
>  2 files changed, 18 insertions(+), 18 deletions(-)
>
> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
> index 8ab9ac52cc4..1b8555dfa39 100644
> --- a/scripts/qapi/mypy.ini
> +++ b/scripts/qapi/mypy.ini
> @@ -34,11 +34,6 @@ disallow_untyped_defs = False
>  disallow_incomplete_defs = False
>  check_untyped_defs = False
>  
> -[mypy-qapi.source]
> -disallow_untyped_defs = False
> -disallow_incomplete_defs = False
> -check_untyped_defs = False
> -
>  [mypy-qapi.types]
>  disallow_untyped_defs = False
>  disallow_incomplete_defs = False
> diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
> index e97b9a8e15e..1cc6a5b82dc 100644
> --- a/scripts/qapi/source.py
> +++ b/scripts/qapi/source.py
> @@ -11,37 +11,42 @@
>  
>  import copy
>  import sys
> +from typing import List, Optional, TypeVar
>  
>  
>  class QAPISchemaPragma:
> -    def __init__(self):
> +    def __init__(self) -> None:
>          # Are documentation comments required?
>          self.doc_required = False
>          # Whitelist of commands allowed to return a non-dictionary
> -        self.returns_whitelist = []
> +        self.returns_whitelist: List[str] = []
>          # Whitelist of entities allowed to violate case conventions
> -        self.name_case_whitelist = []
> +        self.name_case_whitelist: List[str] = []
>  
>  
>  class QAPISourceInfo:
> -    def __init__(self, fname, line, parent):
> +    T = TypeVar('T', bound='QAPISourceInfo')
> +
> +    def __init__(self: T, fname: str, line: int, parent: Optional[T]):

More ignorant questions (I'm abusing the review process to learn Python
type hints)...

Why do you need to annotate self here, but not elsewhere?

Why do you use T instead of QAPISourceInfo?

>          self.fname = fname
>          self.line = line
>          self.parent = parent
> -        self.pragma = parent.pragma if parent else QAPISchemaPragma()
> -        self.defn_meta = None
> -        self.defn_name = None
> +        self.pragma: QAPISchemaPragma = (
> +            parent.pragma if parent else QAPISchemaPragma()
> +        )

Type inference fail?

> +        self.defn_meta: Optional[str] = None
> +        self.defn_name: Optional[str] = None
>  
> -    def set_defn(self, meta, name):
> +    def set_defn(self, meta: str, name: str) -> None:
>          self.defn_meta = meta
>          self.defn_name = name
>  
> -    def next_line(self):
> +    def next_line(self: T) -> T:
>          info = copy.copy(self)
>          info.line += 1
>          return info
>  
> -    def loc(self):
> +    def loc(self) -> str:
>          if self.fname is None:
>              return sys.argv[0]
>          ret = self.fname
> @@ -49,13 +54,13 @@ def loc(self):
>              ret += ':%d' % self.line
>          return ret
>  
> -    def in_defn(self):
> +    def in_defn(self) -> str:
>          if self.defn_name:
>              return "%s: In %s '%s':\n" % (self.fname,
>                                            self.defn_meta, self.defn_name)
>          return ''
>  
> -    def include_path(self):
> +    def include_path(self) -> str:
>          ret = ''
>          parent = self.parent
>          while parent:
> @@ -63,5 +68,5 @@ def include_path(self):
>              parent = parent.parent
>          return ret
>  
> -    def __str__(self):
> +    def __str__(self) -> str:
>          return self.include_path() + self.in_defn() + self.loc()

Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> The edge case is that if the name is '', this expression returns a
> string instead of a bool, which violates our declared type.

The edge case is impossible, as discussed in review of v2.  I figure the
type checker can't see that, so we need to help it some.  Can we mention
this in the commit message?

>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  scripts/qapi/gen.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index f2e2746fea5..1bad37fc06b 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
>  
>      @staticmethod
>      def _is_user_module(name):
> -        return name and not name.startswith('./')
> +        return bool(name and not name.startswith('./'))

           return not (name is None or name.startswith('./')

Looks slightly clearer to me.
>  
>      @staticmethod
>      def _is_builtin_module(name):

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Annotations do not change runtime behavior.
> This commit *only* adds annotations.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
>  1 file changed, 57 insertions(+), 47 deletions(-)
>
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index 1bad37fc06b..d0391cd8718 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -17,7 +17,13 @@
>  import errno
>  import os
>  import re
> -from typing import Optional
> +from typing import (
> +    Dict,
> +    Iterator,
> +    List,
> +    Optional,
> +    Tuple,
> +)
>  
>  from .common import (
>      c_fname,
> @@ -29,31 +35,31 @@
>      mcgen,
>  )
>  from .schema import QAPISchemaObjectType, QAPISchemaVisitor
> +from .source import QAPISourceInfo
>  
>  
>  class QAPIGen:
> -
> -    def __init__(self, fname):
> +    def __init__(self, fname: Optional[str]):

I'd expect fname: str.  Can you point me to the spot that passes None?

>          self.fname = fname
>          self._preamble = ''
>          self._body = ''
>  
> -    def preamble_add(self, text):
> +    def preamble_add(self, text: str) -> None:
>          self._preamble += text
>  
> -    def add(self, text):
> +    def add(self, text: str) -> None:
>          self._body += text
>  
> -    def get_content(self):
> +    def get_content(self) -> str:
>          return self._top() + self._preamble + self._body + self._bottom()
>  
> -    def _top(self):
> +    def _top(self) -> str:
>          return ''
>  
> -    def _bottom(self):
> +    def _bottom(self) -> str:
>          return ''
>  
> -    def write(self, output_dir):
> +    def write(self, output_dir: str) -> None:
>          # Include paths starting with ../ are used to reuse modules of the main
>          # schema in specialised schemas. Don't overwrite the files that are
>          # already generated for the main schema.
> @@ -78,7 +84,7 @@ def write(self, output_dir):
>          f.close()
>  
>  
> -def _wrap_ifcond(ifcond, before, after):
> +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
>      if before == after:
>          return after   # suppress empty #if ... #endif
>  
> @@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
>  
>  
>  class QAPIGenCCode(QAPIGen):
> -
> -    def __init__(self, fname):
> +    def __init__(self, fname: Optional[str]):

Likewise.

>          super().__init__(fname)
> -        self._start_if = None
> +        self._start_if: Optional[Tuple[List[str], str, str]] = None
>  
> -    def start_if(self, ifcond):
> +    def start_if(self, ifcond: List[str]) -> None:
>          assert self._start_if is None
>          self._start_if = (ifcond, self._body, self._preamble)
>  
> -    def end_if(self):
> +    def end_if(self) -> None:
>          assert self._start_if
>          self._wrap_ifcond()
>          self._start_if = None
>  
> -    def _wrap_ifcond(self):
> +    def _wrap_ifcond(self) -> None:
>          self._body = _wrap_ifcond(self._start_if[0],
>                                    self._start_if[1], self._body)
>          self._preamble = _wrap_ifcond(self._start_if[0],
>                                        self._start_if[2], self._preamble)
>  
> -    def get_content(self):
> +    def get_content(self) -> str:
>          assert self._start_if is None
>          return super().get_content()
>  
>  
>  class QAPIGenC(QAPIGenCCode):
> -
> -    def __init__(self, fname, blurb, pydoc):
> +    def __init__(self, fname: str, blurb: str, pydoc: str):

Here it's just str.

>          super().__init__(fname)
>          self._blurb = blurb
>          self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
>                                                    re.MULTILINE))
>  
> -    def _top(self):
> +    def _top(self) -> str:
>          return mcgen('''
>  /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
>  
> @@ -167,7 +171,7 @@ def _top(self):
>  ''',
>                       blurb=self._blurb, copyright=self._copyright)
>  
> -    def _bottom(self):
> +    def _bottom(self) -> str:
>          return mcgen('''
>  
>  /* Dummy declaration to prevent empty .o file */
> @@ -177,16 +181,15 @@ def _bottom(self):
>  
>  
>  class QAPIGenH(QAPIGenC):
> -
> -    def _top(self):
> +    def _top(self) -> str:
>          return super()._top() + guardstart(self.fname)
>  
> -    def _bottom(self):
> +    def _bottom(self) -> str:
>          return guardend(self.fname)
>  
>  
>  @contextmanager
> -def ifcontext(ifcond, *args):
> +def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:

Oh, the type hint for a *args is QAPIGenCCode, even though args is a
tuple of excess positional arguments (which are all QAPIGenCCode).

>      """
>      A with-statement context manager that wraps with `start_if()` / `end_if()`.
>  
> @@ -214,8 +217,11 @@ def ifcontext(ifcond, *args):
>  
>  
>  class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
> -
> -    def __init__(self, prefix, what, blurb, pydoc):
> +    def __init__(self,
> +                 prefix: str,
> +                 what: str,
> +                 blurb: str,
> +                 pydoc: str):
>          self._prefix = prefix
>          self._what = what
>          self._genc = QAPIGenC(self._prefix + self._what + '.c',
> @@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc):
>          self._genh = QAPIGenH(self._prefix + self._what + '.h',
>                                blurb, pydoc)
>  
> -    def write(self, output_dir):
> +    def write(self, output_dir: str) -> None:
>          self._genc.write(output_dir)
>          self._genh.write(output_dir)
>  
>  
>  class QAPISchemaModularCVisitor(QAPISchemaVisitor):
> -
> -    def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
> +    def __init__(self,
> +                 prefix: str,
> +                 what: str,
> +                 user_blurb: str,
> +                 builtin_blurb: Optional[str],
> +                 pydoc: str):
>          self._prefix = prefix
>          self._what = what
>          self._user_blurb = user_blurb
>          self._builtin_blurb = builtin_blurb
>          self._pydoc = pydoc
> -        self._genc = None
> -        self._genh = None
> -        self._module = {}
> -        self._main_module = None
> +        self._genc: Optional[QAPIGenC] = None
> +        self._genh: Optional[QAPIGenH] = None
> +        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
> +        self._main_module: Optional[str] = None
>  
>      @staticmethod
> -    def _is_user_module(name):
> +    def _is_user_module(name: Optional[str]) -> bool:
>          return bool(name and not name.startswith('./'))
>  
>      @staticmethod
> -    def _is_builtin_module(name):
> +    def _is_builtin_module(name: Optional[str]) -> bool:
>          return not name
>  
> -    def _module_dirname(self, what, name):
> +    def _module_dirname(self, what: str, name: Optional[str]) -> str:
>          if self._is_user_module(name):
>              return os.path.dirname(name)
>          return ''
>  
> -    def _module_basename(self, what, name):
> +    def _module_basename(self, what: str, name: Optional[str]) -> str:
>          ret = '' if self._is_builtin_module(name) else self._prefix
>          if self._is_user_module(name):
>              basename = os.path.basename(name)
> @@ -266,27 +276,27 @@ def _module_basename(self, what, name):
>              ret += re.sub(r'-', '-' + name + '-', what)
>          return ret
>  
> -    def _module_filename(self, what, name):
> +    def _module_filename(self, what: str, name: Optional[str]) -> str:
>          return os.path.join(self._module_dirname(what, name),
>                              self._module_basename(what, name))
>  
> -    def _add_module(self, name, blurb):
> +    def _add_module(self, name: Optional[str], blurb: str) -> None:
>          basename = self._module_filename(self._what, name)
>          genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
>          genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
>          self._module[name] = (genc, genh)
>          self._genc, self._genh = self._module[name]
>  
> -    def _add_user_module(self, name, blurb):
> +    def _add_user_module(self, name: str, blurb: str) -> None:
>          assert self._is_user_module(name)
>          if self._main_module is None:
>              self._main_module = name
>          self._add_module(name, blurb)
>  
> -    def _add_system_module(self, name, blurb):
> +    def _add_system_module(self, name: Optional[str], blurb: str) -> None:
>          self._add_module(name and './' + name, blurb)
>  
> -    def write(self, output_dir, opt_builtins=False):
> +    def write(self, output_dir: str, opt_builtins: bool = False) -> None:
>          for name in self._module:
>              if self._is_builtin_module(name) and not opt_builtins:
>                  continue
> @@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False):
>              genc.write(output_dir)
>              genh.write(output_dir)
>  
> -    def _begin_system_module(self, name):
> +    def _begin_system_module(self, name: None) -> None:

I figure the type hint None signals a simplifcation opportunity.  No
need to worry about it now.

>          pass
>  
> -    def _begin_user_module(self, name):
> +    def _begin_user_module(self, name: str) -> None:
>          pass
>  
> -    def visit_module(self, name):
> +    def visit_module(self, name: Optional[str]) -> None:
>          if name is None:
>              if self._builtin_blurb:
>                  self._add_system_module(None, self._builtin_blurb)
> @@ -314,7 +324,7 @@ def visit_module(self, name):
>              self._add_user_module(name, self._user_blurb)
>              self._begin_user_module(name)
>  
> -    def visit_include(self, name, info):
> +    def visit_include(self, name: str, info: QAPISourceInfo) -> None:
>          relname = os.path.relpath(self._module_filename(self._what, name),
>                                    os.path.dirname(self._genh.fname))
>          self._genh.preamble_add(mcgen('''

Re: [PATCH v5 29/36] qapi/gen.py: Remove unused parameter

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> module_basename doesn't use the 'what' argument, so remove it.

_module_dirname(), I suppose.

>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/gen.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index d0391cd8718..3624162bb77 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool:
>      def _is_builtin_module(name: Optional[str]) -> bool:
>          return not name
>  
> -    def _module_dirname(self, what: str, name: Optional[str]) -> str:
> +    def _module_dirname(self, name: Optional[str]) -> str:
>          if self._is_user_module(name):
>              return os.path.dirname(name)
>          return ''
> @@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
>          return ret
>  
>      def _module_filename(self, what: str, name: Optional[str]) -> str:
> -        return os.path.join(self._module_dirname(what, name),
> +        return os.path.join(self._module_dirname(name),
>                              self._module_basename(what, name))
>  
>      def _add_module(self, name: Optional[str], blurb: str) -> None:

Re: [PATCH v5 30/36] qapi/gen.py: update write() to be more idiomatic

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Make the file handling here just a tiny bit more idiomatic.
> (I realize this is heavily subjective.)
>
> Use exist_ok=True for os.makedirs and remove the exception,
> use fdopen() to wrap the file descriptor in a File-like object,
> and use a context manager for managing the file pointer.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/gen.py | 25 +++++++++++--------------
>  1 file changed, 11 insertions(+), 14 deletions(-)
>
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index 3624162bb77..579ee283297 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -14,7 +14,6 @@
>  # See the COPYING file in the top-level directory.
>  
>  from contextlib import contextmanager
> -import errno
>  import os
>  import re
>  from typing import (
> @@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None:
>              return
>          pathname = os.path.join(output_dir, self.fname)
>          odir = os.path.dirname(pathname)
> +
>          if odir:
> -            try:
> -                os.makedirs(odir)
> -            except os.error as e:
> -                if e.errno != errno.EEXIST:
> -                    raise
> +            os.makedirs(odir, exist_ok=True)

I wouldn't call this part "heavily subjective".  When wrote the old
code, exist_ok was still off limits (it's new in 3.2).

> +
> +        # use os.open for O_CREAT to create and read a non-existant file
>          fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
> -        f = open(fd, 'r+', encoding='utf-8')
> -        text = self.get_content()
> -        oldtext = f.read(len(text) + 1)
> -        if text != oldtext:
> -            f.seek(0)
> -            f.truncate(0)
> -            f.write(text)
> -        f.close()
> +        with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
> +            text = self.get_content()
> +            oldtext = fp.read(len(text) + 1)
> +            if text != oldtext:
> +                fp.seek(0)
> +                fp.truncate(0)
> +                fp.write(text)
>  
>  
>  def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:

Reviewed-by: Markus Armbruster <armbru@redhat.com>

Re: [PATCH v5 33/36] qapi/types.py: remove one-letter variables

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> "John, if pylint told you to jump off a bridge, would you?"
> Hey, if it looked like fun, I might.
>
> Now that this file is clean, enable pylint checks on this file.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/pylintrc |  1 -
>  scripts/qapi/types.py | 29 +++++++++++++++--------------
>  2 files changed, 15 insertions(+), 15 deletions(-)
>
> diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
> index 8badcb11cda..b3c4cf46dbf 100644
> --- a/scripts/qapi/pylintrc
> +++ b/scripts/qapi/pylintrc
> @@ -6,7 +6,6 @@ ignore-patterns=error.py,
>                  expr.py,
>                  parser.py,
>                  schema.py,
> -                types.py,
>                  visit.py,
>  
>  
> diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
> index 766822feb3a..9d1e79d503d 100644
> --- a/scripts/qapi/types.py
> +++ b/scripts/qapi/types.py
> @@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
>      .array = (const char *const[]) {
>  ''',
>                  c_name=c_name(name))
> -    for m in members:
> -        ret += gen_if(m.ifcond)
> -        index = c_enum_const(name, m.name, prefix)
> +    for member in members:

Let's use @memb.  It's more visually distinct from @members, and
gen_struct_members() already uses it.

> +        ret += gen_if(member.ifcond)
> +        index = c_enum_const(name, member.name, prefix)
>          ret += mcgen('''
>          [%(index)s] = "%(name)s",
>  ''',
> -                     index=index, name=m.name)
> -        ret += gen_endif(m.ifcond)
> +                     index=index, name=member.name)
> +        ret += gen_endif(member.ifcond)
>  
>      ret += mcgen('''
>      },
> @@ -79,13 +79,13 @@ def gen_enum(name: str,
>  ''',
>                  c_name=c_name(name))
>  
> -    for m in enum_members:
> -        ret += gen_if(m.ifcond)
> +    for member in enum_members:
> +        ret += gen_if(member.ifcond)
>          ret += mcgen('''
>      %(c_enum)s,
>  ''',
> -                     c_enum=c_enum_const(name, m.name, prefix))
> -        ret += gen_endif(m.ifcond)
> +                     c_enum=c_enum_const(name, member.name, prefix))
> +        ret += gen_endif(member.ifcond)
>  
>      ret += mcgen('''
>  } %(c_name)s;
> @@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
>      objects_seen.add(name)
>  
>      ret = ''
> -    if variants:
> -        for v in variants.variants:
> -            if isinstance(v.type, QAPISchemaObjectType):
> -                ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
> -                                  v.type.local_members, v.type.variants)
> +    for variant in variants.variants if variants else ():

Let's use @var, for consistency with gen_variants().

> +        obj = variant.type
> +        if not isinstance(obj, QAPISchemaObjectType):
> +            continue
> +        ret += gen_object(obj.name, obj.ifcond, obj.base,
> +                          obj.local_members, obj.variants)
>  
>      ret += mcgen('''

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This is true by design, but not presently able to be expressed in the
> type system. An assertion helps mypy understand our constraints.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/visit.py | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index 14f30c228b7..4f11fd325b8 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -22,7 +22,7 @@
>      mcgen,
>  )
>  from .gen import QAPISchemaModularCVisitor, ifcontext
> -from .schema import QAPISchemaObjectType
> +from .schema import QAPISchemaEnumType, QAPISchemaObjectType
>  
>  
>  def gen_visit_decl(name, scalar=False):
> @@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
>          ret += gen_endif(memb.ifcond)
>  
>      if variants:
> +        tag_member = variants.tag_member
> +        assert isinstance(tag_member.type, QAPISchemaEnumType)

I'm curious: do you need the local variable to make the assertion stick?

> +
>          ret += mcgen('''
>      switch (obj->%(c_name)s) {
>  ''',
> -                     c_name=c_name(variants.tag_member.name))
> +                     c_name=c_name(tag_member.name))
>  
>          for var in variants.variants:
> -            case_str = c_enum_const(variants.tag_member.type.name,
> -                                    var.name,
> -                                    variants.tag_member.type.prefix)
> +            case_str = c_enum_const(tag_member.type.name, var.name,
> +                                    tag_member.type.prefix)
>              ret += gen_if(var.ifcond)
>              if var.type.name == 'q_empty':
>                  # valid variant and nothing to do

Re: [PATCH v5 36/36] qapi/visit.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Annotations do not change runtime behavior.
> This commit *only* adds annotations.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/mypy.ini |  5 ---
>  scripts/qapi/visit.py | 73 +++++++++++++++++++++++++++++++++----------
>  2 files changed, 56 insertions(+), 22 deletions(-)
>
> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
> index 83f19813553..74fc6c82153 100644
> --- a/scripts/qapi/mypy.ini
> +++ b/scripts/qapi/mypy.ini
> @@ -28,8 +28,3 @@ check_untyped_defs = False
>  disallow_untyped_defs = False
>  disallow_incomplete_defs = False
>  check_untyped_defs = False
> -
> -[mypy-qapi.visit]
> -disallow_untyped_defs = False
> -disallow_incomplete_defs = False
> -check_untyped_defs = False
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index e54694e23db..14d4f0b261f 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -13,6 +13,8 @@
>  See the COPYING file in the top-level directory.
>  """
>  
> +from typing import List, Optional
> +
>  from .common import (
>      c_enum_const,
>      c_name,
> @@ -22,10 +24,20 @@
>      mcgen,
>  )
>  from .gen import QAPISchemaModularCVisitor, ifcontext
> -from .schema import QAPISchemaEnumType, QAPISchemaObjectType
> +from .schema import (
> +    QAPISchema,
> +    QAPISchemaEnumMember,
> +    QAPISchemaEnumType,
> +    QAPISchemaFeature,
> +    QAPISchemaObjectType,
> +    QAPISchemaObjectTypeMember,
> +    QAPISchemaType,
> +    QAPISchemaVariants,
> +)
> +from .source import QAPISourceInfo
>  
>  
> -def gen_visit_decl(name, scalar=False):
> +def gen_visit_decl(name: str, scalar: bool = False) -> str:

More abuse of review to learn: even through we initialize with False, we
still need the type hint?

>      c_type = c_name(name) + ' *'
>      if not scalar:
>          c_type += '*'
[...]

Re: [PATCH v5 00/36] qapi: static typing conversion, pt1

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Hi, this series adds static type hints to the QAPI module.
> This is part one!
>
> Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
> Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6
>
> - Requires Python 3.6+
> - Requires mypy 0.770 or newer (for type analysis only)
> - Requires pylint 2.6.0 or newer (for lint checking only)
>
> In general, this series tackles the cleanup of one individual QAPI
> module at a time. Once it passes pylint or mypy checks, those checks are
> enabled for that file.
>
> Type hints are added in patches that add *only* type hints and change no
> other behavior. Any necessary changes to behavior to accommodate typing
> are split out into their own tiny patches.

Neat and pleasant to review.  The care you put into this shows.  Thanks!

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Annotations do not change runtime behavior.
> This commit *only* adds annotations.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> ---
>  scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
>  1 file changed, 57 insertions(+), 47 deletions(-)
>
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index 1bad37fc06b..d0391cd8718 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -17,7 +17,13 @@
>  import errno
>  import os
>  import re
> -from typing import Optional
> +from typing import (
> +    Dict,
> +    Iterator,
> +    List,
> +    Optional,
> +    Tuple,
> +)
>  
>  from .common import (
>      c_fname,
> @@ -29,31 +35,31 @@
>      mcgen,
>  )
>  from .schema import QAPISchemaObjectType, QAPISchemaVisitor
> +from .source import QAPISourceInfo

PATCH 03 has a similar cleanup.  Are there more?  Perhaps a separate
patch doing just this kind of cleanup would make sense.  Up to you.

[...]

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/7/20 4:07 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> This is a minor re-work of the entrypoint script. It isolates a
>> generate() method from the actual command-line mechanism.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> Tested-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>   1 file changed, 62 insertions(+), 23 deletions(-)
>>
>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>> index 541e8c1f55d..117b396a595 100644
>> --- a/scripts/qapi-gen.py
>> +++ b/scripts/qapi-gen.py
>> @@ -1,30 +1,77 @@
>>   #!/usr/bin/env python3
>> -# QAPI generator
>> -#
>> +
>>   # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>   # See the COPYING file in the top-level directory.
>>   
>> +"""
>> +QAPI Generator
>> +
>> +This script is the main entry point for generating C code from the QAPI schema.
>> +"""
>>   
>>   import argparse
>>   import re
>>   import sys
>>   
>>   from qapi.commands import gen_commands
>> +from qapi.error import QAPIError
>>   from qapi.events import gen_events
>>   from qapi.introspect import gen_introspect
>> -from qapi.schema import QAPIError, QAPISchema
>> +from qapi.schema import QAPISchema
>>   from qapi.types import gen_types
>>   from qapi.visit import gen_visit
>>   
>>   
>> -def main(argv):
>> +DEFAULT_OUTPUT_DIR = ''
>> +DEFAULT_PREFIX = ''
>> +
>> +
>> +def generate(schema_file: str,
>> +             output_dir: str,
>> +             prefix: str,
>> +             unmask: bool = False,
>> +             builtins: bool = False) -> None:
>> +    """
>> +    generate uses a given schema to produce C code in the target directory.
>> +
>> +    :param schema_file: The primary QAPI schema file.
>> +    :param output_dir: The output directory to store generated code.
>> +    :param prefix: Optional C-code prefix for symbol names.
>> +    :param unmask: Expose non-ABI names through introspection?
>> +    :param builtins: Generate code for built-in types?
>> +
>> +    :raise QAPIError: On failures.
>> +    """
>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>> +    if match.end() != len(prefix):
>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>> +            prefix[match.end()], prefix)
>> +        raise QAPIError('', None, msg)
>> +
>> +    schema = QAPISchema(schema_file)
>> +    gen_types(schema, output_dir, prefix, builtins)
>> +    gen_visit(schema, output_dir, prefix, builtins)
>> +    gen_commands(schema, output_dir, prefix)
>> +    gen_events(schema, output_dir, prefix)
>> +    gen_introspect(schema, output_dir, prefix, unmask)
>> +
>> +
>> +def main() -> int:
>> +    """
>> +    gapi-gen shell script entrypoint.
> 
> What's a "shell script entrypoint"?
> 
> Python docs talk about "when [...] run as a script":
> https://docs.python.org/3/library/__main__.html
> 
> Similar:
> https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts
> 

"entrypoint" is Python garble for a function that can be registered as a 
callable from the command line.

So in a theoretical setup.py, you'd do something like:

'entry_points': {
   'console_scripts': [
     'qapi-gen = qapi.main:main',
   ]
}

so when I say "shell script entrypoint", I am referring to a shell 
script (I mean: it has a shebang and can be executed by an interactive 
shell process) that calls the entrypoint.

Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to 
just generate that script.

(i.e. doing `pip install qemu.qapi` will install a 'qapi-gen' CLI script 
for you. this is how packages like sphinx create the 'sphinx-build' 
script, etc.)

>> +    Expects arguments via sys.argv, see --help for details.
>> +
>> +    :return: int, 0 on success, 1 on failure.
>> +    """
>>       parser = argparse.ArgumentParser(
>>           description='Generate code from a QAPI schema')
>>       parser.add_argument('-b', '--builtins', action='store_true',
>>                           help="generate code for built-in types")
>> -    parser.add_argument('-o', '--output-dir', action='store', default='',
>> +    parser.add_argument('-o', '--output-dir', action='store',
>> +                        default=DEFAULT_OUTPUT_DIR,
>>                           help="write output to directory OUTPUT_DIR")
>> -    parser.add_argument('-p', '--prefix', action='store', default='',
>> +    parser.add_argument('-p', '--prefix', action='store',
>> +                        default=DEFAULT_PREFIX,
>>                           help="prefix for symbols")
>>       parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>>                           dest='unmask',
>> @@ -32,25 +79,17 @@ def main(argv):
>>       parser.add_argument('schema', action='store')
>>       args = parser.parse_args()
>>   
>> -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
>> -    if match.end() != len(args.prefix):
>> -        print("%s: 'funny character '%s' in argument of --prefix"
>> -              % (sys.argv[0], args.prefix[match.end()]),
>> -              file=sys.stderr)
>> -        sys.exit(1)
>> -
>>       try:
>> -        schema = QAPISchema(args.schema)
>> +        generate(args.schema,
>> +                 output_dir=args.output_dir,
>> +                 prefix=args.prefix,
>> +                 unmask=args.unmask,
>> +                 builtins=args.builtins)
>>       except QAPIError as err:
>> -        print(err, file=sys.stderr)
>> -        exit(1)
>> -
>> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
>> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
>> -    gen_commands(schema, args.output_dir, args.prefix)
>> -    gen_events(schema, args.output_dir, args.prefix)
>> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
>> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
>> +        return 1
>> +    return 0
>>   
>>   
>>   if __name__ == '__main__':
>> -    main(sys.argv)
>> +    sys.exit(main())
> 
> What does sys.exit() really buy us here?  I'm asking because both spots
> in the Python docs I referenced above do without.
> 

It just pushes the sys.exit out of the main function so it can be 
invoked by other machinery. (And takes the return code from main and 
turns it into the return code for the process.)

I don't think it winds up mattering for simple "console_script" entry 
points, but you don't want the called function to exit and deny the 
caller the chance to do their own tidying post-call.

You've already offered a "YAGNI", but it's just the convention I tend to 
stick to for how to structure entry points.

--js

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/7/20 4:12 AM, Markus Armbruster wrote:
> I keep stumbling over things in later patches that turn out to go back
> to this one.
> 
> John Snow <jsnow@redhat.com> writes:
> 
>> This is a minor re-work of the entrypoint script. It isolates a
>> generate() method from the actual command-line mechanism.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> Tested-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>   1 file changed, 62 insertions(+), 23 deletions(-)
>>
>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>> index 541e8c1f55d..117b396a595 100644
>> --- a/scripts/qapi-gen.py
>> +++ b/scripts/qapi-gen.py
>> @@ -1,30 +1,77 @@
>>   #!/usr/bin/env python3
>> -# QAPI generator
>> -#
>> +
>>   # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>   # See the COPYING file in the top-level directory.
>>   
>> +"""
>> +QAPI Generator
>> +
>> +This script is the main entry point for generating C code from the QAPI schema.
> 
> PEP 8: For flowing long blocks of text with fewer structural
> restrictions (docstrings or comments), the line length should be limited
> to 72 characters.
> 

Eugh. OK, but I don't have a good way to check or enforce this, 
admittedly. I have to change my emacs settings to understand this when I 
hit the reflow key. I don't know if the python mode has a context-aware 
reflow length.

("I don't disagree, but I'm not immediately sure right now how I will 
make sure I, or anyone else, complies with this. Low priority as a result?")

>> +"""
>>   
>>   import argparse
>>   import re
>>   import sys
>>   
>>   from qapi.commands import gen_commands
>> +from qapi.error import QAPIError
>>   from qapi.events import gen_events
>>   from qapi.introspect import gen_introspect
>> -from qapi.schema import QAPIError, QAPISchema
>> +from qapi.schema import QAPISchema
>>   from qapi.types import gen_types
>>   from qapi.visit import gen_visit
>>   
>>   
>> -def main(argv):
>> +DEFAULT_OUTPUT_DIR = ''
>> +DEFAULT_PREFIX = ''
>> +
>> +
>> +def generate(schema_file: str,
>> +             output_dir: str,
>> +             prefix: str,
>> +             unmask: bool = False,
>> +             builtins: bool = False) -> None:
>> +    """
>> +    generate uses a given schema to produce C code in the target directory.
> 
> PEP 257: The docstring is a phrase ending in a period.  It prescribes
> the function or method's effect as a command ("Do this", "Return that"),
> not as a description; e.g. don't write "Returns the pathname ...".
> 
> Suggest
> 
>         Generate C code for the given schema into the target directory.
> 

OK. I don't mind trying to foster a consistent tone. I clearly didn't. I 
will add a note to my style guide todo.

I give you permission to change the voice in any of my docstrings, or to 
adjust the phrasing to be more technically accurate as you see fit. You 
are the primary maintainer of this code, of course, and you will know best.

It will be quicker to give you full permission to just change any of the 
docstrings as you see fit than it will be to play review-respin ping-pong.

>> +
>> +    :param schema_file: The primary QAPI schema file.
>> +    :param output_dir: The output directory to store generated code.
>> +    :param prefix: Optional C-code prefix for symbol names.
>> +    :param unmask: Expose non-ABI names through introspection?
>> +    :param builtins: Generate code for built-in types?
>> +
>> +    :raise QAPIError: On failures.
>> +    """
> [...]
> 

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/7/20 3:54 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/6/20 7:51 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> This is a minor re-work of the entrypoint script. It isolates a
>>>> generate() method from the actual command-line mechanism.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>    scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>>    1 file changed, 62 insertions(+), 23 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>>> index 541e8c1f55d..117b396a595 100644
>>>> --- a/scripts/qapi-gen.py
>>>> +++ b/scripts/qapi-gen.py
>>>> @@ -1,30 +1,77 @@
>>>>    #!/usr/bin/env python3
>>>> -# QAPI generator
>>>> -#
>>>> +
>>>>    # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>>    # See the COPYING file in the top-level directory.
>>>>    +"""
>>>> +QAPI Generator
>>>> +
>>>> +This script is the main entry point for generating C code from the QAPI schema.
>>>> +"""
>>>>      import argparse
>>>>    import re
>>>>    import sys
>>>>      from qapi.commands import gen_commands
>>>> +from qapi.error import QAPIError
>>>>    from qapi.events import gen_events
>>>>    from qapi.introspect import gen_introspect
>>>> -from qapi.schema import QAPIError, QAPISchema
>>>> +from qapi.schema import QAPISchema
>>>>    from qapi.types import gen_types
>>>>    from qapi.visit import gen_visit
>>> Unrelated cleanup.  Okay.
>>>
>>>>      
>>>> -def main(argv):
>>>> +DEFAULT_OUTPUT_DIR = ''
>>>> +DEFAULT_PREFIX = ''
>>>> +
>>>> +
>>>> +def generate(schema_file: str,
>>>> +             output_dir: str,
>>>> +             prefix: str,
>>>> +             unmask: bool = False,
>>>> +             builtins: bool = False) -> None:
>>>> +    """
>>>> +    generate uses a given schema to produce C code in the target directory.
>>>> +
>>>> +    :param schema_file: The primary QAPI schema file.
>>>> +    :param output_dir: The output directory to store generated code.
>>>> +    :param prefix: Optional C-code prefix for symbol names.
>>>> +    :param unmask: Expose non-ABI names through introspection?
>>>> +    :param builtins: Generate code for built-in types?
>>>> +
>>>> +    :raise QAPIError: On failures.
>>>> +    """
>>>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>>>> +    if match.end() != len(prefix):
>>>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>>>> +            prefix[match.end()], prefix)
>>>> +        raise QAPIError('', None, msg)
>>> Uh...
>>>       $ python3 scripts/qapi-gen.py --prefix=@ x
>>>       scripts/qapi-gen.py: : funny character '@' in prefix '@'
>>> Unwanted " :".
>>> This is due to a hack: you pass '' for info (*quack*).  Everything
>>> else
>>> passes QAPISourceInfo (I believe).
>>>
>>
>> Quack indeed - why does our base error class require so much
>> information from a specific part of the generation process?
> 
> Because it's not "a base error class", it's a base error class for the
> QAPI schema compiler frontend.
> 

Well. It's the base for every error we /had/.

>> Ah, someone changes this in part 4 so that we have a more generic
>> error class to use as a base when we are missing such information.
> 
> Evolving it to satisfy a need for a more widely usable error class is
> okay.
> 

Yep. It's helpful to keep a very generic form on which we grow other 
errors from, so that things like the entry point can be written legibly.

>> You are witnessing some more future-bleed.
>>> Is it really a good idea to do this in generate?  It's not about
>>> generating code, it's about validating a CLI option.
>>>
>>
>> One might also ask: Is it a good idea to only validate this on a
>> frontend, and not in the implementation?
> 
> Yes, because that's where you can emit the better error message more
> easily.
> 
>      $ python3 scripts/qapi-gen.py --prefix=@ x
>      scripts/qapi-gen.py: 'funny character '@' in argument of --prefix
> 
> is better than
> 
>      $ python3 scripts/qapi-gen.py --prefix=@ x
>      scripts/qapi-gen.py: funny character '@' in prefix '@'
> 
> In generate(), the knowledge where the offending prefix value comes from
> is no longer available.
> 
> To emit this error message, you'd have to raise a sufficiently distinct
> error in generate, catch it in main(), then put the error message
> together somehow.  Bah.
> 
> Aside: there's a stray ' in the old error message.
> 
>> The idea here was to create a function that could be used in a script
>> (for tests, debugging interfaces, other python packages) to do all of
>> the same things that the CLI tool did, just sans the actual CLI.
> 
> YAGNI.
> 

It's useful for testing and debugging to be able to just call it outside 
of the CLI, though. Maybe you won't use it, but I will.

I could always add the prefix check into a tiny function and give the 
good error message in main(), and just assert in generate() if you 
insist on the slightly more specific error message from the CLI script.

>> Wouldn't make sense to allow garbage to flow in from one interface but
>> not the other; so the check is here.
> 
> "@prefix is sane" is a precondition of generate().
> 
> When there's a real risk of preconditions getting violated, or readers
> getting confused about preconditions, check them with assert.
> 
>>>> +
>>>> +    schema = QAPISchema(schema_file)
>>>> +    gen_types(schema, output_dir, prefix, builtins)
>>>> +    gen_visit(schema, output_dir, prefix, builtins)
>>>> +    gen_commands(schema, output_dir, prefix)
>>>> +    gen_events(schema, output_dir, prefix)
>>>> +    gen_introspect(schema, output_dir, prefix, unmask)
>>>> +
>>>> +
>>>> +def main() -> int:
>>>> +    """
>>>> +    gapi-gen shell script entrypoint.
>>>> +    Expects arguments via sys.argv, see --help for details.
>>>> +
>>>> +    :return: int, 0 on success, 1 on failure.
>>>> +    """
>>>>        parser = argparse.ArgumentParser(
>>>>            description='Generate code from a QAPI schema')
>>>>        parser.add_argument('-b', '--builtins', action='store_true',
>>>>                            help="generate code for built-in types")
>>>> -    parser.add_argument('-o', '--output-dir', action='store', default='',
>>>> +    parser.add_argument('-o', '--output-dir', action='store',
>>>> +                        default=DEFAULT_OUTPUT_DIR,
>>>>                            help="write output to directory OUTPUT_DIR")
>>>> -    parser.add_argument('-p', '--prefix', action='store', default='',
>>>> +    parser.add_argument('-p', '--prefix', action='store',
>>>> +                        default=DEFAULT_PREFIX,
>>>>                            help="prefix for symbols")
>>> I don't like the changes to default=, because:
>>> 1. They are only losely related to the patch's purpose.
>>>
>>
>> Subjective, but OK.
>>
>>> 2. They split the definition of the CLI: most of it is here, except for
>>> defaults, which are defined elsewhere.
>>>
>>
>> All of it is in main.py, though! If you were to, say, move generate()
>> elsewhere, it'd look pretty compact as just the CLI frontend, no?
> 
> Same statement is more compact than same screenful is more compact than
> same file :)
> 
>>> 3. The defaults will not change, and nothing else uses the constants.
>>>
>>
>> But, fine. Cleber had the same comment but I wasn't fully on-board,
>> but two folks saying the same thing ...
>>
>>>>        parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>>>>                            dest='unmask',
>>>> @@ -32,25 +79,17 @@ def main(argv):
>>>>        parser.add_argument('schema', action='store')
>>>>        args = parser.parse_args()
>>>>    -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?',
>>>> args.prefix)
>>>> -    if match.end() != len(args.prefix):
>>>> -        print("%s: 'funny character '%s' in argument of --prefix"
>>>> -              % (sys.argv[0], args.prefix[match.end()]),
>>>> -              file=sys.stderr)
>>>> -        sys.exit(1)
>>>> -
>>>>        try:
>>>> -        schema = QAPISchema(args.schema)
>>>> +        generate(args.schema,
>>>> +                 output_dir=args.output_dir,
>>>> +                 prefix=args.prefix,
>>>> +                 unmask=args.unmask,
>>>> +                 builtins=args.builtins)
>>>>        except QAPIError as err:
>>>> -        print(err, file=sys.stderr)
>>>> -        exit(1)
>>>> -
>>>> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
>>>> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
>>>> -    gen_commands(schema, args.output_dir, args.prefix)
>>>> -    gen_events(schema, args.output_dir, args.prefix)
>>>> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
>>>> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
>>>> +        return 1
>>>> +    return 0
>>> Subtle change: you move the gen_FOO() into the try ... except.
>>> Okay;
>>> they don't raise QAPIError, but perhaps worth a mention in the commit
>>> message.
>>>
>>
>> Forbidden future knowledge; I intend them to.
> 
> I don't mind the move.
> 
>>>>      
>>>>    if __name__ == '__main__':
>>>> -    main(sys.argv)
>>>> +    sys.exit(main())
>>> "Python was designed to be easy to understand and fun to use."
>>> Ha ha ha.
>>>
>>
>> I mean, I'm having fun, aren't you?
> 
> So many kinds of fun!  The fun I'm having with this patch hunk is
> mocking "easy and fun" Python for requiring such an elaborate menuett
> just to express "can run as program".
> 
> This emperor has no clothes, either.  And that's funny, isn't it?
> 

Re: [PATCH v5 08/36] qapi: delint using flake8

[John Snow]

On 10/7/20 4:19 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Petty style guide fixes and line length enforcement.  Not a big win, not
>> a big loss, but flake8 passes 100% on the qapi module, which gives us an
>> easy baseline to enforce hereafter.
>>
>> A note on the flake8 exception: flake8 will warn on *any* bare except,
>> but pylint's is context-aware and will suppress the warning if you
>> re-raise the exception.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/.flake8     |  2 ++
>>   scripts/qapi/commands.py |  3 ++-
>>   scripts/qapi/schema.py   |  8 +++++---
>>   scripts/qapi/visit.py    | 16 +++++++++++-----
>>   4 files changed, 20 insertions(+), 9 deletions(-)
>>   create mode 100644 scripts/qapi/.flake8
>>
>> diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
>> new file mode 100644
>> index 00000000000..6b158c68b84
>> --- /dev/null
>> +++ b/scripts/qapi/.flake8
>> @@ -0,0 +1,2 @@
>> +[flake8]
>> +extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
>> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
>> index 64ed5278f93..5dc2f5a9fa8 100644
>> --- a/scripts/qapi/commands.py
>> +++ b/scripts/qapi/commands.py
>> @@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type):
>>   def gen_marshal_output(ret_type):
>>       return mcgen('''
>>   
>> -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
>> +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
>> +                                QObject **ret_out, Error **errp)
>>   {
>>       Visitor *v;
>>   
>> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> index 093f7a38d88..cfc52e1ae44 100644
>> --- a/scripts/qapi/schema.py
>> +++ b/scripts/qapi/schema.py
>> @@ -536,7 +536,7 @@ def set_defined_in(self, name):
>>               v.set_defined_in(name)
>>   
>>       def check(self, schema, seen):
>> -        if not self.tag_member: # flat union
>> +        if not self.tag_member:  # flat union
>>               self.tag_member = seen.get(c_name(self._tag_name))
>>               base = "'base'"
>>               # Pointing to the base type when not implicit would be
>> @@ -821,7 +821,7 @@ def __init__(self, fname):
>>           self._entity_dict = {}
>>           self._module_dict = OrderedDict()
>>           self._schema_dir = os.path.dirname(fname)
>> -        self._make_module(None) # built-ins
>> +        self._make_module(None)  # built-ins
>>           self._make_module(fname)
>>           self._predefining = True
>>           self._def_predefineds()
>> @@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
>>               # But it's not tight: the disjunction need not imply it.  We
>>               # may end up compiling useless wrapper types.
>>               # TODO kill simple unions or implement the disjunction
>> -            assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
>> +
>> +            # pylint: disable=protected-access
>> +            assert (ifcond or []) == typ._ifcond
>>           else:
>>               self._def_entity(QAPISchemaObjectType(
>>                   name, info, None, ifcond, None, None, members, None))
>> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
>> index ea277e7704b..808410d6f1b 100644
>> --- a/scripts/qapi/visit.py
>> +++ b/scripts/qapi/visit.py
>> @@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False):
>>       if not scalar:
>>           c_type += '*'
>>       return mcgen('''
>> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
>> +
>> +bool visit_type_%(c_name)s(Visitor *v, const char *name,
>> +                 %(c_type)sobj, Error **errp);
>>   ''',
>>                    c_name=c_name(name), c_type=c_type)
>>   
>> @@ -125,7 +127,8 @@ def gen_visit_object_members(name, base, members, variants):
>>   def gen_visit_list(name, element_type):
>>       return mcgen('''
>>   
>> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
>> +bool visit_type_%(c_name)s(Visitor *v, const char *name,
>> +                 %(c_name)s **obj, Error **errp)
>>   {
>>       bool ok = false;
>>       %(c_name)s *tail;
>> @@ -158,7 +161,8 @@ def gen_visit_list(name, element_type):
>>   def gen_visit_enum(name):
>>       return mcgen('''
>>   
>> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
>> +bool visit_type_%(c_name)s(Visitor *v, const char *name,
>> +                 %(c_name)s *obj, Error **errp)
>>   {
>>       int value = *obj;
>>       bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
>> @@ -172,7 +176,8 @@ def gen_visit_enum(name):
>>   def gen_visit_alternate(name, variants):
>>       ret = mcgen('''
>>   
>> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
>> +bool visit_type_%(c_name)s
>> +    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
> 
> Please break this line the same way as the ones above:
> 
>     bool visit_type_%(c_name)s(Visitor *v, const char *name,
>                      %(c_name)s **obj, Error **errp)
> 
>>   {
>>       bool ok = false;
>>   
>> @@ -247,7 +252,8 @@ def gen_visit_alternate(name, variants):
>>   def gen_visit_object(name, base, members, variants):
>>       return mcgen('''
>>   
>> -bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
>> +bool visit_type_%(c_name)s
>> +    (Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
>>   {
>>       bool ok = false;
> 
> Likewise.
> 

Whupps. Bad conflict resolution on my part. Didn't mean to do this.

--js

Re: [PATCH v5 15/36] qapi/common.py: add type hint annotations

[John Snow]

On 10/7/20 5:03 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Annotations do not change runtime behavior.
>> This commit *only* adds annotations.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/common.py | 27 ++++++++++++++++-----------
>>   1 file changed, 16 insertions(+), 11 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index 338adedef4f..74a2c001ed9 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -12,6 +12,7 @@
>>   # See the COPYING file in the top-level directory.
>>   
>>   import re
>> +from typing import Optional, Sequence
>>   
>>   
>>   EATSPACE = '\033EATSPACE.'
>> @@ -22,7 +23,7 @@
>>   # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>>   # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>>   # ENUM24_Name -> ENUM24_NAME
>> -def camel_to_upper(value):
>> +def camel_to_upper(value: str) -> str:
>>       c_fun_str = c_name(value, False)
>>       if value.isupper():
>>           return c_fun_str
>> @@ -41,7 +42,9 @@ def camel_to_upper(value):
>>       return new_name.lstrip('_').upper()
>>   
>>   
>> -def c_enum_const(type_name, const_name, prefix=None):
>> +def c_enum_const(type_name: str,
>> +                 const_name: str,
>> +                 prefix: Optional[str] = None) -> str:
>>       if prefix is not None:
>>           type_name = prefix
>>       return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>> @@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
>>   # into substrings of a generated C function name.
>>   # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>>   # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>> -def c_name(name, protect=True):
>> +def c_name(name: str, protect: bool = True) -> str:
>>       # ANSI X3J11/88-090, 3.1.1
>>       c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>>                        'default', 'do', 'double', 'else', 'enum', 'extern',
>> @@ -131,24 +134,24 @@ def decrease(self, amount: int = 4) -> None:
>>   
>>   # Generate @code with @kwds interpolated.
>>   # Obey indent, and strip EATSPACE.
>> -def cgen(code, **kwds):
>> +def cgen(code: str, **kwds: object) -> str:
>>       raw = code % kwds
>>       if indent:
>>           raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
>>       return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>>   
>>   
>> -def mcgen(code, **kwds):
>> +def mcgen(code: str, **kwds: object) -> str:
>>       if code[0] == '\n':
>>           code = code[1:]
>>       return cgen(code, **kwds)
>>   
>>   
>> -def c_fname(filename):
>> +def c_fname(filename: str) -> str:
>>       return re.sub(r'[^A-Za-z0-9_]', '_', filename)
>>   
>>   
>> -def guardstart(name):
>> +def guardstart(name: str) -> str:
>>       return mcgen('''
>>   #ifndef %(name)s
>>   #define %(name)s
>> @@ -157,7 +160,7 @@ def guardstart(name):
>>                    name=c_fname(name).upper())
>>   
>>   
>> -def guardend(name):
>> +def guardend(name: str) -> str:
>>       return mcgen('''
>>   
>>   #endif /* %(name)s */
>> @@ -165,7 +168,7 @@ def guardend(name):
>>                    name=c_fname(name).upper())
>>   
>>   
>> -def gen_if(ifcond):
>> +def gen_if(ifcond: Sequence[str]) -> str:
>>       ret = ''
>>       for ifc in ifcond:
>>           ret += mcgen('''
>> @@ -174,7 +177,7 @@ def gen_if(ifcond):
>>       return ret
>>   
>>   
>> -def gen_endif(ifcond):
>> +def gen_endif(ifcond: Sequence[str]) -> str:
>>       ret = ''
>>       for ifc in reversed(ifcond):
>>           ret += mcgen('''
>> @@ -183,7 +186,9 @@ def gen_endif(ifcond):
>>       return ret
>>   
>>   
>> -def build_params(arg_type, boxed, extra=None):
>> +def build_params(arg_type,
>> +                 boxed: bool,
>> +                 extra: Optional[str] = None) -> str:
>>       ret = ''
>>       sep = ''
>>       if boxed:
> 
> @arg_type is the only parameter left unannotated.  Scratching head...
> aha:
> 
>      qapi/common.py: move build_params into gen.py
>      
>      Including it in common.py creates a circular import dependency; schema
>      relies on common, but common.build_params requires a type annotation
>      from schema. To type this properly, it needs to be moved outside the
>      cycle.
> 
> Let's amend the commit message:
> 
>      Note that build_params() cannot be fully annotated due to import
>      dependency issues.  The commit after next will take care of it.
> 
> If we swap the next two commits, the fix follows immediately.  Better,
> but only worth it if swapping is pretty much effortless.
> 

The reason they're not flipped was to avoid giving the impression that 
docstrings were what was preventing us from enabling the mypy type checking.

The reason I don't flip the comments and the type hints is because 
that's a lot of rebase pain for perceptibly little benefit.

I amended the commit message.

Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate

[John Snow]

On 10/7/20 5:14 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> As docstrings, they'll show up in documentation and IDE help.
>>
>> The docstring style being targeted is the Sphinx documentation
>> style. Sphinx uses an extension of ReST with "domains". We use the
>> (implicit) Python domain, which supports a number of custom "info
>> fields". Those info fields are documented here:
>> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
>>
>> Primarily, we use `:param X: descr`, `:return[s]: descr`, and `:raise[s]
>> Z: when`. Everything else is the Sphinx dialect of ReST.
>>
>> (No, nothing checks or enforces this style that I am aware of. Sphinx
>> either chokes or succeeds, but does not enforce a standard of what is
>> otherwise inside the docstring. Pycharm does highlight when your param
>> fields are not aligned with the actual fields present. It does not
>> highlight missing return or exception statements. There is no existing
>> style guide I am aware of that covers a standard for a minimally
>> acceptable docstring. I am debating writing one.)
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
>>   1 file changed, 39 insertions(+), 14 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index 74a2c001ed9..0ef38ea5fe0 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -15,15 +15,24 @@
>>   from typing import Optional, Sequence
>>   
>>   
>> +#: Sentinel value that causes all space to its right to be removed.
> 
> What's the purpose of : after # ?
> 

Documents this name in Sphinx. We had a small discussion about it, I 
think; "Does using this special form or the docstring make the comment 
visible in any IDE?" (No.)

There's no Python-AST way to document these, but there is a Sphinx way 
to document them, so I did that.

(Doing it like this allows `EATSPACE` to be used as a cross-reference.)

> I'm not sure this is a "sentinel value".  Wikipedia:
> 
>      In computer programming, a sentinel value (also referred to as a
>      flag value, trip value, rogue value, signal value, or dummy data)[1]
>      is a special value in the context of an algorithm which uses its
>      presence as a condition of termination, typically in a loop or
>      recursive algorithm.
> 
>      https://en.wikipedia.org/wiki/Sentinel_value
> 

I really should try to learn English as a second language so I know what 
any of the words I use mean, I guess. I had slipped to a less strict 
usage where it meant more like "placeholder".

> Perhaps
> 
>     # Magic string value that gets removed along with all space to the
>     # right.
> 

This can be written on one line if we gently disregard the 72 column 
limit. (Maybe you already did when you wrote it and my client wrapped 
it. Who knows!)

>>   EATSPACE = '\033EATSPACE.'
>>   POINTER_SUFFIX = ' *' + EATSPACE
>>   _C_NAME_TRANS = str.maketrans('.-', '__')
>>   
>>   
>> -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>> -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>> -# ENUM24_Name -> ENUM24_NAME
>>   def camel_to_upper(value: str) -> str:
>> +    """
>> +    Converts CamelCase to CAMEL_CASE.
>> +
>> +    Examples:
>> +      ENUMName -> ENUM_NAME
>> +      EnumName1 -> ENUM_NAME1
>> +      ENUM_NAME -> ENUM_NAME
>> +      ENUM_NAME1 -> ENUM_NAME1
>> +      ENUM_Name2 -> ENUM_NAME2
>> +      ENUM24_Name -> ENUM24_NAME
>> +    """
> 
> I wonder whether these indented lines get wrapped into one
> unintelligible parapgraph.
> 
> Have you eyeballed the output of Sphinx?
> 

Eyeballed, but didn't validate this specific one. Yeah, it's nonsense.

Examples::

     ENUMName -> ENUM_NAME

etc. works better.

>>       c_fun_str = c_name(value, False)
>>       if value.isupper():
>>           return c_fun_str
>> @@ -45,21 +54,33 @@ def camel_to_upper(value: str) -> str:
>>   def c_enum_const(type_name: str,
>>                    const_name: str,
>>                    prefix: Optional[str] = None) -> str:
>> +    """
>> +    Generate a C enumeration constant name.
>> +
>> +    :param type_name: The name of the enumeration.
>> +    :param const_name: The name of this constant.
>> +    :param prefix: Optional, prefix that overrides the type_name.
>> +    """
>>       if prefix is not None:
>>           type_name = prefix
>>       return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>>   
>>   
>> -# Map @name to a valid C identifier.
>> -# If @protect, avoid returning certain ticklish identifiers (like
>> -# C keywords) by prepending 'q_'.
>> -#
>> -# Used for converting 'name' from a 'name':'type' qapi definition
>> -# into a generated struct member, as well as converting type names
>> -# into substrings of a generated C function name.
>> -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>> -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>>   def c_name(name: str, protect: bool = True) -> str:
>> +    """
>> +    Map ``name`` to a valid C identifier.
>> +
>> +    Used for converting 'name' from a 'name':'type' qapi definition
>> +    into a generated struct member, as well as converting type names
>> +    into substrings of a generated C function name.
>> +
>> +    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>> +    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>> +
>> +    :param name: The name to map.
>> +    :param protect: If true, avoid returning certain ticklish identifiers
>> +                    (like C keywords) by prepending ``q_``.
>> +    """
>>       # ANSI X3J11/88-090, 3.1.1
>>       c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>>                        'default', 'do', 'double', 'else', 'enum', 'extern',
>> @@ -129,12 +150,16 @@ def decrease(self, amount: int = 4) -> None:
>>           self._level -= amount
>>   
>>   
>> +#: Global, current indent level for code generation.
>>   indent = Indentation()
>>   
>>   
>> -# Generate @code with @kwds interpolated.
>> -# Obey indent, and strip EATSPACE.
>>   def cgen(code: str, **kwds: object) -> str:
>> +    """
>> +    Generate ``code`` with ``kwds`` interpolated.
>> +
>> +    Obey `indent`, and strip `EATSPACE`.
>> +    """
>>       raw = code % kwds
>>       if indent:
>>           raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)

Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py

[John Snow]

On 10/7/20 5:21 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Including it in common.py creates a circular import dependency; schema
>> relies on common, but common.build_params requires a type annotation
>> from schema. To type this properly, it needs to be moved outside the
>> cycle.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/commands.py |  9 +++++++--
>>   scripts/qapi/common.py   | 23 -----------------------
>>   scripts/qapi/events.py   |  9 ++-------
>>   scripts/qapi/gen.py      | 31 +++++++++++++++++++++++++++++--
>>   4 files changed, 38 insertions(+), 34 deletions(-)
>>
>> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
>> index 5dc2f5a9fa8..f67393f8713 100644
>> --- a/scripts/qapi/commands.py
>> +++ b/scripts/qapi/commands.py
>> @@ -13,8 +13,13 @@
>>   See the COPYING file in the top-level directory.
>>   """
>>   
>> -from .common import build_params, c_name, mcgen
>> -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
>> +from .common import c_name, mcgen
>> +from .gen import (
>> +    QAPIGenCCode,
>> +    QAPISchemaModularCVisitor,
>> +    build_params,
>> +    ifcontext,
>> +)
>>   
>>   
>>   def gen_command_decl(name, arg_type, boxed, ret_type):
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index 0ef38ea5fe0..9ab0685cc51 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
>>   #endif /* %(cond)s */
>>   ''', cond=ifc)
>>       return ret
>> -
>> -
>> -def build_params(arg_type,
>> -                 boxed: bool,
>> -                 extra: Optional[str] = None) -> str:
>> -    ret = ''
>> -    sep = ''
>> -    if boxed:
>> -        assert arg_type
>> -        ret += '%s arg' % arg_type.c_param_type()
>> -        sep = ', '
>> -    elif arg_type:
>> -        assert not arg_type.variants
>> -        for memb in arg_type.members:
>> -            ret += sep
>> -            sep = ', '
>> -            if memb.optional:
>> -                ret += 'bool has_%s, ' % c_name(memb.name)
>> -            ret += '%s %s' % (memb.type.c_param_type(),
>> -                              c_name(memb.name))
>> -    if extra:
>> -        ret += sep + extra
>> -    return ret if ret else 'void'
>> diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
>> index 6b3afa14d72..f840a62ed92 100644
>> --- a/scripts/qapi/events.py
>> +++ b/scripts/qapi/events.py
>> @@ -12,13 +12,8 @@
>>   See the COPYING file in the top-level directory.
>>   """
>>   
>> -from .common import (
>> -    build_params,
>> -    c_enum_const,
>> -    c_name,
>> -    mcgen,
>> -)
>> -from .gen import QAPISchemaModularCVisitor, ifcontext
>> +from .common import c_enum_const, c_name, mcgen
>> +from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
>>   from .schema import QAPISchemaEnumMember
>>   from .types import gen_enum, gen_enum_lookup
>>   
>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> index 1fed712b43b..f2e2746fea5 100644
>> --- a/scripts/qapi/gen.py
>> +++ b/scripts/qapi/gen.py
>> @@ -2,9 +2,11 @@
>>   #
>>   # QAPI code generation
>>   #
>> -# Copyright (c) 2018-2019 Red Hat Inc.
>> +# Copyright IBM, Corp. 2011
>> +# Copyright (c) 2013-2019 Red Hat Inc.
>>   #
>>   # Authors:
>> +#  Anthony Liguori <aliguori@us.ibm.com>
>>   #  Markus Armbruster <armbru@redhat.com>
>>   #  Marc-André Lureau <marcandre.lureau@redhat.com>
>>   #
> 
> The code you move into this file is actually Red Hat's:
> 
> $ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date'
> Author: Markus Armbruster <armbru@redhat.com>
> Date:   Wed Aug 15 21:37:36 2018 +0800
> Author: Marc-André Lureau <marcandre.lureau@redhat.com>
> Date:   Thu Jun 1 16:41:41 2017 +0400
> Author: Eric Blake <eblake@redhat.com>
> Date:   Wed Jul 13 21:50:20 2016 -0600
> Author: Eric Blake <eblake@redhat.com>
> Date:   Wed Jul 13 21:50:19 2016 -0600
> Author: Eric Blake <eblake@redhat.com>
> Date:   Thu Mar 17 16:48:28 2016 -0600
> Author: Markus Armbruster <armbru@redhat.com>
> Date:   Wed Sep 16 13:06:20 2015 +0200
> 
> So the correct update is simply
> 
>     #
>     # QAPI code generation
>     #
>    -# Copyright (c) 2018-2019 Red Hat Inc.
>    +# Copyright (c) 2015-2019 Red Hat Inc.
>     #
>     # Authors:
>     #  Markus Armbruster <armbru@redhat.com>
>     #  Marc-André Lureau <marcandre.lureau@redhat.com>
>     #
> 

OK, thank you. I did the dumbest possibly correct thing. My appetite for 
doing line ownership analysis is ... low.

>> @@ -15,16 +17,18 @@
>>   import errno
>>   import os
>>   import re
>> +from typing import Optional
>>   
>>   from .common import (
>>       c_fname,
>> +    c_name,
>>       gen_endif,
>>       gen_if,
>>       guardend,
>>       guardstart,
>>       mcgen,
>>   )
>> -from .schema import QAPISchemaVisitor
>> +from .schema import QAPISchemaObjectType, QAPISchemaVisitor
>>   
>>   
>>   class QAPIGen:
>> @@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
>>       return out
>>   
>>   
>> +def build_params(arg_type: Optional[QAPISchemaObjectType],
>> +                 boxed: bool,
>> +                 extra: Optional[str] = None) -> str:
>> +    ret = ''
>> +    sep = ''
>> +    if boxed:
>> +        assert arg_type
>> +        ret += '%s arg' % arg_type.c_param_type()
>> +        sep = ', '
>> +    elif arg_type:
>> +        assert not arg_type.variants
>> +        for memb in arg_type.members:
>> +            ret += sep
>> +            sep = ', '
>> +            if memb.optional:
>> +                ret += 'bool has_%s, ' % c_name(memb.name)
>> +            ret += '%s %s' % (memb.type.c_param_type(),
>> +                              c_name(memb.name))
>> +    if extra:
>> +        ret += sep + extra
>> +    return ret if ret else 'void'
>> +
>> +
>>   class QAPIGenCCode(QAPIGen):
>>   
>>       def __init__(self, fname):

Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline

[John Snow]

On 10/7/20 5:25 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Fix two very minor issues,
> 
> What are the two issues?  I'm asking because I can see only one.
> 

Bitrot! The other one got fixed elsewhere and rebase didn't conflict, so 
I didn't update the commit message.

>>                             and then establish a mypy type-checking
>> baseline.
> 
> Any particular reason this can't go before PATCH 15 "qapi/common.py: add
> type hint annotations".
> 

Just the way the chips fell. common.py does not check until this very 
commit.

Adding a mypy config can be shuffled up earlier and earlier by 
determining which checks to ignore and adding them, but I'm not sure who 
that benefits.

>> Like pylint, this should be run from the folder above:
>>
>>   > mypy --config-file=qapi/mypy.ini qapi/
>>
>> This is designed and tested for mypy 0.770 or greater.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Tested-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> Tested-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/mypy.ini  | 60 ++++++++++++++++++++++++++++++++++++++++++
>>   scripts/qapi/schema.py |  3 ++-
>>   2 files changed, 62 insertions(+), 1 deletion(-)
>>   create mode 100644 scripts/qapi/mypy.ini
>>
>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>> new file mode 100644
>> index 00000000000..00fac15dc6e
>> --- /dev/null
>> +++ b/scripts/qapi/mypy.ini
>> @@ -0,0 +1,60 @@
>> +[mypy]
>> +strict = True
>> +strict_optional = False
>> +disallow_untyped_calls = False
>> +python_version = 3.6
>> +
>> +[mypy-qapi.commands]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.error]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.events]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.expr]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.gen]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.introspect]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.parser]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.schema]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.source]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.types]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.visit]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> index 74c6b96d391..483b4b68dff 100644
>> --- a/scripts/qapi/schema.py
>> +++ b/scripts/qapi/schema.py
>> @@ -17,6 +17,7 @@
>>   from collections import OrderedDict
>>   import os
>>   import re
>> +from typing import Optional
>>   
>>   from .common import POINTER_SUFFIX, c_name
>>   from .error import QAPIError, QAPISemError
>> @@ -25,7 +26,7 @@
>>   
>>   
>>   class QAPISchemaEntity:
>> -    meta = None
>> +    meta: Optional[str] = None
>>   
>>       def __init__(self, name, info, doc, ifcond=None, features=None):
>>           assert name is None or isinstance(name, str)

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[John Snow]

On 10/7/20 7:32 AM, Markus Armbruster wrote:
> Ignorant question: what's the difference between -> None (like here) and
> nothing (like __init__() above?

This came up in Cleber's review, too.

Mypy supports a gradual typing paradigm; it is designed to facilitate 
what we are doing here: the gradual adding of types until we are able to 
enforce static typing everywhere.

To that end, mypy uses a simple heuristic to determine if a function is 
"typed" or "untyped": did you use any type annotations for it?

Meanwhile, __init__ never returns anything. You do not need to annotate 
its return type. mypy knows what the return type is and must be. In the 
case of __init__ with no parameters, it is both untyped and strictly typed!

Annotating the return type for no-parameter init methods convinces mypy 
that this is a strictly typed method. It doesn't do this automatically, 
because doing so might enable more checks than you were ready for simply 
because mypy was able to accurately surmise the typing of just __init__.

So it's just a little flip switch to enable strict typing, really.

--js

https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[John Snow]

On 10/7/20 7:49 AM, Markus Armbruster wrote:
> Looks like commit message of PATCH 24 has an answer.
> 
> Copy to all the commits that omit -> None similarly?

Probably not needed.

It's covered by the class basics section in the mypy manual; 
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

and if you should happen to omit annotations for __init__ entirely as a 
novice, you will be treated to messages such as these:

qapi/source.py:18: error: Function is missing a return type annotation
qapi/source.py:18: note: Use "-> None" if function does not return a value
Found 1 error in 1 file (checked 14 source files)

Pretty good error!

There's no error if you DO explicitly add a -> None from __init__, but 
at worst it's just extraneous (but correct) information.

I could add a note to the style guide that I prefer omitting the return 
from __init__. I like omitting as much as I possibly can.

(You'll notice I don't always type every local, either -- when local 
inference is accurate, I leave it alone.)

--js

Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy

[John Snow]

On 10/7/20 7:37 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/mypy.ini | 5 -----
>>   1 file changed, 5 deletions(-)
>>
>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>> index 5df11e53fd1..8ab9ac52cc4 100644
>> --- a/scripts/qapi/mypy.ini
>> +++ b/scripts/qapi/mypy.ini
>> @@ -4,11 +4,6 @@ strict_optional = False
>>   disallow_untyped_calls = False
>>   python_version = 3.6
>>   
>> -[mypy-qapi.commands]
>> -disallow_untyped_defs = False
>> -disallow_incomplete_defs = False
>> -check_untyped_defs = False
>> -
>>   [mypy-qapi.error]
>>   disallow_untyped_defs = False
>>   disallow_incomplete_defs = False
> 
> The equivalent change for  events.py you squashed into the commit adding
> type hints.  Any particular reason for not doing the same here?
> 

Just making my life easier for re-arranging commits and rebasing. By 
separating them out whenever I had > 1 fix patch prior, I was able to 
freely re-arrange and re-order the prior fix patches.

They can be squashed on commit if desired, but for my own sake and 
inability to predict review comments, they are separate.

--js

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

[John Snow]

On 10/7/20 7:55 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Annotations do not change runtime behavior.
>> This commit *only* adds annotations.
>>
>> A note on typing of __init__: mypy requires init functions with no
>> parameters to document a return type of None to be considered fully
>> typed. In the case when there are input parameters, None may be omitted.
>>
>> Since __init__ may never return any value, it is preferred to omit the
>> return annotation whenever possible.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> Tested-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/mypy.ini  |  5 -----
>>   scripts/qapi/source.py | 31 ++++++++++++++++++-------------
>>   2 files changed, 18 insertions(+), 18 deletions(-)
>>
>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>> index 8ab9ac52cc4..1b8555dfa39 100644
>> --- a/scripts/qapi/mypy.ini
>> +++ b/scripts/qapi/mypy.ini
>> @@ -34,11 +34,6 @@ disallow_untyped_defs = False
>>   disallow_incomplete_defs = False
>>   check_untyped_defs = False
>>   
>> -[mypy-qapi.source]
>> -disallow_untyped_defs = False
>> -disallow_incomplete_defs = False
>> -check_untyped_defs = False
>> -
>>   [mypy-qapi.types]
>>   disallow_untyped_defs = False
>>   disallow_incomplete_defs = False
>> diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
>> index e97b9a8e15e..1cc6a5b82dc 100644
>> --- a/scripts/qapi/source.py
>> +++ b/scripts/qapi/source.py
>> @@ -11,37 +11,42 @@
>>   
>>   import copy
>>   import sys
>> +from typing import List, Optional, TypeVar
>>   
>>   
>>   class QAPISchemaPragma:
>> -    def __init__(self):
>> +    def __init__(self) -> None:
>>           # Are documentation comments required?
>>           self.doc_required = False
>>           # Whitelist of commands allowed to return a non-dictionary
>> -        self.returns_whitelist = []
>> +        self.returns_whitelist: List[str] = []
>>           # Whitelist of entities allowed to violate case conventions
>> -        self.name_case_whitelist = []
>> +        self.name_case_whitelist: List[str] = []
>>   
>>   
>>   class QAPISourceInfo:
>> -    def __init__(self, fname, line, parent):
>> +    T = TypeVar('T', bound='QAPISourceInfo')
>> +
>> +    def __init__(self: T, fname: str, line: int, parent: Optional[T]):
> 
> More ignorant questions (I'm abusing the review process to learn Python
> type hints)...
> 
> Why do you need to annotate self here, but not elsewhere?
> 

This is admittedly me being a little extra, but I thought it was a good 
way to show a pattern for people who maybe hadn't been exposed to it yet.

This is a pattern that allows for subclassing. I am stating that this 
__init__ method takes a parent of the same type as itself, whatever that 
happens to actually be.

T is a TypeVar that we can use for Generics. By declaring the type of 
self as that TypeVar, we bind T to self's type. When we annotate the 
parent as a T, we are enforcing that the parent we receive is of the 
same type as ourselves.

(Yep, we don't actually subclass QAPISourceInfo and I don't have plans 
to. It felt slightly icky to hard-code the class type name, though. I 
try to avoid that whenever I can. I'm not sure I would go so far as to 
call it a code smell or an antipattern, but it's something I tend to 
avoid anyway.)

> Why do you use T instead of QAPISourceInfo?
> 
>>           self.fname = fname
>>           self.line = line
>>           self.parent = parent
>> -        self.pragma = parent.pragma if parent else QAPISchemaPragma()
>> -        self.defn_meta = None
>> -        self.defn_name = None
>> +        self.pragma: QAPISchemaPragma = (
>> +            parent.pragma if parent else QAPISchemaPragma()
>> +        )
> 
> Type inference fail?
> 

Yes:

qapi/source.py:34: error: Cannot determine type of 'pragma'
Found 1 error in 1 file (checked 14 source files)

>> +        self.defn_meta: Optional[str] = None
>> +        self.defn_name: Optional[str] = None
>>   
>> -    def set_defn(self, meta, name):
>> +    def set_defn(self, meta: str, name: str) -> None:
>>           self.defn_meta = meta
>>           self.defn_name = name
>>   
>> -    def next_line(self):
>> +    def next_line(self: T) -> T:
>>           info = copy.copy(self)
>>           info.line += 1
>>           return info
>>   
>> -    def loc(self):
>> +    def loc(self) -> str:
>>           if self.fname is None:
>>               return sys.argv[0]
>>           ret = self.fname
>> @@ -49,13 +54,13 @@ def loc(self):
>>               ret += ':%d' % self.line
>>           return ret
>>   
>> -    def in_defn(self):
>> +    def in_defn(self) -> str:
>>           if self.defn_name:
>>               return "%s: In %s '%s':\n" % (self.fname,
>>                                             self.defn_meta, self.defn_name)
>>           return ''
>>   
>> -    def include_path(self):
>> +    def include_path(self) -> str:
>>           ret = ''
>>           parent = self.parent
>>           while parent:
>> @@ -63,5 +68,5 @@ def include_path(self):
>>               parent = parent.parent
>>           return ret
>>   
>> -    def __str__(self):
>> +    def __str__(self) -> str:
>>           return self.include_path() + self.in_defn() + self.loc()

Re: [PATCH v5 26/36] qapi/gen.py: Fix edge-case of _is_user_module

[John Snow]

On 10/7/20 8:02 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> The edge case is that if the name is '', this expression returns a
>> string instead of a bool, which violates our declared type.
> 
> The edge case is impossible, as discussed in review of v2.  I figure the
> type checker can't see that, so we need to help it some.  Can we mention
> this in the commit message?
> 

Sure. It's quite possible we will be able to better model and constrain 
these types, but that will come after all of these patches.

Mypy just has no way of knowing that '' is forbidden. That constraint 
does not exist in the type system we have here at present.

You could, by the way, create a type called NonEmptyString and create a 
function that casts from str to NonEmptyString by means of an assertion 
and then annotating the return type.

Anyway, I'll update the commit message for now.

>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>>   scripts/qapi/gen.py | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> index f2e2746fea5..1bad37fc06b 100644
>> --- a/scripts/qapi/gen.py
>> +++ b/scripts/qapi/gen.py
>> @@ -243,7 +243,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
>>   
>>       @staticmethod
>>       def _is_user_module(name):
>> -        return name and not name.startswith('./')
>> +        return bool(name and not name.startswith('./'))
> 
>             return not (name is None or name.startswith('./')
> 
> Looks slightly clearer to me.
>>   
>>       @staticmethod
>>       def _is_builtin_module(name):

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

[John Snow]

On 10/7/20 8:21 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Annotations do not change runtime behavior.
>> This commit *only* adds annotations.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
>>   1 file changed, 57 insertions(+), 47 deletions(-)
>>
>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> index 1bad37fc06b..d0391cd8718 100644
>> --- a/scripts/qapi/gen.py
>> +++ b/scripts/qapi/gen.py
>> @@ -17,7 +17,13 @@
>>   import errno
>>   import os
>>   import re
>> -from typing import Optional
>> +from typing import (
>> +    Dict,
>> +    Iterator,
>> +    List,
>> +    Optional,
>> +    Tuple,
>> +)
>>   
>>   from .common import (
>>       c_fname,
>> @@ -29,31 +35,31 @@
>>       mcgen,
>>   )
>>   from .schema import QAPISchemaObjectType, QAPISchemaVisitor
>> +from .source import QAPISourceInfo
>>   
>>   
>>   class QAPIGen:
>> -
>> -    def __init__(self, fname):
>> +    def __init__(self, fname: Optional[str]):
> 
> I'd expect fname: str.  Can you point me to the spot that passes None?
> 

qapi/commands.py:        self._regy = QAPIGenCCode(None)

Good time to mention again: I am disabling strict none checks for now in 
this conversion.

That means we can pass Optional[T] to functions expecting T and mypy 
will not complain. This should obviously be fixed long-term, but the 
cleanups involve things that are a higher class of finnicky.

If this alarms you, good! We'll fix it in time, but it doesn't break 
anything any worse than it already was.

If this check was disabled, you would be able to edit the Optional[str] 
to str and see what broke. But because I use this affordance for now, 
you would see no difference.

>>           self.fname = fname
>>           self._preamble = ''
>>           self._body = ''
>>   
>> -    def preamble_add(self, text):
>> +    def preamble_add(self, text: str) -> None:
>>           self._preamble += text
>>   
>> -    def add(self, text):
>> +    def add(self, text: str) -> None:
>>           self._body += text
>>   
>> -    def get_content(self):
>> +    def get_content(self) -> str:
>>           return self._top() + self._preamble + self._body + self._bottom()
>>   
>> -    def _top(self):
>> +    def _top(self) -> str:
>>           return ''
>>   
>> -    def _bottom(self):
>> +    def _bottom(self) -> str:
>>           return ''
>>   
>> -    def write(self, output_dir):
>> +    def write(self, output_dir: str) -> None:
>>           # Include paths starting with ../ are used to reuse modules of the main
>>           # schema in specialised schemas. Don't overwrite the files that are
>>           # already generated for the main schema.
>> @@ -78,7 +84,7 @@ def write(self, output_dir):
>>           f.close()
>>   
>>   
>> -def _wrap_ifcond(ifcond, before, after):
>> +def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
>>       if before == after:
>>           return after   # suppress empty #if ... #endif
>>   
>> @@ -118,40 +124,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
>>   
>>   
>>   class QAPIGenCCode(QAPIGen):
>> -
>> -    def __init__(self, fname):
>> +    def __init__(self, fname: Optional[str]):
> 
> Likewise.
> 
>>           super().__init__(fname)
>> -        self._start_if = None
>> +        self._start_if: Optional[Tuple[List[str], str, str]] = None
>>   
>> -    def start_if(self, ifcond):
>> +    def start_if(self, ifcond: List[str]) -> None:
>>           assert self._start_if is None
>>           self._start_if = (ifcond, self._body, self._preamble)
>>   
>> -    def end_if(self):
>> +    def end_if(self) -> None:
>>           assert self._start_if
>>           self._wrap_ifcond()
>>           self._start_if = None
>>   
>> -    def _wrap_ifcond(self):
>> +    def _wrap_ifcond(self) -> None:
>>           self._body = _wrap_ifcond(self._start_if[0],
>>                                     self._start_if[1], self._body)
>>           self._preamble = _wrap_ifcond(self._start_if[0],
>>                                         self._start_if[2], self._preamble)
>>   
>> -    def get_content(self):
>> +    def get_content(self) -> str:
>>           assert self._start_if is None
>>           return super().get_content()
>>   
>>   
>>   class QAPIGenC(QAPIGenCCode):
>> -
>> -    def __init__(self, fname, blurb, pydoc):
>> +    def __init__(self, fname: str, blurb: str, pydoc: str):
> 
> Here it's just str.
> 

That's right.

>>           super().__init__(fname)
>>           self._blurb = blurb
>>           self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
>>                                                     re.MULTILINE))
>>   
>> -    def _top(self):
>> +    def _top(self) -> str:
>>           return mcgen('''
>>   /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
>>   
>> @@ -167,7 +171,7 @@ def _top(self):
>>   ''',
>>                        blurb=self._blurb, copyright=self._copyright)
>>   
>> -    def _bottom(self):
>> +    def _bottom(self) -> str:
>>           return mcgen('''
>>   
>>   /* Dummy declaration to prevent empty .o file */
>> @@ -177,16 +181,15 @@ def _bottom(self):
>>   
>>   
>>   class QAPIGenH(QAPIGenC):
>> -
>> -    def _top(self):
>> +    def _top(self) -> str:
>>           return super()._top() + guardstart(self.fname)
>>   
>> -    def _bottom(self):
>> +    def _bottom(self) -> str:
>>           return guardend(self.fname)
>>   
>>   
>>   @contextmanager
>> -def ifcontext(ifcond, *args):
>> +def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:
> 
> Oh, the type hint for a *args is QAPIGenCCode, even though args is a
> tuple of excess positional arguments (which are all QAPIGenCCode).
> 

Yeah, it's the way type hints interact with the special '*' syntax. This 
is an Iterable of QAPIGenCCode.

**kwargs works the same way: you annotate the values, and the keys are 
assumed to be str.

>>       """
>>       A with-statement context manager that wraps with `start_if()` / `end_if()`.
>>   
>> @@ -214,8 +217,11 @@ def ifcontext(ifcond, *args):
>>   
>>   
>>   class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
>> -
>> -    def __init__(self, prefix, what, blurb, pydoc):
>> +    def __init__(self,
>> +                 prefix: str,
>> +                 what: str,
>> +                 blurb: str,
>> +                 pydoc: str):
>>           self._prefix = prefix
>>           self._what = what
>>           self._genc = QAPIGenC(self._prefix + self._what + '.c',
>> @@ -223,38 +229,42 @@ def __init__(self, prefix, what, blurb, pydoc):
>>           self._genh = QAPIGenH(self._prefix + self._what + '.h',
>>                                 blurb, pydoc)
>>   
>> -    def write(self, output_dir):
>> +    def write(self, output_dir: str) -> None:
>>           self._genc.write(output_dir)
>>           self._genh.write(output_dir)
>>   
>>   
>>   class QAPISchemaModularCVisitor(QAPISchemaVisitor):
>> -
>> -    def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
>> +    def __init__(self,
>> +                 prefix: str,
>> +                 what: str,
>> +                 user_blurb: str,
>> +                 builtin_blurb: Optional[str],
>> +                 pydoc: str):
>>           self._prefix = prefix
>>           self._what = what
>>           self._user_blurb = user_blurb
>>           self._builtin_blurb = builtin_blurb
>>           self._pydoc = pydoc
>> -        self._genc = None
>> -        self._genh = None
>> -        self._module = {}
>> -        self._main_module = None
>> +        self._genc: Optional[QAPIGenC] = None
>> +        self._genh: Optional[QAPIGenH] = None
>> +        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
>> +        self._main_module: Optional[str] = None
>>   
>>       @staticmethod
>> -    def _is_user_module(name):
>> +    def _is_user_module(name: Optional[str]) -> bool:
>>           return bool(name and not name.startswith('./'))
>>   
>>       @staticmethod
>> -    def _is_builtin_module(name):
>> +    def _is_builtin_module(name: Optional[str]) -> bool:
>>           return not name
>>   
>> -    def _module_dirname(self, what, name):
>> +    def _module_dirname(self, what: str, name: Optional[str]) -> str:
>>           if self._is_user_module(name):
>>               return os.path.dirname(name)
>>           return ''
>>   
>> -    def _module_basename(self, what, name):
>> +    def _module_basename(self, what: str, name: Optional[str]) -> str:
>>           ret = '' if self._is_builtin_module(name) else self._prefix
>>           if self._is_user_module(name):
>>               basename = os.path.basename(name)
>> @@ -266,27 +276,27 @@ def _module_basename(self, what, name):
>>               ret += re.sub(r'-', '-' + name + '-', what)
>>           return ret
>>   
>> -    def _module_filename(self, what, name):
>> +    def _module_filename(self, what: str, name: Optional[str]) -> str:
>>           return os.path.join(self._module_dirname(what, name),
>>                               self._module_basename(what, name))
>>   
>> -    def _add_module(self, name, blurb):
>> +    def _add_module(self, name: Optional[str], blurb: str) -> None:
>>           basename = self._module_filename(self._what, name)
>>           genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
>>           genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
>>           self._module[name] = (genc, genh)
>>           self._genc, self._genh = self._module[name]
>>   
>> -    def _add_user_module(self, name, blurb):
>> +    def _add_user_module(self, name: str, blurb: str) -> None:
>>           assert self._is_user_module(name)
>>           if self._main_module is None:
>>               self._main_module = name
>>           self._add_module(name, blurb)
>>   
>> -    def _add_system_module(self, name, blurb):
>> +    def _add_system_module(self, name: Optional[str], blurb: str) -> None:
>>           self._add_module(name and './' + name, blurb)
>>   
>> -    def write(self, output_dir, opt_builtins=False):
>> +    def write(self, output_dir: str, opt_builtins: bool = False) -> None:
>>           for name in self._module:
>>               if self._is_builtin_module(name) and not opt_builtins:
>>                   continue
>> @@ -294,13 +304,13 @@ def write(self, output_dir, opt_builtins=False):
>>               genc.write(output_dir)
>>               genh.write(output_dir)
>>   
>> -    def _begin_system_module(self, name):
>> +    def _begin_system_module(self, name: None) -> None:
> 
> I figure the type hint None signals a simplifcation opportunity.  No
> need to worry about it now.
> 

Yes, the module name stuff in general has a smell to it. I didn't look 
deeper, but there's an opportunity for cleaning it up. I realize the 
type signature here has parity with _begin_user_module, but I noticed it 
was *never* called with anything that wasn't a literal None, so I added 
the stricter type.

>>           pass
>>   
>> -    def _begin_user_module(self, name):
>> +    def _begin_user_module(self, name: str) -> None:
>>           pass
>>   
>> -    def visit_module(self, name):
>> +    def visit_module(self, name: Optional[str]) -> None:
>>           if name is None:
>>               if self._builtin_blurb:
>>                   self._add_system_module(None, self._builtin_blurb)
>> @@ -314,7 +324,7 @@ def visit_module(self, name):
>>               self._add_user_module(name, self._user_blurb)
>>               self._begin_user_module(name)
>>   
>> -    def visit_include(self, name, info):
>> +    def visit_include(self, name: str, info: QAPISourceInfo) -> None:
>>           relname = os.path.relpath(self._module_filename(self._what, name),
>>                                     os.path.dirname(self._genh.fname))
>>           self._genh.preamble_add(mcgen('''

Re: [PATCH v5 29/36] qapi/gen.py: Remove unused parameter

[John Snow]

On 10/7/20 8:22 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> module_basename doesn't use the 'what' argument, so remove it.
> 
> _module_dirname(), I suppose.
> 

My brain just translates stuff I didn't ask it to all the time. (Fixed.)

>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/gen.py | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> index d0391cd8718..3624162bb77 100644
>> --- a/scripts/qapi/gen.py
>> +++ b/scripts/qapi/gen.py
>> @@ -259,7 +259,7 @@ def _is_user_module(name: Optional[str]) -> bool:
>>       def _is_builtin_module(name: Optional[str]) -> bool:
>>           return not name
>>   
>> -    def _module_dirname(self, what: str, name: Optional[str]) -> str:
>> +    def _module_dirname(self, name: Optional[str]) -> str:
>>           if self._is_user_module(name):
>>               return os.path.dirname(name)
>>           return ''
>> @@ -277,7 +277,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
>>           return ret
>>   
>>       def _module_filename(self, what: str, name: Optional[str]) -> str:
>> -        return os.path.join(self._module_dirname(what, name),
>> +        return os.path.join(self._module_dirname(name),
>>                               self._module_basename(what, name))
>>   
>>       def _add_module(self, name: Optional[str], blurb: str) -> None:

Re: [PATCH v5 30/36] qapi/gen.py: update write() to be more idiomatic

[John Snow]

On 10/7/20 8:32 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Make the file handling here just a tiny bit more idiomatic.
>> (I realize this is heavily subjective.)
>>
>> Use exist_ok=True for os.makedirs and remove the exception,
>> use fdopen() to wrap the file descriptor in a File-like object,
>> and use a context manager for managing the file pointer.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/gen.py | 25 +++++++++++--------------
>>   1 file changed, 11 insertions(+), 14 deletions(-)
>>
>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> index 3624162bb77..579ee283297 100644
>> --- a/scripts/qapi/gen.py
>> +++ b/scripts/qapi/gen.py
>> @@ -14,7 +14,6 @@
>>   # See the COPYING file in the top-level directory.
>>   
>>   from contextlib import contextmanager
>> -import errno
>>   import os
>>   import re
>>   from typing import (
>> @@ -67,21 +66,19 @@ def write(self, output_dir: str) -> None:
>>               return
>>           pathname = os.path.join(output_dir, self.fname)
>>           odir = os.path.dirname(pathname)
>> +
>>           if odir:
>> -            try:
>> -                os.makedirs(odir)
>> -            except os.error as e:
>> -                if e.errno != errno.EEXIST:
>> -                    raise
>> +            os.makedirs(odir, exist_ok=True)
> 
> I wouldn't call this part "heavily subjective".  When wrote the old
> code, exist_ok was still off limits (it's new in 3.2).
> 

It's cool if you agree, I just realize that what people consider 
idiomatic is subjective unless it's enforcable by a tool. This isn't.

"I made this look more like if I wrote it, which caused Dopamine" is a 
bad commit message. (But more true.)

>> +
>> +        # use os.open for O_CREAT to create and read a non-existant file
>>           fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
>> -        f = open(fd, 'r+', encoding='utf-8')
>> -        text = self.get_content()
>> -        oldtext = f.read(len(text) + 1)
>> -        if text != oldtext:
>> -            f.seek(0)
>> -            f.truncate(0)
>> -            f.write(text)
>> -        f.close()
>> +        with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
>> +            text = self.get_content()
>> +            oldtext = fp.read(len(text) + 1)
>> +            if text != oldtext:
>> +                fp.seek(0)
>> +                fp.truncate(0)
>> +                fp.write(text)
>>   
>>   
>>   def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
> 
> Reviewed-by: Markus Armbruster <armbru@redhat.com>
> 

Thanks, though :)

Re: [PATCH v5 33/36] qapi/types.py: remove one-letter variables

[John Snow]

On 10/7/20 8:42 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> "John, if pylint told you to jump off a bridge, would you?"
>> Hey, if it looked like fun, I might.
>>
>> Now that this file is clean, enable pylint checks on this file.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/pylintrc |  1 -
>>   scripts/qapi/types.py | 29 +++++++++++++++--------------
>>   2 files changed, 15 insertions(+), 15 deletions(-)
>>
>> diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
>> index 8badcb11cda..b3c4cf46dbf 100644
>> --- a/scripts/qapi/pylintrc
>> +++ b/scripts/qapi/pylintrc
>> @@ -6,7 +6,6 @@ ignore-patterns=error.py,
>>                   expr.py,
>>                   parser.py,
>>                   schema.py,
>> -                types.py,
>>                   visit.py,
>>   
>>   
>> diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
>> index 766822feb3a..9d1e79d503d 100644
>> --- a/scripts/qapi/types.py
>> +++ b/scripts/qapi/types.py
>> @@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
>>       .array = (const char *const[]) {
>>   ''',
>>                   c_name=c_name(name))
>> -    for m in members:
>> -        ret += gen_if(m.ifcond)
>> -        index = c_enum_const(name, m.name, prefix)
>> +    for member in members:
> 
> Let's use @memb.  It's more visually distinct from @members, and
> gen_struct_members() already uses it.
> 

OK.

>> +        ret += gen_if(member.ifcond)
>> +        index = c_enum_const(name, member.name, prefix)
>>           ret += mcgen('''
>>           [%(index)s] = "%(name)s",
>>   ''',
>> -                     index=index, name=m.name)
>> -        ret += gen_endif(m.ifcond)
>> +                     index=index, name=member.name)
>> +        ret += gen_endif(member.ifcond)
>>   
>>       ret += mcgen('''
>>       },
>> @@ -79,13 +79,13 @@ def gen_enum(name: str,
>>   ''',
>>                   c_name=c_name(name))
>>   
>> -    for m in enum_members:
>> -        ret += gen_if(m.ifcond)
>> +    for member in enum_members:
>> +        ret += gen_if(member.ifcond)
>>           ret += mcgen('''
>>       %(c_enum)s,
>>   ''',
>> -                     c_enum=c_enum_const(name, m.name, prefix))
>> -        ret += gen_endif(m.ifcond)
>> +                     c_enum=c_enum_const(name, member.name, prefix))
>> +        ret += gen_endif(member.ifcond)
>>   
>>       ret += mcgen('''
>>   } %(c_name)s;
>> @@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
>>       objects_seen.add(name)
>>   
>>       ret = ''
>> -    if variants:
>> -        for v in variants.variants:
>> -            if isinstance(v.type, QAPISchemaObjectType):
>> -                ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
>> -                                  v.type.local_members, v.type.variants)
>> +    for variant in variants.variants if variants else ():
> 
> Let's use @var, for consistency with gen_variants().
> 

You're the boss.

(Said only when I don't actually still kinda sorta want to do it my way. 
Then you're the boss, but in the bad way. Like I am the petulant 
teenager working at a Burger King and I am having a disagreement with 
the Assistant Manager over the proper way to stock the ketchup packets.)

>> +        obj = variant.type
>> +        if not isinstance(obj, QAPISchemaObjectType):
>> +            continue
>> +        ret += gen_object(obj.name, obj.ifcond, obj.base,
>> +                          obj.local_members, obj.variants)
>>   
>>       ret += mcgen('''

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

[John Snow]

On 10/7/20 8:43 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> This is true by design, but not presently able to be expressed in the
>> type system. An assertion helps mypy understand our constraints.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/visit.py | 12 +++++++-----
>>   1 file changed, 7 insertions(+), 5 deletions(-)
>>
>> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
>> index 14f30c228b7..4f11fd325b8 100644
>> --- a/scripts/qapi/visit.py
>> +++ b/scripts/qapi/visit.py
>> @@ -22,7 +22,7 @@
>>       mcgen,
>>   )
>>   from .gen import QAPISchemaModularCVisitor, ifcontext
>> -from .schema import QAPISchemaObjectType
>> +from .schema import QAPISchemaEnumType, QAPISchemaObjectType
>>   
>>   
>>   def gen_visit_decl(name, scalar=False):
>> @@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
>>           ret += gen_endif(memb.ifcond)
>>   
>>       if variants:
>> +        tag_member = variants.tag_member
>> +        assert isinstance(tag_member.type, QAPISchemaEnumType)
> 
> I'm curious: do you need the local variable to make the assertion stick?
> 

No, but it only sticks to the binding and not the data. i.e. assertions 
to downcast work on the *name*.

(This comes up somewhere in the schema.py patches where I make a change 
that looks completely pointless, but it makes mypy happy.)

I could have left it alone. I just saw a lot of repeated multi-dots and 
habitually created a temporary local for the purpose.

>> +
>>           ret += mcgen('''
>>       switch (obj->%(c_name)s) {
>>   ''',
>> -                     c_name=c_name(variants.tag_member.name))
>> +                     c_name=c_name(tag_member.name))
>>   
>>           for var in variants.variants:
>> -            case_str = c_enum_const(variants.tag_member.type.name,
>> -                                    var.name,
>> -                                    variants.tag_member.type.prefix)
>> +            case_str = c_enum_const(tag_member.type.name, var.name,
>> +                                    tag_member.type.prefix)
>>               ret += gen_if(var.ifcond)
>>               if var.type.name == 'q_empty':
>>                   # valid variant and nothing to do

Re: [PATCH v5 36/36] qapi/visit.py: add type hint annotations

[John Snow]

On 10/7/20 9:00 AM, Markus Armbruster wrote:
> More abuse of review to learn: even through we initialize with False, we
> still need the type hint?

Dunno.

jsnow@scv ~/s/q/scripts (python-qapi-cleanup-pt1)> mypy 
--config-file=qapi/mypy.ini qapi/;
qapi/visit.py:40: error: Function is missing a type annotation for one 
or more arguments
Found 1 error in 1 file (checked 14 source files)

...yup.

It also serves the purpose of constraining the type to also avoid unions 
with things that might be false-ish. Looks silly, but maybe future mypy 
installments will afford us more chances to use inference. It doesn't 
right now.

--js

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

[John Snow]

On 10/7/20 9:20 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Annotations do not change runtime behavior.
>> This commit *only* adds annotations.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
>>   1 file changed, 57 insertions(+), 47 deletions(-)
>>
>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> index 1bad37fc06b..d0391cd8718 100644
>> --- a/scripts/qapi/gen.py
>> +++ b/scripts/qapi/gen.py
>> @@ -17,7 +17,13 @@
>>   import errno
>>   import os
>>   import re
>> -from typing import Optional
>> +from typing import (
>> +    Dict,
>> +    Iterator,
>> +    List,
>> +    Optional,
>> +    Tuple,
>> +)
>>   
>>   from .common import (
>>       c_fname,
>> @@ -29,31 +35,31 @@
>>       mcgen,
>>   )
>>   from .schema import QAPISchemaObjectType, QAPISchemaVisitor
>> +from .source import QAPISourceInfo
> 
> PATCH 03 has a similar cleanup.  Are there more?  Perhaps a separate
> patch doing just this kind of cleanup would make sense.  Up to you.
> 
> [...]
> 

This isn't a cleanup, I am just importing QAPISourceInfo to use for an 
annotation. It's relevant and required for this patch, and doesn't make 
sense on its own.

Patch 03 ... Oh, you mean identifying the correct location of QAPIError. 
Uh... nah? I think that was the only case of that one changing. Not 
worth pulling out or naming, I think.

Re: [PATCH v5 02/36] qapi: modify docstrings to be sphinx-compatible

[John Snow]

On 10/7/20 3:24 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/6/20 7:21 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> A precise style guide and a package-wide overhaul is forthcoming pending
>>>> further discussion and consensus. At present, we are avoiding obvious
>>>> errors that cause sphinx documentation build problems.
>>>>
>>>> A preliminary style guide is loosely based on PEP 257 and Sphinx
>>>> Autodoc. It is chosen for interoperability with our existing Sphinx
>>>> framework, and because it has loose recognition in the Pycharm IDE.
>>>>
>>>> - Use Triple-double quotes (""").
>>>> - Opening and closing quotes appear on their own lines for multi-line docs.
>>>> - A single-sentence summary should be the first line of the docstring.
>>>> - A blank line follows.
>>>> - Further prose, if needed, is written next and can be multiple paragraphs,
>>>>     contain RST markup, etc.
>>>> - The :param x: desc, :returns: desc, and :raises z: desc info fields follow.
>>> Mandatory when they apply?
>>>
>>
>> Subject of debate...
>>
>> - Some people really hate obvious docstring comments.
>> - Some people really like the consistency.
>>
>> Which type of developer am I? Guess it depends on when you ask.
>>
>> Figured we'd hash that out when I go to write a style guide document.
> 
> Fair enough.
> 
> If I stop reading after the first paragraph, the patch matches
> expectations built by the commit message.
> 

(:

> If I speed-read, the first paragraph barely registers, but the second
> makes me slow down, giving me the mistaken idea that this patch is about
> converting to a preliminary style guide.  It's not, it's about getting
> Sphinx errors out of the way.
> 
> I figure you didn't stop after the first paragraph because you felt a
> need to explain why you resolve the "obvious errors" the way you do.
> 
> Perhaps:
> 
>      qapi: modify docstrings to be sphinx-compatible
> 
>      A precise style guide and a package-wide overhaul is forthcoming
>      pending further discussion and consensus. For now, merely avoid
>      obvious errors that cause Sphinx documentation build problems, using a
>      style loosely based on PEP 257 and Sphinx Autodoc. It is chosen for
>      interoperability with our existing Sphinx framework, and because it
>      has loose recognition in the Pycharm IDE.
> 
>      [...]
>     
> 
>>>> - Additional examples, diagrams, or other metadata follows below.
>>>>
>>>> See also:
>>>>
>>>> https://www.python.org/dev/peps/pep-0257/
>>>> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
> 
> Blank line here, by convention.
> 

Wonder why my script didn't do that. Eh.

>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>>    scripts/qapi/gen.py    | 6 ++++--
>>>>    scripts/qapi/parser.py | 1 +
>>>>    2 files changed, 5 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>>>> index ca66c82b5b8..dc7b94aa115 100644
>>>> --- a/scripts/qapi/gen.py
>>>> +++ b/scripts/qapi/gen.py
>>>> @@ -154,9 +154,11 @@ def _bottom(self):
>>>>      @contextmanager
>>>>    def ifcontext(ifcond, *args):
>>>> -    """A 'with' statement context manager to wrap with start_if()/end_if()
>>>> +    """
>>>> +    A with-statement context manager that wraps with `start_if()` / `end_if()`.
>>>>    -    *args: any number of QAPIGenCCode
>>>> +    :param ifcond: A list of conditionals, passed to `start_if()`.
>>>> +    :param args: any number of `QAPIGenCCode`.
>>>>          Example::
>>>>    diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>>>> index 9d1a3e2eea9..31bc2e6dca9 100644
>>>> --- a/scripts/qapi/parser.py
>>>> +++ b/scripts/qapi/parser.py
>>>> @@ -381,6 +381,7 @@ def append(self, line):
>>>>              The way that the line is dealt with depends on which
>>>> part of
>>>>            the documentation we're parsing right now:
>>>> +
>>>>            * The body section: ._append_line is ._append_body_line
>>>>            * An argument section: ._append_line is ._append_args_line
>>>>            * A features section: ._append_line is ._append_features_line
>>> I'm asking because you're not adding :param line: here.
>>>
>>
>> Yeah, it's not necessary to test the syntax of what else I've written
>> with sphinx, so I didn't add it. VERY TECHNICALLY this blurb isn't
>> required at all and could be deleted. You can do so if you'd like; it
>> will just show up later in some other patch or series more designed to
>> fix formatting.
> 
> I recommend (but do not demand) to strictly limit this commit to
> "avoiding obvious errors that cause sphinx documentation build
> problems."
> 

OK, I'll drop this bit for now, but I will keep the new annotations for 
ifcontext, because... well. Why do it twice.

>>> Same for several other functions in this file.
>>> In schema.py:
>>>       class QAPISchemaMember:
>>>           """ Represents object members, enum members and features """
>>> Are the spaces next to """ okay?
>>>
>>
>> Ideally cleaned up, but that's not a goal of this patch or series.
> 
> Got it.
> 

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

[John Snow]

On 10/7/20 4:50 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Code style tools really dislike the use of global keywords, because it
>> generally involves re-binding the name at runtime which can have strange
>> effects depending on when and how that global name is referenced in
>> other modules.
>>
>> Make a little indent level manager instead.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 
> Intentation is a job for QAPIGen (and its subtypes).  But if this patch
> is easier to achieve this series' goal, I don't mind.
> 

I agree, but refactoring it properly is beyond my capacity right now.

This was the dumbest thing I could do to get pylint/mypy passing, which 
required the elimination (or suppression) of the global keyword.

Creating a stateful object was the fastest way from A to B.

>> ---
>>   scripts/qapi/common.py | 49 ++++++++++++++++++++++++++++--------------
>>   scripts/qapi/visit.py  |  7 +++---
>>   2 files changed, 36 insertions(+), 20 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index cee63eb95c7..b35318b72cf 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -93,33 +93,50 @@ def c_name(name, protect=True):
>>   pointer_suffix = ' *' + eatspace
>>   
>>   
>> -def genindent(count):
>> -    ret = ''
>> -    for _ in range(count):
>> -        ret += ' '
>> -    return ret
>> +class Indentation:
>> +    """
>> +    Indentation level management.
>>   
>> +    :param initial: Initial number of spaces, default 0.
>> +    """
>> +    def __init__(self, initial: int = 0) -> None:
>> +        self._level = initial
>>   
>> -indent_level = 0
>> +    def __int__(self) -> int:
>> +        return self._level
>>   
>> +    def __repr__(self) -> str:
>> +        return "{}({:d})".format(type(self).__name__, self._level)
>>   
>> -def push_indent(indent_amount=4):
>> -    global indent_level
>> -    indent_level += indent_amount
>> +    def __str__(self) -> str:
>> +        """Return the current indentation as a string of spaces."""
>> +        return ' ' * self._level
>>   
>> +    def __bool__(self) -> bool:
>> +        """True when there is a non-zero indentation."""
>> +        return bool(self._level)
>>   
>> -def pop_indent(indent_amount=4):
>> -    global indent_level
>> -    indent_level -= indent_amount
>> +    def increase(self, amount: int = 4) -> None:
>> +        """Increase the indentation level by ``amount``, default 4."""
>> +        self._level += amount
>> +
>> +    def decrease(self, amount: int = 4) -> None:
>> +        """Decrease the indentation level by ``amount``, default 4."""
>> +        if self._level < amount:
>> +            raise ArithmeticError(
>> +                f"Can't remove {amount:d} spaces from {self!r}")
> 
> Raise a fancy error when there's an actual need for it.  You're not
> coding a framework thousands of people you never heard of will put to
> uses you cannot imagine.
> 

It's not fancy, it's just a normal built-in exception, like 
AssertionError or any other:

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ArithmeticError: Can't remove 4 spaces from Indent(0)

vs

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
AssertionError: Can't remove 4 spaces from Indent(0)


I feel like it's kind of a lateral move? I realize you feel this is an 
overfancy class doing what we hope is a temporary job, and that I have 
overengineered the hell out of a tiny do-nothing class... but I suppose 
that's also why I feel weird changing it around so much to accomplish so 
little.

Differences in style, I suppose.

Feel free to change it around to suit your tastes, I don't think it's 
worth spending a lot of ping-pong time on this paintsink in particular.

--js

>> +        self._level -= amount
>> +
>> +
>> +indent = Indentation()
>>   
>>   
>>   # Generate @code with @kwds interpolated.
>> -# Obey indent_level, and strip eatspace.
>> +# Obey indent, and strip eatspace.
>>   def cgen(code, **kwds):
>>       raw = code % kwds
>> -    if indent_level:
>> -        indent = genindent(indent_level)
>> -        raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
>> +    if indent:
>> +        raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
>>       return re.sub(re.escape(eatspace) + r' *', '', raw)
>>   
>>   
>> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
>> index 808410d6f1b..14f30c228b7 100644
>> --- a/scripts/qapi/visit.py
>> +++ b/scripts/qapi/visit.py
>> @@ -18,9 +18,8 @@
>>       c_name,
>>       gen_endif,
>>       gen_if,
>> +    indent,
>>       mcgen,
>> -    pop_indent,
>> -    push_indent,
>>   )
>>   from .gen import QAPISchemaModularCVisitor, ifcontext
>>   from .schema import QAPISchemaObjectType
>> @@ -69,7 +68,7 @@ def gen_visit_object_members(name, base, members, variants):
>>       if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
>>   ''',
>>                            name=memb.name, c_name=c_name(memb.name))
>> -            push_indent()
>> +            indent.increase()
>>           ret += mcgen('''
>>       if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
>>           return false;
>> @@ -78,7 +77,7 @@ def gen_visit_object_members(name, base, members, variants):
>>                        c_type=memb.type.c_name(), name=memb.name,
>>                        c_name=c_name(memb.name))
>>           if memb.optional:
>> -            pop_indent()
>> +            indent.decrease()
>>               ret += mcgen('''
>>       }
>>   ''')

Re: [PATCH v5 17/36] qapi/common.py: move build_params into gen.py

[Eduardo Habkost]

On Wed, Oct 07, 2020 at 11:21:51AM +0200, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
> > Including it in common.py creates a circular import dependency; schema
> > relies on common, but common.build_params requires a type annotation
> > from schema. To type this properly, it needs to be moved outside the
> > cycle.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  scripts/qapi/commands.py |  9 +++++++--
> >  scripts/qapi/common.py   | 23 -----------------------
> >  scripts/qapi/events.py   |  9 ++-------
> >  scripts/qapi/gen.py      | 31 +++++++++++++++++++++++++++++--
> >  4 files changed, 38 insertions(+), 34 deletions(-)
> >
> > diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
> > index 5dc2f5a9fa8..f67393f8713 100644
> > --- a/scripts/qapi/commands.py
> > +++ b/scripts/qapi/commands.py
> > @@ -13,8 +13,13 @@
> >  See the COPYING file in the top-level directory.
> >  """
> >  
> > -from .common import build_params, c_name, mcgen
> > -from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
> > +from .common import c_name, mcgen
> > +from .gen import (
> > +    QAPIGenCCode,
> > +    QAPISchemaModularCVisitor,
> > +    build_params,
> > +    ifcontext,
> > +)
> >  
> >  
> >  def gen_command_decl(name, arg_type, boxed, ret_type):
> > diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> > index 0ef38ea5fe0..9ab0685cc51 100644
> > --- a/scripts/qapi/common.py
> > +++ b/scripts/qapi/common.py
> > @@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
> >  #endif /* %(cond)s */
> >  ''', cond=ifc)
> >      return ret
> > -
> > -
> > -def build_params(arg_type,
> > -                 boxed: bool,
> > -                 extra: Optional[str] = None) -> str:
> > -    ret = ''
> > -    sep = ''
> > -    if boxed:
> > -        assert arg_type
> > -        ret += '%s arg' % arg_type.c_param_type()
> > -        sep = ', '
> > -    elif arg_type:
> > -        assert not arg_type.variants
> > -        for memb in arg_type.members:
> > -            ret += sep
> > -            sep = ', '
> > -            if memb.optional:
> > -                ret += 'bool has_%s, ' % c_name(memb.name)
> > -            ret += '%s %s' % (memb.type.c_param_type(),
> > -                              c_name(memb.name))
> > -    if extra:
> > -        ret += sep + extra
> > -    return ret if ret else 'void'
> > diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
> > index 6b3afa14d72..f840a62ed92 100644
> > --- a/scripts/qapi/events.py
> > +++ b/scripts/qapi/events.py
> > @@ -12,13 +12,8 @@
> >  See the COPYING file in the top-level directory.
> >  """
> >  
> > -from .common import (
> > -    build_params,
> > -    c_enum_const,
> > -    c_name,
> > -    mcgen,
> > -)
> > -from .gen import QAPISchemaModularCVisitor, ifcontext
> > +from .common import c_enum_const, c_name, mcgen
> > +from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
> >  from .schema import QAPISchemaEnumMember
> >  from .types import gen_enum, gen_enum_lookup
> >  
> > diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> > index 1fed712b43b..f2e2746fea5 100644
> > --- a/scripts/qapi/gen.py
> > +++ b/scripts/qapi/gen.py
> > @@ -2,9 +2,11 @@
> >  #
> >  # QAPI code generation
> >  #
> > -# Copyright (c) 2018-2019 Red Hat Inc.
> > +# Copyright IBM, Corp. 2011
> > +# Copyright (c) 2013-2019 Red Hat Inc.
> >  #
> >  # Authors:
> > +#  Anthony Liguori <aliguori@us.ibm.com>
> >  #  Markus Armbruster <armbru@redhat.com>
> >  #  Marc-André Lureau <marcandre.lureau@redhat.com>
> >  #
> 
> The code you move into this file is actually Red Hat's:
> 
> $ git-log -L174,193:scripts/qapi/common.py master| egrep 'Author|Date'
> Author: Markus Armbruster <armbru@redhat.com>
> Date:   Wed Aug 15 21:37:36 2018 +0800
> Author: Marc-André Lureau <marcandre.lureau@redhat.com>
> Date:   Thu Jun 1 16:41:41 2017 +0400
> Author: Eric Blake <eblake@redhat.com>
> Date:   Wed Jul 13 21:50:20 2016 -0600
> Author: Eric Blake <eblake@redhat.com>
> Date:   Wed Jul 13 21:50:19 2016 -0600
> Author: Eric Blake <eblake@redhat.com>
> Date:   Thu Mar 17 16:48:28 2016 -0600
> Author: Markus Armbruster <armbru@redhat.com>
> Date:   Wed Sep 16 13:06:20 2015 +0200
> 
> So the correct update is simply
> 
>    #
>    # QAPI code generation
>    #
>   -# Copyright (c) 2018-2019 Red Hat Inc.
>   +# Copyright (c) 2015-2019 Red Hat Inc.
>    #
>    # Authors:
>    #  Markus Armbruster <armbru@redhat.com>
>    #  Marc-André Lureau <marcandre.lureau@redhat.com>
>    #

I am pretty sure build_params() below is a derivative work of
generate_command_decl(), added in commit c17d9908a942 ("qapi: add
qapi-commands.py code generator"), copyright IBM Corp.  I'm not a
lawyer, though.

Because figuring that out is complicated, when I move or copy
code between files, I either:
(a) copy no copyright notices from the other file, or
(b) copy all copyright notices from the other files.

(The above is a description of what I do, but it is not a
suggestion on how to proceed on this case.)

> 
> > @@ -15,16 +17,18 @@
> >  import errno
> >  import os
> >  import re
> > +from typing import Optional
> >  
> >  from .common import (
> >      c_fname,
> > +    c_name,
> >      gen_endif,
> >      gen_if,
> >      guardend,
> >      guardstart,
> >      mcgen,
> >  )
> > -from .schema import QAPISchemaVisitor
> > +from .schema import QAPISchemaObjectType, QAPISchemaVisitor
> >  
> >  
> >  class QAPIGen:
> > @@ -90,6 +94,29 @@ def _wrap_ifcond(ifcond, before, after):
> >      return out
> >  
> >  
> > +def build_params(arg_type: Optional[QAPISchemaObjectType],
> > +                 boxed: bool,
> > +                 extra: Optional[str] = None) -> str:
> > +    ret = ''
> > +    sep = ''
> > +    if boxed:
> > +        assert arg_type
> > +        ret += '%s arg' % arg_type.c_param_type()
> > +        sep = ', '
> > +    elif arg_type:
> > +        assert not arg_type.variants
> > +        for memb in arg_type.members:
> > +            ret += sep
> > +            sep = ', '
> > +            if memb.optional:
> > +                ret += 'bool has_%s, ' % c_name(memb.name)
> > +            ret += '%s %s' % (memb.type.c_param_type(),
> > +                              c_name(memb.name))
> > +    if extra:
> > +        ret += sep + extra
> > +    return ret if ret else 'void'
> > +
> > +
> >  class QAPIGenCCode(QAPIGen):
> >  
> >      def __init__(self, fname):

-- 
Eduardo

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

[Eduardo Habkost]

On Wed, Oct 07, 2020 at 02:08:33PM -0400, John Snow wrote:
> On 10/7/20 4:50 AM, Markus Armbruster wrote:
> > John Snow <jsnow@redhat.com> writes:
> > 
> > > Code style tools really dislike the use of global keywords, because it
> > > generally involves re-binding the name at runtime which can have strange
> > > effects depending on when and how that global name is referenced in
> > > other modules.
> > > 
> > > Make a little indent level manager instead.
> > > 
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> > > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > 
> > Intentation is a job for QAPIGen (and its subtypes).  But if this patch
> > is easier to achieve this series' goal, I don't mind.
> > 
> 
> I agree, but refactoring it properly is beyond my capacity right now.
> 
> This was the dumbest thing I could do to get pylint/mypy passing, which
> required the elimination (or suppression) of the global keyword.
> 
> Creating a stateful object was the fastest way from A to B.

An even dumber solution could be:

  indent_prefixes = []
  def push_indent(amount=4):
      """Add `amount` spaces to indentation prefix"""
      indent_prefixes.push(' '*amount)
  def pop_indent():
      """Revert the most recent push_indent() call"""
      indent_prefixes.pop()
  def genindent():
      """Return the current indentation prefix"""
      return ''.join(indent_prefixes)

No global keyword involved, and the only stateful object is a
dumb list.

-- 
Eduardo

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 3:54 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 10/6/20 7:51 AM, Markus Armbruster wrote:
>>>> John Snow <jsnow@redhat.com> writes:
>>>>
>>>>> This is a minor re-work of the entrypoint script. It isolates a
>>>>> generate() method from the actual command-line mechanism.
>>>>>
>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>>> ---
>>>>>    scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>>>    1 file changed, 62 insertions(+), 23 deletions(-)
>>>>>
>>>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>>>> index 541e8c1f55d..117b396a595 100644
>>>>> --- a/scripts/qapi-gen.py
>>>>> +++ b/scripts/qapi-gen.py
>>>>> @@ -1,30 +1,77 @@
>>>>>    #!/usr/bin/env python3
>>>>> -# QAPI generator
>>>>> -#
>>>>> +
>>>>>    # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>>>    # See the COPYING file in the top-level directory.
>>>>>    +"""
>>>>> +QAPI Generator
>>>>> +
>>>>> +This script is the main entry point for generating C code from the QAPI schema.
>>>>> +"""
>>>>>      import argparse
>>>>>    import re
>>>>>    import sys
>>>>>      from qapi.commands import gen_commands
>>>>> +from qapi.error import QAPIError
>>>>>    from qapi.events import gen_events
>>>>>    from qapi.introspect import gen_introspect
>>>>> -from qapi.schema import QAPIError, QAPISchema
>>>>> +from qapi.schema import QAPISchema
>>>>>    from qapi.types import gen_types
>>>>>    from qapi.visit import gen_visit
>>>> Unrelated cleanup.  Okay.
>>>>
>>>>>      -def main(argv):
>>>>> +DEFAULT_OUTPUT_DIR = ''
>>>>> +DEFAULT_PREFIX = ''
>>>>> +
>>>>> +
>>>>> +def generate(schema_file: str,
>>>>> +             output_dir: str,
>>>>> +             prefix: str,
>>>>> +             unmask: bool = False,
>>>>> +             builtins: bool = False) -> None:
>>>>> +    """
>>>>> +    generate uses a given schema to produce C code in the target directory.
>>>>> +
>>>>> +    :param schema_file: The primary QAPI schema file.
>>>>> +    :param output_dir: The output directory to store generated code.
>>>>> +    :param prefix: Optional C-code prefix for symbol names.
>>>>> +    :param unmask: Expose non-ABI names through introspection?
>>>>> +    :param builtins: Generate code for built-in types?
>>>>> +
>>>>> +    :raise QAPIError: On failures.
>>>>> +    """
>>>>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>>>>> +    if match.end() != len(prefix):
>>>>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>>>>> +            prefix[match.end()], prefix)
>>>>> +        raise QAPIError('', None, msg)
>>>> Uh...
>>>>       $ python3 scripts/qapi-gen.py --prefix=@ x
>>>>       scripts/qapi-gen.py: : funny character '@' in prefix '@'
>>>> Unwanted " :".
>>>> This is due to a hack: you pass '' for info (*quack*).  Everything
>>>> else
>>>> passes QAPISourceInfo (I believe).
>>>>
>>>
>>> Quack indeed - why does our base error class require so much
>>> information from a specific part of the generation process?
>> Because it's not "a base error class", it's a base error class for
>> the
>> QAPI schema compiler frontend.
>> 
>
> Well. It's the base for every error we /had/.

You asked why the class has the init it has, and I answered :)

>>> Ah, someone changes this in part 4 so that we have a more generic
>>> error class to use as a base when we are missing such information.
>> Evolving it to satisfy a need for a more widely usable error class
>> is
>> okay.
>> 
>
> Yep. It's helpful to keep a very generic form on which we grow other
> errors from, so that things like the entry point can be written
> legibly.

If you have a non-trivial error message format convention, you have a
use for a function formatting error messages.

If you have a separation between diagnose and report of errors, you you
have a use for a transport from diagnose to report.  In Python, that's
raise.

The existing error message in main() has neither.

The existing error class QAPIError caters for the existing users.

>>> You are witnessing some more future-bleed.
>>>> Is it really a good idea to do this in generate?  It's not about
>>>> generating code, it's about validating a CLI option.
>>>>
>>>
>>> One might also ask: Is it a good idea to only validate this on a
>>> frontend, and not in the implementation?
>> Yes, because that's where you can emit the better error message more
>> easily.
>>      $ python3 scripts/qapi-gen.py --prefix=@ x
>>      scripts/qapi-gen.py: 'funny character '@' in argument of --prefix
>> is better than
>>      $ python3 scripts/qapi-gen.py --prefix=@ x
>>      scripts/qapi-gen.py: funny character '@' in prefix '@'
>> In generate(), the knowledge where the offending prefix value comes
>> from
>> is no longer available.
>> To emit this error message, you'd have to raise a sufficiently
>> distinct
>> error in generate, catch it in main(), then put the error message
>> together somehow.  Bah.
>> Aside: there's a stray ' in the old error message.
>> 
>>> The idea here was to create a function that could be used in a script
>>> (for tests, debugging interfaces, other python packages) to do all of
>>> the same things that the CLI tool did, just sans the actual CLI.
>> YAGNI.
>> 
>
> It's useful for testing and debugging to be able to just call it
> outside of the CLI, though. Maybe you won't use it, but I will.

For testing and debugging, treating "prefix is sane" as a precondition
is fine.  I wouldn't even bother checking it.  A check would catch
accidents, and these accidents seem vanishingly unlikely to me.

Evidence: we did without *any* prefix checking for *years*.  I added it
in commit 1cf47a15f18 just for completeness.

> I could always add the prefix check into a tiny function and give the
> good error message in main(), and just assert in generate() if you 
> insist on the slightly more specific error message from the CLI script.

If you genuinely think a check is needed there, that's the way to go.

>>> Wouldn't make sense to allow garbage to flow in from one interface but
>>> not the other; so the check is here.
>> "@prefix is sane" is a precondition of generate().
>> When there's a real risk of preconditions getting violated, or
>> readers
>> getting confused about preconditions, check them with assert.
[...]

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 4:07 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> This is a minor re-work of the entrypoint script. It isolates a
>>> generate() method from the actual command-line mechanism.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>   1 file changed, 62 insertions(+), 23 deletions(-)
>>>
>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>> index 541e8c1f55d..117b396a595 100644
>>> --- a/scripts/qapi-gen.py
>>> +++ b/scripts/qapi-gen.py
>>> @@ -1,30 +1,77 @@
>>>   #!/usr/bin/env python3
>>> -# QAPI generator
>>> -#
>>> +
>>>   # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>   # See the COPYING file in the top-level directory.
>>>   +"""
>>> +QAPI Generator
>>> +
>>> +This script is the main entry point for generating C code from the QAPI schema.
>>> +"""
>>>     import argparse
>>>   import re
>>>   import sys
>>>     from qapi.commands import gen_commands
>>> +from qapi.error import QAPIError
>>>   from qapi.events import gen_events
>>>   from qapi.introspect import gen_introspect
>>> -from qapi.schema import QAPIError, QAPISchema
>>> +from qapi.schema import QAPISchema
>>>   from qapi.types import gen_types
>>>   from qapi.visit import gen_visit
>>>     
>>> -def main(argv):
>>> +DEFAULT_OUTPUT_DIR = ''
>>> +DEFAULT_PREFIX = ''
>>> +
>>> +
>>> +def generate(schema_file: str,
>>> +             output_dir: str,
>>> +             prefix: str,
>>> +             unmask: bool = False,
>>> +             builtins: bool = False) -> None:
>>> +    """
>>> +    generate uses a given schema to produce C code in the target directory.
>>> +
>>> +    :param schema_file: The primary QAPI schema file.
>>> +    :param output_dir: The output directory to store generated code.
>>> +    :param prefix: Optional C-code prefix for symbol names.
>>> +    :param unmask: Expose non-ABI names through introspection?
>>> +    :param builtins: Generate code for built-in types?
>>> +
>>> +    :raise QAPIError: On failures.
>>> +    """
>>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>>> +    if match.end() != len(prefix):
>>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>>> +            prefix[match.end()], prefix)
>>> +        raise QAPIError('', None, msg)
>>> +
>>> +    schema = QAPISchema(schema_file)
>>> +    gen_types(schema, output_dir, prefix, builtins)
>>> +    gen_visit(schema, output_dir, prefix, builtins)
>>> +    gen_commands(schema, output_dir, prefix)
>>> +    gen_events(schema, output_dir, prefix)
>>> +    gen_introspect(schema, output_dir, prefix, unmask)
>>> +
>>> +
>>> +def main() -> int:
>>> +    """
>>> +    gapi-gen shell script entrypoint.
>> What's a "shell script entrypoint"?
>> Python docs talk about "when [...] run as a script":
>> https://docs.python.org/3/library/__main__.html
>> Similar:
>> https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts
>> 
>
> "entrypoint" is Python garble for a function that can be registered as
> a callable from the command line.
>
> So in a theoretical setup.py, you'd do something like:
>
> 'entry_points': {
>   'console_scripts': [
>     'qapi-gen = qapi.main:main',
>   ]
> }
>
> so when I say "shell script entrypoint", I am referring to a shell
> script (I mean: it has a shebang and can be executed by an interactive 
> shell process) that calls the entrypoint.

It can be executed by any process.  See execve(2):

       pathname must be either a binary executable, or a script starting  with
       a line of the form:

           #!interpreter [optional-arg]

       For details of the latter case, see "Interpreter scripts" below.

"Entry point" makes sense in Python context, "script entry point" also
makes sense (since every Python program is a script, script is
redundant, but not wrong).  "Shell script entry point" is misleading.

> Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to
> just generate that script.
>
> (i.e. doing `pip install qemu.qapi` will install a 'qapi-gen' CLI
> script for you. this is how packages like sphinx create the
> 'sphinx-build' script, etc.)
>
>>> +    Expects arguments via sys.argv, see --help for details.
>>> +
>>> +    :return: int, 0 on success, 1 on failure.
>>> +    """
>>>       parser = argparse.ArgumentParser(
>>>           description='Generate code from a QAPI schema')
>>>       parser.add_argument('-b', '--builtins', action='store_true',
>>>                           help="generate code for built-in types")
>>> -    parser.add_argument('-o', '--output-dir', action='store', default='',
>>> +    parser.add_argument('-o', '--output-dir', action='store',
>>> +                        default=DEFAULT_OUTPUT_DIR,
>>>                           help="write output to directory OUTPUT_DIR")
>>> -    parser.add_argument('-p', '--prefix', action='store', default='',
>>> +    parser.add_argument('-p', '--prefix', action='store',
>>> +                        default=DEFAULT_PREFIX,
>>>                           help="prefix for symbols")
>>>       parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>>>                           dest='unmask',
>>> @@ -32,25 +79,17 @@ def main(argv):
>>>       parser.add_argument('schema', action='store')
>>>       args = parser.parse_args()
>>>   -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?',
>>> args.prefix)
>>> -    if match.end() != len(args.prefix):
>>> -        print("%s: 'funny character '%s' in argument of --prefix"
>>> -              % (sys.argv[0], args.prefix[match.end()]),
>>> -              file=sys.stderr)
>>> -        sys.exit(1)
>>> -
>>>       try:
>>> -        schema = QAPISchema(args.schema)
>>> +        generate(args.schema,
>>> +                 output_dir=args.output_dir,
>>> +                 prefix=args.prefix,
>>> +                 unmask=args.unmask,
>>> +                 builtins=args.builtins)
>>>       except QAPIError as err:
>>> -        print(err, file=sys.stderr)
>>> -        exit(1)
>>> -
>>> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
>>> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
>>> -    gen_commands(schema, args.output_dir, args.prefix)
>>> -    gen_events(schema, args.output_dir, args.prefix)
>>> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
>>> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
>>> +        return 1
>>> +    return 0
>>>     
>>>   if __name__ == '__main__':
>>> -    main(sys.argv)
>>> +    sys.exit(main())
>> What does sys.exit() really buy us here?  I'm asking because both
>> spots
>> in the Python docs I referenced above do without.
>> 
>
> It just pushes the sys.exit out of the main function so it can be
> invoked by other machinery. (And takes the return code from main and 
> turns it into the return code for the process.)
>
> I don't think it winds up mattering for simple "console_script" entry
> points, but you don't want the called function to exit and deny the 
> caller the chance to do their own tidying post-call.
>
> You've already offered a "YAGNI", but it's just the convention I tend
> to stick to for how to structure entry points.

I'm not questioning the conventional if __name__ == '__main__' menuett.
I wonder why *we* need sys.exit() where the examples in the Python docs
don't.

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 4:12 AM, Markus Armbruster wrote:
>> I keep stumbling over things in later patches that turn out to go back
>> to this one.
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> This is a minor re-work of the entrypoint script. It isolates a
>>> generate() method from the actual command-line mechanism.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>   1 file changed, 62 insertions(+), 23 deletions(-)
>>>
>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>> index 541e8c1f55d..117b396a595 100644
>>> --- a/scripts/qapi-gen.py
>>> +++ b/scripts/qapi-gen.py
>>> @@ -1,30 +1,77 @@
>>>   #!/usr/bin/env python3
>>> -# QAPI generator
>>> -#
>>> +
>>>   # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>   # See the COPYING file in the top-level directory.
>>>   +"""
>>> +QAPI Generator
>>> +
>>> +This script is the main entry point for generating C code from the QAPI schema.
>> PEP 8: For flowing long blocks of text with fewer structural
>> restrictions (docstrings or comments), the line length should be limited
>> to 72 characters.
>> 
>
> Eugh. OK, but I don't have a good way to check or enforce this,
> admittedly. I have to change my emacs settings to understand this when
> I hit the reflow key. I don't know if the python mode has a
> context-aware reflow length.
>
> ("I don't disagree, but I'm not immediately sure right now how I will
> make sure I, or anyone else, complies with this. Low priority as a
> result?")

Emacs Python mode is close enough by default: fill-paragraph (bound to
M-q) uses variable fill-column, which defaults to 70.  If you want the
extra two columns PEP 8 grants you, I can show you how to bump it to 72
just for Python mode.

You can use fill-paragraph for code, too.  I don't myself, because I
disagree with its line breaking decisions too often (and so does PEP 8).
A better Python mode would break code lines more neatly, and with the
width defaulting to 79.

>>> +"""
>>>     import argparse
>>>   import re
>>>   import sys
>>>     from qapi.commands import gen_commands
>>> +from qapi.error import QAPIError
>>>   from qapi.events import gen_events
>>>   from qapi.introspect import gen_introspect
>>> -from qapi.schema import QAPIError, QAPISchema
>>> +from qapi.schema import QAPISchema
>>>   from qapi.types import gen_types
>>>   from qapi.visit import gen_visit
>>>     
>>> -def main(argv):
>>> +DEFAULT_OUTPUT_DIR = ''
>>> +DEFAULT_PREFIX = ''
>>> +
>>> +
>>> +def generate(schema_file: str,
>>> +             output_dir: str,
>>> +             prefix: str,
>>> +             unmask: bool = False,
>>> +             builtins: bool = False) -> None:
>>> +    """
>>> +    generate uses a given schema to produce C code in the target directory.
>> PEP 257: The docstring is a phrase ending in a period.  It
>> prescribes
>> the function or method's effect as a command ("Do this", "Return that"),
>> not as a description; e.g. don't write "Returns the pathname ...".
>> Suggest
>>         Generate C code for the given schema into the target
>> directory.
>> 
>
> OK. I don't mind trying to foster a consistent tone. I clearly
> didn't. I will add a note to my style guide todo.
>
> I give you permission to change the voice in any of my docstrings, or
> to adjust the phrasing to be more technically accurate as you see
> fit. You are the primary maintainer of this code, of course, and you
> will know best.
>
> It will be quicker to give you full permission to just change any of
> the docstrings as you see fit than it will be to play review-respin
> ping-pong.

Me rewriting your commits without your consent is putting words in your
mouth, which I don't want to do.

We can still reduce ping-pong: whenever I can, I don't just say "this
needs improvement", I propose improvements.  If you disagree, we talk.
Else, if you have to respin, you make a reasonable effort to take them.
Else, the remaining improvements are trivial (because no respin), and
I'll make them in my tree.

>>> +
>>> +    :param schema_file: The primary QAPI schema file.
>>> +    :param output_dir: The output directory to store generated code.
>>> +    :param prefix: Optional C-code prefix for symbol names.
>>> +    :param unmask: Expose non-ABI names through introspection?
>>> +    :param builtins: Generate code for built-in types?
>>> +
>>> +    :raise QAPIError: On failures.
>>> +    """
>> [...]
>> 

Re: [PATCH v5 16/36] qapi/common.py: Convert comments into docstrings, and elaborate

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 5:14 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> As docstrings, they'll show up in documentation and IDE help.
>>>
>>> The docstring style being targeted is the Sphinx documentation
>>> style. Sphinx uses an extension of ReST with "domains". We use the
>>> (implicit) Python domain, which supports a number of custom "info
>>> fields". Those info fields are documented here:
>>> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
>>>
>>> Primarily, we use `:param X: descr`, `:return[s]: descr`, and `:raise[s]
>>> Z: when`. Everything else is the Sphinx dialect of ReST.
>>>
>>> (No, nothing checks or enforces this style that I am aware of. Sphinx
>>> either chokes or succeeds, but does not enforce a standard of what is
>>> otherwise inside the docstring. Pycharm does highlight when your param
>>> fields are not aligned with the actual fields present. It does not
>>> highlight missing return or exception statements. There is no existing
>>> style guide I am aware of that covers a standard for a minimally
>>> acceptable docstring. I am debating writing one.)
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>   scripts/qapi/common.py | 53 +++++++++++++++++++++++++++++++-----------
>>>   1 file changed, 39 insertions(+), 14 deletions(-)
>>>
>>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>>> index 74a2c001ed9..0ef38ea5fe0 100644
>>> --- a/scripts/qapi/common.py
>>> +++ b/scripts/qapi/common.py
>>> @@ -15,15 +15,24 @@
>>>   from typing import Optional, Sequence
>>>     
>>> +#: Sentinel value that causes all space to its right to be removed.
>> What's the purpose of : after # ?
>> 
>
> Documents this name in Sphinx. We had a small discussion about it, I
> think; "Does using this special form or the docstring make the comment 
> visible in any IDE?" (No.)
>
> There's no Python-AST way to document these, but there is a Sphinx way
> to document them, so I did that.
>
> (Doing it like this allows `EATSPACE` to be used as a cross-reference.)

Thanks.

Consider pointing this out when you write the comment & doc string part
of our Python style guide.

>> I'm not sure this is a "sentinel value".  Wikipedia:
>>      In computer programming, a sentinel value (also referred to as a
>>      flag value, trip value, rogue value, signal value, or dummy data)[1]
>>      is a special value in the context of an algorithm which uses its
>>      presence as a condition of termination, typically in a loop or
>>      recursive algorithm.
>>      https://en.wikipedia.org/wiki/Sentinel_value
>> 
>
> I really should try to learn English as a second language so I know
> what any of the words I use mean, I guess. I had slipped to a less
> strict usage where it meant more like "placeholder".
>
>> Perhaps
>>     # Magic string value that gets removed along with all space to
>> the
>>     # right.
>> 
>
> This can be written on one line if we gently disregard the 72 column
> limit. (Maybe you already did when you wrote it and my client wrapped 
> it. Who knows!)

Drop the period and it fits ;-P

You could also drop "value" without loss.

[...]

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

[Markus Armbruster]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Wed, Oct 07, 2020 at 02:08:33PM -0400, John Snow wrote:
>> On 10/7/20 4:50 AM, Markus Armbruster wrote:
>> > John Snow <jsnow@redhat.com> writes:
>> > 
>> > > Code style tools really dislike the use of global keywords, because it
>> > > generally involves re-binding the name at runtime which can have strange
>> > > effects depending on when and how that global name is referenced in
>> > > other modules.
>> > > 
>> > > Make a little indent level manager instead.
>> > > 
>> > > Signed-off-by: John Snow <jsnow@redhat.com>
>> > > Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>> > > Reviewed-by: Cleber Rosa <crosa@redhat.com>
>> > 
>> > Intentation is a job for QAPIGen (and its subtypes).  But if this patch
>> > is easier to achieve this series' goal, I don't mind.
>> > 
>> 
>> I agree, but refactoring it properly is beyond my capacity right now.
>> 
>> This was the dumbest thing I could do to get pylint/mypy passing, which
>> required the elimination (or suppression) of the global keyword.
>> 
>> Creating a stateful object was the fastest way from A to B.
>
> An even dumber solution could be:
>
>   indent_prefixes = []
>   def push_indent(amount=4):
>       """Add `amount` spaces to indentation prefix"""
>       indent_prefixes.push(' '*amount)
>   def pop_indent():
>       """Revert the most recent push_indent() call"""
>       indent_prefixes.pop()
>   def genindent():
>       """Return the current indentation prefix"""
>       return ''.join(indent_prefixes)
>
> No global keyword involved, and the only stateful object is a
> dumb list.

Ha, this is Dumb with a capital D!  I respect that :)

John, I'm not asking you to switch.  Use your judgement.

Re: [PATCH v5 18/36] qapi: establish mypy type-checking baseline

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 5:25 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Fix two very minor issues,
>> What are the two issues?  I'm asking because I can see only one.
>> 
>
> Bitrot! The other one got fixed elsewhere and rebase didn't conflict,
> so I didn't update the commit message.
>
>>>                             and then establish a mypy type-checking
>>> baseline.
>> Any particular reason this can't go before PATCH 15 "qapi/common.py: add
>> type hint annotations".
>> 
>
> Just the way the chips fell. common.py does not check until this very
> commit.
>
> Adding a mypy config can be shuffled up earlier and earlier by
> determining which checks to ignore and adding them, but I'm not sure
> who that benefits.

I like symmetry, but it's okay as is.  I asked to make sure I'm not
missing something more subtle.

[...]

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 7:32 AM, Markus Armbruster wrote:
>> Ignorant question: what's the difference between -> None (like here) and
>> nothing (like __init__() above?
>
> This came up in Cleber's review, too.
>
> Mypy supports a gradual typing paradigm; it is designed to facilitate
> what we are doing here: the gradual adding of types until we are able
> to enforce static typing everywhere.
>
> To that end, mypy uses a simple heuristic to determine if a function
> is "typed" or "untyped": did you use any type annotations for it?
>
> Meanwhile, __init__ never returns anything. You do not need to
> annotate its return type. mypy knows what the return type is and must
> be. In the case of __init__ with no parameters, it is both untyped and
> strictly typed!
>
> Annotating the return type for no-parameter init methods convinces
> mypy that this is a strictly typed method. It doesn't do this
> automatically, because doing so might enable more checks than you were
> ready for simply because mypy was able to accurately surmise the
> typing of just __init__.
>
> So it's just a little flip switch to enable strict typing, really.
>
> --js
>
> https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

I now understand we can omit -> None, except when it's the only type
hint, because omitting it then would make it untyped, which is not what
we want.

Is this just for __init__(), or is it for any function that doesn't
return anything?

Re: [PATCH v5 23/36] qapi/commands.py: enable checking with mypy

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 7:37 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>   scripts/qapi/mypy.ini | 5 -----
>>>   1 file changed, 5 deletions(-)
>>>
>>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>>> index 5df11e53fd1..8ab9ac52cc4 100644
>>> --- a/scripts/qapi/mypy.ini
>>> +++ b/scripts/qapi/mypy.ini
>>> @@ -4,11 +4,6 @@ strict_optional = False
>>>   disallow_untyped_calls = False
>>>   python_version = 3.6
>>>   -[mypy-qapi.commands]
>>> -disallow_untyped_defs = False
>>> -disallow_incomplete_defs = False
>>> -check_untyped_defs = False
>>> -
>>>   [mypy-qapi.error]
>>>   disallow_untyped_defs = False
>>>   disallow_incomplete_defs = False
>> The equivalent change for  events.py you squashed into the commit
>> adding
>> type hints.  Any particular reason for not doing the same here?
>> 
>
> Just making my life easier for re-arranging commits and rebasing. By
> separating them out whenever I had > 1 fix patch prior, I was able to 
> freely re-arrange and re-order the prior fix patches.
>
> They can be squashed on commit if desired, but for my own sake and
> inability to predict review comments, they are separate.

I only asked because some are separate, and some are not.

Squashing on commit is an easy way toward consistency here.

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 7:55 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Annotations do not change runtime behavior.
>>> This commit *only* adds annotations.
>>>
>>> A note on typing of __init__: mypy requires init functions with no
>>> parameters to document a return type of None to be considered fully
>>> typed. In the case when there are input parameters, None may be omitted.
>>>
>>> Since __init__ may never return any value, it is preferred to omit the
>>> return annotation whenever possible.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>   scripts/qapi/mypy.ini  |  5 -----
>>>   scripts/qapi/source.py | 31 ++++++++++++++++++-------------
>>>   2 files changed, 18 insertions(+), 18 deletions(-)
>>>
>>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>>> index 8ab9ac52cc4..1b8555dfa39 100644
>>> --- a/scripts/qapi/mypy.ini
>>> +++ b/scripts/qapi/mypy.ini
>>> @@ -34,11 +34,6 @@ disallow_untyped_defs = False
>>>   disallow_incomplete_defs = False
>>>   check_untyped_defs = False
>>>   -[mypy-qapi.source]
>>> -disallow_untyped_defs = False
>>> -disallow_incomplete_defs = False
>>> -check_untyped_defs = False
>>> -
>>>   [mypy-qapi.types]
>>>   disallow_untyped_defs = False
>>>   disallow_incomplete_defs = False
>>> diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
>>> index e97b9a8e15e..1cc6a5b82dc 100644
>>> --- a/scripts/qapi/source.py
>>> +++ b/scripts/qapi/source.py
>>> @@ -11,37 +11,42 @@
>>>     import copy
>>>   import sys
>>> +from typing import List, Optional, TypeVar
>>>     
>>>   class QAPISchemaPragma:
>>> -    def __init__(self):
>>> +    def __init__(self) -> None:
>>>           # Are documentation comments required?
>>>           self.doc_required = False
>>>           # Whitelist of commands allowed to return a non-dictionary
>>> -        self.returns_whitelist = []
>>> +        self.returns_whitelist: List[str] = []
>>>           # Whitelist of entities allowed to violate case conventions
>>> -        self.name_case_whitelist = []
>>> +        self.name_case_whitelist: List[str] = []
>>>     
>>>   class QAPISourceInfo:
>>> -    def __init__(self, fname, line, parent):
>>> +    T = TypeVar('T', bound='QAPISourceInfo')
>>> +
>>> +    def __init__(self: T, fname: str, line: int, parent: Optional[T]):
>>
>> More ignorant questions (I'm abusing the review process to learn Python
>> type hints)...
>> 
>> Why do you need to annotate self here, but not elsewhere?
>
> This is admittedly me being a little extra, but I thought it was a
> good way to show a pattern for people who maybe hadn't been exposed to
> it yet.
>
> This is a pattern that allows for subclassing. I am stating that this
> __init__ method takes a parent of the same type as itself, whatever
> that happens to actually be.
>
> T is a TypeVar that we can use for Generics. By declaring the type of
> self as that TypeVar, we bind T to self's type. When we annotate the 
> parent as a T, we are enforcing that the parent we receive is of the
> same type as ourselves.
>
> (Yep, we don't actually subclass QAPISourceInfo and I don't have plans
> to. It felt slightly icky to hard-code the class type name, though. I 
> try to avoid that whenever I can. I'm not sure I would go so far as to
> call it a code smell or an antipattern, but it's something I tend to 
> avoid anyway.)

Say I define class QSISub as a direct subclass of QAPISourceInfo, and
let it inherit __init__().  What's the type of QSISub.__init__()'s
parameter @parent?

According to my reading of your explanation, it's QSISub.  Correct?

If we used QAPISourceInfo instead of T for @parent, then it would be
QAPISourceInfo.  Correct?

Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really
needs to be a QSISub.  We can't know when we write QAPISourceInfo.  But
we don't *have* to get this exactly right for all future subclasses,
because I can always override __init__() when inheritance doesn't give
me the __init__() I want.  Correct?

Say I override __init__(), and have it call super().__init__().  I have
to pass it a QAPISourceInfo @parent.  A QSISub will do (it's a subtype).
Correct?

One more: is bound='QAPISourceInfo' strictly needed?

>> Why do you use T instead of QAPISourceInfo?

[...]

Re: [PATCH v5 27/36] qapi/gen.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 9:20 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Annotations do not change runtime behavior.
>>> This commit *only* adds annotations.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>   scripts/qapi/gen.py | 104 ++++++++++++++++++++++++--------------------
>>>   1 file changed, 57 insertions(+), 47 deletions(-)
>>>
>>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>>> index 1bad37fc06b..d0391cd8718 100644
>>> --- a/scripts/qapi/gen.py
>>> +++ b/scripts/qapi/gen.py
>>> @@ -17,7 +17,13 @@
>>>   import errno
>>>   import os
>>>   import re
>>> -from typing import Optional
>>> +from typing import (
>>> +    Dict,
>>> +    Iterator,
>>> +    List,
>>> +    Optional,
>>> +    Tuple,
>>> +)
>>>     from .common import (
>>>       c_fname,
>>> @@ -29,31 +35,31 @@
>>>       mcgen,
>>>   )
>>>   from .schema import QAPISchemaObjectType, QAPISchemaVisitor
>>> +from .source import QAPISourceInfo
>> PATCH 03 has a similar cleanup.  Are there more?  Perhaps a separate
>> patch doing just this kind of cleanup would make sense.  Up to you.
>> [...]
>> 
>
> This isn't a cleanup, I am just importing QAPISourceInfo to use for an
> annotation. It's relevant and required for this patch, and doesn't
> make sense on its own.

I was mistaken.  A case of patch-review-eye...

> Patch 03 ... Oh, you mean identifying the correct location of
> QAPIError. Uh... nah? I think that was the only case of that one
> changing. Not worth pulling out or naming, I think.

Agree.

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 8:43 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> This is true by design, but not presently able to be expressed in the
>>> type system. An assertion helps mypy understand our constraints.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>   scripts/qapi/visit.py | 12 +++++++-----
>>>   1 file changed, 7 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
>>> index 14f30c228b7..4f11fd325b8 100644
>>> --- a/scripts/qapi/visit.py
>>> +++ b/scripts/qapi/visit.py
>>> @@ -22,7 +22,7 @@
>>>       mcgen,
>>>   )
>>>   from .gen import QAPISchemaModularCVisitor, ifcontext
>>> -from .schema import QAPISchemaObjectType
>>> +from .schema import QAPISchemaEnumType, QAPISchemaObjectType
>>>     
>>>   def gen_visit_decl(name, scalar=False):
>>> @@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
>>>           ret += gen_endif(memb.ifcond)
>>>         if variants:
>>> +        tag_member = variants.tag_member
>>> +        assert isinstance(tag_member.type, QAPISchemaEnumType)
>> I'm curious: do you need the local variable to make the assertion
>> stick?
>> 
>
> No, but it only sticks to the binding and not the
> data. i.e. assertions to downcast work on the *name*.

Would it stick to variants.tag_member, too?

I'm asking to learn, not to find a reason to make you change your patch.

> (This comes up somewhere in the schema.py patches where I make a
> change that looks completely pointless, but it makes mypy happy.)
>
> I could have left it alone. I just saw a lot of repeated multi-dots
> and habitually created a temporary local for the purpose.

Matter of taste.  Long chains of dots make the code hard to read because
they are so long.  Temporary variable make it hard to read because you
have to remember what they mean.  Tradeoff.  I come up with cases I find
hard to decide all too often.

In case the local variable isn't needed for mypy: when you throw in
something that isn't needed for the patch's stated purpose, it's best to
mention it in the commit message, because not mentioning it is a review
comment magnet :)

Put yourself in the reviewers shoes.  Your lovingly crafted commit
message puts him into a positive mood.  He nods along while reading your
obvious patch at a good pace.  And then he runs smack into the
unexpected unrelated part, and stops: oh, what's going on here?  Back up
some and read more slowly to make sure I understand.

>>> +
>>>           ret += mcgen('''
>>>       switch (obj->%(c_name)s) {
>>>   ''',
>>> -                     c_name=c_name(variants.tag_member.name))
>>> +                     c_name=c_name(tag_member.name))
>>>             for var in variants.variants:
>>> -            case_str = c_enum_const(variants.tag_member.type.name,
>>> -                                    var.name,
>>> -                                    variants.tag_member.type.prefix)
>>> +            case_str = c_enum_const(tag_member.type.name, var.name,
>>> +                                    tag_member.type.prefix)
>>>               ret += gen_if(var.ifcond)
>>>               if var.type.name == 'q_empty':
>>>                   # valid variant and nothing to do

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/7/20 7:49 AM, Markus Armbruster wrote:
>> Looks like commit message of PATCH 24 has an answer.
>> Copy to all the commits that omit -> None similarly?
>
> Probably not needed.
>
> It's covered by the class basics section in the mypy manual;
> https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods
>
> and if you should happen to omit annotations for __init__ entirely as
> a novice, you will be treated to messages such as these:
>
> qapi/source.py:18: error: Function is missing a return type annotation
> qapi/source.py:18: note: Use "-> None" if function does not return a value
> Found 1 error in 1 file (checked 14 source files)
>
> Pretty good error!
>
> There's no error if you DO explicitly add a -> None from __init__, but
> at worst it's just extraneous (but correct) information.

Let me apply the zero-one-infinity rule:

* Zero: explain it in none of the commit messages, i.e. dumb down PATCH
  24.

* One: explain it in one.  Do it in the first one, please (here, I
  think).

* Infinity: explain it in every one.

Up to you!

> I could add a note to the style guide that I prefer omitting the
> return from __init__. I like omitting as much as I possibly can.
>
> (You'll notice I don't always type every local, either -- when local
> inference is accurate, I leave it alone.)

Type inference can save us from repeating the obvious over and over, and
that's lovely.

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[John Snow]

On 10/8/20 3:41 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/7/20 7:32 AM, Markus Armbruster wrote:
>>> Ignorant question: what's the difference between -> None (like here) and
>>> nothing (like __init__() above?
>>
>> This came up in Cleber's review, too.
>>
>> Mypy supports a gradual typing paradigm; it is designed to facilitate
>> what we are doing here: the gradual adding of types until we are able
>> to enforce static typing everywhere.
>>
>> To that end, mypy uses a simple heuristic to determine if a function
>> is "typed" or "untyped": did you use any type annotations for it?
>>
>> Meanwhile, __init__ never returns anything. You do not need to
>> annotate its return type. mypy knows what the return type is and must
>> be. In the case of __init__ with no parameters, it is both untyped and
>> strictly typed!
>>
>> Annotating the return type for no-parameter init methods convinces
>> mypy that this is a strictly typed method. It doesn't do this
>> automatically, because doing so might enable more checks than you were
>> ready for simply because mypy was able to accurately surmise the
>> typing of just __init__.
>>
>> So it's just a little flip switch to enable strict typing, really.
>>
>> --js
>>
>> https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods
> 
> I now understand we can omit -> None, except when it's the only type
> hint, because omitting it then would make it untyped, which is not what
> we want.
> 
> Is this just for __init__(), or is it for any function that doesn't
> return anything?
> 

*just* __init__. Some other dunders have types that could be assumed, 
but aren't. I asked about this for some other well-known but 
hard-to-type dunders like __exit__ on the Typing-SIG list. Not a strong 
response yet, I might push again later.

--js

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

[John Snow]

On 10/8/20 5:06 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/7/20 8:43 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> This is true by design, but not presently able to be expressed in the
>>>> type system. An assertion helps mypy understand our constraints.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>    scripts/qapi/visit.py | 12 +++++++-----
>>>>    1 file changed, 7 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
>>>> index 14f30c228b7..4f11fd325b8 100644
>>>> --- a/scripts/qapi/visit.py
>>>> +++ b/scripts/qapi/visit.py
>>>> @@ -22,7 +22,7 @@
>>>>        mcgen,
>>>>    )
>>>>    from .gen import QAPISchemaModularCVisitor, ifcontext
>>>> -from .schema import QAPISchemaObjectType
>>>> +from .schema import QAPISchemaEnumType, QAPISchemaObjectType
>>>>      
>>>>    def gen_visit_decl(name, scalar=False):
>>>> @@ -84,15 +84,17 @@ def gen_visit_object_members(name, base, members, variants):
>>>>            ret += gen_endif(memb.ifcond)
>>>>          if variants:
>>>> +        tag_member = variants.tag_member
>>>> +        assert isinstance(tag_member.type, QAPISchemaEnumType)
>>> I'm curious: do you need the local variable to make the assertion
>>> stick?
>>>
>>
>> No, but it only sticks to the binding and not the
>> data. i.e. assertions to downcast work on the *name*.
> 
> Would it stick to variants.tag_member, too?
> 

As long as variants isn't re-bound in that scope, it should stick within 
that scope, yeah. (I think. Didn't test. It's my expectation.)

> I'm asking to learn, not to find a reason to make you change your patch.
> 
>> (This comes up somewhere in the schema.py patches where I make a
>> change that looks completely pointless, but it makes mypy happy.)
>>
>> I could have left it alone. I just saw a lot of repeated multi-dots
>> and habitually created a temporary local for the purpose.
> 
> Matter of taste.  Long chains of dots make the code hard to read because
> they are so long.  Temporary variable make it hard to read because you
> have to remember what they mean.  Tradeoff.  I come up with cases I find
> hard to decide all too often.
> 
> In case the local variable isn't needed for mypy: when you throw in
> something that isn't needed for the patch's stated purpose, it's best to
> mention it in the commit message, because not mentioning it is a review
> comment magnet :)
> 
> Put yourself in the reviewers shoes.  Your lovingly crafted commit
> message puts him into a positive mood.  He nods along while reading your
> obvious patch at a good pace.  And then he runs smack into the
> unexpected unrelated part, and stops: oh, what's going on here?  Back up
> some and read more slowly to make sure I understand.
> 

Yeah, understood. Well, part of it is knowing the review style of your 
reviewer too. It's been a while since we've personally swapped patches.

The actual process for much of this was: "There is an error! let me fix 
that error."

And then I do that, but I do so using idioms and patterns I'm familiar 
or comfortable with. And then post-hoc, not 100% of them were 
necessarily required, strictly, to fix the problem. Most of this series 
was written "one file at a time", and then split out post-fact into 
little per-warning changes.

I didn't even send part 1 until I finished typing the *entire* package, 
to make sure that things were as correct as possible from the very first 
commit.

I often don't really even notice that little changes aren't strictly 
necessary until they're challenged, it's usually not a conscious choice 
to try and sneak stuff in.

Still always trying to find a balance between "Easy to maintain and 
iterate" and "easy to review." Tough line for me.

>>>> +
>>>>            ret += mcgen('''
>>>>        switch (obj->%(c_name)s) {
>>>>    ''',
>>>> -                     c_name=c_name(variants.tag_member.name))
>>>> +                     c_name=c_name(tag_member.name))
>>>>              for var in variants.variants:
>>>> -            case_str = c_enum_const(variants.tag_member.type.name,
>>>> -                                    var.name,
>>>> -                                    variants.tag_member.type.prefix)
>>>> +            case_str = c_enum_const(tag_member.type.name, var.name,
>>>> +                                    tag_member.type.prefix)
>>>>                ret += gen_if(var.ifcond)
>>>>                if var.type.name == 'q_empty':
>>>>                    # valid variant and nothing to do

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[John Snow]

On 10/8/20 5:16 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/7/20 7:49 AM, Markus Armbruster wrote:
>>> Looks like commit message of PATCH 24 has an answer.
>>> Copy to all the commits that omit -> None similarly?
>>
>> Probably not needed.
>>
>> It's covered by the class basics section in the mypy manual;
>> https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods
>>
>> and if you should happen to omit annotations for __init__ entirely as
>> a novice, you will be treated to messages such as these:
>>
>> qapi/source.py:18: error: Function is missing a return type annotation
>> qapi/source.py:18: note: Use "-> None" if function does not return a value
>> Found 1 error in 1 file (checked 14 source files)
>>
>> Pretty good error!
>>
>> There's no error if you DO explicitly add a -> None from __init__, but
>> at worst it's just extraneous (but correct) information.
> 
> Let me apply the zero-one-infinity rule:
> 
> * Zero: explain it in none of the commit messages, i.e. dumb down PATCH
>    24.
> 
> * One: explain it in one.  Do it in the first one, please (here, I
>    think).
> 
> * Infinity: explain it in every one.
> 
> Up to you!
> 

I'm just bad at predicting which things people will want explained. I 
know people don't read the cover letters already, so I'd rather go for 
less instead of more.

I think you and Cleber each noticed a different angle of this: Cleber 
noticed the first time I *did* annotate __init__'s return and you 
noticed the first time I *didn't*.

I'll just add it here too, but I have doubts it will be useful reference 
once it's merged. I guess it doesn't hurt to add it either, I just find 
it difficult to predict what reviewers will want.

>> I could add a note to the style guide that I prefer omitting the
>> return from __init__. I like omitting as much as I possibly can.
>>
>> (You'll notice I don't always type every local, either -- when local
>> inference is accurate, I leave it alone.)
> 
> Type inference can save us from repeating the obvious over and over, and
> that's lovely.
> 

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/8/20 2:51 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/7/20 4:07 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> This is a minor re-work of the entrypoint script. It isolates a
>>>> generate() method from the actual command-line mechanism.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>    scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>>    1 file changed, 62 insertions(+), 23 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>>> index 541e8c1f55d..117b396a595 100644
>>>> --- a/scripts/qapi-gen.py
>>>> +++ b/scripts/qapi-gen.py
>>>> @@ -1,30 +1,77 @@
>>>>    #!/usr/bin/env python3
>>>> -# QAPI generator
>>>> -#
>>>> +
>>>>    # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>>    # See the COPYING file in the top-level directory.
>>>>    +"""
>>>> +QAPI Generator
>>>> +
>>>> +This script is the main entry point for generating C code from the QAPI schema.
>>>> +"""
>>>>      import argparse
>>>>    import re
>>>>    import sys
>>>>      from qapi.commands import gen_commands
>>>> +from qapi.error import QAPIError
>>>>    from qapi.events import gen_events
>>>>    from qapi.introspect import gen_introspect
>>>> -from qapi.schema import QAPIError, QAPISchema
>>>> +from qapi.schema import QAPISchema
>>>>    from qapi.types import gen_types
>>>>    from qapi.visit import gen_visit
>>>>      
>>>> -def main(argv):
>>>> +DEFAULT_OUTPUT_DIR = ''
>>>> +DEFAULT_PREFIX = ''
>>>> +
>>>> +
>>>> +def generate(schema_file: str,
>>>> +             output_dir: str,
>>>> +             prefix: str,
>>>> +             unmask: bool = False,
>>>> +             builtins: bool = False) -> None:
>>>> +    """
>>>> +    generate uses a given schema to produce C code in the target directory.
>>>> +
>>>> +    :param schema_file: The primary QAPI schema file.
>>>> +    :param output_dir: The output directory to store generated code.
>>>> +    :param prefix: Optional C-code prefix for symbol names.
>>>> +    :param unmask: Expose non-ABI names through introspection?
>>>> +    :param builtins: Generate code for built-in types?
>>>> +
>>>> +    :raise QAPIError: On failures.
>>>> +    """
>>>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>>>> +    if match.end() != len(prefix):
>>>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>>>> +            prefix[match.end()], prefix)
>>>> +        raise QAPIError('', None, msg)
>>>> +
>>>> +    schema = QAPISchema(schema_file)
>>>> +    gen_types(schema, output_dir, prefix, builtins)
>>>> +    gen_visit(schema, output_dir, prefix, builtins)
>>>> +    gen_commands(schema, output_dir, prefix)
>>>> +    gen_events(schema, output_dir, prefix)
>>>> +    gen_introspect(schema, output_dir, prefix, unmask)
>>>> +
>>>> +
>>>> +def main() -> int:
>>>> +    """
>>>> +    gapi-gen shell script entrypoint.
>>> What's a "shell script entrypoint"?
>>> Python docs talk about "when [...] run as a script":
>>> https://docs.python.org/3/library/__main__.html
>>> Similar:
>>> https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts
>>>
>>
>> "entrypoint" is Python garble for a function that can be registered as
>> a callable from the command line.
>>
>> So in a theoretical setup.py, you'd do something like:
>>
>> 'entry_points': {
>>    'console_scripts': [
>>      'qapi-gen = qapi.main:main',
>>    ]
>> }
>>
>> so when I say "shell script entrypoint", I am referring to a shell
>> script (I mean: it has a shebang and can be executed by an interactive
>> shell process) that calls the entrypoint.
> 
> It can be executed by any process.  See execve(2):
> 
>         pathname must be either a binary executable, or a script starting  with
>         a line of the form:
> 
>             #!interpreter [optional-arg]
> 
>         For details of the latter case, see "Interpreter scripts" below.
> 
> "Entry point" makes sense in Python context, "script entry point" also
> makes sense (since every Python program is a script, script is
> redundant, but not wrong).  "Shell script entry point" is misleading.
> 
>> Once (if) QAPI migrates to ./python/qemu/qapi, it will be possible to
>> just generate that script.
>>
>> (i.e. doing `pip install qemu.qapi` will install a 'qapi-gen' CLI
>> script for you. this is how packages like sphinx create the
>> 'sphinx-build' script, etc.)
>>
>>>> +    Expects arguments via sys.argv, see --help for details.
>>>> +
>>>> +    :return: int, 0 on success, 1 on failure.
>>>> +    """
>>>>        parser = argparse.ArgumentParser(
>>>>            description='Generate code from a QAPI schema')
>>>>        parser.add_argument('-b', '--builtins', action='store_true',
>>>>                            help="generate code for built-in types")
>>>> -    parser.add_argument('-o', '--output-dir', action='store', default='',
>>>> +    parser.add_argument('-o', '--output-dir', action='store',
>>>> +                        default=DEFAULT_OUTPUT_DIR,
>>>>                            help="write output to directory OUTPUT_DIR")
>>>> -    parser.add_argument('-p', '--prefix', action='store', default='',
>>>> +    parser.add_argument('-p', '--prefix', action='store',
>>>> +                        default=DEFAULT_PREFIX,
>>>>                            help="prefix for symbols")
>>>>        parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
>>>>                            dest='unmask',
>>>> @@ -32,25 +79,17 @@ def main(argv):
>>>>        parser.add_argument('schema', action='store')
>>>>        args = parser.parse_args()
>>>>    -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?',
>>>> args.prefix)
>>>> -    if match.end() != len(args.prefix):
>>>> -        print("%s: 'funny character '%s' in argument of --prefix"
>>>> -              % (sys.argv[0], args.prefix[match.end()]),
>>>> -              file=sys.stderr)
>>>> -        sys.exit(1)
>>>> -
>>>>        try:
>>>> -        schema = QAPISchema(args.schema)
>>>> +        generate(args.schema,
>>>> +                 output_dir=args.output_dir,
>>>> +                 prefix=args.prefix,
>>>> +                 unmask=args.unmask,
>>>> +                 builtins=args.builtins)
>>>>        except QAPIError as err:
>>>> -        print(err, file=sys.stderr)
>>>> -        exit(1)
>>>> -
>>>> -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
>>>> -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
>>>> -    gen_commands(schema, args.output_dir, args.prefix)
>>>> -    gen_events(schema, args.output_dir, args.prefix)
>>>> -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
>>>> +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
>>>> +        return 1
>>>> +    return 0
>>>>      
>>>>    if __name__ == '__main__':
>>>> -    main(sys.argv)
>>>> +    sys.exit(main())
>>> What does sys.exit() really buy us here?  I'm asking because both
>>> spots
>>> in the Python docs I referenced above do without.
>>>
>>
>> It just pushes the sys.exit out of the main function so it can be
>> invoked by other machinery. (And takes the return code from main and
>> turns it into the return code for the process.)
>>
>> I don't think it winds up mattering for simple "console_script" entry
>> points, but you don't want the called function to exit and deny the
>> caller the chance to do their own tidying post-call.
>>
>> You've already offered a "YAGNI", but it's just the convention I tend
>> to stick to for how to structure entry points.
> 
> I'm not questioning the conventional if __name__ == '__main__' menuett.
> I wonder why *we* need sys.exit() where the examples in the Python docs
> don't.
> 

I assume they don't care about setting that return code manually. By 
default it's going to be 0 on normal exit, and non-zero if it raises or 
aborts.

We catch the error to pretty-print it to console instead, and want to 
avoid the stack trace so we return a status code instead.

I assume that's the only difference, really.

--js

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/8/20 2:51 AM, Markus Armbruster wrote:
> It can be executed by any process.  See execve(2):
> 
>         pathname must be either a binary executable, or a script starting  with
>         a line of the form:
> 
>             #!interpreter [optional-arg]
> 
>         For details of the latter case, see "Interpreter scripts" below.
> 
> "Entry point" makes sense in Python context, "script entry point" also
> makes sense (since every Python program is a script, script is
> redundant, but not wrong).  "Shell script entry point" is misleading.

You know, I don't think I was actually explicitly aware that the #! 
shebang was not something the shell actually processed itself. Always 
learning new things.

(No, I don't think I have ever execve'd something that wasn't a binary.)

"entry point" is a little vague, an entry point for what? by whom? I was 
trying to call attention to the idea specifically that main() was 
intended as python's "console script entry point", but used the word 
"shell" instead.

"console script entrypoint" is also a lot of jargon. What I really want 
to communicate is: "When you run `qapi-gen` on your command-line, this 
is the function that runs!"

So I guess something like:

"qapi-gen executable entry point." will suffice. Please further adjust 
to your liking when staging.

--js

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/8/20 3:15 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/7/20 4:12 AM, Markus Armbruster wrote:
>>> I keep stumbling over things in later patches that turn out to go back
>>> to this one.
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> This is a minor re-work of the entrypoint script. It isolates a
>>>> generate() method from the actual command-line mechanism.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>    scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>>    1 file changed, 62 insertions(+), 23 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>>> index 541e8c1f55d..117b396a595 100644
>>>> --- a/scripts/qapi-gen.py
>>>> +++ b/scripts/qapi-gen.py
>>>> @@ -1,30 +1,77 @@
>>>>    #!/usr/bin/env python3
>>>> -# QAPI generator
>>>> -#
>>>> +
>>>>    # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>>    # See the COPYING file in the top-level directory.
>>>>    +"""
>>>> +QAPI Generator
>>>> +
>>>> +This script is the main entry point for generating C code from the QAPI schema.
>>> PEP 8: For flowing long blocks of text with fewer structural
>>> restrictions (docstrings or comments), the line length should be limited
>>> to 72 characters.
>>>
>>
>> Eugh. OK, but I don't have a good way to check or enforce this,
>> admittedly. I have to change my emacs settings to understand this when
>> I hit the reflow key. I don't know if the python mode has a
>> context-aware reflow length.
>>
>> ("I don't disagree, but I'm not immediately sure right now how I will
>> make sure I, or anyone else, complies with this. Low priority as a
>> result?")
> 
> Emacs Python mode is close enough by default: fill-paragraph (bound to
> M-q) uses variable fill-column, which defaults to 70.  If you want the
> extra two columns PEP 8 grants you, I can show you how to bump it to 72
> just for Python mode.
> 
> You can use fill-paragraph for code, too.  I don't myself, because I
> disagree with its line breaking decisions too often (and so does PEP 8).
> A better Python mode would break code lines more neatly, and with the
> width defaulting to 79.
> 

Yeah, how do I set the reflow to 72 for specific modes?

I tend to do a lot of refactoring and "prototyping" in Pycharm, but when 
it comes to bread and butter edits I still prefer emacs. I kinda bounce 
between 'em a lot. Having emacs DTRT is still useful to me.

>>>> +"""
>>>>      import argparse
>>>>    import re
>>>>    import sys
>>>>      from qapi.commands import gen_commands
>>>> +from qapi.error import QAPIError
>>>>    from qapi.events import gen_events
>>>>    from qapi.introspect import gen_introspect
>>>> -from qapi.schema import QAPIError, QAPISchema
>>>> +from qapi.schema import QAPISchema
>>>>    from qapi.types import gen_types
>>>>    from qapi.visit import gen_visit
>>>>      
>>>> -def main(argv):
>>>> +DEFAULT_OUTPUT_DIR = ''
>>>> +DEFAULT_PREFIX = ''
>>>> +
>>>> +
>>>> +def generate(schema_file: str,
>>>> +             output_dir: str,
>>>> +             prefix: str,
>>>> +             unmask: bool = False,
>>>> +             builtins: bool = False) -> None:
>>>> +    """
>>>> +    generate uses a given schema to produce C code in the target directory.
>>> PEP 257: The docstring is a phrase ending in a period.  It
>>> prescribes
>>> the function or method's effect as a command ("Do this", "Return that"),
>>> not as a description; e.g. don't write "Returns the pathname ...".
>>> Suggest
>>>          Generate C code for the given schema into the target
>>> directory.
>>>
>>
>> OK. I don't mind trying to foster a consistent tone. I clearly
>> didn't. I will add a note to my style guide todo.
>>
>> I give you permission to change the voice in any of my docstrings, or
>> to adjust the phrasing to be more technically accurate as you see
>> fit. You are the primary maintainer of this code, of course, and you
>> will know best.
>>
>> It will be quicker to give you full permission to just change any of
>> the docstrings as you see fit than it will be to play review-respin
>> ping-pong.
> 
> Me rewriting your commits without your consent is putting words in your
> mouth, which I don't want to do.
> 
> We can still reduce ping-pong: whenever I can, I don't just say "this
> needs improvement", I propose improvements.  If you disagree, we talk.
> Else, if you have to respin, you make a reasonable effort to take them.
> Else, the remaining improvements are trivial (because no respin), and
> I'll make them in my tree.
> 

I appreciate the consideration. They're not just "my" words, though, 
they are "our" words!

I do give you permission to touch up things like punctuation, voice, 
etc. If I dislike the changes I can always yelp at you post-hoc when you 
post the PR email.

So I'll reiterate that I am definitely happy with such changes -- the 
precision of the technical writing here is not my strong suit as I do 
not know QAPI as intimately as you, and it's not the focus of this 
series. I don't think of it as a personal offense that someone would 
copy-edit these things.

If you're not comfortable doing it, that's OK too, but I'm definitely 
okay with things like comment and docstring edits in particular. It's 
just something that feels hard to predict.

(Anyway, I made this change. There are likely other voice and line 
length changes needed, but I think I will try to address those 
systematically when I lift missing-docstring for pylint. Probably I will 
craft a series that creates a style guide, enables sphinx, lifts 
missing-docstring, and performs this same style of cleanup, going 
module-by-module.)

>>>> +
>>>> +    :param schema_file: The primary QAPI schema file.
>>>> +    :param output_dir: The output directory to store generated code.
>>>> +    :param prefix: Optional C-code prefix for symbol names.
>>>> +    :param unmask: Expose non-ABI names through introspection?
>>>> +    :param builtins: Generate code for built-in types?
>>>> +
>>>> +    :raise QAPIError: On failures.
>>>> +    """
>>> [...]
>>>

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/8/20 1:56 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/7/20 3:54 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> On 10/6/20 7:51 AM, Markus Armbruster wrote:
>>>>> John Snow <jsnow@redhat.com> writes:
>>>>>
>>>>>> This is a minor re-work of the entrypoint script. It isolates a
>>>>>> generate() method from the actual command-line mechanism.
>>>>>>
>>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>>>> ---
>>>>>>     scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>>>>     1 file changed, 62 insertions(+), 23 deletions(-)
>>>>>>
>>>>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>>>>> index 541e8c1f55d..117b396a595 100644
>>>>>> --- a/scripts/qapi-gen.py
>>>>>> +++ b/scripts/qapi-gen.py
>>>>>> @@ -1,30 +1,77 @@
>>>>>>     #!/usr/bin/env python3
>>>>>> -# QAPI generator
>>>>>> -#
>>>>>> +
>>>>>>     # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>>>>     # See the COPYING file in the top-level directory.
>>>>>>     +"""
>>>>>> +QAPI Generator
>>>>>> +
>>>>>> +This script is the main entry point for generating C code from the QAPI schema.
>>>>>> +"""
>>>>>>       import argparse
>>>>>>     import re
>>>>>>     import sys
>>>>>>       from qapi.commands import gen_commands
>>>>>> +from qapi.error import QAPIError
>>>>>>     from qapi.events import gen_events
>>>>>>     from qapi.introspect import gen_introspect
>>>>>> -from qapi.schema import QAPIError, QAPISchema
>>>>>> +from qapi.schema import QAPISchema
>>>>>>     from qapi.types import gen_types
>>>>>>     from qapi.visit import gen_visit
>>>>> Unrelated cleanup.  Okay.
>>>>>
>>>>>>       -def main(argv):
>>>>>> +DEFAULT_OUTPUT_DIR = ''
>>>>>> +DEFAULT_PREFIX = ''
>>>>>> +
>>>>>> +
>>>>>> +def generate(schema_file: str,
>>>>>> +             output_dir: str,
>>>>>> +             prefix: str,
>>>>>> +             unmask: bool = False,
>>>>>> +             builtins: bool = False) -> None:
>>>>>> +    """
>>>>>> +    generate uses a given schema to produce C code in the target directory.
>>>>>> +
>>>>>> +    :param schema_file: The primary QAPI schema file.
>>>>>> +    :param output_dir: The output directory to store generated code.
>>>>>> +    :param prefix: Optional C-code prefix for symbol names.
>>>>>> +    :param unmask: Expose non-ABI names through introspection?
>>>>>> +    :param builtins: Generate code for built-in types?
>>>>>> +
>>>>>> +    :raise QAPIError: On failures.
>>>>>> +    """
>>>>>> +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
>>>>>> +    if match.end() != len(prefix):
>>>>>> +        msg = "funny character '{:s}' in prefix '{:s}'".format(
>>>>>> +            prefix[match.end()], prefix)
>>>>>> +        raise QAPIError('', None, msg)
>>>>> Uh...
>>>>>        $ python3 scripts/qapi-gen.py --prefix=@ x
>>>>>        scripts/qapi-gen.py: : funny character '@' in prefix '@'
>>>>> Unwanted " :".
>>>>> This is due to a hack: you pass '' for info (*quack*).  Everything
>>>>> else
>>>>> passes QAPISourceInfo (I believe).
>>>>>
>>>>
>>>> Quack indeed - why does our base error class require so much
>>>> information from a specific part of the generation process?
>>> Because it's not "a base error class", it's a base error class for
>>> the
>>> QAPI schema compiler frontend.
>>>
>>
>> Well. It's the base for every error we /had/.
> 
> You asked why the class has the init it has, and I answered :)
> 
>>>> Ah, someone changes this in part 4 so that we have a more generic
>>>> error class to use as a base when we are missing such information.
>>> Evolving it to satisfy a need for a more widely usable error class
>>> is
>>> okay.
>>>
>>
>> Yep. It's helpful to keep a very generic form on which we grow other
>> errors from, so that things like the entry point can be written
>> legibly.
> 
> If you have a non-trivial error message format convention, you have a
> use for a function formatting error messages.
> 
> If you have a separation between diagnose and report of errors, you you
> have a use for a transport from diagnose to report.  In Python, that's
> raise.
> 
> The existing error message in main() has neither.
> 
> The existing error class QAPIError caters for the existing users.
> 
>>>> You are witnessing some more future-bleed.
>>>>> Is it really a good idea to do this in generate?  It's not about
>>>>> generating code, it's about validating a CLI option.
>>>>>
>>>>
>>>> One might also ask: Is it a good idea to only validate this on a
>>>> frontend, and not in the implementation?
>>> Yes, because that's where you can emit the better error message more
>>> easily.
>>>       $ python3 scripts/qapi-gen.py --prefix=@ x
>>>       scripts/qapi-gen.py: 'funny character '@' in argument of --prefix
>>> is better than
>>>       $ python3 scripts/qapi-gen.py --prefix=@ x
>>>       scripts/qapi-gen.py: funny character '@' in prefix '@'
>>> In generate(), the knowledge where the offending prefix value comes
>>> from
>>> is no longer available.
>>> To emit this error message, you'd have to raise a sufficiently
>>> distinct
>>> error in generate, catch it in main(), then put the error message
>>> together somehow.  Bah.
>>> Aside: there's a stray ' in the old error message.
>>>
>>>> The idea here was to create a function that could be used in a script
>>>> (for tests, debugging interfaces, other python packages) to do all of
>>>> the same things that the CLI tool did, just sans the actual CLI.
>>> YAGNI.
>>>
>>
>> It's useful for testing and debugging to be able to just call it
>> outside of the CLI, though. Maybe you won't use it, but I will.
> 
> For testing and debugging, treating "prefix is sane" as a precondition
> is fine.  I wouldn't even bother checking it.  A check would catch
> accidents, and these accidents seem vanishingly unlikely to me.
> 
> Evidence: we did without *any* prefix checking for *years*.  I added it
> in commit 1cf47a15f18 just for completeness.
> 
>> I could always add the prefix check into a tiny function and give the
>> good error message in main(), and just assert in generate() if you
>> insist on the slightly more specific error message from the CLI script.
> 
> If you genuinely think a check is needed there, that's the way to go.
> 

A goal of this patch in particular is the clean separation of the CLI 
frontend from the implementation logic behind it, such that you have:

(1) The CLI parsing, and
(2) A function implementing a task the package is capable of performing

The CLI parsing method converts sys.argv arguments into function 
parameter data that can be acted upon by another function in the package.

If checks are exclusive to the process of converting flags to argument 
data, they should stay in the CLI parsing function.

If checks prevent you from passing bad data lower into the program, I 
prefer pushing that check lower into the stack instead, such that the 
module is equivalently useful without the CLI frontend.

This conceptualization might be different from how you perceive QAPI. In 
my case, I consider the public functions and modules themselves to be 
interface that should behave reasonably and with some defensiveness. You 
might only count the CLI script itself in that category.

In this case, the prefix is passed to many different generators, so it 
seems good to check it before we hand it off to all those different places.

You don't like the loss of precision that causes for the CLI frontend 
error, which is reasonable.

I consider the function interface and the CLI frontend "equivalent", and 
so I found it odd to reject bad data in one, but not the other. IOW, I 
do not consider the CLI frontend the only user of generate().

I think it's minor in this case, sure. I am fine with saying "A valid 
prefix is a precondition", but then I want to document and enforce that 
precondition -- so I did wind up using an assert in my pending respin.

--js

>>>> Wouldn't make sense to allow garbage to flow in from one interface but
>>>> not the other; so the check is here.
>>> "@prefix is sane" is a precondition of generate().
>>> When there's a real risk of preconditions getting violated, or
>>> readers
>>> getting confused about preconditions, check them with assert.
> [...]
> 

Re: [PATCH v5 11/36] qapi/common.py: Add indent manager

[John Snow]

On 10/8/20 3:23 AM, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
>> On Wed, Oct 07, 2020 at 02:08:33PM -0400, John Snow wrote:
>>> On 10/7/20 4:50 AM, Markus Armbruster wrote:
>>>> John Snow <jsnow@redhat.com> writes:
>>>>
>>>>> Code style tools really dislike the use of global keywords, because it
>>>>> generally involves re-binding the name at runtime which can have strange
>>>>> effects depending on when and how that global name is referenced in
>>>>> other modules.
>>>>>
>>>>> Make a little indent level manager instead.
>>>>>
>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>>
>>>> Intentation is a job for QAPIGen (and its subtypes).  But if this patch
>>>> is easier to achieve this series' goal, I don't mind.
>>>>
>>>
>>> I agree, but refactoring it properly is beyond my capacity right now.
>>>
>>> This was the dumbest thing I could do to get pylint/mypy passing, which
>>> required the elimination (or suppression) of the global keyword.
>>>
>>> Creating a stateful object was the fastest way from A to B.
>>
>> An even dumber solution could be:
>>
>>    indent_prefixes = []
>>    def push_indent(amount=4):
>>        """Add `amount` spaces to indentation prefix"""
>>        indent_prefixes.push(' '*amount)
>>    def pop_indent():
>>        """Revert the most recent push_indent() call"""
>>        indent_prefixes.pop()
>>    def genindent():
>>        """Return the current indentation prefix"""
>>        return ''.join(indent_prefixes)
>>
>> No global keyword involved, and the only stateful object is a
>> dumb list.
> 
> Ha, this is Dumb with a capital D!  I respect that :)
> 
> John, I'm not asking you to switch.  Use your judgement.
> 

It's something we'll revisit soon, I'm sure. it's not good to be 
managing indent in so many different ways in so many different places.

(I prefer to leave it alone for now to try and press forward with 
accomplishing regression checks on strict typing.)

--js

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/8/20 2:51 AM, Markus Armbruster wrote:
>> It can be executed by any process.  See execve(2):
>>
>>         pathname must be either a binary executable, or a script starting  with
>>         a line of the form:
>>             #!interpreter [optional-arg]
>>         For details of the latter case, see "Interpreter scripts"
>> below.
>>
>> "Entry point" makes sense in Python context, "script entry point" also
>> makes sense (since every Python program is a script, script is
>> redundant, but not wrong).  "Shell script entry point" is misleading.
>
> You know, I don't think I was actually explicitly aware that the #!
> shebang was not something the shell actually processed itself. Always 
> learning new things.
>
> (No, I don't think I have ever execve'd something that wasn't a binary.)

I'm sure you've done it countless times, just not knowingly :)

> "entry point" is a little vague, an entry point for what? by whom? I
> was trying to call attention to the idea specifically that main() was 
> intended as python's "console script entry point", but used the word
> "shell" instead.
>
> "console script entrypoint" is also a lot of jargon. What I really

It is.

If I didn't find "console script" so thoroughly misguided, I'd advie you
to stick to it just because it's what we found in Python's docs.  It's
misguided, because this entry point is the one and only entry point for
*any* kind of Python executable program, be it console, GUI, or
whatever.

> want to communicate is: "When you run `qapi-gen` on your command-line,
> this is the function that runs!"
>
> So I guess something like:
>
> "qapi-gen executable entry point." will suffice. Please further adjust
> to your liking when staging.

Works for me.

Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 10/8/20 3:15 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 10/7/20 4:12 AM, Markus Armbruster wrote:
>>>> I keep stumbling over things in later patches that turn out to go back
>>>> to this one.
>>>> John Snow <jsnow@redhat.com> writes:
>>>>
>>>>> This is a minor re-work of the entrypoint script. It isolates a
>>>>> generate() method from the actual command-line mechanism.
>>>>>
>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>>> ---
>>>>>    scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
>>>>>    1 file changed, 62 insertions(+), 23 deletions(-)
>>>>>
>>>>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>>>>> index 541e8c1f55d..117b396a595 100644
>>>>> --- a/scripts/qapi-gen.py
>>>>> +++ b/scripts/qapi-gen.py
>>>>> @@ -1,30 +1,77 @@
>>>>>    #!/usr/bin/env python3
>>>>> -# QAPI generator
>>>>> -#
>>>>> +
>>>>>    # This work is licensed under the terms of the GNU GPL, version 2 or later.
>>>>>    # See the COPYING file in the top-level directory.
>>>>>    +"""
>>>>> +QAPI Generator
>>>>> +
>>>>> +This script is the main entry point for generating C code from the QAPI schema.
>>>> PEP 8: For flowing long blocks of text with fewer structural
>>>> restrictions (docstrings or comments), the line length should be limited
>>>> to 72 characters.
>>>>
>>>
>>> Eugh. OK, but I don't have a good way to check or enforce this,
>>> admittedly. I have to change my emacs settings to understand this when
>>> I hit the reflow key. I don't know if the python mode has a
>>> context-aware reflow length.
>>>
>>> ("I don't disagree, but I'm not immediately sure right now how I will
>>> make sure I, or anyone else, complies with this. Low priority as a
>>> result?")
>> 
>> Emacs Python mode is close enough by default: fill-paragraph (bound to
>> M-q) uses variable fill-column, which defaults to 70.  If you want the
>> extra two columns PEP 8 grants you, I can show you how to bump it to 72
>> just for Python mode.
>> 
>> You can use fill-paragraph for code, too.  I don't myself, because I
>> disagree with its line breaking decisions too often (and so does PEP 8).
>> A better Python mode would break code lines more neatly, and with the
>> width defaulting to 79.
>
> Yeah, how do I set the reflow to 72 for specific modes?
>
> I tend to do a lot of refactoring and "prototyping" in Pycharm, but
> when it comes to bread and butter edits I still prefer emacs. I kinda
> bounce between 'em a lot. Having emacs DTRT is still useful to me.

In your .emacs:

    (add-hook 'python-mode-hook
              (lambda () (setq fill-column 72)))


Happy to explain this in detail if you're curious.

[...]

Re: [PATCH v5 19/36] qapi/events.py: add type hint annotations

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

[...]
> I just find it difficult to predict what reviewers will want.

Capricious bunch, those reviewers.

[...]

Re: [PATCH v5 34/36] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

[...]
> Still always trying to find a balance between "Easy to maintain and
> iterate" and "easy to review." Tough line for me.

I assure you it's a tough line for everyone.

[...]

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

[John Snow]

On 10/8/20 4:42 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 10/7/20 7:55 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> Annotations do not change runtime behavior.
>>>> This commit *only* adds annotations.
>>>>
>>>> A note on typing of __init__: mypy requires init functions with no
>>>> parameters to document a return type of None to be considered fully
>>>> typed. In the case when there are input parameters, None may be omitted.
>>>>
>>>> Since __init__ may never return any value, it is preferred to omit the
>>>> return annotation whenever possible.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>> ---
>>>>    scripts/qapi/mypy.ini  |  5 -----
>>>>    scripts/qapi/source.py | 31 ++++++++++++++++++-------------
>>>>    2 files changed, 18 insertions(+), 18 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>>>> index 8ab9ac52cc4..1b8555dfa39 100644
>>>> --- a/scripts/qapi/mypy.ini
>>>> +++ b/scripts/qapi/mypy.ini
>>>> @@ -34,11 +34,6 @@ disallow_untyped_defs = False
>>>>    disallow_incomplete_defs = False
>>>>    check_untyped_defs = False
>>>>    -[mypy-qapi.source]
>>>> -disallow_untyped_defs = False
>>>> -disallow_incomplete_defs = False
>>>> -check_untyped_defs = False
>>>> -
>>>>    [mypy-qapi.types]
>>>>    disallow_untyped_defs = False
>>>>    disallow_incomplete_defs = False
>>>> diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
>>>> index e97b9a8e15e..1cc6a5b82dc 100644
>>>> --- a/scripts/qapi/source.py
>>>> +++ b/scripts/qapi/source.py
>>>> @@ -11,37 +11,42 @@
>>>>      import copy
>>>>    import sys
>>>> +from typing import List, Optional, TypeVar
>>>>      
>>>>    class QAPISchemaPragma:
>>>> -    def __init__(self):
>>>> +    def __init__(self) -> None:
>>>>            # Are documentation comments required?
>>>>            self.doc_required = False
>>>>            # Whitelist of commands allowed to return a non-dictionary
>>>> -        self.returns_whitelist = []
>>>> +        self.returns_whitelist: List[str] = []
>>>>            # Whitelist of entities allowed to violate case conventions
>>>> -        self.name_case_whitelist = []
>>>> +        self.name_case_whitelist: List[str] = []
>>>>      
>>>>    class QAPISourceInfo:
>>>> -    def __init__(self, fname, line, parent):
>>>> +    T = TypeVar('T', bound='QAPISourceInfo')
>>>> +
>>>> +    def __init__(self: T, fname: str, line: int, parent: Optional[T]):
>>>
>>> More ignorant questions (I'm abusing the review process to learn Python
>>> type hints)...
>>>
>>> Why do you need to annotate self here, but not elsewhere?
>>
>> This is admittedly me being a little extra, but I thought it was a
>> good way to show a pattern for people who maybe hadn't been exposed to
>> it yet.
>>
>> This is a pattern that allows for subclassing. I am stating that this
>> __init__ method takes a parent of the same type as itself, whatever
>> that happens to actually be.
>>
>> T is a TypeVar that we can use for Generics. By declaring the type of
>> self as that TypeVar, we bind T to self's type. When we annotate the
>> parent as a T, we are enforcing that the parent we receive is of the
>> same type as ourselves.
>>
>> (Yep, we don't actually subclass QAPISourceInfo and I don't have plans
>> to. It felt slightly icky to hard-code the class type name, though. I
>> try to avoid that whenever I can. I'm not sure I would go so far as to
>> call it a code smell or an antipattern, but it's something I tend to
>> avoid anyway.)
> 
> Say I define class QSISub as a direct subclass of QAPISourceInfo, and
> let it inherit __init__().  What's the type of QSISub.__init__()'s
> parameter @parent?
> 
> According to my reading of your explanation, it's QSISub.  Correct?
> 

That's right.

(I'm realizing that this is maybe not a constraint that we should even 
anticipate here, because maybe we don't wish to say that the parent 
should always be of the same type. but hey, it led to a good mypy lesson.

I'm going to edit it to do the simpler thing for now and leave well 
enough alone. There's another chance to see an interesting pattern of 
TypeVars in the error series in part 4 that I think is actually more 
explicitly appropriate.)

> If we used QAPISourceInfo instead of T for @parent, then it would be
> QAPISourceInfo.  Correct?
> 

Yup!

Here's a little sample program that shows what this kind of typing does:

```
from typing import TypeVar, Optional

class Example:
     T = TypeVar('T', bound='Example')
     def __init__(self: T, parent: Optional[T] = None):
         self.parent = parent

class SubExample(Example):
     pass


x = Example()
y = Example(x)
z = SubExample()
a = Example(x)            # OK
b = Example(z)            # OK
c = SubExample(x)         # BZZZT
d = SubExample(z)         # OK
```

If you check this with mypy, you'll get this error:

```
test.py:17: error: Argument 1 to "SubExample" has incompatible type 
"Example"; expected "Optional[SubExample]"
Found 1 error in 1 file (checked 1 source file)
```

> Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really
> needs to be a QSISub.  We can't know when we write QAPISourceInfo.  But
> we don't *have* to get this exactly right for all future subclasses,
> because I can always override __init__() when inheritance doesn't give
> me the __init__() I want.  Correct?
> 

You could, but I suggested on IRC the other day that I am not fully 
comfortable with the LSP rules that mypy (sometimes?) enforces. I tend 
not to want to override init with narrower types if I can avoid it, but 
it is true that we do this quite a lot in the codebase already.

(I believe I have seen mypy throw errors about this on occasion, but I 
can't pinpoint the exact circumstances in which it does. It's a point of 
confusion for me.)

> Say I override __init__(), and have it call super().__init__().  I have
> to pass it a QAPISourceInfo @parent.  A QSISub will do (it's a subtype).
> Correct?
> 
> One more: is bound='QAPISourceInfo' strictly needed?
> 

I'm not sure if bound is strictly required or not. mypy docs just use it 
outright and don't mention why:

https://mypy.readthedocs.io/en/stable/generics.html#generic-methods-and-generic-self

Generally, this constrains a TypeVar to some upper bound, see --

https://mypy.readthedocs.io/en/stable/generics.html#type-variables-with-upper-bounds 


-- but in this case, we're only using that TypeVar for an init method 
that only exists for this type and below, so it might be redundant.

I modified my example program to remove the bound and it appears to fail 
in the same exact way, so it might be pointless in this case. It might 
have a stronger use if you want to re-use the 'T' typevar elsewhere 
where it isn't implicitly bound by the 'self' argument. Maybe it has 
implications for multi-inheritance too, I've not tested it to that level.


Thanks for the good question!

--js

Re: [PATCH v5 24/36] qapi/source.py: add type hint annotations

[John Snow]

On 10/9/20 10:30 AM, John Snow wrote:
> On 10/8/20 4:42 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>>
>>> On 10/7/20 7:55 AM, Markus Armbruster wrote:
>>>> John Snow <jsnow@redhat.com> writes:
>>>>
>>>>> Annotations do not change runtime behavior.
>>>>> This commit *only* adds annotations.
>>>>>
>>>>> A note on typing of __init__: mypy requires init functions with no
>>>>> parameters to document a return type of None to be considered fully
>>>>> typed. In the case when there are input parameters, None may be 
>>>>> omitted.
>>>>>
>>>>> Since __init__ may never return any value, it is preferred to omit the
>>>>> return annotation whenever possible.
>>>>>
>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>>> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>>>>> Reviewed-by: Cleber Rosa <crosa@redhat.com>
>>>>> Tested-by: Cleber Rosa <crosa@redhat.com>
>>>>> ---
>>>>>    scripts/qapi/mypy.ini  |  5 -----
>>>>>    scripts/qapi/source.py | 31 ++++++++++++++++++-------------
>>>>>    2 files changed, 18 insertions(+), 18 deletions(-)
>>>>>
>>>>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>>>>> index 8ab9ac52cc4..1b8555dfa39 100644
>>>>> --- a/scripts/qapi/mypy.ini
>>>>> +++ b/scripts/qapi/mypy.ini
>>>>> @@ -34,11 +34,6 @@ disallow_untyped_defs = False
>>>>>    disallow_incomplete_defs = False
>>>>>    check_untyped_defs = False
>>>>>    -[mypy-qapi.source]
>>>>> -disallow_untyped_defs = False
>>>>> -disallow_incomplete_defs = False
>>>>> -check_untyped_defs = False
>>>>> -
>>>>>    [mypy-qapi.types]
>>>>>    disallow_untyped_defs = False
>>>>>    disallow_incomplete_defs = False
>>>>> diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
>>>>> index e97b9a8e15e..1cc6a5b82dc 100644
>>>>> --- a/scripts/qapi/source.py
>>>>> +++ b/scripts/qapi/source.py
>>>>> @@ -11,37 +11,42 @@
>>>>>      import copy
>>>>>    import sys
>>>>> +from typing import List, Optional, TypeVar
>>>>>    class QAPISchemaPragma:
>>>>> -    def __init__(self):
>>>>> +    def __init__(self) -> None:
>>>>>            # Are documentation comments required?
>>>>>            self.doc_required = False
>>>>>            # Whitelist of commands allowed to return a non-dictionary
>>>>> -        self.returns_whitelist = []
>>>>> +        self.returns_whitelist: List[str] = []
>>>>>            # Whitelist of entities allowed to violate case conventions
>>>>> -        self.name_case_whitelist = []
>>>>> +        self.name_case_whitelist: List[str] = []
>>>>>    class QAPISourceInfo:
>>>>> -    def __init__(self, fname, line, parent):
>>>>> +    T = TypeVar('T', bound='QAPISourceInfo')
>>>>> +
>>>>> +    def __init__(self: T, fname: str, line: int, parent: 
>>>>> Optional[T]):
>>>>
>>>> More ignorant questions (I'm abusing the review process to learn Python
>>>> type hints)...
>>>>
>>>> Why do you need to annotate self here, but not elsewhere?
>>>
>>> This is admittedly me being a little extra, but I thought it was a
>>> good way to show a pattern for people who maybe hadn't been exposed to
>>> it yet.
>>>
>>> This is a pattern that allows for subclassing. I am stating that this
>>> __init__ method takes a parent of the same type as itself, whatever
>>> that happens to actually be.
>>>
>>> T is a TypeVar that we can use for Generics. By declaring the type of
>>> self as that TypeVar, we bind T to self's type. When we annotate the
>>> parent as a T, we are enforcing that the parent we receive is of the
>>> same type as ourselves.
>>>
>>> (Yep, we don't actually subclass QAPISourceInfo and I don't have plans
>>> to. It felt slightly icky to hard-code the class type name, though. I
>>> try to avoid that whenever I can. I'm not sure I would go so far as to
>>> call it a code smell or an antipattern, but it's something I tend to
>>> avoid anyway.)
>>
>> Say I define class QSISub as a direct subclass of QAPISourceInfo, and
>> let it inherit __init__().  What's the type of QSISub.__init__()'s
>> parameter @parent?
>>
>> According to my reading of your explanation, it's QSISub.  Correct?
>>
> 
> That's right.
> 
> (I'm realizing that this is maybe not a constraint that we should even 
> anticipate here, because maybe we don't wish to say that the parent 
> should always be of the same type. but hey, it led to a good mypy lesson.
> 
> I'm going to edit it to do the simpler thing for now and leave well 
> enough alone. There's another chance to see an interesting pattern of 
> TypeVars in the error series in part 4 that I think is actually more 
> explicitly appropriate.)
> 
>> If we used QAPISourceInfo instead of T for @parent, then it would be
>> QAPISourceInfo.  Correct?
>>
> 
> Yup!
> 
> Here's a little sample program that shows what this kind of typing does:
> 
> ```
> from typing import TypeVar, Optional
> 
> class Example:
>      T = TypeVar('T', bound='Example')
>      def __init__(self: T, parent: Optional[T] = None):
>          self.parent = parent
> 
> class SubExample(Example):
>      pass
> 
> 
> x = Example()
> y = Example(x)
> z = SubExample()
> a = Example(x)            # OK
> b = Example(z)            # OK
> c = SubExample(x)         # BZZZT
> d = SubExample(z)         # OK
> ```
> 
> If you check this with mypy, you'll get this error:
> 
> ```
> test.py:17: error: Argument 1 to "SubExample" has incompatible type 
> "Example"; expected "Optional[SubExample]"
> Found 1 error in 1 file (checked 1 source file)
> ```
> 
>> Now, perhaps any QAPISourceInfo will do as @parent, perhaps it really
>> needs to be a QSISub.  We can't know when we write QAPISourceInfo.  But
>> we don't *have* to get this exactly right for all future subclasses,
>> because I can always override __init__() when inheritance doesn't give
>> me the __init__() I want.  Correct?
>>
> 
> You could, but I suggested on IRC the other day that I am not fully 
> comfortable with the LSP rules that mypy (sometimes?) enforces. I tend 
> not to want to override init with narrower types if I can avoid it, but 
> it is true that we do this quite a lot in the codebase already.
> 
> (I believe I have seen mypy throw errors about this on occasion, but I 
> can't pinpoint the exact circumstances in which it does. It's a point of 
> confusion for me.)
> 
>> Say I override __init__(), and have it call super().__init__().  I have
>> to pass it a QAPISourceInfo @parent.  A QSISub will do (it's a subtype).
>> Correct?
>>
>> One more: is bound='QAPISourceInfo' strictly needed?
>>
> 
> I'm not sure if bound is strictly required or not. mypy docs just use it 
> outright and don't mention why:
> 
> https://mypy.readthedocs.io/en/stable/generics.html#generic-methods-and-generic-self 
> 
> 
> Generally, this constrains a TypeVar to some upper bound, see --
> 
> https://mypy.readthedocs.io/en/stable/generics.html#type-variables-with-upper-bounds 
> 
> 
> -- but in this case, we're only using that TypeVar for an init method 
> that only exists for this type and below, so it might be redundant.
> 
> I modified my example program to remove the bound and it appears to fail 
> in the same exact way, so it might be pointless in this case. It might 
> have a stronger use if you want to re-use the 'T' typevar elsewhere 
> where it isn't implicitly bound by the 'self' argument. Maybe it has 
> implications for multi-inheritance too, I've not tested it to that level.
> 

SPOKE TOO SOON, ADDENDUM:

Adding bound= is still necessary to allow type inference to assume the 
minimal base form. It's not needed for my toy example because I don't do 
anything *with* the variable.

In this case, though, the next_line method needs help:

     def next_line(self: T) -> T:
	info = copy.copy(self)
         info.line += 1
         return info

Now, when we return a copy of ourselves, we are going to return 
something that is the same type as ourselves. If we remove the bound 
from the TypeVar here, "info.line" will become an error because mypy 
cannot infer the minimum form.

(It doesn't seem smart enough to notice that the class it is defined in 
does not inherit from anywhere, so it MUST be at least a QAPISourceInfo.)

So I am going to:

(1) Make __init__'s typing less dynamic, because it's not really needed.
(2) Leave the TypeVar in for the benefit of next_line().

--js