[PATCH 00/37] qapi: static typing conversion, pt1

[PATCH 00/37] qapi: static typing conversion, pt1

[John Snow]

Hi, this series starts adding static type hints to the QAPI module. As
you can see, the series started getting quite a bit long, so this is
only a partial conversion that focuses on a handful of the easier files.

The tougher files -- schema.py, expr.py, parser.py -- will each receive
their own series as a follow-up to this one.

Notes:

- This requires Python 3.6+. Python 3.5 is EOL, so let's do that.

- Any patch named "add notational type hints" changes ONLY signatures,
  which have no runtime impact whatsoever. These are big patches,
  but fairly straightforward.

- Most other patches are as bite-sized as possible, generally fixing one
  single warning.

- After patch 6, `flake8 qapi/` should pass 100% on this and every
  future commit.

- After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
  on this and every future commit.

- After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
  100% on this and every future commit.

Preliminary refactoring and prerequisites:

001/37: 'python: Require 3.6+'
002/37: '[DO-NOT-MERGE] qapi: add debugging tools'
003/37: 'qapi-gen: Separate arg-parsing from generation'
004/37: 'qapi: move generator entrypoint into module'
005/37: 'qapi: Remove wildcard includes'
006/37: 'qapi: delint using flake8'
007/37: 'qapi: add pylintrc'

common.py (and params.py):

008/37: 'qapi/common.py: Remove python compatibility workaround'
009/37: 'qapi/common.py: Add indent manager'
010/37: 'qapi/common.py: delint with pylint'
011/37: 'qapi/common.py: Replace one-letter 'c' variable'
012/37: 'qapi/common.py: check with pylint'
013/37: 'qapi/common.py: add notational type hints'
014/37: 'qapi/common.py: Move comments into docstrings'
015/37: 'qapi/common.py: split build_params into new file'
016/37: 'qapi: establish mypy type-checking baseline'

events.py:

017/37: 'qapi/events.py: add notational type hints'
018/37: 'qapi/events.py: Move comments into docstrings'

commands.py:

019/37: 'qapi/commands.py: Don't re-bind to variable of different type'
020/37: 'qapi/commands.py: add notational type hints'
021/37: 'qapi/commands.py: enable checking with mypy'

source.py:

022/37: 'qapi/source.py: add notational type hints'
023/37: 'qapi/source.py: delint with pylint'

gen.py:

024/37: 'qapi/gen.py: Fix edge-case of _is_user_module'
025/37: 'qapi/gen.py: add notational type hints'
026/37: 'qapi/gen.py: Enable checking with mypy'
027/37: 'qapi/gen.py: Remove unused parameter'
028/37: 'qapi/gen.py: update write() to be more idiomatic'
029/37: 'qapi/gen.py: delint with pylint'

introspect.py:

030/37: 'qapi/introspect.py: Add a typed 'extra' structure'
031/37: 'qapi/introspect.py: add _gen_features helper'
032/37: 'qapi/introspect.py: create a typed 'Node' data structure'
033/37: 'qapi/introspect.py: add notational type hints'

types.py:

034/37: 'qapi/types.py: add notational type hints'
035/37: 'qapi/types.py: remove one-letter variables'

visit.py:

036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object'
037/37: 'qapi/visit.py: add notational type hints'

John Snow (37):
  python: Require 3.6+
  [DO-NOT-MERGE] qapi: add debugging tools
  qapi-gen: Separate arg-parsing from generation
  qapi: move generator entrypoint into module
  qapi: Remove wildcard includes
  qapi: delint using flake8
  qapi: add pylintrc
  qapi/common.py: Remove python compatibility workaround
  qapi/common.py: Add indent manager
  qapi/common.py: delint with pylint
  qapi/common.py: Replace one-letter 'c' variable
  qapi/common.py: check with pylint
  qapi/common.py: add notational type hints
  qapi/common.py: Move comments into docstrings
  qapi/common.py: split build_params into new file
  qapi: establish mypy type-checking baseline
  qapi/events.py: add notational type hints
  qapi/events.py: Move comments into docstrings
  qapi/commands.py: Don't re-bind to variable of different type
  qapi/commands.py: add notational type hints
  qapi/commands.py: enable checking with mypy
  qapi/source.py: add notational type hints
  qapi/source.py: delint with pylint
  qapi/gen.py: Fix edge-case of _is_user_module
  qapi/gen.py: add notational type hints
  qapi/gen.py: Enable checking with mypy
  qapi/gen.py: Remove unused parameter
  qapi/gen.py: update write() to be more idiomatic
  qapi/gen.py: delint with pylint
  qapi/introspect.py: Add a typed 'extra' structure
  qapi/introspect.py: add _gen_features helper
  qapi/introspect.py: create a typed 'Node' data structure
  qapi/introspect.py: add notational type hints
  qapi/types.py: add notational type hints
  qapi/types.py: remove one-letter variables
  qapi/visit.py: remove unused parameters from gen_visit_object
  qapi/visit.py: add notational type hints

 docs/conf.py                  |   4 +-
 configure                     |   6 +-
 .readthedocs.yml              |   2 +-
 .travis.yml                   |   8 --
 scripts/qapi-gen.py           |  59 ++--------
 scripts/qapi/.flake8          |   2 +
 scripts/qapi/commands.py      |  85 +++++++++++----
 scripts/qapi/common.py        | 161 +++++++++++++++------------
 scripts/qapi/debug.py         |  78 +++++++++++++
 scripts/qapi/doc.py           |   3 +-
 scripts/qapi/events.py        |  61 ++++++++---
 scripts/qapi/expr.py          |   4 +-
 scripts/qapi/gen.py           | 147 +++++++++++++------------
 scripts/qapi/introspect.py    | 200 +++++++++++++++++++++++-----------
 scripts/qapi/mypy.ini         |  30 +++++
 scripts/qapi/params.py        |  40 +++++++
 scripts/qapi/parser.py        |   4 +-
 scripts/qapi/pylintrc         |  71 ++++++++++++
 scripts/qapi/schema.py        |  33 +++---
 scripts/qapi/script.py        |  91 ++++++++++++++++
 scripts/qapi/source.py        |  34 +++---
 scripts/qapi/types.py         | 125 ++++++++++++++-------
 scripts/qapi/visit.py         | 104 +++++++++++++-----
 tests/qemu-iotests/iotests.py |   2 -
 24 files changed, 947 insertions(+), 407 deletions(-)
 create mode 100644 scripts/qapi/.flake8
 create mode 100644 scripts/qapi/debug.py
 create mode 100644 scripts/qapi/mypy.ini
 create mode 100644 scripts/qapi/params.py
 create mode 100644 scripts/qapi/pylintrc
 create mode 100644 scripts/qapi/script.py

-- 
2.26.2

[PATCH 01/37] python: Require 3.6+

[John Snow]

Python 3.5 is now EOL. Python 3.6 brings with it several nice things:

- Literal string interpolation (PEP 498)
- Syntax for variable annotations (PEP 526)
- Preserving keyword argument order (PEP 468) *

PEP 526 in particular will allow us to convert the QAPI module to the
statically typed subset of the Python language. These static type hints
improve the interactive editing experience in IDEs and help make
refactoring and adding new features faster, easier and safer.

*Note: **kwargs is now guaranteed to preserve keyword ordering, but dict
itself is still allowed to have an arbitrary order based on
implementation. It is not until Python 3.7 that the dict type guarantees
insertion order.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py                  | 4 ++--
 configure                     | 6 +++---
 .readthedocs.yml              | 2 +-
 .travis.yml                   | 8 --------
 tests/qemu-iotests/iotests.py | 2 --
 5 files changed, 6 insertions(+), 16 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index 0dbd90dc11..8aeac40124 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -36,9 +36,9 @@
 # In newer versions of Sphinx this will display nicely; in older versions
 # Sphinx will also produce a Python backtrace but at least the information
 # gets printed...
-if sys.version_info < (3,5):
+if sys.version_info < (3,6):
     raise ConfigError(
-        "QEMU requires a Sphinx that uses Python 3.5 or better\n")
+        "QEMU requires a Sphinx that uses Python 3.6 or better\n")
 
 # The per-manual conf.py will set qemu_docdir for a single-manual build;
 # otherwise set it here if this is an entire-manual-set build.
diff --git a/configure b/configure
index ce27eafb0a..33292500e7 100755
--- a/configure
+++ b/configure
@@ -913,7 +913,7 @@ fi
 
 : ${make=${MAKE-make}}
 
-# We prefer python 3.x. A bare 'python' is traditionally
+# We prefer python 3.6+. A bare 'python' is traditionally
 # python 2.x, but some distros have it as python 3.x, so
 # we check that too
 python=
@@ -1961,8 +1961,8 @@ fi
 
 # Note that if the Python conditional here evaluates True we will exit
 # with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
-  error_exit "Cannot use '$python', Python >= 3.5 is required." \
+if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
+  error_exit "Cannot use '$python', Python >= 3.6 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
diff --git a/.readthedocs.yml b/.readthedocs.yml
index 8355dbc634..7fb7b8dd61 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -17,4 +17,4 @@ formats: all
 # we require for other Python in our codebase (our conf.py
 # enforces this, and some code needs it.)
 python:
-  version: 3.5
+  version: 3.6
diff --git a/.travis.yml b/.travis.yml
index 65341634d0..0dca15a5e8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -267,14 +267,6 @@ jobs:
 
 
     # Python builds
-    - name: "GCC Python 3.5 (x86_64-softmmu)"
-      env:
-        - CONFIG="--target-list=x86_64-softmmu"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
-      language: python
-      python: 3.5
-
-
     - name: "GCC Python 3.6 (x86_64-softmmu)"
       env:
         - CONFIG="--target-list=x86_64-softmmu"
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 91e4a57126..f48460480a 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -40,8 +40,6 @@
 from qemu import qtest
 from qemu.qmp import QMPMessage
 
-assert sys.version_info >= (3, 6)
-
 # Use this logger for logging messages directly from the iotests module
 logger = logging.getLogger('qemu.iotests')
 logger.addHandler(logging.NullHandler())
-- 
2.26.2

[PATCH 02/37] [DO-NOT-MERGE] qapi: add debugging tools

[John Snow]

This adds some really childishly simple debugging tools. Maybe they're
interesting for someone else, too?

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/debug.py | 78 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
 create mode 100644 scripts/qapi/debug.py

diff --git a/scripts/qapi/debug.py b/scripts/qapi/debug.py
new file mode 100644
index 0000000000..bacf5ee180
--- /dev/null
+++ b/scripts/qapi/debug.py
@@ -0,0 +1,78 @@
+"""
+Small debugging facilities for mypy static analysis work.
+(C) 2020 John Snow, for Red Hat, Inc.
+"""
+
+import inspect
+import json
+from typing import Dict, List, Any
+from types import FrameType
+
+
+OBSERVED_TYPES: Dict[str, List[str]] = {}
+
+
+# You have no idea how long it took to find this return type...
+def caller_frame() -> FrameType:
+    """
+    Returns the stack frame of the caller's caller.
+    e.g. foo() -> caller() -> caller_frame() return's foo's stack frame.
+    """
+    stack = inspect.stack()
+    caller = stack[2].frame
+    if caller is None:
+        msg = "Python interpreter does not support stack frame inspection"
+        raise RuntimeError(msg)
+    return caller
+
+
+def _add_type_record(name: str, typestr: str) -> None:
+    seen = OBSERVED_TYPES.setdefault(name, [])
+    if typestr not in seen:
+        seen.append(typestr)
+
+
+def record_type(name: str, value: Any, dict_names: bool = False) -> None:
+    """
+    Record the type of a variable.
+
+    :param name: The name of the variable
+    :param value: The value of the variable
+    """
+    _add_type_record(name, str(type(value)))
+
+    try:
+        for key, subvalue in value.items():
+            subname = f"{name}.{key}" if dict_names else f"{name}.[dict_value]"
+            _add_type_record(subname, str(type(subvalue)))
+        return
+    except AttributeError:
+        # (Wasn't a dict or anything resembling one.)
+        pass
+
+    # str is iterable, but not in the way we want!
+    if isinstance(value, str):
+        return
+
+    try:
+        for elem in value:
+            _add_type_record(f"{name}.[list_elem]", str(type(elem)))
+    except TypeError:
+        # (Wasn't a list or anything else iterable.)
+        pass
+
+
+def show_types() -> None:
+    """
+    Print all of the currently known variable types to stdout.
+    """
+    print(json.dumps(OBSERVED_TYPES, indent=2))
+
+
+def record_locals(show: bool = False, dict_names: bool = False) -> None:
+    caller = caller_frame()
+    name = caller.f_code.co_name
+    for key, value in caller.f_locals.items():
+        record_type(f"{name}.{key}", value, dict_names=dict_names)
+    if show:
+        show_types()
-- 
2.26.2

[PATCH 03/37] qapi-gen: Separate arg-parsing from generation

[John Snow]

This is a minor re-work of the entrypoint script. It isolates a
generate() method from the actual command-line mechanism.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi-gen.py | 87 ++++++++++++++++++++++++++++++++-------------
 1 file changed, 63 insertions(+), 24 deletions(-)

diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 4b03f7d53b..59becba3e1 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -1,9 +1,13 @@
 #!/usr/bin/env python3
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This script is the main entry point for generating C code from the QAPI schema.
+"""
 
 import argparse
 import re
@@ -11,21 +15,65 @@
 
 from qapi.commands import gen_commands
 from qapi.doc import gen_doc
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 
 
-def main(argv):
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match and match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+    gen_doc(schema, output_dir, prefix)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
     parser = argparse.ArgumentParser(
         description='Generate code from a QAPI schema')
     parser.add_argument('-b', '--builtins', action='store_true',
                         help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store', default='',
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
                         help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store', default='',
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
                         help="prefix for symbols")
     parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                         dest='unmask',
@@ -33,26 +81,17 @@ def main(argv):
     parser.add_argument('schema', action='store')
     args = parser.parse_args()
 
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-    if match.end() != len(args.prefix):
-        print("%s: 'funny character '%s' in argument of --prefix"
-              % (sys.argv[0], args.prefix[match.end()]),
-              file=sys.stderr)
-        sys.exit(1)
-
     try:
-        schema = QAPISchema(args.schema)
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
     except QAPIError as err:
-        print(err, file=sys.stderr)
-        exit(1)
-
-    gen_types(schema, args.output_dir, args.prefix, args.builtins)
-    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-    gen_commands(schema, args.output_dir, args.prefix)
-    gen_events(schema, args.output_dir, args.prefix)
-    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
-    gen_doc(schema, args.output_dir, args.prefix)
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
 
 
 if __name__ == '__main__':
-    main(sys.argv)
+    sys.exit(main())
-- 
2.26.2

[PATCH 04/37] qapi: move generator entrypoint into module

[John Snow]

As part of delinting and adding type hints to the QAPI generator, it's
helpful for the entrypoint to be part of the module, only leaving a very
tiny entrypoint shim outside of the module.

As a result, all of the include statements are reworked to be module-aware,
as explicit relative imports.

This is done primarily for the benefit of python tooling (pylint, mypy,
flake8, et al) which otherwise has trouble consistently resolving
"qapi.x" to mean "a sibling file in this folder."

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi-gen.py        | 94 +++-----------------------------------
 scripts/qapi/commands.py   |  4 +-
 scripts/qapi/doc.py        |  2 +-
 scripts/qapi/events.py     |  8 ++--
 scripts/qapi/expr.py       |  4 +-
 scripts/qapi/gen.py        |  4 +-
 scripts/qapi/introspect.py |  8 ++--
 scripts/qapi/parser.py     |  4 +-
 scripts/qapi/schema.py     |  8 ++--
 scripts/qapi/script.py     | 91 ++++++++++++++++++++++++++++++++++++
 scripts/qapi/types.py      |  6 +--
 scripts/qapi/visit.py      |  6 +--
 12 files changed, 124 insertions(+), 115 deletions(-)
 create mode 100644 scripts/qapi/script.py

diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 59becba3e1..e649f8dd44 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -1,97 +1,15 @@
 #!/usr/bin/env python3
-
-# This work is licensed under the terms of the GNU GPL, version 2 or later.
-# See the COPYING file in the top-level directory.
-
 """
-QAPI Generator
+QAPI code generation execution shim.
 
-This script is the main entry point for generating C code from the QAPI schema.
+This file exists primarily to facilitate the running of the QAPI code
+generator without needing to install the python module to the current
+execution environment.
 """
 
-import argparse
-import re
 import sys
 
-from qapi.commands import gen_commands
-from qapi.doc import gen_doc
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
-
-
-DEFAULT_OUTPUT_DIR = ''
-DEFAULT_PREFIX = ''
-
-
-def generate(schema_file: str,
-             output_dir: str,
-             prefix: str,
-             unmask: bool = False,
-             builtins: bool = False) -> None:
-    """
-    generate uses a given schema to produce C code in the target directory.
-
-    :param schema_file: The primary QAPI schema file.
-    :param output_dir: The output directory to store generated code.
-    :param prefix: Optional C-code prefix for symbol names.
-    :param unmask: Expose non-ABI names through introspection?
-    :param builtins: Generate code for built-in types?
-
-    :raise QAPIError: On failures.
-    """
-    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
-    if match and match.end() != len(prefix):
-        msg = "funny character '{:s}' in prefix '{:s}'".format(
-            prefix[match.end()], prefix)
-        raise QAPIError('', None, msg)
-
-    schema = QAPISchema(schema_file)
-    gen_types(schema, output_dir, prefix, builtins)
-    gen_visit(schema, output_dir, prefix, builtins)
-    gen_commands(schema, output_dir, prefix)
-    gen_events(schema, output_dir, prefix)
-    gen_introspect(schema, output_dir, prefix, unmask)
-    gen_doc(schema, output_dir, prefix)
-
-
-def main() -> int:
-    """
-    gapi-gen shell script entrypoint.
-    Expects arguments via sys.argv, see --help for details.
-
-    :return: int, 0 on success, 1 on failure.
-    """
-    parser = argparse.ArgumentParser(
-        description='Generate code from a QAPI schema')
-    parser.add_argument('-b', '--builtins', action='store_true',
-                        help="generate code for built-in types")
-    parser.add_argument('-o', '--output-dir', action='store',
-                        default=DEFAULT_OUTPUT_DIR,
-                        help="write output to directory OUTPUT_DIR")
-    parser.add_argument('-p', '--prefix', action='store',
-                        default=DEFAULT_PREFIX,
-                        help="prefix for symbols")
-    parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
-                        dest='unmask',
-                        help="expose non-ABI names in introspection")
-    parser.add_argument('schema', action='store')
-    args = parser.parse_args()
-
-    try:
-        generate(args.schema,
-                 output_dir=args.output_dir,
-                 prefix=args.prefix,
-                 unmask=args.unmask,
-                 builtins=args.builtins)
-    except QAPIError as err:
-        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
-        return 1
-    return 0
-
+from qapi import script
 
 if __name__ == '__main__':
-    sys.exit(main())
+    sys.exit(script.main())
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 3cf9e1110b..ce5926146a 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,8 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import *
+from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
index 92f584edcf..cbf7076ed9 100644
--- a/scripts/qapi/doc.py
+++ b/scripts/qapi/doc.py
@@ -5,7 +5,7 @@
 """This script produces the documentation of a qapi schema in texinfo format"""
 
 import re
-from qapi.gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc, QAPISchemaVisitor
 
 
 MSG_FMT = """
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index b544af5a1c..0467272438 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,10 +12,10 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember
-from qapi.types import gen_enum, gen_enum_lookup
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember
+from .types import gen_enum, gen_enum_lookup
 
 
 def build_event_send_proto(name, arg_type, boxed):
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 2942520399..03b31ecfc1 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -16,8 +16,8 @@
 
 import re
 from collections import OrderedDict
-from qapi.common import c_name
-from qapi.error import QAPISemError
+from .common import c_name
+from .error import QAPISemError
 
 
 # Names must be letters, numbers, -, and _.  They must start with letter,
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index bf5552a4e7..8df19a0df0 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -17,8 +17,8 @@
 import re
 from contextlib import contextmanager
 
-from qapi.common import *
-from qapi.schema import QAPISchemaVisitor
+from .common import *
+from .schema import QAPISchemaVisitor
 
 
 class QAPIGen:
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 23652be810..2a34cd1e8e 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,10 +10,10 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaMonolithicCVisitor
-from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                         QAPISchemaType)
+from .common import *
+from .gen import QAPISchemaMonolithicCVisitor
+from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
+                     QAPISchemaType)
 
 
 def _make_tree(obj, ifcond, features, extra=None):
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 165925ca72..327cf05736 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -18,8 +18,8 @@
 import re
 from collections import OrderedDict
 
-from qapi.error import QAPIParseError, QAPISemError
-from qapi.source import QAPISourceInfo
+from .error import QAPIParseError, QAPISemError
+from .source import QAPISourceInfo
 
 
 class QAPISchemaParser:
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 78309a00f0..a835ee6fde 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,10 +18,10 @@
 import re
 from collections import OrderedDict
 
-from qapi.common import c_name, pointer_suffix
-from qapi.error import QAPIError, QAPISemError
-from qapi.expr import check_exprs
-from qapi.parser import QAPISchemaParser
+from .common import c_name, pointer_suffix
+from .error import QAPIError, QAPISemError
+from .expr import check_exprs
+from .parser import QAPISchemaParser
 
 
 class QAPISchemaEntity:
diff --git a/scripts/qapi/script.py b/scripts/qapi/script.py
new file mode 100644
index 0000000000..3f8338ade8
--- /dev/null
+++ b/scripts/qapi/script.py
@@ -0,0 +1,91 @@
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+"""
+QAPI Generator
+
+This is the main entry point for generating C code from the QAPI schema.
+"""
+
+import argparse
+import re
+import sys
+
+from .commands import gen_commands
+from .doc import gen_doc
+from .error import QAPIError
+from .events import gen_events
+from .introspect import gen_introspect
+from .schema import QAPISchema
+from .types import gen_types
+from .visit import gen_visit
+
+
+DEFAULT_OUTPUT_DIR = ''
+DEFAULT_PREFIX = ''
+
+
+def generate(schema_file: str,
+             output_dir: str,
+             prefix: str,
+             unmask: bool = False,
+             builtins: bool = False) -> None:
+    """
+    generate uses a given schema to produce C code in the target directory.
+
+    :param schema_file: The primary QAPI schema file.
+    :param output_dir: The output directory to store generated code.
+    :param prefix: Optional C-code prefix for symbol names.
+    :param unmask: Expose non-ABI names through introspection?
+    :param builtins: Generate code for built-in types?
+
+    :raise QAPIError: On failures.
+    """
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+    if match and match.end() != len(prefix):
+        msg = "funny character '{:s}' in prefix '{:s}'".format(
+            prefix[match.end()], prefix)
+        raise QAPIError('', None, msg)
+
+    schema = QAPISchema(schema_file)
+    gen_types(schema, output_dir, prefix, builtins)
+    gen_visit(schema, output_dir, prefix, builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, unmask)
+    gen_doc(schema, output_dir, prefix)
+
+
+def main() -> int:
+    """
+    gapi-gen shell script entrypoint.
+    Expects arguments via sys.argv, see --help for details.
+
+    :return: int, 0 on success, 1 on failure.
+    """
+    parser = argparse.ArgumentParser(
+        description='Generate code from a QAPI schema')
+    parser.add_argument('-b', '--builtins', action='store_true',
+                        help="generate code for built-in types")
+    parser.add_argument('-o', '--output-dir', action='store',
+                        default=DEFAULT_OUTPUT_DIR,
+                        help="write output to directory OUTPUT_DIR")
+    parser.add_argument('-p', '--prefix', action='store',
+                        default=DEFAULT_PREFIX,
+                        help="prefix for symbols")
+    parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
+                        dest='unmask',
+                        help="expose non-ABI names in introspection")
+    parser.add_argument('schema', action='store')
+    args = parser.parse_args()
+
+    try:
+        generate(args.schema,
+                 output_dir=args.output_dir,
+                 prefix=args.prefix,
+                 unmask=args.unmask,
+                 builtins=args.builtins)
+    except QAPIError as err:
+        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+        return 1
+    return 0
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 3640f17cd6..ca9a5aacb3 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,9 +13,9 @@
 # See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
 
 # variants must be emitted before their container; track what has already
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index cdabc5fa28..7850f6e848 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,9 +13,9 @@
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaObjectType
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaObjectType
 
 
 def gen_visit_decl(name, scalar=False):
-- 
2.26.2

[PATCH 05/37] qapi: Remove wildcard includes

[John Snow]

Wildcard includes become hard to manage when refactoring and dealing
with circular dependencies with strictly typed mypy.

Remove them and include things explicitly by name instead.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/commands.py   |  6 +++++-
 scripts/qapi/events.py     |  7 ++++++-
 scripts/qapi/gen.py        | 12 +++++++++---
 scripts/qapi/introspect.py |  7 ++++++-
 scripts/qapi/types.py      |  8 +++++++-
 scripts/qapi/visit.py      | 10 +++++++++-
 6 files changed, 42 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index ce5926146a..e1df0e341f 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,7 +13,11 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    build_params,
+    c_name,
+    mcgen,
+)
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 0467272438..6b3afa14d7 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,7 +12,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    build_params,
+    c_enum_const,
+    c_name,
+    mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 8df19a0df0..11472ba043 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -11,13 +11,19 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-
+from contextlib import contextmanager
 import errno
 import os
 import re
-from contextlib import contextmanager
 
-from .common import *
+from .common import (
+    c_fname,
+    gen_endif,
+    gen_if,
+    guardend,
+    guardstart,
+    mcgen,
+)
 from .schema import QAPISchemaVisitor
 
 
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 2a34cd1e8e..b036fcf9ce 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,12 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+)
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
                      QAPISchemaType)
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index ca9a5aacb3..53b47f9e58 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,7 +13,13 @@
 # See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_enum_const,
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 7850f6e848..ea277e7704 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,7 +13,15 @@
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+    c_enum_const,
+    c_name,
+    gen_endif,
+    gen_if,
+    mcgen,
+    pop_indent,
+    push_indent,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
 
-- 
2.26.2

[PATCH 06/37] qapi: delint using flake8

[John Snow]

Petty style guide fixes and line length enforcement.  Not a big win, not
a big loss, but flake8 passes 100% on the qapi module, which gives us an
easy baseline to enforce hereafter.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/.flake8     |  2 ++
 scripts/qapi/commands.py |  3 ++-
 scripts/qapi/schema.py   |  8 +++++---
 scripts/qapi/visit.py    | 15 ++++++++++-----
 4 files changed, 19 insertions(+), 9 deletions(-)
 create mode 100644 scripts/qapi/.flake8

diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
new file mode 100644
index 0000000000..45d8146f3f
--- /dev/null
+++ b/scripts/qapi/.flake8
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Pylint handles this, but smarter.
\ No newline at end of file
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index e1df0e341f..2e4b4de0fa 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
 def gen_marshal_output(ret_type):
     return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
+                                          Error **errp)
 {
     Visitor *v;
 
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index a835ee6fde..b77e0e56b2 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -536,7 +536,7 @@ def set_defined_in(self, name):
             v.set_defined_in(name)
 
     def check(self, schema, seen):
-        if not self.tag_member: # flat union
+        if not self.tag_member:  # flat union
             self.tag_member = seen.get(c_name(self._tag_name))
             base = "'base'"
             # Pointing to the base type when not implicit would be
@@ -821,7 +821,7 @@ def __init__(self, fname):
         self._entity_dict = {}
         self._module_dict = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
-        self._make_module(None) # built-ins
+        self._make_module(None)  # built-ins
         self._make_module(fname)
         self._predefining = True
         self._def_predefineds()
@@ -965,7 +965,9 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
             # But it's not tight: the disjunction need not imply it.  We
             # may end up compiling useless wrapper types.
             # TODO kill simple unions or implement the disjunction
-            assert (ifcond or []) == typ._ifcond # pylint: disable=protected-access
+
+            # pylint: disable=protected-access
+            assert (ifcond or []) == typ._ifcond
         else:
             self._def_entity(QAPISchemaObjectType(
                 name, info, None, ifcond, None, None, members, None))
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index ea277e7704..31bf46e7f7 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -31,7 +31,8 @@ def gen_visit_decl(name, scalar=False):
     if not scalar:
         c_type += '*'
     return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error **errp);
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj,
+                           Error **errp);
 ''',
                  c_name=c_name(name), c_type=c_type)
 
@@ -125,7 +126,8 @@ def gen_visit_object_members(name, base, members, variants):
 def gen_visit_list(name, element_type):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+                           Error **errp)
 {
     bool ok = false;
     %(c_name)s *tail;
@@ -158,7 +160,8 @@ def gen_visit_list(name, element_type):
 def gen_visit_enum(name):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj,
+                           Error **errp)
 {
     int value = *obj;
     bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +175,8 @@ def gen_visit_enum(name):
 def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+                           Error **errp)
 {
     bool ok = false;
 
@@ -247,7 +251,8 @@ def gen_visit_alternate(name, variants):
 def gen_visit_object(name, base, members, variants):
     return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
+                           Error **errp)
 {
     bool ok = false;
 
-- 
2.26.2

[PATCH 07/37] qapi: add pylintrc

[John Snow]

Add a skeleton pylintrc file. Right now, it ignores quite a few things.
Files will be removed from the known-bad list throughout this and
following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as
that folder is shared by many things, it is going inside the module
folder now.

Due to some bugs in different versions of pylint (2.5.x), pylint does
not correctly recognize when it is being run from "inside" a module, and
must be run *outside* of the module.

Therefore, to run it, you must:

 > cd :/qemu/scripts
 > pylint qapi/ --rcfile=qapi/pylintrc

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)
 create mode 100644 scripts/qapi/pylintrc

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
new file mode 100644
index 0000000000..c2bbb8e8e1
--- /dev/null
+++ b/scripts/qapi/pylintrc
@@ -0,0 +1,74 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the blacklist. The
+# regex matches against base names, not paths.
+ignore-patterns=common.py,
+                doc.py,
+                error.py,
+                expr.py,
+                gen.py,
+                parser.py,
+                schema.py,
+                source.py,
+                types.py,
+                visit.py,
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=fixme,
+        missing-docstring,
+        too-many-arguments,
+        too-many-branches,
+        too-many-statements,
+        too-many-instance-attributes,
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+           j,
+           k,
+           ex,
+           Run,
+           _
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore imports when computing similarities.
+ignore-imports=yes
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]
-- 
2.26.2

[PATCH 08/37] qapi/common.py: Remove python compatibility workaround

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/common.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index ba35abea47..4fb265a8bf 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
     raw = code % kwds
     if indent_level:
         indent = genindent(indent_level)
-        # re.subn() lacks flags support before Python 2.7, use re.compile()
-        raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
-                      indent, raw)
-        raw = raw[0]
+        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
-- 
2.26.2

[PATCH 09/37] qapi/common.py: Add indent manager

[John Snow]

Code style tools really dislike the use of global keywords, because it
generally involves re-binding the name at runtime which can have strange
effects depending on when and how that global name is referenced in
other modules.

Make a little indent level manager instead.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++-------------
 scripts/qapi/visit.py  |  7 +++---
 2 files changed, 38 insertions(+), 19 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 4fb265a8bf..87d87b95e5 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -93,33 +93,53 @@ def c_name(name, protect=True):
 pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-    ret = ''
-    for _ in range(count):
-        ret += ' '
-    return ret
+class Indent:
+    """
+    Indent-level management.
 
+    :param initial: Initial value, default 0.
+    """
+    def __init__(self, initial: int = 0) -> None:
+        self._level = initial
 
-indent_level = 0
+    def __int__(self) -> int:
+        """Return the indent as an integer."""
+        return self._level
 
+    def __repr__(self) -> str:
+        return "{}({:d})".format(type(self).__name__, self._level)
 
-def push_indent(indent_amount=4):
-    global indent_level
-    indent_level += indent_amount
+    def __str__(self) -> str:
+        """Return the indent as a string."""
+        return ' ' * self._level
 
+    def __bool__(self) -> bool:
+        """True when there is a non-zero indent."""
+        return bool(self._level)
 
-def pop_indent(indent_amount=4):
-    global indent_level
-    indent_level -= indent_amount
+    def push(self, amount: int = 4) -> int:
+        """Push `amount` spaces onto the indent, default four."""
+        self._level += amount
+        return self._level
+
+    def pop(self, amount: int = 4) -> int:
+        """Pop `amount` spaces off of the indent, default four."""
+        if self._level < amount:
+            raise ArithmeticError(
+                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
+        self._level -= amount
+        return self._level
+
+
+INDENT = Indent(0)
 
 
 # Generate @code with @kwds interpolated.
 # Obey indent_level, and strip eatspace.
 def cgen(code, **kwds):
     raw = code % kwds
-    if indent_level:
-        indent = genindent(indent_level)
-        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+    if INDENT:
+        raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
     return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 31bf46e7f7..66ce3dc52e 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -19,8 +19,7 @@
     gen_endif,
     gen_if,
     mcgen,
-    pop_indent,
-    push_indent,
+    INDENT,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
@@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants):
     if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
 ''',
                          name=memb.name, c_name=c_name(memb.name))
-            push_indent()
+            INDENT.push()
         ret += mcgen('''
     if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
         return false;
@@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants):
                      c_type=memb.type.c_name(), name=memb.name,
                      c_name=c_name(memb.name))
         if memb.optional:
-            pop_indent()
+            INDENT.pop()
             ret += mcgen('''
     }
 ''')
-- 
2.26.2

[PATCH 10/37] qapi/common.py: delint with pylint

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/common.py | 16 +++++++---------
 scripts/qapi/schema.py | 14 +++++++-------
 2 files changed, 14 insertions(+), 16 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 87d87b95e5..c665e67495 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -14,6 +14,11 @@
 import re
 
 
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+c_name_trans = str.maketrans('.-', '__')
+
+
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-c_name_trans = str.maketrans('.-', '__')
-
-
 # Map @name to a valid C identifier.
 # If @protect, avoid returning certain ticklish identifiers (like
 # C keywords) by prepending 'q_'.
@@ -89,10 +91,6 @@ def c_name(name, protect=True):
     return name
 
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
 class Indent:
     """
     Indent-level management.
@@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey INDENT level, and strip EATSPACE.
 def cgen(code, **kwds):
     raw = code % kwds
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
-    return re.sub(re.escape(eatspace) + r' *', '', raw)
+    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
 def mcgen(code, **kwds):
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index b77e0e56b2..b4921b46c9 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,7 +18,7 @@
 import re
 from collections import OrderedDict
 
-from .common import c_name, pointer_suffix
+from .common import c_name, POINTER_SUFFIX
 from .error import QAPIError, QAPISemError
 from .expr import check_exprs
 from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ def is_implicit(self):
         return True
 
     def c_type(self):
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def json_type(self):
         return 'array'
@@ -430,7 +430,7 @@ def c_name(self):
 
     def c_type(self):
         assert not self.is_implicit()
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def c_unboxed_type(self):
         return c_name(self.name)
@@ -504,7 +504,7 @@ def connect_doc(self, doc=None):
             v.connect_doc(doc)
 
     def c_type(self):
-        return c_name(self.name) + pointer_suffix
+        return c_name(self.name) + POINTER_SUFFIX
 
     def json_type(self):
         return 'value'
@@ -896,7 +896,7 @@ def _def_builtin_type(self, name, json_type, c_type):
         self._make_array_type(name, None)
 
     def _def_predefineds(self):
-        for t in [('str',    'string',  'char' + pointer_suffix),
+        for t in [('str',    'string',  'char' + POINTER_SUFFIX),
                   ('number', 'number',  'double'),
                   ('int',    'int',     'int64_t'),
                   ('int8',   'int',     'int8_t'),
@@ -909,8 +909,8 @@ def _def_predefineds(self):
                   ('uint64', 'int',     'uint64_t'),
                   ('size',   'int',     'uint64_t'),
                   ('bool',   'boolean', 'bool'),
-                  ('any',    'value',   'QObject' + pointer_suffix),
-                  ('null',   'null',    'QNull' + pointer_suffix)]:
+                  ('any',    'value',   'QObject' + POINTER_SUFFIX),
+                  ('null',   'null',    'QNull' + POINTER_SUFFIX)]:
             self._def_builtin_type(*t)
         self.the_empty_object_type = QAPISchemaObjectType(
             'q_empty', None, None, None, None, None, [], None)
-- 
2.26.2

[PATCH 11/37] qapi/common.py: Replace one-letter 'c' variable

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index c665e67495..4c079755d3 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -30,14 +30,14 @@ def camel_to_upper(value):
     new_name = ''
     length = len(c_fun_str)
     for i in range(length):
-        c = c_fun_str[i]
-        # When c is upper and no '_' appears before, do more checks
-        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+        char = c_fun_str[i]
+        # When char is upper and no '_' appears before, do more checks
+        if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
             if i < length - 1 and c_fun_str[i + 1].islower():
                 new_name += '_'
             elif c_fun_str[i - 1].isdigit():
                 new_name += '_'
-        new_name += c
+        new_name += char
     return new_name.lstrip('_').upper()
 
 
-- 
2.26.2

[PATCH 12/37] qapi/common.py: check with pylint

[John Snow]

Remove qapi/common.py from the pylintrc ignore list.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/pylintrc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index c2bbb8e8e1..c800055589 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -2,8 +2,7 @@
 
 # Add files or directories matching the regex patterns to the blacklist. The
 # regex matches against base names, not paths.
-ignore-patterns=common.py,
-                doc.py,
+ignore-patterns=doc.py,
                 error.py,
                 expr.py,
                 gen.py,
-- 
2.26.2

[PATCH 13/37] qapi/common.py: add notational type hints

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/common.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 4c079755d3..af01348b35 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -12,6 +12,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Optional, Union, Sequence
 
 
 EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
     return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+                 const_name: str,
+                 prefix: Optional[str] = None) -> str:
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
 # into substrings of a generated C function name.
 # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
 # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
 
 # Generate @code with @kwds interpolated.
 # Obey INDENT level, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: Union[str, int]) -> str:
     raw = code % kwds
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
     return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: Union[str, int]) -> str:
     if code[0] == '\n':
         code = code[1:]
     return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
     return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
     return mcgen('''
 #ifndef %(name)s
 #define %(name)s
@@ -160,7 +163,7 @@ def guardstart(name):
                  name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
     return mcgen('''
 
 #endif /* %(name)s */
@@ -168,7 +171,7 @@ def guardend(name):
                  name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in ifcond:
         ret += mcgen('''
@@ -177,7 +180,7 @@ def gen_if(ifcond):
     return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
     ret = ''
     for ifc in reversed(ifcond):
         ret += mcgen('''
@@ -186,7 +189,9 @@ def gen_endif(ifcond):
     return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
     ret = ''
     sep = ''
     if boxed:
-- 
2.26.2

[PATCH 14/37] qapi/common.py: Move comments into docstrings

[John Snow]

As docstrings, they'll show up in documentation and IDE help.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------
 1 file changed, 36 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index af01348b35..38d380f0a9 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -20,10 +20,18 @@
 c_name_trans = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
 def camel_to_upper(value: str) -> str:
+    """
+    Converts CamelCase to CAMEL_CASE.
+
+    Examples:
+      ENUMName -> ENUM_NAME
+      EnumName1 -> ENUM_NAME1
+      ENUM_NAME -> ENUM_NAME
+      ENUM_NAME1 -> ENUM_NAME1
+      ENUM_Name2 -> ENUM_NAME2
+      ENUM24_Name -> ENUM24_NAME
+    """
     c_fun_str = c_name(value, False)
     if value.isupper():
         return c_fun_str
@@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str:
 def c_enum_const(type_name: str,
                  const_name: str,
                  prefix: Optional[str] = None) -> str:
+    """
+    Generate a C enumeration constant name.
+
+    :param type_name: The name of the enumeration.
+    :param const_name: The name of this constant.
+    :param prefix: Optional, prefix that overrides the type_name.
+    """
     if prefix is not None:
         type_name = prefix
     return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 def c_name(name: str, protect: bool = True) -> str:
+    """
+    Map `name` to a valid C identifier.
+
+    Used for converting 'name' from a 'name':'type' qapi definition
+    into a generated struct member, as well as converting type names
+    into substrings of a generated C function name.
+
+    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+    :param name: The name to map.
+    :param protect: If true, avoid returning certain ticklish identifiers
+                    (like C keywords) by prepending 'q_'.
+    """
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
                      'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int:
 INDENT = Indent(0)
 
 
-# Generate @code with @kwds interpolated.
-# Obey INDENT level, and strip EATSPACE.
 def cgen(code: str, **kwds: Union[str, int]) -> str:
+    """
+    Generate `code` with `kwds` interpolated.
+    Obey `INDENT`, and strip `EATSPACE`.
+    """
     raw = code % kwds
     if INDENT:
         raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
-- 
2.26.2

[PATCH 15/37] qapi/common.py: split build_params into new file

[John Snow]

Including it in common.py creates a circular import dependency, because
schema relies on common.py. To type build_params properly, it needs to
be moved outside of the chain.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/commands.py |  2 +-
 scripts/qapi/common.py   | 23 -----------------------
 scripts/qapi/events.py   |  2 +-
 scripts/qapi/params.py   | 40 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 42 insertions(+), 25 deletions(-)
 create mode 100644 scripts/qapi/params.py

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 2e4b4de0fa..0c0fe854fe 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -14,10 +14,10 @@
 """
 
 from .common import (
-    build_params,
     c_name,
     mcgen,
 )
+from .params import build_params
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 38d380f0a9..0b1af694e6 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -209,26 +209,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
 #endif /* %(cond)s */
 ''', cond=ifc)
     return ret
-
-
-def build_params(arg_type,
-                 boxed: bool,
-                 extra: Optional[str] = None) -> str:
-    ret = ''
-    sep = ''
-    if boxed:
-        assert arg_type
-        ret += '%s arg' % arg_type.c_param_type()
-        sep = ', '
-    elif arg_type:
-        assert not arg_type.variants
-        for memb in arg_type.members:
-            ret += sep
-            sep = ', '
-            if memb.optional:
-                ret += 'bool has_%s, ' % c_name(memb.name)
-            ret += '%s %s' % (memb.type.c_param_type(),
-                              c_name(memb.name))
-    if extra:
-        ret += sep + extra
-    return ret if ret else 'void'
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 6b3afa14d7..75eda72534 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -13,11 +13,11 @@
 """
 
 from .common import (
-    build_params,
     c_enum_const,
     c_name,
     mcgen,
 )
+from .params import build_params
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
diff --git a/scripts/qapi/params.py b/scripts/qapi/params.py
new file mode 100644
index 0000000000..4d4b02f60d
--- /dev/null
+++ b/scripts/qapi/params.py
@@ -0,0 +1,40 @@
+#
+# QAPI helper library
+#
+# Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2018 Red Hat Inc.
+#
+# Authors:
+#  Anthony Liguori <aliguori@us.ibm.com>
+#  Markus Armbruster <armbru@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from typing import Optional
+
+from .common import c_name
+from .schema import QAPISchemaObjectType
+
+
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+                 boxed: bool,
+                 extra: Optional[str] = None) -> str:
+    ret = ''
+    sep = ''
+    if boxed:
+        assert arg_type
+        ret += '%s arg' % arg_type.c_param_type()
+        sep = ', '
+    elif arg_type:
+        assert not arg_type.variants
+        for memb in arg_type.members:
+            ret += sep
+            sep = ', '
+            if memb.optional:
+                ret += 'bool has_%s, ' % c_name(memb.name)
+            ret += '%s %s' % (memb.type.c_param_type(),
+                              c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret if ret else 'void'
-- 
2.26.2

[PATCH 16/37] qapi: establish mypy type-checking baseline

[John Snow]

Fix two very minor issues, and then establish a mypy type-checking
baseline.

Like pylint, this should be run from the folder above:

 > mypy --config-file=qapi/mypy.ini qapi/

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/doc.py    |  3 +-
 scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/schema.py |  3 +-
 3 files changed, 69 insertions(+), 2 deletions(-)
 create mode 100644 scripts/qapi/mypy.ini

diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
index cbf7076ed9..70f7cdfaa6 100644
--- a/scripts/qapi/doc.py
+++ b/scripts/qapi/doc.py
@@ -5,7 +5,8 @@
 """This script produces the documentation of a qapi schema in texinfo format"""
 
 import re
-from .gen import QAPIGenDoc, QAPISchemaVisitor
+from .gen import QAPIGenDoc
+from .schema import QAPISchemaVisitor
 
 
 MSG_FMT = """
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
new file mode 100644
index 0000000000..a0f2365a53
--- /dev/null
+++ b/scripts/qapi/mypy.ini
@@ -0,0 +1,65 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.doc]
+disallow_subclassing_any = False
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index b4921b46c9..bb0cd717f1 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -17,6 +17,7 @@
 import os
 import re
 from collections import OrderedDict
+from typing import Optional
 
 from .common import c_name, POINTER_SUFFIX
 from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@
 
 
 class QAPISchemaEntity:
-    meta = None
+    meta: Optional[str] = None
 
     def __init__(self, name, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
-- 
2.26.2

[PATCH 17/37] qapi/events.py: add notational type hints

[John Snow]

This is enough to enable type-checking on events.py, so enable that in
this patch, too.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/events.py | 46 ++++++++++++++++++++++++++++++++----------
 scripts/qapi/mypy.ini  |  5 -----
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 75eda72534..449a700ffe 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,6 +12,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List
+
 from .common import (
     c_enum_const,
     c_name,
@@ -19,17 +21,27 @@
 )
 from .params import build_params
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumMember
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+)
+from .source import QAPISourceInfo
 from .types import gen_enum, gen_enum_lookup
 
 
-def build_event_send_proto(name, arg_type, boxed):
+def build_event_send_proto(name: str,
+                           arg_type: QAPISchemaObjectType,
+                           boxed: bool) -> str:
     return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
         'c_name': c_name(name.lower()),
         'param': build_params(arg_type, boxed)}
 
 
-def gen_event_send_decl(name, arg_type, boxed):
+def gen_event_send_decl(name: str,
+                        arg_type: QAPISchemaObjectType,
+                        boxed: bool) -> str:
     return mcgen('''
 
 %(proto)s;
@@ -38,7 +50,7 @@ def gen_event_send_decl(name, arg_type, boxed):
 
 
 # Declare and initialize an object 'qapi' using parameters from build_params()
-def gen_param_var(typ):
+def gen_param_var(typ: QAPISchemaObjectType) -> str:
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
@@ -66,7 +78,11 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
+def gen_event_send(name: str,
+                   arg_type: QAPISchemaObjectType,
+                   boxed: bool,
+                   event_enum_name: str,
+                   event_emit: str) -> str:
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -142,15 +158,15 @@ def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
 
 class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-events',
             ' * Schema-defined QAPI/QMP events', None, __doc__)
         self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-        self._event_enum_members = []
+        self._event_enum_members: List[QAPISchemaEnumMember] = []
         self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         events = self._module_basename('qapi-events', name)
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
@@ -173,7 +189,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         self._add_system_module('emit', ' * QAPI Events emission')
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
@@ -194,7 +210,13 @@ def visit_end(self):
                              event_emit=self._event_emit_name,
                              event_enum=self._event_enum_name))
 
-    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+    def visit_event(self,
+                    name: str,
+                    info: QAPISourceInfo,
+                    ifcond: List[str],
+                    features: List[QAPISchemaFeature],
+                    arg_type: QAPISchemaObjectType,
+                    boxed: bool) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_event_send_decl(name, arg_type, boxed))
             self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -205,7 +227,9 @@ def visit_event(self, name, info, ifcond, features, arg_type, boxed):
         self._event_enum_members.append(QAPISchemaEnumMember(name, None))
 
 
-def gen_events(schema, output_dir, prefix):
+def gen_events(schema: QAPISchema,
+               output_dir: str,
+               prefix: str) -> None:
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index a0f2365a53..b668776d94 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -19,11 +19,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.events]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.expr]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
-- 
2.26.2

[PATCH 18/37] qapi/events.py: Move comments into docstrings

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/events.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 449a700ffe..0b49cf5fea 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -49,8 +49,10 @@ def gen_event_send_decl(name: str,
                  proto=build_event_send_proto(name, arg_type, boxed))
 
 
-# Declare and initialize an object 'qapi' using parameters from build_params()
 def gen_param_var(typ: QAPISchemaObjectType) -> str:
+    """
+    Declare and initialize a qapi object, using parameters from `build_params`.
+    """
     assert not typ.variants
     ret = mcgen('''
     %(c_name)s param = {
-- 
2.26.2

[PATCH 19/37] qapi/commands.py: Don't re-bind to variable of different type

[John Snow]

Mypy isn't a fan of rebinding a variable with a new data type.
It's easy enough to avoid.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/commands.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 0c0fe854fe..b215e58357 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -194,14 +194,12 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     if not options:
         options = ['QCO_NO_OPTIONS']
 
-    options = " | ".join(options)
-
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
                          qmp_marshal_%(c_name)s, %(opts)s);
 ''',
                 name=name, c_name=c_name(name),
-                opts=options)
+                opts=" | ".join(options))
     return ret
 
 
-- 
2.26.2

[PATCH 20/37] qapi/commands.py: add notational type hints

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/commands.py | 72 ++++++++++++++++++++++++++++++----------
 1 file changed, 54 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index b215e58357..3c950da10a 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,15 +13,32 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import Dict, Optional, List, Set
+
 from .common import (
     c_name,
     mcgen,
 )
+from .gen import (
+    QAPIGenC,
+    QAPIGenCCode,
+    QAPISchemaModularCVisitor,
+    ifcontext,
+)
 from .params import build_params
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .schema import (
+    QAPISchema,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaType,
+)
+from .source import QAPISourceInfo
 
 
-def gen_command_decl(name, arg_type, boxed, ret_type):
+def gen_command_decl(name: str,
+                     arg_type: Optional[QAPISchemaObjectType],
+                     boxed: bool,
+                     ret_type: Optional[QAPISchemaType]) -> str:
     return mcgen('''
 %(c_type)s qmp_%(c_name)s(%(params)s);
 ''',
@@ -30,7 +47,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type):
                  params=build_params(arg_type, boxed, 'Error **errp'))
 
 
-def gen_call(name, arg_type, boxed, ret_type):
+def gen_call(name: str,
+             arg_type: Optional[QAPISchemaObjectType],
+             boxed: bool,
+             ret_type: Optional[QAPISchemaType]) -> str:
     ret = ''
 
     argstr = ''
@@ -66,7 +86,7 @@ def gen_call(name, arg_type, boxed, ret_type):
     return ret
 
 
-def gen_marshal_output(ret_type):
+def gen_marshal_output(ret_type: QAPISchemaType) -> str:
     return mcgen('''
 
 static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
@@ -87,19 +107,22 @@ def gen_marshal_output(ret_type):
                  c_type=ret_type.c_type(), c_name=ret_type.c_name())
 
 
-def build_marshal_proto(name):
+def build_marshal_proto(name: str) -> str:
     return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)'
             % c_name(name))
 
 
-def gen_marshal_decl(name):
+def gen_marshal_decl(name: str) -> str:
     return mcgen('''
 %(proto)s;
 ''',
                  proto=build_marshal_proto(name))
 
 
-def gen_marshal(name, arg_type, boxed, ret_type):
+def gen_marshal(name: str,
+                arg_type: Optional[QAPISchemaObjectType],
+                boxed: bool,
+                ret_type: Optional[QAPISchemaType]) -> str:
     have_args = boxed or (arg_type and not arg_type.is_empty())
 
     ret = mcgen('''
@@ -181,7 +204,10 @@ def gen_marshal(name, arg_type, boxed, ret_type):
     return ret
 
 
-def gen_register_command(name, success_response, allow_oob, allow_preconfig):
+def gen_register_command(name: str,
+                         success_response: bool,
+                         allow_oob: bool,
+                         allow_preconfig: bool) -> str:
     options = []
 
     if not success_response:
@@ -203,7 +229,7 @@ def gen_register_command(name, success_response, allow_oob, allow_preconfig):
     return ret
 
 
-def gen_registry(registry, prefix):
+def gen_registry(registry: str, prefix: str) -> str:
     ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -220,15 +246,14 @@ def gen_registry(registry, prefix):
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
-
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-commands',
             ' * Schema-defined QAPI/QMP commands', None, __doc__)
         self._regy = QAPIGenCCode(None)
-        self._visited_ret_types = {}
+        self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {}
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         self._visited_ret_types[self._genc] = set()
         commands = self._module_basename('qapi-commands', name)
         types = self._module_basename('qapi-types', name)
@@ -252,7 +277,7 @@ def _begin_user_module(self, name):
 ''',
                              types=types))
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         self._add_system_module('init', ' * QAPI Commands initialization')
         self._genh.add(mcgen('''
 #include "qapi/qmp/dispatch.h"
@@ -268,9 +293,18 @@ def visit_end(self):
                                       prefix=self._prefix))
         self._genc.add(gen_registry(self._regy.get_content(), self._prefix))
 
-    def visit_command(self, name, info, ifcond, features,
-                      arg_type, ret_type, gen, success_response, boxed,
-                      allow_oob, allow_preconfig):
+    def visit_command(self,
+                      name: str,
+                      info: QAPISourceInfo,
+                      ifcond: List[str],
+                      features: List[QAPISchemaFeature],
+                      arg_type: Optional[QAPISchemaObjectType],
+                      ret_type: Optional[QAPISchemaType],
+                      gen: bool,
+                      success_response: bool,
+                      boxed: bool,
+                      allow_oob: bool,
+                      allow_preconfig: bool) -> None:
         if not gen:
             return
         # FIXME: If T is a user-defined type, the user is responsible
@@ -291,7 +325,9 @@ def visit_command(self, name, info, ifcond, features,
                                                 allow_oob, allow_preconfig))
 
 
-def gen_commands(schema, output_dir, prefix):
+def gen_commands(schema: QAPISchema,
+                 output_dir: str,
+                 prefix: str) -> None:
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir)
-- 
2.26.2

[PATCH 21/37] qapi/commands.py: enable checking with mypy

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index b668776d94..9da1dccef4 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -4,11 +4,6 @@ strict_optional = False
 disallow_untyped_calls = False
 python_version = 3.6
 
-[mypy-qapi.commands]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.doc]
 disallow_subclassing_any = False
 disallow_untyped_defs = False
-- 
2.26.2

[PATCH 22/37] qapi/source.py: add notational type hints

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/mypy.ini  |  5 -----
 scripts/qapi/source.py | 31 ++++++++++++++++++-------------
 2 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 9da1dccef4..43c8bd1973 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -39,11 +39,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.types]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index e97b9a8e15..1cc6a5b82d 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -11,37 +11,42 @@
 
 import copy
 import sys
+from typing import List, Optional, TypeVar
 
 
 class QAPISchemaPragma:
-    def __init__(self):
+    def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
         # Whitelist of commands allowed to return a non-dictionary
-        self.returns_whitelist = []
+        self.returns_whitelist: List[str] = []
         # Whitelist of entities allowed to violate case conventions
-        self.name_case_whitelist = []
+        self.name_case_whitelist: List[str] = []
 
 
 class QAPISourceInfo:
-    def __init__(self, fname, line, parent):
+    T = TypeVar('T', bound='QAPISourceInfo')
+
+    def __init__(self: T, fname: str, line: int, parent: Optional[T]):
         self.fname = fname
         self.line = line
         self.parent = parent
-        self.pragma = parent.pragma if parent else QAPISchemaPragma()
-        self.defn_meta = None
-        self.defn_name = None
+        self.pragma: QAPISchemaPragma = (
+            parent.pragma if parent else QAPISchemaPragma()
+        )
+        self.defn_meta: Optional[str] = None
+        self.defn_name: Optional[str] = None
 
-    def set_defn(self, meta, name):
+    def set_defn(self, meta: str, name: str) -> None:
         self.defn_meta = meta
         self.defn_name = name
 
-    def next_line(self):
+    def next_line(self: T) -> T:
         info = copy.copy(self)
         info.line += 1
         return info
 
-    def loc(self):
+    def loc(self) -> str:
         if self.fname is None:
             return sys.argv[0]
         ret = self.fname
@@ -49,13 +54,13 @@ def loc(self):
             ret += ':%d' % self.line
         return ret
 
-    def in_defn(self):
+    def in_defn(self) -> str:
         if self.defn_name:
             return "%s: In %s '%s':\n" % (self.fname,
                                           self.defn_meta, self.defn_name)
         return ''
 
-    def include_path(self):
+    def include_path(self) -> str:
         ret = ''
         parent = self.parent
         while parent:
@@ -63,5 +68,5 @@ def include_path(self):
             parent = parent.parent
         return ret
 
-    def __str__(self):
+    def __str__(self) -> str:
         return self.include_path() + self.in_defn() + self.loc()
-- 
2.26.2

[PATCH 23/37] qapi/source.py: delint with pylint

[John Snow]

Shush an error and leave a hint for future cleanups when we're allowed
to use Python 3.7+.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/pylintrc  | 1 -
 scripts/qapi/source.py | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index c800055589..c50a6656b7 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -8,7 +8,6 @@ ignore-patterns=doc.py,
                 gen.py,
                 parser.py,
                 schema.py,
-                source.py,
                 types.py,
                 visit.py,
 
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index 1cc6a5b82d..ba991d798f 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -15,6 +15,9 @@
 
 
 class QAPISchemaPragma:
+    # Replace with @dataclass in Python 3.7+
+    # pylint: disable=too-few-public-methods
+
     def __init__(self) -> None:
         # Are documentation comments required?
         self.doc_required = False
-- 
2.26.2

[PATCH 24/37] qapi/gen.py: Fix edge-case of _is_user_module

[John Snow]

The edge case is that if the name is '', this expression returns a
string instead of a bool, which violates our declared type.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/gen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 11472ba043..fc46813d1a 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -221,7 +221,7 @@ def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
 
     @staticmethod
     def _is_user_module(name):
-        return name and not name.startswith('./')
+        return name is not None and not name.startswith('./')
 
     @staticmethod
     def _is_builtin_module(name):
-- 
2.26.2

[PATCH 25/37] qapi/gen.py: add notational type hints

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/gen.py | 105 +++++++++++++++++++++++---------------------
 1 file changed, 56 insertions(+), 49 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index fc46813d1a..c87c84e7c9 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -15,6 +15,7 @@
 import errno
 import os
 import re
+from typing import Dict, Generator, List, Optional, Tuple
 
 from .common import (
     c_fname,
@@ -24,32 +25,34 @@
     guardstart,
     mcgen,
 )
-from .schema import QAPISchemaVisitor
+from .schema import (
+    QAPISchemaVisitor,
+)
+from .source import QAPISourceInfo
 
 
 class QAPIGen:
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):
         self.fname = fname
         self._preamble = ''
         self._body = ''
 
-    def preamble_add(self, text):
+    def preamble_add(self, text: str) -> None:
         self._preamble += text
 
-    def add(self, text):
+    def add(self, text: str) -> None:
         self._body += text
 
-    def get_content(self):
+    def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
-    def _top(self):
+    def _top(self) -> str:
         return ''
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return ''
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         # Include paths starting with ../ are used to reuse modules of the main
         # schema in specialised schemas. Don't overwrite the files that are
         # already generated for the main schema.
@@ -74,7 +77,7 @@ def write(self, output_dir):
         f.close()
 
 
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
     if before == after:
         return after   # suppress empty #if ... #endif
 
@@ -91,40 +94,38 @@ def _wrap_ifcond(ifcond, before, after):
 
 
 class QAPIGenCCode(QAPIGen):
-
-    def __init__(self, fname):
+    def __init__(self, fname: Optional[str]):
         super().__init__(fname)
-        self._start_if = None
+        self._start_if: Optional[Tuple[List[str], str, str]] = None
 
-    def start_if(self, ifcond):
+    def start_if(self, ifcond: List[str]) -> None:
         assert self._start_if is None
         self._start_if = (ifcond, self._body, self._preamble)
 
-    def end_if(self):
+    def end_if(self) -> None:
         assert self._start_if
         self._wrap_ifcond()
         self._start_if = None
 
-    def _wrap_ifcond(self):
+    def _wrap_ifcond(self) -> None:
         self._body = _wrap_ifcond(self._start_if[0],
                                   self._start_if[1], self._body)
         self._preamble = _wrap_ifcond(self._start_if[0],
                                       self._start_if[2], self._preamble)
 
-    def get_content(self):
+    def get_content(self) -> str:
         assert self._start_if is None
         return super().get_content()
 
 
 class QAPIGenC(QAPIGenCCode):
-
-    def __init__(self, fname, blurb, pydoc):
+    def __init__(self, fname: str, blurb: str, pydoc: str):
         super().__init__(fname)
         self._blurb = blurb
         self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
                                                   re.MULTILINE))
 
-    def _top(self):
+    def _top(self) -> str:
         return mcgen('''
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
@@ -140,7 +141,7 @@ def _top(self):
 ''',
                      blurb=self._blurb, copyright=self._copyright)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return mcgen('''
 
 /* Dummy declaration to prevent empty .o file */
@@ -150,16 +151,16 @@ def _bottom(self):
 
 
 class QAPIGenH(QAPIGenC):
-
-    def _top(self):
+    def _top(self) -> str:
         return super()._top() + guardstart(self.fname)
 
-    def _bottom(self):
+    def _bottom(self) -> str:
         return guardend(self.fname)
 
 
 @contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str],
+              *args: QAPIGenCCode) -> Generator[None, None, None]:
     """A 'with' statement context manager to wrap with start_if()/end_if()
 
     *args: any number of QAPIGenCCode
@@ -185,15 +186,17 @@ def ifcontext(ifcond, *args):
 
 
 class QAPIGenDoc(QAPIGen):
-
-    def _top(self):
+    def _top(self) -> str:
         return (super()._top()
                 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
 
 
 class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 blurb: str,
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._genc = QAPIGenC(self._prefix + self._what + '.c',
@@ -201,38 +204,42 @@ def __init__(self, prefix, what, blurb, pydoc):
         self._genh = QAPIGenH(self._prefix + self._what + '.h',
                               blurb, pydoc)
 
-    def write(self, output_dir):
+    def write(self, output_dir: str) -> None:
         self._genc.write(output_dir)
         self._genh.write(output_dir)
 
 
 class QAPISchemaModularCVisitor(QAPISchemaVisitor):
-
-    def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc):
+    def __init__(self,
+                 prefix: str,
+                 what: str,
+                 user_blurb: str,
+                 builtin_blurb: Optional[str],
+                 pydoc: str):
         self._prefix = prefix
         self._what = what
         self._user_blurb = user_blurb
         self._builtin_blurb = builtin_blurb
         self._pydoc = pydoc
-        self._genc = None
-        self._genh = None
-        self._module = {}
-        self._main_module = None
+        self._genc: Optional[QAPIGenC] = None
+        self._genh: Optional[QAPIGenH] = None
+        self._module: Dict[Optional[str], Tuple[QAPIGenC, QAPIGenH]] = {}
+        self._main_module: Optional[str] = None
 
     @staticmethod
-    def _is_user_module(name):
+    def _is_user_module(name: Optional[str]) -> bool:
         return name is not None and not name.startswith('./')
 
     @staticmethod
-    def _is_builtin_module(name):
+    def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what, name):
+    def _module_dirname(self, what: str, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
 
-    def _module_basename(self, what, name):
+    def _module_basename(self, what: str, name: Optional[str]) -> str:
         ret = '' if self._is_builtin_module(name) else self._prefix
         if self._is_user_module(name):
             basename = os.path.basename(name)
@@ -244,27 +251,27 @@ def _module_basename(self, what, name):
             ret += re.sub(r'-', '-' + name + '-', what)
         return ret
 
-    def _module_filename(self, what, name):
+    def _module_filename(self, what: str, name: Optional[str]) -> str:
         return os.path.join(self._module_dirname(what, name),
                             self._module_basename(what, name))
 
-    def _add_module(self, name, blurb):
+    def _add_module(self, name: Optional[str], blurb: str) -> None:
         basename = self._module_filename(self._what, name)
         genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
         genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
         self._module[name] = (genc, genh)
         self._genc, self._genh = self._module[name]
 
-    def _add_user_module(self, name, blurb):
+    def _add_user_module(self, name: str, blurb: str) -> None:
         assert self._is_user_module(name)
         if self._main_module is None:
             self._main_module = name
         self._add_module(name, blurb)
 
-    def _add_system_module(self, name, blurb):
+    def _add_system_module(self, name: Optional[str], blurb: str) -> None:
         self._add_module(name and './' + name, blurb)
 
-    def write(self, output_dir, opt_builtins=False):
+    def write(self, output_dir: str, opt_builtins: bool = False) -> None:
         for name in self._module:
             if self._is_builtin_module(name) and not opt_builtins:
                 continue
@@ -272,13 +279,13 @@ def write(self, output_dir, opt_builtins=False):
             genc.write(output_dir)
             genh.write(output_dir)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:
         pass
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         pass
 
-    def visit_module(self, name):
+    def visit_module(self, name: Optional[str]) -> None:
         if name is None:
             if self._builtin_blurb:
                 self._add_system_module(None, self._builtin_blurb)
@@ -292,7 +299,7 @@ def visit_module(self, name):
             self._add_user_module(name, self._user_blurb)
             self._begin_user_module(name)
 
-    def visit_include(self, name, info):
+    def visit_include(self, name: str, info: QAPISourceInfo) -> None:
         relname = os.path.relpath(self._module_filename(self._what, name),
                                   os.path.dirname(self._genh.fname))
         self._genh.preamble_add(mcgen('''
-- 
2.26.2

[PATCH 26/37] qapi/gen.py: Enable checking with mypy

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 43c8bd1973..dbfeda748c 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -19,11 +19,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.gen]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.introspect]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
-- 
2.26.2

[PATCH 27/37] qapi/gen.py: Remove unused parameter

[John Snow]

module_basename doesn't use the 'what' argument, so remove it.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/gen.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index c87c84e7c9..e873901672 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -234,7 +234,7 @@ def _is_user_module(name: Optional[str]) -> bool:
     def _is_builtin_module(name: Optional[str]) -> bool:
         return not name
 
-    def _module_dirname(self, what: str, name: Optional[str]) -> str:
+    def _module_dirname(self, name: Optional[str]) -> str:
         if self._is_user_module(name):
             return os.path.dirname(name)
         return ''
@@ -252,7 +252,7 @@ def _module_basename(self, what: str, name: Optional[str]) -> str:
         return ret
 
     def _module_filename(self, what: str, name: Optional[str]) -> str:
-        return os.path.join(self._module_dirname(what, name),
+        return os.path.join(self._module_dirname(name),
                             self._module_basename(what, name))
 
     def _add_module(self, name: Optional[str], blurb: str) -> None:
-- 
2.26.2

[PATCH 28/37] qapi/gen.py: update write() to be more idiomatic

[John Snow]

Make the file handling here just a tiny bit more idiomatic.
(I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception,
use fdopen() to wrap the file descriptor in a File-like object,
and use a context manager for managing the file pointer.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/gen.py | 24 ++++++++++--------------
 1 file changed, 10 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index e873901672..bd67f6b069 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -12,7 +12,6 @@
 # See the COPYING file in the top-level directory.
 
 from contextlib import contextmanager
-import errno
 import os
 import re
 from typing import Dict, Generator, List, Optional, Tuple
@@ -60,21 +59,18 @@ def write(self, output_dir: str) -> None:
             return
         pathname = os.path.join(output_dir, self.fname)
         odir = os.path.dirname(pathname)
+
         if odir:
-            try:
-                os.makedirs(odir)
-            except os.error as e:
-                if e.errno != errno.EEXIST:
-                    raise
+            os.makedirs(odir, exist_ok=True)
+
         fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-        f = open(fd, 'r+', encoding='utf-8')
-        text = self.get_content()
-        oldtext = f.read(len(text) + 1)
-        if text != oldtext:
-            f.seek(0)
-            f.truncate(0)
-            f.write(text)
-        f.close()
+        with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+            text = self.get_content()
+            oldtext = fp.read(len(text) + 1)
+            if text != oldtext:
+                fp.seek(0)
+                fp.truncate(0)
+                fp.write(text)
 
 
 def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
-- 
2.26.2

[PATCH 29/37] qapi/gen.py: delint with pylint

[John Snow]

'fp' and 'fd' are self-evident in context, add them to the list of OK
names.

_top and _bottom also need to stay class methods because some users
override the method and need to use `self`. Tell pylint to shush.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/gen.py   | 2 ++
 scripts/qapi/pylintrc | 5 +++--
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index bd67f6b069..0a72aecdd0 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -46,9 +46,11 @@ def get_content(self) -> str:
         return self._top() + self._preamble + self._body + self._bottom()
 
     def _top(self) -> str:
+        # pylint: disable=no-self-use
         return ''
 
     def _bottom(self) -> str:
+        # pylint: disable=no-self-use
         return ''
 
     def write(self, output_dir: str) -> None:
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index c50a6656b7..3aebb0d228 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -5,7 +5,6 @@
 ignore-patterns=doc.py,
                 error.py,
                 expr.py,
-                gen.py,
                 parser.py,
                 schema.py,
                 types.py,
@@ -46,7 +45,9 @@ good-names=i,
            k,
            ex,
            Run,
-           _
+           _,
+           fp,  # fp = open(...)
+           fd,  # fd = os.open(...)
 
 [VARIABLES]
 
-- 
2.26.2

[PATCH 30/37] qapi/introspect.py: Add a typed 'extra' structure

[John Snow]

Typing arbitrarily shaped dicts with mypy is difficult prior to Python
3.8; using explicit structures is nicer.

Since we always define an Extra type now, the return type of _make_tree
simplifies and always returns the tuple.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/introspect.py | 31 +++++++++++++++++++------------
 1 file changed, 19 insertions(+), 12 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index b036fcf9ce..41ca8afc67 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,6 +10,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import (NamedTuple, Optional, Sequence)
+
 from .common import (
     c_name,
     gen_endif,
@@ -21,16 +23,21 @@
                      QAPISchemaType)
 
 
-def _make_tree(obj, ifcond, features, extra=None):
-    if extra is None:
-        extra = {}
-    if ifcond:
-        extra['if'] = ifcond
+class Extra(NamedTuple):
+    """
+    Extra contains data that isn't intended for output by introspection.
+    """
+    comment: Optional[str] = None
+    ifcond: Sequence[str] = tuple()
+
+
+def _make_tree(obj, ifcond, features,
+               extra: Optional[Extra] = None):
+    comment = extra.comment if extra else None
+    extra = Extra(comment, ifcond)
     if features:
-        obj['features'] = [(f.name, {'if': f.ifcond}) for f in features]
-    if extra:
-        return (obj, extra)
-    return obj
+        obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]
+    return (obj, extra)
 
 
 def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -40,8 +47,8 @@ def indent(level):
 
     if isinstance(obj, tuple):
         ifobj, extra = obj
-        ifcond = extra.get('if')
-        comment = extra.get('comment')
+        ifcond = extra.ifcond
+        comment = extra.comment
         ret = ''
         if comment:
             ret += indent(level) + '/* %s */\n' % comment
@@ -168,7 +175,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
             if not self._unmask:
                 # Output a comment to make it easy to map masked names
                 # back to the source when reading the generated output.
-                extra = {'comment': '"%s" = %s' % (self._name(name), name)}
+                extra = Extra(comment=f'"{self._name(name)}" = {name}')
             name = self._name(name)
         obj['name'] = name
         obj['meta-type'] = mtype
-- 
2.26.2

[PATCH 31/37] qapi/introspect.py: add _gen_features helper

[John Snow]

_make_tree doesn't know if it is receiving an object or some other type;
adding features information should arguably be performed by the caller.

This will help us refactor _make_tree more gracefully in the next patch.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/introspect.py | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 41ca8afc67..e1edd0b179 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,7 @@
 See the COPYING file in the top-level directory.
 """
 
-from typing import (NamedTuple, Optional, Sequence)
+from typing import (NamedTuple, List, Optional, Sequence)
 
 from .common import (
     c_name,
@@ -20,7 +20,7 @@
 )
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                     QAPISchemaType)
+                     QAPISchemaFeature, QAPISchemaType)
 
 
 class Extra(NamedTuple):
@@ -31,12 +31,10 @@ class Extra(NamedTuple):
     ifcond: Sequence[str] = tuple()
 
 
-def _make_tree(obj, ifcond, features,
+def _make_tree(obj, ifcond,
                extra: Optional[Extra] = None):
     comment = extra.comment if extra else None
     extra = Extra(comment, ifcond)
-    if features:
-        obj['features'] = [(f.name, Extra(None, f.ifcond)) for f in features]
     return (obj, extra)
 
 
@@ -169,6 +167,10 @@ def _use_type(self, typ):
             return '[' + self._use_type(typ.element_type) + ']'
         return self._name(typ.name)
 
+    @classmethod
+    def _gen_features(cls, features: List[QAPISchemaFeature]):
+        return [_make_tree(f.name, f.ifcond) for f in features]
+
     def _gen_tree(self, name, mtype, obj, ifcond, features):
         extra = None
         if mtype not in ('command', 'event', 'builtin', 'array'):
@@ -179,13 +181,17 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
             name = self._name(name)
         obj['name'] = name
         obj['meta-type'] = mtype
-        self._trees.append(_make_tree(obj, ifcond, features, extra))
+        if features:
+            obj['features'] = self._gen_features(features)
+        self._trees.append(_make_tree(obj, ifcond, extra))
 
     def _gen_member(self, member):
         obj = {'name': member.name, 'type': self._use_type(member.type)}
         if member.optional:
             obj['default'] = None
-        return _make_tree(obj, member.ifcond, member.features)
+        if member.features:
+            obj['features'] = self._gen_features(member.features)
+        return _make_tree(obj, member.ifcond)
 
     def _gen_variants(self, tag_name, variants):
         return {'tag': tag_name,
@@ -193,7 +199,7 @@ def _gen_variants(self, tag_name, variants):
 
     def _gen_variant(self, variant):
         obj = {'case': variant.name, 'type': self._use_type(variant.type)}
-        return _make_tree(obj, variant.ifcond, None)
+        return _make_tree(obj, variant.ifcond)
 
     def visit_builtin_type(self, name, info, json_type):
         self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
-- 
2.26.2

[PATCH 32/37] qapi/introspect.py: create a typed 'Node' data structure

[John Snow]

Replacing the un-typed tuple, add a typed Node that we can add typed
metadata to.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/introspect.py | 53 ++++++++++++++++++++------------------
 1 file changed, 28 insertions(+), 25 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index e1edd0b179..e0f5007ab7 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -31,11 +31,18 @@ class Extra(NamedTuple):
     ifcond: Sequence[str] = tuple()
 
 
-def _make_tree(obj, ifcond,
-               extra: Optional[Extra] = None):
-    comment = extra.comment if extra else None
-    extra = Extra(comment, ifcond)
-    return (obj, extra)
+class Node:
+    """
+    Node generally contains a SchemaInfo-like type (as a dict),
+    But it also used to wrap comments/ifconds around leaf value types.
+    """
+    # Remove after 3.7 adds @dataclass:
+    # pylint: disable=too-few-public-methods
+    def __init__(self, data, ifcond: List[str],
+                 extra: Optional[Extra] = None):
+        self.data = data
+        comment = extra.comment if extra else None
+        self.extra = Extra(comment, ifcond)
 
 
 def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
@@ -43,18 +50,15 @@ def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
     def indent(level):
         return level * 4 * ' '
 
-    if isinstance(obj, tuple):
-        ifobj, extra = obj
-        ifcond = extra.ifcond
-        comment = extra.comment
+    if isinstance(obj, Node):
         ret = ''
-        if comment:
-            ret += indent(level) + '/* %s */\n' % comment
-        if ifcond:
-            ret += gen_if(ifcond)
-        ret += _tree_to_qlit(ifobj, level)
-        if ifcond:
-            ret += '\n' + gen_endif(ifcond)
+        if obj.extra.comment:
+            ret += indent(level) + '/* %s */\n' % obj.extra.comment
+        if obj.extra.ifcond:
+            ret += gen_if(obj.extra.ifcond)
+        ret += _tree_to_qlit(obj.data, level)
+        if obj.extra.ifcond:
+            ret += '\n' + gen_endif(obj.extra.ifcond)
         return ret
 
     ret = ''
@@ -169,7 +173,7 @@ def _use_type(self, typ):
 
     @classmethod
     def _gen_features(cls, features: List[QAPISchemaFeature]):
-        return [_make_tree(f.name, f.ifcond) for f in features]
+        return [Node(f.name, f.ifcond) for f in features]
 
     def _gen_tree(self, name, mtype, obj, ifcond, features):
         extra = None
@@ -183,7 +187,7 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
         obj['meta-type'] = mtype
         if features:
             obj['features'] = self._gen_features(features)
-        self._trees.append(_make_tree(obj, ifcond, extra))
+        self._trees.append(Node(obj, ifcond, extra))
 
     def _gen_member(self, member):
         obj = {'name': member.name, 'type': self._use_type(member.type)}
@@ -191,7 +195,7 @@ def _gen_member(self, member):
             obj['default'] = None
         if member.features:
             obj['features'] = self._gen_features(member.features)
-        return _make_tree(obj, member.ifcond)
+        return Node(obj, member.ifcond)
 
     def _gen_variants(self, tag_name, variants):
         return {'tag': tag_name,
@@ -199,15 +203,14 @@ def _gen_variants(self, tag_name, variants):
 
     def _gen_variant(self, variant):
         obj = {'case': variant.name, 'type': self._use_type(variant.type)}
-        return _make_tree(obj, variant.ifcond)
+        return Node(obj, variant.ifcond)
 
     def visit_builtin_type(self, name, info, json_type):
         self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
 
     def visit_enum_type(self, name, info, ifcond, features, members, prefix):
         self._gen_tree(name, 'enum',
-                       {'values': [_make_tree(m.name, m.ifcond, None)
-                                   for m in members]},
+                       {'values': [Node(m.name, m.ifcond) for m in members]},
                        ifcond, features)
 
     def visit_array_type(self, name, info, ifcond, element_type):
@@ -227,9 +230,9 @@ def visit_object_type_flat(self, name, info, ifcond, features,
     def visit_alternate_type(self, name, info, ifcond, features, variants):
         self._gen_tree(name, 'alternate',
                        {'members': [
-                           _make_tree({'type': self._use_type(m.type)},
-                                      m.ifcond, None)
-                           for m in variants.variants]},
+                           Node({'type': self._use_type(m.type)}, m.ifcond)
+                           for m in variants.variants
+                       ]},
                        ifcond, features)
 
     def visit_command(self, name, info, ifcond, features,
-- 
2.26.2

[PATCH 33/37] qapi/introspect.py: add notational type hints

[John Snow]

Add a tiny hint into schema.py to allow introspect.py to type check, too.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/introspect.py | 123 ++++++++++++++++++++++++++-----------
 scripts/qapi/mypy.ini      |   5 --
 scripts/qapi/schema.py     |   2 +-
 3 files changed, 87 insertions(+), 43 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index e0f5007ab7..4e1ea3a00c 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,16 @@
 See the COPYING file in the top-level directory.
 """
 
-from typing import (NamedTuple, List, Optional, Sequence)
+from typing import (
+    Any,
+    Dict,
+    Generic,
+    List,
+    NamedTuple,
+    Optional,
+    Sequence,
+    TypeVar,
+)
 
 from .common import (
     c_name,
@@ -19,8 +28,23 @@
     mcgen,
 )
 from .gen import QAPISchemaMonolithicCVisitor
-from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
-                     QAPISchemaFeature, QAPISchemaType)
+from .schema import (
+    QAPISchema,
+    QAPISchemaArrayType,
+    QAPISchemaBuiltinType,
+    QAPISchemaEntity,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaType,
+    QAPISchemaVariant,
+    QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
+
+
+_NodeType = TypeVar('_NodeType')
 
 
 class Extra(NamedTuple):
@@ -31,23 +55,24 @@ class Extra(NamedTuple):
     ifcond: Sequence[str] = tuple()
 
 
-class Node:
+class Node(Generic[_NodeType]):
     """
     Node generally contains a SchemaInfo-like type (as a dict),
     But it also used to wrap comments/ifconds around leaf value types.
     """
     # Remove after 3.7 adds @dataclass:
     # pylint: disable=too-few-public-methods
-    def __init__(self, data, ifcond: List[str],
+    def __init__(self, data: _NodeType, ifcond: List[str],
                  extra: Optional[Extra] = None):
         self.data = data
         comment = extra.comment if extra else None
         self.extra = Extra(comment, ifcond)
 
 
-def _tree_to_qlit(obj, level=0, suppress_first_indent=False):
+def _tree_to_qlit(obj: Any, level: int = 0,
+                  suppress_first_indent: bool = False) -> str:
 
-    def indent(level):
+    def indent(level: int) -> str:
         return level * 4 * ' '
 
     if isinstance(obj, Node):
@@ -94,21 +119,20 @@ def indent(level):
     return ret
 
 
-def to_c_string(string):
+def to_c_string(string: str) -> str:
     return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"'
 
 
 class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor):
-
-    def __init__(self, prefix, unmask):
+    def __init__(self, prefix: str, unmask: bool):
         super().__init__(
             prefix, 'qapi-introspect',
             ' * QAPI/QMP schema introspection', __doc__)
         self._unmask = unmask
-        self._schema = None
-        self._trees = []
-        self._used_types = []
-        self._name_map = {}
+        self._schema: Optional[QAPISchema] = None
+        self._trees: List[Node[Dict[str, Any]]] = []
+        self._used_types: List[QAPISchemaType] = []
+        self._name_map: Dict[str, str] = {}
         self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqapi-introspect.h"
@@ -116,10 +140,10 @@ def __init__(self, prefix, unmask):
 ''',
                              prefix=prefix))
 
-    def visit_begin(self, schema):
+    def visit_begin(self, schema: QAPISchema) -> None:
         self._schema = schema
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         # visit the types that are actually used
         for typ in self._used_types:
             typ.visit(self)
@@ -141,18 +165,18 @@ def visit_end(self):
         self._used_types = []
         self._name_map = {}
 
-    def visit_needed(self, entity):
+    def visit_needed(self, entity: QAPISchemaEntity) -> bool:
         # Ignore types on first pass; visit_end() will pick up used types
         return not isinstance(entity, QAPISchemaType)
 
-    def _name(self, name):
+    def _name(self, name: str) -> str:
         if self._unmask:
             return name
         if name not in self._name_map:
             self._name_map[name] = '%d' % len(self._name_map)
         return self._name_map[name]
 
-    def _use_type(self, typ):
+    def _use_type(self, typ: QAPISchemaType) -> str:
         # Map the various integer types to plain int
         if typ.json_type() == 'int':
             typ = self._schema.lookup_type('int')
@@ -172,10 +196,13 @@ def _use_type(self, typ):
         return self._name(typ.name)
 
     @classmethod
-    def _gen_features(cls, features: List[QAPISchemaFeature]):
+    def _gen_features(cls,
+                      features: List[QAPISchemaFeature]) -> List[Node[str]]:
         return [Node(f.name, f.ifcond) for f in features]
 
-    def _gen_tree(self, name, mtype, obj, ifcond, features):
+    def _gen_tree(self, name: str, mtype: str, obj: Dict[str, Any],
+                  ifcond: List[str],
+                  features: Optional[List[QAPISchemaFeature]]) -> None:
         extra = None
         if mtype not in ('command', 'event', 'builtin', 'array'):
             if not self._unmask:
@@ -189,7 +216,8 @@ def _gen_tree(self, name, mtype, obj, ifcond, features):
             obj['features'] = self._gen_features(features)
         self._trees.append(Node(obj, ifcond, extra))
 
-    def _gen_member(self, member):
+    def _gen_member(self, member: QAPISchemaObjectTypeMember,
+                    ) -> Node[Dict[str, Any]]:
         obj = {'name': member.name, 'type': self._use_type(member.type)}
         if member.optional:
             obj['default'] = None
@@ -197,29 +225,39 @@ def _gen_member(self, member):
             obj['features'] = self._gen_features(member.features)
         return Node(obj, member.ifcond)
 
-    def _gen_variants(self, tag_name, variants):
+    def _gen_variants(self, tag_name: str,
+                      variants: List[QAPISchemaVariant]) -> Dict[str, Any]:
         return {'tag': tag_name,
                 'variants': [self._gen_variant(v) for v in variants]}
 
-    def _gen_variant(self, variant):
+    def _gen_variant(self, variant: QAPISchemaVariant) -> Node[Dict[str, Any]]:
         obj = {'case': variant.name, 'type': self._use_type(variant.type)}
         return Node(obj, variant.ifcond)
 
-    def visit_builtin_type(self, name, info, json_type):
+    def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo],
+                           json_type: str) -> None:
         self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None)
 
-    def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+    def visit_enum_type(self, name: str, info: QAPISourceInfo,
+                        ifcond: List[str], features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]) -> None:
         self._gen_tree(name, 'enum',
                        {'values': [Node(m.name, m.ifcond) for m in members]},
                        ifcond, features)
 
-    def visit_array_type(self, name, info, ifcond, element_type):
+    def visit_array_type(self, name: str, info: Optional[QAPISourceInfo],
+                         ifcond: List[str],
+                         element_type: QAPISchemaType) -> None:
         element = self._use_type(element_type)
         self._gen_tree('[' + element + ']', 'array', {'element-type': element},
                        ifcond, None)
 
-    def visit_object_type_flat(self, name, info, ifcond, features,
-                               members, variants):
+    def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo],
+                               ifcond: List[str],
+                               features: List[QAPISchemaFeature],
+                               members: Sequence[QAPISchemaObjectTypeMember],
+                               variants: Optional[QAPISchemaVariants]) -> None:
         obj = {'members': [self._gen_member(m) for m in members]}
         if variants:
             obj.update(self._gen_variants(variants.tag_member.name,
@@ -227,7 +265,10 @@ def visit_object_type_flat(self, name, info, ifcond, features,
 
         self._gen_tree(name, 'object', obj, ifcond, features)
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self, name: str, info: QAPISourceInfo,
+                             ifcond: List[str],
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants) -> None:
         self._gen_tree(name, 'alternate',
                        {'members': [
                            Node({'type': self._use_type(m.type)}, m.ifcond)
@@ -235,24 +276,32 @@ def visit_alternate_type(self, name, info, ifcond, features, variants):
                        ]},
                        ifcond, features)
 
-    def visit_command(self, name, info, ifcond, features,
-                      arg_type, ret_type, gen, success_response, boxed,
-                      allow_oob, allow_preconfig):
+    def visit_command(self, name: str, info: QAPISourceInfo, ifcond: List[str],
+                      features: List[QAPISchemaFeature],
+                      arg_type: QAPISchemaObjectType,
+                      ret_type: Optional[QAPISchemaObjectType], gen: bool,
+                      success_response: bool, boxed: bool, allow_oob: bool,
+                      allow_preconfig: bool) -> None:
         arg_type = arg_type or self._schema.the_empty_object_type
         ret_type = ret_type or self._schema.the_empty_object_type
-        obj = {'arg-type': self._use_type(arg_type),
-               'ret-type': self._use_type(ret_type)}
+        obj: Dict[str, Any] = {
+            'arg-type': self._use_type(arg_type),
+            'ret-type': self._use_type(ret_type)
+        }
         if allow_oob:
             obj['allow-oob'] = allow_oob
         self._gen_tree(name, 'command', obj, ifcond, features)
 
-    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+    def visit_event(self, name: str, info: QAPISourceInfo,
+                    ifcond: List[str], features: List[QAPISchemaFeature],
+                    arg_type: QAPISchemaObjectType, boxed: bool) -> None:
         arg_type = arg_type or self._schema.the_empty_object_type
         self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)},
                        ifcond, features)
 
 
-def gen_introspect(schema, output_dir, prefix, opt_unmask):
+def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str,
+                   opt_unmask: bool) -> None:
     vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
     schema.visit(vis)
     vis.write(output_dir)
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index dbfeda748c..9ce8b56f22 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -19,11 +19,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.introspect]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.parser]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index bb0cd717f1..3023bab44b 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -28,7 +28,7 @@
 class QAPISchemaEntity:
     meta: Optional[str] = None
 
-    def __init__(self, name, info, doc, ifcond=None, features=None):
+    def __init__(self, name: str, info, doc, ifcond=None, features=None):
         assert name is None or isinstance(name, str)
         for f in features or []:
             assert isinstance(f, QAPISchemaFeature)
-- 
2.26.2

[PATCH 34/37] qapi/types.py: add notational type hints

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/types.py | 86 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 9ce8b56f22..3babc380a8 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -29,11 +29,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.types]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.visit]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 53b47f9e58..274d62fac8 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,6 +13,8 @@
 # See the COPYING file in the top-level directory.
 """
 
+from typing import Optional, List
+
 from .common import (
     c_enum_const,
     c_name,
@@ -21,7 +23,16 @@
     mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .source import QAPISourceInfo
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaObjectType,
+    QAPISchemaVariants,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaFeature,
+    QAPISchemaType,
+)
 
 
 # variants must be emitted before their container; track what has already
@@ -29,7 +40,9 @@
 objects_seen = set()
 
 
-def gen_enum_lookup(name, members, prefix=None):
+def gen_enum_lookup(name: str,
+                    members: List[QAPISchemaEnumMember],
+                    prefix: Optional[str] = None) -> str:
     ret = mcgen('''
 
 const QEnumLookup %(c_name)s_lookup = {
@@ -54,7 +67,9 @@ def gen_enum_lookup(name, members, prefix=None):
     return ret
 
 
-def gen_enum(name, members, prefix=None):
+def gen_enum(name: str,
+             members: List[QAPISchemaEnumMember],
+             prefix: Optional[str] = None) -> str:
     # append automatically generated _MAX value
     enum_members = members + [QAPISchemaEnumMember('_MAX', None)]
 
@@ -88,7 +103,7 @@ def gen_enum(name, members, prefix=None):
     return ret
 
 
-def gen_fwd_object_or_array(name):
+def gen_fwd_object_or_array(name: str) -> str:
     return mcgen('''
 
 typedef struct %(c_name)s %(c_name)s;
@@ -96,7 +111,7 @@ def gen_fwd_object_or_array(name):
                  c_name=c_name(name))
 
 
-def gen_array(name, element_type):
+def gen_array(name: str, element_type: QAPISchemaType) -> str:
     return mcgen('''
 
 struct %(c_name)s {
@@ -107,7 +122,7 @@ def gen_array(name, element_type):
                  c_name=c_name(name), c_type=element_type.c_type())
 
 
-def gen_struct_members(members):
+def gen_struct_members(members: List[QAPISchemaObjectTypeMember]) -> str:
     ret = ''
     for memb in members:
         ret += gen_if(memb.ifcond)
@@ -124,7 +139,10 @@ def gen_struct_members(members):
     return ret
 
 
-def gen_object(name, ifcond, base, members, variants):
+def gen_object(name: str, ifcond: List[str],
+               base: Optional[QAPISchemaObjectType],
+               members: List[QAPISchemaObjectTypeMember],
+               variants: Optional[QAPISchemaVariants]) -> str:
     if name in objects_seen:
         return ''
     objects_seen.add(name)
@@ -178,7 +196,7 @@ def gen_object(name, ifcond, base, members, variants):
     return ret
 
 
-def gen_upcast(name, base):
+def gen_upcast(name: str, base: QAPISchemaObjectType) -> str:
     # C makes const-correctness ugly.  We have to cast away const to let
     # this function work for both const and non-const obj.
     return mcgen('''
@@ -191,7 +209,7 @@ def gen_upcast(name, base):
                  c_name=c_name(name), base=base.c_name())
 
 
-def gen_variants(variants):
+def gen_variants(variants: QAPISchemaVariants) -> str:
     ret = mcgen('''
     union { /* union tag is @%(c_name)s */
 ''',
@@ -215,7 +233,7 @@ def gen_variants(variants):
     return ret
 
 
-def gen_type_cleanup_decl(name):
+def gen_type_cleanup_decl(name: str) -> str:
     ret = mcgen('''
 
 void qapi_free_%(c_name)s(%(c_name)s *obj);
@@ -225,7 +243,7 @@ def gen_type_cleanup_decl(name):
     return ret
 
 
-def gen_type_cleanup(name):
+def gen_type_cleanup(name: str) -> str:
     ret = mcgen('''
 
 void qapi_free_%(c_name)s(%(c_name)s *obj)
@@ -247,12 +265,12 @@ def gen_type_cleanup(name):
 
 class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-types', ' * Schema-defined QAPI types',
             ' * Built-in QAPI types', __doc__)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
@@ -263,7 +281,7 @@ def _begin_system_module(self, name):
 #include "qapi/util.h"
 '''))
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
         self._genc.preamble_add(mcgen('''
@@ -277,27 +295,43 @@ def _begin_user_module(self, name):
 #include "qapi/qapi-builtin-types.h"
 '''))
 
-    def visit_begin(self, schema):
+    def visit_begin(self, schema: QAPISchema) -> None:
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
 
-    def _gen_type_cleanup(self, name):
+    def _gen_type_cleanup(self, name: str) -> None:
         self._genh.add(gen_type_cleanup_decl(name))
         self._genc.add(gen_type_cleanup(name))
 
-    def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+    def visit_enum_type(self,
+                        name: str,
+                        info: Optional[QAPISourceInfo],
+                        ifcond: List[str],
+                        features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.preamble_add(gen_enum(name, members, prefix))
             self._genc.add(gen_enum_lookup(name, members, prefix))
 
-    def visit_array_type(self, name, info, ifcond, element_type):
+    def visit_array_type(self,
+                         name: str,
+                         info: Optional[QAPISourceInfo],
+                         ifcond: List[str],
+                         element_type: QAPISchemaType) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.preamble_add(gen_fwd_object_or_array(name))
             self._genh.add(gen_array(name, element_type))
             self._gen_type_cleanup(name)
 
-    def visit_object_type(self, name, info, ifcond, features,
-                          base, members, variants):
+    def visit_object_type(self,
+                          name: str,
+                          info: Optional[QAPISourceInfo],
+                          ifcond: List[str],
+                          features: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> None:
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
@@ -313,7 +347,12 @@ def visit_object_type(self, name, info, ifcond, features,
                 # implicit types won't be directly allocated/freed
                 self._gen_type_cleanup(name)
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self,
+                             name: str,
+                             info: QAPISourceInfo,
+                             ifcond: List[str],
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants) -> None:
         with ifcontext(ifcond, self._genh):
             self._genh.preamble_add(gen_fwd_object_or_array(name))
         self._genh.add(gen_object(name, ifcond, None,
@@ -322,7 +361,10 @@ def visit_alternate_type(self, name, info, ifcond, features, variants):
             self._gen_type_cleanup(name)
 
 
-def gen_types(schema, output_dir, prefix, opt_builtins):
+def gen_types(schema: QAPISchema,
+              output_dir: str,
+              prefix: str,
+              opt_builtins: bool) -> None:
     vis = QAPISchemaGenTypeVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir, opt_builtins)
-- 
2.26.2

[PATCH 35/37] qapi/types.py: remove one-letter variables

[John Snow]

"John, if pylint told you to jump off a bridge, would you?"
Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/pylintrc |  1 -
 scripts/qapi/types.py | 29 +++++++++++++++--------------
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 3aebb0d228..9ca6b60caf 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -7,7 +7,6 @@ ignore-patterns=doc.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                types.py,
                 visit.py,
 
 
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 274d62fac8..a0a06b68c5 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -49,14 +49,14 @@ def gen_enum_lookup(name: str,
     .array = (const char *const[]) {
 ''',
                 c_name=c_name(name))
-    for m in members:
-        ret += gen_if(m.ifcond)
-        index = c_enum_const(name, m.name, prefix)
+    for member in members:
+        ret += gen_if(member.ifcond)
+        index = c_enum_const(name, member.name, prefix)
         ret += mcgen('''
         [%(index)s] = "%(name)s",
 ''',
-                     index=index, name=m.name)
-        ret += gen_endif(m.ifcond)
+                     index=index, name=member.name)
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
     },
@@ -79,13 +79,13 @@ def gen_enum(name: str,
 ''',
                 c_name=c_name(name))
 
-    for m in enum_members:
-        ret += gen_if(m.ifcond)
+    for member in enum_members:
+        ret += gen_if(member.ifcond)
         ret += mcgen('''
     %(c_enum)s,
 ''',
-                     c_enum=c_enum_const(name, m.name, prefix))
-        ret += gen_endif(m.ifcond)
+                     c_enum=c_enum_const(name, member.name, prefix))
+        ret += gen_endif(member.ifcond)
 
     ret += mcgen('''
 } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
     objects_seen.add(name)
 
     ret = ''
-    if variants:
-        for v in variants.variants:
-            if isinstance(v.type, QAPISchemaObjectType):
-                ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
-                                  v.type.local_members, v.type.variants)
+    for variant in variants.variants if variants else ():
+        obj = variant.type
+        if not isinstance(obj, QAPISchemaObjectType):
+            continue
+        ret += gen_object(obj.name, obj.ifcond, obj.base,
+                          obj.local_members, obj.variants)
 
     ret += mcgen('''
 
-- 
2.26.2

[PATCH 36/37] qapi/visit.py: remove unused parameters from gen_visit_object

[John Snow]

And this fixes the pylint report for this file, so make sure we check
this in the future, too.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/pylintrc | 1 -
 scripts/qapi/visit.py | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 9ca6b60caf..9d7a1d285f 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -7,7 +7,6 @@ ignore-patterns=doc.py,
                 expr.py,
                 parser.py,
                 schema.py,
-                visit.py,
 
 
 [MESSAGES CONTROL]
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 66ce3dc52e..ce71659a03 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -247,7 +247,7 @@ def gen_visit_alternate(name, variants):
     return ret
 
 
-def gen_visit_object(name, base, members, variants):
+def gen_visit_object(name):
     return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -340,7 +340,7 @@ def visit_object_type(self, name, info, ifcond, features,
             if not name.startswith('q_'):
                 # only explicit types need an allocating visit
                 self._genh.add(gen_visit_decl(name))
-                self._genc.add(gen_visit_object(name, base, members, variants))
+                self._genc.add(gen_visit_object(name))
 
     def visit_alternate_type(self, name, info, ifcond, features, variants):
         with ifcontext(ifcond, self._genh, self._genc):
-- 
2.26.2

[PATCH 37/37] qapi/visit.py: add notational type hints

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/visit.py | 72 +++++++++++++++++++++++++++++++++----------
 2 files changed, 55 insertions(+), 22 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 3babc380a8..5ab3433c5f 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -28,8 +28,3 @@ check_untyped_defs = False
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
-
-[mypy-qapi.visit]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index ce71659a03..035521476f 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,6 +13,8 @@
 See the COPYING file in the top-level directory.
 """
 
+from typing import List, Optional
+
 from .common import (
     c_enum_const,
     c_name,
@@ -22,10 +24,19 @@
     INDENT,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaVariants,
+    QAPISchemaType,
+)
+from .source import QAPISourceInfo
 
 
-def gen_visit_decl(name, scalar=False):
+def gen_visit_decl(name: str, scalar: bool = False) -> str:
     c_type = c_name(name) + ' *'
     if not scalar:
         c_type += '*'
@@ -36,7 +47,7 @@ def gen_visit_decl(name, scalar=False):
                  c_name=c_name(name), c_type=c_type)
 
 
-def gen_visit_members_decl(name):
+def gen_visit_members_decl(name: str) -> str:
     return mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp);
@@ -44,7 +55,10 @@ def gen_visit_members_decl(name):
                  c_name=c_name(name))
 
 
-def gen_visit_object_members(name, base, members, variants):
+def gen_visit_object_members(name: str,
+                             base: Optional[QAPISchemaObjectType],
+                             members: List[QAPISchemaObjectTypeMember],
+                             variants: Optional[QAPISchemaVariants]) -> str:
     ret = mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
@@ -122,7 +136,7 @@ def gen_visit_object_members(name, base, members, variants):
     return ret
 
 
-def gen_visit_list(name, element_type):
+def gen_visit_list(name: str, element_type: QAPISchemaType) -> str:
     return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -156,7 +170,7 @@ def gen_visit_list(name, element_type):
                  c_name=c_name(name), c_elt_type=element_type.c_name())
 
 
-def gen_visit_enum(name):
+def gen_visit_enum(name: str) -> str:
     return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj,
@@ -171,7 +185,7 @@ def gen_visit_enum(name):
                  c_name=c_name(name))
 
 
-def gen_visit_alternate(name, variants):
+def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
     ret = mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -247,7 +261,7 @@ def gen_visit_alternate(name, variants):
     return ret
 
 
-def gen_visit_object(name):
+def gen_visit_object(name: str) -> str:
     return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj,
@@ -282,12 +296,12 @@ def gen_visit_object(name):
 
 class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix):
+    def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-visit', ' * Schema-defined QAPI visitors',
             ' * Built-in QAPI visitors', __doc__)
 
-    def _begin_system_module(self, name):
+    def _begin_system_module(self, name: None) -> None:
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/error.h"
@@ -299,7 +313,7 @@ def _begin_system_module(self, name):
 
 '''))
 
-    def _begin_user_module(self, name):
+    def _begin_user_module(self, name: str) -> None:
         types = self._module_basename('qapi-types', name)
         visit = self._module_basename('qapi-visit', name)
         self._genc.preamble_add(mcgen('''
@@ -316,18 +330,34 @@ def _begin_user_module(self, name):
 ''',
                                       types=types))
 
-    def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+    def visit_enum_type(self,
+                        name: str,
+                        info: QAPISourceInfo,
+                        ifcond: List[str],
+                        features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name, scalar=True))
             self._genc.add(gen_visit_enum(name))
 
-    def visit_array_type(self, name, info, ifcond, element_type):
+    def visit_array_type(self,
+                         name: str,
+                         info: Optional[QAPISourceInfo],
+                         ifcond: List[str],
+                         element_type: QAPISchemaType) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name))
             self._genc.add(gen_visit_list(name, element_type))
 
-    def visit_object_type(self, name, info, ifcond, features,
-                          base, members, variants):
+    def visit_object_type(self,
+                          name: str,
+                          info: Optional[QAPISourceInfo],
+                          ifcond: List[str],
+                          features: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> None:
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
@@ -342,13 +372,21 @@ def visit_object_type(self, name, info, ifcond, features,
                 self._genh.add(gen_visit_decl(name))
                 self._genc.add(gen_visit_object(name))
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self,
+                             name: str,
+                             info: QAPISourceInfo,
+                             ifcond: List[str],
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name))
             self._genc.add(gen_visit_alternate(name, variants))
 
 
-def gen_visit(schema, output_dir, prefix, opt_builtins):
+def gen_visit(schema: QAPISchema,
+              output_dir: str,
+              prefix: str,
+              opt_builtins: bool) -> None:
     vis = QAPISchemaGenVisitVisitor(prefix)
     schema.visit(vis)
     vis.write(output_dir, opt_builtins)
-- 
2.26.2

Re: [PATCH 01/37] python: Require 3.6+

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Python 3.5 is now EOL. Python 3.6 brings with it several nice things:
>
> - Literal string interpolation (PEP 498)
> - Syntax for variable annotations (PEP 526)
> - Preserving keyword argument order (PEP 468) *
>
> PEP 526 in particular will allow us to convert the QAPI module to the
> statically typed subset of the Python language. These static type hints
> improve the interactive editing experience in IDEs and help make
> refactoring and adding new features faster, easier and safer.
>
> *Note: **kwargs is now guaranteed to preserve keyword ordering, but dict
> itself is still allowed to have an arbitrary order based on
> implementation. It is not until Python 3.7 that the dict type guarantees
> insertion order.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  docs/conf.py                  | 4 ++--
>  configure                     | 6 +++---
>  .readthedocs.yml              | 2 +-
>  .travis.yml                   | 8 --------
>  tests/qemu-iotests/iotests.py | 2 --
>  5 files changed, 6 insertions(+), 16 deletions(-)
>
> diff --git a/docs/conf.py b/docs/conf.py
> index 0dbd90dc11..8aeac40124 100644
> --- a/docs/conf.py
> +++ b/docs/conf.py
> @@ -36,9 +36,9 @@
>  # In newer versions of Sphinx this will display nicely; in older versions
>  # Sphinx will also produce a Python backtrace but at least the information
>  # gets printed...
> -if sys.version_info < (3,5):
> +if sys.version_info < (3,6):
>      raise ConfigError(
> -        "QEMU requires a Sphinx that uses Python 3.5 or better\n")
> +        "QEMU requires a Sphinx that uses Python 3.6 or better\n")
>  
>  # The per-manual conf.py will set qemu_docdir for a single-manual build;
>  # otherwise set it here if this is an entire-manual-set build.
> diff --git a/configure b/configure
> index ce27eafb0a..33292500e7 100755
> --- a/configure
> +++ b/configure
> @@ -913,7 +913,7 @@ fi
>  
>  : ${make=${MAKE-make}}
>  
> -# We prefer python 3.x. A bare 'python' is traditionally
> +# We prefer python 3.6+. A bare 'python' is traditionally

We require

>  # python 2.x, but some distros have it as python 3.x, so
>  # we check that too
>  python=
> @@ -1961,8 +1961,8 @@ fi
>  
>  # Note that if the Python conditional here evaluates True we will exit
>  # with status 1 which is a shell 'false' value.
> -if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
> -  error_exit "Cannot use '$python', Python >= 3.5 is required." \
> +if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
> +  error_exit "Cannot use '$python', Python >= 3.6 is required." \
>        "Use --python=/path/to/python to specify a supported Python."
>  fi
>  
> diff --git a/.readthedocs.yml b/.readthedocs.yml
> index 8355dbc634..7fb7b8dd61 100644
> --- a/.readthedocs.yml
> +++ b/.readthedocs.yml
> @@ -17,4 +17,4 @@ formats: all
>  # we require for other Python in our codebase (our conf.py
>  # enforces this, and some code needs it.)
>  python:
> -  version: 3.5
> +  version: 3.6
> diff --git a/.travis.yml b/.travis.yml
> index 65341634d0..0dca15a5e8 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -267,14 +267,6 @@ jobs:
>  
>  
>      # Python builds
> -    - name: "GCC Python 3.5 (x86_64-softmmu)"
> -      env:
> -        - CONFIG="--target-list=x86_64-softmmu"
> -        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
> -      language: python
> -      python: 3.5
> -
> -
>      - name: "GCC Python 3.6 (x86_64-softmmu)"
>        env:
>          - CONFIG="--target-list=x86_64-softmmu"
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 91e4a57126..f48460480a 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -40,8 +40,6 @@
>  from qemu import qtest
>  from qemu.qmp import QMPMessage
>  
> -assert sys.version_info >= (3, 6)
> -
>  # Use this logger for logging messages directly from the iotests module
>  logger = logging.getLogger('qemu.iotests')
>  logger.addHandler(logging.NullHandler())

Re: [PATCH 01/37] python: Require 3.6+

[Daniel P. Berrangé]

On Tue, Sep 15, 2020 at 06:39:51PM -0400, John Snow wrote:
> Python 3.5 is now EOL. Python 3.6 brings with it several nice things:

NB Upstream EOL is not relevant. We need to base min version changes
on the OS platform support matrix. So it would be better for the
commit message to provide details of the version in the various
distros we target, to show that the min version change is valid.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 04/37] qapi: move generator entrypoint into module

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> As part of delinting and adding type hints to the QAPI generator, it's
> helpful for the entrypoint to be part of the module, only leaving a very
> tiny entrypoint shim outside of the module.
>
> As a result, all of the include statements are reworked to be module-aware,
> as explicit relative imports.

Should this be split into two commits, one for each of the paragraphs
above?

PEP 8 recommends absolute imports, with one exception:

    However, explicit relative imports are an acceptable alternative to
    absolute imports, especially when dealing with complex package
    layouts where using absolute imports would be unnecessarily verbose:

        from . import sibling
        from .sibling import example

    Standard library code should avoid complex package layouts and
    always use absolute imports.

Do you think it covers your use of relative imports?

> This is done primarily for the benefit of python tooling (pylint, mypy,
> flake8, et al) which otherwise has trouble consistently resolving
> "qapi.x" to mean "a sibling file in this folder."

Can you give me an example of some tool having troube?

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi-gen.py        | 94 +++-----------------------------------
>  scripts/qapi/commands.py   |  4 +-
>  scripts/qapi/doc.py        |  2 +-
>  scripts/qapi/events.py     |  8 ++--
>  scripts/qapi/expr.py       |  4 +-
>  scripts/qapi/gen.py        |  4 +-
>  scripts/qapi/introspect.py |  8 ++--
>  scripts/qapi/parser.py     |  4 +-
>  scripts/qapi/schema.py     |  8 ++--
>  scripts/qapi/script.py     | 91 ++++++++++++++++++++++++++++++++++++
>  scripts/qapi/types.py      |  6 +--
>  scripts/qapi/visit.py      |  6 +--
>  12 files changed, 124 insertions(+), 115 deletions(-)
>  create mode 100644 scripts/qapi/script.py

Naming is hard...  main.py?

>
> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
> index 59becba3e1..e649f8dd44 100644
> --- a/scripts/qapi-gen.py
> +++ b/scripts/qapi-gen.py
> @@ -1,97 +1,15 @@
>  #!/usr/bin/env python3
> -
> -# This work is licensed under the terms of the GNU GPL, version 2 or later.
> -# See the COPYING file in the top-level directory.
> -

Keep the license blurb.

[...]

Re: [PATCH 06/37] qapi: delint using flake8

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Petty style guide fixes and line length enforcement.  Not a big win, not
> a big loss, but flake8 passes 100% on the qapi module, which gives us an
> easy baseline to enforce hereafter.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/.flake8     |  2 ++
>  scripts/qapi/commands.py |  3 ++-
>  scripts/qapi/schema.py   |  8 +++++---
>  scripts/qapi/visit.py    | 15 ++++++++++-----
>  4 files changed, 19 insertions(+), 9 deletions(-)
>  create mode 100644 scripts/qapi/.flake8
>
> diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
> new file mode 100644
> index 0000000000..45d8146f3f
> --- /dev/null
> +++ b/scripts/qapi/.flake8
> @@ -0,0 +1,2 @@
> +[flake8]
> +extend-ignore = E722  # Pylint handles this, but smarter.

I guess you mean pylint's W0702 a.k.a. bare-except.  What's wrong with
flake8's E722 compared to pylint's W0702?

> \ No newline at end of file

So put one there :)

> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
> index e1df0e341f..2e4b4de0fa 100644
> --- a/scripts/qapi/commands.py
> +++ b/scripts/qapi/commands.py
> @@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
>  def gen_marshal_output(ret_type):
>      return mcgen('''
>  
> -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
> +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
> +                                          Error **errp)

The continued parameter list may become misalignd in generated C.  E.g.

    static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
                                              Error **errp)
    {
    ...
    }

Do we care?

More of the same below.

>  {
>      Visitor *v;
>  
[...]

Re: [PATCH 07/37] qapi: add pylintrc

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Add a skeleton pylintrc file. Right now, it ignores quite a few things.
> Files will be removed from the known-bad list throughout this and
> following series as they are repaired.
>
> Note: Normally, pylintrc would go in the folder above the module, but as
> that folder is shared by many things, it is going inside the module
> folder now.
>
> Due to some bugs in different versions of pylint (2.5.x), pylint does
> not correctly recognize when it is being run from "inside" a module, and
> must be run *outside* of the module.
>
> Therefore, to run it, you must:
>
>  > cd :/qemu/scripts

-bash: cd: :/qemu/scripts: No such file or directory

;-P

>  > pylint qapi/ --rcfile=qapi/pylintrc

Why not

   $ pylint scripts/qapi --rcfile=scripts/qapi/pylintrc

>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 74 insertions(+)
>  create mode 100644 scripts/qapi/pylintrc
>
> diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
> new file mode 100644
> index 0000000000..c2bbb8e8e1
> --- /dev/null
> +++ b/scripts/qapi/pylintrc
> @@ -0,0 +1,74 @@
> +[MASTER]
> +
> +# Add files or directories matching the regex patterns to the blacklist. The
> +# regex matches against base names, not paths.
> +ignore-patterns=common.py,
> +                doc.py,
> +                error.py,
> +                expr.py,
> +                gen.py,
> +                parser.py,
> +                schema.py,
> +                source.py,
> +                types.py,
> +                visit.py,

Already not ignored:

    __init__.py
    commands.py
    common.py
    debug.py
    events.py
    introspect.py
    script.py

Okay.

> +
> +
> +[MESSAGES CONTROL]
> +
> +# Disable the message, report, category or checker with the given id(s). You
> +# can either give multiple identifiers separated by comma (,) or put this
> +# option multiple times (only on the command line, not in the configuration
> +# file where it should appear only once). You can also use "--disable=all" to
> +# disable everything first and then reenable specific checks. For example, if
> +# you want to run only the similarities checker, you can use "--disable=all
> +# --enable=similarities". If you want to run only the classes checker, but have
> +# no Warning level messages displayed, use "--disable=all --enable=classes
> +# --disable=W".
> +disable=fixme,
> +        missing-docstring,
> +        too-many-arguments,
> +        too-many-branches,
> +        too-many-statements,
> +        too-many-instance-attributes,

I'm fine with disabling these.

> +
> +[REPORTS]
> +
> +[REFACTORING]
> +
> +[MISCELLANEOUS]
> +
> +[LOGGING]
> +
> +[BASIC]
> +
> +# Good variable names which should always be accepted, separated by a comma.
> +good-names=i,
> +           j,
> +           k,
> +           ex,
> +           Run,
> +           _

Isn't this the default?

> +
> +[VARIABLES]
> +
> +[STRING]
> +
> +[SPELLING]
> +
> +[FORMAT]
> +
> +[SIMILARITIES]
> +
> +# Ignore imports when computing similarities.
> +ignore-imports=yes

Why?

> +
> +[TYPECHECK]
> +
> +[CLASSES]
> +
> +[IMPORTS]
> +
> +[DESIGN]
> +
> +[EXCEPTIONS]

Looks like you started with output of --generate-rcfile, 

Re: [PATCH 08/37] qapi/common.py: Remove python compatibility workaround

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/common.py | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index ba35abea47..4fb265a8bf 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -119,10 +119,7 @@ def cgen(code, **kwds):
>      raw = code % kwds
>      if indent_level:
>          indent = genindent(indent_level)
> -        # re.subn() lacks flags support before Python 2.7, use re.compile()
> -        raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
> -                      indent, raw)
> -        raw = raw[0]
> +        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
>      return re.sub(re.escape(eatspace) + r' *', '', raw)

I missed this one in my "qapi: Bye-bye Python 2" series.

Can we use re.sub() instead?

Re: [PATCH 04/37] qapi: move generator entrypoint into module

[John Snow]

On 9/16/20 7:54 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> As part of delinting and adding type hints to the QAPI generator, it's
>> helpful for the entrypoint to be part of the module, only leaving a very
>> tiny entrypoint shim outside of the module.
>>
>> As a result, all of the include statements are reworked to be module-aware,
>> as explicit relative imports.
> 
> Should this be split into two commits, one for each of the paragraphs
> above?
> 

Hmm ... I hadn't considered it was possible, but actually ... I guess I 
can split those out, yeah.

> PEP 8 recommends absolute imports, with one exception:
> 
>      However, explicit relative imports are an acceptable alternative to
>      absolute imports, especially when dealing with complex package
>      layouts where using absolute imports would be unnecessarily verbose:
> 
>          from . import sibling
>          from .sibling import example
> 
>      Standard library code should avoid complex package layouts and
>      always use absolute imports.
> 
> Do you think it covers your use of relative imports?
> 

Admittedly I am going by trial and error; my experience is that the 
relative imports behave the nicest.

There is a historical fear of explicit relative imports because they are 
"new" and years of Python2 compatibility rendered many afraid of them. 
It is advice safely ignored in my opinion.

Using absolute imports (e.g. from qapi.sibling import foo) gets messy in 
virtual environments when you have *installed* the module in question: 
it becomes ambiguous as to *which* qapi you meant: the one in this 
folder, or the one installed to the environment?

Python, mypy, pylint, flake8 etc disagree sometimes, or can get 
confused; thinking there are two copies of each module.

Just my experience that relative imports seem to give me the least trouble.

>> This is done primarily for the benefit of python tooling (pylint, mypy,
>> flake8, et al) which otherwise has trouble consistently resolving
>> "qapi.x" to mean "a sibling file in this folder."
> 
> Can you give me an example of some tool having troube?
> 

I'd have to code up some examples. I have some hobby code unrelated to 
QEMU where I struggled to get flake8, mypy, and pylint all cooperating 
with an import regime until I gave up and used explicit relative imports.

>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi-gen.py        | 94 +++-----------------------------------
>>   scripts/qapi/commands.py   |  4 +-
>>   scripts/qapi/doc.py        |  2 +-
>>   scripts/qapi/events.py     |  8 ++--
>>   scripts/qapi/expr.py       |  4 +-
>>   scripts/qapi/gen.py        |  4 +-
>>   scripts/qapi/introspect.py |  8 ++--
>>   scripts/qapi/parser.py     |  4 +-
>>   scripts/qapi/schema.py     |  8 ++--
>>   scripts/qapi/script.py     | 91 ++++++++++++++++++++++++++++++++++++
>>   scripts/qapi/types.py      |  6 +--
>>   scripts/qapi/visit.py      |  6 +--
>>   12 files changed, 124 insertions(+), 115 deletions(-)
>>   create mode 100644 scripts/qapi/script.py
> 
> Naming is hard...  main.py?
> 

I was thinking of changing this myself, so this convinced me.

>>
>> diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
>> index 59becba3e1..e649f8dd44 100644
>> --- a/scripts/qapi-gen.py
>> +++ b/scripts/qapi-gen.py
>> @@ -1,97 +1,15 @@
>>   #!/usr/bin/env python3
>> -
>> -# This work is licensed under the terms of the GNU GPL, version 2 or later.
>> -# See the COPYING file in the top-level directory.
>> -
> 
> Keep the license blurb.
> 

This is a mistake. I tried to convince git to "move" the old file and 
then add a "new" file to preserve history, but of course that's not how 
git manages file histories, so it didn't work.

TLDR: I didn't delete the license blurb, I just didn't "add" it again.
I'll "fix" that.

> [...]
> 

Re: [PATCH 06/37] qapi: delint using flake8

[John Snow]

On 9/16/20 8:12 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Petty style guide fixes and line length enforcement.  Not a big win, not
>> a big loss, but flake8 passes 100% on the qapi module, which gives us an
>> easy baseline to enforce hereafter.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/.flake8     |  2 ++
>>   scripts/qapi/commands.py |  3 ++-
>>   scripts/qapi/schema.py   |  8 +++++---
>>   scripts/qapi/visit.py    | 15 ++++++++++-----
>>   4 files changed, 19 insertions(+), 9 deletions(-)
>>   create mode 100644 scripts/qapi/.flake8
>>
>> diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
>> new file mode 100644
>> index 0000000000..45d8146f3f
>> --- /dev/null
>> +++ b/scripts/qapi/.flake8
>> @@ -0,0 +1,2 @@
>> +[flake8]
>> +extend-ignore = E722  # Pylint handles this, but smarter.
> 
> I guess you mean pylint's W0702 a.k.a. bare-except.  What's wrong with
> flake8's E722 compared to pylint's W0702?
> 

Flake8 will warn on *any* bare except, but Pylint's is context-aware and 
will suppress the warning if you re-raise the exception.

I don't actually think this comes up in the qapi code base, but it does 
come up in the ./python/qemu code base.

(One of my goals is unifying the lint checking regime for both packages.)

>> \ No newline at end of file
> 
> So put one there :)
> 

Whupps, okay.

>> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
>> index e1df0e341f..2e4b4de0fa 100644
>> --- a/scripts/qapi/commands.py
>> +++ b/scripts/qapi/commands.py
>> @@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
>>   def gen_marshal_output(ret_type):
>>       return mcgen('''
>>   
>> -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
>> +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
>> +                                          Error **errp)
> 
> The continued parameter list may become misalignd in generated C.  E.g.
> 
>      static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
>                                                Error **errp)
>      {
>      ...
>      }
> 
> Do we care?
> 

Yeah, I don't know. Do we?

It actually seemed more annoying to try and get flake8 to make an 
exception for these handful of examples.

Path of least resistance led me here, but I can try and appease both 
systems if you'd prefer.

> More of the same below.
> 
>>   {
>>       Visitor *v;
>>   
> [...]
> 

Re: [PATCH 07/37] qapi: add pylintrc

[John Snow]

On 9/16/20 8:30 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Add a skeleton pylintrc file. Right now, it ignores quite a few things.
>> Files will be removed from the known-bad list throughout this and
>> following series as they are repaired.
>>
>> Note: Normally, pylintrc would go in the folder above the module, but as
>> that folder is shared by many things, it is going inside the module
>> folder now.
>>
>> Due to some bugs in different versions of pylint (2.5.x), pylint does
>> not correctly recognize when it is being run from "inside" a module, and
>> must be run *outside* of the module.
>>
>> Therefore, to run it, you must:
>>
>>   > cd :/qemu/scripts
> 
> -bash: cd: :/qemu/scripts: No such file or directory
> 
> ;-P
> 
>>   > pylint qapi/ --rcfile=qapi/pylintrc
> 
> Why not
> 
>     $ pylint scripts/qapi --rcfile=scripts/qapi/pylintrc
> 

No reason I'm aware of, I have just been testing with CWD at the scripts 
dir myself because of how python imports work.

If it works this way, enjoy!

>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 74 insertions(+)
>>   create mode 100644 scripts/qapi/pylintrc
>>
>> diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
>> new file mode 100644
>> index 0000000000..c2bbb8e8e1
>> --- /dev/null
>> +++ b/scripts/qapi/pylintrc
>> @@ -0,0 +1,74 @@
>> +[MASTER]
>> +
>> +# Add files or directories matching the regex patterns to the blacklist. The
>> +# regex matches against base names, not paths.
>> +ignore-patterns=common.py,
>> +                doc.py,
>> +                error.py,
>> +                expr.py,
>> +                gen.py,
>> +                parser.py,
>> +                schema.py,
>> +                source.py,
>> +                types.py,
>> +                visit.py,
> 
> Already not ignored:
> 
>      __init__.py
>      commands.py
>      common.py
>      debug.py
>      events.py
>      introspect.py
>      script.py
> 
> Okay.
> 
>> +
>> +
>> +[MESSAGES CONTROL]
>> +
>> +# Disable the message, report, category or checker with the given id(s). You
>> +# can either give multiple identifiers separated by comma (,) or put this
>> +# option multiple times (only on the command line, not in the configuration
>> +# file where it should appear only once). You can also use "--disable=all" to
>> +# disable everything first and then reenable specific checks. For example, if
>> +# you want to run only the similarities checker, you can use "--disable=all
>> +# --enable=similarities". If you want to run only the classes checker, but have
>> +# no Warning level messages displayed, use "--disable=all --enable=classes
>> +# --disable=W".
>> +disable=fixme,
>> +        missing-docstring,
>> +        too-many-arguments,
>> +        too-many-branches,
>> +        too-many-statements,
>> +        too-many-instance-attributes,
> 
> I'm fine with disabling these.
> 

I'd like to enable missing-docstring eventually, but that's not for today.

>> +
>> +[REPORTS]
>> +
>> +[REFACTORING]
>> +
>> +[MISCELLANEOUS]
>> +
>> +[LOGGING]
>> +
>> +[BASIC]
>> +
>> +# Good variable names which should always be accepted, separated by a comma.
>> +good-names=i,
>> +           j,
>> +           k,
>> +           ex,
>> +           Run,
>> +           _
> 
> Isn't this the default?
> 

Yes. I could omit it until I need to use good-names later on in the 
series, but I thought it would look odd to add the defaults at that point.

So it's a minor bit of prescience here.

>> +
>> +[VARIABLES]
>> +
>> +[STRING]
>> +
>> +[SPELLING]
>> +
>> +[FORMAT]
>> +
>> +[SIMILARITIES]
>> +
>> +# Ignore imports when computing similarities.
>> +ignore-imports=yes
> 
> Why?
> 

We don't care if import statements are similar to those in other files. 
It's uninteresting entirely.

(It matches on from typing import ... that exceed four lines, which I do 
regularly by the end of the series.)

>> +
>> +[TYPECHECK]
>> +
>> +[CLASSES]
>> +
>> +[IMPORTS]
>> +
>> +[DESIGN]
>> +
>> +[EXCEPTIONS]
> 
> Looks like you started with output of --generate-rcfile,
> 

I did,

Re: [PATCH 08/37] qapi/common.py: Remove python compatibility workaround

[John Snow]

On 9/16/20 8:34 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/common.py | 5 +----
>>   1 file changed, 1 insertion(+), 4 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index ba35abea47..4fb265a8bf 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -119,10 +119,7 @@ def cgen(code, **kwds):
>>       raw = code % kwds
>>       if indent_level:
>>           indent = genindent(indent_level)
>> -        # re.subn() lacks flags support before Python 2.7, use re.compile()
>> -        raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
>> -                      indent, raw)
>> -        raw = raw[0]
>> +        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
>>       return re.sub(re.escape(eatspace) + r' *', '', raw)
> 
> I missed this one in my "qapi: Bye-bye Python 2" series.
> 
> Can we use re.sub() instead?
> 

Yup.

Re: [PATCH 09/37] qapi/common.py: Add indent manager

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Code style tools really dislike the use of global keywords, because it
> generally involves re-binding the name at runtime which can have strange
> effects depending on when and how that global name is referenced in
> other modules.
>
> Make a little indent level manager instead.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++-------------
>  scripts/qapi/visit.py  |  7 +++---
>  2 files changed, 38 insertions(+), 19 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 4fb265a8bf..87d87b95e5 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -93,33 +93,53 @@ def c_name(name, protect=True):
>  pointer_suffix = ' *' + eatspace
>  
>  
> -def genindent(count):
> -    ret = ''
> -    for _ in range(count):
> -        ret += ' '
> -    return ret
> +class Indent:

Since this class name will be used just once...  Indentation or
IndentationManager?

> +    """
> +    Indent-level management.
>  
> +    :param initial: Initial value, default 0.

The only caller passes 0.

Let's drop the parameter, and hardcode the default initial value until
we have an actual use for another initial value.

> +    """
> +    def __init__(self, initial: int = 0) -> None:
> +        self._level = initial
>  
> -indent_level = 0
> +    def __int__(self) -> int:
> +        """Return the indent as an integer."""

Isn't "as an integer" obvious?

Let's replace "the indent" by "the indentation" globally.

> +        return self._level
>  
> +    def __repr__(self) -> str:
> +        return "{}({:d})".format(type(self).__name__, self._level)

Is __repr__ needed?

>  
> -def push_indent(indent_amount=4):
> -    global indent_level
> -    indent_level += indent_amount
> +    def __str__(self) -> str:
> +        """Return the indent as a string."""
> +        return ' ' * self._level
>  
> +    def __bool__(self) -> bool:
> +        """True when there is a non-zero indent."""
> +        return bool(self._level)

This one lets us shorten

    if int(INDENT):

to

    if INDENT:

There's just one instance.  Marginal.  I'm not rejecting it.

> -def pop_indent(indent_amount=4):
> -    global indent_level
> -    indent_level -= indent_amount
> +    def push(self, amount: int = 4) -> int:
> +        """Push `amount` spaces onto the indent, default four."""
> +        self._level += amount
> +        return self._level
> +
> +    def pop(self, amount: int = 4) -> int:
> +        """Pop `amount` spaces off of the indent, default four."""
> +        if self._level < amount:
> +            raise ArithmeticError(
> +                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
> +        self._level -= amount
> +        return self._level

The push / pop names never made sense.  The functions don't push onto /
pop from a stack, they increment / decrement a counter, and so do the
methods.  Good opportunity to fix the naming.

The @amount parameter has never been used.  I don't mind keeping it.

> +
> +
> +INDENT = Indent(0)

Uh, does this really qualify as a constant?  It looks quite variable to
me...

>  # Generate @code with @kwds interpolated.
>  # Obey indent_level, and strip eatspace.
>  def cgen(code, **kwds):
>      raw = code % kwds
> -    if indent_level:
> -        indent = genindent(indent_level)
> -        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
> +    if INDENT:
> +        raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
>      return re.sub(re.escape(eatspace) + r' *', '', raw)
>  
>  
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index 31bf46e7f7..66ce3dc52e 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -19,8 +19,7 @@
>      gen_endif,
>      gen_if,
>      mcgen,
> -    pop_indent,
> -    push_indent,
> +    INDENT,
>  )
>  from .gen import QAPISchemaModularCVisitor, ifcontext
>  from .schema import QAPISchemaObjectType
> @@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants):
>      if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
>  ''',
>                           name=memb.name, c_name=c_name(memb.name))
> -            push_indent()
> +            INDENT.push()
>          ret += mcgen('''
>      if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
>          return false;
> @@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants):
>                       c_type=memb.type.c_name(), name=memb.name,
>                       c_name=c_name(memb.name))
>          if memb.optional:
> -            pop_indent()
> +            INDENT.pop()
>              ret += mcgen('''
>      }
>  ''')

Re: [PATCH 09/37] qapi/common.py: Add indent manager

[John Snow]

On 9/16/20 11:13 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Code style tools really dislike the use of global keywords, because it
>> generally involves re-binding the name at runtime which can have strange
>> effects depending on when and how that global name is referenced in
>> other modules.
>>
>> Make a little indent level manager instead.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++-------------
>>   scripts/qapi/visit.py  |  7 +++---
>>   2 files changed, 38 insertions(+), 19 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index 4fb265a8bf..87d87b95e5 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -93,33 +93,53 @@ def c_name(name, protect=True):
>>   pointer_suffix = ' *' + eatspace
>>   
>>   
>> -def genindent(count):
>> -    ret = ''
>> -    for _ in range(count):
>> -        ret += ' '
>> -    return ret
>> +class Indent:
> 
> Since this class name will be used just once...  Indentation or
> IndentationManager?
> 

Indentation is fine, if you'd like. IndentationManager makes me feel 
ashamed for writing this patch, like I am punishing myself with Java.

>> +    """
>> +    Indent-level management.
>>   
>> +    :param initial: Initial value, default 0.
> 
> The only caller passes 0.
> 
> Let's drop the parameter, and hardcode the default initial value until
> we have an actual use for another initial value.
> 

The parameter is nice because it gives meaning to the output of repr(), 
see below.

>> +    """
>> +    def __init__(self, initial: int = 0) -> None:
>> +        self._level = initial
>>   
>> -indent_level = 0
>> +    def __int__(self) -> int:
>> +        """Return the indent as an integer."""
> 
> Isn't "as an integer" obvious?

Just a compulsion to write complete sentences that got lost in the sea 
of many patches. I'll nix this one, because dunder methods do not need 
to be documented.

> 
> Let's replace "the indent" by "the indentation" globally.
> 

They're both cromulent as nouns and one is shorter.

I'll switch in good faith; do you prefer "Indentation" as a noun?

>> +        return self._level
>>   
>> +    def __repr__(self) -> str:
>> +        return "{}({:d})".format(type(self).__name__, self._level)
> 
> Is __repr__ needed?
> 

Yes; it's used in the underflow exception , and it is nice when using 
the python shell interactively.

repr(Indent(4)) == "Indent(4)"

>>   
>> -def push_indent(indent_amount=4):
>> -    global indent_level
>> -    indent_level += indent_amount
>> +    def __str__(self) -> str:
>> +        """Return the indent as a string."""
>> +        return ' ' * self._level
>>   
>> +    def __bool__(self) -> bool:
>> +        """True when there is a non-zero indent."""
>> +        return bool(self._level)
> 
> This one lets us shorten
> 
>      if int(INDENT):
> 
> to
> 
>      if INDENT:
> 
> There's just one instance.  Marginal.  I'm not rejecting it.
> 

Yep, it's trivial in either direction. Still, because it's a custom 
type, I thought I'd be courteous and have it support bool().

>> -def pop_indent(indent_amount=4):
>> -    global indent_level
>> -    indent_level -= indent_amount
>> +    def push(self, amount: int = 4) -> int:
>> +        """Push `amount` spaces onto the indent, default four."""
>> +        self._level += amount
>> +        return self._level
>> +
>> +    def pop(self, amount: int = 4) -> int:
>> +        """Pop `amount` spaces off of the indent, default four."""
>> +        if self._level < amount:
>> +            raise ArithmeticError(
>> +                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
>> +        self._level -= amount
>> +        return self._level
> 
> The push / pop names never made sense.  The functions don't push onto /
> pop from a stack, they increment / decrement a counter, and so do the
> methods.  Good opportunity to fix the naming.
> 

OK.

I briefly thought about using __isub__ and __iadd__ to support e.g. 
indent += 4, but actually it'd be annoying to have to specify 4 everywhere.

Since you didn't suggest anything, I am going to change it to 'increase' 
and 'decrease'.

> The @amount parameter has never been used.  I don't mind keeping it.
> 
I'll keep it, because I like having the default amount show up in the 
signature instead of as a magic constant in the function body.

>> +
>> +
>> +INDENT = Indent(0)
> 
> Uh, does this really qualify as a constant?  It looks quite variable to
> me...
> 

Ever make a mistake? I thought I did once, but I was mistaken.

>>   # Generate @code with @kwds interpolated.
>>   # Obey indent_level, and strip eatspace.
>>   def cgen(code, **kwds):
>>       raw = code % kwds
>> -    if indent_level:
>> -        indent = genindent(indent_level)
>> -        raw, _ = re.subn(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
>> +    if INDENT:
>> +        raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
>>       return re.sub(re.escape(eatspace) + r' *', '', raw)
>>   
>>   
>> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
>> index 31bf46e7f7..66ce3dc52e 100644
>> --- a/scripts/qapi/visit.py
>> +++ b/scripts/qapi/visit.py
>> @@ -19,8 +19,7 @@
>>       gen_endif,
>>       gen_if,
>>       mcgen,
>> -    pop_indent,
>> -    push_indent,
>> +    INDENT,
>>   )
>>   from .gen import QAPISchemaModularCVisitor, ifcontext
>>   from .schema import QAPISchemaObjectType
>> @@ -68,7 +67,7 @@ def gen_visit_object_members(name, base, members, variants):
>>       if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
>>   ''',
>>                            name=memb.name, c_name=c_name(memb.name))
>> -            push_indent()
>> +            INDENT.push()
>>           ret += mcgen('''
>>       if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
>>           return false;
>> @@ -77,7 +76,7 @@ def gen_visit_object_members(name, base, members, variants):
>>                        c_type=memb.type.c_name(), name=memb.name,
>>                        c_name=c_name(memb.name))
>>           if memb.optional:
>> -            pop_indent()
>> +            INDENT.pop()
>>               ret += mcgen('''
>>       }
>>   ''')

Re: [PATCH 00/37] qapi: static typing conversion, pt1

[John Snow]

On 9/15/20 6:39 PM, John Snow wrote:
> Hi, this series starts adding static type hints to the QAPI module. As
> you can see, the series started getting quite a bit long, so this is
> only a partial conversion that focuses on a handful of the easier files.
> 

Updated series with current feedback and pushed to gitlab. Awaiting 
feedback on patches 10-37 before re-spinning.

https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/

--js

Re: [PATCH 04/37] qapi: move generator entrypoint into module

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/16/20 7:54 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> As part of delinting and adding type hints to the QAPI generator, it's
>>> helpful for the entrypoint to be part of the module, only leaving a very
>>> tiny entrypoint shim outside of the module.
>>>
>>> As a result, all of the include statements are reworked to be module-aware,
>>> as explicit relative imports.
>> Should this be split into two commits, one for each of the
>> paragraphs
>> above?
>> 
>
> Hmm ... I hadn't considered it was possible, but actually ... I guess
> I can split those out, yeah.
>
>> PEP 8 recommends absolute imports, with one exception:
>>      However, explicit relative imports are an acceptable
>> alternative to
>>      absolute imports, especially when dealing with complex package
>>      layouts where using absolute imports would be unnecessarily verbose:
>>          from . import sibling
>>          from .sibling import example
>>      Standard library code should avoid complex package layouts and
>>      always use absolute imports.
>> Do you think it covers your use of relative imports?
>> 
>
> Admittedly I am going by trial and error; my experience is that the
> relative imports behave the nicest.
>
> There is a historical fear of explicit relative imports because they
> are "new" and years of Python2 compatibility

Having to struggle with that for years was bound to damage brains.

>                                              rendered many afraid of
> them. It is advice safely ignored in my opinion.
>
> Using absolute imports (e.g. from qapi.sibling import foo) gets messy
> in virtual environments when you have *installed* the module in
> question: it becomes ambiguous as to *which* qapi you meant: the one
> in this folder, or the one installed to the environment?
>
> Python, mypy, pylint, flake8 etc disagree sometimes, or can get
> confused; thinking there are two copies of each module.
>
> Just my experience that relative imports seem to give me the least trouble.
>
>>> This is done primarily for the benefit of python tooling (pylint, mypy,
>>> flake8, et al) which otherwise has trouble consistently resolving
>>> "qapi.x" to mean "a sibling file in this folder."
>> Can you give me an example of some tool having troube?
>> 
>
> I'd have to code up some examples. I have some hobby code unrelated to
> QEMU where I struggled to get flake8, mypy, and pylint all cooperating 
> with an import regime until I gave up and used explicit relative imports.

I'd make the switch when we actually do run into trouble.

But I'm willing to take your advice and switch now.

[...]

Re: [PATCH 06/37] qapi: delint using flake8

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/16/20 8:12 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Petty style guide fixes and line length enforcement.  Not a big win, not
>>> a big loss, but flake8 passes 100% on the qapi module, which gives us an
>>> easy baseline to enforce hereafter.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>   scripts/qapi/.flake8     |  2 ++
>>>   scripts/qapi/commands.py |  3 ++-
>>>   scripts/qapi/schema.py   |  8 +++++---
>>>   scripts/qapi/visit.py    | 15 ++++++++++-----
>>>   4 files changed, 19 insertions(+), 9 deletions(-)
>>>   create mode 100644 scripts/qapi/.flake8
>>>
>>> diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
>>> new file mode 100644
>>> index 0000000000..45d8146f3f
>>> --- /dev/null
>>> +++ b/scripts/qapi/.flake8
>>> @@ -0,0 +1,2 @@
>>> +[flake8]
>>> +extend-ignore = E722  # Pylint handles this, but smarter.
>> I guess you mean pylint's W0702 a.k.a. bare-except.  What's wrong
>> with
>> flake8's E722 compared to pylint's W0702?
>> 
>
> Flake8 will warn on *any* bare except, but Pylint's is context-aware
> and will suppress the warning if you re-raise the exception.

Should this information be worked into the comment?

> I don't actually think this comes up in the qapi code base, but it
> does come up in the ./python/qemu code base.
>
> (One of my goals is unifying the lint checking regime for both packages.)
>
>>> \ No newline at end of file
>> So put one there :)
>> 
>
> Whupps, okay.
>
>>> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
>>> index e1df0e341f..2e4b4de0fa 100644
>>> --- a/scripts/qapi/commands.py
>>> +++ b/scripts/qapi/commands.py
>>> @@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
>>>   def gen_marshal_output(ret_type):
>>>       return mcgen('''
>>>   -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
>>> QObject **ret_out, Error **errp)
>>> +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
>>> +                                          Error **errp)
>> The continued parameter list may become misalignd in generated C.
>> E.g.
>>      static void qmp_marshal_output_BlockInfoList(BlockInfoList
>> *ret_in, QObject **ret_out,
>>                                                Error **errp)
>>      {
>>      ...
>>      }
>> Do we care?
>> 
>
> Yeah, I don't know. Do we?

I care, but I also care for automated style checks.

> It actually seemed more annoying to try and get flake8 to make an
> exception for these handful of examples.
>
> Path of least resistance led me here, but I can try and appease both
> systems if you'd prefer.

Up to now, I ran the style checkers manually, and this was just one of
several complaints to ignore, so I left the code alone.

If it gets in the way of running them automatically, and messing up the
generated code slightly is the easiest way to get it out of the way,
then I can accept the slight mess.

>> More of the same below.
>> 
>>>   {
>>>       Visitor *v;
>>>   
>> [...]
>> 

Re: [PATCH 07/37] qapi: add pylintrc

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/16/20 8:30 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Add a skeleton pylintrc file. Right now, it ignores quite a few things.
>>> Files will be removed from the known-bad list throughout this and
>>> following series as they are repaired.
>>>
>>> Note: Normally, pylintrc would go in the folder above the module, but as
>>> that folder is shared by many things, it is going inside the module
>>> folder now.
>>>
>>> Due to some bugs in different versions of pylint (2.5.x), pylint does
>>> not correctly recognize when it is being run from "inside" a module, and
>>> must be run *outside* of the module.
>>>
>>> Therefore, to run it, you must:
>>>
>>>   > cd :/qemu/scripts
>> -bash: cd: :/qemu/scripts: No such file or directory
>> ;-P
>> 
>>>   > pylint qapi/ --rcfile=qapi/pylintrc
>> Why not
>>     $ pylint scripts/qapi --rcfile=scripts/qapi/pylintrc
>> 
>
> No reason I'm aware of, I have just been testing with CWD at the
> scripts dir myself because of how python imports work.
>
> If it works this way, enjoy!

It works, and it simplifies the commmit message.

>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>   scripts/qapi/pylintrc | 74 +++++++++++++++++++++++++++++++++++++++++++
>>>   1 file changed, 74 insertions(+)
>>>   create mode 100644 scripts/qapi/pylintrc
>>>
>>> diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
>>> new file mode 100644
>>> index 0000000000..c2bbb8e8e1
>>> --- /dev/null
>>> +++ b/scripts/qapi/pylintrc
>>> @@ -0,0 +1,74 @@
>>> +[MASTER]
>>> +
>>> +# Add files or directories matching the regex patterns to the blacklist. The
>>> +# regex matches against base names, not paths.
>>> +ignore-patterns=common.py,
>>> +                doc.py,
>>> +                error.py,
>>> +                expr.py,
>>> +                gen.py,
>>> +                parser.py,
>>> +                schema.py,
>>> +                source.py,
>>> +                types.py,
>>> +                visit.py,
>> Already not ignored:
>>      __init__.py
>>      commands.py
>>      common.py
>>      debug.py
>>      events.py
>>      introspect.py
>>      script.py
>> Okay.
>> 
>>> +
>>> +
>>> +[MESSAGES CONTROL]
>>> +
>>> +# Disable the message, report, category or checker with the given id(s). You
>>> +# can either give multiple identifiers separated by comma (,) or put this
>>> +# option multiple times (only on the command line, not in the configuration
>>> +# file where it should appear only once). You can also use "--disable=all" to
>>> +# disable everything first and then reenable specific checks. For example, if
>>> +# you want to run only the similarities checker, you can use "--disable=all
>>> +# --enable=similarities". If you want to run only the classes checker, but have
>>> +# no Warning level messages displayed, use "--disable=all --enable=classes
>>> +# --disable=W".
>>> +disable=fixme,
>>> +        missing-docstring,
>>> +        too-many-arguments,
>>> +        too-many-branches,
>>> +        too-many-statements,
>>> +        too-many-instance-attributes,
>> I'm fine with disabling these.
>> 
>
> I'd like to enable missing-docstring eventually, but that's not for today.

Understood.

>>> +
>>> +[REPORTS]
>>> +
>>> +[REFACTORING]
>>> +
>>> +[MISCELLANEOUS]
>>> +
>>> +[LOGGING]
>>> +
>>> +[BASIC]
>>> +
>>> +# Good variable names which should always be accepted, separated by a comma.
>>> +good-names=i,
>>> +           j,
>>> +           k,
>>> +           ex,
>>> +           Run,
>>> +           _
>> Isn't this the default?
>> 
>
> Yes. I could omit it until I need to use good-names later on in the
> series, but I thought it would look odd to add the defaults at that
> point.
>
> So it's a minor bit of prescience here.

Matter of taste.  No objection.

>>> +
>>> +[VARIABLES]
>>> +
>>> +[STRING]
>>> +
>>> +[SPELLING]
>>> +
>>> +[FORMAT]
>>> +
>>> +[SIMILARITIES]
>>> +
>>> +# Ignore imports when computing similarities.
>>> +ignore-imports=yes
>> Why?
>> 
>
> We don't care if import statements are similar to those in other
> files. It's uninteresting entirely.
>
> (It matches on from typing import ... that exceed four lines, which I
> do regularly by the end of the series.)

What about something like

     # Ignore imports when computing similarities, because import
     # statements being similar is uninteresting entirely

>>> +
>>> +[TYPECHECK]
>>> +
>>> +[CLASSES]
>>> +
>>> +[IMPORTS]
>>> +
>>> +[DESIGN]
>>> +
>>> +[EXCEPTIONS]
>> Looks like you started with output of --generate-rcfile,
>> 
>
> I did,

Let's mention that in the commit message.  Education opportunity :)

Re: [PATCH 09/37] qapi/common.py: Add indent manager

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/16/20 11:13 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Code style tools really dislike the use of global keywords, because it
>>> generally involves re-binding the name at runtime which can have strange
>>> effects depending on when and how that global name is referenced in
>>> other modules.
>>>
>>> Make a little indent level manager instead.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>   scripts/qapi/common.py | 50 +++++++++++++++++++++++++++++-------------
>>>   scripts/qapi/visit.py  |  7 +++---
>>>   2 files changed, 38 insertions(+), 19 deletions(-)
>>>
>>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>>> index 4fb265a8bf..87d87b95e5 100644
>>> --- a/scripts/qapi/common.py
>>> +++ b/scripts/qapi/common.py
>>> @@ -93,33 +93,53 @@ def c_name(name, protect=True):
>>>   pointer_suffix = ' *' + eatspace
>>>     
>>> -def genindent(count):
>>> -    ret = ''
>>> -    for _ in range(count):
>>> -        ret += ' '
>>> -    return ret
>>> +class Indent:
>> Since this class name will be used just once...  Indentation or
>> IndentationManager?
>> 
>
> Indentation is fine, if you'd like. IndentationManager makes me feel
> ashamed for writing this patch, like I am punishing myself with Java.

Hah!  Point taken.

>>> +    """
>>> +    Indent-level management.
>>>   +    :param initial: Initial value, default 0.
>> The only caller passes 0.
>> Let's drop the parameter, and hardcode the default initial value
>> until
>> we have an actual use for another initial value.
>> 
>
> The parameter is nice because it gives meaning to the output of
> repr(), see below.
>
>>> +    """
>>> +    def __init__(self, initial: int = 0) -> None:
>>> +        self._level = initial
>>>   -indent_level = 0
>>> +    def __int__(self) -> int:
>>> +        """Return the indent as an integer."""
>> Isn't "as an integer" obvious?
>
> Just a compulsion to write complete sentences that got lost in the sea
> of many patches. I'll nix this one, because dunder methods do not need 
> to be documented.
>
>> Let's replace "the indent" by "the indentation" globally.
>> 
>
> They're both cromulent as nouns and one is shorter.
>
> I'll switch in good faith; do you prefer "Indentation" as a noun?

Use of "indent" as a noun was new to me, but what do I know; you're the
native speaker.  Use your judgement.  Applies to the class name, too.

>>> +        return self._level
>>>   +    def __repr__(self) -> str:
>>> +        return "{}({:d})".format(type(self).__name__, self._level)
>> Is __repr__ needed?
>> 
>
> Yes; it's used in the underflow exception , and it is nice when using
> the python shell interactively.
>
> repr(Indent(4)) == "Indent(4)"

Meh.  There's another three dozen classes for you to put lipstick on :)

>>>   -def push_indent(indent_amount=4):
>>> -    global indent_level
>>> -    indent_level += indent_amount
>>> +    def __str__(self) -> str:
>>> +        """Return the indent as a string."""
>>> +        return ' ' * self._level
>>>   +    def __bool__(self) -> bool:
>>> +        """True when there is a non-zero indent."""
>>> +        return bool(self._level)
>> This one lets us shorten
>>      if int(INDENT):
>> to
>>      if INDENT:
>> There's just one instance.  Marginal.  I'm not rejecting it.
>> 
>
> Yep, it's trivial in either direction. Still, because it's a custom
> type, I thought I'd be courteous and have it support bool().
>
>>> -def pop_indent(indent_amount=4):
>>> -    global indent_level
>>> -    indent_level -= indent_amount
>>> +    def push(self, amount: int = 4) -> int:
>>> +        """Push `amount` spaces onto the indent, default four."""
>>> +        self._level += amount
>>> +        return self._level
>>> +
>>> +    def pop(self, amount: int = 4) -> int:
>>> +        """Pop `amount` spaces off of the indent, default four."""
>>> +        if self._level < amount:
>>> +            raise ArithmeticError(
>>> +                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))

I think assert(amount <= self._level) would do just fine.

>>> +        self._level -= amount
>>> +        return self._level
>> The push / pop names never made sense.  The functions don't push
>> onto /
>> pop from a stack, they increment / decrement a counter, and so do the
>> methods.  Good opportunity to fix the naming.
>> 
>
> OK.
>
> I briefly thought about using __isub__ and __iadd__ to support
> e.g. indent += 4, but actually it'd be annoying to have to specify 4
> everywhere.
>
> Since you didn't suggest anything, I am going to change it to
> 'increase' and 'decrease'.

Works for me.  So would inc and dec.

>> The @amount parameter has never been used.  I don't mind keeping it.
>> 
> I'll keep it, because I like having the default amount show up in the
> signature instead of as a magic constant in the function body.
>
>>> +
>>> +
>>> +INDENT = Indent(0)
>> Uh, does this really qualify as a constant?  It looks quite variable
>> to
>> me...
>> 
>
> Ever make a mistake? I thought I did once, but I was mistaken.

"If I had any humility, I'd be perfect!"

Re: [PATCH 10/37] qapi/common.py: delint with pylint

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/common.py | 16 +++++++---------
>  scripts/qapi/schema.py | 14 +++++++-------
>  2 files changed, 14 insertions(+), 16 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 87d87b95e5..c665e67495 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -14,6 +14,11 @@
>  import re
>  
>  
> +EATSPACE = '\033EATSPACE.'
> +POINTER_SUFFIX = ' *' + EATSPACE
> +c_name_trans = str.maketrans('.-', '__')
> +
> +

You rename and move.  pylint gripes about the names, but it doesn't
actually ask for the move, as far as I can tell.  Can you explain why
you move?

>  # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>  # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>  # ENUM24_Name -> ENUM24_NAME
> @@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
>      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>  
>  
> -c_name_trans = str.maketrans('.-', '__')
> -
> -
>  # Map @name to a valid C identifier.
>  # If @protect, avoid returning certain ticklish identifiers (like
>  # C keywords) by prepending 'q_'.
> @@ -89,10 +91,6 @@ def c_name(name, protect=True):
>      return name
>  
>  
> -eatspace = '\033EATSPACE.'
> -pointer_suffix = ' *' + eatspace
> -
> -
>  class Indent:
>      """
>      Indent-level management.
> @@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
>  
>  
>  # Generate @code with @kwds interpolated.
> -# Obey indent_level, and strip eatspace.
> +# Obey INDENT level, and strip EATSPACE.

Is the change to INDENT intentional?

>  def cgen(code, **kwds):
>      raw = code % kwds
>      if INDENT:
>          raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
> -    return re.sub(re.escape(eatspace) + r' *', '', raw)
> +    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>  
>  
>  def mcgen(code, **kwds):
[...]

Re: [PATCH 11/37] qapi/common.py: Replace one-letter 'c' variable

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/common.py | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index c665e67495..4c079755d3 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -30,14 +30,14 @@ def camel_to_upper(value):
>      new_name = ''
>      length = len(c_fun_str)
>      for i in range(length):
> -        c = c_fun_str[i]
> -        # When c is upper and no '_' appears before, do more checks
> -        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
> +        char = c_fun_str[i]
> +        # When char is upper and no '_' appears before, do more checks

Good opportunity to tweak the language: "When char is upper case".

> +        if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
>              if i < length - 1 and c_fun_str[i + 1].islower():
>                  new_name += '_'
>              elif c_fun_str[i - 1].isdigit():
>                  new_name += '_'
> -        new_name += c
> +        new_name += char
>      return new_name.lstrip('_').upper()

Re: [PATCH 13/37] qapi/common.py: add notational type hints

[Markus Armbruster]

Question on the subject line: what makes a type hint notational?

John Snow <jsnow@redhat.com> writes:

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/common.py | 27 ++++++++++++++++-----------
>  1 file changed, 16 insertions(+), 11 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 4c079755d3..af01348b35 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -12,6 +12,7 @@
>  # See the COPYING file in the top-level directory.
>  
>  import re
> +from typing import Optional, Union, Sequence
>  
>  
>  EATSPACE = '\033EATSPACE.'
> @@ -22,7 +23,7 @@
>  # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>  # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>  # ENUM24_Name -> ENUM24_NAME
> -def camel_to_upper(value):
> +def camel_to_upper(value: str) -> str:
>      c_fun_str = c_name(value, False)
>      if value.isupper():
>          return c_fun_str
> @@ -41,7 +42,9 @@ def camel_to_upper(value):
>      return new_name.lstrip('_').upper()
>  
>  
> -def c_enum_const(type_name, const_name, prefix=None):
> +def c_enum_const(type_name: str,
> +                 const_name: str,
> +                 prefix: Optional[str] = None) -> str:
>      if prefix is not None:
>          type_name = prefix
>      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
> @@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
>  # into substrings of a generated C function name.
>  # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>  # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
> -def c_name(name, protect=True):
> +def c_name(name: str, protect: bool = True) -> str:
>      # ANSI X3J11/88-090, 3.1.1
>      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>                       'default', 'do', 'double', 'else', 'enum', 'extern',
> @@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
>  
>  # Generate @code with @kwds interpolated.
>  # Obey INDENT level, and strip EATSPACE.
> -def cgen(code, **kwds):
> +def cgen(code: str, **kwds: Union[str, int]) -> str:

Hmm.

The @kwds values can be anything, provided they match the conversion
specifiers in @code:

>      raw = code % kwds

Your type hint adds a restriction that wasn't there before.

Is there a better way?

>      if INDENT:
>          raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
>      return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>  
>  
> -def mcgen(code, **kwds):
> +def mcgen(code: str, **kwds: Union[str, int]) -> str:

Likewise.

>      if code[0] == '\n':
>          code = code[1:]
>      return cgen(code, **kwds)
>  
>  
> -def c_fname(filename):
> +def c_fname(filename: str) -> str:
>      return re.sub(r'[^A-Za-z0-9_]', '_', filename)
>  
>  
> -def guardstart(name):
> +def guardstart(name: str) -> str:
>      return mcgen('''
>  #ifndef %(name)s
>  #define %(name)s
> @@ -160,7 +163,7 @@ def guardstart(name):
>                   name=c_fname(name).upper())
>  
>  
> -def guardend(name):
> +def guardend(name: str) -> str:
>      return mcgen('''
>  
>  #endif /* %(name)s */
> @@ -168,7 +171,7 @@ def guardend(name):
>                   name=c_fname(name).upper())
>  
>  
> -def gen_if(ifcond):
> +def gen_if(ifcond: Sequence[str]) -> str:
>      ret = ''
>      for ifc in ifcond:
>          ret += mcgen('''
> @@ -177,7 +180,7 @@ def gen_if(ifcond):
>      return ret
>  
>  
> -def gen_endif(ifcond):
> +def gen_endif(ifcond: Sequence[str]) -> str:
>      ret = ''
>      for ifc in reversed(ifcond):
>          ret += mcgen('''
> @@ -186,7 +189,9 @@ def gen_endif(ifcond):
>      return ret
>  
>  
> -def build_params(arg_type, boxed, extra=None):
> +def build_params(arg_type,
> +                 boxed: bool,
> +                 extra: Optional[str] = None) -> str:
>      ret = ''
>      sep = ''
>      if boxed:

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> As docstrings, they'll show up in documentation and IDE help.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------
>  1 file changed, 36 insertions(+), 14 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index af01348b35..38d380f0a9 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -20,10 +20,18 @@
>  c_name_trans = str.maketrans('.-', '__')
>  
>  
> -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
> -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
> -# ENUM24_Name -> ENUM24_NAME
>  def camel_to_upper(value: str) -> str:
> +    """
> +    Converts CamelCase to CAMEL_CASE.
> +
> +    Examples:
> +      ENUMName -> ENUM_NAME
> +      EnumName1 -> ENUM_NAME1
> +      ENUM_NAME -> ENUM_NAME
> +      ENUM_NAME1 -> ENUM_NAME1
> +      ENUM_Name2 -> ENUM_NAME2
> +      ENUM24_Name -> ENUM24_NAME
> +    """
>      c_fun_str = c_name(value, False)
>      if value.isupper():
>          return c_fun_str
> @@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str:
>  def c_enum_const(type_name: str,
>                   const_name: str,
>                   prefix: Optional[str] = None) -> str:
> +    """
> +    Generate a C enumeration constant name.
> +
> +    :param type_name: The name of the enumeration.
> +    :param const_name: The name of this constant.
> +    :param prefix: Optional, prefix that overrides the type_name.
> +    """

Not actually a move.  Suggest to retitle

    qapi/common: Turn comments in docstrings, and add more

>      if prefix is not None:
>          type_name = prefix
>      return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>  
>  
> -# Map @name to a valid C identifier.
> -# If @protect, avoid returning certain ticklish identifiers (like
> -# C keywords) by prepending 'q_'.
> -#
> -# Used for converting 'name' from a 'name':'type' qapi definition
> -# into a generated struct member, as well as converting type names
> -# into substrings of a generated C function name.
> -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
> -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>  def c_name(name: str, protect: bool = True) -> str:
> +    """
> +    Map `name` to a valid C identifier.
> +
> +    Used for converting 'name' from a 'name':'type' qapi definition
> +    into a generated struct member, as well as converting type names
> +    into substrings of a generated C function name.
> +
> +    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
> +    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
> +
> +    :param name: The name to map.
> +    :param protect: If true, avoid returning certain ticklish identifiers
> +                    (like C keywords) by prepending 'q_'.
> +    """
>      # ANSI X3J11/88-090, 3.1.1
>      c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>                       'default', 'do', 'double', 'else', 'enum', 'extern',
> @@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int:
>  INDENT = Indent(0)
>  
>  
> -# Generate @code with @kwds interpolated.
> -# Obey INDENT level, and strip EATSPACE.
>  def cgen(code: str, **kwds: Union[str, int]) -> str:
> +    """
> +    Generate `code` with `kwds` interpolated.
> +    Obey `INDENT`, and strip `EATSPACE`.
> +    """
>      raw = code % kwds
>      if INDENT:
>          raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)

Can you point to documentation on the docstring conventions and markup
to use?

Re: [PATCH 15/37] qapi/common.py: split build_params into new file

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Including it in common.py creates a circular import dependency, because
> schema relies on common.py. To type build_params properly, it needs to
> be moved outside of the chain.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/commands.py |  2 +-
>  scripts/qapi/common.py   | 23 -----------------------
>  scripts/qapi/events.py   |  2 +-
>  scripts/qapi/params.py   | 40 ++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 42 insertions(+), 25 deletions(-)
>  create mode 100644 scripts/qapi/params.py

Ugh.

Would moving it gen.py work?

Re: [PATCH 06/37] qapi: delint using flake8

[John Snow]

On 9/17/20 3:54 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/16/20 8:12 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> Petty style guide fixes and line length enforcement.  Not a big win, not
>>>> a big loss, but flake8 passes 100% on the qapi module, which gives us an
>>>> easy baseline to enforce hereafter.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>>    scripts/qapi/.flake8     |  2 ++
>>>>    scripts/qapi/commands.py |  3 ++-
>>>>    scripts/qapi/schema.py   |  8 +++++---
>>>>    scripts/qapi/visit.py    | 15 ++++++++++-----
>>>>    4 files changed, 19 insertions(+), 9 deletions(-)
>>>>    create mode 100644 scripts/qapi/.flake8
>>>>
>>>> diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
>>>> new file mode 100644
>>>> index 0000000000..45d8146f3f
>>>> --- /dev/null
>>>> +++ b/scripts/qapi/.flake8
>>>> @@ -0,0 +1,2 @@
>>>> +[flake8]
>>>> +extend-ignore = E722  # Pylint handles this, but smarter.
>>> I guess you mean pylint's W0702 a.k.a. bare-except.  What's wrong
>>> with
>>> flake8's E722 compared to pylint's W0702?
>>>
>>
>> Flake8 will warn on *any* bare except, but Pylint's is context-aware
>> and will suppress the warning if you re-raise the exception.
> 
> Should this information be worked into the comment?
> 

I'll improve it a little, but I'll add what I wrote above to the commit 
message.

>> I don't actually think this comes up in the qapi code base, but it
>> does come up in the ./python/qemu code base.
>>
>> (One of my goals is unifying the lint checking regime for both packages.)
>>
>>>> \ No newline at end of file
>>> So put one there :)
>>>
>>
>> Whupps, okay.
>>
>>>> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
>>>> index e1df0e341f..2e4b4de0fa 100644
>>>> --- a/scripts/qapi/commands.py
>>>> +++ b/scripts/qapi/commands.py
>>>> @@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
>>>>    def gen_marshal_output(ret_type):
>>>>        return mcgen('''
>>>>    -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
>>>> QObject **ret_out, Error **errp)
>>>> +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
>>>> +                                          Error **errp)
>>> The continued parameter list may become misalignd in generated C.
>>> E.g.
>>>       static void qmp_marshal_output_BlockInfoList(BlockInfoList
>>> *ret_in, QObject **ret_out,
>>>                                                 Error **errp)
>>>       {
>>>       ...
>>>       }
>>> Do we care?
>>>
>>
>> Yeah, I don't know. Do we?
> 
> I care, but I also care for automated style checks.
> 
>> It actually seemed more annoying to try and get flake8 to make an
>> exception for these handful of examples.
>>
>> Path of least resistance led me here, but I can try and appease both
>> systems if you'd prefer.
> 
> Up to now, I ran the style checkers manually, and this was just one of
> several complaints to ignore, so I left the code alone.
> 
> If it gets in the way of running them automatically, and messing up the
> generated code slightly is the easiest way to get it out of the way,
> then I can accept the slight mess.
> 

I changed this a little to put all the args on the next line, which is 
slightly unusual but works okay.

I think that's a fine middle ground, because the alternative (to me) is 
to start using abstracted code generation tokens in a tree structure, 
etc etc etc.

Embedded templates are always gonna look kinda nasty, I think, because 
you're trying to fight style guidelines in two languages simultaneously 
and it's never gonna quite work out exactly how you want without some 
pretty complex abstraction mechanisms that are well beyond the power we 
need right now.

>>> More of the same below.
>>>
>>>>    {
>>>>        Visitor *v;
>>>>    
>>> [...]
>>>

Re: [PATCH 07/37] qapi: add pylintrc

[John Snow]

On 9/17/20 3:58 AM, Markus Armbruster wrote:
>> We don't care if import statements are similar to those in other
>> files. It's uninteresting entirely.
>>
>> (It matches on from typing import ... that exceed four lines, which I
>> do regularly by the end of the series.)
> What about something like
> 
>       # Ignore imports when computing similarities, because import
>       # statements being similar is uninteresting entirely
> 

OK. I left the "default" comments that pylint itself wrote. I'm now writing:

"Ignore import statements themselves when computing similarities."

Re: [PATCH 09/37] qapi/common.py: Add indent manager

[John Snow]

On 9/17/20 4:07 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/16/20 11:13 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>


>>> Let's replace "the indent" by "the indentation" globally.
>>>
>>
>> They're both cromulent as nouns and one is shorter.
>>
>> I'll switch in good faith; do you prefer "Indentation" as a noun?
> 
> Use of "indent" as a noun was new to me, but what do I know; you're the
> native speaker.  Use your judgement.  Applies to the class name, too.
> 

I made the change; see gitlab commits or wait for v2 to see if it reads 
better to you.

>>>> +        return self._level
>>>>    +    def __repr__(self) -> str:
>>>> +        return "{}({:d})".format(type(self).__name__, self._level)
>>> Is __repr__ needed?
>>>
>>
>> Yes; it's used in the underflow exception , and it is nice when using
>> the python shell interactively.
>>
>> repr(Indent(4)) == "Indent(4)"
> 
> Meh.  There's another three dozen classes for you to put lipstick on :)
> 

We'll get to them in due time. For now, please admire the lipstick.


>>>> +    def pop(self, amount: int = 4) -> int:
>>>> +        """Pop `amount` spaces off of the indent, default four."""
>>>> +        if self._level < amount:
>>>> +            raise ArithmeticError(
>>>> +                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
> 
> I think assert(amount <= self._level) would do just fine.
> 

Secretly, asserts can be disabled in Python just like they can in C code.

My habit: if it's something that should already be true given the nature 
of how the code is laid out, use an assertion. If I am preventing an 
erroneous state (Especially from callers in other modules), explicitly 
raise an exception.

(See the gitlab branch for the updated version of this patch, which has 
changed the code slightly.)

>>>> +        self._level -= amount
>>>> +        return self._level
>>> The push / pop names never made sense.  The functions don't push
>>> onto /
>>> pop from a stack, they increment / decrement a counter, and so do the
>>> methods.  Good opportunity to fix the naming.
>>>
>>
>> OK.
>>
>> I briefly thought about using __isub__ and __iadd__ to support
>> e.g. indent += 4, but actually it'd be annoying to have to specify 4
>> everywhere.
>>
>> Since you didn't suggest anything, I am going to change it to
>> 'increase' and 'decrease'.
> 
> Works for me.  So would inc and dec.
> 

increase and decrease it is.

>>> The @amount parameter has never been used.  I don't mind keeping it.
>>>
>> I'll keep it, because I like having the default amount show up in the
>> signature instead of as a magic constant in the function body.
>>
>>>> +
>>>> +
>>>> +INDENT = Indent(0)
>>> Uh, does this really qualify as a constant?  It looks quite variable
>>> to
>>> me...
>>>
>>
>> Ever make a mistake? I thought I did once, but I was mistaken.
> 
> "If I had any humility, I'd be perfect!"
> 

:)

Re: [PATCH 10/37] qapi/common.py: delint with pylint

[John Snow]

On 9/17/20 10:15 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/common.py | 16 +++++++---------
>>   scripts/qapi/schema.py | 14 +++++++-------
>>   2 files changed, 14 insertions(+), 16 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index 87d87b95e5..c665e67495 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -14,6 +14,11 @@
>>   import re
>>   
>>   
>> +EATSPACE = '\033EATSPACE.'
>> +POINTER_SUFFIX = ' *' + EATSPACE
>> +c_name_trans = str.maketrans('.-', '__')
>> +
>> +
> 
> You rename and move.  pylint gripes about the names, but it doesn't
> actually ask for the move, as far as I can tell.  Can you explain why
> you move?
> 

Preference. I like constants and globals at the top so you can audit any 
code that runs at import time in one place. Since they are externally 
visible objects, having them near other "header" style information makes 
sense to me.

>>   # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>>   # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>>   # ENUM24_Name -> ENUM24_NAME
>> @@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
>>       return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>>   
>>   
>> -c_name_trans = str.maketrans('.-', '__')

(This one winds up being a constant, so I renamed it in my v2.)

>> -
>> -
>>   # Map @name to a valid C identifier.
>>   # If @protect, avoid returning certain ticklish identifiers (like
>>   # C keywords) by prepending 'q_'.
>> @@ -89,10 +91,6 @@ def c_name(name, protect=True):
>>       return name
>>   
>>   
>> -eatspace = '\033EATSPACE.'
>> -pointer_suffix = ' *' + eatspace
>> -
>> -
>>   class Indent:
>>       """
>>       Indent-level management.
>> @@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
>>   
>>   
>>   # Generate @code with @kwds interpolated.
>> -# Obey indent_level, and strip eatspace.
>> +# Obey INDENT level, and strip EATSPACE.
> 
> Is the change to INDENT intentional?
> 

Kind of, but it's either late (should have been with the indent manager 
patch) or early (Should be with the patch that moves comments into 
docstrings.)

When this comment becomes a docstring, I use `INDENT` to indicate it as 
a proper object. This in and of itself is prescient, as we are not using 
sphinx/apidoc to generate any documentation about the QAPI package yet.

(The pending v2 uses `indent` after you pointed out that it was not a 
constant.)

>>   def cgen(code, **kwds):
>>       raw = code % kwds
>>       if INDENT:
>>           raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
>> -    return re.sub(re.escape(eatspace) + r' *', '', raw)
>> +    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>>   
>>   
>>   def mcgen(code, **kwds):
> [...]
> 

Re: [PATCH 11/37] qapi/common.py: Replace one-letter 'c' variable

[John Snow]

On 9/17/20 10:17 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/common.py | 8 ++++----
>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index c665e67495..4c079755d3 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -30,14 +30,14 @@ def camel_to_upper(value):
>>       new_name = ''
>>       length = len(c_fun_str)
>>       for i in range(length):
>> -        c = c_fun_str[i]
>> -        # When c is upper and no '_' appears before, do more checks
>> -        if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
>> +        char = c_fun_str[i]
>> +        # When char is upper and no '_' appears before, do more checks
> 
> Good opportunity to tweak the language: "When char is upper case".
> 

ACK, change made.

Re: [PATCH 13/37] qapi/common.py: add notational type hints

[John Snow]

On 9/17/20 10:32 AM, Markus Armbruster wrote:
> Question on the subject line: what makes a type hint notational?
> 

My cover letter explains that every time I use this phrase, I mean to 
state that "This patch adds exclusively type notations and makes no 
functional changes to the runtime operation whatsoever."

i.e. notations-only.

> John Snow <jsnow@redhat.com> writes:
> 
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/common.py | 27 ++++++++++++++++-----------
>>   1 file changed, 16 insertions(+), 11 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index 4c079755d3..af01348b35 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -12,6 +12,7 @@
>>   # See the COPYING file in the top-level directory.
>>   
>>   import re
>> +from typing import Optional, Union, Sequence
>>   
>>   
>>   EATSPACE = '\033EATSPACE.'
>> @@ -22,7 +23,7 @@
>>   # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>>   # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>>   # ENUM24_Name -> ENUM24_NAME
>> -def camel_to_upper(value):
>> +def camel_to_upper(value: str) -> str:
>>       c_fun_str = c_name(value, False)
>>       if value.isupper():
>>           return c_fun_str
>> @@ -41,7 +42,9 @@ def camel_to_upper(value):
>>       return new_name.lstrip('_').upper()
>>   
>>   
>> -def c_enum_const(type_name, const_name, prefix=None):
>> +def c_enum_const(type_name: str,
>> +                 const_name: str,
>> +                 prefix: Optional[str] = None) -> str:
>>       if prefix is not None:
>>           type_name = prefix
>>       return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>> @@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
>>   # into substrings of a generated C function name.
>>   # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>>   # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>> -def c_name(name, protect=True):
>> +def c_name(name: str, protect: bool = True) -> str:
>>       # ANSI X3J11/88-090, 3.1.1
>>       c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>>                        'default', 'do', 'double', 'else', 'enum', 'extern',
>> @@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
>>   
>>   # Generate @code with @kwds interpolated.
>>   # Obey INDENT level, and strip EATSPACE.
>> -def cgen(code, **kwds):
>> +def cgen(code: str, **kwds: Union[str, int]) -> str:
> 
> Hmm.
> 
> The @kwds values can be anything, provided they match the conversion
> specifiers in @code:
> 
>>       raw = code % kwds
> 
> Your type hint adds a restriction that wasn't there before.
> 
> Is there a better way?
> 

Maybe there are format-like type annotation tricks you can do to enforce 
this, but I did not research them. I tried to resist "improving" our 
usage of the old % formatter prematurely. I may do a wholesale f-string 
conversion at some point, but not now, it's not important.

In practice, we pass strings and integers. This typing *is* artificially 
restrictive, though. We can declare the type to be "Any" and allow the 
function to fail or succeed at runtime if you'd prefer.

>>       if INDENT:
>>           raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
>>       return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>>   
>>   
>> -def mcgen(code, **kwds):
>> +def mcgen(code: str, **kwds: Union[str, int]) -> str:
> 
> Likewise.
> 

Unresearched idea: It's possible that we can subclass the 
string.Formatter class and extend it to perform our special variable 
replacements (chomping EATSPACE, etc.)

And *maybe* because it inherits from the standard formatter, we would 
benefit from any analysis Mypy performs on such things.

Basically, replace mcgen/cgen with class CFormatter(string.Formatter).

(maybe. assume that none of what I just said will work or is feasible.)

--js

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

[John Snow]

On 9/17/20 10:37 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> As docstrings, they'll show up in documentation and IDE help.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/common.py | 50 ++++++++++++++++++++++++++++++------------
>>   1 file changed, 36 insertions(+), 14 deletions(-)
>>
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index af01348b35..38d380f0a9 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -20,10 +20,18 @@
>>   c_name_trans = str.maketrans('.-', '__')
>>   
>>   
>> -# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>> -# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>> -# ENUM24_Name -> ENUM24_NAME
>>   def camel_to_upper(value: str) -> str:
>> +    """
>> +    Converts CamelCase to CAMEL_CASE.
>> +
>> +    Examples:
>> +      ENUMName -> ENUM_NAME
>> +      EnumName1 -> ENUM_NAME1
>> +      ENUM_NAME -> ENUM_NAME
>> +      ENUM_NAME1 -> ENUM_NAME1
>> +      ENUM_Name2 -> ENUM_NAME2
>> +      ENUM24_Name -> ENUM24_NAME
>> +    """
>>       c_fun_str = c_name(value, False)
>>       if value.isupper():
>>           return c_fun_str
>> @@ -45,21 +53,33 @@ def camel_to_upper(value: str) -> str:
>>   def c_enum_const(type_name: str,
>>                    const_name: str,
>>                    prefix: Optional[str] = None) -> str:
>> +    """
>> +    Generate a C enumeration constant name.
>> +
>> +    :param type_name: The name of the enumeration.
>> +    :param const_name: The name of this constant.
>> +    :param prefix: Optional, prefix that overrides the type_name.
>> +    """
> 
> Not actually a move.  Suggest to retitle
> 
>      qapi/common: Turn comments in docstrings, and add more
> 

OK.

>>       if prefix is not None:
>>           type_name = prefix
>>       return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>>   
>>   
>> -# Map @name to a valid C identifier.
>> -# If @protect, avoid returning certain ticklish identifiers (like
>> -# C keywords) by prepending 'q_'.
>> -#
>> -# Used for converting 'name' from a 'name':'type' qapi definition
>> -# into a generated struct member, as well as converting type names
>> -# into substrings of a generated C function name.
>> -# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>> -# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>>   def c_name(name: str, protect: bool = True) -> str:
>> +    """
>> +    Map `name` to a valid C identifier.
>> +
>> +    Used for converting 'name' from a 'name':'type' qapi definition
>> +    into a generated struct member, as well as converting type names
>> +    into substrings of a generated C function name.
>> +
>> +    '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>> +    protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>> +
>> +    :param name: The name to map.
>> +    :param protect: If true, avoid returning certain ticklish identifiers
>> +                    (like C keywords) by prepending 'q_'.
>> +    """
>>       # ANSI X3J11/88-090, 3.1.1
>>       c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>>                        'default', 'do', 'double', 'else', 'enum', 'extern',
>> @@ -135,9 +155,11 @@ def pop(self, amount: int = 4) -> int:
>>   INDENT = Indent(0)
>>   
>>   
>> -# Generate @code with @kwds interpolated.
>> -# Obey INDENT level, and strip EATSPACE.
>>   def cgen(code: str, **kwds: Union[str, int]) -> str:
>> +    """
>> +    Generate `code` with `kwds` interpolated.
>> +    Obey `INDENT`, and strip `EATSPACE`.
>> +    """
>>       raw = code % kwds
>>       if INDENT:
>>           raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
> 
> Can you point to documentation on the docstring conventions and markup
> to use?
> 

Short answer: No.

Long answer:

It's actually completely arbitrary, with major competing de-facto 
standards. Their primary function is to be stored to the __doc__ 
attribute of a module/class/method and can be displayed when using the 
interactive function help(...).

https://www.python.org/dev/peps/pep-0257/ covers docstrings only in an 
extremely broad sense. In summary, it asks:

- Use full sentences that end in periods
- Use the triple-double quote style
- multi-line docstrings should have their closing quote on a line by itself
- multi-line docstrings should use a one-sentence summary, a blank line, 
and then a more elaborate description.

It recommends you document arguments, types, return values and types, 
exceptions and so on but does not dictate a format. Two popular 
conventions are the google-style [1] and the NumPy-style [2] docstring 
formats.

I write docstrings assuming we will be using *Sphinx* as our 
documentation tool. Sphinx does not read docstrings at all by default, 
but *autodoc* does. Autodoc assumes your docstrings are written in the 
Sphinx dialect of ReST.

What you really want to look for is the "Python domain" documentation in 
Sphinx: 
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html

Autodoc will annotate function/method docstrings with "py:function" or 
"py:method" as appropriate, but the actual contents of the block are 
still up to you.

For those, you want to look up the Python domain info field lists that 
are supported by Sphinx, which are here: 
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Taken together with PEP 257, you generally want something like this:

"""
Function f converts uncertainty into happiness.

Function f only works on days that do not end in 'y'. Caution should be 
used when integrating this function into threaded code.

:param uncertainty: All of your doubts.
:raise RuntimeError: When it is a day ending in 'y'.
:return: Your newfound happiness.
"""

I use the single-backtick as the Sphinx-ese "default role" resolver, 
which generally should resolve to a reference to some Python entity. The 
double-backtick is used to do formatted text for things like string 
literals and so on.



Coffee break.



Having said this, I have not found any tool to date that actually 
*checks* these comments for consistency. The pycharm IDE interactively 
highlights them when it senses that you have made a mistake, but that 
cannot be worked into our CI process that I know of - it's a proprietary 
checker.

So right now, they're just plaintext that I am writing to approximate 
the Sphinx style until such time as I enable autodoc for the module and 
fine-tune the actual formatting and so on.

--js



[1] 
https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html

[2] 
https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy

Re: [PATCH 15/37] qapi/common.py: split build_params into new file

[John Snow]

On 9/17/20 10:42 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Including it in common.py creates a circular import dependency, because
>> schema relies on common.py. To type build_params properly, it needs to
>> be moved outside of the chain.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/commands.py |  2 +-
>>   scripts/qapi/common.py   | 23 -----------------------
>>   scripts/qapi/events.py   |  2 +-
>>   scripts/qapi/params.py   | 40 ++++++++++++++++++++++++++++++++++++++++
>>   4 files changed, 42 insertions(+), 25 deletions(-)
>>   create mode 100644 scripts/qapi/params.py
> 
> Ugh.
> 
> Would moving it gen.py work?
> 

Actually, yes.

I have an experimental patch way, way later in the series that does this:

1. Leaves common.py with *just* functions and constants used by 
schema.py: c_name, POINTER_SUFFIX, and transitively EATSPACE.

2. Splits gen_c.py out of gen.py, putting all of the C-specific 
generator classes in there.

3. Adds params.py and the C-specific bits of common.py into gen_c.py.


In effect, you get gen_c.py with all of the C-specific bits in it, all 
of the other code-generation modules import from gen_c (marking them 
obviously as C code generators), and schema.py and other parsing friends 
import only the tiny common.py for c_name().


I'll adjust this patch to stash this in gen.py for now. It's too 
disruptive to shift the other refactor around in my stack.

--js

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

[Eduardo Habkost]

On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
[...]
> Having said this, I have not found any tool to date that actually *checks*
> these comments for consistency. The pycharm IDE interactively highlights
> them when it senses that you have made a mistake, but that cannot be worked
> into our CI process that I know of - it's a proprietary checker.
> 
> So right now, they're just plaintext that I am writing to approximate the
> Sphinx style until such time as I enable autodoc for the module and
> fine-tune the actual formatting and so on.

After applying this series, I only had to make two small tweaks
to make Sphinx + autodoc happy with the docstrings you wrote.

With the following patch, "make sphinxdocs" will generate the
QAPI Python module documentation at docs/devel/qapi.html.

I had to explicitly skip qapi/doc.py because autodoc thinks the
string constants are documentation strings.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 docs/conf.py           |  5 +++-
 docs/devel/index.rst   |  1 +
 docs/devel/qapi.rst    | 67 ++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/common.py |  2 +-
 scripts/qapi/gen.py    |  2 +-
 5 files changed, 74 insertions(+), 3 deletions(-)
 create mode 100644 docs/devel/qapi.rst

diff --git a/docs/conf.py b/docs/conf.py
index 8aeac40124..85be0e1860 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -54,6 +54,9 @@ except NameError:
 #
 sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
 
+# Make scripts/qapi module available for autodoc
+sys.path.insert(0, os.path.join(qemu_docdir, "../scripts"))
+
 
 # -- General configuration ------------------------------------------------
 
@@ -67,7 +70,7 @@ needs_sphinx = '1.6'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile']
+extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile', 'sphinx.ext.autodoc']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 04773ce076..a4d2cb9893 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -31,3 +31,4 @@ Contents:
    reset
    s390-dasd-ipl
    clocks
+   qapi
diff --git a/docs/devel/qapi.rst b/docs/devel/qapi.rst
new file mode 100644
index 0000000000..9130ef84c6
--- /dev/null
+++ b/docs/devel/qapi.rst
@@ -0,0 +1,67 @@
+QAPI Python module reference
+============================
+
+.. automodule:: qapi
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.commands
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.common
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.debug
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.error
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.events
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.expr
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.gen
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.introspect
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.params
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.parser
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.schema
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.script
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.source
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.types
+   :members:
+   :undoc-members:
+
+.. automodule:: qapi.visit
+   :members:
+   :undoc-members:
+
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 0b1af694e6..7c8c4cb846 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -78,7 +78,7 @@ def c_name(name: str, protect: bool = True) -> str:
 
     :param name: The name to map.
     :param protect: If true, avoid returning certain ticklish identifiers
-                    (like C keywords) by prepending 'q_'.
+                    (like C keywords) by prepending ``q_``.
     """
     # ANSI X3J11/88-090, 3.1.1
     c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 0a72aecdd0..cf33732256 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -161,7 +161,7 @@ def ifcontext(ifcond: List[str],
               *args: QAPIGenCCode) -> Generator[None, None, None]:
     """A 'with' statement context manager to wrap with start_if()/end_if()
 
-    *args: any number of QAPIGenCCode
+    :param args: any number of QAPIGenCCode
 
     Example::
 
-- 
2.26.2

-- 
Eduardo

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

[John Snow]

On 9/17/20 3:14 PM, Eduardo Habkost wrote:
> On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
> [...]
>> Having said this, I have not found any tool to date that actually *checks*
>> these comments for consistency. The pycharm IDE interactively highlights
>> them when it senses that you have made a mistake, but that cannot be worked
>> into our CI process that I know of - it's a proprietary checker.
>>
>> So right now, they're just plaintext that I am writing to approximate the
>> Sphinx style until such time as I enable autodoc for the module and
>> fine-tune the actual formatting and so on.
> 
> After applying this series, I only had to make two small tweaks
> to make Sphinx + autodoc happy with the docstrings you wrote.
> 
> With the following patch, "make sphinxdocs" will generate the
> QAPI Python module documentation at docs/devel/qapi.html.
> 
> I had to explicitly skip qapi/doc.py because autodoc thinks the
> string constants are documentation strings.
> 

Awesome!

I think I am going to delay explicitly pursuing writing a manual for the 
QAPI parser for now, but it's good to know I am not too far off. I'm 
going to target the mypy conversions first, because they can be invasive 
and full of churn.

When I get there, though ... I am thinking I should add this as 
Devel/QAPI Parser.

> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>   docs/conf.py           |  5 +++-
>   docs/devel/index.rst   |  1 +
>   docs/devel/qapi.rst    | 67 ++++++++++++++++++++++++++++++++++++++++++
>   scripts/qapi/common.py |  2 +-
>   scripts/qapi/gen.py    |  2 +-
>   5 files changed, 74 insertions(+), 3 deletions(-)
>   create mode 100644 docs/devel/qapi.rst
> 
> diff --git a/docs/conf.py b/docs/conf.py
> index 8aeac40124..85be0e1860 100644
> --- a/docs/conf.py
> +++ b/docs/conf.py
> @@ -54,6 +54,9 @@ except NameError:
>   #
>   sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
>   
> +# Make scripts/qapi module available for autodoc
> +sys.path.insert(0, os.path.join(qemu_docdir, "../scripts"))
> +
>   
>   # -- General configuration ------------------------------------------------
>   
> @@ -67,7 +70,7 @@ needs_sphinx = '1.6'
>   # Add any Sphinx extension module names here, as strings. They can be
>   # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
>   # ones.
> -extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile']
> +extensions = ['kerneldoc', 'qmp_lexer', 'hxtool', 'depfile', 'sphinx.ext.autodoc']
>   
>   # Add any paths that contain templates here, relative to this directory.
>   templates_path = ['_templates']
> diff --git a/docs/devel/index.rst b/docs/devel/index.rst
> index 04773ce076..a4d2cb9893 100644
> --- a/docs/devel/index.rst
> +++ b/docs/devel/index.rst
> @@ -31,3 +31,4 @@ Contents:
>      reset
>      s390-dasd-ipl
>      clocks
> +   qapi
> diff --git a/docs/devel/qapi.rst b/docs/devel/qapi.rst
> new file mode 100644
> index 0000000000..9130ef84c6
> --- /dev/null
> +++ b/docs/devel/qapi.rst
> @@ -0,0 +1,67 @@
> +QAPI Python module reference
> +============================
> +
> +.. automodule:: qapi
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.commands
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.common
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.debug
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.error
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.events
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.expr
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.gen
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.introspect
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.params
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.parser
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.schema
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.script
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.source
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.types
> +   :members:
> +   :undoc-members:
> +
> +.. automodule:: qapi.visit
> +   :members:
> +   :undoc-members:
> +
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 0b1af694e6..7c8c4cb846 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -78,7 +78,7 @@ def c_name(name: str, protect: bool = True) -> str:
>   
>       :param name: The name to map.
>       :param protect: If true, avoid returning certain ticklish identifiers
> -                    (like C keywords) by prepending 'q_'.
> +                    (like C keywords) by prepending ``q_``.
>       """
>       # ANSI X3J11/88-090, 3.1.1
>       c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index 0a72aecdd0..cf33732256 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -161,7 +161,7 @@ def ifcontext(ifcond: List[str],
>                 *args: QAPIGenCCode) -> Generator[None, None, None]:
>       """A 'with' statement context manager to wrap with start_if()/end_if()
>   
> -    *args: any number of QAPIGenCCode
> +    :param args: any number of QAPIGenCCode
>   
>       Example::
>   
> 

Re: [PATCH 15/37] qapi/common.py: split build_params into new file

[John Snow]

On 9/17/20 10:42 AM, Markus Armbruster wrote:
> Ugh.
> 
> Would moving it gen.py work?

Done.

Re: [PATCH 13/37] qapi/common.py: add notational type hints

[John Snow]

On 9/17/20 2:18 PM, John Snow wrote:
> Your type hint adds a restriction that wasn't there before.
> 
> Is there a better way?

I've settled on using the `object` type for now, which is slightly more 
restrictive than `Any`.

`Any` and `object` both allow any type of argument, but `Any` 
effectively creates a gradually typed boundary in which you are 
implicitly casting to whatever typed boundary it enters next.

`object` is an explicit cast to the most abstracted type; any further 
usage that is not supported the base object will be rejected.

This is nice, because if someone adds a call at or below the mcgen 
level, mypy will certainly complain that they are assuming too much 
about the type -- and they would be!

The rest of my series will now take this approach: prefer `object` to `Any`.

--js

Re: [PATCH 00/37] qapi: static typing conversion, pt1

[John Snow]

On 9/15/20 6:39 PM, John Snow wrote:
> Hi, this series starts adding static type hints to the QAPI module. As
> you can see, the series started getting quite a bit long, so this is
> only a partial conversion that focuses on a handful of the easier files.
> 
> The tougher files -- schema.py, expr.py, parser.py -- will each receive
> their own series as a follow-up to this one.
> 
> Notes:
> 
> - This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
> 
> - Any patch named "add notational type hints" changes ONLY signatures,
>    which have no runtime impact whatsoever. These are big patches,
>    but fairly straightforward.
> 
> - Most other patches are as bite-sized as possible, generally fixing one
>    single warning.
> 
> - After patch 6, `flake8 qapi/` should pass 100% on this and every
>    future commit.
> 
> - After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
>    on this and every future commit.
> 
> - After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
>    100% on this and every future commit.
> 

Addressed today's feedback and pushed to gitlab.
Let's call it "Version 1.3".

https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/

Waiting on feedback for 16-37 before re-authoring to list.

--js

Re: [PATCH 00/37] qapi: static typing conversion, pt1

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/15/20 6:39 PM, John Snow wrote:
>> Hi, this series starts adding static type hints to the QAPI module. As
>> you can see, the series started getting quite a bit long, so this is
>> only a partial conversion that focuses on a handful of the easier files.
>> The tougher files -- schema.py, expr.py, parser.py -- will each
>> receive
>> their own series as a follow-up to this one.
>> Notes:
>> - This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
>> - Any patch named "add notational type hints" changes ONLY
>> signatures,
>>    which have no runtime impact whatsoever. These are big patches,
>>    but fairly straightforward.
>> - Most other patches are as bite-sized as possible, generally fixing
>> one
>>    single warning.
>> - After patch 6, `flake8 qapi/` should pass 100% on this and every
>>    future commit.
>> - After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass
>> 100%
>>    on this and every future commit.
>> - After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should
>> pass
>>    100% on this and every future commit.
>> 
>
> Addressed today's feedback and pushed to gitlab.
> Let's call it "Version 1.3".
>
> https://gitlab.com/jsnow/qemu/-/commits/python-qapi-cleanup-pt1/
>
> Waiting on feedback for 16-37 before re-authoring to list.

If 03-12 don't actually depend on Python 3.6, you may want to respin
these for me to queue right away.

Re: [PATCH 06/37] qapi: delint using flake8

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/17/20 3:54 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 9/16/20 8:12 AM, Markus Armbruster wrote:
>>>> John Snow <jsnow@redhat.com> writes:
>>>>
>>>>> Petty style guide fixes and line length enforcement.  Not a big win, not
>>>>> a big loss, but flake8 passes 100% on the qapi module, which gives us an
>>>>> easy baseline to enforce hereafter.
>>>>>
>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
[...]
>>>>> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
>>>>> index e1df0e341f..2e4b4de0fa 100644
>>>>> --- a/scripts/qapi/commands.py
>>>>> +++ b/scripts/qapi/commands.py
>>>>> @@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
>>>>>    def gen_marshal_output(ret_type):
>>>>>        return mcgen('''
>>>>>    -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
>>>>> QObject **ret_out, Error **errp)
>>>>> +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
>>>>> +                                          Error **errp)
>>>> The continued parameter list may become misalignd in generated C.
>>>> E.g.
>>>>       static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
>>>>                                                 Error **errp)
>>>>       {
>>>>       ...
>>>>       }
>>>> Do we care?
>>>>
>>>
>>> Yeah, I don't know. Do we?
>> I care, but I also care for automated style checks.
>> 
>>> It actually seemed more annoying to try and get flake8 to make an
>>> exception for these handful of examples.
>>>
>>> Path of least resistance led me here, but I can try and appease both
>>> systems if you'd prefer.
>> Up to now, I ran the style checkers manually, and this was just one
>> of
>> several complaints to ignore, so I left the code alone.
>> If it gets in the way of running them automatically, and messing up
>> the
>> generated code slightly is the easiest way to get it out of the way,
>> then I can accept the slight mess.
>> 
>
> I changed this a little to put all the args on the next line, which is
> slightly unusual but works okay.

I think it's slightly more unusual than the non-matching indentation
was.

Yet another way to skin this cat:

    static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
                                    QObject **ret_out, Error **errp)

Now the second line is not aligned with the left parenthesis both in the
Python source and in the generated C.

> I think that's a fine middle ground, because the alternative (to me)
> is to start using abstracted code generation tokens in a tree
> structure, etc etc etc.
>
> Embedded templates are always gonna look kinda nasty, I think, because
> you're trying to fight style guidelines in two languages
> simultaneously and it's never gonna quite work out exactly how you
> want without some pretty complex abstraction mechanisms that are well
> beyond the power we need right now.

The thought "screw this, pile the output through /usr/bin/indent" has
crossed my mind more than once.

Re: [PATCH 09/37] qapi/common.py: Add indent manager

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/17/20 4:07 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 9/16/20 11:13 AM, Markus Armbruster wrote:
>>>> John Snow <jsnow@redhat.com> writes:
>>>>
>
>
>>>> Let's replace "the indent" by "the indentation" globally.
>>>>
>>>
>>> They're both cromulent as nouns and one is shorter.
>>>
>>> I'll switch in good faith; do you prefer "Indentation" as a noun?
>> Use of "indent" as a noun was new to me, but what do I know; you're
>> the
>> native speaker.  Use your judgement.  Applies to the class name, too.
>> 
>
> I made the change; see gitlab commits or wait for v2 to see if it
> reads better to you.
>
>>>>> +        return self._level
>>>>>    +    def __repr__(self) -> str:
>>>>> +        return "{}({:d})".format(type(self).__name__, self._level)
>>>> Is __repr__ needed?
>>>>
>>>
>>> Yes; it's used in the underflow exception , and it is nice when using
>>> the python shell interactively.
>>>
>>> repr(Indent(4)) == "Indent(4)"
>> Meh.  There's another three dozen classes for you to put lipstick on
>> :)
>> 
>
> We'll get to them in due time. For now, please admire the lipstick.

If I take off my glasses and step six feet back, I just might be able to
overlook it.

>>>>> +    def pop(self, amount: int = 4) -> int:
>>>>> +        """Pop `amount` spaces off of the indent, default four."""
>>>>> +        if self._level < amount:
>>>>> +            raise ArithmeticError(
>>>>> +                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
>> I think assert(amount <= self._level) would do just fine.
>> 
>
> Secretly, asserts can be disabled in Python just like they can in C code.

There are compilers that let you switch off array bounds checking.
Would you advocate manual bounds checking to protect people from their
own folly?

> My habit: if it's something that should already be true given the
> nature of how the code is laid out, use an assertion. If I am
> preventing an erroneous state (Especially from callers in other
> modules), explicitly raise an exception.

I check function preconditions ruthlessly with assert.  There's no sane
way to recover anyway.

Without a way to recover, the only benefit is crashing more prettily.
If the error is six levels deep in a some fancy-pants framework, then
prettier crashes might actually help someone finding the error slightly
faster.  But it ain't.

My final argument is local consistency: use of assertions to check
preconditions is pervasive in scripts/qapi/.

> (See the gitlab branch for the updated version of this patch, which
> has changed the code slightly.)

[...]

Re: [PATCH 10/37] qapi/common.py: delint with pylint

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/17/20 10:15 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>   scripts/qapi/common.py | 16 +++++++---------
>>>   scripts/qapi/schema.py | 14 +++++++-------
>>>   2 files changed, 14 insertions(+), 16 deletions(-)
>>>
>>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>>> index 87d87b95e5..c665e67495 100644
>>> --- a/scripts/qapi/common.py
>>> +++ b/scripts/qapi/common.py
>>> @@ -14,6 +14,11 @@
>>>   import re
>>>     
>>> +EATSPACE = '\033EATSPACE.'
>>> +POINTER_SUFFIX = ' *' + EATSPACE
>>> +c_name_trans = str.maketrans('.-', '__')
>>> +
>>> +
>> You rename and move.  pylint gripes about the names, but it doesn't
>> actually ask for the move, as far as I can tell.  Can you explain why
>> you move?
>> 
>
> Preference. I like constants and globals at the top so you can audit
> any code that runs at import time in one place.

I can buy this argument.

>                                                 Since they are
> externally visible objects, having them near other "header" style
> information makes sense to me.

This one I find unconvincing.  Functions and classes are just as
visible.

Mention the move in the commit message, along with the (convincing part
of the) rationale.

Aside: EATSPACE and c_name_trans are actually local to common.py.
EATSPACE sort of leaks out only via the contract of mcgen().  The
contract could be rephrased in terms of POINTER_SUFFIX.  Not sure it's
worthwhile.

>>>   # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>>>   # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>>>   # ENUM24_Name -> ENUM24_NAME
>>> @@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
>>>       return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>>>     
>>> -c_name_trans = str.maketrans('.-', '__')
>
> (This one winds up being a constant, so I renamed it in my v2.)
>
>>> -
>>> -
>>>   # Map @name to a valid C identifier.
>>>   # If @protect, avoid returning certain ticklish identifiers (like
>>>   # C keywords) by prepending 'q_'.
>>> @@ -89,10 +91,6 @@ def c_name(name, protect=True):
>>>       return name
>>>     
>>> -eatspace = '\033EATSPACE.'
>>> -pointer_suffix = ' *' + eatspace
>>> -
>>> -
>>>   class Indent:
>>>       """
>>>       Indent-level management.
>>> @@ -135,12 +133,12 @@ def pop(self, amount: int = 4) -> int:
>>>     
>>>   # Generate @code with @kwds interpolated.
>>> -# Obey indent_level, and strip eatspace.
>>> +# Obey INDENT level, and strip EATSPACE.
>> Is the change to INDENT intentional?
>> 
>
> Kind of, but it's either late (should have been with the indent
> manager patch) or early (Should be with the patch that moves comments
> into docstrings.)
>
> When this comment becomes a docstring, I use `INDENT` to indicate it
> as a proper object. This in and of itself is prescient, as we are not
> using sphinx/apidoc to generate any documentation about the QAPI
> package yet.
>
> (The pending v2 uses `indent` after you pointed out that it was not a
> constant.)
>
>>>   def cgen(code, **kwds):
>>>       raw = code % kwds
>>>       if INDENT:
>>>           raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
>>> -    return re.sub(re.escape(eatspace) + r' *', '', raw)
>>> +    return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>>>     
>>>   def mcgen(code, **kwds):
>> [...]
>> 

Re: [PATCH 13/37] qapi/common.py: add notational type hints

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/17/20 10:32 AM, Markus Armbruster wrote:
>> Question on the subject line: what makes a type hint notational?
>> 
>
> My cover letter explains that every time I use this phrase, I mean to
> state that "This patch adds exclusively type notations and makes no 
> functional changes to the runtime operation whatsoever."
>
> i.e. notations-only.

By the time I get to PATCH 13, details explained in the cover letter
have been flushed from my memory.  Moreover, the cover letter won't make
it into Git.  Best to repeat them right in the commit message.  Perhaps:

    qapi/common; Add type hints

    Type hints do not change behavior.

>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>   scripts/qapi/common.py | 27 ++++++++++++++++-----------
>>>   1 file changed, 16 insertions(+), 11 deletions(-)
>>>
>>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>>> index 4c079755d3..af01348b35 100644
>>> --- a/scripts/qapi/common.py
>>> +++ b/scripts/qapi/common.py
>>> @@ -12,6 +12,7 @@
>>>   # See the COPYING file in the top-level directory.
>>>     import re
>>> +from typing import Optional, Union, Sequence
>>>     
>>>   EATSPACE = '\033EATSPACE.'
>>> @@ -22,7 +23,7 @@
>>>   # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>>>   # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>>>   # ENUM24_Name -> ENUM24_NAME
>>> -def camel_to_upper(value):
>>> +def camel_to_upper(value: str) -> str:
>>>       c_fun_str = c_name(value, False)
>>>       if value.isupper():
>>>           return c_fun_str
>>> @@ -41,7 +42,9 @@ def camel_to_upper(value):
>>>       return new_name.lstrip('_').upper()
>>>     
>>> -def c_enum_const(type_name, const_name, prefix=None):
>>> +def c_enum_const(type_name: str,
>>> +                 const_name: str,
>>> +                 prefix: Optional[str] = None) -> str:
>>>       if prefix is not None:
>>>           type_name = prefix
>>>       return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>>> @@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
>>>   # into substrings of a generated C function name.
>>>   # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>>>   # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>>> -def c_name(name, protect=True):
>>> +def c_name(name: str, protect: bool = True) -> str:
>>>       # ANSI X3J11/88-090, 3.1.1
>>>       c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>>>                        'default', 'do', 'double', 'else', 'enum', 'extern',
>>> @@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
>>>     # Generate @code with @kwds interpolated.
>>>   # Obey INDENT level, and strip EATSPACE.
>>> -def cgen(code, **kwds):
>>> +def cgen(code: str, **kwds: Union[str, int]) -> str:
>> Hmm.
>> The @kwds values can be anything, provided they match the conversion
>> specifiers in @code:
>> 
>>>       raw = code % kwds
>> Your type hint adds a restriction that wasn't there before.
>> Is there a better way?
>> 
>
> Maybe there are format-like type annotation tricks you can do to
> enforce this, but I did not research them. I tried to resist
> "improving" our usage of the old % formatter prematurely. I may do a
> wholesale f-string conversion at some point, but not now, it's not
> important.
>
> In practice, we pass strings and integers. This typing *is*
> artificially restrictive, though. We can declare the type to be "Any"
> and allow the function to fail or succeed at runtime if you'd prefer.
>
>>>       if INDENT:
>>>           raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
>>>       return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>>>     
>>> -def mcgen(code, **kwds):
>>> +def mcgen(code: str, **kwds: Union[str, int]) -> str:
>> Likewise.
>> 
>
> Unresearched idea: It's possible that we can subclass the
> string.Formatter class and extend it to perform our special variable 
> replacements (chomping EATSPACE, etc.)
>
> And *maybe* because it inherits from the standard formatter, we would
> benefit from any analysis Mypy performs on such things.
>
> Basically, replace mcgen/cgen with class CFormatter(string.Formatter).
>
> (maybe. assume that none of what I just said will work or is feasible.)

Sounds worth exploring.  No need to do it now, of course.

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

[Markus Armbruster]

Ignorant question: why does this come after PATCH 13 "qapi/common.py:
add notational type hints", but before all the other patches adding type
hints?

John Snow <jsnow@redhat.com> writes:

> Fix two very minor issues, and then establish a mypy type-checking
> baseline.
>
> Like pylint, this should be run from the folder above:
>
>  > mypy --config-file=qapi/mypy.ini qapi/

I get:

    $ mypy --config-file qapi/mypy.ini qapi
    qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
    qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
    Found 1 error in 1 file (checked 18 source files)

Is this expected?

In case it matters:

    $ mypy --version
    mypy 0.761

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/doc.py    |  3 +-
>  scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
>  scripts/qapi/schema.py |  3 +-
>  3 files changed, 69 insertions(+), 2 deletions(-)
>  create mode 100644 scripts/qapi/mypy.ini
>
> diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
> index cbf7076ed9..70f7cdfaa6 100644
> --- a/scripts/qapi/doc.py
> +++ b/scripts/qapi/doc.py
> @@ -5,7 +5,8 @@
>  """This script produces the documentation of a qapi schema in texinfo format"""
>  
>  import re
> -from .gen import QAPIGenDoc, QAPISchemaVisitor
> +from .gen import QAPIGenDoc
> +from .schema import QAPISchemaVisitor

Your mypy doesn't like such lazy imports?  Mine seems not to care.

>  
>  
>  MSG_FMT = """
> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
> new file mode 100644
> index 0000000000..a0f2365a53
> --- /dev/null
> +++ b/scripts/qapi/mypy.ini
> @@ -0,0 +1,65 @@
> +[mypy]
> +strict = True
> +strict_optional = False
> +disallow_untyped_calls = False
> +python_version = 3.6
> +
> +[mypy-qapi.commands]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.doc]
> +disallow_subclassing_any = False
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +
> +[mypy-qapi.error]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.events]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.expr]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.gen]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.introspect]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.parser]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.schema]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.source]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.types]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False
> +
> +[mypy-qapi.visit]
> +disallow_untyped_defs = False
> +disallow_incomplete_defs = False
> +check_untyped_defs = False

Greek to me.  I'll learn in due time.

> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index b4921b46c9..bb0cd717f1 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -17,6 +17,7 @@
>  import os
>  import re
>  from collections import OrderedDict
> +from typing import Optional
>  
>  from .common import c_name, POINTER_SUFFIX
>  from .error import QAPIError, QAPISemError
> @@ -25,7 +26,7 @@
>  
>  
>  class QAPISchemaEntity:
> -    meta = None
> +    meta: Optional[str] = None
>  
>      def __init__(self, name, info, doc, ifcond=None, features=None):
>          assert name is None or isinstance(name, str)

Re: [PATCH 00/37] qapi: static typing conversion, pt1

[Philippe Mathieu-Daudé]

Cc'ing Ben who offered to help with having the QEMU python
scripts packaged (and this series is a step toward that).

On 9/16/20 12:39 AM, John Snow wrote:
> Hi, this series starts adding static type hints to the QAPI module. As
> you can see, the series started getting quite a bit long, so this is
> only a partial conversion that focuses on a handful of the easier files.
> 
> The tougher files -- schema.py, expr.py, parser.py -- will each receive
> their own series as a follow-up to this one.
> 
> Notes:
> 
> - This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
> 
> - Any patch named "add notational type hints" changes ONLY signatures,
>   which have no runtime impact whatsoever. These are big patches,
>   but fairly straightforward.
> 
> - Most other patches are as bite-sized as possible, generally fixing one
>   single warning.
> 
> - After patch 6, `flake8 qapi/` should pass 100% on this and every
>   future commit.
> 
> - After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
>   on this and every future commit.
> 
> - After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
>   100% on this and every future commit.
> 
> Preliminary refactoring and prerequisites:
> 
> 001/37: 'python: Require 3.6+'
> 002/37: '[DO-NOT-MERGE] qapi: add debugging tools'
> 003/37: 'qapi-gen: Separate arg-parsing from generation'
> 004/37: 'qapi: move generator entrypoint into module'
> 005/37: 'qapi: Remove wildcard includes'
> 006/37: 'qapi: delint using flake8'
> 007/37: 'qapi: add pylintrc'
> 
> common.py (and params.py):
> 
> 008/37: 'qapi/common.py: Remove python compatibility workaround'
> 009/37: 'qapi/common.py: Add indent manager'
> 010/37: 'qapi/common.py: delint with pylint'
> 011/37: 'qapi/common.py: Replace one-letter 'c' variable'
> 012/37: 'qapi/common.py: check with pylint'
> 013/37: 'qapi/common.py: add notational type hints'
> 014/37: 'qapi/common.py: Move comments into docstrings'
> 015/37: 'qapi/common.py: split build_params into new file'
> 016/37: 'qapi: establish mypy type-checking baseline'
> 
> events.py:
> 
> 017/37: 'qapi/events.py: add notational type hints'
> 018/37: 'qapi/events.py: Move comments into docstrings'
> 
> commands.py:
> 
> 019/37: 'qapi/commands.py: Don't re-bind to variable of different type'
> 020/37: 'qapi/commands.py: add notational type hints'
> 021/37: 'qapi/commands.py: enable checking with mypy'
> 
> source.py:
> 
> 022/37: 'qapi/source.py: add notational type hints'
> 023/37: 'qapi/source.py: delint with pylint'
> 
> gen.py:
> 
> 024/37: 'qapi/gen.py: Fix edge-case of _is_user_module'
> 025/37: 'qapi/gen.py: add notational type hints'
> 026/37: 'qapi/gen.py: Enable checking with mypy'
> 027/37: 'qapi/gen.py: Remove unused parameter'
> 028/37: 'qapi/gen.py: update write() to be more idiomatic'
> 029/37: 'qapi/gen.py: delint with pylint'
> 
> introspect.py:
> 
> 030/37: 'qapi/introspect.py: Add a typed 'extra' structure'
> 031/37: 'qapi/introspect.py: add _gen_features helper'
> 032/37: 'qapi/introspect.py: create a typed 'Node' data structure'
> 033/37: 'qapi/introspect.py: add notational type hints'
> 
> types.py:
> 
> 034/37: 'qapi/types.py: add notational type hints'
> 035/37: 'qapi/types.py: remove one-letter variables'
> 
> visit.py:
> 
> 036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object'
> 037/37: 'qapi/visit.py: add notational type hints'

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

[John Snow]

On 9/18/20 7:55 AM, Markus Armbruster wrote:
> Ignorant question: why does this come after PATCH 13 "qapi/common.py:
> add notational type hints", but before all the other patches adding type
> hints?
> 
> John Snow <jsnow@redhat.com> writes:
> 
>> Fix two very minor issues, and then establish a mypy type-checking
>> baseline.
>>
>> Like pylint, this should be run from the folder above:
>>
>>   > mypy --config-file=qapi/mypy.ini qapi/
> 
> I get:
> 
>      $ mypy --config-file qapi/mypy.ini qapi
>      qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
>      qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
>      Found 1 error in 1 file (checked 18 source files)
> 
> Is this expected?
> 

Nope.

> In case it matters:
> 
>      $ mypy --version
>      mypy 0.761
> 

I am using mypy 0.782.

I will investigate to see if there is an *easy* win to allow older 
versions to work.

In the meantime, please consider trying this:

pip install --user mypy==0.782
~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/

>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   scripts/qapi/doc.py    |  3 +-
>>   scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
>>   scripts/qapi/schema.py |  3 +-
>>   3 files changed, 69 insertions(+), 2 deletions(-)
>>   create mode 100644 scripts/qapi/mypy.ini
>>
>> diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
>> index cbf7076ed9..70f7cdfaa6 100644
>> --- a/scripts/qapi/doc.py
>> +++ b/scripts/qapi/doc.py
>> @@ -5,7 +5,8 @@
>>   """This script produces the documentation of a qapi schema in texinfo format"""
>>   
>>   import re
>> -from .gen import QAPIGenDoc, QAPISchemaVisitor
>> +from .gen import QAPIGenDoc
>> +from .schema import QAPISchemaVisitor
> 
> Your mypy doesn't like such lazy imports?  Mine seems not to care.
> 

Yeah, it specifically complained that no such definition existed in that 
file.

>>   
>>   
>>   MSG_FMT = """
>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>> new file mode 100644
>> index 0000000000..a0f2365a53
>> --- /dev/null
>> +++ b/scripts/qapi/mypy.ini
>> @@ -0,0 +1,65 @@
>> +[mypy]
>> +strict = True
>> +strict_optional = False
>> +disallow_untyped_calls = False
>> +python_version = 3.6
>> +
>> +[mypy-qapi.commands]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.doc]
>> +disallow_subclassing_any = False
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +
>> +[mypy-qapi.error]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.events]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.expr]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.gen]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.introspect]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.parser]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.schema]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.source]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.types]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
>> +
>> +[mypy-qapi.visit]
>> +disallow_untyped_defs = False
>> +disallow_incomplete_defs = False
>> +check_untyped_defs = False
> 
> Greek to me.  I'll learn in due time.
> 

I am using these options:

--strict, which is effectively -Wall.

--no-strict-optional, which disables type checking errors on conflict 
between Optional[T] and T. Namely, when you initialize a class field to 
None and set that variable after initialization, callers must be 
prepared to see if that field was None or not. We do that effectively 
nowhere.

As Python will surely explode in a noticeable way if we got an 
unexpected 'None', I am just suppressing these warnings "for now".

--allow-untyped-calls silences errors in files that have calls to 
functions in files I still have not typed. By the end of the series, 
this option goes away, because there's nothing untyped left.


For each untyped file, we are actually starting with all of the above 
options and then layering these options on top. Any egregious typing 
errors present in these "ignored" files will be spotted.

To get the bad files to pass, we only need three options:

allow untyped defs -- Simply permits us to have functions without 
annotations.

allow incomplete defs -- allows functions that are only partially typed.

check untyped defs = False -- Don't try to type check untyped definitions.

>> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> index b4921b46c9..bb0cd717f1 100644
>> --- a/scripts/qapi/schema.py
>> +++ b/scripts/qapi/schema.py
>> @@ -17,6 +17,7 @@
>>   import os
>>   import re
>>   from collections import OrderedDict
>> +from typing import Optional
>>   
>>   from .common import c_name, POINTER_SUFFIX
>>   from .error import QAPIError, QAPISemError
>> @@ -25,7 +26,7 @@
>>   
>>   
>>   class QAPISchemaEntity:
>> -    meta = None
>> +    meta: Optional[str] = None
>>   
>>       def __init__(self, name, info, doc, ifcond=None, features=None):
>>           assert name is None or isinstance(name, str)

Re: [PATCH 00/37] qapi: static typing conversion, pt1

[John Snow]

On 9/18/20 9:07 AM, Philippe Mathieu-Daudé wrote:
> Cc'ing Ben who offered to help with having the QEMU python
> scripts packaged (and this series is a step toward that).
> 

FWIW, I want to move ./python/qemu/ into ./python/qemu/core, and then 
move ./scripts/qapi to ./python/qemu/qapi.

I will leave the qapi-gen.py stub script behind for compatibility.

Once I've done that, I can add package setup scripts.

I have a long series adding mypy type hints and so forth to 
./python/qemu and adding packaging glue there too, but I shelved it as 
5.1 released and right now I am focusing on QAPI.

> On 9/16/20 12:39 AM, John Snow wrote:
>> Hi, this series starts adding static type hints to the QAPI module. As
>> you can see, the series started getting quite a bit long, so this is
>> only a partial conversion that focuses on a handful of the easier files.
>>
>> The tougher files -- schema.py, expr.py, parser.py -- will each receive
>> their own series as a follow-up to this one.
>>
>> Notes:
>>
>> - This requires Python 3.6+. Python 3.5 is EOL, so let's do that.
>>
>> - Any patch named "add notational type hints" changes ONLY signatures,
>>    which have no runtime impact whatsoever. These are big patches,
>>    but fairly straightforward.
>>
>> - Most other patches are as bite-sized as possible, generally fixing one
>>    single warning.
>>
>> - After patch 6, `flake8 qapi/` should pass 100% on this and every
>>    future commit.
>>
>> - After patch 7, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
>>    on this and every future commit.
>>
>> - After patch 16, `mypy --config-file=qapi/mypy.ini qapi/` should pass
>>    100% on this and every future commit.
>>
>> Preliminary refactoring and prerequisites:
>>
>> 001/37: 'python: Require 3.6+'
>> 002/37: '[DO-NOT-MERGE] qapi: add debugging tools'
>> 003/37: 'qapi-gen: Separate arg-parsing from generation'
>> 004/37: 'qapi: move generator entrypoint into module'
>> 005/37: 'qapi: Remove wildcard includes'
>> 006/37: 'qapi: delint using flake8'
>> 007/37: 'qapi: add pylintrc'
>>
>> common.py (and params.py):
>>
>> 008/37: 'qapi/common.py: Remove python compatibility workaround'
>> 009/37: 'qapi/common.py: Add indent manager'
>> 010/37: 'qapi/common.py: delint with pylint'
>> 011/37: 'qapi/common.py: Replace one-letter 'c' variable'
>> 012/37: 'qapi/common.py: check with pylint'
>> 013/37: 'qapi/common.py: add notational type hints'
>> 014/37: 'qapi/common.py: Move comments into docstrings'
>> 015/37: 'qapi/common.py: split build_params into new file'
>> 016/37: 'qapi: establish mypy type-checking baseline'
>>
>> events.py:
>>
>> 017/37: 'qapi/events.py: add notational type hints'
>> 018/37: 'qapi/events.py: Move comments into docstrings'
>>
>> commands.py:
>>
>> 019/37: 'qapi/commands.py: Don't re-bind to variable of different type'
>> 020/37: 'qapi/commands.py: add notational type hints'
>> 021/37: 'qapi/commands.py: enable checking with mypy'
>>
>> source.py:
>>
>> 022/37: 'qapi/source.py: add notational type hints'
>> 023/37: 'qapi/source.py: delint with pylint'
>>
>> gen.py:
>>
>> 024/37: 'qapi/gen.py: Fix edge-case of _is_user_module'
>> 025/37: 'qapi/gen.py: add notational type hints'
>> 026/37: 'qapi/gen.py: Enable checking with mypy'
>> 027/37: 'qapi/gen.py: Remove unused parameter'
>> 028/37: 'qapi/gen.py: update write() to be more idiomatic'
>> 029/37: 'qapi/gen.py: delint with pylint'
>>
>> introspect.py:
>>
>> 030/37: 'qapi/introspect.py: Add a typed 'extra' structure'
>> 031/37: 'qapi/introspect.py: add _gen_features helper'
>> 032/37: 'qapi/introspect.py: create a typed 'Node' data structure'
>> 033/37: 'qapi/introspect.py: add notational type hints'
>>
>> types.py:
>>
>> 034/37: 'qapi/types.py: add notational type hints'
>> 035/37: 'qapi/types.py: remove one-letter variables'
>>
>> visit.py:
>>
>> 036/37: 'qapi/visit.py: remove unused parameters from gen_visit_object'
>> 037/37: 'qapi/visit.py: add notational type hints'
> 

Re: [PATCH 13/37] qapi/common.py: add notational type hints

[John Snow]

On 9/18/20 7:14 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/17/20 10:32 AM, Markus Armbruster wrote:
>>> Question on the subject line: what makes a type hint notational?
>>>
>>
>> My cover letter explains that every time I use this phrase, I mean to
>> state that "This patch adds exclusively type notations and makes no
>> functional changes to the runtime operation whatsoever."
>>
>> i.e. notations-only.
> 
> By the time I get to PATCH 13, details explained in the cover letter
> have been flushed from my memory.  Moreover, the cover letter won't make
> it into Git.  Best to repeat them right in the commit message.  Perhaps:
> 
>      qapi/common; Add type hints
> 
>      Type hints do not change behavior.
> 

ACK, done.

>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>>    scripts/qapi/common.py | 27 ++++++++++++++++-----------
>>>>    1 file changed, 16 insertions(+), 11 deletions(-)
>>>>
>>>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>>>> index 4c079755d3..af01348b35 100644
>>>> --- a/scripts/qapi/common.py
>>>> +++ b/scripts/qapi/common.py
>>>> @@ -12,6 +12,7 @@
>>>>    # See the COPYING file in the top-level directory.
>>>>      import re
>>>> +from typing import Optional, Union, Sequence
>>>>      
>>>>    EATSPACE = '\033EATSPACE.'
>>>> @@ -22,7 +23,7 @@
>>>>    # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
>>>>    # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
>>>>    # ENUM24_Name -> ENUM24_NAME
>>>> -def camel_to_upper(value):
>>>> +def camel_to_upper(value: str) -> str:
>>>>        c_fun_str = c_name(value, False)
>>>>        if value.isupper():
>>>>            return c_fun_str
>>>> @@ -41,7 +42,9 @@ def camel_to_upper(value):
>>>>        return new_name.lstrip('_').upper()
>>>>      
>>>> -def c_enum_const(type_name, const_name, prefix=None):
>>>> +def c_enum_const(type_name: str,
>>>> +                 const_name: str,
>>>> +                 prefix: Optional[str] = None) -> str:
>>>>        if prefix is not None:
>>>>            type_name = prefix
>>>>        return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
>>>> @@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
>>>>    # into substrings of a generated C function name.
>>>>    # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
>>>>    # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
>>>> -def c_name(name, protect=True):
>>>> +def c_name(name: str, protect: bool = True) -> str:
>>>>        # ANSI X3J11/88-090, 3.1.1
>>>>        c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
>>>>                         'default', 'do', 'double', 'else', 'enum', 'extern',
>>>> @@ -134,24 +137,24 @@ def pop(self, amount: int = 4) -> int:
>>>>      # Generate @code with @kwds interpolated.
>>>>    # Obey INDENT level, and strip EATSPACE.
>>>> -def cgen(code, **kwds):
>>>> +def cgen(code: str, **kwds: Union[str, int]) -> str:
>>> Hmm.
>>> The @kwds values can be anything, provided they match the conversion
>>> specifiers in @code:
>>>
>>>>        raw = code % kwds
>>> Your type hint adds a restriction that wasn't there before.
>>> Is there a better way?
>>>
>>
>> Maybe there are format-like type annotation tricks you can do to
>> enforce this, but I did not research them. I tried to resist
>> "improving" our usage of the old % formatter prematurely. I may do a
>> wholesale f-string conversion at some point, but not now, it's not
>> important.
>>
>> In practice, we pass strings and integers. This typing *is*
>> artificially restrictive, though. We can declare the type to be "Any"
>> and allow the function to fail or succeed at runtime if you'd prefer.
>>

I went with the 'object' type in new revisions.

>>>>        if INDENT:
>>>>            raw, _ = re.subn(r'^(?!(#|$))', str(INDENT), raw, flags=re.MULTILINE)
>>>>        return re.sub(re.escape(EATSPACE) + r' *', '', raw)
>>>>      
>>>> -def mcgen(code, **kwds):
>>>> +def mcgen(code: str, **kwds: Union[str, int]) -> str:
>>> Likewise.
>>>
>>
>> Unresearched idea: It's possible that we can subclass the
>> string.Formatter class and extend it to perform our special variable
>> replacements (chomping EATSPACE, etc.)
>>
>> And *maybe* because it inherits from the standard formatter, we would
>> benefit from any analysis Mypy performs on such things.
>>
>> Basically, replace mcgen/cgen with class CFormatter(string.Formatter).
>>
>> (maybe. assume that none of what I just said will work or is feasible.)
> 
> Sounds worth exploring.  No need to do it now, of course.
> 

Re: [PATCH 09/37] qapi/common.py: Add indent manager

[John Snow]

On 9/18/20 6:55 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:

>> We'll get to them in due time. For now, please admire the lipstick.
> 
> If I take off my glasses and step six feet back, I just might be able to
> overlook it.
> 

I consider writing a nice __repr__ good habit, I'd prefer not to delete 
it just because the rest of our code doesn't do so yet. (Give me time.)

I spend a lot of my time in the interactive python console: having nice 
__repr__ methods is a good habit, not an unsightly blemish.

>>>>>> +    def pop(self, amount: int = 4) -> int:
>>>>>> +        """Pop `amount` spaces off of the indent, default four."""
>>>>>> +        if self._level < amount:
>>>>>> +            raise ArithmeticError(
>>>>>> +                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
>>> I think assert(amount <= self._level) would do just fine.
>>>
>>
>> Secretly, asserts can be disabled in Python just like they can in C code.
> 
> There are compilers that let you switch off array bounds checking.
> Would you advocate manual bounds checking to protect people from their
> own folly?
> 
>> My habit: if it's something that should already be true given the
>> nature of how the code is laid out, use an assertion. If I am
>> preventing an erroneous state (Especially from callers in other
>> modules), explicitly raise an exception.
> 
> I check function preconditions ruthlessly with assert.  There's no sane
> way to recover anyway.
> 
> Without a way to recover, the only benefit is crashing more prettily.
> If the error is six levels deep in a some fancy-pants framework, then
> prettier crashes might actually help someone finding the error slightly
> faster.  But it ain't.
> 
> My final argument is local consistency: use of assertions to check
> preconditions is pervasive in scripts/qapi/.
> 

You're right that there's no safe recovery from an error such as this. 
The only actual difference is whether you see AssertionError or 
ArithmeticError.

One can be disabled (But you rightly shouldn't), the other can't. One 
has more semantic meaning and information to it.

I prefer what I've currently written.

--js

Re: [PATCH 06/37] qapi: delint using flake8

[John Snow]

On 9/18/20 6:33 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/17/20 3:54 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> On 9/16/20 8:12 AM, Markus Armbruster wrote:
>>>>> John Snow <jsnow@redhat.com> writes:
>>>>>
>>>>>> Petty style guide fixes and line length enforcement.  Not a big win, not
>>>>>> a big loss, but flake8 passes 100% on the qapi module, which gives us an
>>>>>> easy baseline to enforce hereafter.
>>>>>>
>>>>>> Signed-off-by: John Snow <jsnow@redhat.com>
> [...]
>>>>>> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
>>>>>> index e1df0e341f..2e4b4de0fa 100644
>>>>>> --- a/scripts/qapi/commands.py
>>>>>> +++ b/scripts/qapi/commands.py
>>>>>> @@ -69,7 +69,8 @@ def gen_call(name, arg_type, boxed, ret_type):
>>>>>>     def gen_marshal_output(ret_type):
>>>>>>         return mcgen('''
>>>>>>     -static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
>>>>>> QObject **ret_out, Error **errp)
>>>>>> +static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
>>>>>> +                                          Error **errp)
>>>>> The continued parameter list may become misalignd in generated C.
>>>>> E.g.
>>>>>        static void qmp_marshal_output_BlockInfoList(BlockInfoList *ret_in, QObject **ret_out,
>>>>>                                                  Error **errp)
>>>>>        {
>>>>>        ...
>>>>>        }
>>>>> Do we care?
>>>>>
>>>>
>>>> Yeah, I don't know. Do we?
>>> I care, but I also care for automated style checks.
>>>
>>>> It actually seemed more annoying to try and get flake8 to make an
>>>> exception for these handful of examples.
>>>>
>>>> Path of least resistance led me here, but I can try and appease both
>>>> systems if you'd prefer.
>>> Up to now, I ran the style checkers manually, and this was just one
>>> of
>>> several complaints to ignore, so I left the code alone.
>>> If it gets in the way of running them automatically, and messing up
>>> the
>>> generated code slightly is the easiest way to get it out of the way,
>>> then I can accept the slight mess.
>>>
>>
>> I changed this a little to put all the args on the next line, which is
>> slightly unusual but works okay.
> 
> I think it's slightly more unusual than the non-matching indentation
> was.
> 
> Yet another way to skin this cat:
> 
>      static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
>                                      QObject **ret_out, Error **errp)
> 
> Now the second line is not aligned with the left parenthesis both in the
> Python source and in the generated C.
> 

Your wish is my command!
(Except in the other replies where I am arguing with you.)

>> I think that's a fine middle ground, because the alternative (to me)
>> is to start using abstracted code generation tokens in a tree
>> structure, etc etc etc.
>>
>> Embedded templates are always gonna look kinda nasty, I think, because
>> you're trying to fight style guidelines in two languages
>> simultaneously and it's never gonna quite work out exactly how you
>> want without some pretty complex abstraction mechanisms that are well
>> beyond the power we need right now.
> 
> The thought "screw this, pile the output through /usr/bin/indent" has
> crossed my mind more than once.
> 
> 

Bigger fish to fry, but with other languages like rust looming, making 
the core generation facilities nicer might be ...nice. Not for this series.

Two approaches in general would make sense to me:

1. Building an AST for C instead of strings, and rendering the AST.
2. Writing a templating engine that doesn't break the Python indentation 
flow by hoisting them into module constants and improving the rendering 
logic.

--js

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

[John Snow]

On 9/18/20 7:55 AM, Markus Armbruster wrote:
> Ignorant question: why does this come after PATCH 13 "qapi/common.py:
> add notational type hints", but before all the other patches adding type
> hints?
> 
> John Snow <jsnow@redhat.com> writes:
> 
>> Fix two very minor issues, and then establish a mypy type-checking
>> baseline.
>>
>> Like pylint, this should be run from the folder above:
>>
>>   > mypy --config-file=qapi/mypy.ini qapi/
> 
> I get:
> 
>      $ mypy --config-file qapi/mypy.ini qapi
>      qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
>      qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
>      Found 1 error in 1 file (checked 18 source files)
> 
> Is this expected?
> 
> In case it matters:
> 
>      $ mypy --version
>      mypy 0.761
> 

(Warning; FiSH and stgit ahead)

cd ~/src/qemu/scripts
pipenv --python 3.6
pipenv shell
pip install pylint flake8

### Testing mypy 0.770

pip install mypy==0.770
stg goto qapi-establish-mypy-type

while true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/; 
and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end

pip uninstall mypy

###



0.782 - OK
0.770 - OK
0.760 - FAIL, Fixable*
0.750 - OK*
0.740 - OK*
0.730 - OK*
0.720 - OK*
0.710 - OK** (Does not recognize 'no_implicit_reexport' option)
0.700 - OK*** (Not compatible with bleeding edge pylint/flake8)
0.670 - OK***
0.660 - OK***
0.650 - OK***
0.641 - OK***
0.630 - Fails again.



0.760 doesn't support strict in the config file (It needs component 
options), and it wants a few extra annotations where its inference power 
is weaker. Well, easy enough to fix up.

0.630 fails again and insists that __init__ should have a return type 
annotation of None. Modern mypy is smart enough to know that's what that 
type is supposed to be. Arbitrarily, this is my cutoff for what seems 
reasonable to even want to support.

I still find the lack of "strict=true" in the config file irritating and 
might ask to target 0.770 or newer. There should be no reason we can't 
install that in a venv for CI to chew on.

Humbly ask I take the lazy way out and document that we support mypy >= 
0.770.

--js

Re: [PATCH 06/37] qapi: delint using flake8

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/18/20 6:33 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
[...]
>>> Embedded templates are always gonna look kinda nasty, I think, because
>>> you're trying to fight style guidelines in two languages
>>> simultaneously and it's never gonna quite work out exactly how you
>>> want without some pretty complex abstraction mechanisms that are well
>>> beyond the power we need right now.
>> The thought "screw this, pile the output through /usr/bin/indent"
>> has
>> crossed my mind more than once.
>> 
>
> Bigger fish to fry, but with other languages like rust looming, making
> the core generation facilities nicer might be ...nice. Not for this
> series.

Definitely not.

The way we generate C now is (close to) the stupidest way that could
possibly work.  Stupid is *good*, until it creates actual problems.

Any improvement will have to balance readability vs. hackability.

What we have now is often hard to read, but pretty easy to hack.  May
sound like a contradiction, but it's been my experience.

> Two approaches in general would make sense to me:
>
> 1. Building an AST for C instead of strings, and rendering the AST.

I fear the need to define an AST for C makes this uneconomical.

I've missed the ease of generating Lisp in Lisp many times.

> 2. Writing a templating engine that doesn't break the Python
> indentation flow by hoisting them into module constants and improving
> the rendering logic.

Too vague for me to judge, happy to look at PoC patches.

Re: [PATCH 09/37] qapi/common.py: Add indent manager

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/18/20 6:55 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>
>>> We'll get to them in due time. For now, please admire the lipstick.
>> If I take off my glasses and step six feet back, I just might be
>> able to
>> overlook it.
>> 
>
> I consider writing a nice __repr__ good habit, I'd prefer not to
> delete it just because the rest of our code doesn't do so yet. (Give
> me time.)
>
> I spend a lot of my time in the interactive python console: having
> nice __repr__ methods is a good habit, not an unsightly blemish.
>
>>>>>>> +    def pop(self, amount: int = 4) -> int:
>>>>>>> +        """Pop `amount` spaces off of the indent, default four."""
>>>>>>> +        if self._level < amount:
>>>>>>> +            raise ArithmeticError(
>>>>>>> +                "Can't pop {:d} spaces from {:s}".format(amount, repr(self)))
>>>> I think assert(amount <= self._level) would do just fine.
>>>>
>>>
>>> Secretly, asserts can be disabled in Python just like they can in C code.
>> There are compilers that let you switch off array bounds checking.
>> Would you advocate manual bounds checking to protect people from their
>> own folly?
>> 
>>> My habit: if it's something that should already be true given the
>>> nature of how the code is laid out, use an assertion. If I am
>>> preventing an erroneous state (Especially from callers in other
>>> modules), explicitly raise an exception.
>> I check function preconditions ruthlessly with assert.  There's no
>> sane
>> way to recover anyway.
>> Without a way to recover, the only benefit is crashing more
>> prettily.
>> If the error is six levels deep in a some fancy-pants framework, then
>> prettier crashes might actually help someone finding the error slightly
>> faster.  But it ain't.
>> My final argument is local consistency: use of assertions to check
>> preconditions is pervasive in scripts/qapi/.
>> 
>
> You're right that there's no safe recovery from an error such as
> this. The only actual difference is whether you see AssertionError or 
> ArithmeticError.
>
> One can be disabled (But you rightly shouldn't), the other can't. One
> has more semantic meaning and information to it.

YAGNI.

> I prefer what I've currently written.

Where personal preference collides with local consistency, I'm with
local consistency.

You can get the two in line: change everything to your preference.

You signalled intent to do that for __repr__(): "Give me time".
Alright, having such __repr__() is obviously more important / useful to
you than avoiding the extra code is to me.

I received no such signal for checking preconditions.  Good, because I'd
go "are you serious?" :)

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/18/20 7:55 AM, Markus Armbruster wrote:
>> Ignorant question: why does this come after PATCH 13 "qapi/common.py:
>> add notational type hints", but before all the other patches adding type
>> hints?
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Fix two very minor issues, and then establish a mypy type-checking
>>> baseline.
>>>
>>> Like pylint, this should be run from the folder above:
>>>
>>>   > mypy --config-file=qapi/mypy.ini qapi/
>> I get:
>>      $ mypy --config-file qapi/mypy.ini qapi
>>      qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
>>      qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
>>      Found 1 error in 1 file (checked 18 source files)
>> Is this expected?
>> 
>
> Nope.
>
>> In case it matters:
>>      $ mypy --version
>>      mypy 0.761
>> 
>
> I am using mypy 0.782.
>
> I will investigate to see if there is an *easy* win to allow older
> versions to work.
>
> In the meantime, please consider trying this:
>
> pip install --user mypy==0.782
> ~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/

I'll consider dragging my feet until upgrading Fedora gives it to me for
free.

The less I interact with package managers, the happier I am.

>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>   scripts/qapi/doc.py    |  3 +-
>>>   scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
>>>   scripts/qapi/schema.py |  3 +-
>>>   3 files changed, 69 insertions(+), 2 deletions(-)
>>>   create mode 100644 scripts/qapi/mypy.ini
>>>
>>> diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
>>> index cbf7076ed9..70f7cdfaa6 100644
>>> --- a/scripts/qapi/doc.py
>>> +++ b/scripts/qapi/doc.py
>>> @@ -5,7 +5,8 @@
>>>   """This script produces the documentation of a qapi schema in texinfo format"""
>>>     import re
>>> -from .gen import QAPIGenDoc, QAPISchemaVisitor
>>> +from .gen import QAPIGenDoc
>>> +from .schema import QAPISchemaVisitor
>> Your mypy doesn't like such lazy imports?  Mine seems not to care.
>> 
>
> Yeah, it specifically complained that no such definition existed in
> that file.

I sense a certain wobbliness in mypy.  Perhaps to be expected from a
tool with major version zero.  There's a risk that developers' local
"make check" and our gating CI differ too much.  We'll see.

>>>     
>>>   MSG_FMT = """
>>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>>> new file mode 100644
>>> index 0000000000..a0f2365a53
>>> --- /dev/null
>>> +++ b/scripts/qapi/mypy.ini
>>> @@ -0,0 +1,65 @@
>>> +[mypy]
>>> +strict = True
>>> +strict_optional = False
>>> +disallow_untyped_calls = False
>>> +python_version = 3.6
>>> +
>>> +[mypy-qapi.commands]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.doc]
>>> +disallow_subclassing_any = False
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +
>>> +[mypy-qapi.error]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.events]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.expr]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.gen]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.introspect]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.parser]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.schema]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.source]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.types]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>>> +
>>> +[mypy-qapi.visit]
>>> +disallow_untyped_defs = False
>>> +disallow_incomplete_defs = False
>>> +check_untyped_defs = False
>> Greek to me.  I'll learn in due time.
>> 
>
> I am using these options:
>
> --strict, which is effectively -Wall.
>
> --no-strict-optional, which disables type checking errors on conflict
>   between Optional[T] and T. Namely, when you initialize a class field
>  to None and set that variable after initialization, callers must be 
> prepared to see if that field was None or not. We do that effectively
> nowhere.
>
> As Python will surely explode in a noticeable way if we got an
> unexpected 'None', I am just suppressing these warnings "for now".

Okay.

We may want to rethink how we define and initialize these variables.  We
initialize mostly to keep pylint happy.  We initialize to None precisely
to make use before the real initialization explode.

> --allow-untyped-calls silences errors in files that have calls to
>   functions in files I still have not typed. By the end of the series, 
> this option goes away, because there's nothing untyped left.
>
>
> For each untyped file, we are actually starting with all of the above
> options and then layering these options on top. Any egregious typing 
> errors present in these "ignored" files will be spotted.
>
> To get the bad files to pass, we only need three options:
>
> allow untyped defs -- Simply permits us to have functions without
> annotations.
>
> allow incomplete defs -- allows functions that are only partially typed.
>
> check untyped defs = False -- Don't try to type check untyped definitions.

Thanks!

[...]

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/18/20 7:55 AM, Markus Armbruster wrote:
>> Ignorant question: why does this come after PATCH 13 "qapi/common.py:
>> add notational type hints", but before all the other patches adding type
>> hints?
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> Fix two very minor issues, and then establish a mypy type-checking
>>> baseline.
>>>
>>> Like pylint, this should be run from the folder above:
>>>
>>>   > mypy --config-file=qapi/mypy.ini qapi/
>> I get:
>>      $ mypy --config-file qapi/mypy.ini qapi
>>      qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
>>      qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
>>      Found 1 error in 1 file (checked 18 source files)
>> Is this expected?
>> In case it matters:
>>      $ mypy --version
>>      mypy 0.761
>> 
>
> (Warning; FiSH and stgit ahead)
>
> cd ~/src/qemu/scripts
> pipenv --python 3.6
> pipenv shell
> pip install pylint flake8
>
> ### Testing mypy 0.770
>
> pip install mypy==0.770
> stg goto qapi-establish-mypy-type
>
> while true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/;
> and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end
>
> pip uninstall mypy
>
> ###
>
>
>
> 0.782 - OK
> 0.770 - OK
> 0.760 - FAIL, Fixable*
> 0.750 - OK*
> 0.740 - OK*
> 0.730 - OK*
> 0.720 - OK*
> 0.710 - OK** (Does not recognize 'no_implicit_reexport' option)
> 0.700 - OK*** (Not compatible with bleeding edge pylint/flake8)
> 0.670 - OK***
> 0.660 - OK***
> 0.650 - OK***
> 0.641 - OK***
> 0.630 - Fails again.
>
>
>
> 0.760 doesn't support strict in the config file (It needs component
> options), and it wants a few extra annotations where its inference
> power is weaker. Well, easy enough to fix up.
>
> 0.630 fails again and insists that __init__ should have a return type
> annotation of None. Modern mypy is smart enough to know that's what
> that type is supposed to be. Arbitrarily, this is my cutoff for what
> seems reasonable to even want to support.
>
> I still find the lack of "strict=true" in the config file irritating
> and might ask to target 0.770 or newer. There should be no reason we
> can't install that in a venv for CI to chew on.
>
> Humbly ask I take the lazy way out and document that we support mypy
> >= 0.770.

Our "supported build platforms" policy puts hard limits on the minimum
versions for tools the build requires.

Mypy is not such a tool.  I hope we get to the point where we can have
"make check" run it, but skipping the test when we don't have a
sufficiently modern mypy feels okay to me, as long as our gating CI
still guards the master branch.

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

[John Snow]

On 9/21/20 4:05 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/18/20 7:55 AM, Markus Armbruster wrote:
>>> Ignorant question: why does this come after PATCH 13 "qapi/common.py:
>>> add notational type hints", but before all the other patches adding type
>>> hints?
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> Fix two very minor issues, and then establish a mypy type-checking
>>>> baseline.
>>>>
>>>> Like pylint, this should be run from the folder above:
>>>>
>>>>    > mypy --config-file=qapi/mypy.ini qapi/
>>> I get:
>>>       $ mypy --config-file qapi/mypy.ini qapi
>>>       qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
>>>       qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
>>>       Found 1 error in 1 file (checked 18 source files)
>>> Is this expected?
>>>
>>
>> Nope.
>>
>>> In case it matters:
>>>       $ mypy --version
>>>       mypy 0.761
>>>
>>
>> I am using mypy 0.782.
>>
>> I will investigate to see if there is an *easy* win to allow older
>> versions to work.
>>
>> In the meantime, please consider trying this:
>>
>> pip install --user mypy==0.782
>> ~/.local/bin/mypy --config-file=qapi/mypy.ini qapi/
> 
> I'll consider dragging my feet until upgrading Fedora gives it to me for
> free.
> 
> The less I interact with package managers, the happier I am.
> 

I'll probably be working on making sure CI runs in a virtual environment 
with specifically pinned versions.

We don't need mypy to run or build, so I have been less worried about 
requiring bleeding edge versions for development and regression testing.

>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>>    scripts/qapi/doc.py    |  3 +-
>>>>    scripts/qapi/mypy.ini  | 65 ++++++++++++++++++++++++++++++++++++++++++
>>>>    scripts/qapi/schema.py |  3 +-
>>>>    3 files changed, 69 insertions(+), 2 deletions(-)
>>>>    create mode 100644 scripts/qapi/mypy.ini
>>>>
>>>> diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
>>>> index cbf7076ed9..70f7cdfaa6 100644
>>>> --- a/scripts/qapi/doc.py
>>>> +++ b/scripts/qapi/doc.py
>>>> @@ -5,7 +5,8 @@
>>>>    """This script produces the documentation of a qapi schema in texinfo format"""
>>>>      import re
>>>> -from .gen import QAPIGenDoc, QAPISchemaVisitor
>>>> +from .gen import QAPIGenDoc
>>>> +from .schema import QAPISchemaVisitor
>>> Your mypy doesn't like such lazy imports?  Mine seems not to care.
>>>
>>
>> Yeah, it specifically complained that no such definition existed in
>> that file.
> 
> I sense a certain wobbliness in mypy.  Perhaps to be expected from a
> tool with major version zero.  There's a risk that developers' local
> "make check" and our gating CI differ too much.  We'll see.
> 

It is indeed very cutting edge. It wasn't really viable to use until 
Python 3.6.

>>>>      
>>>>    MSG_FMT = """
>>>> diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
>>>> new file mode 100644
>>>> index 0000000000..a0f2365a53
>>>> --- /dev/null
>>>> +++ b/scripts/qapi/mypy.ini
>>>> @@ -0,0 +1,65 @@
>>>> +[mypy]
>>>> +strict = True
>>>> +strict_optional = False
>>>> +disallow_untyped_calls = False
>>>> +python_version = 3.6
>>>> +
>>>> +[mypy-qapi.commands]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.doc]
>>>> +disallow_subclassing_any = False
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +
>>>> +[mypy-qapi.error]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.events]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.expr]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.gen]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.introspect]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.parser]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.schema]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.source]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.types]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>>> +
>>>> +[mypy-qapi.visit]
>>>> +disallow_untyped_defs = False
>>>> +disallow_incomplete_defs = False
>>>> +check_untyped_defs = False
>>> Greek to me.  I'll learn in due time.
>>>
>>
>> I am using these options:
>>
>> --strict, which is effectively -Wall.
>>
>> --no-strict-optional, which disables type checking errors on conflict
>>    between Optional[T] and T. Namely, when you initialize a class field
>>   to None and set that variable after initialization, callers must be
>> prepared to see if that field was None or not. We do that effectively
>> nowhere.
>>
>> As Python will surely explode in a noticeable way if we got an
>> unexpected 'None', I am just suppressing these warnings "for now".
> 
> Okay.
> 
> We may want to rethink how we define and initialize these variables.  We
> initialize mostly to keep pylint happy.  We initialize to None precisely
> to make use before the real initialization explode.
> 

The pattern we have is roughly correct; it's just that we rely on that 
implicit explosion, which mypy will warn about if I don't disable this 
option.

It's not worth addressing in our first pass, but if you look at 
schema.py's @property ifcond code, that's the type of thing that would 
"fix" this warning -- basically access shims that add a "real" error 
message when it's None, but convert the return type from Optional[T] to T.

It's later stuff.

>> --allow-untyped-calls silences errors in files that have calls to
>>    functions in files I still have not typed. By the end of the series,
>> this option goes away, because there's nothing untyped left.
>>
>>
>> For each untyped file, we are actually starting with all of the above
>> options and then layering these options on top. Any egregious typing
>> errors present in these "ignored" files will be spotted.
>>
>> To get the bad files to pass, we only need three options:
>>
>> allow untyped defs -- Simply permits us to have functions without
>> annotations.
>>
>> allow incomplete defs -- allows functions that are only partially typed.
>>
>> check untyped defs = False -- Don't try to type check untyped definitions.
> 
> Thanks!
> 
> [...]
> 

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

[John Snow]

On 9/21/20 4:05 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/18/20 7:55 AM, Markus Armbruster wrote:
>>> Ignorant question: why does this come after PATCH 13 "qapi/common.py:
>>> add notational type hints", but before all the other patches adding type
>>> hints?
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> Fix two very minor issues, and then establish a mypy type-checking
>>>> baseline.
>>>>
>>>> Like pylint, this should be run from the folder above:
>>>>
>>>>    > mypy --config-file=qapi/mypy.ini qapi/
>>> I get:
>>>       $ mypy --config-file qapi/mypy.ini qapi
>>>       qapi/mypy.ini: [mypy]: Strict mode is not supported in configuration files: specify individual flags instead (see 'mypy -h' for the list of flags enabled in strict mode)
>>>       qapi/types.py:29: error: Need type annotation for 'objects_seen' (hint: "objects_seen: Set[<type>] = ...")
>>>       Found 1 error in 1 file (checked 18 source files)
>>> Is this expected?
>>> In case it matters:
>>>       $ mypy --version
>>>       mypy 0.761
>>>
>>
>> (Warning; FiSH and stgit ahead)
>>
>> cd ~/src/qemu/scripts
>> pipenv --python 3.6
>> pipenv shell
>> pip install pylint flake8
>>
>> ### Testing mypy 0.770
>>
>> pip install mypy==0.770
>> stg goto qapi-establish-mypy-type
>>
>> while true; and flake8 qapi/; and pylint --rcfile=qapi/pylintrc qapi/;
>> and mypy --config-file=qapi/mypy.ini qapi/; and stg push; end
>>
>> pip uninstall mypy
>>
>> ###
>>
>>
>>
>> 0.782 - OK
>> 0.770 - OK
>> 0.760 - FAIL, Fixable*
>> 0.750 - OK*
>> 0.740 - OK*
>> 0.730 - OK*
>> 0.720 - OK*
>> 0.710 - OK** (Does not recognize 'no_implicit_reexport' option)
>> 0.700 - OK*** (Not compatible with bleeding edge pylint/flake8)
>> 0.670 - OK***
>> 0.660 - OK***
>> 0.650 - OK***
>> 0.641 - OK***
>> 0.630 - Fails again.
>>
>>
>>
>> 0.760 doesn't support strict in the config file (It needs component
>> options), and it wants a few extra annotations where its inference
>> power is weaker. Well, easy enough to fix up.
>>
>> 0.630 fails again and insists that __init__ should have a return type
>> annotation of None. Modern mypy is smart enough to know that's what
>> that type is supposed to be. Arbitrarily, this is my cutoff for what
>> seems reasonable to even want to support.
>>
>> I still find the lack of "strict=true" in the config file irritating
>> and might ask to target 0.770 or newer. There should be no reason we
>> can't install that in a venv for CI to chew on.
>>
>> Humbly ask I take the lazy way out and document that we support mypy
>>> = 0.770.
> 
> Our "supported build platforms" policy puts hard limits on the minimum
> versions for tools the build requires.
> 
> Mypy is not such a tool.  I hope we get to the point where we can have
> "make check" run it, but skipping the test when we don't have a
> sufficiently modern mypy feels okay to me, as long as our gating CI
> still guards the master branch.
> 

Yes, that's been my view. These tools *are* new and they *do* change 
often. Chasing a wide compatibility window with them is prohibitively 
difficult. Luckily, they're not needed for running the packages at all.

If I do add linting to 'make check', it is going to be through a virtual 
environment that deploys *specific versions* of these tools, and it will 
happen transparently. The VM tests already do this.

--js

Re: [PATCH 06/37] qapi: delint using flake8

[John Snow]

On 9/21/20 3:31 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/18/20 6:33 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
> [...]

>> 2. Writing a templating engine that doesn't break the Python
>> indentation flow by hoisting them into module constants and improving
>> the rendering logic.
> 
> Too vague for me to judge, happy to look at PoC patches.
> 

Okay, some other day. Not important enough presently. It's just hard to 
stop brainstorming ideas as I learn about and touch different areas of code.

--js

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/17/20 3:14 PM, Eduardo Habkost wrote:
>> On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
>> [...]
>>> Having said this, I have not found any tool to date that actually *checks*
>>> these comments for consistency. The pycharm IDE interactively highlights
>>> them when it senses that you have made a mistake, but that cannot be worked
>>> into our CI process that I know of - it's a proprietary checker.
>>>
>>> So right now, they're just plaintext that I am writing to approximate the
>>> Sphinx style until such time as I enable autodoc for the module and
>>> fine-tune the actual formatting and so on.

You are deliberately trying to "approximate" Sphinx style, and ...

>> After applying this series, I only had to make two small tweaks
>> to make Sphinx + autodoc happy with the docstrings you wrote.
>> With the following patch, "make sphinxdocs" will generate the
>> QAPI Python module documentation at docs/devel/qapi.html.
>> I had to explicitly skip qapi/doc.py because autodoc thinks the
>> string constants are documentation strings.
>> 
>
> Awesome!

... actually almost nail it.

Please mention your choice of style in the commit message.

> I think I am going to delay explicitly pursuing writing a manual for
> the QAPI parser for now, but it's good to know I am not too far
> off. I'm going to target the mypy conversions first, because they can
> be invasive and full of churn.
>
> When I get there, though ... I am thinking I should add this as
> Devel/QAPI Parser.

Doing "actually Sphinx style" instead of "an approximation of Sphinx
style" would reduce churn later on.  So, if it's not too distracting...

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

[John Snow]

On 9/24/20 11:06 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/17/20 3:14 PM, Eduardo Habkost wrote:
>>> On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
>>> [...]
>>>> Having said this, I have not found any tool to date that actually *checks*
>>>> these comments for consistency. The pycharm IDE interactively highlights
>>>> them when it senses that you have made a mistake, but that cannot be worked
>>>> into our CI process that I know of - it's a proprietary checker.
>>>>
>>>> So right now, they're just plaintext that I am writing to approximate the
>>>> Sphinx style until such time as I enable autodoc for the module and
>>>> fine-tune the actual formatting and so on.
> 
> You are deliberately trying to "approximate" Sphinx style, and ...
> 
>>> After applying this series, I only had to make two small tweaks
>>> to make Sphinx + autodoc happy with the docstrings you wrote.
>>> With the following patch, "make sphinxdocs" will generate the
>>> QAPI Python module documentation at docs/devel/qapi.html.
>>> I had to explicitly skip qapi/doc.py because autodoc thinks the
>>> string constants are documentation strings.
>>>
>>
>> Awesome!
> 
> ... actually almost nail it.
> 
> Please mention your choice of style in the commit message.
> 

OK, I'll try to summarize it. I guess I'll do it in this commit message, 
but it's not likely to be too visible in the future that way, depending 
on how you run git blame and what you're looking at.

I want to resurface my other python style patches soon; perhaps a python 
coding style document should go in alongside those patches.

>> I think I am going to delay explicitly pursuing writing a manual for
>> the QAPI parser for now, but it's good to know I am not too far
>> off. I'm going to target the mypy conversions first, because they can
>> be invasive and full of churn.
>>
>> When I get there, though ... I am thinking I should add this as
>> Devel/QAPI Parser.
> 
> Doing "actually Sphinx style" instead of "an approximation of Sphinx
> style" would reduce churn later on.  So, if it's not too distracting...
> 

Yes, I just mean in terms of rebasing, docstrings and signatures fight 
with each other for context and make re-spinning 125 patches a major 
chore. I wasn't prepared to have the debate on if we wanted to add 
Python code into the Sphinx generator and have that entire discussion yet.

So, I figured it would be a separate series afterwards. I mentioned 
somewhere else that I anticipated doing it when I removed the 
"missing-docstring" warning.

I will investigate doing it (using Eduardo's patches) as a starting 
point while reviews continue to filter in. If I figure it out in time, I 
might squash the formatting changes in, but leave the sphinx enablement 
patches themselves out.

--js

Re: [PATCH 16/37] qapi: establish mypy type-checking baseline

[Eduardo Habkost]

On Mon, Sep 21, 2020 at 10:05:24AM +0200, Markus Armbruster wrote:
[...]
> I sense a certain wobbliness in mypy.  Perhaps to be expected from a
> tool with major version zero.  There's a risk that developers' local
> "make check" and our gating CI differ too much.  We'll see.

This possibility shouldn't exist.  If you don't have the required
version in your system, the validation script must know how to
create a virtualenv and install the right version inside it.

-- 
Eduardo

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 9/24/20 11:06 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 9/17/20 3:14 PM, Eduardo Habkost wrote:
>>>> On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
>>>> [...]
>>>>> Having said this, I have not found any tool to date that actually *checks*
>>>>> these comments for consistency. The pycharm IDE interactively highlights
>>>>> them when it senses that you have made a mistake, but that cannot be worked
>>>>> into our CI process that I know of - it's a proprietary checker.
>>>>>
>>>>> So right now, they're just plaintext that I am writing to approximate the
>>>>> Sphinx style until such time as I enable autodoc for the module and
>>>>> fine-tune the actual formatting and so on.
>> You are deliberately trying to "approximate" Sphinx style, and ...
>> 
>>>> After applying this series, I only had to make two small tweaks
>>>> to make Sphinx + autodoc happy with the docstrings you wrote.
>>>> With the following patch, "make sphinxdocs" will generate the
>>>> QAPI Python module documentation at docs/devel/qapi.html.
>>>> I had to explicitly skip qapi/doc.py because autodoc thinks the
>>>> string constants are documentation strings.
>>>>
>>>
>>> Awesome!
>> ... actually almost nail it.
>> Please mention your choice of style in the commit message.
>> 
>
> OK, I'll try to summarize it. I guess I'll do it in this commit
> message, but it's not likely to be too visible in the future that way,
> depending on how you run git blame and what you're looking at.

Thanks!

> I want to resurface my other python style patches soon; perhaps a
> python coding style document should go in alongside those patches.
>
>>> I think I am going to delay explicitly pursuing writing a manual for
>>> the QAPI parser for now, but it's good to know I am not too far
>>> off. I'm going to target the mypy conversions first, because they can
>>> be invasive and full of churn.
>>>
>>> When I get there, though ... I am thinking I should add this as
>>> Devel/QAPI Parser.
>> Doing "actually Sphinx style" instead of "an approximation of Sphinx
>> style" would reduce churn later on.  So, if it's not too distracting...
>> 
>
> Yes, I just mean in terms of rebasing, docstrings and signatures fight
> with each other for context and make re-spinning 125 patches a major 
> chore. I wasn't prepared to have the debate on if we wanted to add
> Python code into the Sphinx generator and have that entire discussion
> yet.
>
> So, I figured it would be a separate series afterwards. I mentioned
> somewhere else that I anticipated doing it when I removed the 
> "missing-docstring" warning.
>
> I will investigate doing it (using Eduardo's patches) as a starting
> point while reviews continue to filter in. If I figure it out in time,
> I might squash the formatting changes in, but leave the sphinx
> enablement patches themselves out.

Use your judgement.

Re: [PATCH 14/37] qapi/common.py: Move comments into docstrings

[John Snow]

On 9/25/20 3:49 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 9/24/20 11:06 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> On 9/17/20 3:14 PM, Eduardo Habkost wrote:
>>>>> On Thu, Sep 17, 2020 at 02:44:53PM -0400, John Snow wrote:
>>>>> [...]
>>>>>> Having said this, I have not found any tool to date that actually *checks*
>>>>>> these comments for consistency. The pycharm IDE interactively highlights
>>>>>> them when it senses that you have made a mistake, but that cannot be worked
>>>>>> into our CI process that I know of - it's a proprietary checker.
>>>>>>
>>>>>> So right now, they're just plaintext that I am writing to approximate the
>>>>>> Sphinx style until such time as I enable autodoc for the module and
>>>>>> fine-tune the actual formatting and so on.
>>> You are deliberately trying to "approximate" Sphinx style, and ...
>>>
>>>>> After applying this series, I only had to make two small tweaks
>>>>> to make Sphinx + autodoc happy with the docstrings you wrote.
>>>>> With the following patch, "make sphinxdocs" will generate the
>>>>> QAPI Python module documentation at docs/devel/qapi.html.
>>>>> I had to explicitly skip qapi/doc.py because autodoc thinks the
>>>>> string constants are documentation strings.
>>>>>
>>>>
>>>> Awesome!
>>> ... actually almost nail it.
>>> Please mention your choice of style in the commit message.
>>>
>>
>> OK, I'll try to summarize it. I guess I'll do it in this commit
>> message, but it's not likely to be too visible in the future that way,
>> depending on how you run git blame and what you're looking at.
> 
> Thanks!
> 
>> I want to resurface my other python style patches soon; perhaps a
>> python coding style document should go in alongside those patches.
>>
>>>> I think I am going to delay explicitly pursuing writing a manual for
>>>> the QAPI parser for now, but it's good to know I am not too far
>>>> off. I'm going to target the mypy conversions first, because they can
>>>> be invasive and full of churn.
>>>>
>>>> When I get there, though ... I am thinking I should add this as
>>>> Devel/QAPI Parser.
>>> Doing "actually Sphinx style" instead of "an approximation of Sphinx
>>> style" would reduce churn later on.  So, if it's not too distracting...
>>>
>>
>> Yes, I just mean in terms of rebasing, docstrings and signatures fight
>> with each other for context and make re-spinning 125 patches a major
>> chore. I wasn't prepared to have the debate on if we wanted to add
>> Python code into the Sphinx generator and have that entire discussion
>> yet.
>>
>> So, I figured it would be a separate series afterwards. I mentioned
>> somewhere else that I anticipated doing it when I removed the
>> "missing-docstring" warning.
>>
>> I will investigate doing it (using Eduardo's patches) as a starting
>> point while reviews continue to filter in. If I figure it out in time,
>> I might squash the formatting changes in, but leave the sphinx
>> enablement patches themselves out.
> 

I wound up figuring it out before I sent V3 out :)

> Use your judgement.
> 

Yep. You can see how this played out in V3: I ensure that nothing is 
"wrong" but I haven't converted the entire submodule yet.

A thing to clarify for the list as well: Nothing validates these 
docstrings in the content-aware sense; nothing makes sure you documented 
the right parameter names, return values, exceptions, etc.

The only validation we can perform automatically right now that I am 
aware of is:

1. Did we have a docstring, yes/no?
2. Is anything referred to in `backticks` a valid reference?
3. Did that docstring parse as Sphinx-dialect ReST?

I would like to improve this situation someday, but it's not a task for 
the near future.

It would be very, very cool if autodoc itself was able to parse the 
integrity of the docstring and perform some simple validity checks, namely:

1. Every parameter has a :param FOO: in the docstring somewhere
2. Every :param FOO: matches a real parameter.
3. If the function has a return value, it is described with :returns:
4. If the function raises an exception, it must be mentioned with :raises:.

With perhaps some kind of exception for "simple" functions (How do I 
measure that?) where a one-line description is likely efficient enough.

--js