[PATCH 00/20] qapi: new sphinx qapi domain pre-requisites

[PATCH 00/20] qapi: new sphinx qapi domain pre-requisites

[John Snow]

Howdy - this patch series is the first batch of patches meant to prepare
the QAPI documentation for a new Sphinx module that adds
cross-references, an index, improved inlining, elision of types unseen
on the wire, and other goodies.

This series addresses just existing code and documentation that needs to
be changed and doesn't introduce anything new just yet - except the rST
conversion of Notes and Examples sections, which DOES impact the
existing QAPI documentation generation.

If you're CC'd on this series, it's *probably* because I've adjusted
some QAPI documentation that you're the maintainer of - In most cases,
these changes are purely mechanical (converting QAPI sections into pure
rST) and probably nothing too interesting. In a small handful of cases
(patches 15-17), I've been a bit more invasive and you may want to take
a quick peek.

Overview:

Patches 1-3: linter/typing cleanup
Patches 4-12: QAPI generator fixes/miscellany
Patch 13: qapidoc.py fix (to prepare for rST conversion)
Patches 14-20: QAPI documentation modifications, rST conversion

Sorry,
--js

John Snow (20):
  [DO-NOT-MERGE]: Add some ad-hoc linting helpers.
  qapi: linter fixups
  docs/qapidoc: delint a tiny portion of the module
  qapi/parser: preserve indentation in QAPIDoc sections
  qapi/parser: adjust info location for doc body section
  qapi/parser: fix comment parsing immediately following a doc block
  qapi/parser: add semantic 'kind' parameter to QAPIDoc.Section
  qapi/parser: differentiate intro and outro paragraphs
  qapi/parser: add undocumented stub members to all_sections
  qapi/schema: add __iter__ method to QAPISchemaVariants
  qapi/schema: add doc_visible property to QAPISchemaDefinition
  qapi/source: allow multi-line QAPISourceInfo advancing
  docs/qapidoc: fix nested parsing under untagged sections
  qapi: fix non-compliant JSON examples
  qapi: remove developer factoring comments from QAPI doc blocks
  qapi: rewrite StatsFilter comment
  qapi: rewrite BlockExportOptions doc block
  qapi: ensure all errors sections are uniformly typset
  qapi: convert "Note" sections to plain rST
  qapi: convert "Example" sections to rST

 docs/sphinx/qapidoc.py                        |  62 ++++--
 qapi/acpi.json                                |   6 +-
 qapi/audio.json                               |   5 +-
 qapi/block-core.json                          | 195 ++++++++++--------
 qapi/block-export.json                        |  16 +-
 qapi/block.json                               |  62 +++---
 qapi/char.json                                |  53 +++--
 qapi/control.json                             |  32 +--
 qapi/crypto.json                              |  33 ++-
 qapi/dump.json                                |  14 +-
 qapi/introspect.json                          |   6 +-
 qapi/machine-target.json                      |  29 +--
 qapi/machine.json                             | 138 +++++++------
 qapi/migration.json                           | 159 +++++++++-----
 qapi/misc-target.json                         |  33 ++-
 qapi/misc.json                                | 139 +++++++------
 qapi/net.json                                 |  49 +++--
 qapi/pci.json                                 |  11 +-
 qapi/qapi-schema.json                         |   6 +-
 qapi/qdev.json                                |  45 ++--
 qapi/qom.json                                 |  69 +++----
 qapi/replay.json                              |  12 +-
 qapi/rocker.json                              |  30 +--
 qapi/run-state.json                           |  63 +++---
 qapi/sockets.json                             |  10 +-
 qapi/stats.json                               |  30 ++-
 qapi/tpm.json                                 |   9 +-
 qapi/trace.json                               |   6 +-
 qapi/transaction.json                         |  13 +-
 qapi/ui.json                                  | 107 +++++-----
 qapi/virtio.json                              |  50 ++---
 qapi/yank.json                                |   6 +-
 qga/qapi-schema.json                          |  48 ++---
 scripts/qapi-lint.sh                          |  51 +++++
 scripts/qapi/Makefile                         |   5 +
 scripts/qapi/introspect.py                    |  12 +-
 scripts/qapi/parser.py                        | 104 ++++++++--
 scripts/qapi/schema.py                        |  54 ++++-
 scripts/qapi/source.py                        |   4 +-
 scripts/qapi/types.py                         |   4 +-
 scripts/qapi/visit.py                         |   9 +-
 tests/qapi-schema/doc-empty-section.err       |   2 +-
 tests/qapi-schema/doc-empty-section.json      |   2 +-
 tests/qapi-schema/doc-good.json               |  18 +-
 tests/qapi-schema/doc-good.out                |  61 +++---
 tests/qapi-schema/doc-good.txt                |  31 +--
 .../qapi-schema/doc-interleaved-section.json  |   2 +-
 47 files changed, 1152 insertions(+), 753 deletions(-)
 create mode 100755 scripts/qapi-lint.sh
 create mode 100644 scripts/qapi/Makefile

-- 
2.44.0

[PATCH 01/20] [DO-NOT-MERGE]: Add some ad-hoc linting helpers.

[John Snow]

These aren't ready for upstream inclusion, because they do not properly
manage version dependencies, execution environment and so on. These are
just the tools I use in my Own Special Environment :tm: for testing and
debugging.

They've been tested only on Fedora 38 for right now, which means:

Python 3.11.4-1.fc38
pylint 2.17.4-2.fc38
mypy 1.4.0-1.fc38
isort 5.12.0-1.fc38
flake8 5.0.3-2.fc38

"Soon" :tm: I'll move the qapi generator code under the python/
directory to take advantage of the more robust linting infrastructure
there, but that's going to happen after this series. Sorry about that!

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi-lint.sh  | 51 +++++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/Makefile |  5 +++++
 2 files changed, 56 insertions(+)
 create mode 100755 scripts/qapi-lint.sh
 create mode 100644 scripts/qapi/Makefile

diff --git a/scripts/qapi-lint.sh b/scripts/qapi-lint.sh
new file mode 100755
index 00000000000..528b31627eb
--- /dev/null
+++ b/scripts/qapi-lint.sh
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+set -e
+
+if [[ -f qapi/.flake8 ]]; then
+    echo "flake8 --config=qapi/.flake8 qapi/"
+    flake8 --config=qapi/.flake8 qapi/
+fi
+if [[ -f qapi/pylintrc ]]; then
+    echo "pylint --rcfile=qapi/pylintrc qapi/"
+    pylint --rcfile=qapi/pylintrc qapi/
+fi
+if [[ -f qapi/mypy.ini ]]; then
+    echo "mypy --config-file=qapi/mypy.ini qapi/"
+    mypy --config-file=qapi/mypy.ini qapi/
+fi
+
+if [[ -f qapi/.isort.cfg ]]; then
+    pushd qapi
+    echo "isort -c ."
+    isort -c .
+    popd
+fi
+
+if [[ -f ../docs/sphinx/qapi-domain.py ]]; then
+    pushd ../docs/sphinx
+
+    echo "mypy --strict qapi-domain.py"
+    mypy --strict qapi-domain.py
+    echo "flake8 qapi-domain.py --max-line-length=99"
+    flake8 qapi-domain.py --max-line-length=99
+    echo "isort qapi-domain.py"
+    isort qapi-domain.py
+    echo "black --check qapi-domain.py"
+    black --check qapi-domain.py
+
+    echo "flake8 qapidoc.py --max-line-length=99"
+    flake8 qapidoc.py --max-line-length=99
+    echo "isort qapidoc.py"
+    isort qapidoc.py
+    echo "black --check qapidoc.py"
+    black --check qapidoc.py
+
+    popd
+fi
+
+pushd ../build
+make -j13
+make check-qapi-schema
+make docs
+make sphinxdocs
+popd
diff --git a/scripts/qapi/Makefile b/scripts/qapi/Makefile
new file mode 100644
index 00000000000..314e8a5505e
--- /dev/null
+++ b/scripts/qapi/Makefile
@@ -0,0 +1,5 @@
+check:
+	isort -c .
+	flake8 .
+	cd .. && pylint --rcfile=qapi/pylintrc qapi
+	cd .. && mypy -p qapi --config-file=qapi/mypy.ini
-- 
2.44.0

[PATCH 02/20] qapi: linter fixups

[John Snow]

Fix minor irritants to pylint/flake8 et al.

(Yes, these need to be guarded by the Python tests. That's a work in
progress, a series that's quite likely to follow once I finish this
Sphinx project. Please pardon the temporary irritation.)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/introspect.py | 8 ++++----
 scripts/qapi/schema.py     | 6 +++---
 scripts/qapi/visit.py      | 5 +++--
 3 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 86c075a6ad2..ac14b20f308 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -27,8 +27,8 @@
 from .schema import (
     QAPISchema,
     QAPISchemaAlternatives,
-    QAPISchemaBranches,
     QAPISchemaArrayType,
+    QAPISchemaBranches,
     QAPISchemaBuiltinType,
     QAPISchemaEntity,
     QAPISchemaEnumMember,
@@ -233,9 +233,9 @@ def _use_type(self, typ: QAPISchemaType) -> str:
             typ = type_int
         elif (isinstance(typ, QAPISchemaArrayType) and
               typ.element_type.json_type() == 'int'):
-            type_intList = self._schema.lookup_type('intList')
-            assert type_intList
-            typ = type_intList
+            type_intlist = self._schema.lookup_type('intList')
+            assert type_intlist
+            typ = type_intlist
         # Add type to work queue if new
         if typ not in self._used_types:
             self._used_types.append(typ)
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 721c470d2b8..d65c35f6ee6 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -730,6 +730,7 @@ def set_defined_in(self, name: str) -> None:
         for v in self.variants:
             v.set_defined_in(name)
 
+    # pylint: disable=unused-argument
     def check(
             self, schema: QAPISchema, seen: Dict[str, QAPISchemaMember]
     ) -> None:
@@ -1166,7 +1167,7 @@ def _def_definition(self, defn: QAPISchemaDefinition) -> None:
                 defn.info, "%s is already defined" % other_defn.describe())
         self._entity_dict[defn.name] = defn
 
-    def lookup_entity(self,name: str) -> Optional[QAPISchemaEntity]:
+    def lookup_entity(self, name: str) -> Optional[QAPISchemaEntity]:
         return self._entity_dict.get(name)
 
     def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
@@ -1302,11 +1303,10 @@ def _make_implicit_object_type(
         name = 'q_obj_%s-%s' % (name, role)
         typ = self.lookup_entity(name)
         if typ:
-            assert(isinstance(typ, QAPISchemaObjectType))
+            assert isinstance(typ, QAPISchemaObjectType)
             # The implicit object type has multiple users.  This can
             # only be a duplicate definition, which will be flagged
             # later.
-            pass
         else:
             self._def_definition(QAPISchemaObjectType(
                 name, info, None, ifcond, None, None, members, None))
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index e766acaac92..12f92e429f6 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -280,8 +280,9 @@ def gen_visit_alternate(name: str,
         abort();
     default:
         assert(visit_is_input(v));
-        error_setg(errp, "Invalid parameter type for '%%s', expected: %(name)s",
-                         name ? name : "null");
+        error_setg(errp,
+                   "Invalid parameter type for '%%s', expected: %(name)s",
+                   name ? name : "null");
         /* Avoid passing invalid *obj to qapi_free_%(c_name)s() */
         g_free(*obj);
         *obj = NULL;
-- 
2.44.0

[PATCH 03/20] docs/qapidoc: delint a tiny portion of the module

[John Snow]

In the coming patches, it's helpful to have a linting baseline. However,
there's no need to shuffle around the deck chairs too much, because most
of this code will be removed once the new qapidoc generator (the
"transmogrifier") is in place.

To ease my pain: just turn off the black auto-formatter for most, but
not all, of qapidoc.py. This will help ensure that *new* code follows a
coding standard without bothering too much with cleaning up the existing
code.

For manual checking for now, try "black --check qapidoc.py" from the
docs/sphinx directory. "pip install black" (without root permissions) if
you do not have it installed otherwise.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index f270b494f01..1655682d4c7 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -28,28 +28,30 @@
 import re
 
 from docutils import nodes
+from docutils.parsers.rst import Directive, directives
 from docutils.statemachine import ViewList
-from docutils.parsers.rst import directives, Directive
-from sphinx.errors import ExtensionError
-from sphinx.util.nodes import nested_parse_with_titles
-import sphinx
-from qapi.gen import QAPISchemaVisitor
 from qapi.error import QAPIError, QAPISemError
+from qapi.gen import QAPISchemaVisitor
 from qapi.schema import QAPISchema
 
+import sphinx
+from sphinx.errors import ExtensionError
+from sphinx.util.nodes import nested_parse_with_titles
+
 
 # Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
 # use switch_source_input. Check borrowed from kerneldoc.py.
-Use_SSI = sphinx.__version__[:3] >= '1.7'
+Use_SSI = sphinx.__version__[:3] >= "1.7"
 if Use_SSI:
     from sphinx.util.docutils import switch_source_input
 else:
     from sphinx.ext.autodoc import AutodocReporter
 
 
-__version__ = '1.0'
+__version__ = "1.0"
 
 
+# fmt: off
 # Function borrowed from pydash, which is under the MIT license
 def intersperse(iterable, separator):
     """Yield the members of *iterable* interspersed with *separator*."""
-- 
2.44.0

[PATCH 04/20] qapi/parser: preserve indentation in QAPIDoc sections

[John Snow]

Prior to this patch, a section like this:

@name: lorem ipsum
   dolor sit amet
     consectetur adipiscing elit

would be parsed as:

"lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"

We want to preserve the indentation for even the first body line so that
the entire block can be parsed directly as rST. This patch would now
parse that segment as:

"lorem ipsum\n   dolor sit amet\n     consectetur adipiscing elit"

This understandably breaks qapidoc.py; so a new function is added there
to re-dedent the text. Once the new generator is merged, this function
will not be needed any longer and can be dropped.

(I verified this patch changes absolutely nothing by comparing the
md5sums of the QMP ref html pages both before and after the change, so
it's certified inert. QAPI test output has been updated to reflect the
new strategy of preserving indents for rST.)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py         | 36 +++++++++++++++++++++++++++++-----
 scripts/qapi/parser.py         |  8 ++++++--
 tests/qapi-schema/doc-good.out | 32 +++++++++++++++---------------
 3 files changed, 53 insertions(+), 23 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 1655682d4c7..2e3ffcbafb7 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -26,6 +26,7 @@
 
 import os
 import re
+import textwrap
 
 from docutils import nodes
 from docutils.parsers.rst import Directive, directives
@@ -51,6 +52,28 @@
 __version__ = "1.0"
 
 
+def dedent(text: str) -> str:
+    # Temporary: In service of the new QAPI domain, the QAPI doc parser
+    # now preserves indents in args/members/features text. QAPIDoc does
+    # not handle this well, so undo that change here.
+
+    # QAPIDoc is being rewritten and will be replaced soon,
+    # but this function is here in the interim as transition glue.
+
+    lines = text.splitlines(True)
+    if len(lines) > 1:
+        if re.match(r"\s+", lines[0]):
+            # First line is indented; description started on
+            # the line after the name. dedent the whole block.
+            return textwrap.dedent(text)
+        else:
+            # Descr started on same line. Dedent line 2+.
+            return lines[0] + textwrap.dedent("".join(lines[1:]))
+    else:
+        # Descr was a single line; dedent entire line.
+        return textwrap.dedent(text)
+
+
 # fmt: off
 # Function borrowed from pydash, which is under the MIT license
 def intersperse(iterable, separator):
@@ -169,7 +192,7 @@ def _nodes_for_members(self, doc, what, base=None, branches=None):
             term = self._nodes_for_one_member(section.member)
             # TODO drop fallbacks when undocumented members are outlawed
             if section.text:
-                defn = section.text
+                defn = dedent(section.text)
             else:
                 defn = [nodes.Text('Not documented')]
 
@@ -207,7 +230,7 @@ def _nodes_for_enum_values(self, doc):
                 termtext.extend(self._nodes_for_ifcond(section.member.ifcond))
             # TODO drop fallbacks when undocumented members are outlawed
             if section.text:
-                defn = section.text
+                defn = dedent(section.text)
             else:
                 defn = [nodes.Text('Not documented')]
 
@@ -242,7 +265,7 @@ def _nodes_for_features(self, doc):
         dlnode = nodes.definition_list()
         for section in doc.features.values():
             dlnode += self._make_dlitem(
-                [nodes.literal('', section.member.name)], section.text)
+                [nodes.literal('', section.member.name)], dedent(section.text))
             seen_item = True
 
         if not seen_item:
@@ -265,9 +288,12 @@ def _nodes_for_sections(self, doc):
                 continue
             snode = self._make_section(section.tag)
             if section.tag and section.tag.startswith('Example'):
-                snode += self._nodes_for_example(section.text)
+                snode += self._nodes_for_example(dedent(section.text))
             else:
-                self._parse_text_into_node(section.text, snode)
+                self._parse_text_into_node(
+                    dedent(section.text) if section.tag else section.text,
+                    snode,
+                )
             nodelist.append(snode)
         return nodelist
 
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 7b13a583ac1..8cdd5334ec6 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -448,7 +448,10 @@ def get_doc_indented(self, doc: 'QAPIDoc') -> Optional[str]:
         indent = must_match(r'\s*', line).end()
         if not indent:
             return line
-        doc.append_line(line[indent:])
+
+        # Preserve the indent, it's needed for rST formatting.
+        doc.append_line(line)
+
         prev_line_blank = False
         while True:
             self.accept(False)
@@ -465,7 +468,8 @@ def get_doc_indented(self, doc: 'QAPIDoc') -> Optional[str]:
                     self,
                     "unexpected de-indent (expected at least %d spaces)" %
                     indent)
-            doc.append_line(line[indent:])
+            # Again, preserve the indent for ReST.
+            doc.append_line(line)
             prev_line_blank = True
 
     def get_doc_paragraph(self, doc: 'QAPIDoc') -> Optional[str]:
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 716a9a41026..435f6e6d768 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -117,8 +117,8 @@ doc symbol=Base
     body=
 
     arg=base1
-description starts on a new line,
-minimally indented
+ description starts on a new line,
+ minimally indented
 doc symbol=Variant1
     body=
 A paragraph
@@ -145,8 +145,8 @@ doc symbol=Alternate
 
     arg=i
 description starts on the same line
-remainder indented the same
-@b is undocumented
+    remainder indented the same
+    @b is undocumented
     arg=b
 
     feature=alt-feat
@@ -158,11 +158,11 @@ doc symbol=cmd
     body=
 
     arg=arg1
-description starts on a new line,
-indented
+    description starts on a new line,
+    indented
     arg=arg2
 description starts on the same line
-remainder indented differently
+    remainder indented differently
     arg=arg3
 
     feature=cmd-feat1
@@ -178,16 +178,16 @@ some
     section=TODO
 frobnicate
     section=Notes
-- Lorem ipsum dolor sit amet
-- Ut enim ad minim veniam
+ - Lorem ipsum dolor sit amet
+ - Ut enim ad minim veniam
 
-Duis aute irure dolor
+ Duis aute irure dolor
     section=Example
--> in
-<- out
+ -> in
+ <- out
     section=Examples
-- *verbatim*
-- {braces}
+ - *verbatim*
+ - {braces}
     section=Since
 2.10
 doc symbol=cmd-boxed
@@ -198,9 +198,9 @@ a feature
     feature=cmd-feat2
 another feature
     section=Example
--> in
+ -> in
 
-<- out
+ <- out
 doc symbol=EVT_BOXED
     body=
 
-- 
2.44.0

[PATCH 05/20] qapi/parser: adjust info location for doc body section

[John Snow]

Instead of using the info object for the doc block as a whole, update
the info pointer for each call to ensure_untagged_section when the
existing section is otherwise empty. This way, Sphinx error information
will match precisely to where the text actually starts.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 8cdd5334ec6..41b9319e5cb 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -662,8 +662,13 @@ def end(self) -> None:
 
     def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
         if self.all_sections and not self.all_sections[-1].tag:
-            # extend current section
-            self.all_sections[-1].text += '\n'
+            section = self.all_sections[-1]
+            # Section is empty so far; update info to start *here*.
+            if not section.text:
+                section.info = info
+            else:
+                # extend current section
+                self.all_sections[-1].text += '\n'
             return
         # start new section
         section = self.Section(info)
-- 
2.44.0

[PATCH 06/20] qapi/parser: fix comment parsing immediately following a doc block

[John Snow]

If a comment immediately follows a doc block, the parser doesn't ignore
that token appropriately. Fix that.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 41b9319e5cb..161768b8b96 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -587,7 +587,7 @@ def get_doc(self) -> 'QAPIDoc':
                 line = self.get_doc_line()
                 first = False
 
-        self.accept(False)
+        self.accept()
         doc.end()
         return doc
 
-- 
2.44.0

[PATCH 07/20] qapi/parser: add semantic 'kind' parameter to QAPIDoc.Section

[John Snow]

When iterating all_sections, this is helpful to be able to distinguish
"members" from "features"; the only other way to do so is to
cross-reference these sections against QAPIDoc.args or QAPIDoc.features,
but if the desired end goal for QAPIDoc is to remove everything except
all_sections, we need *something* accessible to distinguish them.

To keep types simple, add this semantic parameter to the base Section
and not just ArgSection; we can use this to filter out paragraphs and
tagged sections, too.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 161768b8b96..cf4cbca1c1f 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -613,21 +613,27 @@ class QAPIDoc:
 
     class Section:
         # pylint: disable=too-few-public-methods
-        def __init__(self, info: QAPISourceInfo,
-                     tag: Optional[str] = None):
+        def __init__(
+            self,
+            info: QAPISourceInfo,
+            tag: Optional[str] = None,
+            kind: str = 'paragraph',
+        ):
             # section source info, i.e. where it begins
             self.info = info
             # section tag, if any ('Returns', '@name', ...)
             self.tag = tag
             # section text without tag
             self.text = ''
+            # section type - {paragraph, feature, member, tagged}
+            self.kind = kind
 
         def append_line(self, line: str) -> None:
             self.text += line + '\n'
 
     class ArgSection(Section):
-        def __init__(self, info: QAPISourceInfo, tag: str):
-            super().__init__(info, tag)
+        def __init__(self, info: QAPISourceInfo, tag: str, kind: str):
+            super().__init__(info, tag, kind)
             self.member: Optional['QAPISchemaMember'] = None
 
         def connect(self, member: 'QAPISchemaMember') -> None:
@@ -676,7 +682,7 @@ def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
         self.all_sections.append(section)
 
     def new_tagged_section(self, info: QAPISourceInfo, tag: str) -> None:
-        section = self.Section(info, tag)
+        section = self.Section(info, tag, "tagged")
         if tag == 'Returns':
             if self.returns:
                 raise QAPISemError(
@@ -696,20 +702,21 @@ def new_tagged_section(self, info: QAPISourceInfo, tag: str) -> None:
         self.all_sections.append(section)
 
     def _new_description(self, info: QAPISourceInfo, name: str,
+                         kind: str,
                          desc: Dict[str, ArgSection]) -> None:
         if not name:
             raise QAPISemError(info, "invalid parameter name")
         if name in desc:
             raise QAPISemError(info, "'%s' parameter name duplicated" % name)
-        section = self.ArgSection(info, '@' + name)
+        section = self.ArgSection(info, '@' + name, kind)
         self.all_sections.append(section)
         desc[name] = section
 
     def new_argument(self, info: QAPISourceInfo, name: str) -> None:
-        self._new_description(info, name, self.args)
+        self._new_description(info, name, 'member', self.args)
 
     def new_feature(self, info: QAPISourceInfo, name: str) -> None:
-        self._new_description(info, name, self.features)
+        self._new_description(info, name, 'feature', self.features)
 
     def append_line(self, line: str) -> None:
         self.all_sections[-1].append_line(line)
@@ -722,7 +729,7 @@ def connect_member(self, member: 'QAPISchemaMember') -> None:
                                    "%s '%s' lacks documentation"
                                    % (member.role, member.name))
             self.args[member.name] = QAPIDoc.ArgSection(
-                self.info, '@' + member.name)
+                self.info, '@' + member.name, 'member')
         self.args[member.name].connect(member)
 
     def connect_feature(self, feature: 'QAPISchemaFeature') -> None:
-- 
2.44.0

[PATCH 08/20] qapi/parser: differentiate intro and outro paragraphs

[John Snow]

Add a semantic tag to paragraphs that appear *before* tagged
sections/members/features and those that appear after. This will control
how they are inlined when doc sections are merged and flattened.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index cf4cbca1c1f..b1794f71e12 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -503,6 +503,10 @@ def get_doc(self) -> 'QAPIDoc':
             self.accept(False)
             line = self.get_doc_line()
             no_more_args = False
+            # Paragraphs before members/features/tagged are "intro" paragraphs.
+            # Any appearing subsequently are "outro" paragraphs.
+            # This is only semantic metadata for the doc generator.
+            intro = True
 
             while line is not None:
                 # Blank lines
@@ -532,6 +536,7 @@ def get_doc(self) -> 'QAPIDoc':
                         raise QAPIParseError(
                             self, 'feature descriptions expected')
                     no_more_args = True
+                    intro = False
                 elif match := self._match_at_name_colon(line):
                     # description
                     if no_more_args:
@@ -547,6 +552,7 @@ def get_doc(self) -> 'QAPIDoc':
                             doc.append_line(text)
                         line = self.get_doc_indented(doc)
                     no_more_args = True
+                    intro = False
                 elif match := re.match(
                         r'(Returns|Errors|Since|Notes?|Examples?|TODO): *',
                         line):
@@ -557,13 +563,14 @@ def get_doc(self) -> 'QAPIDoc':
                         doc.append_line(text)
                     line = self.get_doc_indented(doc)
                     no_more_args = True
+                    intro = False
                 elif line.startswith('='):
                     raise QAPIParseError(
                         self,
                         "unexpected '=' markup in definition documentation")
                 else:
                     # tag-less paragraph
-                    doc.ensure_untagged_section(self.info)
+                    doc.ensure_untagged_section(self.info, intro)
                     doc.append_line(line)
                     line = self.get_doc_paragraph(doc)
         else:
@@ -617,7 +624,7 @@ def __init__(
             self,
             info: QAPISourceInfo,
             tag: Optional[str] = None,
-            kind: str = 'paragraph',
+            kind: str = 'intro-paragraph',
         ):
             # section source info, i.e. where it begins
             self.info = info
@@ -625,7 +632,7 @@ def __init__(
             self.tag = tag
             # section text without tag
             self.text = ''
-            # section type - {paragraph, feature, member, tagged}
+            # section type - {<intro|outro>-paragraph, feature, member, tagged}
             self.kind = kind
 
         def append_line(self, line: str) -> None:
@@ -666,7 +673,11 @@ def end(self) -> None:
                 raise QAPISemError(
                     section.info, "text required after '%s:'" % section.tag)
 
-    def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
+    def ensure_untagged_section(
+        self,
+        info: QAPISourceInfo,
+        intro: bool = True,
+    ) -> None:
         if self.all_sections and not self.all_sections[-1].tag:
             section = self.all_sections[-1]
             # Section is empty so far; update info to start *here*.
@@ -677,7 +688,8 @@ def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
                 self.all_sections[-1].text += '\n'
             return
         # start new section
-        section = self.Section(info)
+        kind = ("intro" if intro else "outro") + "-paragraph"
+        section = self.Section(info, kind=kind)
         self.sections.append(section)
         self.all_sections.append(section)
 
-- 
2.44.0

[PATCH 09/20] qapi/parser: add undocumented stub members to all_sections

[John Snow]

This helps simplify the doc generator if it doesn't have to check for
undocumented members.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index b1794f71e12..3cd8e7ee295 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -740,8 +740,24 @@ def connect_member(self, member: 'QAPISchemaMember') -> None:
                 raise QAPISemError(member.info,
                                    "%s '%s' lacks documentation"
                                    % (member.role, member.name))
-            self.args[member.name] = QAPIDoc.ArgSection(
-                self.info, '@' + member.name, 'member')
+
+            # Insert stub documentation section for missing member docs.
+            section = QAPIDoc.ArgSection(
+                self.info, f"@{member.name}", "member")
+            self.args[member.name] = section
+
+            # Determine where to insert stub doc.
+            index = 0
+            for i, sect in enumerate(self.all_sections):
+                # insert after these:
+                if sect.kind in ('intro-paragraph', 'member'):
+                    index = i + 1
+                # but before these:
+                elif sect.kind in ('tagged', 'feature', 'outro-paragraph'):
+                    index = i
+                    break
+            self.all_sections.insert(index, section)
+
         self.args[member.name].connect(member)
 
     def connect_feature(self, feature: 'QAPISchemaFeature') -> None:
-- 
2.44.0

[PATCH 10/20] qapi/schema: add __iter__ method to QAPISchemaVariants

[John Snow]

This just makes it easier to do something like:

for var in variants:
    ...

Instead of the more cumbersome and repetitive:

for var in variants.variants:
    ...

Especially in conjunction with entities that aren't guaranteed to have
variants. Compare:

for var in variants.variants if variants else []:
    ...

against:

for var in variants or []:
    ...

Update callsites to reflect the new usage pattern.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py     | 2 +-
 scripts/qapi/introspect.py | 4 ++--
 scripts/qapi/schema.py     | 8 ++++++--
 scripts/qapi/types.py      | 4 ++--
 scripts/qapi/visit.py      | 4 ++--
 5 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 2e3ffcbafb7..34e95bd168d 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -204,7 +204,7 @@ def _nodes_for_members(self, doc, what, base=None, branches=None):
                                         None)
 
         if branches:
-            for v in branches.variants:
+            for v in branches:
                 if v.type.name == 'q_empty':
                     continue
                 assert not v.type.is_implicit()
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index ac14b20f308..6ec34e055d3 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -342,7 +342,7 @@ def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo],
         }
         if branches:
             obj['tag'] = branches.tag_member.name
-            obj['variants'] = [self._gen_variant(v) for v in branches.variants]
+            obj['variants'] = [self._gen_variant(v) for v in branches]
         self._gen_tree(name, 'object', obj, ifcond, features)
 
     def visit_alternate_type(self, name: str, info: Optional[QAPISourceInfo],
@@ -353,7 +353,7 @@ def visit_alternate_type(self, name: str, info: Optional[QAPISourceInfo],
             name, 'alternate',
             {'members': [Annotated({'type': self._use_type(m.type)},
                                    m.ifcond)
-                         for m in alternatives.variants]},
+                         for m in alternatives]},
             ifcond, features
         )
 
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index d65c35f6ee6..e15e64ea8cb 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -26,6 +26,7 @@
     Any,
     Callable,
     Dict,
+    Iterator,
     List,
     Optional,
     Union,
@@ -669,7 +670,7 @@ def check(self, schema: QAPISchema) -> None:
         # so we have to check for potential name collisions ourselves.
         seen: Dict[str, QAPISchemaMember] = {}
         types_seen: Dict[str, str] = {}
-        for v in self.alternatives.variants:
+        for v in self.alternatives:
             v.check_clash(self.info, seen)
             qtype = v.type.alternate_qtype()
             if not qtype:
@@ -700,7 +701,7 @@ def check(self, schema: QAPISchema) -> None:
     def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
-        for v in self.alternatives.variants:
+        for v in self.alternatives:
             v.connect_doc(doc)
 
     def c_type(self) -> str:
@@ -726,6 +727,9 @@ def __init__(
         self.tag_member: QAPISchemaObjectTypeMember
         self.variants = variants
 
+    def __iter__(self) -> Iterator[QAPISchemaVariant]:
+        return iter(self.variants)
+
     def set_defined_in(self, name: str) -> None:
         for v in self.variants:
             v.set_defined_in(name)
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 0dd0b00ada3..ad36b55488f 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -166,7 +166,7 @@ def gen_object(name: str, ifcond: QAPISchemaIfCond,
     objects_seen.add(name)
 
     ret = ''
-    for var in variants.variants if variants else ():
+    for var in variants or ():
         obj = var.type
         if not isinstance(obj, QAPISchemaObjectType):
             continue
@@ -234,7 +234,7 @@ def gen_variants(variants: QAPISchemaVariants) -> str:
 ''',
                 c_name=c_name(variants.tag_member.name))
 
-    for var in variants.variants:
+    for var in variants:
         if var.type.name == 'q_empty':
             continue
         ret += var.ifcond.gen_if()
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 12f92e429f6..1eca452378c 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -141,7 +141,7 @@ def gen_visit_object_members(name: str,
 ''',
                      c_name=c_name(tag_member.name))
 
-        for var in branches.variants:
+        for var in branches:
             case_str = c_enum_const(tag_member.type.name, var.name,
                                     tag_member.type.prefix)
             ret += var.ifcond.gen_if()
@@ -246,7 +246,7 @@ def gen_visit_alternate(name: str,
 ''',
                 c_name=c_name(name))
 
-    for var in alternatives.variants:
+    for var in alternatives:
         ret += var.ifcond.gen_if()
         ret += mcgen('''
     case %(case)s:
-- 
2.44.0

[PATCH 11/20] qapi/schema: add doc_visible property to QAPISchemaDefinition

[John Snow]

The intent here is to mark only certain definitions as visible in the
end-user docs.

All commands and events are inherently visible. Everything else is
visible only if it is a member (or a branch member) of a type that is
visible, or if it is named as a return type for a command.

Notably, this excludes arg_type for commands and events, and any
base_types specified for structures/unions. Those objects may still be
marked visible if they are named as members from a visible type.

This does not necessarily match the data revealed by introspection: in
this case, we want anything that we are cross-referencing in generated
documentation to be available to target.

Some internal and built-in types may be marked visible with this
approach, but if they do not have a documentation block, they'll be
skipped by the generator anyway. This includes array types and built-in
primitives which do not get their own documentation objects.

This information is not yet used by qapidoc, which continues to render
documentation exactly as it has. This information will be used by the
new qapidoc (the "transmogrifier"), to be introduced later. The new
generator verifies that all of the objects that should be rendered *are*
by failing if any cross-references are missing, verifying everything is
in place.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index e15e64ea8cb..6025b4e9354 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -131,6 +131,7 @@ def __init__(
         self.doc = doc
         self._ifcond = ifcond or QAPISchemaIfCond()
         self.features = features or []
+        self.doc_visible = False
 
     def __repr__(self) -> str:
         return "<%s:%s at 0x%x>" % (type(self).__name__, self.name,
@@ -146,6 +147,10 @@ def check(self, schema: QAPISchema) -> None:
         for f in self.features:
             f.check_clash(self.info, seen)
 
+    def mark_visible(self, mark_self: bool = True) -> None:
+        if mark_self:
+            self.doc_visible = True
+
     def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
@@ -483,6 +488,10 @@ def check(self, schema: QAPISchema) -> None:
             self.info.defn_meta if self.info else None)
         assert not isinstance(self.element_type, QAPISchemaArrayType)
 
+    def mark_visible(self, mark_self: bool = True) -> None:
+        super().mark_visible(mark_self)
+        self.element_type.mark_visible()
+
     def set_module(self, schema: QAPISchema) -> None:
         self._set_module(schema, self.element_type.info)
 
@@ -607,6 +616,17 @@ def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         for m in self.local_members:
             m.connect_doc(doc)
 
+    def mark_visible(self, mark_self: bool = True) -> None:
+        # Mark this object and its members as visible in the user-facing docs.
+        if self.doc_visible:
+            return
+
+        super().mark_visible(mark_self)
+        for m in self.members:
+            m.type.mark_visible()
+        for var in self.branches or []:
+            var.type.mark_visible(False)
+
     def is_implicit(self) -> bool:
         # See QAPISchema._make_implicit_object_type(), as well as
         # _def_predefineds()
@@ -698,6 +718,11 @@ def check(self, schema: QAPISchema) -> None:
                         % (v.describe(self.info), types_seen[qt]))
                 types_seen[qt] = v.name
 
+    def mark_visible(self, mark_self: bool = True) -> None:
+        super().mark_visible(mark_self)
+        for var in self.alternatives:
+            var.type.mark_visible()
+
     def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
@@ -1056,6 +1081,13 @@ def check(self, schema: QAPISchema) -> None:
                         "command's 'returns' cannot take %s"
                         % self.ret_type.describe())
 
+    def mark_visible(self, mark_self: bool = True) -> None:
+        super().mark_visible(mark_self)
+        if self.arg_type:
+            self.arg_type.mark_visible(False)
+        if self.ret_type:
+            self.ret_type.mark_visible()
+
     def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
@@ -1112,6 +1144,11 @@ def check(self, schema: QAPISchema) -> None:
                     self.info,
                     "conditional event arguments require 'boxed': true")
 
+    def mark_visible(self, mark_self: bool = True) -> None:
+        super().mark_visible(mark_self)
+        if self.arg_type:
+            self.arg_type.mark_visible(False)
+
     def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
@@ -1488,6 +1525,9 @@ def check(self) -> None:
             ent.set_module(self)
         for doc in self.docs:
             doc.check()
+        for ent in self._entity_list:
+            if isinstance(ent, (QAPISchemaCommand, QAPISchemaEvent)):
+                ent.mark_visible()
 
     def visit(self, visitor: QAPISchemaVisitor) -> None:
         visitor.visit_begin(self)
-- 
2.44.0

[PATCH 12/20] qapi/source: allow multi-line QAPISourceInfo advancing

[John Snow]

This is for the sake of the new rST generator (the "transmogrifier") so
we can advance multiple lines on occasion while keeping the
generated<-->source mappings accurate.

next_line now simply takes an optional n parameter which chooses the
number of lines to advance.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/source.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index 7b379fdc925..ffdc3f482ac 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -47,9 +47,9 @@ def set_defn(self, meta: str, name: str) -> None:
         self.defn_meta = meta
         self.defn_name = name
 
-    def next_line(self: T) -> T:
+    def next_line(self: T, n: int = 1) -> T:
         info = copy.copy(self)
-        info.line += 1
+        info.line += n
         return info
 
     def loc(self) -> str:
-- 
2.44.0

[PATCH 13/20] docs/qapidoc: fix nested parsing under untagged sections

[John Snow]

Sphinx does not like sections without titles, because it wants to
convert every section into a reference. When there is no title, it
struggles to do this and transforms the tree inproperly.

Depending on the rST used, this may result in an assertion error deep in
the docutils HTMLWriter.

When parsing an untagged section (free paragraphs), skip making a hollow
section and instead append the parse results to the prior section.

Many Bothans died to bring us this information.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 34e95bd168d..cfc0cf169ef 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -286,14 +286,20 @@ def _nodes_for_sections(self, doc):
             if section.tag and section.tag == 'TODO':
                 # Hide TODO: sections
                 continue
+
+            if not section.tag:
+                # Sphinx cannot handle sectionless titles;
+                # Instead, just append the results to the prior section.
+                container = nodes.container()
+                self._parse_text_into_node(section.text, container)
+                nodelist += container.children
+                continue
+
             snode = self._make_section(section.tag)
-            if section.tag and section.tag.startswith('Example'):
+            if section.tag.startswith('Example'):
                 snode += self._nodes_for_example(dedent(section.text))
             else:
-                self._parse_text_into_node(
-                    dedent(section.text) if section.tag else section.text,
-                    snode,
-                )
+                self._parse_text_into_node(dedent(section.text), snode)
             nodelist.append(snode)
         return nodelist
 
-- 
2.44.0

[PATCH 14/20] qapi: fix non-compliant JSON examples

[John Snow]

If we parse all examples as QMP, we need them to conform to a standard
so that they render correctly. Once the QMP lexer is active for
examples, these will produce warning messages and fail the build.

The QMP lexer still supports elisions, but they must be represented as
the value "...", so two examples have been adjusted to support that
format here.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/control.json   | 3 ++-
 qapi/machine.json   | 2 +-
 qapi/migration.json | 2 +-
 qapi/misc.json      | 3 ++-
 qapi/net.json       | 6 +++---
 qapi/rocker.json    | 2 +-
 qapi/ui.json        | 2 +-
 7 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/qapi/control.json b/qapi/control.json
index 6bdbf077c2e..10c906fa0e7 100644
--- a/qapi/control.json
+++ b/qapi/control.json
@@ -145,7 +145,8 @@
 #             },
 #             {
 #                "name":"system_powerdown"
-#             }
+#             },
+#             ...
 #          ]
 #        }
 #
diff --git a/qapi/machine.json b/qapi/machine.json
index bce6e1bbc41..64a77557571 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -1057,7 +1057,7 @@
 #            "vcpus-count": 1 },
 #          { "props": { "core-id": 0 }, "type": "POWER8-spapr-cpu-core",
 #            "vcpus-count": 1, "qom-path": "/machine/unattached/device[0]"}
-#        ]}'
+#        ]}
 #
 #     For pc machine type started with -smp 1,maxcpus=2:
 #
diff --git a/qapi/migration.json b/qapi/migration.json
index a351fd37143..89047d46c7c 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -2078,7 +2078,7 @@
 # Example:
 #
 #     -> {"execute": "calc-dirty-rate", "arguments": {"calc-time": 1,
-#                                                     'sample-pages': 512} }
+#                                                     "sample-pages": 512} }
 #     <- { "return": {} }
 #
 #     Measure dirty rate using dirty bitmap for 500 milliseconds:
diff --git a/qapi/misc.json b/qapi/misc.json
index ec30e5c570a..4b41e15dcd4 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -287,7 +287,8 @@
 #
 # Example:
 #
-#     -> { "execute": "get-win32-socket", "arguments": { "info": "abcd123..", fdname": "skclient" } }
+#     -> { "execute": "get-win32-socket",
+#          "arguments": { "info": "abcd123..", "fdname": "skclient" } }
 #     <- { "return": {} }
 ##
 { 'command': 'get-win32-socket', 'data': {'info': 'str', 'fdname': 'str'}, 'if': 'CONFIG_WIN32' }
diff --git a/qapi/net.json b/qapi/net.json
index 0f5a259475e..c19df435a53 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -1003,9 +1003,9 @@
 #
 # Example:
 #
-#     <- { 'event': 'NETDEV_STREAM_DISCONNECTED',
-#          'data': {'netdev-id': 'netdev0'},
-#          'timestamp': {'seconds': 1663330937, 'microseconds': 526695} }
+#     <- { "event": "NETDEV_STREAM_DISCONNECTED",
+#          "data": {"netdev-id": "netdev0"},
+#          "timestamp": {"seconds": 1663330937, "microseconds": 526695} }
 ##
 { 'event': 'NETDEV_STREAM_DISCONNECTED',
   'data': { 'netdev-id': 'str' } }
diff --git a/qapi/rocker.json b/qapi/rocker.json
index 5635cf174fd..f5225eb62cc 100644
--- a/qapi/rocker.json
+++ b/qapi/rocker.json
@@ -250,7 +250,7 @@
 #                       "action": {"goto-tbl": 10},
 #                       "mask": {"in-pport": 4294901760}
 #                      },
-#                      {...more...},
+#                      {...},
 #        ]}
 ##
 { 'command': 'query-rocker-of-dpa-flows',
diff --git a/qapi/ui.json b/qapi/ui.json
index f610bce118a..c12f5292571 100644
--- a/qapi/ui.json
+++ b/qapi/ui.json
@@ -361,7 +361,7 @@
 #                    "channel-id": 0,
 #                    "tls": false
 #                 },
-#                 [ ... more channels follow ... ]
+#                 ...
 #              ]
 #           }
 #        }
-- 
2.44.0

[PATCH 15/20] qapi: remove developer factoring comments from QAPI doc blocks

[John Snow]

This is part of a project to overhaul the QMP reference manual. One goal
of this overhaul is to "inline" inherited argument sections into command
reference sections. A consequence of this design decision is that
inherited doc block sections need to be merged with the inheritor's
sections.

When documentation is written for types whose primary purpose is to be
inherited by other types, we need to know how to merge those paragraphs
into the descendent. Much of the time, these paragraphs aren't actually
useful to end users.

Either remove information that's of little to no use to *either*
audience, and convert what's left to garden-variety comments to prevent
it from showing up in rendered documentation.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/audio.json        |  5 ++---
 qapi/block-core.json   | 47 ++++++++++++++++++------------------------
 qapi/block-export.json | 10 ++++-----
 qapi/char.json         |  5 ++---
 qapi/crypto.json       | 33 ++++++++++++-----------------
 qapi/machine.json      | 10 ++++-----
 qapi/net.json          |  7 ++-----
 qapi/qom.json          | 30 +++++++++++----------------
 qapi/ui.json           | 14 -------------
 9 files changed, 59 insertions(+), 102 deletions(-)

diff --git a/qapi/audio.json b/qapi/audio.json
index 519697c0cd8..ee09cd231b6 100644
--- a/qapi/audio.json
+++ b/qapi/audio.json
@@ -10,11 +10,10 @@
 # = Audio
 ##
 
-##
-# @AudiodevPerDirectionOptions:
-#
 # General audio backend options that are used for both playback and
 # recording.
+##
+# @AudiodevPerDirectionOptions:
 #
 # @mixing-engine: use QEMU's mixing engine to mix all streams inside
 #     QEMU and convert audio formats when not supported by the
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 746d1694c25..64fe5240cc9 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -267,10 +267,9 @@
       'file': 'ImageInfoSpecificFileWrapper'
   } }
 
-##
-# @BlockNodeInfo:
-#
 # Information about a QEMU image file
+##
+# @BlockNodeInfo:
 #
 # @filename: name of the image file
 #
@@ -1494,8 +1493,6 @@
 ##
 # @BlockdevSnapshotSync:
 #
-# Either @device or @node-name must be set but not both.
-#
 # @device: the name of the device to take a snapshot of.
 #
 # @node-name: graph node name to generate the snapshot from (Since
@@ -1512,6 +1509,9 @@
 #
 # @mode: whether and how QEMU should create a new image, default is
 #     'absolute-paths'.
+#
+# Note: Either @device or @node-name must be set but not both.
+#
 ##
 { 'struct': 'BlockdevSnapshotSync',
   'data': { '*device': 'str', '*node-name': 'str',
@@ -2139,10 +2139,9 @@
   'data': 'DriveMirror',
   'allow-preconfig': true }
 
-##
-# @DriveMirror:
-#
 # A set of parameters describing drive mirror setup.
+##
+# @DriveMirror:
 #
 # @job-id: identifier for the newly-created block job.  If omitted,
 #     the device name will be used.  (Since 2.7)
@@ -2553,10 +2552,9 @@
             '*auto-finalize': 'bool', '*auto-dismiss': 'bool' },
   'allow-preconfig': true }
 
-##
-# @BlockIOThrottle:
-#
 # A set of parameters describing block throttling.
+##
+# @BlockIOThrottle:
 #
 # @device: Block device name
 #
@@ -3073,10 +3071,9 @@
 { 'struct': 'BlockJobChangeOptionsMirror',
   'data': { 'copy-mode' : 'MirrorCopyMode' } }
 
-##
-# @BlockJobChangeOptions:
-#
 # Block job options that can be changed after job creation.
+##
+# @BlockJobChangeOptions:
 #
 # @id: The job identifier
 #
@@ -3332,11 +3329,10 @@
   'data': { 'dir': 'str', '*fat-type': 'int', '*floppy': 'bool',
             '*label': 'str', '*rw': 'bool' } }
 
-##
-# @BlockdevOptionsGenericFormat:
-#
 # Driver specific block device options for image format that have no
 # option besides their data source.
+##
+# @BlockdevOptionsGenericFormat:
 #
 # @file: reference to or definition of the data source block device
 #
@@ -3363,11 +3359,10 @@
   'data': { '*key-secret': 'str',
             '*header': 'BlockdevRef'} }
 
-##
-# @BlockdevOptionsGenericCOWFormat:
-#
 # Driver specific block device options for image format that have no
 # option besides their data source and an optional backing file.
+##
+# @BlockdevOptionsGenericCOWFormat:
 #
 # @backing: reference to or definition of the backing file block
 #     device, null disables the backing file entirely.  Defaults to
@@ -4385,11 +4380,10 @@
             '*page-cache-size': 'int',
             '*debug': 'int' } }
 
-##
-# @BlockdevOptionsCurlBase:
-#
 # Driver specific block device options shared by all protocols
 # supported by the curl backend.
+##
+# @BlockdevOptionsCurlBase:
 #
 # @url: URL of the image file
 #
@@ -4645,11 +4639,10 @@
   'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap',
             '*on-cbw-error': 'OnCbwError', '*cbw-timeout': 'uint32' } }
 
-##
-# @BlockdevOptions:
-#
 # Options for creating a block device.  Many options are available for
-# all block devices, independent of the block driver:
+# all block devices, independent of the block driver.
+##
+# @BlockdevOptions:
 #
 # @driver: block driver name
 #
diff --git a/qapi/block-export.json b/qapi/block-export.json
index 3919a2d5b9d..dc328097a94 100644
--- a/qapi/block-export.json
+++ b/qapi/block-export.json
@@ -77,11 +77,10 @@
             '*max-connections': 'uint32' },
   'allow-preconfig': true }
 
-##
-# @BlockExportOptionsNbdBase:
-#
 # An NBD block export (common options shared between nbd-server-add
 # and the NBD branch of block-export-add).
+##
+# @BlockExportOptionsNbdBase:
 #
 # @name: Export name.  If unspecified, the @device parameter is used
 #     as the export name.  (Since 2.12)
@@ -213,10 +212,9 @@
             '*logical-block-size': 'size',
             '*serial': 'str' } }
 
-##
-# @NbdServerAddOptions:
-#
 # An NBD block export, per legacy nbd-server-add command.
+##
+# @NbdServerAddOptions:
 #
 # @device: The device name or node name of the node to be exported
 #
diff --git a/qapi/char.json b/qapi/char.json
index 777dde55d97..ab4c23976ed 100644
--- a/qapi/char.json
+++ b/qapi/char.json
@@ -189,10 +189,9 @@
   'data': {'device': 'str', 'size': 'int', '*format': 'DataFormat'},
   'returns': 'str' }
 
-##
-# @ChardevCommon:
-#
 # Configuration shared across all chardev backends
+##
+# @ChardevCommon:
 #
 # @logfile: The name of a logfile to save output
 #
diff --git a/qapi/crypto.json b/qapi/crypto.json
index e102be337bb..d4ac5f23c8c 100644
--- a/qapi/crypto.json
+++ b/qapi/crypto.json
@@ -163,10 +163,9 @@
 #  'prefix': 'QCRYPTO_BLOCK_FORMAT',
   'data': ['qcow', 'luks']}
 
-##
-# @QCryptoBlockOptionsBase:
-#
 # The common options that apply to all full disk encryption formats
+##
+# @QCryptoBlockOptionsBase:
 #
 # @format: the encryption format
 #
@@ -175,10 +174,9 @@
 { 'struct': 'QCryptoBlockOptionsBase',
   'data': { 'format': 'QCryptoBlockFormat' }}
 
-##
-# @QCryptoBlockOptionsQCow:
-#
 # The options that apply to QCow/QCow2 AES-CBC encryption format
+##
+# @QCryptoBlockOptionsQCow:
 #
 # @key-secret: the ID of a QCryptoSecret object providing the
 #     decryption key.  Mandatory except when probing image for
@@ -189,11 +187,10 @@
 { 'struct': 'QCryptoBlockOptionsQCow',
   'data': { '*key-secret': 'str' }}
 
+# The options that apply to LUKS encryption format:
 ##
 # @QCryptoBlockOptionsLUKS:
 #
-# The options that apply to LUKS encryption format
-#
 # @key-secret: the ID of a QCryptoSecret object providing the
 #     decryption key.  Mandatory except when probing image for
 #     metadata only.
@@ -203,10 +200,9 @@
 { 'struct': 'QCryptoBlockOptionsLUKS',
   'data': { '*key-secret': 'str' }}
 
-##
-# @QCryptoBlockCreateOptionsLUKS:
-#
 # The options that apply to LUKS encryption format initialization
+##
+# @QCryptoBlockCreateOptionsLUKS:
 #
 # @cipher-alg: the cipher algorithm for data encryption Currently
 #     defaults to 'aes-256'.
@@ -268,11 +264,10 @@
   'data': { 'qcow': 'QCryptoBlockOptionsQCow',
             'luks': 'QCryptoBlockCreateOptionsLUKS' } }
 
-##
-# @QCryptoBlockInfoBase:
-#
 # The common information that applies to all full disk encryption
 # formats
+##
+# @QCryptoBlockInfoBase:
 #
 # @format: the encryption format
 #
@@ -424,10 +419,9 @@
   'data': {
           'luks': 'QCryptoBlockAmendOptionsLUKS' } }
 
-##
-# @SecretCommonProperties:
-#
 # Properties for objects of classes derived from secret-common.
+##
+# @SecretCommonProperties:
 #
 # @loaded: if true, the secret is loaded immediately when applying
 #     this option and will probably fail when processing the next
@@ -490,10 +484,9 @@
   'base': 'SecretCommonProperties',
   'data': { 'serial': 'int32' } }
 
-##
-# @TlsCredsProperties:
-#
 # Properties for objects of classes derived from tls-creds.
+##
+# @TlsCredsProperties:
 #
 # @verify-peer: if true the peer credentials will be verified once the
 #     handshake is completed.  This is a no-op for anonymous
diff --git a/qapi/machine.json b/qapi/machine.json
index 64a77557571..35cca12ba41 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -497,10 +497,9 @@
 { 'enum': 'NumaOptionsType',
   'data': [ 'node', 'dist', 'cpu', 'hmat-lb', 'hmat-cache' ] }
 
-##
-# @NumaOptions:
-#
 # A discriminated record of NUMA options.  (for OptsVisitor)
+##
+# @NumaOptions:
 #
 # @type: NUMA option type
 #
@@ -1198,10 +1197,9 @@
 { 'event': 'BALLOON_CHANGE',
   'data': { 'actual': 'int' } }
 
-##
-# @HvBalloonInfo:
-#
 # hv-balloon guest-provided memory status information.
+##
+# @HvBalloonInfo:
 #
 # @committed: the amount of memory in use inside the guest plus the
 #     amount of the memory unusable inside the guest (ballooned out,
diff --git a/qapi/net.json b/qapi/net.json
index c19df435a53..dc616d010f0 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -722,10 +722,9 @@
             { 'name': 'vmnet-shared', 'if': 'CONFIG_VMNET' },
             { 'name': 'vmnet-bridged', 'if': 'CONFIG_VMNET' }] }
 
-##
-# @Netdev:
-#
 # Captures the configuration of a network device.
+##
+# @Netdev:
 #
 # @id: identifier for monitor commands.
 #
@@ -894,8 +893,6 @@
 ##
 # @AnnounceParameters:
 #
-# Parameters for self-announce timers
-#
 # @initial: Initial delay (in ms) before sending the first GARP/RARP
 #     announcement
 #
diff --git a/qapi/qom.json b/qapi/qom.json
index 38dde6d785a..8f0601859b1 100644
--- a/qapi/qom.json
+++ b/qapi/qom.json
@@ -272,11 +272,10 @@
             '*max_queue_size': 'uint32',
             '*vnet_hdr_support': 'bool' } }
 
-##
-# @CryptodevBackendProperties:
-#
 # Properties for cryptodev-backend and cryptodev-backend-builtin
 # objects.
+##
+# @CryptodevBackendProperties:
 #
 # @queues: the number of queues for the cryptodev backend.  Ignored
 #     for cryptodev-backend and must be 1 for
@@ -338,10 +337,9 @@
 { 'enum': 'NetfilterInsert',
   'data': [ 'before', 'behind' ] }
 
-##
-# @NetfilterProperties:
-#
 # Properties for objects of classes derived from netfilter.
+##
+# @NetfilterProperties:
 #
 # @netdev: id of the network device backend to filter
 #
@@ -516,10 +514,9 @@
             '*repeat': 'bool',
             '*grab-toggle': 'GrabToggleKeys' } }
 
-##
-# @EventLoopBaseProperties:
-#
 # Common properties for event loops
+##
+# @EventLoopBaseProperties:
 #
 # @aio-max-batch: maximum number of requests in a batch for the AIO
 #     engine, 0 means that the engine will use its default.
@@ -576,10 +573,9 @@
   'base': 'EventLoopBaseProperties',
   'data': {} }
 
-##
-# @MemoryBackendProperties:
-#
 # Properties for objects of classes derived from memory-backend.
+##
+# @MemoryBackendProperties:
 #
 # @merge: if true, mark the memory as mergeable (default depends on
 #     the machine type)
@@ -826,10 +822,9 @@
   'data': { 'pci-dev': 'str',
             'node': 'uint32' } }
 
-##
-# @RngProperties:
-#
 # Properties for objects of classes derived from rng.
+##
+# @RngProperties:
 #
 # @opened: if true, the device is opened immediately when applying
 #     this option and will probably fail when processing the next
@@ -1008,10 +1003,9 @@
     { 'name': 'x-vfio-user-server', 'features': [ 'unstable' ] }
   ] }
 
-##
-# @ObjectOptions:
-#
 # Describes the options of a user creatable QOM object.
+##
+# @ObjectOptions:
 #
 # @qom-type: the class name for the object to be created
 #
diff --git a/qapi/ui.json b/qapi/ui.json
index c12f5292571..2d0aa407aca 100644
--- a/qapi/ui.json
+++ b/qapi/ui.json
@@ -39,8 +39,6 @@
 ##
 # @SetPasswordOptions:
 #
-# Options for set_password.
-#
 # @protocol:
 #     - 'vnc' to modify the VNC server password
 #     - 'spice' to modify the Spice server password
@@ -94,8 +92,6 @@
 ##
 # @ExpirePasswordOptions:
 #
-# General options for expire_password.
-#
 # @protocol:
 #     - 'vnc' to modify the VNC server expiration
 #     - 'spice' to modify the Spice server expiration
@@ -206,8 +202,6 @@
 ##
 # @SpiceBasicInfo:
 #
-# The basic information for SPICE network connection
-#
 # @host: IP address
 #
 # @port: port number
@@ -469,8 +463,6 @@
 ##
 # @VncBasicInfo:
 #
-# The basic information for vnc network connection
-#
 # @host: IP address
 #
 # @service: The service name of the vnc port.  This may depend on the
@@ -1598,8 +1590,6 @@
 ##
 # @DisplayReloadOptions:
 #
-# Options of the display configuration reload.
-#
 # @type: Specify the display type.
 #
 # Since: 6.0
@@ -1641,8 +1631,6 @@
 ##
 # @DisplayUpdateOptionsVNC:
 #
-# Specify the VNC reload options.
-#
 # @addresses: If specified, change set of addresses to listen for
 #     connections.  Addresses configured for websockets are not
 #     touched.
@@ -1655,8 +1643,6 @@
 ##
 # @DisplayUpdateOptions:
 #
-# Options of the display configuration reload.
-#
 # @type: Specify the display type.
 #
 # Since: 7.1
-- 
2.44.0

[PATCH 16/20] qapi: rewrite StatsFilter comment

[John Snow]

Rewrite the StatsFilter intro paragraph to be more meaningful to
end-users when it is inlined in generated documentation.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/stats.json | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/qapi/stats.json b/qapi/stats.json
index 578b52c7ef7..c4a9f3ff70e 100644
--- a/qapi/stats.json
+++ b/qapi/stats.json
@@ -112,10 +112,6 @@
 ##
 # @StatsFilter:
 #
-# The arguments to the query-stats command; specifies a target for
-# which to request statistics and optionally the required subset of
-# information for that target.
-#
 # @target: the kind of objects to query.  Note that each possible
 #          target may enable additional filtering options
 #
@@ -183,8 +179,8 @@
 # Return runtime-collected statistics for objects such as the VM or
 # its vCPUs.
 #
-# The arguments are a StatsFilter and specify the provider and objects
-# to return statistics about.
+# The arguments specify a target for which to request statistics and
+# optionally the required subset of information for that target.
 #
 # Returns: a list of StatsResult, one for each provider and object
 #     (e.g., for each vCPU).
-- 
2.44.0

[PATCH 17/20] qapi: rewrite BlockExportOptions doc block

[John Snow]

Rephrase this paragraph so that it can apply to any commands that
inherit from this object.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/block-export.json | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/qapi/block-export.json b/qapi/block-export.json
index dc328097a94..550763a9f6a 100644
--- a/qapi/block-export.json
+++ b/qapi/block-export.json
@@ -342,9 +342,6 @@
 ##
 # @BlockExportOptions:
 #
-# Describes a block export, i.e. how single node should be exported on
-# an external interface.
-#
 # @type: Block export type
 #
 # @id: A unique identifier for the block export (across all export
@@ -396,6 +393,9 @@
 #
 # Creates a new block export.
 #
+# Arguments describe a block export, i.e. how single node should be
+# exported on an external interface.
+#
 # Since: 5.2
 ##
 { 'command': 'block-export-add',
-- 
2.44.0

[PATCH 18/20] qapi: ensure all errors sections are uniformly typset

[John Snow]

Transactions have the only instance of an Errors section that isn't a
rST list; turn it into one.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/transaction.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/qapi/transaction.json b/qapi/transaction.json
index 5749c133d4a..07afc269d54 100644
--- a/qapi/transaction.json
+++ b/qapi/transaction.json
@@ -235,7 +235,7 @@
 #     additional detail.
 #
 # Errors:
-#     Any errors from commands in the transaction
+#     - Any errors from commands in the transaction
 #
 # Note: The transaction aborts on the first failure.  Therefore, there
 #     will be information on only one failed operation returned in an
-- 
2.44.0

[PATCH 19/20] qapi: convert "Note" sections to plain rST

[John Snow]

We do not need a dedicated section for notes. By eliminating a specially
parsed section, these notes can be treated as normal rST paragraphs in
the new QMP reference manual, and can be placed and styled much more
flexibly.

Update the QAPI parser to now prohibit special "Note" sections while
suggesting a new syntax.

The exact formatting to use is a matter of taste, but a good candidate
is simply:

.. note:: lorem ipsum ...

but there are other choices, too. The Sphinx readthedocs theme offers
theming for the following forms (capitalization unimportant); all are
adorned with a (!) symbol in the title bar for rendered HTML docs.

These are rendered in orange:

.. Attention:: ...
.. Caution:: ...
.. WARNING:: ...

These are rendered in red:

.. DANGER:: ...
.. Error:: ...

These are rendered in green:

.. Hint:: ...
.. Important:: ...
.. Tip:: ...

These are rendered in blue:

.. Note:: ...
.. admonition:: custom title

   admonition body text

This patch uses ".. notes::" almost everywhere, with just two "caution"
directives. ".. admonition:: notes" is used in a few places where we had
an ordered list of multiple notes.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/block-core.json                          | 30 +++----
 qapi/block.json                               |  2 +-
 qapi/char.json                                | 12 +--
 qapi/control.json                             | 15 ++--
 qapi/dump.json                                |  2 +-
 qapi/introspect.json                          |  6 +-
 qapi/machine-target.json                      | 26 +++---
 qapi/machine.json                             | 47 +++++-----
 qapi/migration.json                           | 12 +--
 qapi/misc.json                                | 88 +++++++++----------
 qapi/net.json                                 |  6 +-
 qapi/pci.json                                 |  7 +-
 qapi/qdev.json                                | 30 +++----
 qapi/qom.json                                 | 19 ++--
 qapi/rocker.json                              | 16 ++--
 qapi/run-state.json                           | 18 ++--
 qapi/sockets.json                             | 10 +--
 qapi/stats.json                               | 22 ++---
 qapi/transaction.json                         |  8 +-
 qapi/ui.json                                  | 29 +++---
 qapi/virtio.json                              | 12 +--
 qga/qapi-schema.json                          | 48 +++++-----
 scripts/qapi/parser.py                        |  9 ++
 tests/qapi-schema/doc-empty-section.err       |  2 +-
 tests/qapi-schema/doc-empty-section.json      |  2 +-
 tests/qapi-schema/doc-good.json               |  6 +-
 tests/qapi-schema/doc-good.out                | 10 ++-
 tests/qapi-schema/doc-good.txt                | 14 ++-
 .../qapi-schema/doc-interleaved-section.json  |  2 +-
 29 files changed, 258 insertions(+), 252 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 64fe5240cc9..530af40404d 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1510,7 +1510,7 @@
 # @mode: whether and how QEMU should create a new image, default is
 #     'absolute-paths'.
 #
-# Note: Either @device or @node-name must be set but not both.
+# .. note:: Either @device or @node-name must be set but not both.
 #
 ##
 { 'struct': 'BlockdevSnapshotSync',
@@ -1616,9 +1616,9 @@
 #
 # @unstable: Member @x-perf is experimental.
 #
-# Note: @on-source-error and @on-target-error only affect background
-#     I/O.  If an error occurs during a guest write request, the
-#     device's rerror/werror actions will be used.
+# .. note:: @on-source-error and @on-target-error only affect background
+#    I/O.  If an error occurs during a guest write request, the device's
+#    rerror/werror actions will be used.
 #
 # Since: 4.2
 ##
@@ -5534,8 +5534,8 @@
 #     after this event and must be repaired (Since 2.2; before, every
 #     BLOCK_IMAGE_CORRUPTED event was fatal)
 #
-# Note: If action is "stop", a STOP event will eventually follow the
-#     BLOCK_IO_ERROR event.
+# .. note:: If action is "stop", a STOP event will eventually follow the
+#    BLOCK_IO_ERROR event.
 #
 # Example:
 #
@@ -5581,8 +5581,8 @@
 #     field is a debugging aid for humans, it should not be parsed by
 #     applications) (since: 2.2)
 #
-# Note: If action is "stop", a STOP event will eventually follow the
-#     BLOCK_IO_ERROR event
+# .. note:: If action is "stop", a STOP event will eventually follow the
+#    BLOCK_IO_ERROR event.
 #
 # Since: 0.13
 #
@@ -5720,8 +5720,8 @@
 #
 # @speed: rate limit, bytes per second
 #
-# Note: The "ready to complete" status is always reset by a
-#     @BLOCK_JOB_ERROR event
+# .. note:: The "ready to complete" status is always reset by a
+#    @BLOCK_JOB_ERROR event.
 #
 # Since: 1.3
 #
@@ -5974,7 +5974,7 @@
 #
 # @sectors-count: failed read operation sector count
 #
-# Note: This event is rate-limited.
+# .. note:: This event is rate-limited.
 #
 # Since: 2.0
 #
@@ -6005,7 +6005,7 @@
 #
 # @sectors-count: failed read operation sector count
 #
-# Note: This event is rate-limited.
+# .. note:: This event is rate-limited.
 #
 # Since: 2.0
 #
@@ -6037,9 +6037,9 @@
 #
 # @name: the name of the internal snapshot to be created
 #
-# Notes: In transaction, if @name is empty, or any snapshot matching
-#     @name exists, the operation will fail.  Only some image formats
-#     support it, for example, qcow2, and rbd.
+# .. note:: In a transaction, if @name is empty or any snapshot matching
+#    @name exists, the operation will fail.  Only some image formats
+#    support it; for example, qcow2, and rbd.
 #
 # Since: 1.7
 ##
diff --git a/qapi/block.json b/qapi/block.json
index 5de99fe09d9..ea81d9e1921 100644
--- a/qapi/block.json
+++ b/qapi/block.json
@@ -113,7 +113,7 @@
 # Errors:
 #     - If @device is not a valid block device, DeviceNotFound
 #
-# Notes: Ejecting a device with no media results in success
+# .. note:: Ejecting a device with no media results in success.
 #
 # Since: 0.14
 #
diff --git a/qapi/char.json b/qapi/char.json
index ab4c23976ed..0f39c2d5cdf 100644
--- a/qapi/char.json
+++ b/qapi/char.json
@@ -21,8 +21,8 @@
 #     backend (e.g. with the chardev=... option) is in open or closed
 #     state (since 2.1)
 #
-# Notes: @filename is encoded using the QEMU command line character
-#     device encoding.  See the QEMU man page for details.
+# .. note:: @filename is encoded using the QEMU command line character
+#    device encoding.  See the QEMU man page for details.
 #
 # Since: 0.14
 ##
@@ -387,9 +387,9 @@
 #
 # @rows: console height, in chars
 #
-# Note: the options are only effective when the VNC or SDL graphical
-#     display backend is active.  They are ignored with the GTK,
-#     Spice, VNC and D-Bus display backends.
+# .. note:: The options are only effective when the VNC or SDL graphical
+#    display backend is active.  They are ignored with the GTK, Spice,
+#    VNC and D-Bus display backends.
 #
 # Since: 1.5
 ##
@@ -805,7 +805,7 @@
 #
 # @open: true if the guest has opened the virtio-serial port
 #
-# Note: This event is rate-limited.
+# .. note:: This event is rate-limited.
 #
 # Since: 2.1
 #
diff --git a/qapi/control.json b/qapi/control.json
index 10c906fa0e7..2498e5dd6ba 100644
--- a/qapi/control.json
+++ b/qapi/control.json
@@ -22,14 +22,13 @@
 #          "arguments": { "enable": [ "oob" ] } }
 #     <- { "return": {} }
 #
-# Notes: This command is valid exactly when first connecting: it must
-#     be issued before any other command will be accepted, and will
-#     fail once the monitor is accepting other commands.  (see qemu
-#     docs/interop/qmp-spec.rst)
+# .. note:: This command is valid exactly when first connecting: it must
+#    be issued before any other command will be accepted, and will fail
+#    once the monitor is accepting other commands.
+#    (see :doc:`/interop/qmp-spec`)
 #
-#     The QMP client needs to explicitly enable QMP capabilities,
-#     otherwise all the QMP capabilities will be turned off by
-#     default.
+# .. note:: The QMP client needs to explicitly enable QMP capabilities,
+#    otherwise all the QMP capabilities will be turned off by default.
 #
 # Since: 0.13
 ##
@@ -150,7 +149,7 @@
 #          ]
 #        }
 #
-# Note: This example has been shortened as the real response is too
+#     Note: This example has been shortened as the real response is too
 #     long.
 ##
 { 'command': 'query-commands', 'returns': ['CommandInfo'],
diff --git a/qapi/dump.json b/qapi/dump.json
index 2fa9504d864..f9aee7ea1dd 100644
--- a/qapi/dump.json
+++ b/qapi/dump.json
@@ -90,7 +90,7 @@
 #     and @length is not allowed to be specified with non-elf @format
 #     at the same time (since 2.0)
 #
-# Note: All boolean arguments default to false
+# .. note:: All boolean arguments default to false.
 #
 # Since: 1.2
 #
diff --git a/qapi/introspect.json b/qapi/introspect.json
index b041b02ba8c..b15052ec21a 100644
--- a/qapi/introspect.json
+++ b/qapi/introspect.json
@@ -41,9 +41,9 @@
 #     names are guaranteed to be unique (no name will be duplicated
 #     with different meta-types).
 #
-# Note: the QAPI schema is also used to help define *internal*
-#     interfaces, by defining QAPI types.  These are not part of the
-#     QMP wire ABI, and therefore not returned by this command.
+# .. note:: The QAPI schema is also used to help define *internal*
+#    interfaces, by defining QAPI types.  These are not part of the QMP
+#    wire ABI, and therefore not returned by this command.
 #
 # Since: 2.5
 ##
diff --git a/qapi/machine-target.json b/qapi/machine-target.json
index 29428530923..a8d9ec87f59 100644
--- a/qapi/machine-target.json
+++ b/qapi/machine-target.json
@@ -49,15 +49,15 @@
 #     to be migration-safe, but allows tooling to get an insight and
 #     work with model details.
 #
-# Note: When a non-migration-safe CPU model is expanded in static
-#     mode, some features enabled by the CPU model may be omitted,
-#     because they can't be implemented by a static CPU model
-#     definition (e.g. cache info passthrough and PMU passthrough in
-#     x86). If you need an accurate representation of the features
-#     enabled by a non-migration-safe CPU model, use @full.  If you
-#     need a static representation that will keep ABI compatibility
-#     even when changing QEMU version or machine-type, use @static
-#     (but keep in mind that some features may be omitted).
+# .. note:: When a non-migration-safe CPU model is expanded in static
+#    mode, some features enabled by the CPU model may be omitted,
+#    because they can't be implemented by a static CPU model definition
+#    (e.g. cache info passthrough and PMU passthrough in x86). If you
+#    need an accurate representation of the features enabled by a
+#    non-migration-safe CPU model, use @full.  If you need a static
+#    representation that will keep ABI compatibility even when changing
+#    QEMU version or machine-type, use @static (but keep in mind that
+#    some features may be omitted).
 #
 # Since: 2.8
 ##
@@ -175,8 +175,8 @@
 #     - if a model contains an unknown cpu definition name, unknown
 #       properties or properties with wrong types.
 #
-# Note: this command isn't specific to s390x, but is only implemented
-#     on this architecture currently.
+# .. note:: This command isn't specific to s390x, but is only
+#    implemented on this architecture currently.
 #
 # Since: 2.8
 ##
@@ -229,8 +229,8 @@
 #     - if a model contains an unknown cpu definition name, unknown
 #       properties or properties with wrong types.
 #
-# Note: this command isn't specific to s390x, but is only implemented
-#     on this architecture currently.
+# .. note:: This command isn't specific to s390x, but is only
+#    implemented on this architecture currently.
 #
 # Since: 2.8
 ##
diff --git a/qapi/machine.json b/qapi/machine.json
index 35cca12ba41..e9c9bef940d 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -24,9 +24,9 @@
 #
 # @avr: since 5.1
 #
-# Notes: The resulting QMP strings can be appended to the
-#     "qemu-system-" prefix to produce the corresponding QEMU
-#     executable name.  This is true even for "qemu-system-x86_64".
+# .. note:: The resulting QMP strings can be appended to the
+#    "qemu-system-" prefix to produce the corresponding QEMU executable
+#    name.  This is true even for "qemu-system-x86_64".
 #
 # Since: 3.0
 ##
@@ -305,8 +305,9 @@
 #
 # Since: 0.14
 #
-# Notes: If no UUID was specified for the guest, a null UUID is
-#     returned.
+# .. note:: If no UUID was specified for the guest, a null UUID is
+#    returned.
+#
 ##
 { 'struct': 'UuidInfo', 'data': {'UUID': 'str'} }
 
@@ -367,10 +368,10 @@
 #
 # Since: 0.14
 #
-# Notes: A guest may or may not respond to this command.  This command
-#     returning does not indicate that a guest has accepted the
-#     request or that it has shut down.  Many guests will respond to
-#     this command by prompting the user in some way.
+# .. note:: A guest may or may not respond to this command.  This
+#    command returning does not indicate that a guest has accepted the
+#    request or that it has shut down.  Many guests will respond to this
+#    command by prompting the user in some way.
 #
 # Example:
 #
@@ -389,8 +390,8 @@
 #
 # Since: 1.1
 #
-# Note: prior to 4.0, this command does nothing in case the guest
-#     isn't suspended.
+# .. note:: Prior to 4.0, this command does nothing in case the guest
+#    isn't suspended.
 #
 # Example:
 #
@@ -440,8 +441,8 @@
 #
 # Since: 0.14
 #
-# Note: prior to 2.1, this command was only supported for x86 and s390
-#     VMs
+# .. note:: Prior to 2.1, this command was only supported for x86 and
+#    s390 VMs
 #
 # Example:
 #
@@ -838,7 +839,7 @@
 #
 # Since: 0.14
 #
-# Notes: Errors were not reliably returned until 1.1
+# .. caution:: Errors were not reliably returned until 1.1.
 #
 # Example:
 #
@@ -864,7 +865,7 @@
 #
 # Since: 0.14
 #
-# Notes: Errors were not reliably returned until 1.1
+# .. caution:: Errors were not reliably returned until 1.1.
 #
 # Example:
 #
@@ -994,8 +995,8 @@
 #
 # @thread-id: thread number within the core the CPU  belongs to
 #
-# Note: management should be prepared to pass through additional
-#     properties with device_add.
+# .. note:: Management should be prepared to pass through additional
+#    properties with device_add.
 #
 # Since: 2.7
 ##
@@ -1122,9 +1123,9 @@
 #       the KVM kernel module cannot support it, KVMMissingCap
 #     - If no balloon device is present, DeviceNotActive
 #
-# Notes: This command just issues a request to the guest.  When it
-#     returns, the balloon size may not have changed.  A guest can
-#     change the balloon size independent of this command.
+# .. note:: This command just issues a request to the guest.  When it
+#    returns, the balloon size may not have changed.  A guest can change
+#    the balloon size independent of this command.
 #
 # Since: 0.14
 #
@@ -1184,7 +1185,7 @@
 # @actual: the logical size of the VM in bytes Formula used:
 #     logical_vm_size = vm_ram_size - balloon_size
 #
-# Note: this event is rate-limited.
+# .. note:: This event is rate-limited.
 #
 # Since: 1.2
 #
@@ -1246,7 +1247,7 @@
 # Emitted when the hv-balloon driver receives a "STATUS" message from
 # the guest.
 #
-# Note: this event is rate-limited.
+# .. note:: This event is rate-limited.
 #
 # Since: 8.2
 #
@@ -1591,7 +1592,7 @@
 #
 # @qom-path: path to the device object in the QOM tree (since 6.2)
 #
-# Note: this event is rate-limited.
+# .. note:: This event is rate-limited.
 #
 # Since: 5.1
 #
diff --git a/qapi/migration.json b/qapi/migration.json
index 89047d46c7c..a7b8ff138e3 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -1428,8 +1428,8 @@
 #
 # Cancel the current executing migration process.
 #
-# Notes: This command succeeds even if there is no migration process
-#     running.
+# .. note:: This command succeeds even if there is no migration process
+#    running.
 #
 # Since: 0.14
 #
@@ -1561,16 +1561,16 @@
 #
 # Since: 0.14
 #
-# Notes:
+# .. admonition:: Notes
 #
 #     1. The 'query-migrate' command should be used to check
 #        migration's progress and final result (this information is
 #        provided by the 'status' member)
 #
-#     2. All boolean arguments default to false
+#     2. All boolean arguments default to false.
 #
 #     3. The user Monitor's "detach" argument is invalid in QMP and
-#        should not be used
+#        should not be used.
 #
 #     4. The uri argument should have the Uniform Resource Identifier
 #        of default destination VM. This connection will be bound to
@@ -1644,7 +1644,7 @@
 #
 # Since: 2.3
 #
-# Notes:
+# .. admonition:: Notes
 #
 #     1. It's a bad idea to use a string for the uri, but it needs to
 #        stay compatible with -incoming and the format of the uri is
diff --git a/qapi/misc.json b/qapi/misc.json
index 4b41e15dcd4..b04efbadec6 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -103,9 +103,9 @@
 #
 # Returns a list of information about each iothread.
 #
-# Note: this list excludes the QEMU main loop thread, which is not
-#     declared using the -object iothread command-line option.  It is
-#     always the main thread of the process.
+# .. note:: This list excludes the QEMU main loop thread, which is not
+#    declared using the ``-object iothread`` command-line option.  It is
+#    always the main thread of the process.
 #
 # Returns: a list of @IOThreadInfo for each iothread
 #
@@ -136,13 +136,13 @@
 #
 # Since: 0.14
 #
-# Notes: This function will succeed even if the guest is already in
-#     the stopped state.  In "inmigrate" state, it will ensure that
-#     the guest remains paused once migration finishes, as if the -S
-#     option was passed on the command line.
+# .. note:: This function will succeed even if the guest is already in
+#    the stopped state.  In "inmigrate" state, it will ensure that the
+#    guest remains paused once migration finishes, as if the ``-S``
+#    option was passed on the command line.
 #
-#     In the "suspended" state, it will completely stop the VM and
-#     cause a transition to the "paused" state.  (Since 9.0)
+#    In the "suspended" state, it will completely stop the VM and cause
+#    a transition to the "paused" state.  (Since 9.0)
 #
 # Example:
 #
@@ -158,15 +158,15 @@
 #
 # Since: 0.14
 #
-# Notes: This command will succeed if the guest is currently running.
-#     It will also succeed if the guest is in the "inmigrate" state;
-#     in this case, the effect of the command is to make sure the
-#     guest starts once migration finishes, removing the effect of the
-#     -S command line option if it was passed.
+# .. note:: This command will succeed if the guest is currently running.
+#    It will also succeed if the guest is in the "inmigrate" state; in
+#    this case, the effect of the command is to make sure the guest
+#    starts once migration finishes, removing the effect of the ``-S``
+#    command line option if it was passed.
 #
-#     If the VM was previously suspended, and not been reset or woken,
-#     this command will transition back to the "suspended" state.
-#     (Since 9.0)
+#    If the VM was previously suspended, and not been reset or woken,
+#    this command will transition back to the "suspended" state.  (Since
+#    9.0)
 #
 # Example:
 #
@@ -219,18 +219,18 @@
 #
 # Since: 0.14
 #
-# Notes: This command only exists as a stop-gap.  Its use is highly
-#     discouraged.  The semantics of this command are not guaranteed:
-#     this means that command names, arguments and responses can
-#     change or be removed at ANY time.  Applications that rely on
-#     long term stability guarantees should NOT use this command.
+# .. note:: This command only exists as a stop-gap.  Its use is highly
+#    discouraged.  The semantics of this command are not guaranteed:
+#    this means that command names, arguments and responses can change
+#    or be removed at ANY time.  Applications that rely on long term
+#    stability guarantees should NOT use this command.
 #
-#     Known limitations:
+#    Known limitations:
 #
-#     * This command is stateless, this means that commands that
-#       depend on state information (such as getfd) might not work
+#    * This command is stateless, this means that commands that
+#      depend on state information (such as getfd) might not work.
 #
-#     * Commands that prompt the user for data don't currently work
+#    * Commands that prompt the user for data don't currently work.
 #
 # Example:
 #
@@ -252,11 +252,11 @@
 #
 # Since: 0.14
 #
-# Notes: If @fdname already exists, the file descriptor assigned to it
-#     will be closed and replaced by the received file descriptor.
+# .. note:: If @fdname already exists, the file descriptor assigned to
+#    it will be closed and replaced by the received file descriptor.
 #
-#     The 'closefd' command can be used to explicitly close the file
-#     descriptor when it is no longer needed.
+#    The 'closefd' command can be used to explicitly close the file
+#    descriptor when it is no longer needed.
 #
 # Example:
 #
@@ -279,11 +279,11 @@
 #
 # Since: 8.0
 #
-# Notes: If @fdname already exists, the file descriptor assigned to it
-#     will be closed and replaced by the received file descriptor.
+# .. note:: If @fdname already exists, the file descriptor assigned to
+#    it will be closed and replaced by the received file descriptor.
 #
-#     The 'closefd' command can be used to explicitly close the file
-#     descriptor when it is no longer needed.
+#    The 'closefd' command can be used to explicitly close the file
+#    descriptor when it is no longer needed.
 #
 # Example:
 #
@@ -339,10 +339,9 @@
 #     - If file descriptor was not received, GenericError
 #     - If @fdset-id is a negative value, GenericError
 #
-# Notes:
-#     The list of fd sets is shared by all monitor connections.
+# .. note:: The list of fd sets is shared by all monitor connections.
 #
-#     If @fdset-id is not specified, a new fd set will be created.
+# .. note:: If @fdset-id is not specified, a new fd set will be created.
 #
 # Since: 1.2
 #
@@ -370,11 +369,10 @@
 #
 # Since: 1.2
 #
-# Notes:
-#     The list of fd sets is shared by all monitor connections.
+# .. note:: The list of fd sets is shared by all monitor connections.
 #
-#     If @fd is not specified, all file descriptors in @fdset-id will
-#     be removed.
+# .. note:: If @fd is not specified, all file descriptors in @fdset-id
+#    will be removed.
 #
 # Example:
 #
@@ -420,7 +418,7 @@
 #
 # Since: 1.2
 #
-# Note: The list of fd sets is shared by all monitor connections.
+# .. note:: The list of fd sets is shared by all monitor connections.
 #
 # Example:
 #
@@ -561,9 +559,9 @@
 #
 # @qom-path: path to the RTC object in the QOM tree
 #
-# Note: This event is rate-limited.  It is not guaranteed that the RTC
-#     in the system implements this event, or even that the system has
-#     an RTC at all.
+# .. note:: This event is rate-limited.  It is not guaranteed that the
+#    RTC in the system implements this event, or even that the system
+#    has an RTC at all.
 #
 # Since: 0.13
 #
diff --git a/qapi/net.json b/qapi/net.json
index dc616d010f0..4ac7fdc7e6c 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -22,9 +22,9 @@
 #
 # Since: 0.14
 #
-# Notes: Not all network adapters support setting link status.  This
-#     command will succeed even if the network adapter does not
-#     support link status notification.
+# .. note:: Not all network adapters support setting link status.  This
+#    command will succeed even if the network adapter does not support
+#    link status notification.
 #
 # Example:
 #
diff --git a/qapi/pci.json b/qapi/pci.json
index 08bf6958634..f51159a2c4c 100644
--- a/qapi/pci.json
+++ b/qapi/pci.json
@@ -146,8 +146,8 @@
 #
 # @regions: a list of the PCI I/O regions associated with the device
 #
-# Notes: the contents of @class_info.desc are not stable and should
-#     only be treated as informational.
+# .. note:: The contents of @class_info.desc are not stable and should
+#    only be treated as informational.
 #
 # Since: 0.14
 ##
@@ -311,7 +311,8 @@
 #           ]
 #        }
 #
-# Note: This example has been shortened as the real response is too
+#     Note: This example has been shortened as the real response is too
 #     long.
+#
 ##
 { 'command': 'query-pci', 'returns': ['PciInfo'] }
diff --git a/qapi/qdev.json b/qapi/qdev.json
index facaa0bc6a2..d031fc3590d 100644
--- a/qapi/qdev.json
+++ b/qapi/qdev.json
@@ -20,9 +20,9 @@
 # Returns: a list of ObjectPropertyInfo describing a devices
 #     properties
 #
-# Note: objects can create properties at runtime, for example to
-#     describe links between different devices and/or objects.  These
-#     properties are not included in the output of this command.
+# .. note:: Objects can create properties at runtime, for example to
+#    describe links between different devices and/or objects.  These
+#    properties are not included in the output of this command.
 #
 # Since: 1.2
 ##
@@ -51,7 +51,7 @@
 #     supports JSON syntax without the reference counting leak that
 #     broke hot-unplug
 #
-# Notes:
+# .. admonition:: Notes
 #
 #     1. Additional arguments depend on the type.
 #
@@ -59,8 +59,8 @@
 #        the 'docs/qdev-device-use.txt' file.
 #
 #     3. It's possible to list device properties by running QEMU with
-#        the "-device DEVICE,help" command-line argument, where DEVICE
-#        is the device's name
+#        the ``-device DEVICE,help`` command-line argument, where DEVICE
+#        is the device's name.
 #
 # Example:
 #
@@ -92,15 +92,15 @@
 # Errors:
 #     - If @id is not a valid device, DeviceNotFound
 #
-# Notes: When this command completes, the device may not be removed
-#     from the guest.  Hot removal is an operation that requires guest
-#     cooperation.  This command merely requests that the guest begin
-#     the hot removal process.  Completion of the device removal
-#     process is signaled with a DEVICE_DELETED event.  Guest reset
-#     will automatically complete removal for all devices.  If a
-#     guest-side error in the hot removal process is detected, the
-#     device will not be removed and a DEVICE_UNPLUG_GUEST_ERROR event
-#     is sent.  Some errors cannot be detected.
+# .. note:: When this command completes, the device may not be removed
+#    from the guest.  Hot removal is an operation that requires guest
+#    cooperation.  This command merely requests that the guest begin the
+#    hot removal process.  Completion of the device removal process is
+#    signaled with a DEVICE_DELETED event.  Guest reset will
+#    automatically complete removal for all devices.  If a guest-side
+#    error in the hot removal process is detected, the device will not
+#    be removed and a DEVICE_UNPLUG_GUEST_ERROR event is sent.  Some
+#    errors cannot be detected.
 #
 # Since: 0.14
 #
diff --git a/qapi/qom.json b/qapi/qom.json
index 8f0601859b1..e927f4a3c5d 100644
--- a/qapi/qom.json
+++ b/qapi/qom.json
@@ -195,12 +195,12 @@
 #
 # @typename: the type name of an object
 #
-# Note: objects can create properties at runtime, for example to
-#     describe links between different devices and/or objects.  These
-#     properties are not included in the output of this command.
-#
 # Returns: a list of ObjectPropertyInfo describing object properties
 #
+# .. note:: Objects can create properties at runtime, for example to
+#    describe links between different devices and/or objects.  These
+#    properties are not included in the output of this command.
+#
 # Since: 2.12
 ##
 { 'command': 'qom-list-properties',
@@ -608,12 +608,11 @@
 #     older to allow migration with newer QEMU versions.
 #     (default: false generally, but true for machine types <= 4.0)
 #
-# Note: prealloc=true and reserve=false cannot be set at the same
-#     time.  With reserve=true, the behavior depends on the operating
-#     system: for example, Linux will not reserve swap space for
-#     shared file mappings -- "not applicable". In contrast,
-#     reserve=false will bail out if it cannot be configured
-#     accordingly.
+# .. note:: prealloc=true and reserve=false cannot be set at the same
+#    time.  With reserve=true, the behavior depends on the operating
+#    system: for example, Linux will not reserve swap space for shared
+#    file mappings -- "not applicable". In contrast, reserve=false will
+#    bail out if it cannot be configured accordingly.
 #
 # Since: 2.1
 ##
diff --git a/qapi/rocker.json b/qapi/rocker.json
index f5225eb62cc..9f95e638309 100644
--- a/qapi/rocker.json
+++ b/qapi/rocker.json
@@ -138,8 +138,8 @@
 #
 # @ip-dst: IP header destination address
 #
-# Note: optional members may or may not appear in the flow key
-#     depending if they're relevant to the flow key.
+# .. note:: Optional members may or may not appear in the flow key
+#    depending if they're relevant to the flow key.
 #
 # Since: 2.4
 ##
@@ -168,8 +168,8 @@
 #
 # @ip-tos: IP header TOS field
 #
-# Note: optional members may or may not appear in the flow mask
-#     depending if they're relevant to the flow mask.
+# .. note:: Optional members may or may not appear in the flow mask
+#    depending if they're relevant to the flow mask.
 #
 # Since: 2.4
 ##
@@ -195,8 +195,8 @@
 #
 # @out-pport: physical output port
 #
-# Note: optional members may or may not appear in the flow action
-#     depending if they're relevant to the flow action.
+# .. note:: Optional members may or may not appear in the flow action
+#    depending if they're relevant to the flow action.
 #
 # Since: 2.4
 ##
@@ -288,8 +288,8 @@
 #
 # @ttl-check: perform TTL check
 #
-# Note: optional members may or may not appear in the group depending
-#     if they're relevant to the group type.
+# .. note:: Optional members may or may not appear in the group depending
+#    if they're relevant to the group type.
 #
 # Since: 2.4
 ##
diff --git a/qapi/run-state.json b/qapi/run-state.json
index f8773f23b29..252d7d6afa7 100644
--- a/qapi/run-state.json
+++ b/qapi/run-state.json
@@ -146,9 +146,9 @@
 # @reason: The @ShutdownCause which resulted in the SHUTDOWN.
 #     (since 4.0)
 #
-# Note: If the command-line option "-no-shutdown" has been specified,
-#     qemu will not exit, and a STOP event will eventually follow the
-#     SHUTDOWN event
+# .. note:: If the command-line option ``-no-shutdown`` has been
+#    specified, qemu will not exit, and a STOP event will eventually
+#    follow the SHUTDOWN event.
 #
 # Since: 0.12
 #
@@ -247,8 +247,8 @@
 # saved on disk, for example, S4 state, which is sometimes called
 # hibernate state
 #
-# Note: QEMU shuts down (similar to event @SHUTDOWN) when entering
-#     this state
+# .. note:: QEMU shuts down (similar to event @SHUTDOWN) when entering
+#    this state.
 #
 # Since: 1.2
 #
@@ -281,11 +281,11 @@
 #
 # @action: action that has been taken
 #
-# Note: If action is "reset", "shutdown", or "pause" the WATCHDOG
-#     event is followed respectively by the RESET, SHUTDOWN, or STOP
-#     events
+# .. note:: If action is "reset", "shutdown", or "pause" the WATCHDOG
+#    event is followed respectively by the RESET, SHUTDOWN, or STOP
+#    events.
 #
-# Note: This event is rate-limited.
+# .. note:: This event is rate-limited.
 #
 # Since: 0.13
 #
diff --git a/qapi/sockets.json b/qapi/sockets.json
index aa97c897687..f46113ab1b8 100644
--- a/qapi/sockets.json
+++ b/qapi/sockets.json
@@ -104,8 +104,8 @@
 #
 # @port: port
 #
-# Note: string types are used to allow for possible future hostname or
-#     service resolution support.
+# .. note:: string types are used to allow for possible future hostname
+#    or service resolution support.
 #
 # Since: 2.8
 ##
@@ -179,9 +179,9 @@
 #
 # @type: Transport type
 #
-# Note: This type is deprecated in favor of SocketAddress.  The
-#     difference between SocketAddressLegacy and SocketAddress is that
-#     the latter has fewer {} on the wire.
+# .. note:: This type is deprecated in favor of SocketAddress.  The
+#    difference between SocketAddressLegacy and SocketAddress is that
+#    the latter has fewer ``{}`` on the wire.
 #
 # Since: 1.3
 ##
diff --git a/qapi/stats.json b/qapi/stats.json
index c4a9f3ff70e..683929b2322 100644
--- a/qapi/stats.json
+++ b/qapi/stats.json
@@ -254,17 +254,17 @@
 #
 # @provider: a provider to restrict the query to.
 #
-# Note: runtime-collected statistics and their names fall outside
-#     QEMU's usual deprecation policies.  QEMU will try to keep the
-#     set of available data stable, together with their names, but
-#     will not guarantee stability at all costs; the same is true of
-#     providers that source statistics externally, e.g. from Linux.
-#     For example, if the same value is being tracked with different
-#     names on different architectures or by different providers, one
-#     of them might be renamed.  A statistic might go away if an
-#     algorithm is changed or some code is removed; changing a default
-#     might cause previously useful statistics to always report 0.
-#     Such changes, however, are expected to be rare.
+# .. note:: runtime-collected statistics and their names fall outside
+#    QEMU's usual deprecation policies.  QEMU will try to keep the set
+#    of available data stable, together with their names, but will not
+#    guarantee stability at all costs; the same is true of providers
+#    that source statistics externally, e.g. from Linux.  For example,
+#    if the same value is being tracked with different names on
+#    different architectures or by different providers, one of them
+#    might be renamed.  A statistic might go away if an algorithm is
+#    changed or some code is removed; changing a default might cause
+#    previously useful statistics to always report 0.  Such changes,
+#    however, are expected to be rare.
 #
 # Since: 7.1
 ##
diff --git a/qapi/transaction.json b/qapi/transaction.json
index 07afc269d54..bcb05fdedd6 100644
--- a/qapi/transaction.json
+++ b/qapi/transaction.json
@@ -237,10 +237,10 @@
 # Errors:
 #     - Any errors from commands in the transaction
 #
-# Note: The transaction aborts on the first failure.  Therefore, there
-#     will be information on only one failed operation returned in an
-#     error condition, and subsequent actions will not have been
-#     attempted.
+# .. note:: The transaction aborts on the first failure.  Therefore,
+#    there will be information on only one failed operation returned in
+#    an error condition, and subsequent actions will not have been
+#    attempted.
 #
 # Since: 1.1
 #
diff --git a/qapi/ui.json b/qapi/ui.json
index 2d0aa407aca..ec72998e28e 100644
--- a/qapi/ui.json
+++ b/qapi/ui.json
@@ -103,11 +103,10 @@
 #     - '+INT' where INT is the number of seconds from now (integer)
 #     - 'INT' where INT is the absolute time in seconds
 #
-# Notes: Time is relative to the server and currently there is no way
-#     to coordinate server time with client time.  It is not
-#     recommended to use the absolute time version of the @time
-#     parameter unless you're sure you are on the same machine as the
-#     QEMU instance.
+# .. note:: Time is relative to the server and currently there is no way
+#    to coordinate server time with client time.  It is not recommended
+#    to use the absolute time version of the @time parameter unless
+#    you're sure you are on the same machine as the QEMU instance.
 #
 # Since: 7.0
 ##
@@ -268,7 +267,7 @@
 # @unknown: No information is available about mouse mode used by the
 #     spice server.
 #
-# Note: spice/enums.h has a SpiceMouseMode already, hence the name.
+# .. note:: spice/enums.h has a SpiceMouseMode already, hence the name.
 #
 # Since: 1.1
 ##
@@ -697,9 +696,9 @@
 #
 # Since: 1.1
 #
-# Notes: An empty password in this command will set the password to
-#     the empty string.  Existing clients are unaffected by executing
-#     this command.
+# .. note:: An empty password in this command will set the password to
+#    the empty string.  Existing clients are unaffected by executing
+#    this command.
 ##
 { 'command': 'change-vnc-password',
   'data': { 'password': 'str' },
@@ -714,8 +713,8 @@
 #
 # @client: client information
 #
-# Note: This event is emitted before any authentication takes place,
-#     thus the authentication ID is not provided
+# .. note:: This event is emitted before any authentication takes place,
+#    thus the authentication ID is not provided.
 #
 # Since: 0.13
 #
@@ -1260,10 +1259,10 @@
 #
 # Since: 2.6
 #
-# Note: The consoles are visible in the qom tree, under
-#     /backend/console[$index]. They have a device link and head
-#     property, so it is possible to map which console belongs to
-#     which device and display.
+# .. note:: The consoles are visible in the qom tree, under
+#    ``/backend/console[$index]``. They have a device link and head
+#    property, so it is possible to map which console belongs to which
+#    device and display.
 #
 # Examples:
 #
diff --git a/qapi/virtio.json b/qapi/virtio.json
index 74fc27c7029..b91f3cdd0df 100644
--- a/qapi/virtio.json
+++ b/qapi/virtio.json
@@ -559,12 +559,12 @@
 #
 # Returns: VirtQueueStatus of the VirtQueue
 #
-# Notes: last_avail_idx will not be displayed in the case where the
-#     selected VirtIODevice has a running vhost device and the
-#     VirtIODevice VirtQueue index (queue) does not exist for the
-#     corresponding vhost device vhost_virtqueue.  Also,
-#     shadow_avail_idx will not be displayed in the case where the
-#     selected VirtIODevice has a running vhost device.
+# .. note:: last_avail_idx will not be displayed in the case where the
+#    selected VirtIODevice has a running vhost device and the
+#    VirtIODevice VirtQueue index (queue) does not exist for the
+#    corresponding vhost device vhost_virtqueue.  Also, shadow_avail_idx
+#    will not be displayed in the case where the selected VirtIODevice
+#    has a running vhost device.
 #
 # Since: 7.2
 #
diff --git a/qga/qapi-schema.json b/qga/qapi-schema.json
index b3de1fb6b3a..1273d85bb5f 100644
--- a/qga/qapi-schema.json
+++ b/qga/qapi-schema.json
@@ -422,8 +422,9 @@
 # Returns: GuestFsfreezeStatus ("thawed", "frozen", etc., as defined
 #     below)
 #
-# Note: This may fail to properly report the current state as a result
-#     of some other guest processes having issued an fs freeze/thaw.
+# .. note:: This may fail to properly report the current state as a
+#    result of some other guest processes having issued an fs
+#    freeze/thaw.
 #
 # Since: 0.15.0
 ##
@@ -443,9 +444,9 @@
 #
 # Returns: Number of file systems currently frozen.
 #
-# Note: On Windows, the command is implemented with the help of a
-#     Volume Shadow-copy Service DLL helper.  The frozen state is
-#     limited for up to 10 seconds by VSS.
+# .. note:: On Windows, the command is implemented with the help of a
+#    Volume Shadow-copy Service DLL helper.  The frozen state is limited
+#    for up to 10 seconds by VSS.
 #
 # Since: 0.15.0
 ##
@@ -479,10 +480,10 @@
 #
 # Returns: Number of file systems thawed by this call
 #
-# Note: if return value does not match the previous call to
-#     guest-fsfreeze-freeze, this likely means some freezable
-#     filesystems were unfrozen before this call, and that the
-#     filesystem state may have changed before issuing this command.
+# .. note:: If the return value does not match the previous call to
+#    guest-fsfreeze-freeze, this likely means some freezable filesystems
+#    were unfrozen before this call, and that the filesystem state may
+#    have changed before issuing this command.
 #
 # Since: 0.15.0
 ##
@@ -560,8 +561,8 @@
 # Errors:
 #     - If suspend to disk is not supported, Unsupported
 #
-# Notes: It's strongly recommended to issue the guest-sync command
-#     before sending commands when the guest resumes
+# .. note:: It's strongly recommended to issue the guest-sync command
+#    before sending commands when the guest resumes.
 #
 # Since: 1.1
 ##
@@ -596,8 +597,8 @@
 # Errors:
 #     - If suspend to ram is not supported, Unsupported
 #
-# Notes: It's strongly recommended to issue the guest-sync command
-#     before sending commands when the guest resumes
+# .. note:: It's strongly recommended to issue the guest-sync command
+#    before sending commands when the guest resumes.
 #
 # Since: 1.1
 ##
@@ -631,8 +632,8 @@
 # Errors:
 #     - If hybrid suspend is not supported, Unsupported
 #
-# Notes: It's strongly recommended to issue the guest-sync command
-#     before sending commands when the guest resumes
+# .. note:: It's strongly recommended to issue the guest-sync command
+#    before sending commands when the guest resumes.
 #
 # Since: 1.1
 ##
@@ -1461,16 +1462,15 @@
 #     * POSIX: as defined by os-release(5)
 #     * Windows: contains string "server" or "client"
 #
-# Notes: On POSIX systems the fields @id, @name, @pretty-name,
-#     @version, @version-id, @variant and @variant-id follow the
-#     definition specified in os-release(5). Refer to the manual page
-#     for exact description of the fields.  Their values are taken
-#     from the os-release file.  If the file is not present in the
-#     system, or the values are not present in the file, the fields
-#     are not included.
+# .. note:: On POSIX systems the fields @id, @name, @pretty-name,
+#    @version, @version-id, @variant and @variant-id follow the
+#    definition specified in os-release(5). Refer to the manual page for
+#    exact description of the fields.  Their values are taken from the
+#    os-release file.  If the file is not present in the system, or the
+#    values are not present in the file, the fields are not included.
 #
-#     On Windows the values are filled from information gathered from
-#     the system.
+#    On Windows the values are filled from information gathered from
+#    the system.
 #
 # Since: 2.10
 ##
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 3cd8e7ee295..8b1da96124e 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -557,6 +557,15 @@ def get_doc(self) -> 'QAPIDoc':
                         r'(Returns|Errors|Since|Notes?|Examples?|TODO): *',
                         line):
                     # tagged section
+
+                    if 'Note' in match.group(1):
+                        emsg = (
+                            f"The '{match.group(1)}' section is deprecated."
+                            " Please use rST's '.. note::' directive or "
+                            "another suitable admonition instead."
+                        )
+                        raise QAPIParseError(self, emsg)
+
                     doc.new_tagged_section(self.info, match.group(1))
                     text = line[match.end():]
                     if text:
diff --git a/tests/qapi-schema/doc-empty-section.err b/tests/qapi-schema/doc-empty-section.err
index 5f03a6d733f..711a0d629c2 100644
--- a/tests/qapi-schema/doc-empty-section.err
+++ b/tests/qapi-schema/doc-empty-section.err
@@ -1 +1 @@
-doc-empty-section.json:6: text required after 'Note:'
+doc-empty-section.json:6: text required after 'Errors:'
diff --git a/tests/qapi-schema/doc-empty-section.json b/tests/qapi-schema/doc-empty-section.json
index f3384e9a3bb..f179d3eff6d 100644
--- a/tests/qapi-schema/doc-empty-section.json
+++ b/tests/qapi-schema/doc-empty-section.json
@@ -3,6 +3,6 @@
 ##
 # @foo:
 #
-# Note:
+# Errors:
 ##
 { 'command': 'foo', 'data': {'a': 'int'} }
diff --git a/tests/qapi-schema/doc-good.json b/tests/qapi-schema/doc-good.json
index de38a386e8f..0a294eb324e 100644
--- a/tests/qapi-schema/doc-good.json
+++ b/tests/qapi-schema/doc-good.json
@@ -29,7 +29,7 @@
 # - Second list
 #   Note: still in list
 #
-# Note: not in list
+# .. note:: not in list
 #
 # 1. Third list
 #    is numbered
@@ -155,7 +155,7 @@
 # @cmd-feat1: a feature
 # @cmd-feat2: another feature
 #
-# Note: @arg3 is undocumented
+# .. note:: @arg3 is undocumented
 #
 # Returns: @Object
 #
@@ -163,7 +163,7 @@
 #
 # TODO: frobnicate
 #
-# Notes:
+# .. admonition:: Notes
 #
 #  - Lorem ipsum dolor sit amet
 #  - Ut enim ad minim veniam
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 435f6e6d768..2c9b4e419cb 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -76,7 +76,7 @@ Not in list
 - Second list
   Note: still in list
 
-Note: not in list
+.. note:: not in list
 
 1. Third list
    is numbered
@@ -169,15 +169,17 @@ description starts on the same line
 a feature
     feature=cmd-feat2
 another feature
-    section=Note
-@arg3 is undocumented
+    section=None
+.. note:: @arg3 is undocumented
     section=Returns
 @Object
     section=Errors
 some
     section=TODO
 frobnicate
-    section=Notes
+    section=None
+.. admonition:: Notes
+
  - Lorem ipsum dolor sit amet
  - Ut enim ad minim veniam
 
diff --git a/tests/qapi-schema/doc-good.txt b/tests/qapi-schema/doc-good.txt
index 847db70412d..b89f35d5476 100644
--- a/tests/qapi-schema/doc-good.txt
+++ b/tests/qapi-schema/doc-good.txt
@@ -17,7 +17,9 @@ Not in list
 
 * Second list Note: still in list
 
-Note: not in list
+Note:
+
+  not in list
 
 1. Third list is numbered
 
@@ -193,11 +195,9 @@ Features
 "cmd-feat2"
    another feature
 
+Note:
 
-Note
-~~~~
-
-"arg3" is undocumented
+  "arg3" is undocumented
 
 
 Returns
@@ -211,9 +211,7 @@ Errors
 
 some
 
-
-Notes
-~~~~~
+Notes:
 
 * Lorem ipsum dolor sit amet
 
diff --git a/tests/qapi-schema/doc-interleaved-section.json b/tests/qapi-schema/doc-interleaved-section.json
index adb29e98daa..b26bc0bbb79 100644
--- a/tests/qapi-schema/doc-interleaved-section.json
+++ b/tests/qapi-schema/doc-interleaved-section.json
@@ -10,7 +10,7 @@
 #
 #           bao
 #
-# Note: a section.
+# Returns: a section.
 #
 # @foobar: catch this
 #
-- 
2.44.0

[PATCH 20/20] qapi: convert "Example" sections to rST

[John Snow]

Eliminate the "Example" sections in QAPI doc blocks, converting them
into QMP example code blocks. This is generally done by converting
"Example:" or "Examples:" lines into ".. code-block:: QMP" lines.

This patch does also allow for the use of the rST syntax "Example::" by
exempting double-colon syntax from the QAPI doc parser, but that form is
not used by this conversion patch. The phrase "Example" here is not
special, it is the double-colon syntax that transforms the following
block into a code-block. By default, *this* form does not apply QMP
highlighting.

This patch has several benefits:

1. Example sections can now be written more arbitrarily, mixing
   explanatory paragraphs and code blocks however desired.

2. Example sections can now use fully arbitrary rST.

3. All code blocks are now lexed and validated as QMP; increasing
   usability of the docs and ensuring validity of example snippets.

4. Each code-block can be captioned independently without bypassing the
   QMP lexer/validator.

For any sections with more than one example, examples are split up into
multiple code-block regions. If annotations are present, those
annotations are converted into code-block captions instead, e.g.

```
Examples:

   1. Lorem Ipsum

   -> { "foo": "bar" }
```

Is rewritten as:

```
.. code-block:: QMP
   :caption: Example: Lorem Ipsum

   -> { "foo": "bar" }
```

This process was only semi-automated:

1. Replace "Examples?:" sections with sed:

sed -i 's|# Example:|# .. code-block:: QMP|' *.json
sed -i 's|# Examples:|# .. code-block:: QMP|' *.json

2. Identify sections that no longer parse successfully by attempting the
   doc build, convert annotations into captions manually.
   (Tedious, oh well.)

3. Add captions where still needed:

sed -zi 's|# .. code-block:: QMP\n#\n|# .. code-block:: QMP\n#    :caption: Example\n#\n|g' *.json

Not fully ideal, but hopefully not something that has to be done very
often. (Or ever again.)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/acpi.json                  |   6 +-
 qapi/block-core.json            | 120 ++++++++++++++++----------
 qapi/block.json                 |  60 +++++++------
 qapi/char.json                  |  36 ++++++--
 qapi/control.json               |  16 ++--
 qapi/dump.json                  |  12 ++-
 qapi/machine-target.json        |   3 +-
 qapi/machine.json               |  79 ++++++++++-------
 qapi/migration.json             | 145 +++++++++++++++++++++++---------
 qapi/misc-target.json           |  33 +++++---
 qapi/misc.json                  |  48 +++++++----
 qapi/net.json                   |  30 +++++--
 qapi/pci.json                   |   6 +-
 qapi/qapi-schema.json           |   6 +-
 qapi/qdev.json                  |  15 +++-
 qapi/qom.json                   |  20 +++--
 qapi/replay.json                |  12 ++-
 qapi/rocker.json                |  12 ++-
 qapi/run-state.json             |  45 ++++++----
 qapi/tpm.json                   |   9 +-
 qapi/trace.json                 |   6 +-
 qapi/transaction.json           |   3 +-
 qapi/ui.json                    |  62 +++++++++-----
 qapi/virtio.json                |  38 +++++----
 qapi/yank.json                  |   6 +-
 scripts/qapi/parser.py          |  15 +++-
 tests/qapi-schema/doc-good.json |  12 +--
 tests/qapi-schema/doc-good.out  |  17 ++--
 tests/qapi-schema/doc-good.txt  |  17 +---
 29 files changed, 574 insertions(+), 315 deletions(-)

diff --git a/qapi/acpi.json b/qapi/acpi.json
index aa4dbe57943..3da01f1b7fc 100644
--- a/qapi/acpi.json
+++ b/qapi/acpi.json
@@ -111,7 +111,8 @@
 #
 # Since: 2.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-acpi-ospm-status" }
 #     <- { "return": [ { "device": "d1", "slot": "0", "slot-type": "DIMM", "source": 1, "status": 0},
@@ -131,7 +132,8 @@
 #
 # Since: 2.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "ACPI_DEVICE_OST",
 #          "data": { "info": { "device": "d1", "slot": "0",
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 530af40404d..bb0447207df 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -763,7 +763,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-block" }
 #     <- {
@@ -1167,7 +1168,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-blockstats" }
 #     <- {
@@ -1460,7 +1462,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block_resize",
 #          "arguments": { "device": "scratch", "size": 1073741824 } }
@@ -1678,7 +1681,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-snapshot-sync",
 #          "arguments": { "device": "ide-hd0",
@@ -1711,7 +1715,8 @@
 #
 # Since: 2.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-add",
 #          "arguments": { "driver": "qcow2",
@@ -1857,7 +1862,8 @@
 #
 # Since: 1.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-commit",
 #          "arguments": { "device": "virtio0",
@@ -1895,7 +1901,8 @@
 #
 # Since: 1.6
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "drive-backup",
 #          "arguments": { "device": "drive0",
@@ -1921,7 +1928,8 @@
 #
 # Since: 2.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-backup",
 #          "arguments": { "device": "src-id",
@@ -1945,7 +1953,8 @@
 #
 # Since: 2.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-named-block-nodes" }
 #     <- { "return": [ { "ro":false,
@@ -2126,7 +2135,8 @@
 #
 # Since: 1.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "drive-mirror",
 #          "arguments": { "device": "ide-hd0",
@@ -2302,7 +2312,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-dirty-bitmap-add",
 #          "arguments": { "node": "drive0", "name": "bitmap0" } }
@@ -2326,7 +2337,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-dirty-bitmap-remove",
 #          "arguments": { "node": "drive0", "name": "bitmap0" } }
@@ -2349,7 +2361,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-dirty-bitmap-clear",
 #          "arguments": { "node": "drive0", "name": "bitmap0" } }
@@ -2370,7 +2383,8 @@
 #
 # Since: 4.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-dirty-bitmap-enable",
 #          "arguments": { "node": "drive0", "name": "bitmap0" } }
@@ -2391,7 +2405,8 @@
 #
 # Since: 4.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-dirty-bitmap-disable",
 #          "arguments": { "node": "drive0", "name": "bitmap0" } }
@@ -2423,7 +2438,8 @@
 #
 # Since: 4.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-dirty-bitmap-merge",
 #          "arguments": { "node": "drive0", "target": "bitmap0",
@@ -2532,7 +2548,8 @@
 #
 # Since: 2.6
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-mirror",
 #          "arguments": { "device": "ide-hd0",
@@ -2856,7 +2873,8 @@
 #
 # Since: 1.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-stream",
 #          "arguments": { "device": "virtio0",
@@ -4790,7 +4808,8 @@
 #
 # Since: 2.9
 #
-# Examples:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-add",
 #          "arguments": {
@@ -4804,6 +4823,9 @@
 #         }
 #     <- { "return": {} }
 #
+# .. code-block:: QMP
+#    :caption: Example
+#
 #     -> { "execute": "blockdev-add",
 #          "arguments": {
 #               "driver": "qcow2",
@@ -4888,7 +4910,8 @@
 #
 # Since: 2.9
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-add",
 #          "arguments": {
@@ -5537,7 +5560,8 @@
 # .. note:: If action is "stop", a STOP event will eventually follow the
 #    BLOCK_IO_ERROR event.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "BLOCK_IMAGE_CORRUPTED",
 #          "data": { "device": "", "node-name": "drive", "fatal": false,
@@ -5586,7 +5610,8 @@
 #
 # Since: 0.13
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "BLOCK_IO_ERROR",
 #          "data": { "device": "ide0-hd1",
@@ -5626,7 +5651,8 @@
 #
 # Since: 1.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "BLOCK_JOB_COMPLETED",
 #          "data": { "type": "stream", "device": "virtio-disk0",
@@ -5661,7 +5687,8 @@
 #
 # Since: 1.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "BLOCK_JOB_CANCELLED",
 #          "data": { "type": "stream", "device": "virtio-disk0",
@@ -5690,7 +5717,8 @@
 #
 # Since: 1.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "BLOCK_JOB_ERROR",
 #          "data": { "device": "ide0-hd1",
@@ -5725,7 +5753,8 @@
 #
 # Since: 1.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "BLOCK_JOB_READY",
 #          "data": { "device": "drive0", "type": "mirror", "speed": 0,
@@ -5753,7 +5782,8 @@
 #
 # Since: 2.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "BLOCK_JOB_PENDING",
 #          "data": { "type": "mirror", "id": "backup_1" },
@@ -5827,7 +5857,8 @@
 #
 # Since: 2.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block-set-write-threshold",
 #          "arguments": { "node-name": "mydev",
@@ -5872,9 +5903,8 @@
 #
 # Since: 2.7
 #
-# Examples:
-#
-#     1. Add a new node to a quorum
+# .. code-block:: QMP
+#    :caption: Example: Add a new node to a quorum
 #
 #     -> { "execute": "blockdev-add",
 #          "arguments": {
@@ -5888,7 +5918,8 @@
 #                         "node": "new_node" } }
 #     <- { "return": {} }
 #
-#     2. Delete a quorum's node
+# .. code-block:: QMP
+#    :caption: Example: Delete a quorum's node
 #
 #     -> { "execute": "x-blockdev-change",
 #          "arguments": { "parent": "disk1",
@@ -5924,16 +5955,16 @@
 #
 # Since: 2.12
 #
-# Examples:
-#
-#     1. Move a node into an IOThread
+# .. code-block:: QMP
+#    :caption: Example: Move a node into an IOThread
 #
 #     -> { "execute": "x-blockdev-set-iothread",
 #          "arguments": { "node-name": "disk1",
 #                         "iothread": "iothread0" } }
 #     <- { "return": {} }
 #
-#     2. Move a node into the main loop
+# .. code-block:: QMP
+#    :caption: Example: Move a node into the main loop
 #
 #     -> { "execute": "x-blockdev-set-iothread",
 #          "arguments": { "node-name": "disk1",
@@ -5978,7 +6009,8 @@
 #
 # Since: 2.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "QUORUM_FAILURE",
 #          "data": { "reference": "usr1", "sector-num": 345435, "sectors-count": 5 },
@@ -6009,16 +6041,16 @@
 #
 # Since: 2.0
 #
-# Examples:
-#
-#     1. Read operation
+# .. code-block:: QMP
+#    :caption: Example: Read operation
 #
 #     <- { "event": "QUORUM_REPORT_BAD",
 #          "data": { "node-name": "node0", "sector-num": 345435, "sectors-count": 5,
 #                    "type": "read" },
 #          "timestamp": { "seconds": 1344522075, "microseconds": 745528 } }
 #
-#     2. Flush operation
+# .. code-block:: QMP
+#    :caption: Example: Flush operation
 #
 #     <- { "event": "QUORUM_REPORT_BAD",
 #          "data": { "node-name": "node0", "sector-num": 0, "sectors-count": 2097120,
@@ -6066,7 +6098,8 @@
 #
 # Since: 1.7
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-snapshot-internal-sync",
 #          "arguments": { "device": "ide-hd0",
@@ -6105,7 +6138,8 @@
 #
 # Since: 1.7
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-snapshot-delete-internal-sync",
 #          "arguments": { "device": "ide-hd0",
diff --git a/qapi/block.json b/qapi/block.json
index ea81d9e1921..c67bda708a2 100644
--- a/qapi/block.json
+++ b/qapi/block.json
@@ -117,7 +117,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "eject", "arguments": { "id": "ide1-0-1" } }
 #     <- { "return": {} }
@@ -161,7 +162,8 @@
 #
 # Since: 2.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-open-tray",
 #          "arguments": { "id": "ide0-1-0" } }
@@ -199,7 +201,8 @@
 #
 # Since: 2.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-close-tray",
 #          "arguments": { "id": "ide0-1-0" } }
@@ -231,7 +234,8 @@
 #
 # Since: 2.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-remove-medium",
 #          "arguments": { "id": "ide0-1-0" } }
@@ -272,7 +276,8 @@
 #
 # Since: 2.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "blockdev-add",
 #          "arguments": {
@@ -342,9 +347,8 @@
 #
 # Since: 2.5
 #
-# Examples:
-#
-#     1. Change a removable medium
+# .. code-block:: QMP
+#    :caption: Example: Change a removable medium
 #
 #     -> { "execute": "blockdev-change-medium",
 #          "arguments": { "id": "ide0-1-0",
@@ -352,7 +356,8 @@
 #                         "format": "raw" } }
 #     <- { "return": {} }
 #
-#     2. Load a read-only medium into a writable drive
+# .. code-block:: QMP
+#    :caption: Example: Load a read-only medium into a writable drive
 #
 #     -> { "execute": "blockdev-change-medium",
 #          "arguments": { "id": "floppyA",
@@ -397,7 +402,8 @@
 #
 # Since: 1.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "DEVICE_TRAY_MOVED",
 #          "data": { "device": "ide1-cd0",
@@ -421,7 +427,8 @@
 #
 # Since: 3.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "PR_MANAGER_STATUS_CHANGED",
 #          "data": { "id": "pr-helper0",
@@ -463,7 +470,8 @@
 #
 # Since: 1.1
 #
-# Examples:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "block_set_io_throttle",
 #          "arguments": { "id": "virtio-blk-pci0/virtio-backend",
@@ -483,6 +491,9 @@
 #                         "iops_size": 0 } }
 #     <- { "return": {} }
 #
+# .. code-block:: QMP
+#    :caption: Example
+#
 #     -> { "execute": "block_set_io_throttle",
 #          "arguments": { "id": "ide0-1-0",
 #                         "bps": 1000000,
@@ -543,29 +554,27 @@
 #
 # Since: 4.0
 #
-# Example:
-#
-#     Set new histograms for all io types with intervals
-#     [0, 10), [10, 50), [50, 100), [100, +inf):
+# .. code-block:: QMP
+#    :caption: Example:
+#      Set new histograms for all io types with intervals
+#      [0, 10), [10, 50), [50, 100), [100, +inf):
 #
 #     -> { "execute": "block-latency-histogram-set",
 #          "arguments": { "id": "drive0",
 #                         "boundaries": [10, 50, 100] } }
 #     <- { "return": {} }
 #
-# Example:
-#
-#     Set new histogram only for write, other histograms will remain
-#     not changed (or not created):
+# .. code-block:: QMP
+#    :caption: Example: Set new histogram only for write, other
+#       histograms will remain not changed (or not created):
 #
 #     -> { "execute": "block-latency-histogram-set",
 #          "arguments": { "id": "drive0",
 #                         "boundaries-write": [10, 50, 100] } }
 #     <- { "return": {} }
 #
-# Example:
-#
-#     Set new histograms with the following intervals:
+# .. code-block:: QMP
+#    :caption: Example: Set new histograms with the following intervals:
 #       read, flush: [0, 10), [10, 50), [50, 100), [100, +inf)
 #       write: [0, 1000), [1000, 5000), [5000, +inf)
 #
@@ -575,9 +584,8 @@
 #                         "boundaries-write": [1000, 5000] } }
 #     <- { "return": {} }
 #
-# Example:
-#
-#     Remove all latency histograms:
+# .. code-block:: QMP
+#    :caption: Example: Remove all latency histograms:
 #
 #     -> { "execute": "block-latency-histogram-set",
 #          "arguments": { "id": "drive0" } }
diff --git a/qapi/char.json b/qapi/char.json
index 0f39c2d5cdf..a119bbb70b5 100644
--- a/qapi/char.json
+++ b/qapi/char.json
@@ -40,7 +40,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-chardev" }
 #     <- {
@@ -86,7 +87,8 @@
 #
 # Since: 2.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-chardev-backends" }
 #     <- {
@@ -141,7 +143,8 @@
 #
 # Since: 1.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "ringbuf-write",
 #          "arguments": { "device": "foo",
@@ -177,7 +180,8 @@
 #
 # Since: 1.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "ringbuf-read",
 #          "arguments": { "device": "foo",
@@ -698,19 +702,26 @@
 #
 # Since: 1.4
 #
-# Examples:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute" : "chardev-add",
 #          "arguments" : { "id" : "foo",
 #                          "backend" : { "type" : "null", "data" : {} } } }
 #     <- { "return": {} }
 #
+# .. code-block:: QMP
+#    :caption: Example
+#
 #     -> { "execute" : "chardev-add",
 #          "arguments" : { "id" : "bar",
 #                          "backend" : { "type" : "file",
 #                                        "data" : { "out" : "/tmp/bar.log" } } } }
 #     <- { "return": {} }
 #
+# .. code-block:: QMP
+#    :caption: Example
+#
 #     -> { "execute" : "chardev-add",
 #          "arguments" : { "id" : "baz",
 #                          "backend" : { "type" : "pty", "data" : {} } } }
@@ -734,13 +745,17 @@
 #
 # Since: 2.10
 #
-# Examples:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute" : "chardev-change",
 #          "arguments" : { "id" : "baz",
 #                          "backend" : { "type" : "pty", "data" : {} } } }
 #     <- { "return": { "pty" : "/dev/pty/42" } }
 #
+# .. code-block:: QMP
+#    :caption: Example
+#
 #     -> {"execute" : "chardev-change",
 #         "arguments" : {
 #             "id" : "charchannel2",
@@ -771,7 +786,8 @@
 #
 # Since: 1.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "chardev-remove", "arguments": { "id" : "foo" } }
 #     <- { "return": {} }
@@ -788,7 +804,8 @@
 #
 # Since: 2.10
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "chardev-send-break", "arguments": { "id" : "foo" } }
 #     <- { "return": {} }
@@ -809,7 +826,8 @@
 #
 # Since: 2.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "VSERPORT_CHANGE",
 #          "data": { "id": "channel0", "open": true },
diff --git a/qapi/control.json b/qapi/control.json
index 2498e5dd6ba..938c94eb3ce 100644
--- a/qapi/control.json
+++ b/qapi/control.json
@@ -16,7 +16,8 @@
 #     the QMP greeting message.  If the field is not provided, it
 #     means no QMP capabilities will be enabled.  (since 2.12)
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "qmp_capabilities",
 #          "arguments": { "enable": [ "oob" ] } }
@@ -97,7 +98,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-version" }
 #     <- {
@@ -134,7 +136,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-commands" }
 #     <- {
@@ -149,8 +152,8 @@
 #          ]
 #        }
 #
-#     Note: This example has been shortened as the real response is too
-#     long.
+# This example has been shortened as the real response is too long.
+#
 ##
 { 'command': 'query-commands', 'returns': ['CommandInfo'],
   'allow-preconfig': true }
@@ -165,7 +168,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "quit" }
 #     <- { "return": {} }
diff --git a/qapi/dump.json b/qapi/dump.json
index f9aee7ea1dd..758dc90755a 100644
--- a/qapi/dump.json
+++ b/qapi/dump.json
@@ -94,7 +94,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "dump-guest-memory",
 #          "arguments": { "paging": false, "protocol": "fd:dump" } }
@@ -150,7 +151,8 @@
 #
 # Since: 2.6
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-dump" }
 #     <- { "return": { "status": "active", "completed": 1024000,
@@ -171,7 +173,8 @@
 #
 # Since: 2.6
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "DUMP_COMPLETED",
 #          "data": { "result": { "total": 1090650112, "status": "completed",
@@ -202,7 +205,8 @@
 #
 # Since: 2.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-dump-guest-memory-capability" }
 #     <- { "return": { "formats":
diff --git a/qapi/machine-target.json b/qapi/machine-target.json
index a8d9ec87f59..5d31408abfa 100644
--- a/qapi/machine-target.json
+++ b/qapi/machine-target.json
@@ -475,7 +475,8 @@
 #
 # Since: 8.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "CPU_POLARIZATION_CHANGE",
 #          "data": { "polarization": "horizontal" },
diff --git a/qapi/machine.json b/qapi/machine.json
index e9c9bef940d..42f65ce1525 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -104,7 +104,8 @@
 #
 # Since: 2.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-cpus-fast" }
 #     <- { "return": [
@@ -221,7 +222,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-machines", "arguments": { "compat-props": true } }
 #     <- { "return": [
@@ -320,7 +322,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-uuid" }
 #     <- { "return": { "UUID": "550e8400-e29b-41d4-a716-446655440000" } }
@@ -354,7 +357,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "system_reset" }
 #     <- { "return": {} }
@@ -373,7 +377,8 @@
 #    request or that it has shut down.  Many guests will respond to this
 #    command by prompting the user in some way.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "system_powerdown" }
 #     <- { "return": {} }
@@ -393,7 +398,8 @@
 # .. note:: Prior to 4.0, this command does nothing in case the guest
 #    isn't suspended.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "system_wakeup" }
 #     <- { "return": {} }
@@ -444,7 +450,8 @@
 # .. note:: Prior to 2.1, this command was only supported for x86 and
 #    s390 VMs
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "inject-nmi" }
 #     <- { "return": {} }
@@ -473,7 +480,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-kvm" }
 #     <- { "return": { "enabled": true, "present": true } }
@@ -841,7 +849,8 @@
 #
 # .. caution:: Errors were not reliably returned until 1.1.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "memsave",
 #          "arguments": { "val": 10,
@@ -867,7 +876,8 @@
 #
 # .. caution:: Errors were not reliably returned until 1.1.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "pmemsave",
 #          "arguments": { "val": 10,
@@ -928,7 +938,8 @@
 #
 # Since: 2.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-memdev" }
 #     <- { "return": [
@@ -1046,10 +1057,8 @@
 #
 # Since: 2.7
 #
-# Examples:
-#
-#     For pseries machine type started with -smp 2,cores=2,maxcpus=4
-#     -cpu POWER8:
+# .. code-block:: QMP
+#    :caption: Example: For pseries machine type started with ``-smp 2,cores=2,maxcpus=4 -cpu POWER8``:
 #
 #     -> { "execute": "query-hotpluggable-cpus" }
 #     <- {"return": [
@@ -1059,7 +1068,8 @@
 #            "vcpus-count": 1, "qom-path": "/machine/unattached/device[0]"}
 #        ]}
 #
-#     For pc machine type started with -smp 1,maxcpus=2:
+# .. code-block:: QMP
+#    :caption: Example: For pc machine type started with ``-smp 1,maxcpus=2``:
 #
 #     -> { "execute": "query-hotpluggable-cpus" }
 #     <- {"return": [
@@ -1074,8 +1084,8 @@
 #          }
 #        ]}
 #
-#     For s390x-virtio-ccw machine type started with -smp 1,maxcpus=2
-#     -cpu qemu (Since: 2.11):
+# .. code-block:: QMP
+#    :caption: Example: For s390x-virtio-ccw machine type started with ``-smp 1,maxcpus=2 -cpu qemu`` (Since: 2.11):
 #
 #     -> { "execute": "query-hotpluggable-cpus" }
 #     <- {"return": [
@@ -1129,12 +1139,14 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "balloon", "arguments": { "value": 536870912 } }
 #     <- { "return": {} }
 #
-#     With a 2.5GiB guest this command inflated the ballon to 3GiB.
+# With a 2.5GiB guest this command inflated the ballon to 3GiB.
+#
 ##
 { 'command': 'balloon', 'data': {'value': 'int'} }
 
@@ -1165,7 +1177,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-balloon" }
 #     <- { "return": {
@@ -1189,7 +1202,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "BALLOON_CHANGE",
 #          "data": { "actual": 944766976 },
@@ -1230,7 +1244,8 @@
 #
 # Since: 8.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-hv-balloon-status-report" }
 #     <- { "return": {
@@ -1251,7 +1266,8 @@
 #
 # Since: 8.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "HV_BALLOON_STATUS_REPORT",
 #          "data": { "committed": 816640000, "available": 3333054464 },
@@ -1283,7 +1299,8 @@
 # Return the amount of initially allocated and present hotpluggable
 # (if enabled) memory in bytes.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-memory-size-summary" }
 #     <- { "return": { "base-memory": 4294967296, "plugged-memory": 0 } }
@@ -1562,7 +1579,8 @@
 #
 # Since: 2.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-memory-devices" }
 #     <- { "return": [ { "data":
@@ -1596,7 +1614,8 @@
 #
 # Since: 5.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "MEMORY_DEVICE_SIZE_CHANGE",
 #          "data": { "id": "vm0", "size": 1073741824,
@@ -1622,7 +1641,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "MEM_UNPLUG_ERROR",
 #          "data": { "device": "dimm1",
@@ -1882,7 +1902,8 @@
 #
 # Since: 7.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "dumpdtb" }
 #          "arguments": { "filename": "fdt.dtb" } }
diff --git a/qapi/migration.json b/qapi/migration.json
index a7b8ff138e3..a30d519d7c1 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -283,14 +283,14 @@
 #
 # Since: 0.14
 #
-# Examples:
-#
-#     1. Before the first migration
+# .. code-block:: QMP
+#    :caption: Example: Before the first migration
 #
 #     -> { "execute": "query-migrate" }
 #     <- { "return": {} }
 #
-#     2. Migration is done and has succeeded
+# .. code-block:: QMP
+#    :caption: Example: Migration is done and has succeeded
 #
 #     -> { "execute": "query-migrate" }
 #     <- { "return": {
@@ -310,12 +310,14 @@
 #          }
 #        }
 #
-#     3. Migration is done and has failed
+# .. code-block:: QMP
+#    :caption: Example: Migration is done and has failed
 #
 #     -> { "execute": "query-migrate" }
 #     <- { "return": { "status": "failed" } }
 #
-#     4. Migration is being performed:
+# .. code-block:: QMP
+#    :caption: Example: Migration is being performed
 #
 #     -> { "execute": "query-migrate" }
 #     <- {
@@ -336,7 +338,35 @@
 #           }
 #        }
 #
-#     5. Migration is being performed and XBZRLE is active:
+# .. code-block:: QMP
+#    :caption: Example: Migration is being performed and XBZRLE is active
+#
+#     -> { "execute": "query-migrate" }
+#     <- {
+#           "return":{
+#              "status":"active",
+#              "total-time":12345,
+#              "setup-time":12345,
+#              "expected-downtime":12345,
+#              "ram":{
+#                 "total":1057024,
+#                 "remaining":1053304,
+#                 "transferred":3720,
+#                 "duplicate":123,
+#                 "normal":123,
+#                 "normal-bytes":123456,
+#                 "dirty-sync-count":15
+#              },
+#              "disk":{
+#                 "total":20971520,
+#                 "remaining":20880384,
+#                 "transferred":91136
+#              }
+#           }
+#        }
+#
+# .. code-block:: QMP
+#    :caption: Example: Migration is being performed and XBZRLE is active
 #
 #     -> { "execute": "query-migrate" }
 #     <- {
@@ -510,7 +540,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate-set-capabilities" , "arguments":
 #          { "capabilities": [ { "capability": "xbzrle", "state": true } ] } }
@@ -528,7 +559,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-migrate-capabilities" }
 #     <- { "return": [
@@ -1030,7 +1062,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate-set-parameters" ,
 #          "arguments": { "multifd-channels": 5 } }
@@ -1228,7 +1261,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-migrate-parameters" }
 #     <- { "return": {
@@ -1252,7 +1286,8 @@
 #
 # Since: 2.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate-start-postcopy" }
 #     <- { "return": {} }
@@ -1268,7 +1303,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- {"timestamp": {"seconds": 1432121972, "microseconds": 744001},
 #         "event": "MIGRATION",
@@ -1287,7 +1323,8 @@
 #
 # Since: 2.6
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "timestamp": {"seconds": 1449669631, "microseconds": 239225},
 #           "event": "MIGRATION_PASS", "data": {"pass": 2} }
@@ -1371,7 +1408,8 @@
 #
 # Since: 3.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "timestamp": {"seconds": 2032141960, "microseconds": 417172},
 #          "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } }
@@ -1414,7 +1452,8 @@
 #
 # Since: 2.8
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "x-colo-lost-heartbeat" }
 #     <- { "return": {} }
@@ -1433,7 +1472,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate_cancel" }
 #     <- { "return": {} }
@@ -1449,7 +1489,8 @@
 #
 # Since: 2.11
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate-continue" , "arguments":
 #          { "state": "pre-switchover" } }
@@ -1582,7 +1623,8 @@
 #     6. The 'uri' and 'channels' arguments are mutually exclusive;
 #        exactly one of the two should be present.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
 #     <- { "return": {} }
@@ -1661,7 +1703,8 @@
 #     5. The 'uri' and 'channels' arguments are mutually exclusive;
 #        exactly one of the two should be present.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate-incoming",
 #          "arguments": { "uri": "tcp:0:4446" } }
@@ -1712,7 +1755,8 @@
 #
 # Since: 1.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "xen-save-devices-state",
 #          "arguments": { "filename": "/tmp/save" } }
@@ -1730,7 +1774,8 @@
 #
 # Since: 1.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "xen-set-global-dirty-log",
 #          "arguments": { "enable": true } }
@@ -1750,7 +1795,8 @@
 #
 # Since: 2.7
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "xen-load-devices-state",
 #          "arguments": { "filename": "/tmp/resume" } }
@@ -1770,7 +1816,8 @@
 # @failover: true to do failover, false to stop.  Cannot be specified
 #     if 'enable' is true.  Default value is false.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "xen-set-replication",
 #          "arguments": {"enable": true, "primary": false} }
@@ -1805,7 +1852,8 @@
 #
 # Returns: A @ReplicationStatus object showing the status.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-xen-replication-status" }
 #     <- { "return": { "error": false } }
@@ -1821,7 +1869,8 @@
 #
 # Xen uses this command to notify replication to trigger a checkpoint.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "xen-colo-do-checkpoint" }
 #     <- { "return": {} }
@@ -1859,7 +1908,8 @@
 #
 # Returns: A @COLOStatus object showing the status.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-colo-status" }
 #     <- { "return": { "mode": "primary", "last-mode": "none", "reason": "request" } }
@@ -1877,7 +1927,8 @@
 #
 # @uri: the URI to be used for the recovery of migration stream.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate-recover",
 #          "arguments": { "uri": "tcp:192.168.1.200:12345" } }
@@ -1894,7 +1945,8 @@
 #
 # Pause a migration.  Currently it only supports postcopy.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "migrate-pause" }
 #     <- { "return": {} }
@@ -1915,7 +1967,8 @@
 #
 # Since: 4.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "UNPLUG_PRIMARY",
 #          "data": { "device-id": "hostdev0" },
@@ -2075,13 +2128,15 @@
 #
 # Since: 5.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example:
 #
 #     -> {"execute": "calc-dirty-rate", "arguments": {"calc-time": 1,
 #                                                     "sample-pages": 512} }
 #     <- { "return": {} }
 #
-#     Measure dirty rate using dirty bitmap for 500 milliseconds:
+# .. code-block:: QMP
+#    :caption: Example: Measure dirty rate using dirty bitmap for 500 milliseconds:
 #
 #     -> {"execute": "calc-dirty-rate", "arguments": {"calc-time": 500,
 #         "calc-time-unit": "millisecond", "mode": "dirty-bitmap"} }
@@ -2103,15 +2158,15 @@
 #
 # Since: 5.2
 #
-# Examples:
-#
-#     1. Measurement is in progress:
+# .. code-block:: QMP
+#    :caption: Example: Measurement is in progress
 #
 #     <- {"status": "measuring", "sample-pages": 512,
 #         "mode": "page-sampling", "start-time": 1693900454, "calc-time": 10,
 #         "calc-time-unit": "second"}
 #
-#     2. Measurement has been completed:
+# .. code-block:: QMP
+#    :caption: Example: Measurement has been completed
 #
 #     <- {"status": "measured", "sample-pages": 512, "dirty-rate": 108,
 #         "mode": "page-sampling", "start-time": 1693900454, "calc-time": 10,
@@ -2154,7 +2209,8 @@
 #
 # Since: 7.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> {"execute": "set-vcpu-dirty-limit"}
 #         "arguments": { "dirty-rate": 200,
@@ -2178,7 +2234,8 @@
 #
 # Since: 7.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> {"execute": "cancel-vcpu-dirty-limit"},
 #         "arguments": { "cpu-index": 1 } }
@@ -2195,7 +2252,8 @@
 #
 # Since: 7.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> {"execute": "query-vcpu-dirty-limit"}
 #     <- {"return": [
@@ -2259,7 +2317,8 @@
 #
 # If @tag already exists, an error will be reported
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "snapshot-save",
 #          "arguments": {
@@ -2329,7 +2388,8 @@
 # device nodes that can have changed since the original @snapshot-save
 # command execution.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "snapshot-load",
 #          "arguments": {
@@ -2390,7 +2450,8 @@
 # to determine completion and to fetch details of any errors that
 # arise.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "snapshot-delete",
 #          "arguments": {
diff --git a/qapi/misc-target.json b/qapi/misc-target.json
index 4e0a6492a9a..ccbed866195 100644
--- a/qapi/misc-target.json
+++ b/qapi/misc-target.json
@@ -11,7 +11,8 @@
 #
 # Since: 2.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "rtc-reset-reinjection" }
 #     <- { "return": {} }
@@ -89,7 +90,8 @@
 #
 # Since: 2.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-sev" }
 #     <- { "return": { "enabled": true, "api-major" : 0, "api-minor" : 0,
@@ -120,7 +122,8 @@
 #
 # Since: 2.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-sev-launch-measure" }
 #     <- { "return": { "data": "4l8LXeNlSPUDlXPJG5966/8%YZ" } }
@@ -165,7 +168,8 @@
 #
 # Since: 2.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-sev-capabilities" }
 #     <- { "return": { "pdh": "8CCDD8DDD", "cert-chain": "888CCCDDDEE",
@@ -219,7 +223,8 @@
 #
 # Since: 6.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute" : "query-sev-attestation-report",
 #                      "arguments": { "mnonce": "aaaaaaa" } }
@@ -239,7 +244,8 @@
 #
 # Since: 2.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "dump-skeys",
 #          "arguments": { "filename": "/tmp/skeys" } }
@@ -284,7 +290,8 @@
 #
 # Since: 2.6
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-gic-capabilities" }
 #     <- { "return": [{ "version": 2, "emulated": true, "kernel": false },
@@ -342,7 +349,8 @@
 #
 # Since: 6.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-sgx" }
 #     <- { "return": { "sgx": true, "sgx1" : true, "sgx2" : true,
@@ -361,7 +369,8 @@
 #
 # Since: 6.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-sgx-capabilities" }
 #     <- { "return": { "sgx": true, "sgx1" : true, "sgx2" : true,
@@ -436,7 +445,8 @@
 #
 # Since: 8.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "xen-event-list" }
 #     <- { "return": [
@@ -474,7 +484,8 @@
 #
 # Since: 8.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "xen-event-inject", "arguments": { "port": 1 } }
 #     <- { "return": { } }
diff --git a/qapi/misc.json b/qapi/misc.json
index b04efbadec6..39bf8e30df1 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -30,7 +30,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "add_client", "arguments": { "protocol": "vnc",
 #                                                  "fdname": "myclient" } }
@@ -60,7 +61,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-name" }
 #     <- { "return": { "name": "qemu-name" } }
@@ -111,7 +113,8 @@
 #
 # Since: 2.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-iothreads" }
 #     <- { "return": [
@@ -144,7 +147,8 @@
 #    In the "suspended" state, it will completely stop the VM and cause
 #    a transition to the "paused" state.  (Since 9.0)
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "stop" }
 #     <- { "return": {} }
@@ -168,7 +172,8 @@
 #    this command will transition back to the "suspended" state.  (Since
 #    9.0)
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "cont" }
 #     <- { "return": {} }
@@ -192,7 +197,8 @@
 #
 # Since: 3.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "x-exit-preconfig" }
 #     <- { "return": {} }
@@ -232,7 +238,8 @@
 #
 #    * Commands that prompt the user for data don't currently work.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "human-monitor-command",
 #          "arguments": { "command-line": "info kvm" } }
@@ -258,7 +265,8 @@
 #    The 'closefd' command can be used to explicitly close the file
 #    descriptor when it is no longer needed.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "getfd", "arguments": { "fdname": "fd1" } }
 #     <- { "return": {} }
@@ -285,7 +293,8 @@
 #    The 'closefd' command can be used to explicitly close the file
 #    descriptor when it is no longer needed.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "get-win32-socket",
 #          "arguments": { "info": "abcd123..", "fdname": "skclient" } }
@@ -302,7 +311,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "closefd", "arguments": { "fdname": "fd1" } }
 #     <- { "return": {} }
@@ -345,7 +355,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "add-fd", "arguments": { "fdset-id": 1 } }
 #     <- { "return": { "fdset-id": 1, "fd": 3 } }
@@ -374,7 +385,8 @@
 # .. note:: If @fd is not specified, all file descriptors in @fdset-id
 #    will be removed.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "remove-fd", "arguments": { "fdset-id": 1, "fd": 3 } }
 #     <- { "return": {} }
@@ -420,7 +432,8 @@
 #
 # .. note:: The list of fd sets is shared by all monitor connections.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-fdsets" }
 #     <- { "return": [
@@ -523,7 +536,8 @@
 #
 # Since: 1.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-command-line-options",
 #          "arguments": { "option": "option-rom" } }
@@ -565,7 +579,8 @@
 #
 # Since: 0.13
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "RTC_CHANGE",
 #          "data": { "offset": 78 },
@@ -592,7 +607,8 @@
 #
 # Since: 7.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "VFU_CLIENT_HANGUP",
 #          "data": { "vfu-id": "vfu1",
diff --git a/qapi/net.json b/qapi/net.json
index 4ac7fdc7e6c..adef3e6633d 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -26,7 +26,8 @@
 #    command will succeed even if the network adapter does not support
 #    link status notification.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "set_link",
 #          "arguments": { "name": "e1000.0", "up": false } }
@@ -46,7 +47,8 @@
 # Errors:
 #     - If @type is not a valid network backend, DeviceNotFound
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "netdev_add",
 #          "arguments": { "type": "user", "id": "netdev1",
@@ -68,7 +70,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "netdev_del", "arguments": { "id": "netdev1" } }
 #     <- { "return": {} }
@@ -835,7 +838,8 @@
 #
 # Since: 1.6
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-rx-filter", "arguments": { "name": "vnet0" } }
 #     <- { "return": [
@@ -880,7 +884,8 @@
 #
 # Since: 1.6
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "NIC_RX_FILTER_CHANGED",
 #          "data": { "name": "vnet0",
@@ -927,7 +932,8 @@
 # switches.  This can be useful when network bonds fail-over the
 # active slave.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "announce-self",
 #          "arguments": {
@@ -952,7 +958,8 @@
 #
 # Since: 4.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "FAILOVER_NEGOTIATED",
 #          "data": { "device-id": "net1" },
@@ -972,7 +979,8 @@
 #
 # Since: 7.2
 #
-# Examples:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "NETDEV_STREAM_CONNECTED",
 #          "data": { "netdev-id": "netdev0",
@@ -980,6 +988,9 @@
 #                              "host": "::1", "type": "inet" } },
 #          "timestamp": { "seconds": 1666269863, "microseconds": 311222 } }
 #
+# .. code-block:: QMP
+#    :caption: Example
+#
 #     <- { "event": "NETDEV_STREAM_CONNECTED",
 #          "data": { "netdev-id": "netdev0",
 #                    "addr": { "path": "/tmp/qemu0", "type": "unix" } },
@@ -998,7 +1009,8 @@
 #
 # Since: 7.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "NETDEV_STREAM_DISCONNECTED",
 #          "data": {"netdev-id": "netdev0"},
diff --git a/qapi/pci.json b/qapi/pci.json
index f51159a2c4c..9192212661b 100644
--- a/qapi/pci.json
+++ b/qapi/pci.json
@@ -182,7 +182,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-pci" }
 #     <- { "return": [
@@ -311,8 +312,7 @@
 #           ]
 #        }
 #
-#     Note: This example has been shortened as the real response is too
-#     long.
+# This example has been shortened as the real response is too long.
 #
 ##
 { 'command': 'query-pci', 'returns': ['PciInfo'] }
diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
index 5e33da7228f..66fbcbd3619 100644
--- a/qapi/qapi-schema.json
+++ b/qapi/qapi-schema.json
@@ -20,11 +20,7 @@
 # understand.  However, in real protocol usage, they're emitted as a
 # single line.
 #
-# Also, the following notation is used to denote data flow:
-#
-# Example:
-#
-# ::
+# Also, the following notation is used to denote data flow::
 #
 #   -> data issued by the Client
 #   <- Server data response
diff --git a/qapi/qdev.json b/qapi/qdev.json
index d031fc3590d..cfe403fea20 100644
--- a/qapi/qdev.json
+++ b/qapi/qdev.json
@@ -62,7 +62,8 @@
 #        the ``-device DEVICE,help`` command-line argument, where DEVICE
 #        is the device's name.
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "device_add",
 #          "arguments": { "driver": "e1000", "id": "net1",
@@ -104,12 +105,16 @@
 #
 # Since: 0.14
 #
-# Examples:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "device_del",
 #          "arguments": { "id": "net1" } }
 #     <- { "return": {} }
 #
+# .. code-block:: QMP
+#    :caption: Example
+#
 #     -> { "execute": "device_del",
 #          "arguments": { "id": "/machine/peripheral-anon/device[0]" } }
 #     <- { "return": {} }
@@ -130,7 +135,8 @@
 #
 # Since: 1.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "DEVICE_DELETED",
 #          "data": { "device": "virtio-net-pci-0",
@@ -152,7 +158,8 @@
 #
 # Since: 6.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "DEVICE_UNPLUG_GUEST_ERROR",
 #          "data": { "device": "core1",
diff --git a/qapi/qom.json b/qapi/qom.json
index e927f4a3c5d..80357537b65 100644
--- a/qapi/qom.json
+++ b/qapi/qom.json
@@ -59,7 +59,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "qom-list",
 #          "arguments": { "path": "/chardevs" } }
@@ -104,16 +105,16 @@
 #
 # Since: 1.2
 #
-# Examples:
-#
-#     1. Use absolute path
+# .. code-block:: QMP
+#    :caption: Example: Use absolute path
 #
 #     -> { "execute": "qom-get",
 #          "arguments": { "path": "/machine/unattached/device[0]",
 #                         "property": "hotplugged" } }
 #     <- { "return": false }
 #
-#     2. Use partial path
+# .. code-block:: QMP
+#    :caption: Example: Use partial path
 #
 #     -> { "execute": "qom-get",
 #          "arguments": { "path": "unattached/sysbus",
@@ -139,7 +140,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "qom-set",
 #          "arguments": { "path": "/machine",
@@ -1080,7 +1082,8 @@
 #
 # Since: 2.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "object-add",
 #          "arguments": { "qom-type": "rng-random", "id": "rng1",
@@ -1102,7 +1105,8 @@
 #
 # Since: 2.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "object-del", "arguments": { "id": "rng1" } }
 #     <- { "return": {} }
diff --git a/qapi/replay.json b/qapi/replay.json
index d3559f9c8f7..f0351b6d9c1 100644
--- a/qapi/replay.json
+++ b/qapi/replay.json
@@ -54,7 +54,8 @@
 #
 # Since: 5.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-replay" }
 #     <- { "return": { "mode": "play", "filename": "log.rr", "icount": 220414 } }
@@ -76,7 +77,8 @@
 #
 # Since: 5.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "replay-break", "arguments": { "icount": 220414 } }
 #     <- { "return": {} }
@@ -91,7 +93,8 @@
 #
 # Since: 5.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "replay-delete-break" }
 #     <- { "return": {} }
@@ -112,7 +115,8 @@
 #
 # Since: 5.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "replay-seek", "arguments": { "icount": 220414 } }
 #     <- { "return": {} }
diff --git a/qapi/rocker.json b/qapi/rocker.json
index 9f95e638309..da5cc2b9e01 100644
--- a/qapi/rocker.json
+++ b/qapi/rocker.json
@@ -30,7 +30,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-rocker", "arguments": { "name": "sw1" } }
 #     <- { "return": {"name": "sw1", "ports": 2, "id": 1327446905938}}
@@ -98,7 +99,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-rocker-ports", "arguments": { "name": "sw1" } }
 #     <- { "return": [ {"duplex": "full", "enabled": true, "name": "sw1.1",
@@ -240,7 +242,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-rocker-of-dpa-flows",
 #          "arguments": { "name": "sw1" } }
@@ -315,7 +318,8 @@
 #
 # Since: 2.4
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-rocker-of-dpa-groups",
 #          "arguments": { "name": "sw1" } }
diff --git a/qapi/run-state.json b/qapi/run-state.json
index 252d7d6afa7..6b0322de4d7 100644
--- a/qapi/run-state.json
+++ b/qapi/run-state.json
@@ -123,7 +123,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-status" }
 #     <- { "return": { "running": true,
@@ -152,7 +153,8 @@
 #
 # Since: 0.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "SHUTDOWN",
 #          "data": { "guest": true, "reason": "guest-shutdown" },
@@ -168,7 +170,8 @@
 #
 # Since: 0.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "POWERDOWN",
 #          "timestamp": { "seconds": 1267040730, "microseconds": 682951 } }
@@ -189,7 +192,8 @@
 #
 # Since: 0.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "RESET",
 #          "data": { "guest": false, "reason": "guest-reset" },
@@ -204,7 +208,8 @@
 #
 # Since: 0.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "STOP",
 #          "timestamp": { "seconds": 1267041730, "microseconds": 281295 } }
@@ -218,7 +223,8 @@
 #
 # Since: 0.12
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "RESUME",
 #          "timestamp": { "seconds": 1271770767, "microseconds": 582542 } }
@@ -233,7 +239,8 @@
 #
 # Since: 1.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "SUSPEND",
 #          "timestamp": { "seconds": 1344456160, "microseconds": 309119 } }
@@ -252,7 +259,8 @@
 #
 # Since: 1.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "SUSPEND_DISK",
 #          "timestamp": { "seconds": 1344456160, "microseconds": 309119 } }
@@ -267,7 +275,8 @@
 #
 # Since: 1.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "WAKEUP",
 #          "timestamp": { "seconds": 1344522075, "microseconds": 745528 } }
@@ -289,7 +298,8 @@
 #
 # Since: 0.13
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "WATCHDOG",
 #          "data": { "action": "reset" },
@@ -382,7 +392,8 @@
 #
 # Since: 2.11
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "watchdog-set-action",
 #          "arguments": { "action": "inject-nmi" } }
@@ -406,7 +417,8 @@
 #
 # Since: 6.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "set-action",
 #          "arguments": { "reboot": "shutdown",
@@ -433,7 +445,8 @@
 #
 # Since: 1.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "GUEST_PANICKED",
 #          "data": { "action": "pause" },
@@ -453,7 +466,8 @@
 #
 # Since: 5.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "GUEST_CRASHLOADED",
 #          "data": { "action": "run" },
@@ -597,7 +611,8 @@
 #
 # Since: 5.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "MEMORY_FAILURE",
 #          "data": { "recipient": "hypervisor",
diff --git a/qapi/tpm.json b/qapi/tpm.json
index 1577b5c259d..07b7b5bd9fd 100644
--- a/qapi/tpm.json
+++ b/qapi/tpm.json
@@ -31,7 +31,8 @@
 #
 # Since: 1.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-tpm-models" }
 #     <- { "return": [ "tpm-tis", "tpm-crb", "tpm-spapr" ] }
@@ -62,7 +63,8 @@
 #
 # Since: 1.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-tpm-types" }
 #     <- { "return": [ "passthrough", "emulator" ] }
@@ -168,7 +170,8 @@
 #
 # Since: 1.5
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-tpm" }
 #     <- { "return":
diff --git a/qapi/trace.json b/qapi/trace.json
index 043d12f83e0..3af64027759 100644
--- a/qapi/trace.json
+++ b/qapi/trace.json
@@ -64,7 +64,8 @@
 #
 # Since: 2.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "trace-event-get-state",
 #          "arguments": { "name": "qemu_memalign" } }
@@ -94,7 +95,8 @@
 #
 # Since: 2.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "trace-event-set-state",
 #          "arguments": { "name": "qemu_memalign", "enable": true } }
diff --git a/qapi/transaction.json b/qapi/transaction.json
index bcb05fdedd6..3f06ce9c1c6 100644
--- a/qapi/transaction.json
+++ b/qapi/transaction.json
@@ -244,7 +244,8 @@
 #
 # Since: 1.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "transaction",
 #          "arguments": { "actions": [
diff --git a/qapi/ui.json b/qapi/ui.json
index ec72998e28e..856b3779d77 100644
--- a/qapi/ui.json
+++ b/qapi/ui.json
@@ -81,7 +81,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "set_password", "arguments": { "protocol": "vnc",
 #                                                    "password": "secret" } }
@@ -140,7 +141,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "expire_password", "arguments": { "protocol": "vnc",
 #                                                       "time": "+60" } }
@@ -182,7 +184,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "screendump",
 #          "arguments": { "filename": "/tmp/image" } }
@@ -324,7 +327,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-spice" }
 #     <- { "return": {
@@ -373,7 +377,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "timestamp": {"seconds": 1290688046, "microseconds": 388707},
 #          "event": "SPICE_CONNECTED",
@@ -399,7 +404,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "timestamp": {"seconds": 1290688046, "microseconds": 417172},
 #          "event": "SPICE_INITIALIZED",
@@ -426,7 +432,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "timestamp": {"seconds": 1290688046, "microseconds": 388707},
 #          "event": "SPICE_DISCONNECTED",
@@ -447,7 +454,8 @@
 #
 # Since: 1.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "timestamp": {"seconds": 1290688046, "microseconds": 417172},
 #          "event": "SPICE_MIGRATE_COMPLETED" }
@@ -653,7 +661,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-vnc" }
 #     <- { "return": {
@@ -718,7 +727,8 @@
 #
 # Since: 0.13
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "VNC_CONNECTED",
 #          "data": {
@@ -745,7 +755,8 @@
 #
 # Since: 0.13
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <-  { "event": "VNC_INITIALIZED",
 #           "data": {
@@ -771,7 +782,8 @@
 #
 # Since: 0.13
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     <- { "event": "VNC_DISCONNECTED",
 #          "data": {
@@ -819,7 +831,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-mice" }
 #     <- { "return": [
@@ -1028,7 +1041,8 @@
 #
 # Since: 1.3
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "send-key",
 #          "arguments": { "keys": [ { "type": "qcode", "data": "ctrl" },
@@ -1264,9 +1278,8 @@
 #    property, so it is possible to map which console belongs to which
 #    device and display.
 #
-# Examples:
-#
-#     1. Press left mouse button.
+# .. code-block:: QMP
+#    :caption: Example: Press left mouse button.
 #
 #     -> { "execute": "input-send-event",
 #         "arguments": { "device": "video0",
@@ -1280,7 +1293,8 @@
 #                        "data" : { "down": false, "button": "left" } } ] } }
 #     <- { "return": {} }
 #
-#     2. Press ctrl-alt-del.
+# .. code-block:: QMP
+#    :caption: Example: Press ctrl-alt-del.
 #
 #     -> { "execute": "input-send-event",
 #          "arguments": { "events": [
@@ -1292,7 +1306,8 @@
 #               "key": {"type": "qcode", "data": "delete" } } } ] } }
 #     <- { "return": {} }
 #
-#     3. Move mouse pointer to absolute coordinates (20000, 400).
+# .. code-block:: QMP
+#    :caption: Example: Move mouse pointer to absolute coordinates (20000, 400).
 #
 #     -> { "execute": "input-send-event" ,
 #       "arguments": { "events": [
@@ -1605,7 +1620,8 @@
 #
 # Since: 6.0
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "display-reload",
 #          "arguments": { "type": "vnc", "tls-certs": true  } }
@@ -1658,7 +1674,8 @@
 #
 # Since: 7.1
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "display-update",
 #          "arguments": { "type": "vnc", "addresses":
@@ -1689,7 +1706,8 @@
 #
 # Since: 0.14
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "client_migrate_info",
 #          "arguments": { "protocol": "spice",
diff --git a/qapi/virtio.json b/qapi/virtio.json
index b91f3cdd0df..adf5def9dd6 100644
--- a/qapi/virtio.json
+++ b/qapi/virtio.json
@@ -34,7 +34,8 @@
 #
 # Since: 7.2
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "x-query-virtio" }
 #     <- { "return": [
@@ -203,9 +204,8 @@
 #
 # Since: 7.2
 #
-# Examples:
-#
-#     1. Poll for the status of virtio-crypto (no vhost-crypto active)
+# .. code-block:: QMP
+#    :caption: Example: Poll for the status of virtio-crypto (no vhost-crypto active)
 #
 #     -> { "execute": "x-query-virtio-status",
 #          "arguments": { "path": "/machine/peripheral/crypto0/virtio-backend" }
@@ -261,7 +261,8 @@
 #          }
 #        }
 #
-#     2. Poll for the status of virtio-net (vhost-net is active)
+# .. code-block:: QMP
+#    :caption: Example: Poll for the status of virtio-net (vhost-net is active)
 #
 #     -> { "execute": "x-query-virtio-status",
 #          "arguments": { "path": "/machine/peripheral-anon/device[1]/virtio-backend" }
@@ -568,9 +569,8 @@
 #
 # Since: 7.2
 #
-# Examples:
-#
-#     1. Get VirtQueueStatus for virtio-vsock (vhost-vsock running)
+# .. code-block:: QMP
+#    :caption: Example: Get VirtQueueStatus for virtio-vsock (vhost-vsock running)
 #
 #     -> { "execute": "x-query-virtio-queue-status",
 #          "arguments": { "path": "/machine/peripheral/vsock0/virtio-backend",
@@ -593,7 +593,8 @@
 #          }
 #        }
 #
-#     2. Get VirtQueueStatus for virtio-serial (no vhost)
+# .. code-block:: QMP
+#    :caption: Example: Get VirtQueueStatus for virtio-serial (no vhost)
 #
 #     -> { "execute": "x-query-virtio-queue-status",
 #          "arguments": { "path": "/machine/peripheral-anon/device[0]/virtio-backend",
@@ -690,9 +691,8 @@
 #
 # Since: 7.2
 #
-# Examples:
-#
-#     1. Get vhost_virtqueue status for vhost-crypto
+# .. code-block:: QMP
+#    :caption: Example: Get vhost_virtqueue status for vhost-crypto
 #
 #     -> { "execute": "x-query-virtio-vhost-queue-status",
 #          "arguments": { "path": "/machine/peripheral/crypto0/virtio-backend",
@@ -715,7 +715,8 @@
 #          }
 #        }
 #
-#     2. Get vhost_virtqueue status for vhost-vsock
+# .. code-block:: QMP
+#    :caption: Example: Get vhost_virtqueue status for vhost-vsock
 #
 #     -> { "execute": "x-query-virtio-vhost-queue-status",
 #          "arguments": { "path": "/machine/peripheral/vsock0/virtio-backend",
@@ -839,9 +840,8 @@
 #
 # Since: 7.2
 #
-# Examples:
-#
-#     1. Introspect on virtio-net's VirtQueue 0 at index 5
+# .. code-block:: QMP
+#    :caption: Example: Introspect on virtio-net's VirtQueue 0 at index 5
 #
 #     -> { "execute": "x-query-virtio-queue-element",
 #          "arguments": { "path": "/machine/peripheral-anon/device[1]/virtio-backend",
@@ -870,7 +870,8 @@
 #          }
 #        }
 #
-#     2. Introspect on virtio-crypto's VirtQueue 1 at head
+# .. code-block:: QMP
+#    :caption: Example: Introspect on virtio-crypto's VirtQueue 1 at head
 #
 #     -> { "execute": "x-query-virtio-queue-element",
 #          "arguments": { "path": "/machine/peripheral/crypto0/virtio-backend",
@@ -898,7 +899,8 @@
 #          }
 #        }
 #
-#     3. Introspect on virtio-scsi's VirtQueue 2 at head
+# .. code-block:: QMP
+#    :caption: Example: Introspect on virtio-scsi's VirtQueue 2 at head
 #
 #     -> { "execute": "x-query-virtio-queue-element",
 #          "arguments": { "path": "/machine/peripheral-anon/device[2]/virtio-backend",
diff --git a/qapi/yank.json b/qapi/yank.json
index 89f2f4d199b..a64b77e846c 100644
--- a/qapi/yank.json
+++ b/qapi/yank.json
@@ -81,7 +81,8 @@
 # Errors:
 #     - If any of the YankInstances doesn't exist, DeviceNotFound
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "yank",
 #          "arguments": {
@@ -104,7 +105,8 @@
 #
 # Returns: list of @YankInstance
 #
-# Example:
+# .. code-block:: QMP
+#    :caption: Example
 #
 #     -> { "execute": "query-yank" }
 #     <- { "return": [
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 8b1da96124e..afc0b444034 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -554,9 +554,12 @@ def get_doc(self) -> 'QAPIDoc':
                     no_more_args = True
                     intro = False
                 elif match := re.match(
-                        r'(Returns|Errors|Since|Notes?|Examples?|TODO): *',
+                        r'(Returns|Errors|Since|Notes?|Examples?(?!::)|TODO)'
+                        r': *',
                         line):
-                    # tagged section
+                    # tagged section.
+                    # Examples sections followed by two colons are excluded;
+                    # those are raw rST syntax!
 
                     if 'Note' in match.group(1):
                         emsg = (
@@ -566,6 +569,14 @@ def get_doc(self) -> 'QAPIDoc':
                         )
                         raise QAPIParseError(self, emsg)
 
+                    if match.group(1).startswith("Example"):
+                        emsg = (
+                            f"The '{match.group(1)}' section is deprecated. "
+                            "Please use rST's '.. code-block:: QMP' directive,"
+                            " 'Example::', or other suitable markup instead."
+                        )
+                        raise QAPIParseError(self, emsg)
+
                     doc.new_tagged_section(self.info, match.group(1))
                     text = line[match.end():]
                     if text:
diff --git a/tests/qapi-schema/doc-good.json b/tests/qapi-schema/doc-good.json
index 0a294eb324e..57e2e591938 100644
--- a/tests/qapi-schema/doc-good.json
+++ b/tests/qapi-schema/doc-good.json
@@ -46,11 +46,12 @@
 #
 # Duis aute irure dolor
 #
-# Example:
+# .. code-block:: QMP
 #
 # -> in
 # <- out
-# Examples:
+# .. code-block::
+#
 # - *verbatim*
 # - {braces}
 ##
@@ -170,12 +171,13 @@
 #
 #  Duis aute irure dolor
 #
-# Example:
+# .. code-block::
 #
 #  -> in
 #  <- out
 #
-# Examples:
+# .. code-block::
+#
 #  - *verbatim*
 #  - {braces}
 #
@@ -194,7 +196,7 @@
 # @cmd-feat1: a feature
 # @cmd-feat2: another feature
 #
-# Example:
+# .. code-block::
 #
 #  -> in
 #
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 2c9b4e419cb..cd967c8b2e0 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -93,11 +93,12 @@ Notes:
 
 Duis aute irure dolor
 
-Example:
+.. code-block:: QMP
 
 -> in
 <- out
-Examples:
+.. code-block::
+
 - *verbatim*
 - {braces}
 doc symbol=Enum
@@ -184,10 +185,14 @@ frobnicate
  - Ut enim ad minim veniam
 
  Duis aute irure dolor
-    section=Example
+
+.. code-block::
+
  -> in
  <- out
-    section=Examples
+
+.. code-block::
+
  - *verbatim*
  - {braces}
     section=Since
@@ -199,7 +204,9 @@ If you're bored enough to read this, go see a video of boxed cats
 a feature
     feature=cmd-feat2
 another feature
-    section=Example
+    section=None
+.. code-block::
+
  -> in
 
  <- out
diff --git a/tests/qapi-schema/doc-good.txt b/tests/qapi-schema/doc-good.txt
index b89f35d5476..808ca7f73a1 100644
--- a/tests/qapi-schema/doc-good.txt
+++ b/tests/qapi-schema/doc-good.txt
@@ -33,9 +33,10 @@ Returns: the King Since: the first age Notes:
 
 Duis aute irure dolor
 
-Example:
+-> in <- out .. code-block:
 
--> in <- out Examples: - *verbatim* - {braces}
+   - *verbatim*
+   - {braces}
 
 
 "Enum" (Enum)
@@ -219,17 +220,9 @@ Notes:
 
 Duis aute irure dolor
 
-
-Example
-~~~~~~~
-
    -> in
    <- out
 
-
-Examples
-~~~~~~~~
-
    - *verbatim*
    - {braces}
 
@@ -260,10 +253,6 @@ Features
 "cmd-feat2"
    another feature
 
-
-Example
-~~~~~~~
-
    -> in
 
    <- out
-- 
2.44.0

Re: [PATCH 02/20] qapi: linter fixups

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Fix minor irritants to pylint/flake8 et al.
>
> (Yes, these need to be guarded by the Python tests. That's a work in
> progress, a series that's quite likely to follow once I finish this
> Sphinx project. Please pardon the temporary irritation.)

No worries; one step at a time.

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/introspect.py | 8 ++++----
>  scripts/qapi/schema.py     | 6 +++---
>  scripts/qapi/visit.py      | 5 +++--
>  3 files changed, 10 insertions(+), 9 deletions(-)
>
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index 86c075a6ad2..ac14b20f308 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -27,8 +27,8 @@
>  from .schema import (
>      QAPISchema,
>      QAPISchemaAlternatives,
> -    QAPISchemaBranches,
>      QAPISchemaArrayType,
> +    QAPISchemaBranches,
>      QAPISchemaBuiltinType,
>      QAPISchemaEntity,
>      QAPISchemaEnumMember,
> @@ -233,9 +233,9 @@ def _use_type(self, typ: QAPISchemaType) -> str:
>              typ = type_int
>          elif (isinstance(typ, QAPISchemaArrayType) and
>                typ.element_type.json_type() == 'int'):
> -            type_intList = self._schema.lookup_type('intList')
> -            assert type_intList
> -            typ = type_intList
> +            type_intlist = self._schema.lookup_type('intList')
> +            assert type_intlist
> +            typ = type_intlist

I named the variable after the type.  Suppressing the linter complaint
just to keep that name isn't worthwhile.  I might have picked
type_int_list instead.  No need to change it now.

>          # Add type to work queue if new
>          if typ not in self._used_types:
>              self._used_types.append(typ)
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 721c470d2b8..d65c35f6ee6 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -730,6 +730,7 @@ def set_defined_in(self, name: str) -> None:
>          for v in self.variants:
>              v.set_defined_in(name)
>  
> +    # pylint: disable=unused-argument
>      def check(
>              self, schema: QAPISchema, seen: Dict[str, QAPISchemaMember]
>      ) -> None:
> @@ -1166,7 +1167,7 @@ def _def_definition(self, defn: QAPISchemaDefinition) -> None:
>                  defn.info, "%s is already defined" % other_defn.describe())
>          self._entity_dict[defn.name] = defn
>  
> -    def lookup_entity(self,name: str) -> Optional[QAPISchemaEntity]:
> +    def lookup_entity(self, name: str) -> Optional[QAPISchemaEntity]:
>          return self._entity_dict.get(name)
>  
>      def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
> @@ -1302,11 +1303,10 @@ def _make_implicit_object_type(
>          name = 'q_obj_%s-%s' % (name, role)
>          typ = self.lookup_entity(name)
>          if typ:
> -            assert(isinstance(typ, QAPISchemaObjectType))
> +            assert isinstance(typ, QAPISchemaObjectType)
>              # The implicit object type has multiple users.  This can
>              # only be a duplicate definition, which will be flagged
>              # later.
> -            pass
>          else:
>              self._def_definition(QAPISchemaObjectType(
>                  name, info, None, ifcond, None, None, members, None))
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index e766acaac92..12f92e429f6 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -280,8 +280,9 @@ def gen_visit_alternate(name: str,
>          abort();
>      default:
>          assert(visit_is_input(v));
> -        error_setg(errp, "Invalid parameter type for '%%s', expected: %(name)s",
> -                         name ? name : "null");
> +        error_setg(errp,
> +                   "Invalid parameter type for '%%s', expected: %(name)s",
> +                   name ? name : "null");
>          /* Avoid passing invalid *obj to qapi_free_%(c_name)s() */
>          g_free(*obj);
>          *obj = NULL;

This is mostly lint I neglected to pick off in my recent work.  Thanks
for taking care of it!

Reviewed-by: Markus Armbruster <armbru@redhat.com>

Re: [PATCH 03/20] docs/qapidoc: delint a tiny portion of the module

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> In the coming patches, it's helpful to have a linting baseline. However,
> there's no need to shuffle around the deck chairs too much, because most
> of this code will be removed once the new qapidoc generator (the
> "transmogrifier") is in place.
>
> To ease my pain: just turn off the black auto-formatter for most, but
> not all, of qapidoc.py. This will help ensure that *new* code follows a
> coding standard without bothering too much with cleaning up the existing
> code.
>
> For manual checking for now, try "black --check qapidoc.py" from the
> docs/sphinx directory. "pip install black" (without root permissions) if
> you do not have it installed otherwise.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  docs/sphinx/qapidoc.py | 16 +++++++++-------
>  1 file changed, 9 insertions(+), 7 deletions(-)
>
> diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> index f270b494f01..1655682d4c7 100644
> --- a/docs/sphinx/qapidoc.py
> +++ b/docs/sphinx/qapidoc.py
> @@ -28,28 +28,30 @@
>  import re
>  
>  from docutils import nodes
> +from docutils.parsers.rst import Directive, directives
>  from docutils.statemachine import ViewList
> -from docutils.parsers.rst import directives, Directive
> -from sphinx.errors import ExtensionError
> -from sphinx.util.nodes import nested_parse_with_titles
> -import sphinx
> -from qapi.gen import QAPISchemaVisitor
>  from qapi.error import QAPIError, QAPISemError
> +from qapi.gen import QAPISchemaVisitor
>  from qapi.schema import QAPISchema
>  
> +import sphinx
> +from sphinx.errors import ExtensionError
> +from sphinx.util.nodes import nested_parse_with_titles
> +

Exchanges old pylint gripe

    docs/sphinx/qapidoc.py:45:4: C0412: Imports from package sphinx are not grouped (ungrouped-imports)

for new gripes

    docs/sphinx/qapidoc.py:37:0: C0411: third party import "import sphinx" should be placed before "from qapi.error import QAPIError, QAPISemError" (wrong-import-order)
    docs/sphinx/qapidoc.py:38:0: C0411: third party import "from sphinx.errors import ExtensionError" should be placed before "from qapi.error import QAPIError, QAPISemError" (wrong-import-order)
    docs/sphinx/qapidoc.py:39:0: C0411: third party import "from sphinx.util.nodes import nested_parse_with_titles" should be placed before "from qapi.error import QAPIError, QAPISemError" (wrong-import-order)

Easy enough to fix.

>  
>  # Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
>  # use switch_source_input. Check borrowed from kerneldoc.py.
> -Use_SSI = sphinx.__version__[:3] >= '1.7'
> +Use_SSI = sphinx.__version__[:3] >= "1.7"
>  if Use_SSI:
>      from sphinx.util.docutils import switch_source_input
>  else:
>      from sphinx.ext.autodoc import AutodocReporter
>  
>  
> -__version__ = '1.0'
> +__version__ = "1.0"
>  
>  
> +# fmt: off

I figure this tells black to keep quiet for the remainder of the file.
Worth a comment, I think.

>  # Function borrowed from pydash, which is under the MIT license
>  def intersperse(iterable, separator):
>      """Yield the members of *iterable* interspersed with *separator*."""

With my comments addressed
Reviewed-by: Markus Armbruster <armbru@redhat.com>

Re: [PATCH 04/20] qapi/parser: preserve indentation in QAPIDoc sections

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Prior to this patch, a section like this:
>
> @name: lorem ipsum
>    dolor sit amet
>      consectetur adipiscing elit
>
> would be parsed as:
>
> "lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"
>
> We want to preserve the indentation for even the first body line so that
> the entire block can be parsed directly as rST. This patch would now
> parse that segment as:
>
> "lorem ipsum\n   dolor sit amet\n     consectetur adipiscing elit"

I'm afraid it's less than clear *why* we want to parse the entire block
directly as rST.  I have just enough insight into what you've built on
top of this series to hazard a guess.  Bear with me while I try to
explain it.

We first parse the doc comment with parser.py into an internal
representation.  The structural parts become objects, and the remainder
becomes text attributes of these objects.  Currently, parser.py
carefully adjusts indentation for these text attributes.  Why?  I'll get
to that.

For your example, parser.py creates an ArgSection object, and sets its
@text member to

    "lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"

Printing this string gives us

    lorem ipsum
    dolor sit amet
      consectetur adipiscing elit

qapidoc.py then transforms parser.py's IR into Sphinx IR.  The objects
become (more complicated) Sphinx objects, and their text attributes get
re-parsed as rST text into still more Sphinx objects.

This re-parse rejects your example with "Unexpected indentation."

Let me use a slightly different one:

    # @name: lorem ipsum
    #    dolor sit amet
    #    consectetur adipiscing elit

Results in this @text member

    lorem ipsum
    dolor sit amet
    consectetur adipiscing elit

which is re-parsed as paragraph, i.e. exactly what we want.

> This understandably breaks qapidoc.py;

Without indentation adjustment, we'd get

    lorem ipsum
       dolor sit amet
       consectetur adipiscing elit

which would be re-parsed as a definition list, I guess.  This isn't what
we want.

>                                        so a new function is added there
> to re-dedent the text.

Your patch moves the indentation adjustment to another place.  No
functional change.

You move it so you can branch off your new rendering pipeline before the
indentation adjustment, because ...

>                        Once the new generator is merged, this function
> will not be needed any longer and can be dropped.

... yours doesn't want it.

I believe it doesn't want it, because it generates rST (with a QAPI
extension) instead of Sphinx objects.  For your example, something like

    :arg name: lorem ipsum
       dolor sit amet
         consectetur adipiscing elit

For mine:

    :arg name: lorem ipsum
       dolor sit amet
       consectetur adipiscing elit

Fair?

The transition from the doc comment to (extended) rST is straightforward
for these examples.

I hope it'll be as straightforward (and thus predictable) in other
cases, too.

> (I verified this patch changes absolutely nothing by comparing the
> md5sums of the QMP ref html pages both before and after the change, so
> it's certified inert. QAPI test output has been updated to reflect the
> new strategy of preserving indents for rST.)
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  docs/sphinx/qapidoc.py         | 36 +++++++++++++++++++++++++++++-----
>  scripts/qapi/parser.py         |  8 ++++++--
>  tests/qapi-schema/doc-good.out | 32 +++++++++++++++---------------
>  3 files changed, 53 insertions(+), 23 deletions(-)
>
> diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> index 1655682d4c7..2e3ffcbafb7 100644
> --- a/docs/sphinx/qapidoc.py
> +++ b/docs/sphinx/qapidoc.py
> @@ -26,6 +26,7 @@
>  
>  import os
>  import re
> +import textwrap
>  
>  from docutils import nodes
>  from docutils.parsers.rst import Directive, directives
> @@ -51,6 +52,28 @@
>  __version__ = "1.0"
>  
>  
> +def dedent(text: str) -> str:
> +    # Temporary: In service of the new QAPI domain, the QAPI doc parser
> +    # now preserves indents in args/members/features text. QAPIDoc does
> +    # not handle this well, so undo that change here.
> +
> +    # QAPIDoc is being rewritten and will be replaced soon,
> +    # but this function is here in the interim as transition glue.

I'm not sure we need the comment.

> +
> +    lines = text.splitlines(True)
> +    if len(lines) > 1:
> +        if re.match(r"\s+", lines[0]):
> +            # First line is indented; description started on
> +            # the line after the name. dedent the whole block.
> +            return textwrap.dedent(text)
> +        else:

pylint gripes

    docs/sphinx/qapidoc.py:65:8: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)

> +            # Descr started on same line. Dedent line 2+.
> +            return lines[0] + textwrap.dedent("".join(lines[1:]))
> +    else:
> +        # Descr was a single line; dedent entire line.
> +        return textwrap.dedent(text)
> +
> +
>  # fmt: off
>  # Function borrowed from pydash, which is under the MIT license
>  def intersperse(iterable, separator):
> @@ -169,7 +192,7 @@ def _nodes_for_members(self, doc, what, base=None, branches=None):
>              term = self._nodes_for_one_member(section.member)
>              # TODO drop fallbacks when undocumented members are outlawed
>              if section.text:
> -                defn = section.text
> +                defn = dedent(section.text)
>              else:
>                  defn = [nodes.Text('Not documented')]
>  
> @@ -207,7 +230,7 @@ def _nodes_for_enum_values(self, doc):
>                  termtext.extend(self._nodes_for_ifcond(section.member.ifcond))
>              # TODO drop fallbacks when undocumented members are outlawed
>              if section.text:
> -                defn = section.text
> +                defn = dedent(section.text)
>              else:
>                  defn = [nodes.Text('Not documented')]
>  
> @@ -242,7 +265,7 @@ def _nodes_for_features(self, doc):
>          dlnode = nodes.definition_list()
>          for section in doc.features.values():
>              dlnode += self._make_dlitem(
> -                [nodes.literal('', section.member.name)], section.text)
> +                [nodes.literal('', section.member.name)], dedent(section.text))
>              seen_item = True
>  
>          if not seen_item:
> @@ -265,9 +288,12 @@ def _nodes_for_sections(self, doc):
>                  continue
>              snode = self._make_section(section.tag)
>              if section.tag and section.tag.startswith('Example'):
> -                snode += self._nodes_for_example(section.text)
> +                snode += self._nodes_for_example(dedent(section.text))
>              else:
> -                self._parse_text_into_node(section.text, snode)
> +                self._parse_text_into_node(
> +                    dedent(section.text) if section.tag else section.text,
> +                    snode,
> +                )
>              nodelist.append(snode)
>          return nodelist
>  
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index 7b13a583ac1..8cdd5334ec6 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -448,7 +448,10 @@ def get_doc_indented(self, doc: 'QAPIDoc') -> Optional[str]:
>          indent = must_match(r'\s*', line).end()
>          if not indent:
>              return line
> -        doc.append_line(line[indent:])
> +
> +        # Preserve the indent, it's needed for rST formatting.

I'm not sure we need the comment.

> +        doc.append_line(line)
> +
>          prev_line_blank = False
>          while True:
>              self.accept(False)
> @@ -465,7 +468,8 @@ def get_doc_indented(self, doc: 'QAPIDoc') -> Optional[str]:
>                      self,
>                      "unexpected de-indent (expected at least %d spaces)" %
>                      indent)
> -            doc.append_line(line[indent:])
> +            # Again, preserve the indent for ReST.

Likewise.

If you want to document the fact that .get_doc_indented() preserves
indentation, a function comment or doc string would the proper place.

> +            doc.append_line(line)
>              prev_line_blank = True
>  
>      def get_doc_paragraph(self, doc: 'QAPIDoc') -> Optional[str]:

Correctness argument:

0. This patch merely moves an indentation adjustment from parser.py to
   qapidoc.py.

   Checks out: you remove indentation adjustment in parser.py, add it in
   qapidoc.py, and that's it.

1. The new indentation adjuster in qapidoc.py behaves just like the old
   one in parser.py did.

   parser.py's .get_doc_indented() is used to parse what follows certain
   sections' first line.  It's users store the text on this first line,
   if any, with leading whitespace stripped, then call
   .get_doc_indented() to eat the section's remaining lines.  All
   non-blank lines eaten must be indented at least as much as the first
   non-blank line.  .get_doc_indented() appends the lines eaten to the
   stored text with the first non-blank line's indentation stripped from
   all the non-blank lines.

   Your patch drops this stripping of indentation from non-first lines.
   This is what must now be done in qapidoc.py.

   If the section's first line has no text, the first non-blank line's
   indentation must be stripped from all non-blank lines.

   If the section's first line has text, the next non-blank line's
   indentation must be stripped from all lines but the first.

   How can qapidoc.py detect whether the section's first line has text?
   Fortunately, such a line will be unindented, i.e. start with a
   non-blank character, and all remaining lines will be blank or
   indented.

   qapidoc.py's dedent() seems to remove indentation common to all
   non-blank lines.  Except when there are multiple lines, and the first
   one is not indented, it removes common indentation from the remaining
   lines.

   Common indentation works, because all lines are indented at least as
   much as the one whose indentation we want to remove.

   The conditional doesn't quite match the "if the section's first line"
   above.  This simpler dedent() does, and also works in my testing:

def dedent(text: str) -> str:
    lines = text.splitlines(True)
    if re.match(r"\s+", lines[0]):
        # First line is indented; description started on
        # the line after the name. dedent the whole block.
        return textwrap.dedent(text)
    # Descr started on same line. Dedent line 2+.
    return lines[0] + textwrap.dedent("".join(lines[1:]))

   What do you think?

2. The new one is applied exactly when the old one was.

   The old one is applied to the sections starting with @FOO: and the
   tagged sections (Returns:, Errors:, ...).

   The new one is applied in ._nodes_for_members(),
   ._nodes_for_enum_values(), _nodes_for_features(), and
   ._nodes_for_sections().

   It is not applied to the text of untagged sections, including the
   body section.

   Good.

> diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
> index 716a9a41026..435f6e6d768 100644
> --- a/tests/qapi-schema/doc-good.out
> +++ b/tests/qapi-schema/doc-good.out
> @@ -117,8 +117,8 @@ doc symbol=Base
>      body=
>  
>      arg=base1
> -description starts on a new line,
> -minimally indented
> + description starts on a new line,
> + minimally indented
>  doc symbol=Variant1
>      body=
>  A paragraph
> @@ -145,8 +145,8 @@ doc symbol=Alternate
>  
>      arg=i
>  description starts on the same line
> -remainder indented the same
> -@b is undocumented
> +    remainder indented the same
> +    @b is undocumented
>      arg=b
>  
>      feature=alt-feat
> @@ -158,11 +158,11 @@ doc symbol=cmd
>      body=
>  
>      arg=arg1
> -description starts on a new line,
> -indented
> +    description starts on a new line,
> +    indented
>      arg=arg2
>  description starts on the same line
> -remainder indented differently
> +    remainder indented differently
>      arg=arg3
>  
>      feature=cmd-feat1
> @@ -178,16 +178,16 @@ some
>      section=TODO
>  frobnicate
>      section=Notes
> -- Lorem ipsum dolor sit amet
> -- Ut enim ad minim veniam
> + - Lorem ipsum dolor sit amet
> + - Ut enim ad minim veniam
>  
> -Duis aute irure dolor
> + Duis aute irure dolor
>      section=Example
> --> in
> -<- out
> + -> in
> + <- out
>      section=Examples
> -- *verbatim*
> -- {braces}
> + - *verbatim*
> + - {braces}
>      section=Since
>  2.10
>  doc symbol=cmd-boxed
> @@ -198,9 +198,9 @@ a feature
>      feature=cmd-feat2
>  another feature
>      section=Example
> --> in
> + -> in
>  
> -<- out
> + <- out
>  doc symbol=EVT_BOXED
>      body=

The indentation change is nicely visible here.

Re: [PATCH 03/20] docs/qapidoc: delint a tiny portion of the module

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 3873 bytes --]

On Wed, May 15, 2024, 5:17 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > In the coming patches, it's helpful to have a linting baseline. However,
> > there's no need to shuffle around the deck chairs too much, because most
> > of this code will be removed once the new qapidoc generator (the
> > "transmogrifier") is in place.
> >
> > To ease my pain: just turn off the black auto-formatter for most, but
> > not all, of qapidoc.py. This will help ensure that *new* code follows a
> > coding standard without bothering too much with cleaning up the existing
> > code.
> >
> > For manual checking for now, try "black --check qapidoc.py" from the
> > docs/sphinx directory. "pip install black" (without root permissions) if
> > you do not have it installed otherwise.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  docs/sphinx/qapidoc.py | 16 +++++++++-------
> >  1 file changed, 9 insertions(+), 7 deletions(-)
> >
> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> > index f270b494f01..1655682d4c7 100644
> > --- a/docs/sphinx/qapidoc.py
> > +++ b/docs/sphinx/qapidoc.py
> > @@ -28,28 +28,30 @@
> >  import re
> >
> >  from docutils import nodes
> > +from docutils.parsers.rst import Directive, directives
> >  from docutils.statemachine import ViewList
> > -from docutils.parsers.rst import directives, Directive
> > -from sphinx.errors import ExtensionError
> > -from sphinx.util.nodes import nested_parse_with_titles
> > -import sphinx
> > -from qapi.gen import QAPISchemaVisitor
> >  from qapi.error import QAPIError, QAPISemError
> > +from qapi.gen import QAPISchemaVisitor
> >  from qapi.schema import QAPISchema
> >
> > +import sphinx
> > +from sphinx.errors import ExtensionError
> > +from sphinx.util.nodes import nested_parse_with_titles
> > +
>
> Exchanges old pylint gripe
>
>     docs/sphinx/qapidoc.py:45:4: C0412: Imports from package sphinx are
> not grouped (ungrouped-imports)
>
> for new gripes
>
>     docs/sphinx/qapidoc.py:37:0: C0411: third party import "import sphinx"
> should be placed before "from qapi.error import QAPIError, QAPISemError"
> (wrong-import-order)
>     docs/sphinx/qapidoc.py:38:0: C0411: third party import "from
> sphinx.errors import ExtensionError" should be placed before "from
> qapi.error import QAPIError, QAPISemError" (wrong-import-order)
>     docs/sphinx/qapidoc.py:39:0: C0411: third party import "from
> sphinx.util.nodes import nested_parse_with_titles" should be placed before
> "from qapi.error import QAPIError, QAPISemError" (wrong-import-order)
>
> Easy enough to fix.
>

This is a problem where our sphinx directory is colliding with the sphinx
namespace and different versions of the tooling disagree with the
assessment.

I'll try to fix this without renaming our directory, but I'm worried that
might be the most robust solution.


> >
> >  # Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
> >  # use switch_source_input. Check borrowed from kerneldoc.py.
> > -Use_SSI = sphinx.__version__[:3] >= '1.7'
> > +Use_SSI = sphinx.__version__[:3] >= "1.7"
> >  if Use_SSI:
> >      from sphinx.util.docutils import switch_source_input
> >  else:
> >      from sphinx.ext.autodoc import AutodocReporter
> >
> >
> > -__version__ = '1.0'
> > +__version__ = "1.0"
> >
> >
> > +# fmt: off
>
> I figure this tells black to keep quiet for the remainder of the file.
> Worth a comment, I think.
>

It does, yes. Want an inline comment here?


> >  # Function borrowed from pydash, which is under the MIT license
> >  def intersperse(iterable, separator):
> >      """Yield the members of *iterable* interspersed with *separator*."""
>
> With my comments addressed
> Reviewed-by: Markus Armbruster <armbru@redhat.com>
>
>

[-- Attachment #2: Type: text/html, Size: 5455 bytes --]

Re: [PATCH 04/20] qapi/parser: preserve indentation in QAPIDoc sections

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 15511 bytes --]

On Wed, May 15, 2024, 7:50 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > Prior to this patch, a section like this:
> >
> > @name: lorem ipsum
> >    dolor sit amet
> >      consectetur adipiscing elit
> >
> > would be parsed as:
> >
> > "lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"
> >
> > We want to preserve the indentation for even the first body line so that
> > the entire block can be parsed directly as rST. This patch would now
> > parse that segment as:
> >
> > "lorem ipsum\n   dolor sit amet\n     consectetur adipiscing elit"
>
> I'm afraid it's less than clear *why* we want to parse the entire block
> directly as rST.  I have just enough insight into what you've built on
> top of this series to hazard a guess.  Bear with me while I try to
> explain it.
>

My own summary: qapidoc expects a paragraph, the new generator expects a
block.


> We first parse the doc comment with parser.py into an internal
> representation.  The structural parts become objects, and the remainder
> becomes text attributes of these objects.  Currently, parser.py
> carefully adjusts indentation for these text attributes.  Why?  I'll get
> to that.
>
> For your example, parser.py creates an ArgSection object, and sets its
> @text member to
>
>     "lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"
>
> Printing this string gives us
>
>     lorem ipsum
>     dolor sit amet
>       consectetur adipiscing elit
>
> qapidoc.py then transforms parser.py's IR into Sphinx IR.  The objects
> become (more complicated) Sphinx objects, and their text attributes get
> re-parsed as rST text into still more Sphinx objects.
>
> This re-parse rejects your example with "Unexpected indentation."
>

Specifically, it'd be an unexpected *unindent*; the indent lacking on the
first *two* lines is the problem.


> Let me use a slightly different one:
>
>     # @name: lorem ipsum
>     #    dolor sit amet
>     #    consectetur adipiscing elit
>
> Results in this @text member
>
>     lorem ipsum
>     dolor sit amet
>     consectetur adipiscing elit
>
> which is re-parsed as paragraph, i.e. exactly what we want.
>

It's what we used to want, anyway.


> > This understandably breaks qapidoc.py;
>
> Without indentation adjustment, we'd get
>
>     lorem ipsum
>        dolor sit amet
>        consectetur adipiscing elit
>
> which would be re-parsed as a definition list, I guess.  This isn't what
> we want.
>
> >                                        so a new function is added there
> > to re-dedent the text.
>
> Your patch moves the indentation adjustment to another place.  No
> functional change.
>
> You move it so you can branch off your new rendering pipeline before the
> indentation adjustment, because ...
>
> >                        Once the new generator is merged, this function
> > will not be needed any longer and can be dropped.
>
> ... yours doesn't want it.
>
> I believe it doesn't want it, because it generates rST (with a QAPI
> extension) instead of Sphinx objects.  For your example, something like
>
>     :arg name: lorem ipsum
>        dolor sit amet
>          consectetur adipiscing elit
>
> For mine:
>
>     :arg name: lorem ipsum
>        dolor sit amet
>        consectetur adipiscing elit
>
> Fair?
>

Not quite;

Old parsing, new generator:

:arg type name: lorem ipsum
dolor sit amet
  consectetur apidiscing elit

This is wrong - continuations of a field list must be indented. Unlike
paragraphs, we want indents to "keep the block".

New parsing, new generator:

:arg type name: lorem ipsum
   dolor sit amet
     consectetur apidiscing elit

indent is preserved, maintaining the block-level element.

I don't have to re-add indents and any nested block elements will be
preserved correctly. i.e. you can use code examples, nested lists, etc. in
argument definitions.

The goal here was "Do not treat this as a paragraph, treat it directly as
rST and do not modify it needlessly."

It's a lot simpler than trying to manage the indent and injecting spaces
manually - and adding a temporary dedent to scheduled-for-demolition code
seemed the nicer place to add the hack.


> The transition from the doc comment to (extended) rST is straightforward
> for these examples.
>
> I hope it'll be as straightforward (and thus predictable) in other
> cases, too.
>
> > (I verified this patch changes absolutely nothing by comparing the
> > md5sums of the QMP ref html pages both before and after the change, so
> > it's certified inert. QAPI test output has been updated to reflect the
> > new strategy of preserving indents for rST.)
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  docs/sphinx/qapidoc.py         | 36 +++++++++++++++++++++++++++++-----
> >  scripts/qapi/parser.py         |  8 ++++++--
> >  tests/qapi-schema/doc-good.out | 32 +++++++++++++++---------------
> >  3 files changed, 53 insertions(+), 23 deletions(-)
> >
> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> > index 1655682d4c7..2e3ffcbafb7 100644
> > --- a/docs/sphinx/qapidoc.py
> > +++ b/docs/sphinx/qapidoc.py
> > @@ -26,6 +26,7 @@
> >
> >  import os
> >  import re
> > +import textwrap
> >
> >  from docutils import nodes
> >  from docutils.parsers.rst import Directive, directives
> > @@ -51,6 +52,28 @@
> >  __version__ = "1.0"
> >
> >
> > +def dedent(text: str) -> str:
> > +    # Temporary: In service of the new QAPI domain, the QAPI doc parser
> > +    # now preserves indents in args/members/features text. QAPIDoc does
> > +    # not handle this well, so undo that change here.
> > +
> > +    # QAPIDoc is being rewritten and will be replaced soon,
> > +    # but this function is here in the interim as transition glue.
>
> I'm not sure we need the comment.
>

OK. Guess I'd rather overcomment than undercomment... easier to delete than
add :)


> > +
> > +    lines = text.splitlines(True)
> > +    if len(lines) > 1:
> > +        if re.match(r"\s+", lines[0]):
> > +            # First line is indented; description started on
> > +            # the line after the name. dedent the whole block.
> > +            return textwrap.dedent(text)
> > +        else:
>
> pylint gripes
>
>     docs/sphinx/qapidoc.py:65:8: R1705: Unnecessary "else" after "return",
> remove the "else" and de-indent the code inside it (no-else-return)
>

Interesting. What pylint version?


> > +            # Descr started on same line. Dedent line 2+.
> > +            return lines[0] + textwrap.dedent("".join(lines[1:]))
> > +    else:
> > +        # Descr was a single line; dedent entire line.
> > +        return textwrap.dedent(text)
> > +
> > +
> >  # fmt: off
> >  # Function borrowed from pydash, which is under the MIT license
> >  def intersperse(iterable, separator):
> > @@ -169,7 +192,7 @@ def _nodes_for_members(self, doc, what, base=None,
> branches=None):
> >              term = self._nodes_for_one_member(section.member)
> >              # TODO drop fallbacks when undocumented members are outlawed
> >              if section.text:
> > -                defn = section.text
> > +                defn = dedent(section.text)
> >              else:
> >                  defn = [nodes.Text('Not documented')]
> >
> > @@ -207,7 +230,7 @@ def _nodes_for_enum_values(self, doc):
> >
> termtext.extend(self._nodes_for_ifcond(section.member.ifcond))
> >              # TODO drop fallbacks when undocumented members are outlawed
> >              if section.text:
> > -                defn = section.text
> > +                defn = dedent(section.text)
> >              else:
> >                  defn = [nodes.Text('Not documented')]
> >
> > @@ -242,7 +265,7 @@ def _nodes_for_features(self, doc):
> >          dlnode = nodes.definition_list()
> >          for section in doc.features.values():
> >              dlnode += self._make_dlitem(
> > -                [nodes.literal('', section.member.name)], section.text)
> > +                [nodes.literal('', section.member.name)],
> dedent(section.text))
> >              seen_item = True
> >
> >          if not seen_item:
> > @@ -265,9 +288,12 @@ def _nodes_for_sections(self, doc):
> >                  continue
> >              snode = self._make_section(section.tag)
> >              if section.tag and section.tag.startswith('Example'):
> > -                snode += self._nodes_for_example(section.text)
> > +                snode += self._nodes_for_example(dedent(section.text))
> >              else:
> > -                self._parse_text_into_node(section.text, snode)
> > +                self._parse_text_into_node(
> > +                    dedent(section.text) if section.tag else
> section.text,
> > +                    snode,
> > +                )
> >              nodelist.append(snode)
> >          return nodelist
> >
> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> > index 7b13a583ac1..8cdd5334ec6 100644
> > --- a/scripts/qapi/parser.py
> > +++ b/scripts/qapi/parser.py
> > @@ -448,7 +448,10 @@ def get_doc_indented(self, doc: 'QAPIDoc') ->
> Optional[str]:
> >          indent = must_match(r'\s*', line).end()
> >          if not indent:
> >              return line
> > -        doc.append_line(line[indent:])
> > +
> > +        # Preserve the indent, it's needed for rST formatting.
>
> I'm not sure we need the comment.
>

OK.


> > +        doc.append_line(line)
> > +
> >          prev_line_blank = False
> >          while True:
> >              self.accept(False)
> > @@ -465,7 +468,8 @@ def get_doc_indented(self, doc: 'QAPIDoc') ->
> Optional[str]:
> >                      self,
> >                      "unexpected de-indent (expected at least %d
> spaces)" %
> >                      indent)
> > -            doc.append_line(line[indent:])
> > +            # Again, preserve the indent for ReST.
>
> Likewise.
>
> If you want to document the fact that .get_doc_indented() preserves
> indentation, a function comment or doc string would the proper place.
>

Got it.


> > +            doc.append_line(line)
> >              prev_line_blank = True
> >
> >      def get_doc_paragraph(self, doc: 'QAPIDoc') -> Optional[str]:
>
> Correctness argument:
>
> 0. This patch merely moves an indentation adjustment from parser.py to
>    qapidoc.py.
>
>    Checks out: you remove indentation adjustment in parser.py, add it in
>    qapidoc.py, and that's it.
>
> 1. The new indentation adjuster in qapidoc.py behaves just like the old
>    one in parser.py did.
>
>    parser.py's .get_doc_indented() is used to parse what follows certain
>    sections' first line.  It's users store the text on this first line,
>    if any, with leading whitespace stripped, then call
>    .get_doc_indented() to eat the section's remaining lines.  All
>    non-blank lines eaten must be indented at least as much as the first
>    non-blank line.  .get_doc_indented() appends the lines eaten to the
>    stored text with the first non-blank line's indentation stripped from
>    all the non-blank lines.
>
>    Your patch drops this stripping of indentation from non-first lines.
>    This is what must now be done in qapidoc.py.
>
>    If the section's first line has no text, the first non-blank line's
>    indentation must be stripped from all non-blank lines.
>
>    If the section's first line has text, the next non-blank line's
>    indentation must be stripped from all lines but the first.
>
>    How can qapidoc.py detect whether the section's first line has text?
>    Fortunately, such a line will be unindented, i.e. start with a
>    non-blank character, and all remaining lines will be blank or
>    indented.
>
>    qapidoc.py's dedent() seems to remove indentation common to all
>    non-blank lines.  Except when there are multiple lines, and the first
>    one is not indented, it removes common indentation from the remaining
>    lines.
>
>    Common indentation works, because all lines are indented at least as
>    much as the one whose indentation we want to remove.
>
>    The conditional doesn't quite match the "if the section's first line"
>    above.  This simpler dedent() does, and also works in my testing:
>
> def dedent(text: str) -> str:
>     lines = text.splitlines(True)
>     if re.match(r"\s+", lines[0]):
>         # First line is indented; description started on
>         # the line after the name. dedent the whole block.
>         return textwrap.dedent(text)
>     # Descr started on same line. Dedent line 2+.
>     return lines[0] + textwrap.dedent("".join(lines[1:]))
>
>    What do you think?
>

I try not to on most days.

I'll check it out, though in practice the generated documents were already
identical, so... I'll try yours and verify that's still true. If so, sure!


> 2. The new one is applied exactly when the old one was.
>
>    The old one is applied to the sections starting with @FOO: and the
>    tagged sections (Returns:, Errors:, ...).
>
>    The new one is applied in ._nodes_for_members(),
>    ._nodes_for_enum_values(), _nodes_for_features(), and
>    ._nodes_for_sections().
>
>    It is not applied to the text of untagged sections, including the
>    body section.
>
>    Good.
>
> > diff --git a/tests/qapi-schema/doc-good.out
> b/tests/qapi-schema/doc-good.out
> > index 716a9a41026..435f6e6d768 100644
> > --- a/tests/qapi-schema/doc-good.out
> > +++ b/tests/qapi-schema/doc-good.out
> > @@ -117,8 +117,8 @@ doc symbol=Base
> >      body=
> >
> >      arg=base1
> > -description starts on a new line,
> > -minimally indented
> > + description starts on a new line,
> > + minimally indented
> >  doc symbol=Variant1
> >      body=
> >  A paragraph
> > @@ -145,8 +145,8 @@ doc symbol=Alternate
> >
> >      arg=i
> >  description starts on the same line
> > -remainder indented the same
> > -@b is undocumented
> > +    remainder indented the same
> > +    @b is undocumented
> >      arg=b
> >
> >      feature=alt-feat
> > @@ -158,11 +158,11 @@ doc symbol=cmd
> >      body=
> >
> >      arg=arg1
> > -description starts on a new line,
> > -indented
> > +    description starts on a new line,
> > +    indented
> >      arg=arg2
> >  description starts on the same line
> > -remainder indented differently
> > +    remainder indented differently
> >      arg=arg3
> >
> >      feature=cmd-feat1
> > @@ -178,16 +178,16 @@ some
> >      section=TODO
> >  frobnicate
> >      section=Notes
> > -- Lorem ipsum dolor sit amet
> > -- Ut enim ad minim veniam
> > + - Lorem ipsum dolor sit amet
> > + - Ut enim ad minim veniam
> >
> > -Duis aute irure dolor
> > + Duis aute irure dolor
> >      section=Example
> > --> in
> > -<- out
> > + -> in
> > + <- out
> >      section=Examples
> > -- *verbatim*
> > -- {braces}
> > + - *verbatim*
> > + - {braces}
> >      section=Since
> >  2.10
> >  doc symbol=cmd-boxed
> > @@ -198,9 +198,9 @@ a feature
> >      feature=cmd-feat2
> >  another feature
> >      section=Example
> > --> in
> > + -> in
> >
> > -<- out
> > + <- out
> >  doc symbol=EVT_BOXED
> >      body=
>
> The indentation change is nicely visible here.
>
>

[-- Attachment #2: Type: text/html, Size: 21424 bytes --]

Re: [PATCH 04/20] qapi/parser: preserve indentation in QAPIDoc sections

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, May 15, 2024, 7:50 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Prior to this patch, a section like this:
>> >
>> > @name: lorem ipsum
>> >    dolor sit amet
>> >      consectetur adipiscing elit
>> >
>> > would be parsed as:
>> >
>> > "lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"
>> >
>> > We want to preserve the indentation for even the first body line so that
>> > the entire block can be parsed directly as rST. This patch would now
>> > parse that segment as:
>> >
>> > "lorem ipsum\n   dolor sit amet\n     consectetur adipiscing elit"
>>
>> I'm afraid it's less than clear *why* we want to parse the entire block
>> directly as rST.  I have just enough insight into what you've built on
>> top of this series to hazard a guess.  Bear with me while I try to
>> explain it.
>>
>
> My own summary: qapidoc expects a paragraph, the new generator expects a
> block.
>
>
>> We first parse the doc comment with parser.py into an internal
>> representation.  The structural parts become objects, and the remainder
>> becomes text attributes of these objects.  Currently, parser.py
>> carefully adjusts indentation for these text attributes.  Why?  I'll get
>> to that.
>>
>> For your example, parser.py creates an ArgSection object, and sets its
>> @text member to
>>
>>     "lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"
>>
>> Printing this string gives us
>>
>>     lorem ipsum
>>     dolor sit amet
>>       consectetur adipiscing elit
>>
>> qapidoc.py then transforms parser.py's IR into Sphinx IR.  The objects
>> become (more complicated) Sphinx objects, and their text attributes get
>> re-parsed as rST text into still more Sphinx objects.
>>
>> This re-parse rejects your example with "Unexpected indentation."
>>
>
> Specifically, it'd be an unexpected *unindent*; the indent lacking on the
> first *two* lines is the problem.
>
>
>> Let me use a slightly different one:
>>
>>     # @name: lorem ipsum
>>     #    dolor sit amet
>>     #    consectetur adipiscing elit
>>
>> Results in this @text member
>>
>>     lorem ipsum
>>     dolor sit amet
>>     consectetur adipiscing elit
>>
>> which is re-parsed as paragraph, i.e. exactly what we want.
>>
>
> It's what we used to want, anyway.

Yes, I'm describing the current state here.

>> > This understandably breaks qapidoc.py;
>>
>> Without indentation adjustment, we'd get
>>
>>     lorem ipsum
>>        dolor sit amet
>>        consectetur adipiscing elit
>>
>> which would be re-parsed as a definition list, I guess.  This isn't what
>> we want.
>>
>> >                                        so a new function is added there
>> > to re-dedent the text.
>>
>> Your patch moves the indentation adjustment to another place.  No
>> functional change.
>>
>> You move it so you can branch off your new rendering pipeline before the
>> indentation adjustment, because ...
>>
>> >                        Once the new generator is merged, this function
>> > will not be needed any longer and can be dropped.
>>
>> ... yours doesn't want it.
>>
>> I believe it doesn't want it, because it generates rST (with a QAPI
>> extension) instead of Sphinx objects.  For your example, something like
>>
>>     :arg name: lorem ipsum
>>        dolor sit amet
>>          consectetur adipiscing elit
>>
>> For mine:
>>
>>     :arg name: lorem ipsum
>>        dolor sit amet
>>        consectetur adipiscing elit
>>
>> Fair?
>>
>
> Not quite;
>
> Old parsing, new generator:
>
> :arg type name: lorem ipsum
> dolor sit amet
>   consectetur apidiscing elit
>
> This is wrong - continuations of a field list must be indented. Unlike
> paragraphs, we want indents to "keep the block".
>
> New parsing, new generator:
>
> :arg type name: lorem ipsum
>    dolor sit amet
>      consectetur apidiscing elit
>
> indent is preserved, maintaining the block-level element.
>
> I don't have to re-add indents and any nested block elements will be
> preserved correctly. i.e. you can use code examples, nested lists, etc. in
> argument definitions.
>
> The goal here was "Do not treat this as a paragraph, treat it directly as
> rST and do not modify it needlessly."
>
> It's a lot simpler than trying to manage the indent and injecting spaces
> manually - and adding a temporary dedent to scheduled-for-demolition code
> seemed the nicer place to add the hack.

Understand.

A bit more rationale in the commit message would be nice.  Perhaps start
with current state ("we deintent"), then describe the patch ("move the
deindent"), then rationale "to get it out of the way of a new thingy I
wrote, and intend to post soonish", then "which will replace qapidoc.py
entirely".

>> The transition from the doc comment to (extended) rST is straightforward
>> for these examples.
>>
>> I hope it'll be as straightforward (and thus predictable) in other
>> cases, too.
>>
>> > (I verified this patch changes absolutely nothing by comparing the
>> > md5sums of the QMP ref html pages both before and after the change, so
>> > it's certified inert. QAPI test output has been updated to reflect the
>> > new strategy of preserving indents for rST.)
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  docs/sphinx/qapidoc.py         | 36 +++++++++++++++++++++++++++++-----
>> >  scripts/qapi/parser.py         |  8 ++++++--
>> >  tests/qapi-schema/doc-good.out | 32 +++++++++++++++---------------
>> >  3 files changed, 53 insertions(+), 23 deletions(-)
>> >
>> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
>> > index 1655682d4c7..2e3ffcbafb7 100644
>> > --- a/docs/sphinx/qapidoc.py
>> > +++ b/docs/sphinx/qapidoc.py
>> > @@ -26,6 +26,7 @@
>> >
>> >  import os
>> >  import re
>> > +import textwrap
>> >
>> >  from docutils import nodes
>> >  from docutils.parsers.rst import Directive, directives
>> > @@ -51,6 +52,28 @@
>> >  __version__ = "1.0"
>> >
>> >
>> > +def dedent(text: str) -> str:
>> > +    # Temporary: In service of the new QAPI domain, the QAPI doc parser
>> > +    # now preserves indents in args/members/features text. QAPIDoc does
>> > +    # not handle this well, so undo that change here.
>> > +
>> > +    # QAPIDoc is being rewritten and will be replaced soon,
>> > +    # but this function is here in the interim as transition glue.
>>
>> I'm not sure we need the comment.
>
> OK. Guess I'd rather overcomment than undercomment... easier to delete than
> add :)

Right :)

>> > +
>> > +    lines = text.splitlines(True)
>> > +    if len(lines) > 1:
>> > +        if re.match(r"\s+", lines[0]):
>> > +            # First line is indented; description started on
>> > +            # the line after the name. dedent the whole block.
>> > +            return textwrap.dedent(text)
>> > +        else:
>>
>> pylint gripes
>>
>>     docs/sphinx/qapidoc.py:65:8: R1705: Unnecessary "else" after "return",
>> remove the "else" and de-indent the code inside it (no-else-return)
>>
>
> Interesting. What pylint version?

Fedora 39

$ pylint --version
pylint 3.0.4
astroid 3.0.3
Python 3.12.3 (main, Apr 17 2024, 00:00:00) [GCC 13.2.1 20240316 (Red Hat 13.2.1-7)]

>> > +            # Descr started on same line. Dedent line 2+.
>> > +            return lines[0] + textwrap.dedent("".join(lines[1:]))
>> > +    else:
>> > +        # Descr was a single line; dedent entire line.
>> > +        return textwrap.dedent(text)
>> > +
>> > +
>> >  # fmt: off
>> >  # Function borrowed from pydash, which is under the MIT license
>> >  def intersperse(iterable, separator):
>> > @@ -169,7 +192,7 @@ def _nodes_for_members(self, doc, what, base=None, branches=None):
>> >              term = self._nodes_for_one_member(section.member)
>> >              # TODO drop fallbacks when undocumented members are outlawed
>> >              if section.text:
>> > -                defn = section.text
>> > +                defn = dedent(section.text)
>> >              else:
>> >                  defn = [nodes.Text('Not documented')]
>> >
>> > @@ -207,7 +230,7 @@ def _nodes_for_enum_values(self, doc):
>> >                  termtext.extend(self._nodes_for_ifcond(section.member.ifcond))
>> >              # TODO drop fallbacks when undocumented members are outlawed
>> >              if section.text:
>> > -                defn = section.text
>> > +                defn = dedent(section.text)
>> >              else:
>> >                  defn = [nodes.Text('Not documented')]
>> >
>> > @@ -242,7 +265,7 @@ def _nodes_for_features(self, doc):
>> >          dlnode = nodes.definition_list()
>> >          for section in doc.features.values():
>> >              dlnode += self._make_dlitem(
>> > -                [nodes.literal('', section.member.name)], section.text)
>> > +                [nodes.literal('', section.member.name)], dedent(section.text))
>> >              seen_item = True
>> >
>> >          if not seen_item:
>> > @@ -265,9 +288,12 @@ def _nodes_for_sections(self, doc):
>> >                  continue
>> >              snode = self._make_section(section.tag)
>> >              if section.tag and section.tag.startswith('Example'):
>> > -                snode += self._nodes_for_example(section.text)
>> > +                snode += self._nodes_for_example(dedent(section.text))
>> >              else:
>> > -                self._parse_text_into_node(section.text, snode)
>> > +                self._parse_text_into_node(
>> > +                    dedent(section.text) if section.tag else section.text,
>> > +                    snode,
>> > +                )
>> >              nodelist.append(snode)
>> >          return nodelist
>> >
>> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>> > index 7b13a583ac1..8cdd5334ec6 100644
>> > --- a/scripts/qapi/parser.py
>> > +++ b/scripts/qapi/parser.py
>> > @@ -448,7 +448,10 @@ def get_doc_indented(self, doc: 'QAPIDoc') -> Optional[str]:
>> >          indent = must_match(r'\s*', line).end()
>> >          if not indent:
>> >              return line
>> > -        doc.append_line(line[indent:])
>> > +
>> > +        # Preserve the indent, it's needed for rST formatting.
>>
>> I'm not sure we need the comment.
>>
>
> OK.
>
>
>> > +        doc.append_line(line)
>> > +
>> >          prev_line_blank = False
>> >          while True:
>> >              self.accept(False)
>> > @@ -465,7 +468,8 @@ def get_doc_indented(self, doc: 'QAPIDoc') -> Optional[str]:
>> >                      self,
>> >                      "unexpected de-indent (expected at least %d spaces)" %
>> >                      indent)
>> > -            doc.append_line(line[indent:])
>> > +            # Again, preserve the indent for ReST.
>>
>> Likewise.
>>
>> If you want to document the fact that .get_doc_indented() preserves
>> indentation, a function comment or doc string would the proper place.
>>
>
> Got it.
>
>
>> > +            doc.append_line(line)
>> >              prev_line_blank = True
>> >
>> >      def get_doc_paragraph(self, doc: 'QAPIDoc') -> Optional[str]:
>>
>> Correctness argument:
>>
>> 0. This patch merely moves an indentation adjustment from parser.py to
>>    qapidoc.py.
>>
>>    Checks out: you remove indentation adjustment in parser.py, add it in
>>    qapidoc.py, and that's it.
>>
>> 1. The new indentation adjuster in qapidoc.py behaves just like the old
>>    one in parser.py did.
>>
>>    parser.py's .get_doc_indented() is used to parse what follows certain
>>    sections' first line.  It's users store the text on this first line,
>>    if any, with leading whitespace stripped, then call
>>    .get_doc_indented() to eat the section's remaining lines.  All
>>    non-blank lines eaten must be indented at least as much as the first
>>    non-blank line.  .get_doc_indented() appends the lines eaten to the
>>    stored text with the first non-blank line's indentation stripped from
>>    all the non-blank lines.
>>
>>    Your patch drops this stripping of indentation from non-first lines.
>>    This is what must now be done in qapidoc.py.
>>
>>    If the section's first line has no text, the first non-blank line's
>>    indentation must be stripped from all non-blank lines.
>>
>>    If the section's first line has text, the next non-blank line's
>>    indentation must be stripped from all lines but the first.
>>
>>    How can qapidoc.py detect whether the section's first line has text?
>>    Fortunately, such a line will be unindented, i.e. start with a
>>    non-blank character, and all remaining lines will be blank or
>>    indented.
>>
>>    qapidoc.py's dedent() seems to remove indentation common to all
>>    non-blank lines.  Except when there are multiple lines, and the first
>>    one is not indented, it removes common indentation from the remaining
>>    lines.
>>
>>    Common indentation works, because all lines are indented at least as
>>    much as the one whose indentation we want to remove.
>>
>>    The conditional doesn't quite match the "if the section's first line"
>>    above.  This simpler dedent() does, and also works in my testing:
>>
>> def dedent(text: str) -> str:
>>     lines = text.splitlines(True)
>>     if re.match(r"\s+", lines[0]):
>>         # First line is indented; description started on
>>         # the line after the name. dedent the whole block.
>>         return textwrap.dedent(text)
>>     # Descr started on same line. Dedent line 2+.
>>     return lines[0] + textwrap.dedent("".join(lines[1:]))
>>
>>    What do you think?
>>
>
> I try not to on most days.

I always say "me thinking will cost you extra" ;)

> I'll check it out, though in practice the generated documents were already
> identical, so... I'll try yours and verify that's still true. If so, sure!
>
>
>> 2. The new one is applied exactly when the old one was.
>>
>>    The old one is applied to the sections starting with @FOO: and the
>>    tagged sections (Returns:, Errors:, ...).
>>
>>    The new one is applied in ._nodes_for_members(),
>>    ._nodes_for_enum_values(), _nodes_for_features(), and
>>    ._nodes_for_sections().
>>
>>    It is not applied to the text of untagged sections, including the
>>    body section.
>>
>>    Good.

Thanks!

[...]

Re: [PATCH 03/20] docs/qapidoc: delint a tiny portion of the module

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 4876 bytes --]

On Wed, May 15, 2024 at 5:17 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > In the coming patches, it's helpful to have a linting baseline. However,
> > there's no need to shuffle around the deck chairs too much, because most
> > of this code will be removed once the new qapidoc generator (the
> > "transmogrifier") is in place.
> >
> > To ease my pain: just turn off the black auto-formatter for most, but
> > not all, of qapidoc.py. This will help ensure that *new* code follows a
> > coding standard without bothering too much with cleaning up the existing
> > code.
> >
> > For manual checking for now, try "black --check qapidoc.py" from the
> > docs/sphinx directory. "pip install black" (without root permissions) if
> > you do not have it installed otherwise.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  docs/sphinx/qapidoc.py | 16 +++++++++-------
> >  1 file changed, 9 insertions(+), 7 deletions(-)
> >
> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> > index f270b494f01..1655682d4c7 100644
> > --- a/docs/sphinx/qapidoc.py
> > +++ b/docs/sphinx/qapidoc.py
> > @@ -28,28 +28,30 @@
> >  import re
> >
> >  from docutils import nodes
> > +from docutils.parsers.rst import Directive, directives
> >  from docutils.statemachine import ViewList
> > -from docutils.parsers.rst import directives, Directive
> > -from sphinx.errors import ExtensionError
> > -from sphinx.util.nodes import nested_parse_with_titles
> > -import sphinx
> > -from qapi.gen import QAPISchemaVisitor
> >  from qapi.error import QAPIError, QAPISemError
> > +from qapi.gen import QAPISchemaVisitor
> >  from qapi.schema import QAPISchema
> >
> > +import sphinx
> > +from sphinx.errors import ExtensionError
> > +from sphinx.util.nodes import nested_parse_with_titles
> > +
>
> Exchanges old pylint gripe
>
>     docs/sphinx/qapidoc.py:45:4: C0412: Imports from package sphinx are
> not grouped (ungrouped-imports)
>
> for new gripes
>
>     docs/sphinx/qapidoc.py:37:0: C0411: third party import "import sphinx"
> should be placed before "from qapi.error import QAPIError, QAPISemError"
> (wrong-import-order)
>     docs/sphinx/qapidoc.py:38:0: C0411: third party import "from
> sphinx.errors import ExtensionError" should be placed before "from
> qapi.error import QAPIError, QAPISemError" (wrong-import-order)
>     docs/sphinx/qapidoc.py:39:0: C0411: third party import "from
> sphinx.util.nodes import nested_parse_with_titles" should be placed before
> "from qapi.error import QAPIError, QAPISemError" (wrong-import-order)
>
> Easy enough to fix.
>

I believe these errors are caused by the fact that the tools are confused
about the "sphinx" namespace - some interpret them as being the local
"module", the docs/sphinx/ directory, and others believe them to be the
third party external package.

I have not been using pylint on docs/sphinx/ files because of the
difficulty of managing imports - this environment is generally beyond the
reaches of my python borgcube and at present I don't have plans to
integrate it.

At the moment, I am using black, isort and flake8 for qapidoc.py and
they're happy with it. I am not using mypy because I never did the typing
boogaloo with qapidoc.py and I won't be bothering - except for any new code
I write, which *will* bother. By the end of the new transmogrifier,
qapidoc.py *will* strictly typecheck.

pylint may prove to be an issue with the imports, though. isort also seems
to misunderstand "sphinx, the stuff in this folder" and "sphinx, the stuff
in a third party package" and so I'm afraid I don't have any good ability
to get pylint to play along, here.

Pleading for "Sorry, this sucks and I can't figure out how to solve it
quickly". Maybe a future project, apologies.


> >
> >  # Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
> >  # use switch_source_input. Check borrowed from kerneldoc.py.
> > -Use_SSI = sphinx.__version__[:3] >= '1.7'
> > +Use_SSI = sphinx.__version__[:3] >= "1.7"
> >  if Use_SSI:
> >      from sphinx.util.docutils import switch_source_input
> >  else:
> >      from sphinx.ext.autodoc import AutodocReporter
> >
> >
> > -__version__ = '1.0'
> > +__version__ = "1.0"
> >
> >
> > +# fmt: off
>
> I figure this tells black to keep quiet for the remainder of the file.
> Worth a comment, I think.
>
> >  # Function borrowed from pydash, which is under the MIT license
> >  def intersperse(iterable, separator):
> >      """Yield the members of *iterable* interspersed with *separator*."""
>
> With my comments addressed
> Reviewed-by: Markus Armbruster <armbru@redhat.com>
>

^ Dropping this unless you're okay with the weird import orders owing to
the strange import paradigm in the sphinx folder.r

[-- Attachment #2: Type: text/html, Size: 6257 bytes --]

Re: [PATCH 04/20] qapi/parser: preserve indentation in QAPIDoc sections

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 16751 bytes --]

On Wed, May 15, 2024 at 10:18 AM Markus Armbruster <armbru@redhat.com>
wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > On Wed, May 15, 2024, 7:50 AM Markus Armbruster <armbru@redhat.com>
> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > Prior to this patch, a section like this:
> >> >
> >> > @name: lorem ipsum
> >> >    dolor sit amet
> >> >      consectetur adipiscing elit
> >> >
> >> > would be parsed as:
> >> >
> >> > "lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"
> >> >
> >> > We want to preserve the indentation for even the first body line so
> that
> >> > the entire block can be parsed directly as rST. This patch would now
> >> > parse that segment as:
> >> >
> >> > "lorem ipsum\n   dolor sit amet\n     consectetur adipiscing elit"
> >>
> >> I'm afraid it's less than clear *why* we want to parse the entire block
> >> directly as rST.  I have just enough insight into what you've built on
> >> top of this series to hazard a guess.  Bear with me while I try to
> >> explain it.
> >>
> >
> > My own summary: qapidoc expects a paragraph, the new generator expects a
> > block.
> >
> >
> >> We first parse the doc comment with parser.py into an internal
> >> representation.  The structural parts become objects, and the remainder
> >> becomes text attributes of these objects.  Currently, parser.py
> >> carefully adjusts indentation for these text attributes.  Why?  I'll get
> >> to that.
> >>
> >> For your example, parser.py creates an ArgSection object, and sets its
> >> @text member to
> >>
> >>     "lorem ipsum\ndolor sit amet\n  consectetur adipiscing elit"
> >>
> >> Printing this string gives us
> >>
> >>     lorem ipsum
> >>     dolor sit amet
> >>       consectetur adipiscing elit
> >>
> >> qapidoc.py then transforms parser.py's IR into Sphinx IR.  The objects
> >> become (more complicated) Sphinx objects, and their text attributes get
> >> re-parsed as rST text into still more Sphinx objects.
> >>
> >> This re-parse rejects your example with "Unexpected indentation."
> >>
> >
> > Specifically, it'd be an unexpected *unindent*; the indent lacking on the
> > first *two* lines is the problem.
> >
> >
> >> Let me use a slightly different one:
> >>
> >>     # @name: lorem ipsum
> >>     #    dolor sit amet
> >>     #    consectetur adipiscing elit
> >>
> >> Results in this @text member
> >>
> >>     lorem ipsum
> >>     dolor sit amet
> >>     consectetur adipiscing elit
> >>
> >> which is re-parsed as paragraph, i.e. exactly what we want.
> >>
> >
> > It's what we used to want, anyway.
>
> Yes, I'm describing the current state here.
>
> >> > This understandably breaks qapidoc.py;
> >>
> >> Without indentation adjustment, we'd get
> >>
> >>     lorem ipsum
> >>        dolor sit amet
> >>        consectetur adipiscing elit
> >>
> >> which would be re-parsed as a definition list, I guess.  This isn't what
> >> we want.
> >>
> >> >                                        so a new function is added
> there
> >> > to re-dedent the text.
> >>
> >> Your patch moves the indentation adjustment to another place.  No
> >> functional change.
> >>
> >> You move it so you can branch off your new rendering pipeline before the
> >> indentation adjustment, because ...
> >>
> >> >                        Once the new generator is merged, this function
> >> > will not be needed any longer and can be dropped.
> >>
> >> ... yours doesn't want it.
> >>
> >> I believe it doesn't want it, because it generates rST (with a QAPI
> >> extension) instead of Sphinx objects.  For your example, something like
> >>
> >>     :arg name: lorem ipsum
> >>        dolor sit amet
> >>          consectetur adipiscing elit
> >>
> >> For mine:
> >>
> >>     :arg name: lorem ipsum
> >>        dolor sit amet
> >>        consectetur adipiscing elit
> >>
> >> Fair?
> >>
> >
> > Not quite;
> >
> > Old parsing, new generator:
> >
> > :arg type name: lorem ipsum
> > dolor sit amet
> >   consectetur apidiscing elit
> >
> > This is wrong - continuations of a field list must be indented. Unlike
> > paragraphs, we want indents to "keep the block".
> >
> > New parsing, new generator:
> >
> > :arg type name: lorem ipsum
> >    dolor sit amet
> >      consectetur apidiscing elit
> >
> > indent is preserved, maintaining the block-level element.
> >
> > I don't have to re-add indents and any nested block elements will be
> > preserved correctly. i.e. you can use code examples, nested lists, etc.
> in
> > argument definitions.
> >
> > The goal here was "Do not treat this as a paragraph, treat it directly as
> > rST and do not modify it needlessly."
> >
> > It's a lot simpler than trying to manage the indent and injecting spaces
> > manually - and adding a temporary dedent to scheduled-for-demolition code
> > seemed the nicer place to add the hack.
>
> Understand.
>
> A bit more rationale in the commit message would be nice.  Perhaps start
> with current state ("we deintent"), then describe the patch ("move the
> deindent"), then rationale "to get it out of the way of a new thingy I
> wrote, and intend to post soonish", then "which will replace qapidoc.py
> entirely".
>

Updated with more info than you require. I hear that American third graders
get a free trip to Pizza Hut if they read at least 20 of my commit messages
in a school year.


>
> >> The transition from the doc comment to (extended) rST is straightforward
> >> for these examples.
> >>
> >> I hope it'll be as straightforward (and thus predictable) in other
> >> cases, too.
> >>
> >> > (I verified this patch changes absolutely nothing by comparing the
> >> > md5sums of the QMP ref html pages both before and after the change, so
> >> > it's certified inert. QAPI test output has been updated to reflect the
> >> > new strategy of preserving indents for rST.)
> >> >
> >> > Signed-off-by: John Snow <jsnow@redhat.com>
> >> > ---
> >> >  docs/sphinx/qapidoc.py         | 36
> +++++++++++++++++++++++++++++-----
> >> >  scripts/qapi/parser.py         |  8 ++++++--
> >> >  tests/qapi-schema/doc-good.out | 32 +++++++++++++++---------------
> >> >  3 files changed, 53 insertions(+), 23 deletions(-)
> >> >
> >> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> >> > index 1655682d4c7..2e3ffcbafb7 100644
> >> > --- a/docs/sphinx/qapidoc.py
> >> > +++ b/docs/sphinx/qapidoc.py
> >> > @@ -26,6 +26,7 @@
> >> >
> >> >  import os
> >> >  import re
> >> > +import textwrap
> >> >
> >> >  from docutils import nodes
> >> >  from docutils.parsers.rst import Directive, directives
> >> > @@ -51,6 +52,28 @@
> >> >  __version__ = "1.0"
> >> >
> >> >
> >> > +def dedent(text: str) -> str:
> >> > +    # Temporary: In service of the new QAPI domain, the QAPI doc
> parser
> >> > +    # now preserves indents in args/members/features text. QAPIDoc
> does
> >> > +    # not handle this well, so undo that change here.
> >> > +
> >> > +    # QAPIDoc is being rewritten and will be replaced soon,
> >> > +    # but this function is here in the interim as transition glue.
> >>
> >> I'm not sure we need the comment.
> >
> > OK. Guess I'd rather overcomment than undercomment... easier to delete
> than
> > add :)
>
> Right :)
>
> >> > +
> >> > +    lines = text.splitlines(True)
> >> > +    if len(lines) > 1:
> >> > +        if re.match(r"\s+", lines[0]):
> >> > +            # First line is indented; description started on
> >> > +            # the line after the name. dedent the whole block.
> >> > +            return textwrap.dedent(text)
> >> > +        else:
> >>
> >> pylint gripes
> >>
> >>     docs/sphinx/qapidoc.py:65:8: R1705: Unnecessary "else" after
> "return",
> >> remove the "else" and de-indent the code inside it (no-else-return)
> >>
> >
> > Interesting. What pylint version?
>
> Fedora 39
>
> $ pylint --version
> pylint 3.0.4
> astroid 3.0.3
> Python 3.12.3 (main, Apr 17 2024, 00:00:00) [GCC 13.2.1 20240316 (Red Hat
> 13.2.1-7)]
>

Realized from the prior patch I was not actually running pylint on this
file, for good reason. Nonetheless, I did fix this particular gripe by
using your rewrite.

My current best attempt at reducing the noise:

*from inside the docs/sphinx directory*:

> PYTHONPATH=/home/jsnow/src/qemu/scripts/ pylint --rc-file
../../scripts/qapi/pylintrc qapidoc.py
************* Module qapidoc
qapidoc.py:45:0: C0103: Constant name "Use_SSI" doesn't conform to
UPPER_CASE naming style (invalid-name)
qapidoc.py:49:4: E0611: No name 'AutodocReporter' in module
'sphinx.ext.autodoc' (no-name-in-module)
qapidoc.py:77:10: R1708: Do not raise StopIteration in generator, use
return statement instead (stop-iteration-return)
qapidoc.py:567:11: R1735: Consider using '{"version": __version__,
"parallel_read_safe": True, "parallel_write_safe": True, ... }' instead of
a call to 'dict'. (use-dict-literal)

 Hm, actually, maybe this is tractable...


> >> > +            # Descr started on same line. Dedent line 2+.
> >> > +            return lines[0] + textwrap.dedent("".join(lines[1:]))
> >> > +    else:
> >> > +        # Descr was a single line; dedent entire line.
> >> > +        return textwrap.dedent(text)
> >> > +
> >> > +
> >> >  # fmt: off
> >> >  # Function borrowed from pydash, which is under the MIT license
> >> >  def intersperse(iterable, separator):
> >> > @@ -169,7 +192,7 @@ def _nodes_for_members(self, doc, what,
> base=None, branches=None):
> >> >              term = self._nodes_for_one_member(section.member)
> >> >              # TODO drop fallbacks when undocumented members are
> outlawed
> >> >              if section.text:
> >> > -                defn = section.text
> >> > +                defn = dedent(section.text)
> >> >              else:
> >> >                  defn = [nodes.Text('Not documented')]
> >> >
> >> > @@ -207,7 +230,7 @@ def _nodes_for_enum_values(self, doc):
> >> >
> termtext.extend(self._nodes_for_ifcond(section.member.ifcond))
> >> >              # TODO drop fallbacks when undocumented members are
> outlawed
> >> >              if section.text:
> >> > -                defn = section.text
> >> > +                defn = dedent(section.text)
> >> >              else:
> >> >                  defn = [nodes.Text('Not documented')]
> >> >
> >> > @@ -242,7 +265,7 @@ def _nodes_for_features(self, doc):
> >> >          dlnode = nodes.definition_list()
> >> >          for section in doc.features.values():
> >> >              dlnode += self._make_dlitem(
> >> > -                [nodes.literal('', section.member.name)],
> section.text)
> >> > +                [nodes.literal('', section.member.name)],
> dedent(section.text))
> >> >              seen_item = True
> >> >
> >> >          if not seen_item:
> >> > @@ -265,9 +288,12 @@ def _nodes_for_sections(self, doc):
> >> >                  continue
> >> >              snode = self._make_section(section.tag)
> >> >              if section.tag and section.tag.startswith('Example'):
> >> > -                snode += self._nodes_for_example(section.text)
> >> > +                snode +=
> self._nodes_for_example(dedent(section.text))
> >> >              else:
> >> > -                self._parse_text_into_node(section.text, snode)
> >> > +                self._parse_text_into_node(
> >> > +                    dedent(section.text) if section.tag else
> section.text,
> >> > +                    snode,
> >> > +                )
> >> >              nodelist.append(snode)
> >> >          return nodelist
> >> >
> >> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> >> > index 7b13a583ac1..8cdd5334ec6 100644
> >> > --- a/scripts/qapi/parser.py
> >> > +++ b/scripts/qapi/parser.py
> >> > @@ -448,7 +448,10 @@ def get_doc_indented(self, doc: 'QAPIDoc') ->
> Optional[str]:
> >> >          indent = must_match(r'\s*', line).end()
> >> >          if not indent:
> >> >              return line
> >> > -        doc.append_line(line[indent:])
> >> > +
> >> > +        # Preserve the indent, it's needed for rST formatting.
> >>
> >> I'm not sure we need the comment.
> >>
> >
> > OK.
> >
> >
> >> > +        doc.append_line(line)
> >> > +
> >> >          prev_line_blank = False
> >> >          while True:
> >> >              self.accept(False)
> >> > @@ -465,7 +468,8 @@ def get_doc_indented(self, doc: 'QAPIDoc') ->
> Optional[str]:
> >> >                      self,
> >> >                      "unexpected de-indent (expected at least %d
> spaces)" %
> >> >                      indent)
> >> > -            doc.append_line(line[indent:])
> >> > +            # Again, preserve the indent for ReST.
> >>
> >> Likewise.
> >>
> >> If you want to document the fact that .get_doc_indented() preserves
> >> indentation, a function comment or doc string would the proper place.
> >>
> >
> > Got it.
> >
> >
> >> > +            doc.append_line(line)
> >> >              prev_line_blank = True
> >> >
> >> >      def get_doc_paragraph(self, doc: 'QAPIDoc') -> Optional[str]:
> >>
> >> Correctness argument:
> >>
> >> 0. This patch merely moves an indentation adjustment from parser.py to
> >>    qapidoc.py.
> >>
> >>    Checks out: you remove indentation adjustment in parser.py, add it in
> >>    qapidoc.py, and that's it.
> >>
> >> 1. The new indentation adjuster in qapidoc.py behaves just like the old
> >>    one in parser.py did.
> >>
> >>    parser.py's .get_doc_indented() is used to parse what follows certain
> >>    sections' first line.  It's users store the text on this first line,
> >>    if any, with leading whitespace stripped, then call
> >>    .get_doc_indented() to eat the section's remaining lines.  All
> >>    non-blank lines eaten must be indented at least as much as the first
> >>    non-blank line.  .get_doc_indented() appends the lines eaten to the
> >>    stored text with the first non-blank line's indentation stripped from
> >>    all the non-blank lines.
> >>
> >>    Your patch drops this stripping of indentation from non-first lines.
> >>    This is what must now be done in qapidoc.py.
> >>
> >>    If the section's first line has no text, the first non-blank line's
> >>    indentation must be stripped from all non-blank lines.
> >>
> >>    If the section's first line has text, the next non-blank line's
> >>    indentation must be stripped from all lines but the first.
> >>
> >>    How can qapidoc.py detect whether the section's first line has text?
> >>    Fortunately, such a line will be unindented, i.e. start with a
> >>    non-blank character, and all remaining lines will be blank or
> >>    indented.
> >>
> >>    qapidoc.py's dedent() seems to remove indentation common to all
> >>    non-blank lines.  Except when there are multiple lines, and the first
> >>    one is not indented, it removes common indentation from the remaining
> >>    lines.
> >>
> >>    Common indentation works, because all lines are indented at least as
> >>    much as the one whose indentation we want to remove.
> >>
> >>    The conditional doesn't quite match the "if the section's first line"
> >>    above.  This simpler dedent() does, and also works in my testing:
> >>
> >> def dedent(text: str) -> str:
> >>     lines = text.splitlines(True)
> >>     if re.match(r"\s+", lines[0]):
> >>         # First line is indented; description started on
> >>         # the line after the name. dedent the whole block.
> >>         return textwrap.dedent(text)
> >>     # Descr started on same line. Dedent line 2+.
> >>     return lines[0] + textwrap.dedent("".join(lines[1:]))
> >>
> >>    What do you think?
> >>
> >
> > I try not to on most days.
>
> I always say "me thinking will cost you extra" ;)
>
> > I'll check it out, though in practice the generated documents were
> already
> > identical, so... I'll try yours and verify that's still true. If so,
> sure!
>

Yup, it checks out and the md5sums are still the same. Good enough for me.


> >
> >
> >> 2. The new one is applied exactly when the old one was.
> >>
> >>    The old one is applied to the sections starting with @FOO: and the
> >>    tagged sections (Returns:, Errors:, ...).
> >>
> >>    The new one is applied in ._nodes_for_members(),
> >>    ._nodes_for_enum_values(), _nodes_for_features(), and
> >>    ._nodes_for_sections().
> >>
> >>    It is not applied to the text of untagged sections, including the
> >>    body section.
> >>
> >>    Good.
>
> Thanks!
>
> [...]
>
>

[-- Attachment #2: Type: text/html, Size: 22539 bytes --]

Re: [PATCH 03/20] docs/qapidoc: delint a tiny portion of the module

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, May 15, 2024 at 5:17 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > In the coming patches, it's helpful to have a linting baseline. However,
>> > there's no need to shuffle around the deck chairs too much, because most
>> > of this code will be removed once the new qapidoc generator (the
>> > "transmogrifier") is in place.
>> >
>> > To ease my pain: just turn off the black auto-formatter for most, but
>> > not all, of qapidoc.py. This will help ensure that *new* code follows a
>> > coding standard without bothering too much with cleaning up the existing
>> > code.
>> >
>> > For manual checking for now, try "black --check qapidoc.py" from the
>> > docs/sphinx directory. "pip install black" (without root permissions) if
>> > you do not have it installed otherwise.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  docs/sphinx/qapidoc.py | 16 +++++++++-------
>> >  1 file changed, 9 insertions(+), 7 deletions(-)
>> >
>> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
>> > index f270b494f01..1655682d4c7 100644
>> > --- a/docs/sphinx/qapidoc.py
>> > +++ b/docs/sphinx/qapidoc.py
>> > @@ -28,28 +28,30 @@
>> >  import re
>> >
>> >  from docutils import nodes
>> > +from docutils.parsers.rst import Directive, directives
>> >  from docutils.statemachine import ViewList
>> > -from docutils.parsers.rst import directives, Directive
>> > -from sphinx.errors import ExtensionError
>> > -from sphinx.util.nodes import nested_parse_with_titles
>> > -import sphinx
>> > -from qapi.gen import QAPISchemaVisitor
>> >  from qapi.error import QAPIError, QAPISemError
>> > +from qapi.gen import QAPISchemaVisitor
>> >  from qapi.schema import QAPISchema
>> >
>> > +import sphinx
>> > +from sphinx.errors import ExtensionError
>> > +from sphinx.util.nodes import nested_parse_with_titles
>> > +
>>
>> Exchanges old pylint gripe
>>
>>     docs/sphinx/qapidoc.py:45:4: C0412: Imports from package sphinx are
>> not grouped (ungrouped-imports)
>>
>> for new gripes
>>
>>     docs/sphinx/qapidoc.py:37:0: C0411: third party import "import sphinx"
>> should be placed before "from qapi.error import QAPIError, QAPISemError"
>> (wrong-import-order)
>>     docs/sphinx/qapidoc.py:38:0: C0411: third party import "from
>> sphinx.errors import ExtensionError" should be placed before "from
>> qapi.error import QAPIError, QAPISemError" (wrong-import-order)
>>     docs/sphinx/qapidoc.py:39:0: C0411: third party import "from
>> sphinx.util.nodes import nested_parse_with_titles" should be placed before
>> "from qapi.error import QAPIError, QAPISemError" (wrong-import-order)
>>
>> Easy enough to fix.
>>
>
> I believe these errors are caused by the fact that the tools are confused
> about the "sphinx" namespace - some interpret them as being the local
> "module", the docs/sphinx/ directory, and others believe them to be the
> third party external package.
>
> I have not been using pylint on docs/sphinx/ files because of the
> difficulty of managing imports - this environment is generally beyond the
> reaches of my python borgcube and at present I don't have plans to
> integrate it.
>
> At the moment, I am using black, isort and flake8 for qapidoc.py and
> they're happy with it. I am not using mypy because I never did the typing
> boogaloo with qapidoc.py and I won't be bothering - except for any new code
> I write, which *will* bother. By the end of the new transmogrifier,
> qapidoc.py *will* strictly typecheck.
>
> pylint may prove to be an issue with the imports, though. isort also seems
> to misunderstand "sphinx, the stuff in this folder" and "sphinx, the stuff
> in a third party package" and so I'm afraid I don't have any good ability
> to get pylint to play along, here.
>
> Pleading for "Sorry, this sucks and I can't figure out how to solve it
> quickly". Maybe a future project, apologies.

Is this pain we inflict on ourselves by naming the directory "sphinx"?

>> >
>> >  # Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
>> >  # use switch_source_input. Check borrowed from kerneldoc.py.
>> > -Use_SSI = sphinx.__version__[:3] >= '1.7'
>> > +Use_SSI = sphinx.__version__[:3] >= "1.7"
>> >  if Use_SSI:
>> >      from sphinx.util.docutils import switch_source_input
>> >  else:
>> >      from sphinx.ext.autodoc import AutodocReporter
>> >
>> >
>> > -__version__ = '1.0'
>> > +__version__ = "1.0"
>> >
>> >
>> > +# fmt: off
>>
>> I figure this tells black to keep quiet for the remainder of the file.
>> Worth a comment, I think.
>>
>> >  # Function borrowed from pydash, which is under the MIT license
>> >  def intersperse(iterable, separator):
>> >      """Yield the members of *iterable* interspersed with *separator*."""
>>
>> With my comments addressed
>> Reviewed-by: Markus Armbruster <armbru@redhat.com>
>>
>
> ^ Dropping this unless you're okay with the weird import orders owing to
> the strange import paradigm in the sphinx folder.r

Feel free to keep it.

Re: [PATCH 03/20] docs/qapidoc: delint a tiny portion of the module

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 6047 bytes --]

On Wed, May 15, 2024, 1:27 PM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > On Wed, May 15, 2024 at 5:17 AM Markus Armbruster <armbru@redhat.com>
> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > In the coming patches, it's helpful to have a linting baseline.
> However,
> >> > there's no need to shuffle around the deck chairs too much, because
> most
> >> > of this code will be removed once the new qapidoc generator (the
> >> > "transmogrifier") is in place.
> >> >
> >> > To ease my pain: just turn off the black auto-formatter for most, but
> >> > not all, of qapidoc.py. This will help ensure that *new* code follows
> a
> >> > coding standard without bothering too much with cleaning up the
> existing
> >> > code.
> >> >
> >> > For manual checking for now, try "black --check qapidoc.py" from the
> >> > docs/sphinx directory. "pip install black" (without root permissions)
> if
> >> > you do not have it installed otherwise.
> >> >
> >> > Signed-off-by: John Snow <jsnow@redhat.com>
> >> > ---
> >> >  docs/sphinx/qapidoc.py | 16 +++++++++-------
> >> >  1 file changed, 9 insertions(+), 7 deletions(-)
> >> >
> >> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> >> > index f270b494f01..1655682d4c7 100644
> >> > --- a/docs/sphinx/qapidoc.py
> >> > +++ b/docs/sphinx/qapidoc.py
> >> > @@ -28,28 +28,30 @@
> >> >  import re
> >> >
> >> >  from docutils import nodes
> >> > +from docutils.parsers.rst import Directive, directives
> >> >  from docutils.statemachine import ViewList
> >> > -from docutils.parsers.rst import directives, Directive
> >> > -from sphinx.errors import ExtensionError
> >> > -from sphinx.util.nodes import nested_parse_with_titles
> >> > -import sphinx
> >> > -from qapi.gen import QAPISchemaVisitor
> >> >  from qapi.error import QAPIError, QAPISemError
> >> > +from qapi.gen import QAPISchemaVisitor
> >> >  from qapi.schema import QAPISchema
> >> >
> >> > +import sphinx
> >> > +from sphinx.errors import ExtensionError
> >> > +from sphinx.util.nodes import nested_parse_with_titles
> >> > +
> >>
> >> Exchanges old pylint gripe
> >>
> >>     docs/sphinx/qapidoc.py:45:4: C0412: Imports from package sphinx are
> >> not grouped (ungrouped-imports)
> >>
> >> for new gripes
> >>
> >>     docs/sphinx/qapidoc.py:37:0: C0411: third party import "import
> sphinx"
> >> should be placed before "from qapi.error import QAPIError, QAPISemError"
> >> (wrong-import-order)
> >>     docs/sphinx/qapidoc.py:38:0: C0411: third party import "from
> >> sphinx.errors import ExtensionError" should be placed before "from
> >> qapi.error import QAPIError, QAPISemError" (wrong-import-order)
> >>     docs/sphinx/qapidoc.py:39:0: C0411: third party import "from
> >> sphinx.util.nodes import nested_parse_with_titles" should be placed
> before
> >> "from qapi.error import QAPIError, QAPISemError" (wrong-import-order)
> >>
> >> Easy enough to fix.
> >>
> >
> > I believe these errors are caused by the fact that the tools are confused
> > about the "sphinx" namespace - some interpret them as being the local
> > "module", the docs/sphinx/ directory, and others believe them to be the
> > third party external package.
> >
> > I have not been using pylint on docs/sphinx/ files because of the
> > difficulty of managing imports - this environment is generally beyond the
> > reaches of my python borgcube and at present I don't have plans to
> > integrate it.
> >
> > At the moment, I am using black, isort and flake8 for qapidoc.py and
> > they're happy with it. I am not using mypy because I never did the typing
> > boogaloo with qapidoc.py and I won't be bothering - except for any new
> code
> > I write, which *will* bother. By the end of the new transmogrifier,
> > qapidoc.py *will* strictly typecheck.
> >
> > pylint may prove to be an issue with the imports, though. isort also
> seems
> > to misunderstand "sphinx, the stuff in this folder" and "sphinx, the
> stuff
> > in a third party package" and so I'm afraid I don't have any good ability
> > to get pylint to play along, here.
> >
> > Pleading for "Sorry, this sucks and I can't figure out how to solve it
> > quickly". Maybe a future project, apologies.
>
> Is this pain we inflict on ourselves by naming the directory "sphinx"?
>

More or less, yeah. If you check the file from a CWD where there is no
"sphinx" directory, it behaves more normally.

Just not worth renaming it and futzing about for now. However, I did get an
invocation that lets me get a clean pylint run by abusing PYTHONPATH again,
so I have at least one standard baseline we can count on. I updated the
do-not-merge patch to include the special magick incantations.

Maybe in the future I'll make a qemu.plugins submodule instead, but that's
for quite a bit later.


> >> >
> >> >  # Sphinx up to 1.6 uses AutodocReporter; 1.7 and later
> >> >  # use switch_source_input. Check borrowed from kerneldoc.py.
> >> > -Use_SSI = sphinx.__version__[:3] >= '1.7'
> >> > +Use_SSI = sphinx.__version__[:3] >= "1.7"
> >> >  if Use_SSI:
> >> >      from sphinx.util.docutils import switch_source_input
> >> >  else:
> >> >      from sphinx.ext.autodoc import AutodocReporter
> >> >
> >> >
> >> > -__version__ = '1.0'
> >> > +__version__ = "1.0"
> >> >
> >> >
> >> > +# fmt: off
> >>
> >> I figure this tells black to keep quiet for the remainder of the file.
> >> Worth a comment, I think.
> >>
> >> >  # Function borrowed from pydash, which is under the MIT license
> >> >  def intersperse(iterable, separator):
> >> >      """Yield the members of *iterable* interspersed with
> *separator*."""
> >>
> >> With my comments addressed
> >> Reviewed-by: Markus Armbruster <armbru@redhat.com>
> >>
> >
> > ^ Dropping this unless you're okay with the weird import orders owing to
> > the strange import paradigm in the sphinx folder.r
>
> Feel free to keep it.
>
>

[-- Attachment #2: Type: text/html, Size: 8461 bytes --]

Re: [PATCH 05/20] qapi/parser: adjust info location for doc body section

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Instead of using the info object for the doc block as a whole, update
> the info pointer for each call to ensure_untagged_section when the
> existing section is otherwise empty. This way, Sphinx error information
> will match precisely to where the text actually starts.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/parser.py | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index 8cdd5334ec6..41b9319e5cb 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -662,8 +662,13 @@ def end(self) -> None:
>  
>      def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
>          if self.all_sections and not self.all_sections[-1].tag:
> -            # extend current section
> -            self.all_sections[-1].text += '\n'

Before, we always append a newline.

> +            section = self.all_sections[-1]
> +            # Section is empty so far; update info to start *here*.
> +            if not section.text:
> +                section.info = info
> +            else:
> +                # extend current section
> +                self.all_sections[-1].text += '\n'

Afterwards, we append it only when the section already has some text.

The commit message claims the patch only adjusts section.info.  That's a
lie :)

I believe the change makes no difference because .end() strips leading
and trailing newline.

>              return
>          # start new section
>          section = self.Section(info)

You could fix the commit message, but I think backing out the
no-difference change is easier.  The appended patch works in my testing.

Next one.  Your patch changes the meaning of section.info.  Here's its
initialization:

    class Section:
        # pylint: disable=too-few-public-methods
        def __init__(self, info: QAPISourceInfo,
                     tag: Optional[str] = None):
--->        # section source info, i.e. where it begins
            self.info = info
            # section tag, if any ('Returns', '@name', ...)
            self.tag = tag
            # section text without tag
            self.text = ''

The comment is now wrong.  Calls for a thorough review of .info's uses.

The alternative to changing .info's meaning is to add another member
with the meaning you need.  Then we have to review .info's uses to find
out which ones to switch to the new one.

Left for later.


diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 8cdd5334ec..abeae1ca77 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -663,7 +663,10 @@ def end(self) -> None:
     def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
         if self.all_sections and not self.all_sections[-1].tag:
             # extend current section
-            self.all_sections[-1].text += '\n'
+            section = self.all_sections[-1]
+            if not section.text:
+                section.info = info
+            section.text += '\n'
             return
         # start new section
         section = self.Section(info)

Re: [PATCH 06/20] qapi/parser: fix comment parsing immediately following a doc block

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> If a comment immediately follows a doc block, the parser doesn't ignore
> that token appropriately. Fix that.

Reproducer?

>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/parser.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index 41b9319e5cb..161768b8b96 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -587,7 +587,7 @@ def get_doc(self) -> 'QAPIDoc':
>                  line = self.get_doc_line()
>                  first = False
>  
> -        self.accept(False)
> +        self.accept()
>          doc.end()
>          return doc

Can't judge the fix without understanding the problem, and the problem
will be easier to understand for me with a reproducer.

Re: [PATCH 07/20] qapi/parser: add semantic 'kind' parameter to QAPIDoc.Section

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> When iterating all_sections, this is helpful to be able to distinguish
> "members" from "features"; the only other way to do so is to
> cross-reference these sections against QAPIDoc.args or QAPIDoc.features,
> but if the desired end goal for QAPIDoc is to remove everything except
> all_sections, we need *something* accessible to distinguish them.
>
> To keep types simple, add this semantic parameter to the base Section
> and not just ArgSection; we can use this to filter out paragraphs and
> tagged sections, too.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/parser.py | 25 ++++++++++++++++---------
>  1 file changed, 16 insertions(+), 9 deletions(-)
>
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index 161768b8b96..cf4cbca1c1f 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -613,21 +613,27 @@ class QAPIDoc:
>  
>      class Section:
>          # pylint: disable=too-few-public-methods
> -        def __init__(self, info: QAPISourceInfo,
> -                     tag: Optional[str] = None):
> +        def __init__(
> +            self,
> +            info: QAPISourceInfo,
> +            tag: Optional[str] = None,
> +            kind: str = 'paragraph',
> +        ):
>              # section source info, i.e. where it begins
>              self.info = info
>              # section tag, if any ('Returns', '@name', ...)
>              self.tag = tag
>              # section text without tag
>              self.text = ''
> +            # section type - {paragraph, feature, member, tagged}
> +            self.kind = kind

Hmm.  .kind is almost redundant with .tag.

Untagged section:    .kind is 'paragraph', .tag is None

Member description:  .kind is 'member', .tag matches @NAME

Feature description: .kind is 'feature', .tag matches @NAME

Tagged section:      .kind is 'tagged', .tag matches
                          r'Returns|Errors|Since|Notes?|Examples?|TODO'

.kind can directly be derived from .tag except for member and feature
descriptions.  And you want to tell these two apart in a straightforward
manner in later patches, as you explain in your commit message.

If .kind is 'member' or 'feature', then self must be an ArgSection.
Worth a comment?  An assertion?

Some time back, I considered changing .tag for member and feature
descriptions to suitable strings, like your 'member' and 'feature', and
move the member / feature name into ArgSection.  I didn't, because the
benefit wasn't worth the churn at the time.  Perhaps it's worth it now.
Would it result in simpler code than your solution?

Terminology nit: the section you call 'paragraph' isn't actually a
paragraph: it could be several paragraphs.  Best to call it 'untagged',
as in .ensure_untagged_section().

>  
>          def append_line(self, line: str) -> None:
>              self.text += line + '\n'
>  
>      class ArgSection(Section):
> -        def __init__(self, info: QAPISourceInfo, tag: str):
> -            super().__init__(info, tag)
> +        def __init__(self, info: QAPISourceInfo, tag: str, kind: str):
> +            super().__init__(info, tag, kind)
>              self.member: Optional['QAPISchemaMember'] = None
>  
>          def connect(self, member: 'QAPISchemaMember') -> None:

[...]

Re: [PATCH 08/20] qapi/parser: differentiate intro and outro paragraphs

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Add a semantic tag to paragraphs that appear *before* tagged
> sections/members/features and those that appear after. This will control
> how they are inlined when doc sections are merged and flattened.

This future use is not obvious to me now.  I guess the effective way to
help me see it is actual patches, which will come in due time.

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/parser.py | 22 +++++++++++++++++-----
>  1 file changed, 17 insertions(+), 5 deletions(-)
>
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index cf4cbca1c1f..b1794f71e12 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -503,6 +503,10 @@ def get_doc(self) -> 'QAPIDoc':
>              self.accept(False)
>              line = self.get_doc_line()
>              no_more_args = False
> +            # Paragraphs before members/features/tagged are "intro" paragraphs.
> +            # Any appearing subsequently are "outro" paragraphs.
> +            # This is only semantic metadata for the doc generator.

Not sure about the last sentence.  Isn't it true for almost everything
around here?

Also, long line.  

> +            intro = True
>  
>              while line is not None:
>                  # Blank lines
> @@ -532,6 +536,7 @@ def get_doc(self) -> 'QAPIDoc':
>                          raise QAPIParseError(
>                              self, 'feature descriptions expected')
>                      no_more_args = True
> +                    intro = False

After feature descriptions.

>                  elif match := self._match_at_name_colon(line):
>                      # description
>                      if no_more_args:
> @@ -547,6 +552,7 @@ def get_doc(self) -> 'QAPIDoc':
>                              doc.append_line(text)
>                          line = self.get_doc_indented(doc)
>                      no_more_args = True
> +                    intro = False

Or after member descriptions.

>                  elif match := re.match(
>                          r'(Returns|Errors|Since|Notes?|Examples?|TODO): *',
>                          line):
> @@ -557,13 +563,14 @@ def get_doc(self) -> 'QAPIDoc':
>                          doc.append_line(text)
>                      line = self.get_doc_indented(doc)
>                      no_more_args = True
> +                    intro = False

Or after the first tagged section.

Okay, it does what it says on the tin.

>                  elif line.startswith('='):
>                      raise QAPIParseError(
>                          self,
>                          "unexpected '=' markup in definition documentation")
>                  else:
>                      # tag-less paragraph
> -                    doc.ensure_untagged_section(self.info)
> +                    doc.ensure_untagged_section(self.info, intro)
>                      doc.append_line(line)
>                      line = self.get_doc_paragraph(doc)
>          else:
> @@ -617,7 +624,7 @@ def __init__(
>              self,
>              info: QAPISourceInfo,
>              tag: Optional[str] = None,
> -            kind: str = 'paragraph',
> +            kind: str = 'intro-paragraph',

The question "why is this optional?" crossed my mind when reviewing the
previous patch.  I left it unasked, because I felt challenging the
overlap between @kind and @tag was more useful.  However, the new
default value 'intro-paragraph' feels more arbitrary to me than the old
one 'paragraph', and that makes the question pop right back into my
mind.

Unless I'm mistaken, all calls but one @tag and @kind.  Making that one
pass it too feels simpler to me.

Moot if we fuse @tag and @kind, of course.

>          ):
>              # section source info, i.e. where it begins
>              self.info = info
> @@ -625,7 +632,7 @@ def __init__(
>              self.tag = tag
>              # section text without tag
>              self.text = ''
> -            # section type - {paragraph, feature, member, tagged}
> +            # section type - {<intro|outro>-paragraph, feature, member, tagged}

Long line.

>              self.kind = kind
>  
>          def append_line(self, line: str) -> None:
> @@ -666,7 +673,11 @@ def end(self) -> None:
>                  raise QAPISemError(
>                      section.info, "text required after '%s:'" % section.tag)
>  
> -    def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
> +    def ensure_untagged_section(
> +        self,
> +        info: QAPISourceInfo,
> +        intro: bool = True,

Two callers, one passes @info, one doesn't.  Passing it always might be
simpler.

> +    ) -> None:
>          if self.all_sections and not self.all_sections[-1].tag:
>              section = self.all_sections[-1]
>              # Section is empty so far; update info to start *here*.
> @@ -677,7 +688,8 @@ def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
>                  self.all_sections[-1].text += '\n'
>              return
>          # start new section
> -        section = self.Section(info)
> +        kind = ("intro" if intro else "outro") + "-paragraph"
> +        section = self.Section(info, kind=kind)
>          self.sections.append(section)
>          self.all_sections.append(section)

Re: [PATCH 05/20] qapi/parser: adjust info location for doc body section

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 4008 bytes --]

On Thu, May 16, 2024, 1:58 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > Instead of using the info object for the doc block as a whole, update
> > the info pointer for each call to ensure_untagged_section when the
> > existing section is otherwise empty. This way, Sphinx error information
> > will match precisely to where the text actually starts.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/parser.py | 9 +++++++--
> >  1 file changed, 7 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> > index 8cdd5334ec6..41b9319e5cb 100644
> > --- a/scripts/qapi/parser.py
> > +++ b/scripts/qapi/parser.py
> > @@ -662,8 +662,13 @@ def end(self) -> None:
> >
> >      def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
> >          if self.all_sections and not self.all_sections[-1].tag:
> > -            # extend current section
> > -            self.all_sections[-1].text += '\n'
>
> Before, we always append a newline.
>
> > +            section = self.all_sections[-1]
> > +            # Section is empty so far; update info to start *here*.
> > +            if not section.text:
> > +                section.info = info
> > +            else:
> > +                # extend current section
> > +                self.all_sections[-1].text += '\n'
>
> Afterwards, we append it only when the section already has some text.
>
> The commit message claims the patch only adjusts section.info.  That's a
> lie :)
>

Well. It wasn't intentional, so it wasn't a lie... it was just wrong :)


> I believe the change makes no difference because .end() strips leading
> and trailing newline.
>
> >              return
> >          # start new section
> >          section = self.Section(info)
>
> You could fix the commit message, but I think backing out the
> no-difference change is easier.  The appended patch works in my testing.
>
> Next one.  Your patch changes the meaning of section.info.  Here's its
> initialization:
>
>     class Section:
>         # pylint: disable=too-few-public-methods
>         def __init__(self, info: QAPISourceInfo,
>                      tag: Optional[str] = None):
> --->        # section source info, i.e. where it begins
>             self.info = info
>             # section tag, if any ('Returns', '@name', ...)
>             self.tag = tag
>             # section text without tag
>             self.text = ''
>
> The comment is now wrong.  Calls for a thorough review of .info's uses.
>

Hmm... Did I really change its meaning? I guess it's debatable what "where
it begins" means. Does the tagless section start...

## <-- Here?
# Hello! <-- Or here?
##

I assert the *section* starts wherever the first line of text it contains
starts. Nothing else makes any sense.

There is value in recording where the doc block starts, but that's not a
task for the *section* info.

I don't think I understand your feedback.


> The alternative to changing .info's meaning is to add another member
> with the meaning you need.  Then we have to review .info's uses to find
> out which ones to switch to the new one.


> Left for later.
>
>
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index 8cdd5334ec..abeae1ca77 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -663,7 +663,10 @@ def end(self) -> None:
>      def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
>          if self.all_sections and not self.all_sections[-1].tag:
>              # extend current section
> -            self.all_sections[-1].text += '\n'
> +            section = self.all_sections[-1]
> +            if not section.text:
> +                section.info = info
> +            section.text += '\n'
>              return
>          # start new section
>          section = self.Section(info)
>
>

[-- Attachment #2: Type: text/html, Size: 6288 bytes --]

Re: [PATCH 07/20] qapi/parser: add semantic 'kind' parameter to QAPIDoc.Section

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 4880 bytes --]

On Thu, May 16, 2024, 2:18 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > When iterating all_sections, this is helpful to be able to distinguish
> > "members" from "features"; the only other way to do so is to
> > cross-reference these sections against QAPIDoc.args or QAPIDoc.features,
> > but if the desired end goal for QAPIDoc is to remove everything except
> > all_sections, we need *something* accessible to distinguish them.
> >
> > To keep types simple, add this semantic parameter to the base Section
> > and not just ArgSection; we can use this to filter out paragraphs and
> > tagged sections, too.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/parser.py | 25 ++++++++++++++++---------
> >  1 file changed, 16 insertions(+), 9 deletions(-)
> >
> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> > index 161768b8b96..cf4cbca1c1f 100644
> > --- a/scripts/qapi/parser.py
> > +++ b/scripts/qapi/parser.py
> > @@ -613,21 +613,27 @@ class QAPIDoc:
> >
> >      class Section:
> >          # pylint: disable=too-few-public-methods
> > -        def __init__(self, info: QAPISourceInfo,
> > -                     tag: Optional[str] = None):
> > +        def __init__(
> > +            self,
> > +            info: QAPISourceInfo,
> > +            tag: Optional[str] = None,
> > +            kind: str = 'paragraph',
> > +        ):
> >              # section source info, i.e. where it begins
> >              self.info = info
> >              # section tag, if any ('Returns', '@name', ...)
> >              self.tag = tag
> >              # section text without tag
> >              self.text = ''
> > +            # section type - {paragraph, feature, member, tagged}
> > +            self.kind = kind
>
> Hmm.  .kind is almost redundant with .tag.
>

Almost, yes. But the crucial bit is members/features as you notice. That's
the real necessity here that saves a lot of code when relying on *only*
all_sections.

(If you want to remove the other fields leaving only all_sections behind,
this is strictly necessary.)


> Untagged section:    .kind is 'paragraph', .tag is None
>
> Member description:  .kind is 'member', .tag matches @NAME
>
> Feature description: .kind is 'feature', .tag matches @NAME


> Tagged section:      .kind is 'tagged', .tag matches
>                           r'Returns|Errors|Since|Notes?|Examples?|TODO'
>
> .kind can directly be derived from .tag except for member and feature
> descriptions.  And you want to tell these two apart in a straightforward
> manner in later patches, as you explain in your commit message.
>
> If .kind is 'member' or 'feature', then self must be an ArgSection.
> Worth a comment?  An assertion?
>

No real need. The classes don't differ much in practice so there's not much
benefit, and asserting it won't help the static typer out anyway because it
can't remember the inference from string->type anyway.

If you wanted to be FANCY, we could use string literal typing on the field
and restrict valid values per-class, but that's so needless not even I'm
tempted by it.


> Some time back, I considered changing .tag for member and feature
> descriptions to suitable strings, like your 'member' and 'feature', and
> move the member / feature name into ArgSection.  I didn't, because the
> benefit wasn't worth the churn at the time.  Perhaps it's worth it now.
> Would it result in simpler code than your solution?
>

Not considerably, I think. Would just be shuffling around which field names
I touch and where/when.

It might actually just add some lines where I have to assert isinstance to
do type narrowing in the generator.


> Terminology nit: the section you call 'paragraph' isn't actually a
> paragraph: it could be several paragraphs.  Best to call it 'untagged',
> as in .ensure_untagged_section().
>

Oh, I hate when you make a good point. I was avoiding the term because I'm
removing Notes and Examples, and we have plans to eliminate Since ... the
tagged sections are almost going away entirely, leaving just TODO, which we
ignore.

Uhm, can I name it paragraphs? :) or open to other suggestions, incl.
untagged if that's still your preference.


> >
> >          def append_line(self, line: str) -> None:
> >              self.text += line + '\n'
> >
> >      class ArgSection(Section):
> > -        def __init__(self, info: QAPISourceInfo, tag: str):
> > -            super().__init__(info, tag)
> > +        def __init__(self, info: QAPISourceInfo, tag: str, kind: str):
> > +            super().__init__(info, tag, kind)
> >              self.member: Optional['QAPISchemaMember'] = None
> >
> >          def connect(self, member: 'QAPISchemaMember') -> None:
>
> [...]
>
>

[-- Attachment #2: Type: text/html, Size: 7400 bytes --]

Re: [PATCH 08/20] qapi/parser: differentiate intro and outro paragraphs

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 6821 bytes --]

On Thu, May 16, 2024, 5:34 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > Add a semantic tag to paragraphs that appear *before* tagged
> > sections/members/features and those that appear after. This will control
> > how they are inlined when doc sections are merged and flattened.
>
> This future use is not obvious to me now.  I guess the effective way to
> help me see it is actual patches, which will come in due time.
>

Head recursion and tail recursion, respectively :)

* intro
* inherited intro
* members [ancestor-descendent]
* features [ancestor-descendent]
* inherited outro
* outro

Child gets the first and final words. Inherited stuff goes in the sandwich
fillings.

It feels like a simple rule that's easy to internalize. As a bonus, you can
explain it by analogy to Americans as a burger, which is the only metaphor
we understand.


> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/parser.py | 22 +++++++++++++++++-----
> >  1 file changed, 17 insertions(+), 5 deletions(-)
> >
> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> > index cf4cbca1c1f..b1794f71e12 100644
> > --- a/scripts/qapi/parser.py
> > +++ b/scripts/qapi/parser.py
> > @@ -503,6 +503,10 @@ def get_doc(self) -> 'QAPIDoc':
> >              self.accept(False)
> >              line = self.get_doc_line()
> >              no_more_args = False
> > +            # Paragraphs before members/features/tagged are "intro"
> paragraphs.
> > +            # Any appearing subsequently are "outro" paragraphs.
> > +            # This is only semantic metadata for the doc generator.
>
> Not sure about the last sentence.  Isn't it true for almost everything
> around here?
>

I guess I was trying to say "There's no real difference between the two
mechanically, it's purely based on where it appears in the doc block, which
offers only a heuristic for its semantic value- introductory statements or
additional detail."

In my mind: the other "kind" values have some more mechanical difference to
them, but intro/outro don't.


> Also, long line.
>
> > +            intro = True
> >
> >              while line is not None:
> >                  # Blank lines
> > @@ -532,6 +536,7 @@ def get_doc(self) -> 'QAPIDoc':
> >                          raise QAPIParseError(
> >                              self, 'feature descriptions expected')
> >                      no_more_args = True
> > +                    intro = False
>
> After feature descriptions.
>
> >                  elif match := self._match_at_name_colon(line):
> >                      # description
> >                      if no_more_args:
> > @@ -547,6 +552,7 @@ def get_doc(self) -> 'QAPIDoc':
> >                              doc.append_line(text)
> >                          line = self.get_doc_indented(doc)
> >                      no_more_args = True
> > +                    intro = False
>
> Or after member descriptions.
>
> >                  elif match := re.match(
> >                          r'(Returns|Errors|Since|Notes?|Examples?|TODO):
> *',
> >                          line):
> > @@ -557,13 +563,14 @@ def get_doc(self) -> 'QAPIDoc':
> >                          doc.append_line(text)
> >                      line = self.get_doc_indented(doc)
> >                      no_more_args = True
> > +                    intro = False
>
> Or after the first tagged section.
>
> Okay, it does what it says on the tin.
>
> >                  elif line.startswith('='):
> >                      raise QAPIParseError(
> >                          self,
> >                          "unexpected '=' markup in definition
> documentation")
> >                  else:
> >                      # tag-less paragraph
> > -                    doc.ensure_untagged_section(self.info)
> > +                    doc.ensure_untagged_section(self.info, intro)
> >                      doc.append_line(line)
> >                      line = self.get_doc_paragraph(doc)
> >          else:
> > @@ -617,7 +624,7 @@ def __init__(
> >              self,
> >              info: QAPISourceInfo,
> >              tag: Optional[str] = None,
> > -            kind: str = 'paragraph',
> > +            kind: str = 'intro-paragraph',
>
> The question "why is this optional?" crossed my mind when reviewing the
> previous patch.  I left it unasked, because I felt challenging the
> overlap between @kind and @tag was more useful.  However, the new
> default value 'intro-paragraph' feels more arbitrary to me than the old
> one 'paragraph', and that makes the question pop right back into my
> mind.
>

Just "don't break API" habit, nothing more. I can make it mandatory.


> Unless I'm mistaken, all calls but one @tag and @kind.  Making that one
> pass it too feels simpler to me.
>
> Moot if we fuse @tag and @kind, of course.


> >          ):
> >              # section source info, i.e. where it begins
> >              self.info = info
> > @@ -625,7 +632,7 @@ def __init__(
> >              self.tag = tag
> >              # section text without tag
> >              self.text = ''
> > -            # section type - {paragraph, feature, member, tagged}
> > +            # section type - {<intro|outro>-paragraph, feature, member,
> tagged}
>
> Long line.


Oops, default for black is longer. Forgot to enable the "I still use email
patches as part of my penance" setting


> >              self.kind = kind
> >
> >          def append_line(self, line: str) -> None:
> > @@ -666,7 +673,11 @@ def end(self) -> None:
> >                  raise QAPISemError(
> >                      section.info, "text required after '%s:'" %
> section.tag)
> >
> > -    def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
> > +    def ensure_untagged_section(
> > +        self,
> > +        info: QAPISourceInfo,
> > +        intro: bool = True,
>
> Two callers, one passes @info, one doesn't.  Passing it always might be
> simpler.
>

Okeydokey.


> > +    ) -> None:
> >          if self.all_sections and not self.all_sections[-1].tag:
> >              section = self.all_sections[-1]
> >              # Section is empty so far; update info to start *here*.
> > @@ -677,7 +688,8 @@ def ensure_untagged_section(self, info:
> QAPISourceInfo) -> None:
> >                  self.all_sections[-1].text += '\n'
> >              return
> >          # start new section
> > -        section = self.Section(info)
> > +        kind = ("intro" if intro else "outro") + "-paragraph"
> > +        section = self.Section(info, kind=kind)
> >          self.sections.append(section)
> >          self.all_sections.append(section)
>
>

[-- Attachment #2: Type: text/html, Size: 10641 bytes --]

Re: [PATCH 06/20] qapi/parser: fix comment parsing immediately following a doc block

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

On Thu, May 16, 2024 at 2:01 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > If a comment immediately follows a doc block, the parser doesn't ignore
> > that token appropriately. Fix that.
>
> Reproducer?
>
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/parser.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> > index 41b9319e5cb..161768b8b96 100644
> > --- a/scripts/qapi/parser.py
> > +++ b/scripts/qapi/parser.py
> > @@ -587,7 +587,7 @@ def get_doc(self) -> 'QAPIDoc':
> >                  line = self.get_doc_line()
> >                  first = False
> >
> > -        self.accept(False)
> > +        self.accept()
> >          doc.end()
> >          return doc
>
> Can't judge the fix without understanding the problem, and the problem
> will be easier to understand for me with a reproducer.
>

audio.json:

```
##
# = Audio
##

##
# @AudiodevPerDirectionOptions:
#
# General audio backend options that are used for both playback and
# recording.
#
```

Modify this excerpt to have a comment after the "= Audio" header, say for
instance if you were to take out that intro paragraph and transform it into
a comment that preceded the AudiodevPerDirectionOptions doc block.

e.g.

```
##
# = Audio
##

# Lorem Ipsum

##
# @AudiodevPerDirectionOptions:
```

the parser breaks because the line I changed that primes the next token is
still set to "not ignore comments", but that breaks the parser and gives a
rather unhelpful message:

../qapi/audio.json:13:1: junk after '##' at start of documentation comment

Encountered when converting developer commentary from documentation
paragraphs to mere QAPI comments.

--js

[-- Attachment #2: Type: text/html, Size: 3020 bytes --]

Re: [PATCH 08/20] qapi/parser: differentiate intro and outro paragraphs

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 7295 bytes --]

On Thu, May 16, 2024 at 11:06 AM John Snow <jsnow@redhat.com> wrote:

>
>
> On Thu, May 16, 2024, 5:34 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Add a semantic tag to paragraphs that appear *before* tagged
>> > sections/members/features and those that appear after. This will control
>> > how they are inlined when doc sections are merged and flattened.
>>
>> This future use is not obvious to me now.  I guess the effective way to
>> help me see it is actual patches, which will come in due time.
>>
>
> Head recursion and tail recursion, respectively :)
>
> * intro
> * inherited intro
> * members [ancestor-descendent]
> * features [ancestor-descendent]
> * inherited outro
> * outro
>
> Child gets the first and final words. Inherited stuff goes in the sandwich
> fillings.
>
> It feels like a simple rule that's easy to internalize. As a bonus, you
> can explain it by analogy to Americans as a burger, which is the only
> metaphor we understand.
>
>
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  scripts/qapi/parser.py | 22 +++++++++++++++++-----
>> >  1 file changed, 17 insertions(+), 5 deletions(-)
>> >
>> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>> > index cf4cbca1c1f..b1794f71e12 100644
>> > --- a/scripts/qapi/parser.py
>> > +++ b/scripts/qapi/parser.py
>> > @@ -503,6 +503,10 @@ def get_doc(self) -> 'QAPIDoc':
>> >              self.accept(False)
>> >              line = self.get_doc_line()
>> >              no_more_args = False
>> > +            # Paragraphs before members/features/tagged are "intro"
>> paragraphs.
>> > +            # Any appearing subsequently are "outro" paragraphs.
>> > +            # This is only semantic metadata for the doc generator.
>>
>> Not sure about the last sentence.  Isn't it true for almost everything
>> around here?
>>
>
> I guess I was trying to say "There's no real difference between the two
> mechanically, it's purely based on where it appears in the doc block, which
> offers only a heuristic for its semantic value- introductory statements or
> additional detail."
>
> In my mind: the other "kind" values have some more mechanical difference
> to them, but intro/outro don't.
>
>
>> Also, long line.
>>
>> > +            intro = True
>> >
>> >              while line is not None:
>> >                  # Blank lines
>> > @@ -532,6 +536,7 @@ def get_doc(self) -> 'QAPIDoc':
>> >                          raise QAPIParseError(
>> >                              self, 'feature descriptions expected')
>> >                      no_more_args = True
>> > +                    intro = False
>>
>> After feature descriptions.
>>
>> >                  elif match := self._match_at_name_colon(line):
>> >                      # description
>> >                      if no_more_args:
>> > @@ -547,6 +552,7 @@ def get_doc(self) -> 'QAPIDoc':
>> >                              doc.append_line(text)
>> >                          line = self.get_doc_indented(doc)
>> >                      no_more_args = True
>> > +                    intro = False
>>
>> Or after member descriptions.
>>
>> >                  elif match := re.match(
>> >
>> r'(Returns|Errors|Since|Notes?|Examples?|TODO): *',
>> >                          line):
>> > @@ -557,13 +563,14 @@ def get_doc(self) -> 'QAPIDoc':
>> >                          doc.append_line(text)
>> >                      line = self.get_doc_indented(doc)
>> >                      no_more_args = True
>> > +                    intro = False
>>
>> Or after the first tagged section.
>>
>> Okay, it does what it says on the tin.
>>
>> >                  elif line.startswith('='):
>> >                      raise QAPIParseError(
>> >                          self,
>> >                          "unexpected '=' markup in definition
>> documentation")
>> >                  else:
>> >                      # tag-less paragraph
>> > -                    doc.ensure_untagged_section(self.info)
>> > +                    doc.ensure_untagged_section(self.info, intro)
>> >                      doc.append_line(line)
>> >                      line = self.get_doc_paragraph(doc)
>> >          else:
>> > @@ -617,7 +624,7 @@ def __init__(
>> >              self,
>> >              info: QAPISourceInfo,
>> >              tag: Optional[str] = None,
>> > -            kind: str = 'paragraph',
>> > +            kind: str = 'intro-paragraph',
>>
>> The question "why is this optional?" crossed my mind when reviewing the
>> previous patch.  I left it unasked, because I felt challenging the
>> overlap between @kind and @tag was more useful.  However, the new
>> default value 'intro-paragraph' feels more arbitrary to me than the old
>> one 'paragraph', and that makes the question pop right back into my
>> mind.
>>
>
> Just "don't break API" habit, nothing more. I can make it mandatory.
>
>
>> Unless I'm mistaken, all calls but one @tag and @kind.  Making that one
>> pass it too feels simpler to me.
>>
>> Moot if we fuse @tag and @kind, of course.
>
>
>> >          ):
>> >              # section source info, i.e. where it begins
>> >              self.info = info
>> > @@ -625,7 +632,7 @@ def __init__(
>> >              self.tag = tag
>> >              # section text without tag
>> >              self.text = ''
>> > -            # section type - {paragraph, feature, member, tagged}
>> > +            # section type - {<intro|outro>-paragraph, feature,
>> member, tagged}
>>
>> Long line.
>
>
> Oops, default for black is longer. Forgot to enable the "I still use email
> patches as part of my penance" setting
>

Oh, no, this is actually 79 columns on the button. Not picked up by *any*
of our linters. Ehm... can you make the tools enforce them to your
preference instead, please...? What's a "long line"?


>
>
>> >              self.kind = kind
>> >
>> >          def append_line(self, line: str) -> None:
>> > @@ -666,7 +673,11 @@ def end(self) -> None:
>> >                  raise QAPISemError(
>> >                      section.info, "text required after '%s:'" %
>> section.tag)
>> >
>> > -    def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
>> > +    def ensure_untagged_section(
>> > +        self,
>> > +        info: QAPISourceInfo,
>> > +        intro: bool = True,
>>
>> Two callers, one passes @info, one doesn't.  Passing it always might be
>> simpler.
>>
>
> Okeydokey.
>
>
>> > +    ) -> None:
>> >          if self.all_sections and not self.all_sections[-1].tag:
>> >              section = self.all_sections[-1]
>> >              # Section is empty so far; update info to start *here*.
>> > @@ -677,7 +688,8 @@ def ensure_untagged_section(self, info:
>> QAPISourceInfo) -> None:
>> >                  self.all_sections[-1].text += '\n'
>> >              return
>> >          # start new section
>> > -        section = self.Section(info)
>> > +        kind = ("intro" if intro else "outro") + "-paragraph"
>> > +        section = self.Section(info, kind=kind)
>> >          self.sections.append(section)
>> >          self.all_sections.append(section)
>>
>>

[-- Attachment #2: Type: text/html, Size: 11571 bytes --]

Re: [PATCH 00/20] qapi: new sphinx qapi domain pre-requisites

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5527 bytes --]

On Tue, May 14, 2024 at 05:57:19PM -0400, John Snow wrote:
> Howdy - this patch series is the first batch of patches meant to prepare
> the QAPI documentation for a new Sphinx module that adds
> cross-references, an index, improved inlining, elision of types unseen
> on the wire, and other goodies.
> 
> This series addresses just existing code and documentation that needs to
> be changed and doesn't introduce anything new just yet - except the rST
> conversion of Notes and Examples sections, which DOES impact the
> existing QAPI documentation generation.
> 
> If you're CC'd on this series, it's *probably* because I've adjusted
> some QAPI documentation that you're the maintainer of - In most cases,
> these changes are purely mechanical (converting QAPI sections into pure
> rST) and probably nothing too interesting. In a small handful of cases
> (patches 15-17), I've been a bit more invasive and you may want to take
> a quick peek.
> 
> Overview:
> 
> Patches 1-3: linter/typing cleanup
> Patches 4-12: QAPI generator fixes/miscellany
> Patch 13: qapidoc.py fix (to prepare for rST conversion)
> Patches 14-20: QAPI documentation modifications, rST conversion
> 
> Sorry,
> --js
> 
> John Snow (20):
>   [DO-NOT-MERGE]: Add some ad-hoc linting helpers.
>   qapi: linter fixups
>   docs/qapidoc: delint a tiny portion of the module
>   qapi/parser: preserve indentation in QAPIDoc sections
>   qapi/parser: adjust info location for doc body section
>   qapi/parser: fix comment parsing immediately following a doc block
>   qapi/parser: add semantic 'kind' parameter to QAPIDoc.Section
>   qapi/parser: differentiate intro and outro paragraphs
>   qapi/parser: add undocumented stub members to all_sections
>   qapi/schema: add __iter__ method to QAPISchemaVariants
>   qapi/schema: add doc_visible property to QAPISchemaDefinition
>   qapi/source: allow multi-line QAPISourceInfo advancing
>   docs/qapidoc: fix nested parsing under untagged sections
>   qapi: fix non-compliant JSON examples
>   qapi: remove developer factoring comments from QAPI doc blocks
>   qapi: rewrite StatsFilter comment
>   qapi: rewrite BlockExportOptions doc block
>   qapi: ensure all errors sections are uniformly typset
>   qapi: convert "Note" sections to plain rST
>   qapi: convert "Example" sections to rST
> 
>  docs/sphinx/qapidoc.py                        |  62 ++++--
>  qapi/acpi.json                                |   6 +-
>  qapi/audio.json                               |   5 +-
>  qapi/block-core.json                          | 195 ++++++++++--------
>  qapi/block-export.json                        |  16 +-
>  qapi/block.json                               |  62 +++---
>  qapi/char.json                                |  53 +++--
>  qapi/control.json                             |  32 +--
>  qapi/crypto.json                              |  33 ++-
>  qapi/dump.json                                |  14 +-
>  qapi/introspect.json                          |   6 +-
>  qapi/machine-target.json                      |  29 +--
>  qapi/machine.json                             | 138 +++++++------
>  qapi/migration.json                           | 159 +++++++++-----
>  qapi/misc-target.json                         |  33 ++-
>  qapi/misc.json                                | 139 +++++++------
>  qapi/net.json                                 |  49 +++--
>  qapi/pci.json                                 |  11 +-
>  qapi/qapi-schema.json                         |   6 +-
>  qapi/qdev.json                                |  45 ++--
>  qapi/qom.json                                 |  69 +++----
>  qapi/replay.json                              |  12 +-
>  qapi/rocker.json                              |  30 +--
>  qapi/run-state.json                           |  63 +++---
>  qapi/sockets.json                             |  10 +-
>  qapi/stats.json                               |  30 ++-
>  qapi/tpm.json                                 |   9 +-
>  qapi/trace.json                               |   6 +-
>  qapi/transaction.json                         |  13 +-
>  qapi/ui.json                                  | 107 +++++-----
>  qapi/virtio.json                              |  50 ++---
>  qapi/yank.json                                |   6 +-
>  qga/qapi-schema.json                          |  48 ++---
>  scripts/qapi-lint.sh                          |  51 +++++
>  scripts/qapi/Makefile                         |   5 +
>  scripts/qapi/introspect.py                    |  12 +-
>  scripts/qapi/parser.py                        | 104 ++++++++--
>  scripts/qapi/schema.py                        |  54 ++++-
>  scripts/qapi/source.py                        |   4 +-
>  scripts/qapi/types.py                         |   4 +-
>  scripts/qapi/visit.py                         |   9 +-
>  tests/qapi-schema/doc-empty-section.err       |   2 +-
>  tests/qapi-schema/doc-empty-section.json      |   2 +-
>  tests/qapi-schema/doc-good.json               |  18 +-
>  tests/qapi-schema/doc-good.out                |  61 +++---
>  tests/qapi-schema/doc-good.txt                |  31 +--
>  .../qapi-schema/doc-interleaved-section.json  |   2 +-
>  47 files changed, 1152 insertions(+), 753 deletions(-)
>  create mode 100755 scripts/qapi-lint.sh
>  create mode 100644 scripts/qapi/Makefile
> 
> -- 
> 2.44.0
> 
> 

For block-core.json/block-export.json/block.json:

Acked-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 05/20] qapi/parser: adjust info location for doc body section

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, May 16, 2024, 1:58 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Instead of using the info object for the doc block as a whole, update
>> > the info pointer for each call to ensure_untagged_section when the
>> > existing section is otherwise empty. This way, Sphinx error information
>> > will match precisely to where the text actually starts.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  scripts/qapi/parser.py | 9 +++++++--
>> >  1 file changed, 7 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>> > index 8cdd5334ec6..41b9319e5cb 100644
>> > --- a/scripts/qapi/parser.py
>> > +++ b/scripts/qapi/parser.py
>> > @@ -662,8 +662,13 @@ def end(self) -> None:
>> >
>> >      def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
>> >          if self.all_sections and not self.all_sections[-1].tag:
>> > -            # extend current section
>> > -            self.all_sections[-1].text += '\n'
>>
>> Before, we always append a newline.
>>
>> > +            section = self.all_sections[-1]
>> > +            # Section is empty so far; update info to start *here*.
>> > +            if not section.text:
>> > +                section.info = info
>> > +            else:
>> > +                # extend current section
>> > +                self.all_sections[-1].text += '\n'
>>
>> Afterwards, we append it only when the section already has some text.
>>
>> The commit message claims the patch only adjusts section.info.  That's a
>> lie :)
>>
>
> Well. It wasn't intentional, so it wasn't a lie... it was just wrong :)
>
>
>> I believe the change makes no difference because .end() strips leading
>> and trailing newline.
>>
>> >              return
>> >          # start new section
>> >          section = self.Section(info)
>>
>> You could fix the commit message, but I think backing out the
>> no-difference change is easier.  The appended patch works in my testing.
>>
>> Next one.  Your patch changes the meaning of section.info.  Here's its
>> initialization:
>>
>>     class Section:
>>         # pylint: disable=too-few-public-methods
>>         def __init__(self, info: QAPISourceInfo,
>>                      tag: Optional[str] = None):
>> --->        # section source info, i.e. where it begins
>>             self.info = info
>>             # section tag, if any ('Returns', '@name', ...)
>>             self.tag = tag
>>             # section text without tag
>>             self.text = ''
>>
>> The comment is now wrong.  Calls for a thorough review of .info's uses.
>>
>
> Hmm... Did I really change its meaning? I guess it's debatable what "where
> it begins" means. Does the tagless section start...
>
> ## <-- Here?
> # Hello! <-- Or here?
> ##
>
> I assert the *section* starts wherever the first line of text it contains
> starts. Nothing else makes any sense.
>
> There is value in recording where the doc block starts, but that's not a
> task for the *section* info.
>
> I don't think I understand your feedback.

This was before my vacation, and my memory is foggy, ...  I may have
gotten confused back then.  Let me have a fresh look now.

self.info gets initialized in Section.__init__() to whatever info it
gets passed.

Your patch makes .ensure_untagged_section() overwrite this Section.info
when it extends an untagged section that is still empty.  Hmmm.  I'd
prefer .info to remain constant after initialization.

I figure this overwrite can happen only when extenting the body section
QAPIDoc.__init__() creates.  In that case, it adjusts .info from
beginning of doc comment to first non-blank line.

Thoughts?

>> The alternative to changing .info's meaning is to add another member
>> with the meaning you need.  Then we have to review .info's uses to find
>> out which ones to switch to the new one.
>
>
>> Left for later.
>>
>>
>> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>> index 8cdd5334ec..abeae1ca77 100644
>> --- a/scripts/qapi/parser.py
>> +++ b/scripts/qapi/parser.py
>> @@ -663,7 +663,10 @@ def end(self) -> None:
>>      def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
>>          if self.all_sections and not self.all_sections[-1].tag:
>>              # extend current section
>> -            self.all_sections[-1].text += '\n'
>> +            section = self.all_sections[-1]
>> +            if not section.text:
>> +                section.info = info
>> +            section.text += '\n'
>>              return
>>          # start new section
>>          section = self.Section(info)
>>
>>