[Qemu-devel] [PATCH 0/2] qapi2texi: add type information

[Qemu-devel] [PATCH 0/2] qapi2texi: add type information

[Marc-André Lureau]

The type information was dropped from v6 of the "qapi doc generation"
series, since the syntax being used was still under discussion.

However, the documentation without the type information is incomplete
and you have to refer to the json schema for the details. With type
information, you can hopefully only use the generated documentation.

See the second patch summary for a short description of the syntax.

Marc-André Lureau (2):
  qapi2texi: change texi formatters
  qapi2texi: produce type information

 scripts/qapi2texi.py | 114 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 85 insertions(+), 29 deletions(-)

-- 
2.11.0.295.gd7dffce1c

[Qemu-devel] [PATCH 1/2] qapi2texi: change texi formatters

[Marc-André Lureau]

STRUCT_FMT is generic enough, rename it to TYPE_FMT, use it for unions.

Rename COMMAND_FMT to MSG_FMT, since it applies to both commands and
events.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi2texi.py | 46 +++++++++++++++++++---------------------------
 1 file changed, 19 insertions(+), 27 deletions(-)

diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index 83ded95c2d..e1b79c2ad3 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -9,7 +9,7 @@ import sys
 
 import qapi
 
-COMMAND_FMT = """
+MSG_FMT = """
 @deftypefn {type} {{}} {name}
 
 {body}
@@ -18,16 +18,7 @@ COMMAND_FMT = """
 
 """.format
 
-ENUM_FMT = """
-@deftp Enum {name}
-
-{body}
-
-@end deftp
-
-""".format
-
-STRUCT_FMT = """
+TYPE_FMT = """
 @deftp {{{type}}} {name}
 
 {body}
@@ -172,9 +163,9 @@ def texi_body(doc):
 def texi_alternate(expr, doc):
     """Format an alternate to texi"""
     body = texi_body(doc)
-    return STRUCT_FMT(type="Alternate",
-                      name=doc.symbol,
-                      body=body)
+    return TYPE_FMT(type="Alternate",
+                    name=doc.symbol,
+                    body=body)
 
 
 def texi_union(expr, doc):
@@ -186,9 +177,9 @@ def texi_union(expr, doc):
         union = "Simple Union"
 
     body = texi_body(doc)
-    return STRUCT_FMT(type=union,
-                      name=doc.symbol,
-                      body=body)
+    return TYPE_FMT(type=union,
+                    name=doc.symbol,
+                    body=body)
 
 
 def texi_enum(expr, doc):
@@ -197,32 +188,33 @@ def texi_enum(expr, doc):
         if i not in doc.args:
             doc.args[i] = ''
     body = texi_body(doc)
-    return ENUM_FMT(name=doc.symbol,
+    return TYPE_FMT(type="Enum",
+                    name=doc.symbol,
                     body=body)
 
 
 def texi_struct(expr, doc):
     """Format a struct to texi"""
     body = texi_body(doc)
-    return STRUCT_FMT(type="Struct",
-                      name=doc.symbol,
-                      body=body)
+    return TYPE_FMT(type="Struct",
+                    name=doc.symbol,
+                    body=body)
 
 
 def texi_command(expr, doc):
     """Format a command to texi"""
     body = texi_body(doc)
-    return COMMAND_FMT(type="Command",
-                       name=doc.symbol,
-                       body=body)
+    return MSG_FMT(type="Command",
+                   name=doc.symbol,
+                   body=body)
 
 
 def texi_event(expr, doc):
     """Format an event to texi"""
     body = texi_body(doc)
-    return COMMAND_FMT(type="Event",
-                       name=doc.symbol,
-                       body=body)
+    return MSG_FMT(type="Event",
+                   name=doc.symbol,
+                   body=body)
 
 
 def texi_expr(expr, doc):
-- 
2.11.0.295.gd7dffce1c

[Qemu-devel] [PATCH 2/2] qapi2texi: produce type information

[Marc-André Lureau]

Add type information to the generated documentation. Without it the
written documentation is not explicit enough to know how to handle
the various arguments and members.

Array types have the following syntax: type[]. Ex: str[].

- Struct, commands and events use the following members syntax:

  { 'member': type, ('foo': str), ... }

Optional members are under parentheses.

A structure with a base type will have 'BaseStruct +' prepended.

- Alternates use the following syntax:

  [ 'foo': type, 'bar': type, ... ]

- Simple unions use the following syntax:

  { 'type': str, 'data': 'type' = [ 'foo': type, 'bar': type... ] }

- Flat unions use the following syntax:

  BaseStruct + 'discriminator' = [ 'foo': type, 'bar': type... ]

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi2texi.py | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 66 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index e1b79c2ad3..d632c72139 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -10,7 +10,8 @@ import sys
 import qapi
 
 MSG_FMT = """
-@deftypefn {type} {{}} {name}
+@deftypefn {type} {{{ret}}} {name} @
+{{{args}}}
 
 {body}
 
@@ -19,7 +20,8 @@ MSG_FMT = """
 """.format
 
 TYPE_FMT = """
-@deftp {{{type}}} {name}
+@deftp {{{type}}} {name} @
+{{{members}}}
 
 {body}
 
@@ -123,6 +125,36 @@ def texi_format(doc):
     return "\n".join(lines)
 
 
+def texi_type(typ):
+    """Format a type"""
+    if isinstance(typ, list):
+        # must contain single type name
+        typ = "%s[]" % typ[0]
+
+    return '@t{%s}' % typ
+
+
+def texi_args(expr, key="data", fmt="@{%s@}"):
+    """Format the functions/structure/events arguments/members"""
+    if key not in expr:
+        return ""
+    args = expr[key]
+    if isinstance(args, str) or isinstance(args, list):
+        ret = texi_type(args)
+    else:
+        arg_list = []
+        for name, typ in args.iteritems():
+            # optional
+            if name.startswith("*"):
+                name = name[1:]
+                arg_list.append("('%s': %s)" % (name, texi_type(typ)))
+            else:
+                arg_list.append("'%s': %s" % (name, texi_type(typ)))
+        ret = fmt % ", ".join(arg_list)
+
+    return ret
+
+
 def texi_body(doc):
     """
     Format the body of a symbol documentation:
@@ -162,9 +194,11 @@ def texi_body(doc):
 
 def texi_alternate(expr, doc):
     """Format an alternate to texi"""
+    members = texi_args(expr, fmt="[%s]")
     body = texi_body(doc)
     return TYPE_FMT(type="Alternate",
                     name=doc.symbol,
+                    members=members,
                     body=body)
 
 
@@ -172,13 +206,26 @@ def texi_union(expr, doc):
     """Format a union to texi"""
     discriminator = expr.get("discriminator")
     if discriminator:
+        is_flat = True
         union = "Flat Union"
     else:
+        is_flat = False
         union = "Simple Union"
+        discriminator = "type"
 
+    members = ""
+    if is_flat:
+        members += texi_args(expr, "base") + " + "
+    else:
+        members += "@{ 'type': @t{str}, 'data': "
+    members += "'%s' = " % discriminator
+    members += texi_args(expr, "data", fmt="[%s]")
+    if not is_flat:
+        members += " @}"
     body = texi_body(doc)
     return TYPE_FMT(type=union,
                     name=doc.symbol,
+                    members=members,
                     body=body)
 
 
@@ -190,30 +237,47 @@ def texi_enum(expr, doc):
     body = texi_body(doc)
     return TYPE_FMT(type="Enum",
                     name=doc.symbol,
+                    members="",
                     body=body)
 
 
 def texi_struct(expr, doc):
     """Format a struct to texi"""
     body = texi_body(doc)
+    args = texi_args(expr)
+    base = expr.get("base")
+    members = ""
+    if base:
+        members += "%s" % base
+        if args:
+            members += " + "
+    members += args
     return TYPE_FMT(type="Struct",
                     name=doc.symbol,
+                    members=members,
                     body=body)
 
 
 def texi_command(expr, doc):
     """Format a command to texi"""
     body = texi_body(doc)
+    args = texi_args(expr)
+    ret = texi_args(expr, "returns")
     return MSG_FMT(type="Command",
                    name=doc.symbol,
+                   ret=ret,
+                   args=args,
                    body=body)
 
 
 def texi_event(expr, doc):
     """Format an event to texi"""
     body = texi_body(doc)
+    args = texi_args(expr)
     return MSG_FMT(type="Event",
                    name=doc.symbol,
+                   ret="",
+                   args=args,
                    body=body)
 
 
-- 
2.11.0.295.gd7dffce1c

Re: [Qemu-devel] [PATCH 1/2] qapi2texi: change texi formatters

[Markus Armbruster]

I don't like "change FOO" subject lines.  What about;

    qapi: Simplify qapi2texi.py a bit

This way, it's obvious that the commit isn't supposed to change
behavior.  Can touch up on commit.

Marc-André Lureau <marcandre.lureau@redhat.com> writes:

> STRUCT_FMT is generic enough, rename it to TYPE_FMT, use it for unions.
>
> Rename COMMAND_FMT to MSG_FMT, since it applies to both commands and
> events.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  scripts/qapi2texi.py | 46 +++++++++++++++++++---------------------------
>  1 file changed, 19 insertions(+), 27 deletions(-)
>
> diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
> index 83ded95c2d..e1b79c2ad3 100755
> --- a/scripts/qapi2texi.py
> +++ b/scripts/qapi2texi.py
> @@ -9,7 +9,7 @@ import sys
>  
>  import qapi
>  
> -COMMAND_FMT = """
> +MSG_FMT = """

QMP_MSG_FMT?  Your choice, can rename on commit.

>  @deftypefn {type} {{}} {name}
>  
>  {body}
[...]

With the commit message improved:
Reviewed-by: Markus Armbruster <armbru@redhat.com>

Re: [Qemu-devel] [PATCH 2/2] qapi2texi: produce type information

[Markus Armbruster]

Marc-André Lureau <marcandre.lureau@redhat.com> writes:

> Add type information to the generated documentation. Without it the
> written documentation is not explicit enough to know how to handle
> the various arguments and members.

This is actually a regression of sorts: the type information we used to
have in qmp-commands.txt got lost when we replaced it by generated
qemu-qmp-ref.*.

> Array types have the following syntax: type[]. Ex: str[].
>
> - Struct, commands and events use the following members syntax:
>
>   { 'member': type, ('foo': str), ... }
>
> Optional members are under parentheses.
>
> A structure with a base type will have 'BaseStruct +' prepended.
>
> - Alternates use the following syntax:
>
>   [ 'foo': type, 'bar': type, ... ]
>
> - Simple unions use the following syntax:
>
>   { 'type': str, 'data': 'type' = [ 'foo': type, 'bar': type... ] }
>
> - Flat unions use the following syntax:
>
>   BaseStruct + 'discriminator' = [ 'foo': type, 'bar': type... ]
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>

This is a formal language to describe QAPI/QMP from the user's point of
view (the QAPI schema specifies it from the implementor's point of
view).

I intend to implement ideas outlined in my review of "[PATCH v6 05/17]
docs: add master qapi texi files"[*], in particular adding type
information to the argument table.  That will address the stated purpose
of this patch, namely fixing the documentation regression.

We can then compare which fix provides the type information in more
readable form, and whether the additional formal description in your fix
is worth having.

Unfortunately, I need to to some QemuOpts / QAPI work first, because it
blocks features we'd like to have in 2.9.


[*] Message-ID: <87zijp8bxr.fsf@dusky.pond.sub.org>
http://lists.gnu.org/archive/html/qemu-devel/2016-12/msg03002.html