[Qemu-devel] [PATCH RFC v2 00/47] qapi: QMP introspection

[Qemu-devel] [PATCH RFC v2 00/47] qapi: QMP introspection

[Markus Armbruster]

Still RFC, because it has a couple of FIXMEs and TODOs.

* PATCH 01-19 are preliminary fixes and cleanups.  Not really RFC.

* PATCH 20-38 basically replace the interemediate representation.  The
  replacement isn't complete, but fully functional.  See PATCH 21 for
  rationale and future work.

* PATCH 39-44 replace the '**' type bypass by a proper top type called
  'any'.

* PATCH 45 is the introspection RFC.

* PATCH 46-47 are minor variations to hide a bit more detail from
  clients.

I'm afraid the size of my series will strain reviewers.  I did my
level best to split things up into reviewable pieces.

Series depends on my "[PATCH for-2.4 0/2] qom: Fix misuse of Error
API" to pass tests.

You can also fetch it from my git://repo.or.cz/qemu/armbru.git branch
qapi-introspect.

I'll be offline for two weeks starting next Monday.

Bombs away!

v2:
* Old PATCH 01-18 have been merged
* PATCH 19 exploded.  Comparing against RFC v1 is almost certainly a
  waste of time, except perhaps for the QAPI schema introspect.json.
  The awkward alternate Value got replaced by the proper top type
  'any'.
* For even more kaboom, I fixed and cleaned up along the way.

Markus Armbruster (47):
  qapi: Clarify docs on including the same file multiple times
  qapi: Clean up cgen() and mcgen()
  qapi: Simplify guardname()
  qapi-event: Clean up how name of enum QAPIEvent is made
  qapi: Reject -p arguments that break qapi-event.py
  qapi: Drop unused and useless parameters and variables
  qapi: Generate a nicer struct for flat unions
  qapi-visit: Fix generated code when schema has forward refs
  qapi-visit: Replace list implicit_structs by set
  qapi-visit: Fix two name arguments passed to visitors
  tests/qapi-schema: Document alternate's enum lacks visit function
  tests/qapi-schema: Document events with with base don't work
  tests/qapi-schema: Restore test case for flat union base bug
  qapi-tests: New tests for union, alternate command arguments
  qapi: Fix to reject union command arguments
  qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'
  qapi-commands: Inline gen_marshal_output_call()
  qapi-commands: Don't feed output of mcgen() to mcgen() again
  qapi: Generated code cleanup
  qapi: Rename class QAPISchema to QAPISchemaParser
  qapi: New QAPISchema intermediate reperesentation
  qapi: QAPISchema code generation helper methods
  qapi: New QAPISchemaVisitor
  tests/qapi-schema: Convert test harness to QAPISchemaVisitor
  qapi: Make generators work on sorted schema expressions
  qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
  qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
  qapi-commands: Convert to QAPISchemaVisitor
  qapi: Replace dirty is_c_ptr() by method c_null()
  qapi: De-duplicate enum code generation
  qapi-event: Eliminate global variable event_enum_value
  qapi-event: Convert to QAPISchemaVisitor, fixing data with base
  qapi: Clean up after recent conversions to QAPISchemaVisitor
  qapi-visit: Rearrange code a bit
  qapi-commands: Rearrange code
  qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()
  qapi: De-duplicate parameter list generation
  qapi-commands: De-duplicate output marshaling functions
  qapi: Improve built-in type documentation
  qapi: Introduce a first class 'any' type
  qom: Don't use 'gen': false for qom-get, qom-set, object-add
  qapi-schema: Fix up misleading specification of netdev_add
  qmp: Improve netdev_add usage example in the manual
  qapi: Pseudo-type '**' is now unused, drop it
  qapi: New QMP command query-schema for QMP schema introspection
  qapi-introspect: Map all integer types to 'int'
  qapi-introspect: Hide type names

 .gitignore                                         |   1 +
 Makefile                                           |   9 +-
 Makefile.objs                                      |   4 +-
 docs/qapi-code-gen.txt                             |  53 +-
 docs/writing-qmp-commands.txt                      |   8 +-
 include/monitor/monitor.h                          |   3 -
 include/qapi/visitor-impl.h                        |   2 +
 include/qapi/visitor.h                             |   1 +
 monitor.c                                          |  17 +-
 qapi-schema.json                                   |  25 +-
 qapi/introspect.json                               |  69 +++
 qapi/qapi-dealloc-visitor.c                        |   9 +
 qapi/qapi-visit-core.c                             |   6 +
 qapi/qmp-input-visitor.c                           |  11 +
 qapi/qmp-output-visitor.c                          |   9 +
 qmp-commands.hx                                    | 266 +++++----
 qmp.c                                              |  20 +-
 scripts/qapi-commands.py                           | 355 ++++++-----
 scripts/qapi-event.py                              | 243 +++-----
 scripts/qapi-introspect.py                         | 189 ++++++
 scripts/qapi-types.py                              | 378 +++++-------
 scripts/qapi-visit.py                              | 368 +++++-------
 scripts/qapi.py                                    | 658 +++++++++++++++++----
 tests/.gitignore                                   |   1 +
 tests/Makefile                                     |  15 +-
 tests/qapi-schema/alternate-good.out               |  16 +-
 tests/qapi-schema/args-alternate.err               |   1 +
 ...type-bypass-no-gen.exit => args-alternate.exit} |   0
 tests/qapi-schema/args-alternate.json              |   4 +
 .../{type-bypass-no-gen.out => args-alternate.out} |   0
 tests/qapi-schema/args-returns-any.err             |   1 +
 tests/qapi-schema/args-returns-any.exit            |   1 +
 tests/qapi-schema/args-returns-any.json            |   6 +
 .../{type-bypass.err => args-returns-any.out}      |   0
 tests/qapi-schema/args-union.err                   |   1 +
 tests/qapi-schema/args-union.exit                  |   1 +
 tests/qapi-schema/args-union.json                  |   4 +
 tests/qapi-schema/args-union.out                   |   0
 tests/qapi-schema/comments.out                     |   5 +-
 tests/qapi-schema/data-member-array.out            |  14 +-
 tests/qapi-schema/empty.out                        |   4 +-
 tests/qapi-schema/enum-empty.out                   |   5 +-
 tests/qapi-schema/event-case.out                   |   5 +-
 tests/qapi-schema/flat-union-base-star.err         |   2 +-
 tests/qapi-schema/flat-union-base-star.json        |   2 +-
 tests/qapi-schema/flat-union-reverse-define.out    |  22 +-
 tests/qapi-schema/ident-with-escape.out            |   8 +-
 tests/qapi-schema/include-relpath.out              |   5 +-
 tests/qapi-schema/include-repetition.out           |   5 +-
 tests/qapi-schema/include-simple.out               |   5 +-
 tests/qapi-schema/indented-expr.out                |   8 +-
 tests/qapi-schema/qapi-schema-test.json            |  33 +-
 tests/qapi-schema/qapi-schema-test.out             | 187 ++++--
 tests/qapi-schema/returns-int.out                  |   6 +-
 tests/qapi-schema/test-qapi.py                     |  37 +-
 tests/qapi-schema/type-bypass-no-gen.err           |   1 -
 tests/qapi-schema/type-bypass-no-gen.json          |   2 -
 tests/qapi-schema/type-bypass.exit                 |   1 -
 tests/qapi-schema/type-bypass.json                 |   2 -
 tests/qapi-schema/type-bypass.out                  |   3 -
 tests/test-qmp-event.c                             |   2 +-
 tests/test-qmp-input-strict.c                      |   2 +-
 tests/test-qmp-input-visitor.c                     |  82 ++-
 tests/test-qmp-output-visitor.c                    |  55 +-
 64 files changed, 2011 insertions(+), 1247 deletions(-)
 create mode 100644 qapi/introspect.json
 create mode 100644 scripts/qapi-introspect.py
 create mode 100644 tests/qapi-schema/args-alternate.err
 rename tests/qapi-schema/{type-bypass-no-gen.exit => args-alternate.exit} (100%)
 create mode 100644 tests/qapi-schema/args-alternate.json
 rename tests/qapi-schema/{type-bypass-no-gen.out => args-alternate.out} (100%)
 create mode 100644 tests/qapi-schema/args-returns-any.err
 create mode 100644 tests/qapi-schema/args-returns-any.exit
 create mode 100644 tests/qapi-schema/args-returns-any.json
 rename tests/qapi-schema/{type-bypass.err => args-returns-any.out} (100%)
 create mode 100644 tests/qapi-schema/args-union.err
 create mode 100644 tests/qapi-schema/args-union.exit
 create mode 100644 tests/qapi-schema/args-union.json
 create mode 100644 tests/qapi-schema/args-union.out
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.err
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.json
 delete mode 100644 tests/qapi-schema/type-bypass.exit
 delete mode 100644 tests/qapi-schema/type-bypass.json
 delete mode 100644 tests/qapi-schema/type-bypass.out

-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 01/47] qapi: Clarify docs on including the same file multiple times

[Markus Armbruster]

It's idempotent.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 docs/qapi-code-gen.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index 61b5be4..17588c1 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -163,7 +163,7 @@ The QAPI schema definitions can be modularized using the 'include' directive:
 
 The directive is evaluated recursively, and include paths are relative to the
 file using the directive. Multiple includes of the same file are
-safe.  No other keys should appear in the expression, and the include
+idempotent.  No other keys should appear in the expression, and the include
 value should be a string.
 
 As a matter of style, it is a good idea to have all files be
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 02/47] qapi: Clean up cgen() and mcgen()

[Markus Armbruster]

Commit 05dfb26 added eatspace stripping to mcgen().  Move it to
cgen(), just in case somebody gets tempted to use cgen() directly
instead of via mcgen().

cgen() indents blank lines.  No such lines get generated right now,
but fix it anyway.

We use triple-quoted strings for program text, like this:

    '''
    Program text
    any number of lines
    '''

Keeps the program text relatively readable, but puts an extra newline
at either end.  mcgen() "fixes" that by dropping the first and last
line outright.  Drop only the newlines.

This unmasks a bug in qapi-commands.py: four quotes instead of three.
Fix it up.

Output doesn't change

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py |  2 +-
 scripts/qapi.py          | 19 ++++++++++++-------
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index ca22acc..ce51408 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -56,7 +56,7 @@ def gen_sync_call(name, args, ret_type, indent=0):
                 name=c_name(name), args=arglist, retval=retval).rstrip()
     if ret_type:
         ret += "\n" + gen_err_check('local_err')
-        ret += "\n" + mcgen(''''
+        ret += "\n" + mcgen('''
 %(marshal_output_call)s
 ''',
                             marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip()
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 06d7fc2..20383ef 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -943,16 +943,21 @@ def pop_indent(indent_amount=4):
     global indent_level
     indent_level -= indent_amount
 
+# Generate @code with @kwds interpolated.
+# Obey indent_level, and strip eatspace.
 def cgen(code, **kwds):
-    indent = genindent(indent_level)
-    lines = code.split('\n')
-    lines = map(lambda x: indent + x, lines)
-    return '\n'.join(lines) % kwds + '\n'
-
-def mcgen(code, **kwds):
-    raw = cgen('\n'.join(code.split('\n')[1:-1]), **kwds)
+    raw = code % kwds
+    if indent_level:
+        indent = genindent(indent_level)
+        raw = re.subn("^.", indent + '\g<0>', raw, 0, re.MULTILINE)
+        raw = raw[0]
     return re.sub(re.escape(eatspace) + ' *', '', raw)
 
+def mcgen(code, **kwds):
+    if code[0] == '\n':
+        code = code[1:]
+    return cgen(code, **kwds)
+
 def basename(filename):
     return filename.split("/")[-1]
 
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 03/47] qapi: Simplify guardname()

[Markus Armbruster]

The guards around built-in declarations lose their _H.  It never made
much sense anyway.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi.py | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 20383ef..2bbc8ff 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -958,14 +958,9 @@ def mcgen(code, **kwds):
         code = code[1:]
     return cgen(code, **kwds)
 
-def basename(filename):
-    return filename.split("/")[-1]
 
 def guardname(filename):
-    guard = basename(filename).rsplit(".", 1)[0]
-    for substr in [".", " ", "-"]:
-        guard = guard.replace(substr, "_")
-    return guard.upper() + '_H'
+    return c_name(filename, protect=False).upper()
 
 def guardstart(name):
     return mcgen('''
@@ -1035,6 +1030,7 @@ def parse_command_line(extra_options = "", extra_long_options = []):
 
 def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
                 c_comment, h_comment):
+    guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
 
@@ -1067,7 +1063,7 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
 #define %(guard)s
 
 ''',
-                      comment = h_comment, guard = guardname(h_file)))
+                      comment = h_comment, guard = guard))
 
     return (fdef, fdecl)
 
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 04/47] qapi-event: Clean up how name of enum QAPIEvent is made

[Markus Armbruster]

Use c_name() instead of ad hoc code.  Doesn't upcase the -p prefix,
which is an improvement in my book.  Unbreaks prefix containing '.',
but other funny characters remain broken.  To be fixed next.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-event.py  | 2 +-
 tests/test-qmp-event.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 56bc602..cc74f4d 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -267,7 +267,7 @@ fdecl.write(mcgen('''
 
 exprs = parse_schema(input_file)
 
-event_enum_name = prefix.upper().replace('-', '_') + "QAPIEvent"
+event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
 event_enum_values = []
 event_enum_strings = []
 
diff --git a/tests/test-qmp-event.c b/tests/test-qmp-event.c
index 1ee40e1..28f146d 100644
--- a/tests/test-qmp-event.c
+++ b/tests/test-qmp-event.c
@@ -94,7 +94,7 @@ static bool qdict_cmp_simple(QDict *a, QDict *b)
 
 /* This function is hooked as final emit function, which can verify the
    correctness. */
-static void event_test_emit(TEST_QAPIEvent event, QDict *d, Error **errp)
+static void event_test_emit(test_QAPIEvent event, QDict *d, Error **errp)
 {
     QObject *obj;
     QDict *t;
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py

[Markus Armbruster]

qapi-event.py breaks when you ask for a funny prefix like '@'.
Protect it.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 2bbc8ff..ea94ce5 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []):
     for oa in opts:
         o, a = oa
         if o in ("-p", "--prefix"):
+            match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)
+            if match.end() != len(a):
+                print >>sys.stderr, \
+                    "%s: 'funny character '%s' in argument of -prefix" \
+                    % (sys.argv[0], a[match.end()])
+                sys.exit(1)
             prefix = a
         elif o in ("-o", "--output-dir"):
             output_dir = a + "/"
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 06/47] qapi: Drop unused and useless parameters and variables

[Markus Armbruster]

gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it
only as optional argument for push_indent() and pop_indent(), their
default is four, and gen_sync_call()'s only caller passes four.

gen_visitor_input_containers_decl()'s parameter obj is always
"QOBJECT(args)".

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 27 +++++++++++++--------------
 scripts/qapi-event.py    |  1 -
 scripts/qapi-types.py    |  1 -
 scripts/qapi-visit.py    | 47 ++++++++++++++++++++++-------------------------
 scripts/qapi.py          |  1 -
 5 files changed, 35 insertions(+), 42 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index ce51408..69029f5 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -38,7 +38,7 @@ if (local_err) {
 ''')
     return ''
 
-def gen_sync_call(name, args, ret_type, indent=0):
+def gen_sync_call(name, args, ret_type):
     ret = ""
     arglist=""
     retval=""
@@ -48,7 +48,7 @@ def gen_sync_call(name, args, ret_type, indent=0):
         if optional:
             arglist += "has_%s, " % c_name(argname)
         arglist += "%s, " % (c_name(argname))
-    push_indent(indent)
+    push_indent()
     ret = mcgen('''
 %(retval)sqmp_%(name)s(%(args)s&local_err);
 
@@ -60,7 +60,7 @@ def gen_sync_call(name, args, ret_type, indent=0):
 %(marshal_output_call)s
 ''',
                             marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip()
-    pop_indent(indent)
+    pop_indent()
     return ret.rstrip()
 
 
@@ -69,17 +69,16 @@ def gen_marshal_output_call(name, ret_type):
         return ""
     return "qmp_marshal_output_%s(retval, ret, &local_err);" % c_name(name)
 
-def gen_visitor_input_containers_decl(args, obj):
+def gen_visitor_input_containers_decl(args):
     ret = ""
 
     push_indent()
     if len(args) > 0:
         ret += mcgen('''
-QmpInputVisitor *mi = qmp_input_visitor_new_strict(%(obj)s);
+QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
 QapiDeallocVisitor *md;
 Visitor *v;
-''',
-                     obj=obj)
+''')
     pop_indent()
 
     return ret.rstrip()
@@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md);
     pop_indent()
     return ret.rstrip()
 
-def gen_marshal_output(name, args, ret_type, middle_mode):
+def gen_marshal_output(name, ret_type):
     if not ret_type:
         return ""
 
@@ -194,14 +193,14 @@ out:
 
     return ret
 
-def gen_marshal_input_decl(name, args, ret_type, middle_mode):
+def gen_marshal_input_decl(name, middle_mode):
     ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
     if not middle_mode:
         ret = "static " + ret
     return ret
 
 def gen_marshal_input(name, args, ret_type, middle_mode):
-    hdr = gen_marshal_input_decl(name, args, ret_type, middle_mode)
+    hdr = gen_marshal_input_decl(name, middle_mode)
 
     ret = mcgen('''
 %(header)s
@@ -228,7 +227,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
 %(visitor_input_block)s
 
 ''',
-                     visitor_input_containers_decl=gen_visitor_input_containers_decl(args, "QOBJECT(args)"),
+                     visitor_input_containers_decl=gen_visitor_input_containers_decl(args),
                      visitor_input_vars_decl=gen_visitor_input_vars_decl(args),
                      visitor_input_block=gen_visitor_input_block(args))
     else:
@@ -240,7 +239,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
     ret += mcgen('''
 %(sync_call)s
 ''',
-                 sync_call=gen_sync_call(name, args, ret_type, indent=4))
+                 sync_call=gen_sync_call(name, args, ret_type))
     if re.search('^ *goto out\\;', ret, re.MULTILINE):
         ret += mcgen('''
 
@@ -360,11 +359,11 @@ for cmd in commands:
     ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
     fdecl.write(ret)
     if ret_type:
-        ret = gen_marshal_output(cmd['command'], arglist, ret_type, middle_mode) + "\n"
+        ret = gen_marshal_output(cmd['command'], ret_type) + "\n"
         fdef.write(ret)
 
     if middle_mode:
-        fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], arglist, ret_type, middle_mode))
+        fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode))
 
     ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n"
     fdef.write(ret)
diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index cc74f4d..88b0620 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = {
 ''',
                 event_enum_name = event_enum_name)
 
-    i = 0
     for string in event_enum_strings:
         ret += mcgen('''
     "%(string)s",
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index e6eb4b6..4902440 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -108,7 +108,6 @@ def generate_enum_lookup(name, values):
 const char * const %(name)s_lookup[] = {
 ''',
                 name=c_name(name))
-    i = 0
     for value in values:
         index = c_enum_const(name, value)
         ret += mcgen('''
diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 5b99336..e8ee268 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -40,7 +40,6 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
                  c_type=type_name(type))
 
 def generate_visit_struct_fields(name, members, base = None):
-    substructs = []
     ret = ''
 
     if base:
@@ -103,7 +102,7 @@ out:
     return ret
 
 
-def generate_visit_struct_body(name, members):
+def generate_visit_struct_body(name):
     ret = mcgen('''
     Error *err = NULL;
 
@@ -135,14 +134,14 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 ''',
                  name=c_name(name))
 
-    ret += generate_visit_struct_body(name, members)
+    ret += generate_visit_struct_body(name)
 
     ret += mcgen('''
 }
 ''')
     return ret
 
-def generate_visit_list(name, members):
+def generate_visit_list(name):
     return mcgen('''
 
 void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp)
@@ -171,7 +170,7 @@ out:
 ''',
                 name=type_name(name))
 
-def generate_visit_enum(name, members):
+def generate_visit_enum(name):
     return mcgen('''
 
 void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp)
@@ -252,7 +251,7 @@ def generate_visit_union(expr):
     else:
         # There will always be a discriminator in the C switch code, by default
         # it is an enum type generated silently
-        ret = generate_visit_enum(name + 'Kind', members.keys())
+        ret = generate_visit_enum(name + 'Kind')
         disc_type = c_name(name) + 'Kind'
 
     if base:
@@ -340,7 +339,7 @@ out:
 
     return ret
 
-def generate_declaration(name, members, builtin_type=False):
+def generate_declaration(name, builtin_type=False):
     ret = ""
     if not builtin_type:
         name = c_name(name)
@@ -357,7 +356,7 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
 
     return ret
 
-def generate_enum_declaration(name, members):
+def generate_enum_declaration(name):
     ret = mcgen('''
 void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
 ''',
@@ -365,7 +364,7 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
 
     return ret
 
-def generate_decl_enum(name, members):
+def generate_decl_enum(name):
     return mcgen('''
 
 void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
@@ -433,7 +432,7 @@ exprs = parse_schema(input_file)
 # for built-in types in our header files and simply guard them
 fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
 for typename in builtin_types.keys():
-    fdecl.write(generate_declaration(typename, None, builtin_type=True))
+    fdecl.write(generate_declaration(typename, builtin_type=True))
 fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
 
 # ...this doesn't work for cases where we link in multiple objects that
@@ -441,44 +440,42 @@ fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
 # over these cases
 if do_builtins:
     for typename in builtin_types.keys():
-        fdef.write(generate_visit_list(typename, None))
+        fdef.write(generate_visit_list(typename))
 
 for expr in exprs:
     if expr.has_key('struct'):
         ret = generate_visit_struct(expr)
-        ret += generate_visit_list(expr['struct'], expr['data'])
+        ret += generate_visit_list(expr['struct'])
         fdef.write(ret)
 
-        ret = generate_declaration(expr['struct'], expr['data'])
+        ret = generate_declaration(expr['struct'])
         fdecl.write(ret)
     elif expr.has_key('union'):
         ret = generate_visit_union(expr)
-        ret += generate_visit_list(expr['union'], expr['data'])
+        ret += generate_visit_list(expr['union'])
         fdef.write(ret)
 
         enum_define = discriminator_find_enum_define(expr)
         ret = ""
         if not enum_define:
-            ret = generate_decl_enum('%sKind' % expr['union'],
-                                     expr['data'].keys())
-        ret += generate_declaration(expr['union'], expr['data'])
+            ret = generate_decl_enum('%sKind' % expr['union'])
+        ret += generate_declaration(expr['union'])
         fdecl.write(ret)
     elif expr.has_key('alternate'):
         ret = generate_visit_alternate(expr['alternate'], expr['data'])
-        ret += generate_visit_list(expr['alternate'], expr['data'])
+        ret += generate_visit_list(expr['alternate'])
         fdef.write(ret)
 
-        ret = generate_decl_enum('%sKind' % expr['alternate'],
-                                 expr['data'].keys())
-        ret += generate_declaration(expr['alternate'], expr['data'])
+        ret = generate_decl_enum('%sKind' % expr['alternate'])
+        ret += generate_declaration(expr['alternate'])
         fdecl.write(ret)
     elif expr.has_key('enum'):
-        ret = generate_visit_list(expr['enum'], expr['data'])
-        ret += generate_visit_enum(expr['enum'], expr['data'])
+        ret = generate_visit_list(expr['enum'])
+        ret += generate_visit_enum(expr['enum'])
         fdef.write(ret)
 
-        ret = generate_decl_enum(expr['enum'], expr['data'])
-        ret += generate_enum_declaration(expr['enum'], expr['data'])
+        ret = generate_decl_enum(expr['enum'])
+        ret += generate_enum_declaration(expr['enum'])
         fdecl.write(ret)
 
 close_output(fdef, fdecl)
diff --git a/scripts/qapi.py b/scripts/qapi.py
index ea94ce5..99e04f6 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -507,7 +507,6 @@ def check_command(expr, expr_info):
 def check_event(expr, expr_info):
     global events
     name = expr['event']
-    params = expr.get('data')
 
     if name.upper() == 'MAX':
         raise QAPIExprError(expr_info, "Event name 'MAX' cannot be created")
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Markus Armbruster]

The struct generated for a flat union is weird: the members of its
base are at the end, except for the union tag, which is renamed to
'kind' and put at the beginning.

Example: qapi-schema-test.json has

    { 'struct': 'UserDefUnionBase',
      'data': { 'string': 'str', 'enum1': 'EnumOne' } }

    { 'union': 'UserDefFlatUnion',
      'base': 'UserDefUnionBase',
      'discriminator': 'enum1',
      'data': { 'value1' : 'UserDefA',
                'value2' : 'UserDefB',
                'value3' : 'UserDefB' } }

We generate:

    struct UserDefFlatUnion
    {
        EnumOne kind;
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
        char *string;
    };

Change to put all base members at the beginning, unadulterated.  Not
only is this easier to understand, it also permits casting the flat
union to its base, if that should become useful.

We now generate:

    struct UserDefFlatUnion
    {
        char *string;
        EnumOne enum1;
        /* union tag is EnumOne enum1 */
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
    };

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-types.py           | 32 ++++++++++++++++++--------------
 scripts/qapi-visit.py           |  7 +++++--
 tests/test-qmp-input-visitor.c  |  2 +-
 tests/test-qmp-output-visitor.c |  2 +-
 4 files changed, 25 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index 4902440..c6c2786 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -200,12 +200,27 @@ def generate_union(expr, meta):
     ret = mcgen('''
 struct %(name)s
 {
+''',
+                name=name)
+    if base:
+        base_fields = find_struct(base)['data']
+        ret += generate_struct_fields(base_fields)
+        ret += mcgen('''
+    /* union tag is %(discriminator_type_name)s %(c_name)s */
+''',
+                     discriminator_type_name=c_name(discriminator_type_name),
+                     c_name=c_name(discriminator))
+    else:
+        assert not discriminator
+        ret += mcgen('''
     %(discriminator_type_name)s kind;
+''',
+                     discriminator_type_name=c_name(discriminator_type_name))
+
+    ret += mcgen('''
     union {
         void *data;
-''',
-                name=name,
-                discriminator_type_name=c_name(discriminator_type_name))
+''')
 
     for key in typeinfo:
         ret += mcgen('''
@@ -216,17 +231,6 @@ struct %(name)s
 
     ret += mcgen('''
     };
-''')
-
-    if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_struct_fields(base_fields)
-    else:
-        assert not discriminator
-
-    ret += mcgen('''
 };
 ''')
     if meta == 'alternate':
diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index e8ee268..4ec79a6 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -288,20 +288,23 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
                      name=c_name(name))
 
     if not discriminator:
+        tag = 'kind'
         disc_key = "type"
     else:
+        tag = discriminator
         disc_key = discriminator
     ret += mcgen('''
-        visit_type_%(disc_type)s(m, &(*obj)->kind, "%(disc_key)s", &err);
+        visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err);
         if (err) {
             goto out_obj;
         }
         if (!visit_start_union(m, !!(*obj)->data, &err) || err) {
             goto out_obj;
         }
-        switch ((*obj)->kind) {
+        switch ((*obj)->%(c_tag)s) {
 ''',
                  disc_type = disc_type,
+                 c_tag=c_name(tag),
                  disc_key = disc_key)
 
     for key in members:
diff --git a/tests/test-qmp-input-visitor.c b/tests/test-qmp-input-visitor.c
index b961953..b7a87ee 100644
--- a/tests/test-qmp-input-visitor.c
+++ b/tests/test-qmp-input-visitor.c
@@ -313,7 +313,7 @@ static void test_visitor_in_union_flat(TestInputVisitorData *data,
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(err == NULL);
-    g_assert_cmpint(tmp->kind, ==, ENUM_ONE_VALUE1);
+    g_assert_cmpint(tmp->enum1, ==, ENUM_ONE_VALUE1);
     g_assert_cmpstr(tmp->string, ==, "str");
     /* TODO g_assert_cmpint(tmp->integer, ==, 41); */
     g_assert_cmpint(tmp->value1->boolean, ==, true);
diff --git a/tests/test-qmp-output-visitor.c b/tests/test-qmp-output-visitor.c
index 87ba350..338ada0 100644
--- a/tests/test-qmp-output-visitor.c
+++ b/tests/test-qmp-output-visitor.c
@@ -437,7 +437,7 @@ static void test_visitor_out_union_flat(TestOutputVisitorData *data,
     Error *err = NULL;
 
     UserDefFlatUnion *tmp = g_malloc0(sizeof(UserDefFlatUnion));
-    tmp->kind = ENUM_ONE_VALUE1;
+    tmp->enum1 = ENUM_ONE_VALUE1;
     tmp->string = g_strdup("str");
     tmp->value1 = g_malloc0(sizeof(UserDefA));
     /* TODO when generator bug is fixed: tmp->integer = 41; */
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 08/47] qapi-visit: Fix generated code when schema has forward refs

[Markus Armbruster]

The visit_type_implicit_FOO() are generated on demand, right before
their first use.  Used by visit_type_STRUCT_fields() when STRUCT has
base FOO, and by visit_type_UNION() when flat UNION has member a FOO.

If the schema defines FOO after its first use as struct base or flat
union member, visit_type_implicit_FOO() calls
visit_type_implicit_FOO() before its definition, which doesn't
compile.

Rearrange qapi-schema-test.json to demonstrate the bug.

Fix by generating the necessary forward declaration.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-visit.py                   | 15 ++++++++++++++-
 tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++-------------
 tests/qapi-schema/qapi-schema-test.out  | 10 +++++-----
 3 files changed, 36 insertions(+), 19 deletions(-)

diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 4ec79a6..b3a308f 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -17,13 +17,23 @@ from qapi import *
 import re
 
 implicit_structs = []
+struct_fields_seen = set()
 
 def generate_visit_implicit_struct(type):
     global implicit_structs
     if type in implicit_structs:
         return ''
     implicit_structs.append(type)
-    return mcgen('''
+    ret = ''
+    if type not in struct_fields_seen:
+        # Need a forward declaration
+        ret += mcgen('''
+
+static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
+''',
+                     c_type=type_name(type))
+
+    ret += mcgen('''
 
 static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp)
 {
@@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
 }
 ''',
                  c_type=type_name(type))
+    return ret
 
 def generate_visit_struct_fields(name, members, base = None):
+    struct_fields_seen.add(name)
+
     ret = ''
 
     if base:
diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index c7eaa86..ccadb13 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -7,13 +7,13 @@
   'data': { 'enum1': 'EnumOne', '*enum2': 'EnumOne', 'enum3': 'EnumOne', '*enum4': 'EnumOne' } }
 
 # for testing nested structs
-{ 'struct': 'UserDefZero',
-  'data': { 'integer': 'int' } }
-
 { 'struct': 'UserDefOne',
-  'base': 'UserDefZero',
+  'base': 'UserDefZero',        # intentional forward reference
   'data': { 'string': 'str', '*enum1': 'EnumOne' } }
 
+{ 'struct': 'UserDefZero',
+  'data': { 'integer': 'int' } }
+
 { 'struct': 'UserDefTwoDictDict',
   'data': { 'userdef': 'UserDefOne', 'string': 'str' } }
 
@@ -33,29 +33,33 @@
 { 'struct': 'UserDefB',
   'data': { 'integer': 'int' } }
 
-{ 'struct': 'UserDefC',
-  'data': { 'string1': 'str', 'string2': 'str' } }
-
-{ 'struct': 'UserDefUnionBase',
-  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
-
 { 'union': 'UserDefFlatUnion',
-  'base': 'UserDefUnionBase',
+  'base': 'UserDefUnionBase',   # intentional forward reference
   'discriminator': 'enum1',
-  'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', 'value3' : 'UserDefB' } }
+  'data': { 'value1' : 'UserDefA',
+            'value2' : 'UserDefB',
+            'value3' : 'UserDefB' } }
 # FIXME generated struct UserDefFlatUnion has members for direct base
 # UserDefOne, but lacks members for indirect base UserDefZero
 
+{ 'struct': 'UserDefUnionBase',
+  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
+
 # this variant of UserDefFlatUnion defaults to a union that uses fields with
 # allocated types to test corner cases in the cleanup/dealloc visitor
 { 'union': 'UserDefFlatUnion2',
   'base': 'UserDefUnionBase',
   'discriminator': 'enum1',
-  'data': { 'value1' : 'UserDefC', 'value2' : 'UserDefB', 'value3' : 'UserDefA' } }
+  'data': { 'value1' : 'UserDefC', # intentional forward reference
+            'value2' : 'UserDefB',
+            'value3' : 'UserDefA' } }
 
 { 'alternate': 'UserDefAlternate',
   'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } }
 
+{ 'struct': 'UserDefC',
+  'data': { 'string1': 'str', 'string2': 'str' } }
+
 # for testing native lists
 { 'union': 'UserDefNativeListUnion',
   'data': { 'integer': ['int'],
diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
index cf0ccc4..a4291db 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -1,17 +1,17 @@
 [OrderedDict([('enum', 'EnumOne'), ('data', ['value1', 'value2', 'value3'])]),
  OrderedDict([('struct', 'NestedEnumsOne'), ('data', OrderedDict([('enum1', 'EnumOne'), ('*enum2', 'EnumOne'), ('enum3', 'EnumOne'), ('*enum4', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
  OrderedDict([('struct', 'UserDefOne'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('*enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
  OrderedDict([('struct', 'UserDefTwoDictDict'), ('data', OrderedDict([('userdef', 'UserDefOne'), ('string', 'str')]))]),
  OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
  OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
  OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
  OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]),
- OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
- OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
  OrderedDict([('union', 'UserDefFlatUnion'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefA'), ('value2', 'UserDefB'), ('value3', 'UserDefB')]))]),
+ OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
  OrderedDict([('union', 'UserDefFlatUnion2'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefC'), ('value2', 'UserDefB'), ('value3', 'UserDefA')]))]),
  OrderedDict([('alternate', 'UserDefAlternate'), ('data', OrderedDict([('uda', 'UserDefA'), ('s', 'str'), ('i', 'int')]))]),
+ OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
  OrderedDict([('union', 'UserDefNativeListUnion'), ('data', OrderedDict([('integer', ['int']), ('s8', ['int8']), ('s16', ['int16']), ('s32', ['int32']), ('s64', ['int64']), ('u8', ['uint8']), ('u16', ['uint16']), ('u32', ['uint32']), ('u64', ['uint64']), ('number', ['number']), ('boolean', ['bool']), ('string', ['str']), ('sizes', ['size'])]))]),
  OrderedDict([('command', 'user_def_cmd'), ('data', OrderedDict())]),
  OrderedDict([('command', 'user_def_cmd1'), ('data', OrderedDict([('ud1a', 'UserDefOne')]))]),
@@ -39,15 +39,15 @@
  {'enum_name': '__org.qemu_x-Union1Kind', 'enum_values': None},
  {'enum_name': '__org.qemu_x-AltKind', 'enum_values': None}]
 [OrderedDict([('struct', 'NestedEnumsOne'), ('data', OrderedDict([('enum1', 'EnumOne'), ('*enum2', 'EnumOne'), ('enum3', 'EnumOne'), ('*enum4', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
  OrderedDict([('struct', 'UserDefOne'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('*enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
  OrderedDict([('struct', 'UserDefTwoDictDict'), ('data', OrderedDict([('userdef', 'UserDefOne'), ('string', 'str')]))]),
  OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
  OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
  OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
  OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]),
- OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
  OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
  OrderedDict([('struct', 'UserDefOptions'), ('data', OrderedDict([('*i64', ['int']), ('*u64', ['uint64']), ('*u16', ['uint16']), ('*i64x', 'int'), ('*u64x', 'uint64')]))]),
  OrderedDict([('struct', 'EventStructOne'), ('data', OrderedDict([('struct1', 'UserDefOne'), ('string', 'str'), ('*enum2', 'EnumOne')]))]),
  OrderedDict([('struct', '__org.qemu_x-Base'), ('data', OrderedDict([('__org.qemu_x-member1', '__org.qemu_x-Enum')]))]),
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 09/47] qapi-visit: Replace list implicit_structs by set

[Markus Armbruster]

Use set because that's what it is.  While there, rename to
implicit_structs_seen.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-visit.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index b3a308f..9fc040e 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -16,14 +16,13 @@ from ordereddict import OrderedDict
 from qapi import *
 import re
 
-implicit_structs = []
+implicit_structs_seen = set()
 struct_fields_seen = set()
 
 def generate_visit_implicit_struct(type):
-    global implicit_structs
-    if type in implicit_structs:
+    if type in implicit_structs_seen:
         return ''
-    implicit_structs.append(type)
+    implicit_structs_seen.add(type)
     ret = ''
     if type not in struct_fields_seen:
         # Need a forward declaration
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 10/47] qapi-visit: Fix two name arguments passed to visitors

[Markus Armbruster]

The generated code passes mangled schema names to visit_type_enum()
and union's visit_start_struct().  Fix it to pass the names
unadulterated, like we do everywhere else.

Only qapi-schema-test.json actually has names where this makes a
difference: enum __org.qemu_x-Enum, flat union __org.qemu_x-Union2,
simple union __org.qemu_x-Union1 and its implicit enum
__org.qemu_x-Union1Kind.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-visit.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 9fc040e..73f136f 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -185,12 +185,12 @@ out:
 def generate_visit_enum(name):
     return mcgen('''
 
-void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp)
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error **errp)
 {
-    visit_type_enum(m, (int *)obj, %(name)s_lookup, "%(name)s", name, errp);
+    visit_type_enum(m, (int *)obj, %(c_name)s_lookup, "%(name)s", name, errp);
 }
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), name=name)
 
 def generate_visit_alternate(name, members):
     ret = mcgen('''
@@ -278,17 +278,17 @@ def generate_visit_union(expr):
 
     ret += mcgen('''
 
-void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
 
-    visit_start_struct(m, (void **)obj, "%(name)s", name, sizeof(%(name)s), &err);
+    visit_start_struct(m, (void **)obj, "%(name)s", name, sizeof(%(c_name)s), &err);
     if (err) {
         goto out;
     }
     if (*obj) {
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), name=name)
 
     if base:
         ret += mcgen('''
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 11/47] tests/qapi-schema: Document alternate's enum lacks visit function

[Markus Armbruster]

We generate a declaration, but no definition.

The QMP schema has two: Qcow2OverlapChecks and BlockdevRef.  Neither
visit_type_Qcow2OverlapChecksKind() nor visit_type_BlockdevRefKind()
is actually used.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 tests/qapi-schema/qapi-schema-test.json | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index ccadb13..8337ba9 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -56,6 +56,7 @@
 
 { 'alternate': 'UserDefAlternate',
   'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } }
+# FIXME only a declaration of visit_type_UserDefAlternateKind() generated
 
 { 'struct': 'UserDefC',
   'data': { 'string1': 'str', 'string2': 'str' } }
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 12/47] tests/qapi-schema: Document events with with base don't work

[Markus Armbruster]

When event FOO's 'data' is a struct with a base, we consider only the
struct's direct members, and ignore its base.  The generated
qapi_event_send_foo() doesn't take arguments for base members.

No such events currently exist in the QMP schema.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 tests/qapi-schema/qapi-schema-test.json | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index 8337ba9..829dd30 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -128,6 +128,9 @@
 { 'alternate': '__org.qemu_x-Alt',
   'data': { '__org.qemu_x-branch': 'str', 'b': '__org.qemu_x-Base' } }
 { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
+# FIXME generated qapi_event_send___org_qemu_x_event() has only a
+# parameter for data's member __org_qemu_x_member2, none for its base
+# __org.qemu_x-Base's member __org_qemu_x_member1
 { 'command': '__org.qemu_x-command',
   'data': { 'a': ['__org.qemu_x-Enum'], 'b': ['__org.qemu_x-Struct'],
             'c': '__org.qemu_x-Union2', 'd': '__org.qemu_x-Alt' },
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 13/47] tests/qapi-schema: Restore test case for flat union base bug

[Markus Armbruster]

Test case added in commit 2fc0043, and messed up in commit 5223070.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 tests/qapi-schema/qapi-schema-test.json | 5 +++--
 tests/qapi-schema/qapi-schema-test.out  | 8 ++++----
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index 829dd30..a9e5aab 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -31,7 +31,7 @@
   'data': { 'boolean': 'bool' } }
 
 { 'struct': 'UserDefB',
-  'data': { 'integer': 'int' } }
+  'data': { 'intb': 'int' } }
 
 { 'union': 'UserDefFlatUnion',
   'base': 'UserDefUnionBase',   # intentional forward reference
@@ -40,9 +40,10 @@
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
 # FIXME generated struct UserDefFlatUnion has members for direct base
-# UserDefOne, but lacks members for indirect base UserDefZero
+# UserDefUnionBase, but lacks members for indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
+  'base': 'UserDefZero',
   'data': { 'string': 'str', 'enum1': 'EnumOne' } }
 
 # this variant of UserDefFlatUnion defaults to a union that uses fields with
diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
index a4291db..b0b7187 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -6,9 +6,9 @@
  OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
  OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
  OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
- OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]),
+ OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('intb', 'int')]))]),
  OrderedDict([('union', 'UserDefFlatUnion'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefA'), ('value2', 'UserDefB'), ('value3', 'UserDefB')]))]),
- OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefUnionBase'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
  OrderedDict([('union', 'UserDefFlatUnion2'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefC'), ('value2', 'UserDefB'), ('value3', 'UserDefA')]))]),
  OrderedDict([('alternate', 'UserDefAlternate'), ('data', OrderedDict([('uda', 'UserDefA'), ('s', 'str'), ('i', 'int')]))]),
  OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
@@ -45,8 +45,8 @@
  OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
  OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
  OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
- OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]),
- OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('intb', 'int')]))]),
+ OrderedDict([('struct', 'UserDefUnionBase'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
  OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
  OrderedDict([('struct', 'UserDefOptions'), ('data', OrderedDict([('*i64', ['int']), ('*u64', ['uint64']), ('*u16', ['uint16']), ('*i64x', 'int'), ('*u64x', 'uint64')]))]),
  OrderedDict([('struct', 'EventStructOne'), ('data', OrderedDict([('struct1', 'UserDefOne'), ('string', 'str'), ('*enum2', 'EnumOne')]))]),
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

[Markus Armbruster]

A command's 'data' must be a struct type, given either as a
dictionary, or as struct type name.

Existing test case data-int.json covers simple type 'int'.  Add test
cases for type names referring to union and alternate types.

The latter is caught (good), but the former is not (bug).

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 tests/Makefile                        | 3 ++-
 tests/qapi-schema/args-alternate.err  | 1 +
 tests/qapi-schema/args-alternate.exit | 1 +
 tests/qapi-schema/args-alternate.json | 4 ++++
 tests/qapi-schema/args-alternate.out  | 0
 tests/qapi-schema/args-union.err      | 0
 tests/qapi-schema/args-union.exit     | 1 +
 tests/qapi-schema/args-union.json     | 4 ++++
 tests/qapi-schema/args-union.out      | 4 ++++
 9 files changed, 17 insertions(+), 1 deletion(-)
 create mode 100644 tests/qapi-schema/args-alternate.err
 create mode 100644 tests/qapi-schema/args-alternate.exit
 create mode 100644 tests/qapi-schema/args-alternate.json
 create mode 100644 tests/qapi-schema/args-alternate.out
 create mode 100644 tests/qapi-schema/args-union.err
 create mode 100644 tests/qapi-schema/args-union.exit
 create mode 100644 tests/qapi-schema/args-union.json
 create mode 100644 tests/qapi-schema/args-union.out

diff --git a/tests/Makefile b/tests/Makefile
index eff5e11..5a38748 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -225,7 +225,8 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \
 	data-array-empty.json data-array-unknown.json data-int.json \
 	data-unknown.json data-member-unknown.json data-member-array.json \
-	data-member-array-bad.json returns-array-bad.json returns-int.json \
+	data-member-array-bad.json args-union.json args-alternate.json \
+	returns-array-bad.json returns-int.json \
 	returns-unknown.json returns-alternate.json returns-whitelist.json \
 	missing-colon.json missing-comma-list.json missing-comma-object.json \
 	nested-struct-data.json nested-struct-returns.json non-objects.json \
diff --git a/tests/qapi-schema/args-alternate.err b/tests/qapi-schema/args-alternate.err
new file mode 100644
index 0000000..fc90a9e
--- /dev/null
+++ b/tests/qapi-schema/args-alternate.err
@@ -0,0 +1 @@
+tests/qapi-schema/args-alternate.json:4: 'data' for command 'oops' cannot use alternate type 'Alt'
diff --git a/tests/qapi-schema/args-alternate.exit b/tests/qapi-schema/args-alternate.exit
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/tests/qapi-schema/args-alternate.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/args-alternate.json b/tests/qapi-schema/args-alternate.json
new file mode 100644
index 0000000..d2f9199
--- /dev/null
+++ b/tests/qapi-schema/args-alternate.json
@@ -0,0 +1,4 @@
+# we do not allow alternate arguments
+# TODO should we support this?
+{ 'alternate': 'Alt', 'data': { 'case1': 'int', 'case2': 'str' } }
+{ 'command': 'oops', 'data': 'Alt' }
diff --git a/tests/qapi-schema/args-alternate.out b/tests/qapi-schema/args-alternate.out
new file mode 100644
index 0000000..e69de29
diff --git a/tests/qapi-schema/args-union.err b/tests/qapi-schema/args-union.err
new file mode 100644
index 0000000..e69de29
diff --git a/tests/qapi-schema/args-union.exit b/tests/qapi-schema/args-union.exit
new file mode 100644
index 0000000..573541a
--- /dev/null
+++ b/tests/qapi-schema/args-union.exit
@@ -0,0 +1 @@
+0
diff --git a/tests/qapi-schema/args-union.json b/tests/qapi-schema/args-union.json
new file mode 100644
index 0000000..db97ef6
--- /dev/null
+++ b/tests/qapi-schema/args-union.json
@@ -0,0 +1,4 @@
+# FIXME we should reject union arguments
+# TODO should we support this?
+{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
+{ 'command': 'oops', 'data': 'Uni' }
diff --git a/tests/qapi-schema/args-union.out b/tests/qapi-schema/args-union.out
new file mode 100644
index 0000000..907080c
--- /dev/null
+++ b/tests/qapi-schema/args-union.out
@@ -0,0 +1,4 @@
+[OrderedDict([('union', 'Uni'), ('data', OrderedDict([('case1', 'int'), ('case2', 'str')]))]),
+ OrderedDict([('command', 'oops'), ('data', 'Uni')])]
+[{'enum_name': 'UniKind', 'enum_values': None}]
+[]
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 15/47] qapi: Fix to reject union command arguments

[Markus Armbruster]

A command's 'data' must be a struct type, given either as a
dictionary, or as struct type name.

Commit dd883c6 tightened the checking there, but not enough: we still
accept 'union'.  Fix to reject it.

We may want to support union types there, but we'll have to extend
qapi-commands.py for it.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi.py                   | 2 +-
 tests/qapi-schema/args-union.err  | 1 +
 tests/qapi-schema/args-union.exit | 2 +-
 tests/qapi-schema/args-union.json | 2 +-
 tests/qapi-schema/args-union.out  | 4 ----
 5 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 99e04f6..bebaecc 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -495,7 +495,7 @@ def check_command(expr, expr_info):
 
     check_type(expr_info, "'data' for command '%s'" % name,
                expr.get('data'), allow_dict=True, allow_optional=True,
-               allow_metas=['union', 'struct'], allow_star=allow_star)
+               allow_metas=['struct'], allow_star=allow_star)
     returns_meta = ['union', 'struct']
     if name in returns_whitelist:
         returns_meta += ['built-in', 'alternate', 'enum']
diff --git a/tests/qapi-schema/args-union.err b/tests/qapi-schema/args-union.err
index e69de29..1d693d7 100644
--- a/tests/qapi-schema/args-union.err
+++ b/tests/qapi-schema/args-union.err
@@ -0,0 +1 @@
+tests/qapi-schema/args-union.json:4: 'data' for command 'oops' cannot use union type 'Uni'
diff --git a/tests/qapi-schema/args-union.exit b/tests/qapi-schema/args-union.exit
index 573541a..d00491f 100644
--- a/tests/qapi-schema/args-union.exit
+++ b/tests/qapi-schema/args-union.exit
@@ -1 +1 @@
-0
+1
diff --git a/tests/qapi-schema/args-union.json b/tests/qapi-schema/args-union.json
index db97ef6..7bdcbb7 100644
--- a/tests/qapi-schema/args-union.json
+++ b/tests/qapi-schema/args-union.json
@@ -1,4 +1,4 @@
-# FIXME we should reject union arguments
+# we do not allow union arguments
 # TODO should we support this?
 { 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
 { 'command': 'oops', 'data': 'Uni' }
diff --git a/tests/qapi-schema/args-union.out b/tests/qapi-schema/args-union.out
index 907080c..e69de29 100644
--- a/tests/qapi-schema/args-union.out
+++ b/tests/qapi-schema/args-union.out
@@ -1,4 +0,0 @@
-[OrderedDict([('union', 'Uni'), ('data', OrderedDict([('case1', 'int'), ('case2', 'str')]))]),
- OrderedDict([('command', 'oops'), ('data', 'Uni')])]
-[{'enum_name': 'UniKind', 'enum_values': None}]
-[]
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 16/47] qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'

[Markus Armbruster]

gen_err_check() hard-codes 'local_err' instead of substituting the
argument.  Currently harmless, since all callers pass either None or
'local_err'.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 69029f5..3965ca8 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -29,14 +29,15 @@ def generate_command_decl(name, args, ret_type):
                  ret_type=c_type(ret_type), name=c_name(name),
                  args=arglist).strip()
 
-def gen_err_check(errvar):
-    if errvar:
-        return mcgen('''
-if (local_err) {
+def gen_err_check(err):
+    if not err:
+        return ''
+    return mcgen('''
+if (%(err)s) {
     goto out;
 }
-''')
-    return ''
+''',
+                 err=err)
 
 def gen_sync_call(name, args, ret_type):
     ret = ""
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 17/47] qapi-commands: Inline gen_marshal_output_call()

[Markus Armbruster]

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 3965ca8..6de5229 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -57,19 +57,15 @@ def gen_sync_call(name, args, ret_type):
                 name=c_name(name), args=arglist, retval=retval).rstrip()
     if ret_type:
         ret += "\n" + gen_err_check('local_err')
-        ret += "\n" + mcgen('''
-%(marshal_output_call)s
+        ret += mcgen('''
+
+qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
 ''',
-                            marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip()
+                            c_name=c_name(name))
     pop_indent()
     return ret.rstrip()
 
 
-def gen_marshal_output_call(name, ret_type):
-    if not ret_type:
-        return ""
-    return "qmp_marshal_output_%s(retval, ret, &local_err);" % c_name(name)
-
 def gen_visitor_input_containers_decl(args):
     ret = ""
 
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 18/47] qapi-commands: Don't feed output of mcgen() to mcgen() again

[Markus Armbruster]

Multiple passes through mcgen() is prone to produce unwanted blank
lines, which we then combat by sprinkling .rstrip() on top.  Just
don't do it.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 52 +++++++++++++++++++-----------------------------
 1 file changed, 21 insertions(+), 31 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 6de5229..cfbd59c 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -27,7 +27,7 @@ def generate_command_decl(name, args, ret_type):
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
                  ret_type=c_type(ret_type), name=c_name(name),
-                 args=arglist).strip()
+                 args=arglist)
 
 def gen_err_check(err):
     if not err:
@@ -52,19 +52,17 @@ def gen_sync_call(name, args, ret_type):
     push_indent()
     ret = mcgen('''
 %(retval)sqmp_%(name)s(%(args)s&local_err);
-
 ''',
-                name=c_name(name), args=arglist, retval=retval).rstrip()
+                name=c_name(name), args=arglist, retval=retval)
     if ret_type:
-        ret += "\n" + gen_err_check('local_err')
+        ret += gen_err_check('local_err')
         ret += mcgen('''
 
 qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
 ''',
                             c_name=c_name(name))
     pop_indent()
-    return ret.rstrip()
-
+    return ret
 
 def gen_visitor_input_containers_decl(args):
     ret = ""
@@ -78,7 +76,7 @@ Visitor *v;
 ''')
     pop_indent()
 
-    return ret.rstrip()
+    return ret
 
 def gen_visitor_input_vars_decl(args):
     ret = ""
@@ -101,7 +99,7 @@ bool has_%(argname)s = false;
                          argname=c_name(argname), argtype=c_type(argtype))
 
     pop_indent()
-    return ret.rstrip()
+    return ret
 
 def gen_visitor_input_block(args, dealloc=False):
     ret = ""
@@ -155,7 +153,7 @@ visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s);
 qapi_dealloc_visitor_cleanup(md);
 ''')
     pop_indent()
-    return ret.rstrip()
+    return ret
 
 def gen_marshal_output(name, ret_type):
     if not ret_type:
@@ -217,26 +215,17 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
                      retval=retval)
 
     if len(args) > 0:
-        ret += mcgen('''
-%(visitor_input_containers_decl)s
-%(visitor_input_vars_decl)s
-
-%(visitor_input_block)s
-
-''',
-                     visitor_input_containers_decl=gen_visitor_input_containers_decl(args),
-                     visitor_input_vars_decl=gen_visitor_input_vars_decl(args),
-                     visitor_input_block=gen_visitor_input_block(args))
+        ret += gen_visitor_input_containers_decl(args)
+        ret += gen_visitor_input_vars_decl(args) + '\n'
+        ret += gen_visitor_input_block(args) + '\n'
     else:
         ret += mcgen('''
 
     (void)args;
 ''')
 
-    ret += mcgen('''
-%(sync_call)s
-''',
-                 sync_call=gen_sync_call(name, args, ret_type))
+    ret += gen_sync_call(name, args, ret_type)
+
     if re.search('^ *goto out\\;', ret, re.MULTILINE):
         ret += mcgen('''
 
@@ -244,11 +233,11 @@ out:
 ''')
     ret += mcgen('''
     error_propagate(errp, local_err);
-%(visitor_input_block_cleanup)s
+''')
+    ret += gen_visitor_input_block(args, dealloc=True)
+    ret += mcgen('''
 }
-''',
-                 visitor_input_block_cleanup=gen_visitor_input_block(args,
-                                                                     dealloc=True))
+''')
     return ret
 
 def gen_registry(commands):
@@ -268,12 +257,13 @@ qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
     ret = mcgen('''
 static void qmp_init_marshal(void)
 {
-%(registry)s
+''')
+    ret += registry
+    ret += mcgen('''
 }
 
 qapi_init(qmp_init_marshal);
-''',
-                registry=registry.rstrip())
+''')
     return ret
 
 middle_mode = False
@@ -353,7 +343,7 @@ for cmd in commands:
         arglist = cmd['data']
     if cmd.has_key('returns'):
         ret_type = cmd['returns']
-    ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
+    ret = generate_command_decl(cmd['command'], arglist, ret_type)
     fdecl.write(ret)
     if ret_type:
         ret = gen_marshal_output(cmd['command'], ret_type) + "\n"
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

[Markus Armbruster]

Clean up white-space, brace placement, and superfluous

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py |  1 +
 scripts/qapi-event.py    |  3 +--
 scripts/qapi-types.py    | 66 +++++++++++++++++++++++-------------------------
 scripts/qapi-visit.py    |  1 +
 4 files changed, 34 insertions(+), 37 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index cfbd59c..223216d 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
         ret += mcgen('''
 
     (void)args;
+
 ''')
 
     ret += gen_sync_call(name, args, ret_type)
diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 88b0620..7f238df 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[];
                         event_enum_name = event_enum_name)
 
     enum_decl = mcgen('''
-typedef enum %(event_enum_name)s
-{
+typedef enum %(event_enum_name)s {
 ''',
                       event_enum_name = event_enum_name)
 
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index c6c2786..4cc17da 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -15,8 +15,7 @@ from qapi import *
 def generate_fwd_builtin(name):
     return mcgen('''
 
-typedef struct %(name)sList
-{
+typedef struct %(name)sList {
     union {
         %(type)s value;
         uint64_t padding;
@@ -32,8 +31,7 @@ def generate_fwd_struct(name):
 
 typedef struct %(name)s %(name)s;
 
-typedef struct %(name)sList
-{
+typedef struct %(name)sList {
     union {
         %(name)s *value;
         uint64_t padding;
@@ -45,8 +43,8 @@ typedef struct %(name)sList
 
 def generate_fwd_enum_struct(name):
     return mcgen('''
-typedef struct %(name)sList
-{
+
+typedef struct %(name)sList {
     union {
         %(name)s value;
         uint64_t padding;
@@ -79,8 +77,8 @@ def generate_struct(expr):
     base = expr.get('base')
 
     ret = mcgen('''
-struct %(name)s
-{
+
+struct %(name)s {
 ''',
           name=c_name(structname))
 
@@ -105,7 +103,8 @@ struct %(name)s
 
 def generate_enum_lookup(name, values):
     ret = mcgen('''
-const char * const %(name)s_lookup[] = {
+
+const char *const %(name)s_lookup[] = {
 ''',
                 name=c_name(name))
     for value in values:
@@ -119,7 +118,6 @@ const char * const %(name)s_lookup[] = {
     ret += mcgen('''
     [%(max_index)s] = NULL,
 };
-
 ''',
         max_index=max_index)
     return ret
@@ -127,13 +125,14 @@ const char * const %(name)s_lookup[] = {
 def generate_enum(name, values):
     name = c_name(name)
     lookup_decl = mcgen('''
-extern const char * const %(name)s_lookup[];
+
+extern const char *const %(name)s_lookup[];
 ''',
                 name=name)
 
     enum_decl = mcgen('''
-typedef enum %(name)s
-{
+
+typedef enum %(name)s {
 ''',
                 name=name)
 
@@ -155,7 +154,7 @@ typedef enum %(name)s
 ''',
                  name=name)
 
-    return lookup_decl + enum_decl
+    return enum_decl + lookup_decl
 
 def generate_alternate_qtypes(expr):
 
@@ -163,6 +162,7 @@ def generate_alternate_qtypes(expr):
     members = expr['data']
 
     ret = mcgen('''
+
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
@@ -198,8 +198,8 @@ def generate_union(expr, meta):
         discriminator_type_name = '%sKind' % (name)
 
     ret = mcgen('''
-struct %(name)s
-{
+
+struct %(name)s {
 ''',
                 name=name)
     if base:
@@ -317,14 +317,12 @@ fdef.write(mcgen('''
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
-
 ''',
                  prefix=prefix))
 
 fdecl.write(mcgen('''
 #include <stdbool.h>
 #include <stdint.h>
-
 '''))
 
 exprs = parse_schema(input_file)
@@ -335,22 +333,22 @@ for typename in builtin_types.keys():
 fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
 
 for expr in exprs:
-    ret = "\n"
+    ret = ""
     if expr.has_key('struct'):
         ret += generate_fwd_struct(expr['struct'])
     elif expr.has_key('enum'):
-        ret += generate_enum(expr['enum'], expr['data']) + "\n"
+        ret += generate_enum(expr['enum'], expr['data'])
         ret += generate_fwd_enum_struct(expr['enum'])
         fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
     elif expr.has_key('union'):
-        ret += generate_fwd_struct(expr['union']) + "\n"
+        ret += generate_fwd_struct(expr['union'])
         enum_define = discriminator_find_enum_define(expr)
         if not enum_define:
             ret += generate_enum('%sKind' % expr['union'], expr['data'].keys())
             fdef.write(generate_enum_lookup('%sKind' % expr['union'],
                                             expr['data'].keys()))
     elif expr.has_key('alternate'):
-        ret += generate_fwd_struct(expr['alternate']) + "\n"
+        ret += generate_fwd_struct(expr['alternate'])
         ret += generate_enum('%sKind' % expr['alternate'], expr['data'].keys())
         fdef.write(generate_enum_lookup('%sKind' % expr['alternate'],
                                         expr['data'].keys()))
@@ -370,34 +368,32 @@ fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
 # have the functions defined, so we use -b option to provide control
 # over these cases
 if do_builtins:
-    fdef.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))
     for typename in builtin_types.keys():
         fdef.write(generate_type_cleanup(typename + "List"))
-    fdef.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))
 
 for expr in exprs:
-    ret = "\n"
+    ret = ""
     if expr.has_key('struct'):
         ret += generate_struct(expr) + "\n"
         ret += generate_type_cleanup_decl(expr['struct'] + "List")
-        fdef.write(generate_type_cleanup(expr['struct'] + "List") + "\n")
+        fdef.write(generate_type_cleanup(expr['struct'] + "List"))
         ret += generate_type_cleanup_decl(expr['struct'])
-        fdef.write(generate_type_cleanup(expr['struct']) + "\n")
+        fdef.write(generate_type_cleanup(expr['struct']))
     elif expr.has_key('union'):
-        ret += generate_union(expr, 'union')
+        ret += generate_union(expr, 'union') + "\n"
         ret += generate_type_cleanup_decl(expr['union'] + "List")
-        fdef.write(generate_type_cleanup(expr['union'] + "List") + "\n")
+        fdef.write(generate_type_cleanup(expr['union'] + "List"))
         ret += generate_type_cleanup_decl(expr['union'])
-        fdef.write(generate_type_cleanup(expr['union']) + "\n")
+        fdef.write(generate_type_cleanup(expr['union']))
     elif expr.has_key('alternate'):
-        ret += generate_union(expr, 'alternate')
+        ret += generate_union(expr, 'alternate') + "\n"
         ret += generate_type_cleanup_decl(expr['alternate'] + "List")
-        fdef.write(generate_type_cleanup(expr['alternate'] + "List") + "\n")
+        fdef.write(generate_type_cleanup(expr['alternate'] + "List"))
         ret += generate_type_cleanup_decl(expr['alternate'])
-        fdef.write(generate_type_cleanup(expr['alternate']) + "\n")
+        fdef.write(generate_type_cleanup(expr['alternate']))
     elif expr.has_key('enum'):
-        ret += generate_type_cleanup_decl(expr['enum'] + "List")
-        fdef.write(generate_type_cleanup(expr['enum'] + "List") + "\n")
+        ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List")
+        fdef.write(generate_type_cleanup(expr['enum'] + "List"))
     else:
         continue
     fdecl.write(ret)
diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 73f136f..427819a 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -62,6 +62,7 @@ def generate_visit_struct_fields(name, members, base = None):
 static void visit_type_%(name)s_fields(Visitor *m, %(name)s **obj, Error **errp)
 {
     Error *err = NULL;
+
 ''',
                  name=c_name(name))
     push_indent()
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 20/47] qapi: Rename class QAPISchema to QAPISchemaParser

[Markus Armbruster]

I want to name a new class QAPISchema.

While there, make it a new-style class.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index bebaecc..390ccd0 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -103,7 +103,7 @@ class QAPIExprError(Exception):
         return error_path(self.info['parent']) + \
             "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
 
-class QAPISchema:
+class QAPISchemaParser(object):
 
     def __init__(self, fp, previously_included = [], incl_info = None):
         abs_fname = os.path.abspath(fp.name)
@@ -149,8 +149,8 @@ class QAPISchema:
                 except IOError, e:
                     raise QAPIExprError(expr_info,
                                         '%s: %s' % (e.strerror, include))
-                exprs_include = QAPISchema(fobj, previously_included,
-                                           expr_info)
+                exprs_include = QAPISchemaParser(fobj, previously_included,
+                                                 expr_info)
                 self.exprs.extend(exprs_include.exprs)
             else:
                 expr_elem = {'expr': expr,
@@ -755,7 +755,7 @@ def check_exprs(exprs):
 
 def parse_schema(fname):
     try:
-        schema = QAPISchema(open(fname, "r"))
+        schema = QAPISchemaParser(open(fname, "r"))
         return check_exprs(schema.exprs)
     except (QAPISchemaError, QAPIExprError), e:
         print >>sys.stderr, e
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation

[Markus Armbruster]

The QAPI code generators work with a syntax tree (nested dictionaries)
plus a few symbol tables (also dictionaties) on the side.

They have clearly outgrown these simple data structures.  There's lots
of rummaging around in dictionaries, and information is recomputed on
the fly.  For the work I'm going to do, I want more clearly defined
and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the
syntax tree, to make messing with the syntax easier.

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax
tree to create the new internal representation, and finally perform
semantic analysis.

Shortcut: the semantic analysis still relies on existing check_exprs()
to do the actual semantic checking.  All this code needs to move into
the classes.  Mark as TODO.

We generate array types eagerly, even though most of them aren't used.
Mark as TODO.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py       |   2 +-
 scripts/qapi-event.py          |   2 +-
 scripts/qapi-types.py          |   2 +-
 scripts/qapi-visit.py          |   2 +-
 scripts/qapi.py                | 351 +++++++++++++++++++++++++++++++++++++++--
 tests/qapi-schema/test-qapi.py |   2 +-
 6 files changed, 347 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 223216d..3cdbd97 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -276,7 +276,7 @@ for o, a in opts:
     if o in ("-m", "--middle"):
         middle_mode = True
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 commands = filter(lambda expr: expr.has_key('command'), exprs)
 commands = filter(lambda expr: not expr.has_key('gen'), commands)
 
diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 7f238df..aec2d32 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -263,7 +263,7 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
 event_enum_values = []
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index 4cc17da..a48ad9c 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -325,7 +325,7 @@ fdecl.write(mcgen('''
 #include <stdint.h>
 '''))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
 for typename in builtin_types.keys():
diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 427819a..a52a572 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -442,7 +442,7 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 # to avoid header dependency hell, we always generate declarations
 # for built-in types in our header files and simply guard them
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 390ccd0..a80892e 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -302,6 +302,7 @@ class QAPISchemaParser(object):
 
 #
 # Semantic analysis of schema expressions
+# TODO fold into QAPISchema
 #
 
 def find_base_fields(base):
@@ -751,15 +752,347 @@ def check_exprs(exprs):
         else:
             assert False, 'unexpected meta type'
 
-    return map(lambda expr_elem: expr_elem['expr'], exprs)
-
-def parse_schema(fname):
-    try:
-        schema = QAPISchemaParser(open(fname, "r"))
-        return check_exprs(schema.exprs)
-    except (QAPISchemaError, QAPIExprError), e:
-        print >>sys.stderr, e
-        exit(1)
+    return exprs
+
+#
+# Schema compiler frontend
+#
+
+class QAPISchemaEntity(object):
+    def __init__(self, name, info):
+        assert isinstance(name, str)
+        self.name = name
+        self.info = info
+    def check(self, schema):
+        pass
+
+class QAPISchemaType(QAPISchemaEntity):
+    pass
+
+class QAPISchemaBuiltinType(QAPISchemaType):
+    def __init__(self, name):
+        QAPISchemaType.__init__(self, name, None)
+
+class QAPISchemaEnumType(QAPISchemaType):
+    def __init__(self, name, info, values):
+        QAPISchemaType.__init__(self, name, info)
+        for v in values:
+            assert isinstance(v, str)
+        self.values = values
+
+class QAPISchemaArrayType(QAPISchemaType):
+    def __init__(self, name, info, element_type):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(element_type, str)
+        self.element_type_name = element_type
+        self.element_type = None
+    def check(self, schema):
+        self.element_type = schema.lookup_type(self.element_type_name)
+        assert self.element_type
+
+class QAPISchemaObjectType(QAPISchemaType):
+    def __init__(self, name, info, base, local_members, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert base == None or isinstance(base, str)
+        for m in local_members:
+            assert isinstance(m, QAPISchemaObjectTypeMember)
+        if variants != None:
+            assert isinstance(variants, QAPISchemaObjectTypeVariants)
+        self.base_name = base
+        self.base = None
+        self.local_members = local_members
+        self.variants = variants
+        self.members = None
+    def check(self, schema):
+        if self.members:
+            return                      # already checked
+        assert self.members == None     # not running in cycles
+        self.members = False            # mark as being checked
+        if self.base_name:
+            self.base = schema.lookup_type(self.base_name)
+            self.base.check(schema)
+            members = list(self.base.members)
+        else:
+            members = []
+        seen = {}
+        for m in members:
+            seen[m.name] = m
+        for m in self.local_members:
+            m.check(schema, members, seen)
+        if self.variants:
+            self.variants.check(schema, members, seen)
+        self.members = members
+
+class QAPISchemaObjectTypeMember(object):
+    def __init__(self, name, typ, optional):
+        assert isinstance(name, str)
+        assert isinstance(typ, str)
+        assert isinstance(optional, bool)
+        self.name = name
+        self.type_name = typ
+        self.type = None
+        self.optional = optional
+    def check(self, schema, all_members, seen):
+        assert self.name not in seen
+        self.type = schema.lookup_type(self.type_name)
+        assert self.type
+        all_members.append(self)
+        seen[self.name] = self
+
+class QAPISchemaObjectTypeVariants(object):
+    def __init__(self, tag_name, tag_enum, variants):
+        assert tag_name == None or isinstance(tag_name, str)
+        assert tag_enum == None or isinstance(tag_enum, str)
+        for v in variants:
+            assert isinstance(v, QAPISchemaObjectTypeVariant)
+        self.tag_name = tag_name
+        if tag_name:
+            assert not tag_enum
+            self.tag_member = None
+        else:
+            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+                                                         False)
+        self.variants = variants
+    def check(self, schema, members, seen):
+        if self.tag_name:
+            self.tag_member = seen[self.tag_name]
+        else:
+            self.tag_member.check(schema, members, seen)
+        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        for v in self.variants:
+            vseen = dict(seen)
+            v.check(schema, self.tag_member.type, vseen)
+
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+    def __init__(self, name, typ, flat):
+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+        assert isinstance(flat, bool)
+        self.flat = flat
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+        if self.flat:
+            self.type.check(schema)
+            assert isinstance(self.type, QAPISchemaObjectType)
+
+class QAPISchemaAlternateType(QAPISchemaType):
+    def __init__(self, name, info, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(variants, QAPISchemaObjectTypeVariants)
+        assert not variants.tag_name
+        for v in variants.variants:
+            assert not v.flat
+        self.variants = variants
+    def check(self, schema):
+        self.variants.check(schema, [], {})
+
+class QAPISchemaCommand(QAPISchemaEntity):
+    def __init__(self, name, info, args, rets, gen, success_response):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not args or isinstance(args, str)
+        assert not rets or isinstance(rets, str)
+        self.args_name = args
+        self.args = None
+        self.rets_name = rets
+        self.rets = None
+        self.gen = gen
+        self.success_response = success_response
+    def check(self, schema):
+        if self.args_name:
+            self.args = schema.lookup_type(self.args_name)
+            assert isinstance(self.args, QAPISchemaObjectType)
+            assert not self.args.variants # not implemented
+            self.args.check(schema)
+        if self.rets_name:
+            self.rets = schema.lookup_type(self.rets_name)
+            assert isinstance(self.rets, QAPISchemaType)
+            self.rets.check(schema)
+
+class QAPISchemaEvent(QAPISchemaEntity):
+    def __init__(self, name, info, data):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not data or isinstance(data, str)
+        self.data_name = data
+        self.data = None
+    def check(self, schema):
+        if self.data_name:
+            self.data = schema.lookup_type(self.data_name)
+            assert isinstance(self.data, QAPISchemaObjectType)
+            self.data.check(schema)
+
+class QAPISchema(object):
+    def __init__(self, fname):
+        try:
+            self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
+        except (QAPISchemaError, QAPIExprError), err:
+            print >>sys.stderr, err
+            exit(1)
+        self.entity_dict = {}
+        self._def_predefineds()
+        self._def_exprs()
+        self.check()
+
+    def get_exprs(self):
+        return [expr_elem['expr'] for expr_elem in self.exprs]
+
+    def _def_entity(self, ent):
+        assert ent.name not in self.entity_dict
+        self.entity_dict[ent.name] = ent
+
+    def lookup_entity(self, name, typ=None):
+        ent = self.entity_dict.get(name)
+        if typ and not isinstance(ent, typ):
+            return None
+        return ent
+
+    def lookup_type(self, name):
+        return self.lookup_entity(name, QAPISchemaType)
+
+    def _def_builtin_type(self, name):
+        self._def_entity(QAPISchemaBuiltinType(name))
+        if name != '**':
+            self._make_array_type(name) # TODO really needed?
+
+    def _def_predefineds(self):
+        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
+                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
+            self._def_builtin_type(t)
+
+    def _make_implicit_enum_type(self, name, values):
+        name = name + 'Kind'
+        self._def_entity(QAPISchemaEnumType(name, None, values))
+        return name
+
+    def _make_array_type(self, element_type):
+        name = element_type + 'List'
+        if not self.lookup_type(name):
+            self._def_entity(QAPISchemaArrayType(name, None, element_type))
+        return name
+
+    def _make_implicit_object_type(self, name, role, members):
+        name = ':obj-%s-%s' % (name, role)
+        self._def_entity(QAPISchemaObjectType(name, None, None, members, None))
+        return name
+
+    def _def_enum_type(self, expr, info):
+        name = expr['enum']
+        data = expr['data']
+        self._def_entity(QAPISchemaEnumType(name, info, data))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_member(self, name, typ):
+        optional = False
+        if name.startswith('*'):
+            name = name[1:]
+            optional = True
+        if isinstance(typ, list):
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        return QAPISchemaObjectTypeMember(name, typ, optional)
+
+    def _make_members(self, data):
+        return [self._make_member(key, data[key]) for key in data.keys()]
+
+    def _def_struct_type(self, expr, info):
+        name = expr['struct']
+        base = expr.get('base')
+        data = expr['data']
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                              self._make_members(data),
+                                              None))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_flat_variant(self, case, typ):
+        return QAPISchemaObjectTypeVariant(case, typ, True)
+
+    def _make_flat_variants(self, tag_name, data):
+        variants = [self._make_flat_variant(key, data[key])
+                    for key in data.keys()]
+        return QAPISchemaObjectTypeVariants(tag_name, None, variants)
+
+    def _make_simple_variant(self, case, typ):
+        if isinstance(typ, list):
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        return QAPISchemaObjectTypeVariant(case, typ, False)
+
+    def _make_simple_variants(self, type_name, data):
+        variants = [self._make_simple_variant(key, data[key])
+                    for key in data.keys()]
+        enum = self._make_implicit_enum_type(type_name,
+                                             [v.name for v in variants])
+        return QAPISchemaObjectTypeVariants(None, enum, variants)
+
+    def _def_union_type(self, expr, info):
+        name = expr['union']
+        data = expr['data']
+        tag_name = expr.get('discriminator')
+        if tag_name:
+            base = expr['base']
+            variants = self._make_flat_variants(tag_name, data)
+        else:
+            base = None
+            variants = self._make_simple_variants(name, data)
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                              self._make_members(OrderedDict()),
+                                              variants))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_alternate_type(self, expr, info):
+        name = expr['alternate']
+        data = expr['data']
+        self._def_entity(QAPISchemaAlternateType(name, info,
+                                    self._make_simple_variants(name, data)))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_command(self, expr, info):
+        name = expr['command']
+        args = expr.get('data')
+        rets = expr.get('returns')
+        gen = expr.get('gen', True)
+        success_response = expr.get('success-response', True)
+        if args and not isinstance(args, str):
+            args = self._make_implicit_object_type(name, 'args',
+                                                   self._make_members(args))
+        if rets and isinstance(rets, list):
+            assert len(rets) == 1
+            rets = self._make_array_type(rets[0])
+        elif rets and not isinstance(rets, str):
+            rets = self._make_implicit_object_type(name, 'rets',
+                                                   self._make_members(rets))
+        self._def_entity(QAPISchemaCommand(name, info, args, rets, gen,
+                                           success_response))
+
+    def _def_event(self, expr, info):
+        name = expr['event']
+        data = expr.get('data')
+        if data and not isinstance(data, str):
+            data = self._make_implicit_object_type(name, 'data',
+                                                   self._make_members(data))
+        self._def_entity(QAPISchemaEvent(name, info, data))
+
+    def _def_exprs(self):
+        for expr_elem in self.exprs:
+            expr = expr_elem['expr']
+            info = expr_elem['info']
+            if 'enum' in expr:
+                self._def_enum_type(expr, info)
+            elif 'struct' in expr:
+                self._def_struct_type(expr, info)
+            elif 'union' in expr:
+                self._def_union_type(expr, info)
+            elif 'alternate' in expr:
+                self._def_alternate_type(expr, info)
+            elif 'command' in expr:
+                self._def_command(expr, info)
+            elif 'event' in expr:
+                self._def_event(expr, info)
+            else:
+                assert False
+
+    def check(self):
+        for ent in self.entity_dict.values():
+            ent.check(self)
 
 #
 # Code generation helpers
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index 634ef2d..461c713 100644
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -16,7 +16,7 @@ import os
 import sys
 
 try:
-    exprs = parse_schema(sys.argv[1])
+    exprs = QAPISchema(sys.argv[1]).get_exprs()
 except SystemExit:
     raise
 
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

[Markus Armbruster]

New methods c_name(), c_type(), c_null(), json_type(),
alternate_qtype().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 7 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index a80892e..b5fa1be 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -763,15 +763,47 @@ class QAPISchemaEntity(object):
         assert isinstance(name, str)
         self.name = name
         self.info = info
+    def c_name(self):
+        return c_name(self.name)
     def check(self, schema):
         pass
 
 class QAPISchemaType(QAPISchemaEntity):
-    pass
+    def c_type(self, is_param=False):
+        return c_name(self.name) + pointer_suffix
+    def c_null(self):
+        return 'NULL'
+    def json_type(self):
+        return None
+    def alternate_qtype(self):
+        json2qtype = {
+            'string':  'QTYPE_QSTRING',
+            'number':  'QTYPE_QFLOAT',
+            'int':     'QTYPE_QINT',
+            'boolean': 'QTYPE_QBOOL',
+            'object':  'QTYPE_QDICT'
+        }
+        return json2qtype.get(self.json_type())
 
 class QAPISchemaBuiltinType(QAPISchemaType):
-    def __init__(self, name):
+    def __init__(self, name, json_type, c_type, c_null):
         QAPISchemaType.__init__(self, name, None)
+        assert not c_type or isinstance(c_type, str)
+        assert json_type in ('string', 'number', 'int', 'boolean', 'null',
+                             'value')
+        self.json_type_name = json_type
+        self.c_type_name = c_type
+        self.c_null_val = c_null
+    def c_name(self):
+        return self.name
+    def c_type(self, is_param=False):
+        if is_param and self.name == 'str':
+            return 'const ' + self.c_type_name
+        return self.c_type_name
+    def c_null(self):
+        return self.c_null_val
+    def json_type(self):
+        return self.json_type_name
 
 class QAPISchemaEnumType(QAPISchemaType):
     def __init__(self, name, info, values):
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         for v in values:
             assert isinstance(v, str)
         self.values = values
+    def c_type(self, is_param=False):
+        return c_name(self.name)
+    def c_null(self):
+        return c_enum_const(self.name, self.values[0])
+    def json_type(self):
+        return 'string'
 
 class QAPISchemaArrayType(QAPISchemaType):
     def __init__(self, name, info, element_type):
@@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.variants:
             self.variants.check(schema, members, seen)
         self.members = members
+    def c_name(self):
+        assert self.info
+        return QAPISchemaType.c_name(self)
+    def c_type(self, is_param=False):
+        assert self.info
+        return QAPISchemaType.c_type(self)
+    def json_type(self):
+        return 'object'
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -948,15 +994,27 @@ class QAPISchema(object):
     def lookup_type(self, name):
         return self.lookup_entity(name, QAPISchemaType)
 
-    def _def_builtin_type(self, name):
-        self._def_entity(QAPISchemaBuiltinType(name))
+    def _def_builtin_type(self, name, json_type, c_type, c_null):
+        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
         if name != '**':
             self._make_array_type(name) # TODO really needed?
 
     def _def_predefineds(self):
-        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
-                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
-            self._def_builtin_type(t)
+        for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
+                  ('number', 'number',  'double',   '0'),
+                  ('int',    'int',     'int64_t',  '0'),
+                  ('int8',   'int',     'int8_t',   '0'),
+                  ('int16',  'int',     'int16_t',  '0'),
+                  ('int32',  'int',     'int32_t',  '0'),
+                  ('int64',  'int',     'int64_t',  '0'),
+                  ('uint8',  'int',     'uint8_t',  '0'),
+                  ('uint16', 'int',     'uint16_t', '0'),
+                  ('uint32', 'int',     'uint32_t', '0'),
+                  ('uint64', 'int',     'uint64_t', '0'),
+                  ('size',   'int',     'uint64_t', '0'),
+                  ('bool',   'boolean', 'bool',     'false'),
+                  ('**',     'value',   None,       None)]:
+            self._def_builtin_type(*t)
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 23/47] qapi: New QAPISchemaVisitor

[Markus Armbruster]

The visitor will help keeping the code generation code simple and
reasonably separated from QAPISchema details.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi.py | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index b5fa1be..cac7ab5 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -767,6 +767,28 @@ class QAPISchemaEntity(object):
         return c_name(self.name)
     def check(self, schema):
         pass
+    def visit(self, visitor):
+        pass
+
+class QAPISchemaVisitor(object):
+    def visit_begin(self):
+        pass
+    def visit_end(self):
+        pass
+    def visit_builtin_type(self, name, info, json_type):
+        pass
+    def visit_enum_type(self, name, info, values):
+        pass
+    def visit_array_type(self, name, info, element_type):
+        pass
+    def visit_object_type(self, name, info, base, members, variants):
+        pass
+    def visit_alternate_type(self, name, info, variants):
+        pass
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        pass
+    def visit_event(self, name, info, data):
+        pass
 
 class QAPISchemaType(QAPISchemaEntity):
     def c_type(self, is_param=False):
@@ -804,6 +826,8 @@ class QAPISchemaBuiltinType(QAPISchemaType):
         return self.c_null_val
     def json_type(self):
         return self.json_type_name
+    def visit(self, visitor):
+        visitor.visit_builtin_type(self.name, self.info, self.json_type())
 
 class QAPISchemaEnumType(QAPISchemaType):
     def __init__(self, name, info, values):
@@ -817,6 +841,8 @@ class QAPISchemaEnumType(QAPISchemaType):
         return c_enum_const(self.name, self.values[0])
     def json_type(self):
         return 'string'
+    def visit(self, visitor):
+        visitor.visit_enum_type(self.name, self.info, self.values)
 
 class QAPISchemaArrayType(QAPISchemaType):
     def __init__(self, name, info, element_type):
@@ -827,6 +853,8 @@ class QAPISchemaArrayType(QAPISchemaType):
     def check(self, schema):
         self.element_type = schema.lookup_type(self.element_type_name)
         assert self.element_type
+    def visit(self, visitor):
+        visitor.visit_array_type(self.name, self.info, self.element_type)
 
 class QAPISchemaObjectType(QAPISchemaType):
     def __init__(self, name, info, base, local_members, variants):
@@ -868,6 +896,9 @@ class QAPISchemaObjectType(QAPISchemaType):
         return QAPISchemaType.c_type(self)
     def json_type(self):
         return 'object'
+    def visit(self, visitor):
+        visitor.visit_object_type(self.name, self.info,
+                                  self.base, self.local_members, self.variants)
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -931,6 +962,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
     def check(self, schema):
         self.variants.check(schema, [], {})
+    def visit(self, visitor):
+        visitor.visit_alternate_type(self.name, self.info, self.variants)
 
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, args, rets, gen, success_response):
@@ -953,6 +986,9 @@ class QAPISchemaCommand(QAPISchemaEntity):
             self.rets = schema.lookup_type(self.rets_name)
             assert isinstance(self.rets, QAPISchemaType)
             self.rets.check(schema)
+    def visit(self, visitor):
+        visitor.visit_command(self.name, self.info, self.args, self.rets,
+                              self.gen, self.success_response)
 
 class QAPISchemaEvent(QAPISchemaEntity):
     def __init__(self, name, info, data):
@@ -965,6 +1001,8 @@ class QAPISchemaEvent(QAPISchemaEntity):
             self.data = schema.lookup_type(self.data_name)
             assert isinstance(self.data, QAPISchemaObjectType)
             self.data.check(schema)
+    def visit(self, visitor):
+        visitor.visit_event(self.name, self.info, self.data)
 
 class QAPISchema(object):
     def __init__(self, fname):
@@ -1152,6 +1190,12 @@ class QAPISchema(object):
         for ent in self.entity_dict.values():
             ent.check(self)
 
+    def visit(self, visitor):
+        visitor.visit_begin()
+        for name in sorted(self.entity_dict.keys()):
+            self.entity_dict[name].visit(visitor)
+        visitor.visit_end()
+
 #
 # Code generation helpers
 #
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 24/47] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

[Markus Armbruster]

The old code prints the result of parsing (list of expression
dictionaries), and partial results of semantic analysis (list of enum
dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends
are going to use.  Built-in and array types are omitted, because
they're boring.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 tests/qapi-schema/alternate-good.out            |  15 +-
 tests/qapi-schema/comments.out                  |   4 +-
 tests/qapi-schema/data-member-array.out         |  13 +-
 tests/qapi-schema/empty.out                     |   3 -
 tests/qapi-schema/enum-empty.out                |   4 +-
 tests/qapi-schema/event-case.out                |   4 +-
 tests/qapi-schema/flat-union-reverse-define.out |  21 +--
 tests/qapi-schema/ident-with-escape.out         |   7 +-
 tests/qapi-schema/include-relpath.out           |   4 +-
 tests/qapi-schema/include-repetition.out        |   4 +-
 tests/qapi-schema/include-simple.out            |   4 +-
 tests/qapi-schema/indented-expr.out             |   7 +-
 tests/qapi-schema/qapi-schema-test.out          | 186 +++++++++++++++++-------
 tests/qapi-schema/returns-int.out               |   5 +-
 tests/qapi-schema/test-qapi.py                  |  37 ++++-
 tests/qapi-schema/type-bypass.out               |   7 +-
 16 files changed, 210 insertions(+), 115 deletions(-)

diff --git a/tests/qapi-schema/alternate-good.out b/tests/qapi-schema/alternate-good.out
index 99848ee..0cbdfa1 100644
--- a/tests/qapi-schema/alternate-good.out
+++ b/tests/qapi-schema/alternate-good.out
@@ -1,6 +1,9 @@
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))]),
- OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]),
- OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value', 'int'), ('string', 'Enum'), ('struct', 'Data')]))])]
-[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']},
- {'enum_name': 'AltKind', 'enum_values': None}]
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))])]
+alternate Alt
+    case value: int flat=False
+    case string: Enum flat=False
+    case struct: Data flat=False
+enum AltKind ['value', 'string', 'struct']
+object Data
+    member number: int optional=True
+    member name: str optional=True
+enum Enum ['hello', 'world']
diff --git a/tests/qapi-schema/comments.out b/tests/qapi-schema/comments.out
index 4ce3dcf..6161b90 100644
--- a/tests/qapi-schema/comments.out
+++ b/tests/qapi-schema/comments.out
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
diff --git a/tests/qapi-schema/data-member-array.out b/tests/qapi-schema/data-member-array.out
index c39fa25..8911179 100644
--- a/tests/qapi-schema/data-member-array.out
+++ b/tests/qapi-schema/data-member-array.out
@@ -1,5 +1,8 @@
-[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]),
- OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))]),
- OrderedDict([('command', 'okay'), ('data', OrderedDict([('member1', ['int']), ('member2', ['def'])]))])]
-[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}]
-[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))])]
+object :obj-okay-args
+    member member1: intList optional=False
+    member member2: defList optional=False
+enum abc ['a', 'b', 'c']
+object def
+    member array: abcList optional=False
+command okay :obj-okay-args -> None
+   gen=True success_response=True
diff --git a/tests/qapi-schema/empty.out b/tests/qapi-schema/empty.out
index b7f89a4..e69de29 100644
--- a/tests/qapi-schema/empty.out
+++ b/tests/qapi-schema/empty.out
@@ -1,3 +0,0 @@
-[]
-[]
-[]
diff --git a/tests/qapi-schema/enum-empty.out b/tests/qapi-schema/enum-empty.out
index 3b75c16..e09b00f 100644
--- a/tests/qapi-schema/enum-empty.out
+++ b/tests/qapi-schema/enum-empty.out
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'MyEnum'), ('data', [])])]
-[{'enum_name': 'MyEnum', 'enum_values': []}]
-[]
+enum MyEnum []
diff --git a/tests/qapi-schema/event-case.out b/tests/qapi-schema/event-case.out
index 3764bc7..b5ae4c2 100644
--- a/tests/qapi-schema/event-case.out
+++ b/tests/qapi-schema/event-case.out
@@ -1,3 +1 @@
-[OrderedDict([('event', 'oops')])]
-[]
-[]
+event oops None
diff --git a/tests/qapi-schema/flat-union-reverse-define.out b/tests/qapi-schema/flat-union-reverse-define.out
index 1ed7b8a..e156202 100644
--- a/tests/qapi-schema/flat-union-reverse-define.out
+++ b/tests/qapi-schema/flat-union-reverse-define.out
@@ -1,9 +1,12 @@
-[OrderedDict([('union', 'TestUnion'), ('base', 'TestBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'TestTypeA'), ('value2', 'TestTypeB')]))]),
- OrderedDict([('struct', 'TestBase'), ('data', OrderedDict([('enum1', 'TestEnum')]))]),
- OrderedDict([('enum', 'TestEnum'), ('data', ['value1', 'value2'])]),
- OrderedDict([('struct', 'TestTypeA'), ('data', OrderedDict([('string', 'str')]))]),
- OrderedDict([('struct', 'TestTypeB'), ('data', OrderedDict([('integer', 'int')]))])]
-[{'enum_name': 'TestEnum', 'enum_values': ['value1', 'value2']}]
-[OrderedDict([('struct', 'TestBase'), ('data', OrderedDict([('enum1', 'TestEnum')]))]),
- OrderedDict([('struct', 'TestTypeA'), ('data', OrderedDict([('string', 'str')]))]),
- OrderedDict([('struct', 'TestTypeB'), ('data', OrderedDict([('integer', 'int')]))])]
+object TestBase
+    member enum1: TestEnum optional=False
+enum TestEnum ['value1', 'value2']
+object TestTypeA
+    member string: str optional=False
+object TestTypeB
+    member integer: int optional=False
+object TestUnion
+    base TestBase
+    tag enum1
+    case value1: TestTypeA flat=True
+    case value2: TestTypeB flat=True
diff --git a/tests/qapi-schema/ident-with-escape.out b/tests/qapi-schema/ident-with-escape.out
index 4028430..24abf5c 100644
--- a/tests/qapi-schema/ident-with-escape.out
+++ b/tests/qapi-schema/ident-with-escape.out
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'fooA'), ('data', OrderedDict([('bar1', 'str')]))])]
-[]
-[]
+object :obj-fooA-args
+    member bar1: str optional=False
+command fooA :obj-fooA-args -> None
+   gen=True success_response=True
diff --git a/tests/qapi-schema/include-relpath.out b/tests/qapi-schema/include-relpath.out
index 4ce3dcf..6161b90 100644
--- a/tests/qapi-schema/include-relpath.out
+++ b/tests/qapi-schema/include-relpath.out
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
diff --git a/tests/qapi-schema/include-repetition.out b/tests/qapi-schema/include-repetition.out
index 4ce3dcf..6161b90 100644
--- a/tests/qapi-schema/include-repetition.out
+++ b/tests/qapi-schema/include-repetition.out
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
diff --git a/tests/qapi-schema/include-simple.out b/tests/qapi-schema/include-simple.out
index 4ce3dcf..6161b90 100644
--- a/tests/qapi-schema/include-simple.out
+++ b/tests/qapi-schema/include-simple.out
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
diff --git a/tests/qapi-schema/indented-expr.out b/tests/qapi-schema/indented-expr.out
index b5ce915..c5af55a 100644
--- a/tests/qapi-schema/indented-expr.out
+++ b/tests/qapi-schema/indented-expr.out
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'eins')]), OrderedDict([('command', 'zwei')])]
-[]
-[]
+command eins None -> None
+   gen=True success_response=True
+command zwei None -> None
+   gen=True success_response=True
diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
index b0b7187..921d7fb 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -1,55 +1,131 @@
-[OrderedDict([('enum', 'EnumOne'), ('data', ['value1', 'value2', 'value3'])]),
- OrderedDict([('struct', 'NestedEnumsOne'), ('data', OrderedDict([('enum1', 'EnumOne'), ('*enum2', 'EnumOne'), ('enum3', 'EnumOne'), ('*enum4', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefOne'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('*enum1', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
- OrderedDict([('struct', 'UserDefTwoDictDict'), ('data', OrderedDict([('userdef', 'UserDefOne'), ('string', 'str')]))]),
- OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
- OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
- OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
- OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('intb', 'int')]))]),
- OrderedDict([('union', 'UserDefFlatUnion'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefA'), ('value2', 'UserDefB'), ('value3', 'UserDefB')]))]),
- OrderedDict([('struct', 'UserDefUnionBase'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
- OrderedDict([('union', 'UserDefFlatUnion2'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefC'), ('value2', 'UserDefB'), ('value3', 'UserDefA')]))]),
- OrderedDict([('alternate', 'UserDefAlternate'), ('data', OrderedDict([('uda', 'UserDefA'), ('s', 'str'), ('i', 'int')]))]),
- OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
- OrderedDict([('union', 'UserDefNativeListUnion'), ('data', OrderedDict([('integer', ['int']), ('s8', ['int8']), ('s16', ['int16']), ('s32', ['int32']), ('s64', ['int64']), ('u8', ['uint8']), ('u16', ['uint16']), ('u32', ['uint32']), ('u64', ['uint64']), ('number', ['number']), ('boolean', ['bool']), ('string', ['str']), ('sizes', ['size'])]))]),
- OrderedDict([('command', 'user_def_cmd'), ('data', OrderedDict())]),
- OrderedDict([('command', 'user_def_cmd1'), ('data', OrderedDict([('ud1a', 'UserDefOne')]))]),
- OrderedDict([('command', 'user_def_cmd2'), ('data', OrderedDict([('ud1a', 'UserDefOne'), ('*ud1b', 'UserDefOne')])), ('returns', 'UserDefTwo')]),
- OrderedDict([('command', 'user_def_cmd3'), ('data', OrderedDict([('a', 'int'), ('*b', 'int')])), ('returns', 'int')]),
- OrderedDict([('struct', 'UserDefOptions'), ('data', OrderedDict([('*i64', ['int']), ('*u64', ['uint64']), ('*u16', ['uint16']), ('*i64x', 'int'), ('*u64x', 'uint64')]))]),
- OrderedDict([('struct', 'EventStructOne'), ('data', OrderedDict([('struct1', 'UserDefOne'), ('string', 'str'), ('*enum2', 'EnumOne')]))]),
- OrderedDict([('event', 'EVENT_A')]),
- OrderedDict([('event', 'EVENT_B'), ('data', OrderedDict())]),
- OrderedDict([('event', 'EVENT_C'), ('data', OrderedDict([('*a', 'int'), ('*b', 'UserDefOne'), ('c', 'str')]))]),
- OrderedDict([('event', 'EVENT_D'), ('data', OrderedDict([('a', 'EventStructOne'), ('b', 'str'), ('*c', 'str'), ('*enum3', 'EnumOne')]))]),
- OrderedDict([('enum', '__org.qemu_x-Enum'), ('data', ['__org.qemu_x-value'])]),
- OrderedDict([('struct', '__org.qemu_x-Base'), ('data', OrderedDict([('__org.qemu_x-member1', '__org.qemu_x-Enum')]))]),
- OrderedDict([('struct', '__org.qemu_x-Struct'), ('base', '__org.qemu_x-Base'), ('data', OrderedDict([('__org.qemu_x-member2', 'str')]))]),
- OrderedDict([('union', '__org.qemu_x-Union1'), ('data', OrderedDict([('__org.qemu_x-branch', 'str')]))]),
- OrderedDict([('struct', '__org.qemu_x-Struct2'), ('data', OrderedDict([('array', ['__org.qemu_x-Union1'])]))]),
- OrderedDict([('union', '__org.qemu_x-Union2'), ('base', '__org.qemu_x-Base'), ('discriminator', '__org.qemu_x-member1'), ('data', OrderedDict([('__org.qemu_x-value', '__org.qemu_x-Struct2')]))]),
- OrderedDict([('alternate', '__org.qemu_x-Alt'), ('data', OrderedDict([('__org.qemu_x-branch', 'str'), ('b', '__org.qemu_x-Base')]))]),
- OrderedDict([('event', '__ORG.QEMU_X-EVENT'), ('data', '__org.qemu_x-Struct')]),
- OrderedDict([('command', '__org.qemu_x-command'), ('data', OrderedDict([('a', ['__org.qemu_x-Enum']), ('b', ['__org.qemu_x-Struct']), ('c', '__org.qemu_x-Union2'), ('d', '__org.qemu_x-Alt')])), ('returns', '__org.qemu_x-Union1')])]
-[{'enum_name': 'EnumOne', 'enum_values': ['value1', 'value2', 'value3']},
- {'enum_name': '__org.qemu_x-Enum', 'enum_values': ['__org.qemu_x-value']},
- {'enum_name': 'UserDefAlternateKind', 'enum_values': None},
- {'enum_name': 'UserDefNativeListUnionKind', 'enum_values': None},
- {'enum_name': '__org.qemu_x-Union1Kind', 'enum_values': None},
- {'enum_name': '__org.qemu_x-AltKind', 'enum_values': None}]
-[OrderedDict([('struct', 'NestedEnumsOne'), ('data', OrderedDict([('enum1', 'EnumOne'), ('*enum2', 'EnumOne'), ('enum3', 'EnumOne'), ('*enum4', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefOne'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('*enum1', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
- OrderedDict([('struct', 'UserDefTwoDictDict'), ('data', OrderedDict([('userdef', 'UserDefOne'), ('string', 'str')]))]),
- OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
- OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
- OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
- OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('intb', 'int')]))]),
- OrderedDict([('struct', 'UserDefUnionBase'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
- OrderedDict([('struct', 'UserDefOptions'), ('data', OrderedDict([('*i64', ['int']), ('*u64', ['uint64']), ('*u16', ['uint16']), ('*i64x', 'int'), ('*u64x', 'uint64')]))]),
- OrderedDict([('struct', 'EventStructOne'), ('data', OrderedDict([('struct1', 'UserDefOne'), ('string', 'str'), ('*enum2', 'EnumOne')]))]),
- OrderedDict([('struct', '__org.qemu_x-Base'), ('data', OrderedDict([('__org.qemu_x-member1', '__org.qemu_x-Enum')]))]),
- OrderedDict([('struct', '__org.qemu_x-Struct'), ('base', '__org.qemu_x-Base'), ('data', OrderedDict([('__org.qemu_x-member2', 'str')]))]),
- OrderedDict([('struct', '__org.qemu_x-Struct2'), ('data', OrderedDict([('array', ['__org.qemu_x-Union1'])]))])]
+object :obj-EVENT_C-data
+    member a: int optional=True
+    member b: UserDefOne optional=True
+    member c: str optional=False
+object :obj-EVENT_D-data
+    member a: EventStructOne optional=False
+    member b: str optional=False
+    member c: str optional=True
+    member enum3: EnumOne optional=True
+object :obj-__org.qemu_x-command-args
+    member a: __org.qemu_x-EnumList optional=False
+    member b: __org.qemu_x-StructList optional=False
+    member c: __org.qemu_x-Union2 optional=False
+    member d: __org.qemu_x-Alt optional=False
+object :obj-user_def_cmd1-args
+    member ud1a: UserDefOne optional=False
+object :obj-user_def_cmd2-args
+    member ud1a: UserDefOne optional=False
+    member ud1b: UserDefOne optional=True
+object :obj-user_def_cmd3-args
+    member a: int optional=False
+    member b: int optional=True
+event EVENT_A None
+event EVENT_B None
+event EVENT_C :obj-EVENT_C-data
+event EVENT_D :obj-EVENT_D-data
+enum EnumOne ['value1', 'value2', 'value3']
+object EventStructOne
+    member struct1: UserDefOne optional=False
+    member string: str optional=False
+    member enum2: EnumOne optional=True
+object NestedEnumsOne
+    member enum1: EnumOne optional=False
+    member enum2: EnumOne optional=True
+    member enum3: EnumOne optional=False
+    member enum4: EnumOne optional=True
+object UserDefA
+    member boolean: bool optional=False
+alternate UserDefAlternate
+    case uda: UserDefA flat=False
+    case s: str flat=False
+    case i: int flat=False
+enum UserDefAlternateKind ['uda', 's', 'i']
+object UserDefB
+    member intb: int optional=False
+object UserDefC
+    member string1: str optional=False
+    member string2: str optional=False
+object UserDefFlatUnion
+    base UserDefUnionBase
+    tag enum1
+    case value1: UserDefA flat=True
+    case value2: UserDefB flat=True
+    case value3: UserDefB flat=True
+object UserDefFlatUnion2
+    base UserDefUnionBase
+    tag enum1
+    case value1: UserDefC flat=True
+    case value2: UserDefB flat=True
+    case value3: UserDefA flat=True
+object UserDefNativeListUnion
+    case integer: intList flat=False
+    case s8: int8List flat=False
+    case s16: int16List flat=False
+    case s32: int32List flat=False
+    case s64: int64List flat=False
+    case u8: uint8List flat=False
+    case u16: uint16List flat=False
+    case u32: uint32List flat=False
+    case u64: uint64List flat=False
+    case number: numberList flat=False
+    case boolean: boolList flat=False
+    case string: strList flat=False
+    case sizes: sizeList flat=False
+enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes']
+object UserDefOne
+    base UserDefZero
+    member string: str optional=False
+    member enum1: EnumOne optional=True
+object UserDefOptions
+    member i64: intList optional=True
+    member u64: uint64List optional=True
+    member u16: uint16List optional=True
+    member i64x: int optional=True
+    member u64x: uint64 optional=True
+object UserDefTwo
+    member string0: str optional=False
+    member dict1: UserDefTwoDict optional=False
+object UserDefTwoDict
+    member string1: str optional=False
+    member dict2: UserDefTwoDictDict optional=False
+    member dict3: UserDefTwoDictDict optional=True
+object UserDefTwoDictDict
+    member userdef: UserDefOne optional=False
+    member string: str optional=False
+object UserDefUnionBase
+    base UserDefZero
+    member string: str optional=False
+    member enum1: EnumOne optional=False
+object UserDefZero
+    member integer: int optional=False
+event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
+alternate __org.qemu_x-Alt
+    case __org.qemu_x-branch: str flat=False
+    case b: __org.qemu_x-Base flat=False
+enum __org.qemu_x-AltKind ['__org.qemu_x-branch', 'b']
+object __org.qemu_x-Base
+    member __org.qemu_x-member1: __org.qemu_x-Enum optional=False
+enum __org.qemu_x-Enum ['__org.qemu_x-value']
+object __org.qemu_x-Struct
+    base __org.qemu_x-Base
+    member __org.qemu_x-member2: str optional=False
+object __org.qemu_x-Struct2
+    member array: __org.qemu_x-Union1List optional=False
+object __org.qemu_x-Union1
+    case __org.qemu_x-branch: str flat=False
+enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch']
+object __org.qemu_x-Union2
+    base __org.qemu_x-Base
+    tag __org.qemu_x-member1
+    case __org.qemu_x-value: __org.qemu_x-Struct2 flat=True
+command __org.qemu_x-command :obj-__org.qemu_x-command-args -> __org.qemu_x-Union1
+   gen=True success_response=True
+command user_def_cmd None -> None
+   gen=True success_response=True
+command user_def_cmd1 :obj-user_def_cmd1-args -> None
+   gen=True success_response=True
+command user_def_cmd2 :obj-user_def_cmd2-args -> UserDefTwo
+   gen=True success_response=True
+command user_def_cmd3 :obj-user_def_cmd3-args -> int
+   gen=True success_response=True
diff --git a/tests/qapi-schema/returns-int.out b/tests/qapi-schema/returns-int.out
index 70b3ac5..1ac3e1e 100644
--- a/tests/qapi-schema/returns-int.out
+++ b/tests/qapi-schema/returns-int.out
@@ -1,3 +1,2 @@
-[OrderedDict([('command', 'guest-get-time'), ('returns', 'int')])]
-[]
-[]
+command guest-get-time None -> int
+   gen=True success_response=True
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index 461c713..259f515 100644
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -15,11 +15,34 @@ from pprint import pprint
 import os
 import sys
 
-try:
-    exprs = QAPISchema(sys.argv[1]).get_exprs()
-except SystemExit:
-    raise
+class QAPISchemaTestVisitor(QAPISchemaVisitor):
+    def visit_enum_type(self, name, info, values):
+        print 'enum %s %s' % (name, values)
+    def visit_object_type(self, name, info, base, members, variants):
+        print 'object %s' % name
+        if base:
+            print '    base %s' % base.name
+        for m in members:
+            print '    member %s: %s optional=%s' % (m.name, m.type.name,
+                                                     m.optional)
+        self._print_variants(variants)
+    def visit_alternate_type(self, name, info, variants):
+        print 'alternate %s' % name
+        self._print_variants(variants)
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        print 'command %s %s -> %s' % (name, (args and args.name),
+                                       (rets and rets.name))
+        print '   gen=%s success_response=%s' % (gen, success_response)
+    def visit_event(self, name, info, data):
+        print 'event %s %s' % (name, data and data.name)
 
-pprint(exprs)
-pprint(enum_types)
-pprint(struct_types)
+    @staticmethod
+    def _print_variants(variants):
+        if variants:
+            if variants.tag_name:
+                print '    tag %s' % variants.tag_name
+            for v in variants.variants:
+                print '    case %s: %s flat=%s' % (v.name, v.type.name, v.flat)
+
+schema = QAPISchema(sys.argv[1])
+schema.visit(QAPISchemaTestVisitor())
diff --git a/tests/qapi-schema/type-bypass.out b/tests/qapi-schema/type-bypass.out
index eaf20f8..b147864 100644
--- a/tests/qapi-schema/type-bypass.out
+++ b/tests/qapi-schema/type-bypass.out
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'unsafe'), ('data', OrderedDict([('arg', '**')])), ('returns', '**'), ('gen', False)])]
-[]
-[]
+object :obj-unsafe-args
+    member arg: ** optional=False
+command unsafe :obj-unsafe-args -> **
+   gen=False success_response=True
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 25/47] qapi: Make generators work on sorted schema expressions

[Markus Armbruster]

Order of expressions doesn't matter.  QAPISchema().get_exprs() returns
expressions in the order they're parsed.

I'm going to refactor this code.  To check refactorings, I want to
diff the generated code, and for that I need to preserve its order.

Since I don't feel like preserving parse order, I'm changing
get_expr() to return the expressions sorted by name.  This order will
be trivial to preserve.  It also makes the generated code slightly
easier to navigate.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index cac7ab5..20ffdaf 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1017,7 +1017,14 @@ class QAPISchema(object):
         self.check()
 
     def get_exprs(self):
-        return [expr_elem['expr'] for expr_elem in self.exprs]
+        def expr_name(expr):
+            name = expr.get('enum') or expr.get('union') \
+                   or expr.get('alternate') or expr.get('struct') \
+                   or expr.get('command') or expr.get('event')
+            assert name
+            return name
+        return sorted([expr_elem['expr'] for expr_elem in self.exprs],
+                      key=expr_name)
 
     def _def_entity(self, ent):
         assert ent.name not in self.entity_dict
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Markus Armbruster]

Fixes flat unions to get the base's base members.  Test case is from
commit 2fc0043, in qapi-schema-test.json:

    { 'union': 'UserDefFlatUnion',
      'base': 'UserDefUnionBase',
      'discriminator': 'enum1',
      'data': { 'value1' : 'UserDefA',
                'value2' : 'UserDefB',
                'value3' : 'UserDefB' } }

    { 'struct': 'UserDefUnionBase',
      'base': 'UserDefZero',
      'data': { 'string': 'str', 'enum1': 'EnumOne' } }

    { 'struct': 'UserDefZero',
      'data': { 'integer': 'int' } }

Patch's effect on UserDefFlatUnion:

     struct UserDefFlatUnion {
    +    int64_t integer;
         char *string;
         EnumOne enum1;
         /* union tag is EnumOne enum1 */
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
    };

Flat union visitors remain broken.  They'll be fixed next.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-types.py                   | 268 +++++++++++++-------------------
 tests/qapi-schema/qapi-schema-test.json |   4 +-
 2 files changed, 114 insertions(+), 158 deletions(-)

diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index a48ad9c..d6185c6 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -2,88 +2,67 @@
 # QAPI types generator
 #
 # Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2015 Red Hat Inc.
 #
 # Authors:
 #  Anthony Liguori <aliguori@us.ibm.com>
+#  Markus Armbruster <armbru@redhat.com>
 #
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 
-def generate_fwd_builtin(name):
-    return mcgen('''
-
-typedef struct %(name)sList {
-    union {
-        %(type)s value;
-        uint64_t padding;
-    };
-    struct %(name)sList *next;
-} %(name)sList;
-''',
-                 type=c_type(name),
-                 name=name)
-
-def generate_fwd_struct(name):
+def gen_fwd_object_or_array(name):
     return mcgen('''
 
 typedef struct %(name)s %(name)s;
-
-typedef struct %(name)sList {
-    union {
-        %(name)s *value;
-        uint64_t padding;
-    };
-    struct %(name)sList *next;
-} %(name)sList;
 ''',
                  name=c_name(name))
 
-def generate_fwd_enum_struct(name):
+def gen_array(name, element_type):
     return mcgen('''
 
-typedef struct %(name)sList {
+struct %(name)s {
     union {
-        %(name)s value;
+        %(c_type)s value;
         uint64_t padding;
     };
-    struct %(name)sList *next;
-} %(name)sList;
+    struct %(name)s *next;
+};
 ''',
-                 name=c_name(name))
+                 name=c_name(name), c_type=element_type.c_type())
 
-def generate_struct_fields(members):
+def gen_struct_field(name, typ, optional):
     ret = ''
 
-    for argname, argentry, optional in parse_args(members):
-        if optional:
-            ret += mcgen('''
+    if optional:
+        ret += mcgen('''
     bool has_%(c_name)s;
 ''',
-                         c_name=c_name(argname))
-        ret += mcgen('''
+                     c_name=c_name(name))
+    ret += mcgen('''
     %(c_type)s %(c_name)s;
 ''',
-                     c_type=c_type(argentry), c_name=c_name(argname))
-
+                 c_type=typ.c_type(), c_name=c_name(name))
     return ret
 
-def generate_struct(expr):
+def generate_struct_fields(members):
+    ret = ''
 
-    structname = expr.get('struct', "")
-    members = expr['data']
-    base = expr.get('base')
+    for memb in members:
+        ret += gen_struct_field(memb.name, memb.type, memb.optional)
+    return ret
 
+def gen_struct(name, base, members):
     ret = mcgen('''
 
 struct %(name)s {
 ''',
-          name=c_name(structname))
+                name=c_name(name))
 
     if base:
-        ret += generate_struct_fields({'base': base})
+        ret += gen_struct_field('base', base, False)
 
     ret += generate_struct_fields(members)
 
@@ -156,46 +135,38 @@ typedef enum %(name)s {
 
     return enum_decl + lookup_decl
 
-def generate_alternate_qtypes(expr):
+def gen_alternate_qtypes_decl(name):
+    return mcgen('''
 
-    name = expr['alternate']
-    members = expr['data']
+extern const int %(c_name)s_qtypes[];
+''',
+                 c_name=c_name(name))
 
+def gen_alternate_qtypes(name, variants):
     ret = mcgen('''
 
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
 
-    for key in members:
-        qtype = find_alternate_member_qtype(members[key])
-        assert qtype, "Invalid alternate member"
+    for var in variants.variants:
+        qtype = var.type.alternate_qtype()
+        assert qtype
 
         ret += mcgen('''
     [%(qtype)s] = %(enum_const)s,
 ''',
                      qtype = qtype,
-                     enum_const = c_enum_const(name + 'Kind', key))
+                     enum_const=c_enum_const(variants.tag_member.type.name,
+                                             var.name))
 
     ret += mcgen('''
 };
 ''')
     return ret
 
-
-def generate_union(expr, meta):
-
-    name = c_name(expr[meta])
-    typeinfo = expr['data']
-
-    base = expr.get('base')
-    discriminator = expr.get('discriminator')
-
-    enum_define = discriminator_find_enum_define(expr)
-    if enum_define:
-        discriminator_type_name = enum_define['enum_name']
-    else:
-        discriminator_type_name = '%sKind' % (name)
+def gen_union(name, base, variants):
+    name = c_name(name)
 
     ret = mcgen('''
 
@@ -203,56 +174,49 @@ struct %(name)s {
 ''',
                 name=name)
     if base:
-        base_fields = find_struct(base)['data']
-        ret += generate_struct_fields(base_fields)
+        ret += generate_struct_fields(base.members)
         ret += mcgen('''
     /* union tag is %(discriminator_type_name)s %(c_name)s */
 ''',
-                     discriminator_type_name=c_name(discriminator_type_name),
-                     c_name=c_name(discriminator))
+                     discriminator_type_name=c_name(variants.tag_member.type.name),
+                     c_name=c_name(variants.tag_member.name))
     else:
-        assert not discriminator
         ret += mcgen('''
     %(discriminator_type_name)s kind;
 ''',
-                     discriminator_type_name=c_name(discriminator_type_name))
+                     discriminator_type_name=c_name(variants.tag_member.type.name))
 
     ret += mcgen('''
     union {
         void *data;
 ''')
 
-    for key in typeinfo:
+    for var in variants.variants:
         ret += mcgen('''
         %(c_type)s %(c_name)s;
 ''',
-                     c_type=c_type(typeinfo[key]),
-                     c_name=c_name(key))
+                     c_type=var.type.c_type(),
+                     c_name=c_name(var.name))
 
     ret += mcgen('''
     };
 };
 ''')
-    if meta == 'alternate':
-        ret += mcgen('''
-extern const int %(name)s_qtypes[];
-''',
-            name=name)
-
 
     return ret
 
 def generate_type_cleanup_decl(name):
     ret = mcgen('''
-void qapi_free_%(name)s(%(c_type)s obj);
+
+void qapi_free_%(name)s(%(name)s *obj);
 ''',
-                c_type=c_type(name), name=c_name(name))
+                name=c_name(name))
     return ret
 
 def generate_type_cleanup(name):
     ret = mcgen('''
 
-void qapi_free_%(name)s(%(c_type)s obj)
+void qapi_free_%(name)s(%(name)s *obj)
 {
     QapiDeallocVisitor *md;
     Visitor *v;
@@ -267,9 +231,68 @@ void qapi_free_%(name)s(%(c_type)s obj)
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_type=c_type(name), name=c_name(name))
+                name=c_name(name))
     return ret
 
+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.fwdecl = None
+        self.fwdefn = None
+        self.btin = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.fwdecl = ''
+        self.fwdefn = ''
+        self.btin = guardstart('QAPI_TYPES_BUILTIN')
+    def visit_end(self):
+        self.decl = self.fwdecl + self.decl
+        self.fwdecl = None
+        self.defn = self.fwdefn + self.defn
+        self.fwdefn = None
+        # To avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them.
+        self.btin += guardend('QAPI_TYPES_BUILTIN')
+        self.decl = self.btin + self.decl
+        self.btin = None
+        # Doesn't work for cases where we link in multiple objects
+        # that have the functions defined, so generate them only with
+        # option -b (do_builtins).
+    def _gen_type_cleanup(self, name):
+        self.decl += generate_type_cleanup_decl(name)
+        self.defn += generate_type_cleanup(name)
+    def visit_enum_type(self, name, info, values):
+        self.fwdecl += generate_enum(name, values)
+        self.fwdefn += generate_enum_lookup(name, values)
+    def visit_array_type(self, name, info, element_type):
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += gen_fwd_object_or_array(name)
+            self.btin += gen_array(name, element_type)
+            self.btin += generate_type_cleanup_decl(name)
+            if do_builtins:
+                self.defn += generate_type_cleanup(name)
+        else:
+            self.fwdecl += gen_fwd_object_or_array(name)
+            self.decl += gen_array(name, element_type)
+            self._gen_type_cleanup(name)
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:
+            self.fwdecl += gen_fwd_object_or_array(name)
+            if variants:
+                self.decl += gen_union(name, base, variants)
+            else:
+                self.decl += gen_struct(name, base, members)
+            self._gen_type_cleanup(name)
+    def visit_alternate_type(self, name, info, variants):
+        self.fwdecl += gen_fwd_object_or_array(name)
+        self.fwdefn += gen_alternate_qtypes(name, variants)
+        self.decl += gen_union(name, None, variants)
+        self.decl += gen_alternate_qtypes_decl(name)
+        self._gen_type_cleanup(name)
+
 do_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
@@ -325,77 +348,10 @@ fdecl.write(mcgen('''
 #include <stdint.h>
 '''))
 
-exprs = QAPISchema(input_file).get_exprs()
-
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_fwd_builtin(typename))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
-        ret += generate_fwd_struct(expr['struct'])
-    elif expr.has_key('enum'):
-        ret += generate_enum(expr['enum'], expr['data'])
-        ret += generate_fwd_enum_struct(expr['enum'])
-        fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
-    elif expr.has_key('union'):
-        ret += generate_fwd_struct(expr['union'])
-        enum_define = discriminator_find_enum_define(expr)
-        if not enum_define:
-            ret += generate_enum('%sKind' % expr['union'], expr['data'].keys())
-            fdef.write(generate_enum_lookup('%sKind' % expr['union'],
-                                            expr['data'].keys()))
-    elif expr.has_key('alternate'):
-        ret += generate_fwd_struct(expr['alternate'])
-        ret += generate_enum('%sKind' % expr['alternate'], expr['data'].keys())
-        fdef.write(generate_enum_lookup('%sKind' % expr['alternate'],
-                                        expr['data'].keys()))
-        fdef.write(generate_alternate_qtypes(expr))
-    else:
-        continue
-    fdecl.write(ret)
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_type_cleanup_decl(typename + "List"))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_type_cleanup(typename + "List"))
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
-        ret += generate_struct(expr) + "\n"
-        ret += generate_type_cleanup_decl(expr['struct'] + "List")
-        fdef.write(generate_type_cleanup(expr['struct'] + "List"))
-        ret += generate_type_cleanup_decl(expr['struct'])
-        fdef.write(generate_type_cleanup(expr['struct']))
-    elif expr.has_key('union'):
-        ret += generate_union(expr, 'union') + "\n"
-        ret += generate_type_cleanup_decl(expr['union'] + "List")
-        fdef.write(generate_type_cleanup(expr['union'] + "List"))
-        ret += generate_type_cleanup_decl(expr['union'])
-        fdef.write(generate_type_cleanup(expr['union']))
-    elif expr.has_key('alternate'):
-        ret += generate_union(expr, 'alternate') + "\n"
-        ret += generate_type_cleanup_decl(expr['alternate'] + "List")
-        fdef.write(generate_type_cleanup(expr['alternate'] + "List"))
-        ret += generate_type_cleanup_decl(expr['alternate'])
-        fdef.write(generate_type_cleanup(expr['alternate']))
-    elif expr.has_key('enum'):
-        ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List")
-        fdef.write(generate_type_cleanup(expr['enum'] + "List"))
-    else:
-        continue
-    fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenTypeVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index a9e5aab..257b4d4 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -39,8 +39,8 @@
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
-# FIXME generated struct UserDefFlatUnion has members for direct base
-# UserDefUnionBase, but lacks members for indirect base UserDefZero
+# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
+# members of indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Markus Armbruster]

Fixes flat unions to visit the base's base members (the previous
commit merely added them to the struct).  Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

     static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp)
     {
         Error *err = NULL;

    +    visit_type_int(m, &(*obj)->integer, "integer", &err);
    +    if (err) {
    +        goto out;
    +    }
         visit_type_str(m, &(*obj)->string, "string", &err);
         if (err) {
             goto out;

Test cases updated for the bug fix.

Fixes alternates to generate a visitor for their implicit enumeration
type.  None of them are currently used, obviously.  Example:
block-core.json's BlockdevRef now generates
visit_type_BlockdevRefKind().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-visit.py                   | 254 ++++++++++++--------------------
 tests/qapi-schema/qapi-schema-test.json |   3 -
 tests/test-qmp-input-strict.c           |   2 +-
 tests/test-qmp-input-visitor.c          |   4 +-
 4 files changed, 100 insertions(+), 163 deletions(-)

diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index a52a572..135e7c1 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -12,7 +12,6 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 import re
 
@@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type):
         return ''
     implicit_structs_seen.add(type)
     ret = ''
-    if type not in struct_fields_seen:
+    if type.name not in struct_fields_seen:
         # Need a forward declaration
         ret += mcgen('''
 
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
-                     c_type=type_name(type))
+                     c_type=type.c_name())
 
     ret += mcgen('''
 
@@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
     error_propagate(errp, err);
 }
 ''',
-                 c_type=type_name(type))
+                 c_type=type.c_name())
     return ret
 
 def generate_visit_struct_fields(name, members, base = None):
@@ -74,24 +73,24 @@ if (err) {
     goto out;
 }
 ''',
-                     type=type_name(base), c_name=c_name('base'))
+                     type=base.c_name(), c_name=c_name('base'))
 
-    for argname, argentry, optional in parse_args(members):
-        if optional:
+    for memb in members:
+        if memb.optional:
             ret += mcgen('''
 visit_optional(m, &(*obj)->has_%(c_name)s, "%(name)s", &err);
 if (!err && (*obj)->has_%(c_name)s) {
 ''',
-                         c_name=c_name(argname), name=argname)
+                         c_name=c_name(memb.name), name=memb.name)
             push_indent()
 
         ret += mcgen('''
 visit_type_%(type)s(m, &(*obj)->%(c_name)s, "%(name)s", &err);
 ''',
-                     type=type_name(argentry), c_name=c_name(argname),
-                     name=argname)
+                     type=memb.type.c_name(), c_name=c_name(memb.name),
+                     name=memb.name)
 
-        if optional:
+        if memb.optional:
             pop_indent()
             ret += mcgen('''
 }
@@ -132,12 +131,7 @@ def generate_visit_struct_body(name):
 
     return ret
 
-def generate_visit_struct(expr):
-
-    name = expr['struct']
-    members = expr['data']
-    base = expr.get('base')
-
+def gen_visit_struct(name, base, members):
     ret = generate_visit_struct_fields(name, members, base)
 
     ret += mcgen('''
@@ -154,10 +148,10 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 ''')
     return ret
 
-def generate_visit_list(name):
+def gen_visit_list(name, element_type):
     return mcgen('''
 
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp)
+void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
     GenericList *i, **prev;
@@ -170,8 +164,8 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
     for (prev = (GenericList **)obj;
          !err && (i = visit_next_list(m, prev, &err)) != NULL;
          prev = &i) {
-        %(name)sList *native_i = (%(name)sList *)i;
-        visit_type_%(name)s(m, &native_i->value, NULL, &err);
+        %(name)s *native_i = (%(name)s *)i;
+        visit_type_%(c_elt_type)s(m, &native_i->value, NULL, &err);
     }
 
     error_propagate(errp, err);
@@ -181,7 +175,8 @@ out:
     error_propagate(errp, err);
 }
 ''',
-                name=type_name(name))
+                 name=c_name(name),
+                 c_elt_type=element_type.c_name())
 
 def generate_visit_enum(name):
     return mcgen('''
@@ -193,7 +188,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error
 ''',
                  c_name=c_name(name), name=name)
 
-def generate_visit_alternate(name, members):
+def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
@@ -212,25 +207,17 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 ''',
                 name=c_name(name))
 
-    # For alternate, always use the default enum type automatically generated
-    # as name + 'Kind'
-    disc_type = c_name(name) + 'Kind'
-
-    for key in members:
-        assert (members[key] in builtin_types.keys()
-            or find_struct(members[key])
-            or find_union(members[key])
-            or find_enum(members[key])), "Invalid alternate member"
-
-        enum_full_value = c_enum_const(disc_type, key)
+    for var in variants.variants:
+        enum_full_value = c_enum_const(variants.tag_member.type.name,
+                                       var.name)
         ret += mcgen('''
     case %(enum_full_value)s:
         visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, name, &err);
         break;
 ''',
                 enum_full_value = enum_full_value,
-                c_type = type_name(members[key]),
-                c_name = c_name(key))
+                c_type=var.type.c_name(),
+                c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
@@ -247,35 +234,16 @@ out:
 
     return ret
 
-
-def generate_visit_union(expr):
-
-    name = expr['union']
-    members = expr['data']
-
-    base = expr.get('base')
-    discriminator = expr.get('discriminator')
-
-    enum_define = discriminator_find_enum_define(expr)
-    if enum_define:
-        # Use the enum type as discriminator
-        ret = ""
-        disc_type = c_name(enum_define['enum_name'])
-    else:
-        # There will always be a discriminator in the C switch code, by default
-        # it is an enum type generated silently
-        ret = generate_visit_enum(name + 'Kind')
-        disc_type = c_name(name) + 'Kind'
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)
 
-    if discriminator:
-        for key in members:
-            ret += generate_visit_implicit_struct(members[key])
+    for var in variants.variants:
+        if var.flat:
+            ret += generate_visit_implicit_struct(var.type)
 
     ret += mcgen('''
 
@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
 ''',
                      name=c_name(name))
 
-    if not discriminator:
-        tag = 'kind'
-        disc_key = "type"
-    else:
-        tag = discriminator
-        disc_key = discriminator
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions
+        disc_key = 'type'
     ret += mcgen('''
-        visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err);
+        visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err);
         if (err) {
             goto out_obj;
         }
         if (!visit_start_union(m, !!(*obj)->data, &err) || err) {
             goto out_obj;
         }
-        switch ((*obj)->%(c_tag)s) {
+        switch ((*obj)->%(c_name)s) {
 ''',
-                 disc_type = disc_type,
-                 c_tag=c_name(tag),
+                 disc_type=variants.tag_member.type.c_name(),
+                 c_name=c_name(variants.tag_member.name),
                  disc_key = disc_key)
 
-    for key in members:
-        if not discriminator:
+    for var in variants.variants:
+        if not var.flat:
             fmt = 'visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, "data", &err);'
         else:
             fmt = 'visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);'
 
-        enum_full_value = c_enum_const(disc_type, key)
+        enum_full_value = c_enum_const(variants.tag_member.type.name, var.name)
         ret += mcgen('''
         case %(enum_full_value)s:
             ''' + fmt + '''
             break;
 ''',
                 enum_full_value = enum_full_value,
-                c_type=type_name(members[key]),
-                c_name=c_name(key))
+                c_type=var.type.c_name(),
+                c_name=c_name(var.name))
 
     ret += mcgen('''
         default:
@@ -355,37 +321,57 @@ out:
 
     return ret
 
-def generate_declaration(name, builtin_type=False):
-    ret = ""
-    if not builtin_type:
-        name = c_name(name)
-        ret += mcgen('''
-
-void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp);
-''',
-                     name=name)
-
-    ret += mcgen('''
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
-''',
-                 name=name)
-
-    return ret
-
-def generate_enum_declaration(name):
-    ret = mcgen('''
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
-''',
-                name=c_name(name))
-
-    return ret
-
-def generate_decl_enum(name):
+def gen_visit_decl(name, scalar=False):
+    c_type = c_name(name) + ' *'
+    if not scalar:
+        c_type += '*'
     return mcgen('''
-
-void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
+void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), c_type=c_type)
+
+class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.btin = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
+    def visit_end(self):
+        # to avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them
+        self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL')
+        self.decl = self.btin + self.decl
+        self.btin = None
+        # ...this doesn't work for cases where we link in multiple
+        # objects that have the functions defined, so we use
+        # do_builtins (option -b) to provide control
+    def visit_enum_type(self, name, info, values):
+        self.decl += gen_visit_decl(name, scalar=True)
+        self.defn += generate_visit_enum(name)
+    def visit_array_type(self, name, info, element_type):
+        decl = gen_visit_decl(name)
+        defn = gen_visit_list(name, element_type)
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += decl
+            if do_builtins:
+                self.defn += defn
+        else:
+            self.decl += decl
+            self.defn += defn
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:
+            self.decl += gen_visit_decl(name)
+            if variants:
+                self.defn += gen_visit_union(name, base, variants)
+            else:
+                self.defn += gen_visit_struct(name, base, members)
+    def visit_alternate_type(self, name, info, variants):
+        self.decl += gen_visit_decl(name)
+        self.defn += gen_visit_alternate(name, variants)
 
 do_builtins = False
 
@@ -442,56 +428,10 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = QAPISchema(input_file).get_exprs()
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_declaration(typename, builtin_type=True))
-fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_visit_list(typename))
-
-for expr in exprs:
-    if expr.has_key('struct'):
-        ret = generate_visit_struct(expr)
-        ret += generate_visit_list(expr['struct'])
-        fdef.write(ret)
-
-        ret = generate_declaration(expr['struct'])
-        fdecl.write(ret)
-    elif expr.has_key('union'):
-        ret = generate_visit_union(expr)
-        ret += generate_visit_list(expr['union'])
-        fdef.write(ret)
-
-        enum_define = discriminator_find_enum_define(expr)
-        ret = ""
-        if not enum_define:
-            ret = generate_decl_enum('%sKind' % expr['union'])
-        ret += generate_declaration(expr['union'])
-        fdecl.write(ret)
-    elif expr.has_key('alternate'):
-        ret = generate_visit_alternate(expr['alternate'], expr['data'])
-        ret += generate_visit_list(expr['alternate'])
-        fdef.write(ret)
-
-        ret = generate_decl_enum('%sKind' % expr['alternate'])
-        ret += generate_declaration(expr['alternate'])
-        fdecl.write(ret)
-    elif expr.has_key('enum'):
-        ret = generate_visit_list(expr['enum'])
-        ret += generate_visit_enum(expr['enum'])
-        fdef.write(ret)
-
-        ret = generate_decl_enum(expr['enum'])
-        ret += generate_enum_declaration(expr['enum'])
-        fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenVisitVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index 257b4d4..b182174 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -39,8 +39,6 @@
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
-# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
-# members of indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',
@@ -57,7 +55,6 @@
 
 { 'alternate': 'UserDefAlternate',
   'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } }
-# FIXME only a declaration of visit_type_UserDefAlternateKind() generated
 
 { 'struct': 'UserDefC',
   'data': { 'string1': 'str', 'string2': 'str' } }
diff --git a/tests/test-qmp-input-strict.c b/tests/test-qmp-input-strict.c
index 68f855b..00c3e29 100644
--- a/tests/test-qmp-input-strict.c
+++ b/tests/test-qmp-input-strict.c
@@ -167,9 +167,9 @@ static void test_validate_union_flat(TestInputVisitorData *data,
 
     v = validate_test_init(data,
                            "{ 'enum1': 'value1', "
+                           "'integer': 41, "
                            "'string': 'str', "
                            "'boolean': true }");
-    /* TODO when generator bug is fixed, add 'integer': 41 */
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(!err);
diff --git a/tests/test-qmp-input-visitor.c b/tests/test-qmp-input-visitor.c
index b7a87ee..56da3c6 100644
--- a/tests/test-qmp-input-visitor.c
+++ b/tests/test-qmp-input-visitor.c
@@ -307,15 +307,15 @@ static void test_visitor_in_union_flat(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data,
                                 "{ 'enum1': 'value1', "
+                                "'integer': 41, "
                                 "'string': 'str', "
                                 "'boolean': true }");
-    /* TODO when generator bug is fixed, add 'integer': 41 */
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(err == NULL);
     g_assert_cmpint(tmp->enum1, ==, ENUM_ONE_VALUE1);
     g_assert_cmpstr(tmp->string, ==, "str");
-    /* TODO g_assert_cmpint(tmp->integer, ==, 41); */
+    g_assert_cmpint(tmp->integer, ==, 41);
     g_assert_cmpint(tmp->value1->boolean, ==, true);
     qapi_free_UserDefFlatUnion(tmp);
 }
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 28/47] qapi-commands: Convert to QAPISchemaVisitor

[Markus Armbruster]

Output unchanged except for white-space.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 157 ++++++++++++++++++++++++++---------------------
 scripts/qapi.py          |   2 +-
 2 files changed, 87 insertions(+), 72 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 3cdbd97..a225bdd 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -12,21 +12,22 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 import re
 
 def generate_command_decl(name, args, ret_type):
     arglist=""
-    for argname, argtype, optional in parse_args(args):
-        argtype = c_type(argtype, is_param=True)
-        if optional:
-            arglist += "bool has_%s, " % c_name(argname)
-        arglist += "%s %s, " % (argtype, c_name(argname))
+    if args:
+        for memb in args.members:
+            argtype = memb.type.c_type(is_param=True)
+            if memb.optional:
+                arglist += "bool has_%s, " % c_name(memb.name)
+            arglist += "%s %s, " % (argtype, c_name(memb.name))
     return mcgen('''
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
-                 ret_type=c_type(ret_type), name=c_name(name),
+                 ret_type=(ret_type and ret_type.c_type()) or 'void',
+                 name=c_name(name),
                  args=arglist)
 
 def gen_err_check(err):
@@ -45,10 +46,11 @@ def gen_sync_call(name, args, ret_type):
     retval=""
     if ret_type:
         retval = "retval = "
-    for argname, argtype, optional in parse_args(args):
-        if optional:
-            arglist += "has_%s, " % c_name(argname)
-        arglist += "%s, " % (c_name(argname))
+    if args:
+        for memb in args.members:
+            if memb.optional:
+                arglist += "has_%s, " % c_name(memb.name)
+            arglist += "%s, " % c_name(memb.name)
     push_indent()
     ret = mcgen('''
 %(retval)sqmp_%(name)s(%(args)s&local_err);
@@ -68,7 +70,7 @@ def gen_visitor_input_containers_decl(args):
     ret = ""
 
     push_indent()
-    if len(args) > 0:
+    if args:
         ret += mcgen('''
 QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
 QapiDeallocVisitor *md;
@@ -81,22 +83,26 @@ Visitor *v;
 def gen_visitor_input_vars_decl(args):
     ret = ""
     push_indent()
-    for argname, argtype, optional in parse_args(args):
-        if optional:
-            ret += mcgen('''
+
+    if args:
+        for memb in args.members:
+            if memb.optional:
+                ret += mcgen('''
 bool has_%(argname)s = false;
 ''',
-                         argname=c_name(argname))
-        if is_c_ptr(argtype):
-            ret += mcgen('''
+                             argname=c_name(memb.name))
+            if is_c_ptr(memb.type.c_type()):
+                ret += mcgen('''
 %(argtype)s %(argname)s = NULL;
 ''',
-                         argname=c_name(argname), argtype=c_type(argtype))
-        else:
-            ret += mcgen('''
+                             argname=c_name(memb.name),
+                             argtype=memb.type.c_type())
+            else:
+                ret += mcgen('''
 %(argtype)s %(argname)s = {0};
 ''',
-                         argname=c_name(argname), argtype=c_type(argtype))
+                             argname=c_name(memb.name),
+                             argtype=memb.type.c_type())
 
     pop_indent()
     return ret
@@ -106,7 +112,7 @@ def gen_visitor_input_block(args, dealloc=False):
     errparg = '&local_err'
     errarg = 'local_err'
 
-    if len(args) == 0:
+    if not args:
         return ret
 
     push_indent()
@@ -124,25 +130,26 @@ v = qapi_dealloc_get_visitor(md);
 v = qmp_input_get_visitor(mi);
 ''')
 
-    for argname, argtype, optional in parse_args(args):
-        if optional:
+    for memb in args.members:
+        if memb.optional:
             ret += mcgen('''
 visit_optional(v, &has_%(c_name)s, "%(name)s", %(errp)s);
 ''',
-                         c_name=c_name(argname), name=argname, errp=errparg)
+                         c_name=c_name(memb.name), name=memb.name,
+                         errp=errparg)
             ret += gen_err_check(errarg)
             ret += mcgen('''
 if (has_%(c_name)s) {
 ''',
-                         c_name=c_name(argname))
+                         c_name=c_name(memb.name))
             push_indent()
         ret += mcgen('''
 visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s);
 ''',
-                     c_name=c_name(argname), name=argname, argtype=argtype,
-                     visitor=type_name(argtype), errp=errparg)
+                     c_name=c_name(memb.name), name=memb.name,
+                     visitor=memb.type.c_name(), errp=errparg)
         ret += gen_err_check(errarg)
-        if optional:
+        if memb.optional:
             pop_indent()
             ret += mcgen('''
 }
@@ -160,6 +167,7 @@ def gen_marshal_output(name, ret_type):
         return ""
 
     ret = mcgen('''
+
 static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
@@ -183,8 +191,8 @@ out:
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_ret_type=c_type(ret_type), c_name=c_name(name),
-                visitor=type_name(ret_type))
+                c_ret_type=ret_type.c_type(), c_name=c_name(name),
+                visitor=ret_type.c_name())
 
     return ret
 
@@ -198,6 +206,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
     hdr = gen_marshal_input_decl(name, middle_mode)
 
     ret = mcgen('''
+
 %(header)s
 {
     Error *local_err = NULL;
@@ -205,16 +214,16 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
                 header=hdr)
 
     if ret_type:
-        if is_c_ptr(ret_type):
-            retval = "    %s retval = NULL;" % c_type(ret_type)
+        if is_c_ptr(ret_type.c_type()):
+            retval = "    %s retval = NULL;" % ret_type.c_type()
         else:
-            retval = "    %s retval;" % c_type(ret_type)
+            retval = "    %s retval;" % ret_type.c_type()
         ret += mcgen('''
 %(retval)s
 ''',
                      retval=retval)
 
-    if len(args) > 0:
+    if args:
         ret += gen_visitor_input_containers_decl(args)
         ret += gen_visitor_input_vars_decl(args) + '\n'
         ret += gen_visitor_input_block(args) + '\n'
@@ -241,21 +250,23 @@ out:
 ''')
     return ret
 
-def gen_registry(commands):
-    registry=""
+def gen_register_command(name, success_response):
     push_indent()
-    for cmd in commands:
-        options = 'QCO_NO_OPTIONS'
-        if not cmd.get('success-response', True):
-            options = 'QCO_NO_SUCCESS_RESP'
+    options = 'QCO_NO_OPTIONS'
+    if not success_response:
+        options = 'QCO_NO_SUCCESS_RESP'
 
-        registry += mcgen('''
+    ret = mcgen('''
 qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
 ''',
-                     name=cmd['command'], c_name=c_name(cmd['command']),
+                     name=name, c_name=c_name(name),
                      opts=options)
     pop_indent()
+    return ret
+
+def gen_registry(registry):
     ret = mcgen('''
+
 static void qmp_init_marshal(void)
 {
 ''')
@@ -267,6 +278,31 @@ qapi_init(qmp_init_marshal);
 ''')
     return ret
 
+class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.regy = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.regy = ''
+    def visit_end(self):
+        if not middle_mode:
+            self.defn += gen_registry(self.regy)
+            self.regy = None
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        if not gen:
+            return
+        self.decl += generate_command_decl(name, args, rets)
+        if rets:
+            self.defn += gen_marshal_output(name, rets)
+        if middle_mode:
+            self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n'
+        self.defn += gen_marshal_input(name, args, rets, middle_mode)
+        if not middle_mode:
+            self.regy += gen_register_command(name, success_response)
+
 middle_mode = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
@@ -276,10 +312,6 @@ for o, a in opts:
     if o in ("-m", "--middle"):
         middle_mode = True
 
-exprs = QAPISchema(input_file).get_exprs()
-commands = filter(lambda expr: expr.has_key('command'), exprs)
-commands = filter(lambda expr: not expr.has_key('gen'), commands)
-
 c_comment = '''
 /*
  * schema-defined QMP->QAPI command dispatch
@@ -335,29 +367,12 @@ fdecl.write(mcgen('''
 #include "qapi/error.h"
 
 ''',
-                 prefix=prefix))
+                  prefix=prefix))
 
-for cmd in commands:
-    arglist = []
-    ret_type = None
-    if cmd.has_key('data'):
-        arglist = cmd['data']
-    if cmd.has_key('returns'):
-        ret_type = cmd['returns']
-    ret = generate_command_decl(cmd['command'], arglist, ret_type)
-    fdecl.write(ret)
-    if ret_type:
-        ret = gen_marshal_output(cmd['command'], ret_type) + "\n"
-        fdef.write(ret)
-
-    if middle_mode:
-        fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode))
-
-    ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n"
-    fdef.write(ret)
-
-if not middle_mode:
-    ret = gen_registry(commands)
-    fdef.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenCommandVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 20ffdaf..98488be 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1366,7 +1366,7 @@ def c_type(value, is_param=False):
         return c_name(value) + pointer_suffix
 
 def is_c_ptr(value):
-    return c_type(value).endswith(pointer_suffix)
+    return value.endswith(pointer_suffix)
 
 def genindent(count):
     ret = ""
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Markus Armbruster]

is_c_ptr() looks whether the end of the C text for the type looks like
a pointer.  Works, but is fragile.

We now have a better tool: use QAPISchemaType method c_null().  The
initializers for non-pointers become prettier: 0, false or the
enumeration constant with the value 0 instead of {0}.

One place looks suspicious: we initialize pointers, but not
non-pointers.  Either the initialization is superfluous and should be
deleted, or the non-pointers need it as well, or something subtle is
going on and needs a comment.  Since I lack the time to figure it out
now, mark it FIXME.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 19 +++++++------------
 scripts/qapi.py          |  3 ---
 2 files changed, 7 insertions(+), 15 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index a225bdd..d3bddb6 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -91,18 +91,12 @@ def gen_visitor_input_vars_decl(args):
 bool has_%(argname)s = false;
 ''',
                              argname=c_name(memb.name))
-            if is_c_ptr(memb.type.c_type()):
-                ret += mcgen('''
-%(argtype)s %(argname)s = NULL;
+            ret += mcgen('''
+%(c_type)s %(c_name)s = %(c_null)s;
 ''',
-                             argname=c_name(memb.name),
-                             argtype=memb.type.c_type())
-            else:
-                ret += mcgen('''
-%(argtype)s %(argname)s = {0};
-''',
-                             argname=c_name(memb.name),
-                             argtype=memb.type.c_type())
+                         c_name=c_name(memb.name),
+                         c_type=memb.type.c_type(),
+                         c_null=memb.type.c_null())
 
     pop_indent()
     return ret
@@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
                 header=hdr)
 
     if ret_type:
-        if is_c_ptr(ret_type.c_type()):
+        # FIXME fishy: only pointers are initialized
+        if ret_type.c_null() == 'NULL':
             retval = "    %s retval = NULL;" % ret_type.c_type()
         else:
             retval = "    %s retval;" % ret_type.c_type()
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 98488be..dd77a30 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1365,9 +1365,6 @@ def c_type(value, is_param=False):
         assert isinstance(value, str) and value != ""
         return c_name(value) + pointer_suffix
 
-def is_c_ptr(value):
-    return value.endswith(pointer_suffix)
-
 def genindent(count):
     ret = ""
     for i in range(count):
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 30/47] qapi: De-duplicate enum code generation

[Markus Armbruster]

Duplicated in commit 21cd70d.  Yes, we can't import qapi-types, but
that's no excuse.  Move the helpers from qapi-types.py to qapi.py, and
replace the duplicates in qapi-event.py.

The generated event enumeration type gains a _MAX member, and its
lookup table becomes const-correct (see commit 2e4450f).

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-event.py | 67 +++------------------------------------------------
 scripts/qapi-types.py | 55 ------------------------------------------
 scripts/qapi.py       | 55 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 118 deletions(-)

diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index aec2d32..aed45d6 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -153,63 +153,6 @@ def generate_event_implement(api_name, event_name, params):
 
     return ret
 
-
-# Following are the functions that generate an enum type for all defined
-# events, similar to qapi-types.py. Here we already have enum name and
-# values which were generated before and recorded in event_enum_*. It also
-# works around the issue that "import qapi-types" can't work.
-
-def generate_event_enum_decl(event_enum_name, event_enum_values):
-    lookup_decl = mcgen('''
-
-extern const char *%(event_enum_name)s_lookup[];
-''',
-                        event_enum_name = event_enum_name)
-
-    enum_decl = mcgen('''
-typedef enum %(event_enum_name)s {
-''',
-                      event_enum_name = event_enum_name)
-
-    # append automatically generated _MAX value
-    enum_max_value = c_enum_const(event_enum_name, "MAX")
-    enum_values = event_enum_values + [ enum_max_value ]
-
-    i = 0
-    for value in enum_values:
-        enum_decl += mcgen('''
-    %(value)s = %(i)d,
-''',
-                     value = value,
-                     i = i)
-        i += 1
-
-    enum_decl += mcgen('''
-} %(event_enum_name)s;
-''',
-                       event_enum_name = event_enum_name)
-
-    return lookup_decl + enum_decl
-
-def generate_event_enum_lookup(event_enum_name, event_enum_strings):
-    ret = mcgen('''
-
-const char *%(event_enum_name)s_lookup[] = {
-''',
-                event_enum_name = event_enum_name)
-
-    for string in event_enum_strings:
-        ret += mcgen('''
-    "%(string)s",
-''',
-                     string = string)
-
-    ret += mcgen('''
-    NULL,
-};
-''')
-    return ret
-
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
 c_comment = '''
@@ -266,8 +209,7 @@ fdecl.write(mcgen('''
 exprs = QAPISchema(input_file).get_exprs()
 
 event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
-event_enum_values = []
-event_enum_strings = []
+event_names = []
 
 for expr in exprs:
     if expr.has_key('event'):
@@ -286,12 +228,11 @@ for expr in exprs:
         fdef.write(ret)
 
         # Record it, and generate enum later
-        event_enum_values.append(event_enum_value)
-        event_enum_strings.append(event_name)
+        event_names.append(event_name)
 
-ret = generate_event_enum_decl(event_enum_name, event_enum_values)
+ret = generate_enum(event_enum_name, event_names)
 fdecl.write(ret)
-ret = generate_event_enum_lookup(event_enum_name, event_enum_strings)
+ret = generate_enum_lookup(event_enum_name, event_names)
 fdef.write(ret)
 
 close_output(fdef, fdecl)
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index d6185c6..540c359 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -80,61 +80,6 @@ struct %(name)s {
 
     return ret
 
-def generate_enum_lookup(name, values):
-    ret = mcgen('''
-
-const char *const %(name)s_lookup[] = {
-''',
-                name=c_name(name))
-    for value in values:
-        index = c_enum_const(name, value)
-        ret += mcgen('''
-    [%(index)s] = "%(value)s",
-''',
-                     index = index, value = value)
-
-    max_index = c_enum_const(name, 'MAX')
-    ret += mcgen('''
-    [%(max_index)s] = NULL,
-};
-''',
-        max_index=max_index)
-    return ret
-
-def generate_enum(name, values):
-    name = c_name(name)
-    lookup_decl = mcgen('''
-
-extern const char *const %(name)s_lookup[];
-''',
-                name=name)
-
-    enum_decl = mcgen('''
-
-typedef enum %(name)s {
-''',
-                name=name)
-
-    # append automatically generated _MAX value
-    enum_values = values + [ 'MAX' ]
-
-    i = 0
-    for value in enum_values:
-        enum_full_value = c_enum_const(name, value)
-        enum_decl += mcgen('''
-    %(enum_full_value)s = %(i)d,
-''',
-                     enum_full_value = enum_full_value,
-                     i=i)
-        i += 1
-
-    enum_decl += mcgen('''
-} %(name)s;
-''',
-                 name=name)
-
-    return enum_decl + lookup_decl
-
 def gen_alternate_qtypes_decl(name):
     return mcgen('''
 
diff --git a/scripts/qapi.py b/scripts/qapi.py
index dd77a30..699a656 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1417,6 +1417,61 @@ def guardend(name):
 ''',
                  name=guardname(name))
 
+def generate_enum_lookup(name, values):
+    ret = mcgen('''
+
+const char *const %(name)s_lookup[] = {
+''',
+                name=c_name(name))
+    for value in values:
+        index = c_enum_const(name, value)
+        ret += mcgen('''
+    [%(index)s] = "%(value)s",
+''',
+                     index = index, value = value)
+
+    max_index = c_enum_const(name, 'MAX')
+    ret += mcgen('''
+    [%(max_index)s] = NULL,
+};
+''',
+        max_index=max_index)
+    return ret
+
+def generate_enum(name, values):
+    name = c_name(name)
+    lookup_decl = mcgen('''
+
+extern const char *const %(name)s_lookup[];
+''',
+                name=name)
+
+    enum_decl = mcgen('''
+
+typedef enum %(name)s {
+''',
+                name=name)
+
+    # append automatically generated _MAX value
+    enum_values = values + [ 'MAX' ]
+
+    i = 0
+    for value in enum_values:
+        enum_full_value = c_enum_const(name, value)
+        enum_decl += mcgen('''
+    %(enum_full_value)s = %(i)d,
+''',
+                     enum_full_value = enum_full_value,
+                     i=i)
+        i += 1
+
+    enum_decl += mcgen('''
+} %(name)s;
+''',
+                 name=name)
+
+    return enum_decl + lookup_decl
+
 #
 # Common command line parsing
 #
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 31/47] qapi-event: Eliminate global variable event_enum_value

[Markus Armbruster]

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-event.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index aed45d6..537da17 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -137,7 +137,7 @@ def generate_event_implement(api_name, event_name, params):
     emit(%(event_enum_value)s, qmp, &local_err);
 
 """,
-                 event_enum_value = event_enum_value)
+                 event_enum_value = c_enum_const(event_enum_name, event_name))
 
     # step 5: clean up
     if params:
@@ -223,7 +223,6 @@ for expr in exprs:
         fdecl.write(ret)
 
         # We need an enum value per event
-        event_enum_value = c_enum_const(event_enum_name, event_name)
         ret = generate_event_implement(api_name, event_name, params)
         fdef.write(ret)
 
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 32/47] qapi-event: Convert to QAPISchemaVisitor, fixing data with base

[Markus Armbruster]

Fixes events whose data is struct with base to include the struct's
base members.  Test case is qapi-schema-test.json's event
__org.qemu_x-command:

    { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }

    { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member2': 'str' } }

    { 'struct': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }

Patch's effect on generated qapi_event_send___org_qemu_x_event():

    -void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2,
    +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
    +                                        const char *__org_qemu_x_member2,
                                             Error **errp)
     {
         QDict *qmp;
    @@ -224,6 +225,10 @@ void qapi_event_send___org_qemu_x_event(
             goto clean;
         }

    +    visit_type___org_qemu_x_Enum(v, &__org_qemu_x_member1, "__org.qemu_x-member1", &local_err);
    +    if (local_err) {
    +        goto clean;
    +    }
         visit_type_str(v, (char **)&__org_qemu_x_member2, "__org.qemu_x-member2", &local_err);
         if (local_err) {
             goto clean;

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-event.py                   | 87 ++++++++++++++++-----------------
 tests/qapi-schema/qapi-schema-test.json |  3 --
 2 files changed, 43 insertions(+), 47 deletions(-)

diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 537da17..456e590 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -2,28 +2,29 @@
 # QAPI event generator
 #
 # Copyright (c) 2014 Wenchao Xia
+# Copyright (c) 2013-2015 Red Hat Inc.
 #
 # Authors:
 #  Wenchao Xia <wenchaoqemu@gmail.com>
+#  Markus Armbruster <armbru@redhat.com>
 #
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 
-def _generate_event_api_name(event_name, params):
+def _generate_event_api_name(event_name, data):
     api_name = "void qapi_event_send_%s(" % c_name(event_name).lower();
     l = len(api_name)
 
-    if params:
-        for argname, argentry, optional in parse_args(params):
-            if optional:
-                api_name += "bool has_%s,\n" % c_name(argname)
+    if data:
+        for m in data.members:
+            if m.optional:
+                api_name += "bool has_%s,\n" % c_name(m.name)
                 api_name += "".ljust(l)
 
-            api_name += "%s %s,\n" % (c_type(argentry, is_param=True),
-                                      c_name(argname))
+            api_name += "%s %s,\n" % (m.type.c_type(is_param=True),
+                                      c_name(m.name))
             api_name += "".ljust(l)
 
     api_name += "Error **errp)"
@@ -51,7 +52,7 @@ def generate_event_implement(api_name, event_name, params):
 """,
                 api_name = api_name)
 
-    if params:
+    if params and params.members:
         ret += mcgen("""
     QmpOutputVisitor *qov;
     Visitor *v;
@@ -72,7 +73,7 @@ def generate_event_implement(api_name, event_name, params):
                  event_name = event_name)
 
     # step 3: visit the params if params != None
-    if params:
+    if params and params.members:
         ret += mcgen("""
     qov = qmp_output_visitor_new();
     g_assert(qov);
@@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params):
 """,
                 event_name = event_name)
 
-        for argname, argentry, optional in parse_args(params):
-            if optional:
+        for memb in params.members:
+            if memb.optional:
                 ret += mcgen("""
     if (has_%(var)s) {
 """,
-                             var = c_name(argname))
+                             var=c_name(memb.name))
                 push_indent()
 
-            if argentry == "str":
+            if memb.type.name == "str":
                 var_type = "(char **)"
             else:
                 var_type = ""
@@ -109,11 +110,11 @@ def generate_event_implement(api_name, event_name, params):
     }
 """,
                          var_type = var_type,
-                         var = c_name(argname),
-                         type = type_name(argentry),
-                         name = argname)
+                         var=c_name(memb.name),
+                         type=memb.type.c_name(),
+                         name=memb.name)
 
-            if optional:
+            if memb.optional:
                 pop_indent()
                 ret += mcgen("""
     }
@@ -140,7 +141,7 @@ def generate_event_implement(api_name, event_name, params):
                  event_enum_value = c_enum_const(event_enum_name, event_name))
 
     # step 5: clean up
-    if params:
+    if params and params.members:
         ret += mcgen("""
  clean:
     qmp_output_visitor_cleanup(qov);
@@ -153,6 +154,24 @@ def generate_event_implement(api_name, event_name, params):
 
     return ret
 
+class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.event_names = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.event_names = []
+    def visit_end(self):
+        self.decl += generate_enum(event_enum_name, self.event_names)
+        self.defn += generate_enum_lookup(event_enum_name, self.event_names)
+    def visit_event(self, name, info, data):
+        api_name = _generate_event_api_name(name, data)
+        self.decl += generate_event_declaration(api_name)
+        self.defn += generate_event_implement(api_name, name, data)
+        self.event_names.append(name)
+
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
 c_comment = '''
@@ -206,32 +225,12 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = QAPISchema(input_file).get_exprs()
-
 event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
-event_names = []
 
-for expr in exprs:
-    if expr.has_key('event'):
-        event_name = expr['event']
-        params = expr.get('data')
-        if params and len(params) == 0:
-            params = None
-
-        api_name = _generate_event_api_name(event_name, params)
-        ret = generate_event_declaration(api_name)
-        fdecl.write(ret)
-
-        # We need an enum value per event
-        ret = generate_event_implement(api_name, event_name, params)
-        fdef.write(ret)
-
-        # Record it, and generate enum later
-        event_names.append(event_name)
-
-ret = generate_enum(event_enum_name, event_names)
-fdecl.write(ret)
-ret = generate_enum_lookup(event_enum_name, event_names)
-fdef.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenEventVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index b182174..90b4740 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -126,9 +126,6 @@
 { 'alternate': '__org.qemu_x-Alt',
   'data': { '__org.qemu_x-branch': 'str', 'b': '__org.qemu_x-Base' } }
 { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
-# FIXME generated qapi_event_send___org_qemu_x_event() has only a
-# parameter for data's member __org_qemu_x_member2, none for its base
-# __org.qemu_x-Base's member __org_qemu_x_member1
 { 'command': '__org.qemu_x-command',
   'data': { 'a': ['__org.qemu_x-Enum'], 'b': ['__org.qemu_x-Struct'],
             'c': '__org.qemu_x-Union2', 'd': '__org.qemu_x-Alt' },
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 33/47] qapi: Clean up after recent conversions to QAPISchemaVisitor

[Markus Armbruster]

Drop helper functions that are now unused.

Make pylint reasonably happy.

Rename generate_FOO() functions to gen_FOO() for consistency.

Use more consistent and sensible variable names.

Consistently use c_ for mapping keys when their value is a C
identifier or type.

Simplify gen_enum() and gen_visit_union()

Consistently use single quotes for C text string literals.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 109 +++++++++++++++++++------------------
 scripts/qapi-event.py    | 117 +++++++++++++++++++---------------------
 scripts/qapi-types.py    |  68 ++++++++++++-----------
 scripts/qapi-visit.py    | 121 ++++++++++++++++++++---------------------
 scripts/qapi.py          | 138 +++++++++--------------------------------------
 5 files changed, 229 insertions(+), 324 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index d3bddb6..5d11032 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -15,20 +15,20 @@
 from qapi import *
 import re
 
-def generate_command_decl(name, args, ret_type):
-    arglist=""
+def gen_command_decl(name, args, rets):
+    argstr = ''
     if args:
         for memb in args.members:
-            argtype = memb.type.c_type(is_param=True)
             if memb.optional:
-                arglist += "bool has_%s, " % c_name(memb.name)
-            arglist += "%s %s, " % (argtype, c_name(memb.name))
+                argstr += 'bool has_%s, ' % c_name(memb.name)
+            argstr += '%s %s, ' % (memb.type.c_type(is_param=True),
+                                   c_name(memb.name))
     return mcgen('''
-%(ret_type)s qmp_%(name)s(%(args)sError **errp);
+%(c_type)s qmp_%(c_name)s(%(args)sError **errp);
 ''',
-                 ret_type=(ret_type and ret_type.c_type()) or 'void',
-                 name=c_name(name),
-                 args=arglist)
+                 c_type=(rets and rets.c_type()) or 'void',
+                 c_name=c_name(name),
+                 args=argstr)
 
 def gen_err_check(err):
     if not err:
@@ -40,34 +40,37 @@ if (%(err)s) {
 ''',
                  err=err)
 
-def gen_sync_call(name, args, ret_type):
-    ret = ""
-    arglist=""
-    retval=""
-    if ret_type:
-        retval = "retval = "
+def gen_call(name, args, rets):
+    ret = ''
+
+    argstr = ''
     if args:
         for memb in args.members:
             if memb.optional:
-                arglist += "has_%s, " % c_name(memb.name)
-            arglist += "%s, " % c_name(memb.name)
+                argstr += 'has_%s, ' % c_name(memb.name)
+            argstr += '%s, ' % c_name(memb.name)
+
+    lhs = ''
+    if rets:
+        lhs = 'retval = '
+
     push_indent()
     ret = mcgen('''
-%(retval)sqmp_%(name)s(%(args)s&local_err);
+%(lhs)sqmp_%(c_name)s(%(args)s&local_err);
 ''',
-                name=c_name(name), args=arglist, retval=retval)
-    if ret_type:
+                c_name=c_name(name), args=argstr, lhs=lhs)
+    if rets:
         ret += gen_err_check('local_err')
         ret += mcgen('''
 
 qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
 ''',
-                            c_name=c_name(name))
+                     c_name=c_name(name))
     pop_indent()
     return ret
 
 def gen_visitor_input_containers_decl(args):
-    ret = ""
+    ret = ''
 
     push_indent()
     if args:
@@ -81,16 +84,16 @@ Visitor *v;
     return ret
 
 def gen_visitor_input_vars_decl(args):
-    ret = ""
+    ret = ''
     push_indent()
 
     if args:
         for memb in args.members:
             if memb.optional:
                 ret += mcgen('''
-bool has_%(argname)s = false;
+bool has_%(c_name)s = false;
 ''',
-                             argname=c_name(memb.name))
+                             c_name=c_name(memb.name))
             ret += mcgen('''
 %(c_type)s %(c_name)s = %(c_null)s;
 ''',
@@ -102,7 +105,7 @@ bool has_%(argname)s = false;
     return ret
 
 def gen_visitor_input_block(args, dealloc=False):
-    ret = ""
+    ret = ''
     errparg = '&local_err'
     errarg = 'local_err'
 
@@ -113,7 +116,7 @@ def gen_visitor_input_block(args, dealloc=False):
 
     if dealloc:
         errparg = 'NULL'
-        errarg = None;
+        errarg = None
         ret += mcgen('''
 qmp_input_visitor_cleanup(mi);
 md = qapi_dealloc_visitor_new();
@@ -138,10 +141,10 @@ if (has_%(c_name)s) {
                          c_name=c_name(memb.name))
             push_indent()
         ret += mcgen('''
-visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s);
+visit_type_%(c_type)s(v, &%(c_name)s, "%(name)s", %(errp)s);
 ''',
                      c_name=c_name(memb.name), name=memb.name,
-                     visitor=memb.type.c_name(), errp=errparg)
+                     c_type=memb.type.c_name(), errp=errparg)
         ret += gen_err_check(errarg)
         if memb.optional:
             pop_indent()
@@ -156,13 +159,13 @@ qapi_dealloc_visitor_cleanup(md);
     pop_indent()
     return ret
 
-def gen_marshal_output(name, ret_type):
-    if not ret_type:
-        return ""
+def gen_marshal_output(name, rets):
+    if not rets:
+        return ''
 
     ret = mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_cmd_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
     QmpOutputVisitor *mo = qmp_output_visitor_new();
@@ -170,7 +173,7 @@ static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_o
     Visitor *v;
 
     v = qmp_output_get_visitor(mo);
-    visit_type_%(visitor)s(v, &ret_in, "unused", &local_err);
+    visit_type_%(c_name)s(v, &ret_in, "unused", &local_err);
     if (local_err) {
         goto out;
     }
@@ -181,23 +184,23 @@ out:
     qmp_output_visitor_cleanup(mo);
     md = qapi_dealloc_visitor_new();
     v = qapi_dealloc_get_visitor(md);
-    visit_type_%(visitor)s(v, &ret_in, "unused", NULL);
+    visit_type_%(c_name)s(v, &ret_in, "unused", NULL);
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_ret_type=ret_type.c_type(), c_name=c_name(name),
-                visitor=ret_type.c_name())
+                c_type=rets.c_type(), c_cmd_name=c_name(name),
+                c_name=rets.c_name())
 
     return ret
 
-def gen_marshal_input_decl(name, middle_mode):
+def gen_marshal_input_decl(name):
     ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
     if not middle_mode:
-        ret = "static " + ret
+        ret = 'static ' + ret
     return ret
 
-def gen_marshal_input(name, args, ret_type, middle_mode):
-    hdr = gen_marshal_input_decl(name, middle_mode)
+def gen_marshal_input(name, args, rets):
+    hdr = gen_marshal_input_decl(name)
 
     ret = mcgen('''
 
@@ -207,12 +210,12 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
 ''',
                 header=hdr)
 
-    if ret_type:
+    if rets:
         # FIXME fishy: only pointers are initialized
-        if ret_type.c_null() == 'NULL':
-            retval = "    %s retval = NULL;" % ret_type.c_type()
+        if rets.c_null() == 'NULL':
+            retval = '    %s retval = NULL;' % rets.c_type()
         else:
-            retval = "    %s retval;" % ret_type.c_type()
+            retval = '    %s retval;' % rets.c_type()
         ret += mcgen('''
 %(retval)s
 ''',
@@ -229,9 +232,9 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
 
 ''')
 
-    ret += gen_sync_call(name, args, ret_type)
+    ret += gen_call(name, args, rets)
 
-    if re.search('^ *goto out\\;', ret, re.MULTILINE):
+    if re.search('^ *goto out;', ret, re.MULTILINE):
         ret += mcgen('''
 
 out:
@@ -254,8 +257,8 @@ def gen_register_command(name, success_response):
     ret = mcgen('''
 qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
 ''',
-                     name=name, c_name=c_name(name),
-                     opts=options)
+                name=name, c_name=c_name(name),
+                opts=options)
     pop_indent()
     return ret
 
@@ -289,12 +292,12 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
     def visit_command(self, name, info, args, rets, gen, success_response):
         if not gen:
             return
-        self.decl += generate_command_decl(name, args, rets)
+        self.decl += gen_command_decl(name, args, rets)
         if rets:
             self.defn += gen_marshal_output(name, rets)
         if middle_mode:
-            self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n'
-        self.defn += gen_marshal_input(name, args, rets, middle_mode)
+            self.decl += gen_marshal_input_decl(name) + ';\n'
+        self.defn += gen_marshal_input(name, args, rets)
         if not middle_mode:
             self.regy += gen_register_command(name, success_response)
 
@@ -354,7 +357,7 @@ fdef.write(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                prefix=prefix))
+                 prefix=prefix))
 
 fdecl.write(mcgen('''
 #include "%(prefix)sqapi-types.h"
diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 456e590..03bb1ec 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -13,8 +13,8 @@
 
 from qapi import *
 
-def _generate_event_api_name(event_name, data):
-    api_name = "void qapi_event_send_%s(" % c_name(event_name).lower();
+def gen_event_send_proto(name, data):
+    api_name = "void qapi_event_send_%s(" % c_name(name).lower()
     l = len(api_name)
 
     if data:
@@ -28,53 +28,47 @@ def _generate_event_api_name(event_name, data):
             api_name += "".ljust(l)
 
     api_name += "Error **errp)"
-    return api_name;
+    return api_name
 
-
-# Following are the core functions that generate C APIs to emit event.
-
-def generate_event_declaration(api_name):
+def gen_event_send_decl(name, data):
     return mcgen('''
 
-%(api_name)s;
+%(proto)s;
 ''',
-                 api_name = api_name)
+                 proto=gen_event_send_proto(name, data))
 
-def generate_event_implement(api_name, event_name, params):
-    # step 1: declare any variables
-    ret = mcgen("""
+def gen_event_send(name, data):
+    ret = mcgen('''
 
-%(api_name)s
+%(proto)s
 {
     QDict *qmp;
     Error *local_err = NULL;
     QMPEventFuncEmit emit;
-""",
-                api_name = api_name)
+''',
+                proto=gen_event_send_proto(name, data))
 
-    if params and params.members:
-        ret += mcgen("""
+    if data and data.members:
+        ret += mcgen('''
     QmpOutputVisitor *qov;
     Visitor *v;
     QObject *obj;
 
-""")
+''')
 
-    # step 2: check emit function, create a dict
-    ret += mcgen("""
+    ret += mcgen('''
     emit = qmp_event_get_func_emit();
     if (!emit) {
         return;
     }
 
-    qmp = qmp_event_build_dict("%(event_name)s");
+    qmp = qmp_event_build_dict("%(name)s");
 
-""",
-                 event_name = event_name)
+''',
+                 name=name)
 
-    # step 3: visit the params if params != None
-    if params and params.members:
-        ret += mcgen("""
+    if data and data.members:
+        ret += mcgen('''
     qov = qmp_output_visitor_new();
     g_assert(qov);
 
@@ -82,45 +76,46 @@ def generate_event_implement(api_name, event_name, params):
     g_assert(v);
 
     /* Fake visit, as if all members are under a structure */
-    visit_start_struct(v, NULL, "", "%(event_name)s", 0, &local_err);
+    visit_start_struct(v, NULL, "", "%(name)s", 0, &local_err);
     if (local_err) {
         goto clean;
     }
 
-""",
-                event_name = event_name)
+''',
+                     name=name)
 
-        for memb in params.members:
+        for memb in data.members:
             if memb.optional:
-                ret += mcgen("""
-    if (has_%(var)s) {
-""",
-                             var=c_name(memb.name))
+                ret += mcgen('''
+    if (has_%(c_name)s) {
+''',
+                             c_name=c_name(memb.name))
                 push_indent()
 
+            # Ugly: need to cast away the const
             if memb.type.name == "str":
-                var_type = "(char **)"
+                cast = "(char **)"
             else:
-                var_type = ""
+                cast = ""
 
-            ret += mcgen("""
-    visit_type_%(type)s(v, %(var_type)s&%(var)s, "%(name)s", &local_err);
+            ret += mcgen('''
+    visit_type_%(c_type)s(v, %(cast)s&%(c_name)s, "%(name)s", &local_err);
     if (local_err) {
         goto clean;
     }
-""",
-                         var_type = var_type,
-                         var=c_name(memb.name),
-                         type=memb.type.c_name(),
+''',
+                         cast=cast,
+                         c_name=c_name(memb.name),
+                         c_type=memb.type.c_name(),
                          name=memb.name)
 
             if memb.optional:
                 pop_indent()
-                ret += mcgen("""
+                ret += mcgen('''
     }
-""")
+''')
 
-        ret += mcgen("""
+        ret += mcgen('''
 
     visit_end_struct(v, &local_err);
     if (local_err) {
@@ -131,27 +126,24 @@ def generate_event_implement(api_name, event_name, params):
     g_assert(obj != NULL);
 
     qdict_put_obj(qmp, "data", obj);
-""")
+''')
 
-    # step 4: call qmp event api
-    ret += mcgen("""
-    emit(%(event_enum_value)s, qmp, &local_err);
+    ret += mcgen('''
+    emit(%(c_enum)s, qmp, &local_err);
 
-""",
-                 event_enum_value = c_enum_const(event_enum_name, event_name))
+''',
+                 c_enum=c_enum_const(event_enum_name, name))
 
-    # step 5: clean up
-    if params and params.members:
-        ret += mcgen("""
+    if data and data.members:
+        ret += mcgen('''
  clean:
     qmp_output_visitor_cleanup(qov);
-""")
-    ret += mcgen("""
+''')
+    ret += mcgen('''
     error_propagate(errp, local_err);
     QDECREF(qmp);
 }
-""")
-
+''')
     return ret
 
 class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
@@ -164,12 +156,11 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self.defn = ''
         self.event_names = []
     def visit_end(self):
-        self.decl += generate_enum(event_enum_name, self.event_names)
-        self.defn += generate_enum_lookup(event_enum_name, self.event_names)
+        self.decl += gen_enum(event_enum_name, self.event_names)
+        self.defn += gen_enum_lookup(event_enum_name, self.event_names)
     def visit_event(self, name, info, data):
-        api_name = _generate_event_api_name(name, data)
-        self.decl += generate_event_declaration(api_name)
-        self.defn += generate_event_implement(api_name, name, data)
+        self.decl += gen_event_send_decl(name, data)
+        self.defn += gen_event_send(name, data)
         self.event_names.append(name)
 
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index 540c359..b6990a5 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -16,22 +16,22 @@ from qapi import *
 def gen_fwd_object_or_array(name):
     return mcgen('''
 
-typedef struct %(name)s %(name)s;
+typedef struct %(c_name)s %(c_name)s;
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name))
 
 def gen_array(name, element_type):
     return mcgen('''
 
-struct %(name)s {
+struct %(c_name)s {
     union {
         %(c_type)s value;
         uint64_t padding;
     };
-    struct %(name)s *next;
+    struct %(c_name)s *next;
 };
 ''',
-                 name=c_name(name), c_type=element_type.c_type())
+                 c_name=c_name(name), c_type=element_type.c_type())
 
 def gen_struct_field(name, typ, optional):
     ret = ''
@@ -47,7 +47,7 @@ def gen_struct_field(name, typ, optional):
                  c_type=typ.c_type(), c_name=c_name(name))
     return ret
 
-def generate_struct_fields(members):
+def gen_struct_fields(members):
     ret = ''
 
     for memb in members:
@@ -57,20 +57,20 @@ def generate_struct_fields(members):
 def gen_struct(name, base, members):
     ret = mcgen('''
 
-struct %(name)s {
+struct %(c_name)s {
 ''',
-                name=c_name(name))
+                c_name=c_name(name))
 
     if base:
         ret += gen_struct_field('base', base, False)
 
-    ret += generate_struct_fields(members)
+    ret += gen_struct_fields(members)
 
     # Make sure that all structs have at least one field; this avoids
     # potential issues with attempting to malloc space for zero-length structs
     # in C, and also incompatibility with C++ (where an empty struct is size 1).
     if not base and not members:
-            ret += mcgen('''
+        ret += mcgen('''
     char qapi_dummy_field_for_empty_struct;
 ''')
 
@@ -90,9 +90,9 @@ extern const int %(c_name)s_qtypes[];
 def gen_alternate_qtypes(name, variants):
     ret = mcgen('''
 
-const int %(name)s_qtypes[QTYPE_MAX] = {
+const int %(c_name)s_qtypes[QTYPE_MAX] = {
 ''',
-                name=c_name(name))
+                c_name=c_name(name))
 
     for var in variants.variants:
         qtype = var.type.alternate_qtype()
@@ -101,7 +101,7 @@ const int %(name)s_qtypes[QTYPE_MAX] = {
         ret += mcgen('''
     [%(qtype)s] = %(enum_const)s,
 ''',
-                     qtype = qtype,
+                     qtype=qtype,
                      enum_const=c_enum_const(variants.tag_member.type.name,
                                              var.name))
 
@@ -111,25 +111,23 @@ const int %(name)s_qtypes[QTYPE_MAX] = {
     return ret
 
 def gen_union(name, base, variants):
-    name = c_name(name)
-
     ret = mcgen('''
 
-struct %(name)s {
+struct %(c_name)s {
 ''',
-                name=name)
+                c_name=c_name(name))
     if base:
-        ret += generate_struct_fields(base.members)
+        ret += gen_struct_fields(base.members)
         ret += mcgen('''
-    /* union tag is %(discriminator_type_name)s %(c_name)s */
+    /* union tag is %(c_type)s %(c_name)s */
 ''',
-                     discriminator_type_name=c_name(variants.tag_member.type.name),
+                     c_type=c_name(variants.tag_member.type.name),
                      c_name=c_name(variants.tag_member.name))
     else:
         ret += mcgen('''
-    %(discriminator_type_name)s kind;
+    %(c_type)s kind;
 ''',
-                     discriminator_type_name=c_name(variants.tag_member.type.name))
+                     c_type=c_name(variants.tag_member.type.name))
 
     ret += mcgen('''
     union {
@@ -150,18 +148,18 @@ struct %(name)s {
 
     return ret
 
-def generate_type_cleanup_decl(name):
+def gen_type_cleanup_decl(name):
     ret = mcgen('''
 
-void qapi_free_%(name)s(%(name)s *obj);
+void qapi_free_%(c_name)s(%(c_name)s *obj);
 ''',
-                name=c_name(name))
+                c_name=c_name(name))
     return ret
 
-def generate_type_cleanup(name):
+def gen_type_cleanup(name):
     ret = mcgen('''
 
-void qapi_free_%(name)s(%(name)s *obj)
+void qapi_free_%(c_name)s(%(c_name)s *obj)
 {
     QapiDeallocVisitor *md;
     Visitor *v;
@@ -172,11 +170,11 @@ void qapi_free_%(name)s(%(name)s *obj)
 
     md = qapi_dealloc_visitor_new();
     v = qapi_dealloc_get_visitor(md);
-    visit_type_%(name)s(v, &obj, NULL, NULL);
+    visit_type_%(c_name)s(v, &obj, NULL, NULL);
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                name=c_name(name))
+                c_name=c_name(name))
     return ret
 
 class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
@@ -207,18 +205,18 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         # that have the functions defined, so generate them only with
         # option -b (do_builtins).
     def _gen_type_cleanup(self, name):
-        self.decl += generate_type_cleanup_decl(name)
-        self.defn += generate_type_cleanup(name)
+        self.decl += gen_type_cleanup_decl(name)
+        self.defn += gen_type_cleanup(name)
     def visit_enum_type(self, name, info, values):
-        self.fwdecl += generate_enum(name, values)
-        self.fwdefn += generate_enum_lookup(name, values)
+        self.fwdecl += gen_enum(name, values)
+        self.fwdefn += gen_enum_lookup(name, values)
     def visit_array_type(self, name, info, element_type):
         if isinstance(element_type, QAPISchemaBuiltinType):
             self.btin += gen_fwd_object_or_array(name)
             self.btin += gen_array(name, element_type)
-            self.btin += generate_type_cleanup_decl(name)
+            self.btin += gen_type_cleanup_decl(name)
             if do_builtins:
-                self.defn += generate_type_cleanup(name)
+                self.defn += gen_type_cleanup(name)
         else:
             self.fwdecl += gen_fwd_object_or_array(name)
             self.decl += gen_array(name, element_type)
diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 135e7c1..97b1f05 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -18,18 +18,19 @@ import re
 implicit_structs_seen = set()
 struct_fields_seen = set()
 
-def generate_visit_implicit_struct(type):
-    if type in implicit_structs_seen:
+def gen_visit_implicit_struct(typ):
+    if typ in implicit_structs_seen:
         return ''
-    implicit_structs_seen.add(type)
+    implicit_structs_seen.add(typ)
+
     ret = ''
-    if type.name not in struct_fields_seen:
+    if typ.name not in struct_fields_seen:
         # Need a forward declaration
         ret += mcgen('''
 
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
-                     c_type=type.c_name())
+                     c_type=typ.c_name())
 
     ret += mcgen('''
 
@@ -45,35 +46,35 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
     error_propagate(errp, err);
 }
 ''',
-                 c_type=type.c_name())
+                 c_type=typ.c_name())
     return ret
 
-def generate_visit_struct_fields(name, members, base = None):
+def gen_visit_struct_fields(name, base, members):
     struct_fields_seen.add(name)
 
     ret = ''
 
     if base:
-        ret += generate_visit_implicit_struct(base)
+        ret += gen_visit_implicit_struct(base)
 
     ret += mcgen('''
 
-static void visit_type_%(name)s_fields(Visitor *m, %(name)s **obj, Error **errp)
+static void visit_type_%(c_name)s_fields(Visitor *m, %(c_name)s **obj, Error **errp)
 {
     Error *err = NULL;
 
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name))
     push_indent()
 
     if base:
         ret += mcgen('''
-visit_type_implicit_%(type)s(m, &(*obj)->%(c_name)s, &err);
+visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);
 if (err) {
     goto out;
 }
 ''',
-                     type=base.c_name(), c_name=c_name('base'))
+                     c_type=base.c_name(), c_name=c_name('base'))
 
     for memb in members:
         if memb.optional:
@@ -85,9 +86,9 @@ if (!err && (*obj)->has_%(c_name)s) {
             push_indent()
 
         ret += mcgen('''
-visit_type_%(type)s(m, &(*obj)->%(c_name)s, "%(name)s", &err);
+visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, "%(name)s", &err);
 ''',
-                     type=memb.type.c_name(), c_name=c_name(memb.name),
+                     c_type=memb.type.c_name(), c_name=c_name(memb.name),
                      name=memb.name)
 
         if memb.optional:
@@ -102,7 +103,7 @@ if (err) {
 ''')
 
     pop_indent()
-    if re.search('^ *goto out\\;', ret, re.MULTILINE):
+    if re.search('^ *goto out;', ret, re.MULTILINE):
         ret += mcgen('''
 
 out:
@@ -113,8 +114,7 @@ out:
 ''')
     return ret
 
-
-def generate_visit_struct_body(name):
+def gen_visit_struct_body(name):
     ret = mcgen('''
     Error *err = NULL;
 
@@ -128,20 +128,18 @@ def generate_visit_struct_body(name):
     error_propagate(errp, err);
 ''',
                 name=name, c_name=c_name(name))
-
     return ret
 
 def gen_visit_struct(name, base, members):
-    ret = generate_visit_struct_fields(name, members, base)
-
+    ret = gen_visit_struct_fields(name, base, members)
     ret += mcgen('''
 
-void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name))
 
-    ret += generate_visit_struct_body(name)
+    ret += gen_visit_struct_body(name)
 
     ret += mcgen('''
 }
@@ -151,7 +149,7 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 def gen_visit_list(name, element_type):
     return mcgen('''
 
-void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
     GenericList *i, **prev;
@@ -164,7 +162,7 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
     for (prev = (GenericList **)obj;
          !err && (i = visit_next_list(m, prev, &err)) != NULL;
          prev = &i) {
-        %(name)s *native_i = (%(name)s *)i;
+        %(c_name)s *native_i = (%(c_name)s *)i;
         visit_type_%(c_elt_type)s(m, &native_i->value, NULL, &err);
     }
 
@@ -175,10 +173,9 @@ out:
     error_propagate(errp, err);
 }
 ''',
-                 name=c_name(name),
-                 c_elt_type=element_type.c_name())
+                 c_name=c_name(name), c_elt_type=element_type.c_name())
 
-def generate_visit_enum(name):
+def gen_visit_enum(name):
     return mcgen('''
 
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error **errp)
@@ -191,33 +188,32 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error
 def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
-void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
 
-    visit_start_implicit_struct(m, (void**) obj, sizeof(%(name)s), &err);
+    visit_start_implicit_struct(m, (void**) obj, sizeof(%(c_name)s), &err);
     if (err) {
         goto out;
     }
-    visit_get_next_type(m, (int*) &(*obj)->kind, %(name)s_qtypes, name, &err);
+    visit_get_next_type(m, (int*) &(*obj)->kind, %(c_name)s_qtypes, name, &err);
     if (err) {
         goto out_end;
     }
     switch ((*obj)->kind) {
 ''',
-                name=c_name(name))
+                c_name=c_name(name))
 
     for var in variants.variants:
-        enum_full_value = c_enum_const(variants.tag_member.type.name,
-                                       var.name)
         ret += mcgen('''
-    case %(enum_full_value)s:
+    case %(case)s:
         visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, name, &err);
         break;
 ''',
-                enum_full_value = enum_full_value,
-                c_type=var.type.c_name(),
-                c_name=c_name(var.name))
+                     case=c_enum_const(variants.tag_member.type.name,
+                                       var.name),
+                     c_type=var.type.c_name(),
+                     c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
@@ -239,11 +235,11 @@ def gen_visit_union(name, base, variants):
 
     if base:
         members = [m for m in base.members if m != variants.tag_member]
-        ret += generate_visit_struct_fields(name, members)
+        ret += gen_visit_struct_fields(name, None, members)
 
     for var in variants.variants:
         if var.flat:
-            ret += generate_visit_implicit_struct(var.type)
+            ret += gen_visit_implicit_struct(var.type)
 
     ret += mcgen('''
 
@@ -261,19 +257,19 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
 
     if base:
         ret += mcgen('''
-        visit_type_%(name)s_fields(m, obj, &err);
+        visit_type_%(c_name)s_fields(m, obj, &err);
         if (err) {
             goto out_obj;
         }
 ''',
-                     name=c_name(name))
+                     c_name=c_name(name))
 
-    disc_key = variants.tag_member.name
+    tag_key = variants.tag_member.name
     if not variants.tag_name:
         # we pointlessly use a different key for simple unions
-        disc_key = 'type'
+        tag_key = 'type'
     ret += mcgen('''
-        visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err);
+        visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, "%(name)s", &err);
         if (err) {
             goto out_obj;
         }
@@ -282,25 +278,30 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
         }
         switch ((*obj)->%(c_name)s) {
 ''',
-                 disc_type=variants.tag_member.type.c_name(),
+                 c_type=variants.tag_member.type.c_name(),
                  c_name=c_name(variants.tag_member.name),
-                 disc_key = disc_key)
+                 name=tag_key)
 
     for var in variants.variants:
-        if not var.flat:
-            fmt = 'visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, "data", &err);'
+        ret += mcgen('''
+        case %(case)s:
+''',
+                     case=c_enum_const(variants.tag_member.type.name, var.name))
+        if var.flat:
+            ret += mcgen('''
+            visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);
+''',
+                         c_type=var.type.c_name(),
+                         c_name=c_name(var.name))
         else:
-            fmt = 'visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);'
-
-        enum_full_value = c_enum_const(variants.tag_member.type.name, var.name)
+            ret += mcgen('''
+            visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, "data", &err);
+''',
+                         c_type=var.type.c_name(),
+                         c_name=c_name(var.name))
         ret += mcgen('''
-        case %(enum_full_value)s:
-            ''' + fmt + '''
             break;
-''',
-                enum_full_value = enum_full_value,
-                c_type=var.type.c_name(),
-                c_name=c_name(var.name))
+''')
 
     ret += mcgen('''
         default:
@@ -351,7 +352,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         # do_builtins (option -b) to provide control
     def visit_enum_type(self, name, info, values):
         self.decl += gen_visit_decl(name, scalar=True)
-        self.defn += generate_visit_enum(name)
+        self.defn += gen_visit_enum(name)
     def visit_array_type(self, name, info, element_type):
         decl = gen_visit_decl(name)
         defn = gen_visit_list(name, element_type)
@@ -419,7 +420,7 @@ fdef.write(mcgen('''
 #include "qemu-common.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix = prefix))
+                 prefix=prefix))
 
 fdecl.write(mcgen('''
 #include "qapi/visitor.h"
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 699a656..4d47214 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1016,16 +1016,6 @@ class QAPISchema(object):
         self._def_exprs()
         self.check()
 
-    def get_exprs(self):
-        def expr_name(expr):
-            name = expr.get('enum') or expr.get('union') \
-                   or expr.get('alternate') or expr.get('struct') \
-                   or expr.get('command') or expr.get('event')
-            assert name
-            return name
-        return sorted([expr_elem['expr'] for expr_elem in self.exprs],
-                      key=expr_name)
-
     def _def_entity(self, ent):
         assert ent.name not in self.entity_dict
         self.entity_dict[ent.name] = ent
@@ -1207,23 +1197,6 @@ class QAPISchema(object):
 # Code generation helpers
 #
 
-def parse_args(typeinfo):
-    if isinstance(typeinfo, str):
-        struct = find_struct(typeinfo)
-        assert struct != None
-        typeinfo = struct['data']
-
-    for member in typeinfo:
-        argname = member
-        argentry = typeinfo[member]
-        optional = False
-        if member.startswith('*'):
-            argname = member[1:]
-            optional = True
-        # Todo: allow argentry to be OrderedDict, for providing the
-        # value of an optional argument.
-        yield (argname, argentry, optional)
-
 def camel_case(name):
     new_name = ''
     first = True
@@ -1304,67 +1277,9 @@ def c_name(name, protect=True):
         return "q_" + name
     return name.translate(c_name_trans)
 
-# Map type @name to the C typedef name for the list form.
-#
-# ['Name'] -> 'NameList', ['x-Foo'] -> 'x_FooList', ['int'] -> 'intList'
-def c_list_type(name):
-    return type_name(name) + 'List'
-
-# Map type @value to the C typedef form.
-#
-# Used for converting 'type' from a 'member':'type' qapi definition
-# into the alphanumeric portion of the type for a generated C parameter,
-# as well as generated C function names.  See c_type() for the rest of
-# the conversion such as adding '*' on pointer types.
-# 'int' -> 'int', '[x-Foo]' -> 'x_FooList', '__a.b_c' -> '__a_b_c'
-def type_name(value):
-    if type(value) == list:
-        return c_list_type(value[0])
-    if value in builtin_types.keys():
-        return value
-    return c_name(value)
-
 eatspace = '\033EATSPACE.'
 pointer_suffix = ' *' + eatspace
 
-# Map type @name to its C type expression.
-# If @is_param, const-qualify the string type.
-#
-# This function is used for computing the full C type of 'member':'name'.
-# A special suffix is added in c_type() for pointer types, and it's
-# stripped in mcgen(). So please notice this when you check the return
-# value of c_type() outside mcgen().
-def c_type(value, is_param=False):
-    if value == 'str':
-        if is_param:
-            return 'const char' + pointer_suffix
-        return 'char' + pointer_suffix
-
-    elif value == 'int':
-        return 'int64_t'
-    elif (value == 'int8' or value == 'int16' or value == 'int32' or
-          value == 'int64' or value == 'uint8' or value == 'uint16' or
-          value == 'uint32' or value == 'uint64'):
-        return value + '_t'
-    elif value == 'size':
-        return 'uint64_t'
-    elif value == 'bool':
-        return 'bool'
-    elif value == 'number':
-        return 'double'
-    elif type(value) == list:
-        return c_list_type(value[0]) + pointer_suffix
-    elif is_enum(value):
-        return c_name(value)
-    elif value == None:
-        return 'void'
-    elif value in events:
-        return camel_case(value) + 'Event' + pointer_suffix
-    else:
-        # complex type name
-        assert isinstance(value, str) and value != ""
-        return c_name(value) + pointer_suffix
-
 def genindent(count):
     ret = ""
     for i in range(count):
@@ -1417,60 +1332,57 @@ def guardend(name):
 ''',
                  name=guardname(name))
 
-def generate_enum_lookup(name, values):
+def gen_enum_lookup(name, values):
     ret = mcgen('''
 
-const char *const %(name)s_lookup[] = {
+const char *const %(c_name)s_lookup[] = {
 ''',
-                name=c_name(name))
+                c_name=c_name(name))
     for value in values:
         index = c_enum_const(name, value)
         ret += mcgen('''
     [%(index)s] = "%(value)s",
 ''',
-                     index = index, value = value)
+                     index=index, value=value)
 
     max_index = c_enum_const(name, 'MAX')
     ret += mcgen('''
     [%(max_index)s] = NULL,
 };
 ''',
-        max_index=max_index)
+                 max_index=max_index)
     return ret
 
-def generate_enum(name, values):
-    name = c_name(name)
-    lookup_decl = mcgen('''
-
-extern const char *const %(name)s_lookup[];
-''',
-                name=name)
-
-    enum_decl = mcgen('''
-
-typedef enum %(name)s {
-''',
-                name=name)
-
+def gen_enum(name, values):
     # append automatically generated _MAX value
-    enum_values = values + [ 'MAX' ]
+    enum_values = values + ['MAX']
+
+    ret = mcgen('''
+
+typedef enum %(c_name)s {
+''',
+                c_name=c_name(name))
 
     i = 0
     for value in enum_values:
-        enum_full_value = c_enum_const(name, value)
-        enum_decl += mcgen('''
-    %(enum_full_value)s = %(i)d,
+        ret += mcgen('''
+    %(c_enum)s = %(i)d,
 ''',
-                     enum_full_value = enum_full_value,
+                     c_enum=c_enum_const(name, value),
                      i=i)
         i += 1
 
-    enum_decl += mcgen('''
-} %(name)s;
+    ret += mcgen('''
+} %(c_name)s;
 ''',
-                 name=name)
+                 c_name=c_name(name))
 
-    return enum_decl + lookup_decl
+    ret += mcgen('''
+
+extern const char *const %(c_name)s_lookup[];
+''',
+                 c_name=c_name(name))
+    return ret
 
 #
 # Common command line parsing
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 34/47] qapi-visit: Rearrange code a bit

[Markus Armbruster]

Move gen_visit_decl() to a better place.  Inline
generate_visit_struct_body().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-visit.py | 42 ++++++++++++++++--------------------------
 1 file changed, 16 insertions(+), 26 deletions(-)

diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 97b1f05..2813bb3 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -18,6 +18,15 @@ import re
 implicit_structs_seen = set()
 struct_fields_seen = set()
 
+def gen_visit_decl(name, scalar=False):
+    c_type = c_name(name) + ' *'
+    if not scalar:
+        c_type += '*'
+    return mcgen('''
+void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
+''',
+                 c_name=c_name(name), c_type=c_type)
+
 def gen_visit_implicit_struct(typ):
     if typ in implicit_structs_seen:
         return ''
@@ -114,8 +123,12 @@ out:
 ''')
     return ret
 
-def gen_visit_struct_body(name):
-    ret = mcgen('''
+def gen_visit_struct(name, base, members):
+    ret = gen_visit_struct_fields(name, base, members)
+    ret += mcgen('''
+
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
+{
     Error *err = NULL;
 
     visit_start_struct(m, (void **)obj, "%(name)s", name, sizeof(%(c_name)s), &err);
@@ -126,24 +139,10 @@ def gen_visit_struct_body(name):
         visit_end_struct(m, &err);
     }
     error_propagate(errp, err);
+}
 ''',
                 name=name, c_name=c_name(name))
-    return ret
 
-def gen_visit_struct(name, base, members):
-    ret = gen_visit_struct_fields(name, base, members)
-    ret += mcgen('''
-
-void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
-{
-''',
-                 c_name=c_name(name))
-
-    ret += gen_visit_struct_body(name)
-
-    ret += mcgen('''
-}
-''')
     return ret
 
 def gen_visit_list(name, element_type):
@@ -322,15 +321,6 @@ out:
 
     return ret
 
-def gen_visit_decl(name, scalar=False):
-    c_type = c_name(name) + ' *'
-    if not scalar:
-        c_type += '*'
-    return mcgen('''
-void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
-''',
-                 c_name=c_name(name), c_type=c_type)
-
 class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def __init__(self):
         self.decl = None
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 35/47] qapi-commands: Rearrange code

[Markus Armbruster]

Rename gen_marshal_input() to gen_marshal(), because the generated
function marshals both arguments and results.

Rename gen_visitor_input_containers_decl() to gen_marshal_vars(), and
move the other variable declarations there, too.

Rename gen_visitor_input_block()() to gen_marshal_input_visit(), and
rearrange its code slightly.

Rename gen_marshal_input_decl() to gen_marshal_proto(), because the
result isn't a full declaration, unlike gen_command_decl()'s.

New gen_marshal_decl() actually returns a full declaration.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 95 ++++++++++++++++++++++--------------------------
 1 file changed, 43 insertions(+), 52 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 5d11032..728f15b 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -56,6 +56,7 @@ def gen_call(name, args, rets):
 
     push_indent()
     ret = mcgen('''
+
 %(lhs)sqmp_%(c_name)s(%(args)s&local_err);
 ''',
                 c_name=c_name(name), args=argstr, lhs=lhs)
@@ -69,25 +70,31 @@ qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
     pop_indent()
     return ret
 
-def gen_visitor_input_containers_decl(args):
-    ret = ''
+def gen_marshal_vars(args, rets):
+    ret = mcgen('''
+    Error *local_err = NULL;
+''')
 
     push_indent()
+
+    if rets:
+        # FIXME fishy: only pointers are initialized
+        if rets.c_null() == 'NULL':
+            retval = '%s retval = NULL;' % rets.c_type()
+        else:
+            retval = '%s retval;' % rets.c_type()
+        ret += mcgen('''
+%(retval)s
+''',
+                     retval=retval)
+
     if args:
         ret += mcgen('''
 QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
 QapiDeallocVisitor *md;
 Visitor *v;
 ''')
-    pop_indent()
 
-    return ret
-
-def gen_visitor_input_vars_decl(args):
-    ret = ''
-    push_indent()
-
-    if args:
         for memb in args.members:
             if memb.optional:
                 ret += mcgen('''
@@ -100,14 +107,18 @@ bool has_%(c_name)s = false;
                          c_name=c_name(memb.name),
                          c_type=memb.type.c_type(),
                          c_null=memb.type.c_null())
+        ret += '\n'
+    else:
+        ret += mcgen('''
+
+(void)args;
+''')
 
     pop_indent()
     return ret
 
-def gen_visitor_input_block(args, dealloc=False):
+def gen_marshal_input_visit(args, dealloc=False):
     ret = ''
-    errparg = '&local_err'
-    errarg = 'local_err'
 
     if not args:
         return ret
@@ -123,6 +134,8 @@ md = qapi_dealloc_visitor_new();
 v = qapi_dealloc_get_visitor(md);
 ''')
     else:
+        errparg = '&local_err'
+        errarg = 'local_err'
         ret += mcgen('''
 v = qmp_input_get_visitor(mi);
 ''')
@@ -160,10 +173,7 @@ qapi_dealloc_visitor_cleanup(md);
     return ret
 
 def gen_marshal_output(name, rets):
-    if not rets:
-        return ''
-
-    ret = mcgen('''
+    return mcgen('''
 
 static void qmp_marshal_output_%(c_cmd_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
 {
@@ -188,50 +198,31 @@ out:
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_type=rets.c_type(), c_cmd_name=c_name(name),
-                c_name=rets.c_name())
+                 c_type=rets.c_type(), c_cmd_name=c_name(name),
+                 c_name=rets.c_name())
 
-    return ret
-
-def gen_marshal_input_decl(name):
+def gen_marshal_proto(name):
     ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
     if not middle_mode:
         ret = 'static ' + ret
     return ret
 
-def gen_marshal_input(name, args, rets):
-    hdr = gen_marshal_input_decl(name)
+def gen_marshal_decl(name):
+    return mcgen('''
+%(proto)s;
+''',
+                 proto=gen_marshal_proto(name))
 
+def gen_marshal(name, args, rets):
     ret = mcgen('''
 
-%(header)s
+%(proto)s
 {
-    Error *local_err = NULL;
 ''',
-                header=hdr)
-
-    if rets:
-        # FIXME fishy: only pointers are initialized
-        if rets.c_null() == 'NULL':
-            retval = '    %s retval = NULL;' % rets.c_type()
-        else:
-            retval = '    %s retval;' % rets.c_type()
-        ret += mcgen('''
-%(retval)s
-''',
-                     retval=retval)
-
-    if args:
-        ret += gen_visitor_input_containers_decl(args)
-        ret += gen_visitor_input_vars_decl(args) + '\n'
-        ret += gen_visitor_input_block(args) + '\n'
-    else:
-        ret += mcgen('''
-
-    (void)args;
-
-''')
+                proto=gen_marshal_proto(name))
 
+    ret += gen_marshal_vars(args, rets)
+    ret += gen_marshal_input_visit(args)
     ret += gen_call(name, args, rets)
 
     if re.search('^ *goto out;', ret, re.MULTILINE):
@@ -242,7 +233,7 @@ out:
     ret += mcgen('''
     error_propagate(errp, local_err);
 ''')
-    ret += gen_visitor_input_block(args, dealloc=True)
+    ret += gen_marshal_input_visit(args, dealloc=True)
     ret += mcgen('''
 }
 ''')
@@ -296,8 +287,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         if rets:
             self.defn += gen_marshal_output(name, rets)
         if middle_mode:
-            self.decl += gen_marshal_input_decl(name) + ';\n'
-        self.defn += gen_marshal_input(name, args, rets)
+            self.decl += gen_marshal_decl(name)
+        self.defn += gen_marshal(name, args, rets)
         if not middle_mode:
             self.regy += gen_register_command(name, success_response)
 
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 36/47] qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()

[Markus Armbruster]

These functions marshal both input and output.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 docs/qapi-code-gen.txt        |   4 +-
 docs/writing-qmp-commands.txt |   8 +-
 monitor.c                     |   2 +-
 qmp-commands.hx               | 240 +++++++++++++++++++++---------------------
 scripts/qapi-commands.py      |   4 +-
 5 files changed, 129 insertions(+), 129 deletions(-)

diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index 17588c1..83af531 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -740,7 +740,7 @@ Example:
         qapi_dealloc_visitor_cleanup(md);
     }
 
-    static void qmp_marshal_input_my_command(QDict *args, QObject **ret, Error **errp)
+    static void qmp_marshal_my_command(QDict *args, QObject **ret, Error **errp)
     {
         Error *local_err = NULL;
         UserDefOne *retval = NULL;
@@ -774,7 +774,7 @@ Example:
 
     static void qmp_init_marshal(void)
     {
-        qmp_register_command("my-command", qmp_marshal_input_my_command, QCO_NO_OPTIONS);
+        qmp_register_command("my-command", qmp_marshal_my_command, QCO_NO_OPTIONS);
     }
 
     qapi_init(qmp_init_marshal);
diff --git a/docs/writing-qmp-commands.txt b/docs/writing-qmp-commands.txt
index ab1fdd3..c65bdc6 100644
--- a/docs/writing-qmp-commands.txt
+++ b/docs/writing-qmp-commands.txt
@@ -127,7 +127,7 @@ following in the botton:
     {
         .name       = "hello-world",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_hello_world,
+        .mhandler.cmd_new = qmp_marshal_hello_world,
     },
 
 You're done. Now build qemu, run it as suggested in the "Testing" section,
@@ -179,7 +179,7 @@ The last step is to update the qmp-commands.hx file:
     {
         .name       = "hello-world",
         .args_type  = "message:s?",
-        .mhandler.cmd_new = qmp_marshal_input_hello_world,
+        .mhandler.cmd_new = qmp_marshal_hello_world,
     },
 
 Notice that the "args_type" member got our "message" argument. The character
@@ -461,7 +461,7 @@ The last step is to add the correspoding entry in the qmp-commands.hx file:
     {
         .name       = "query-alarm-clock",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_alarm_clock,
+        .mhandler.cmd_new = qmp_marshal_query_alarm_clock,
     },
 
 Time to test the new command. Build qemu, run it as described in the "Testing"
@@ -607,7 +607,7 @@ To test this you have to add the corresponding qmp-commands.hx entry:
     {
         .name       = "query-alarm-methods",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_alarm_methods,
+        .mhandler.cmd_new = qmp_marshal_query_alarm_methods,
     },
 
 Now Build qemu, run it as explained in the "Testing" section and try our new
diff --git a/monitor.c b/monitor.c
index aeea2b5..16672f1 100644
--- a/monitor.c
+++ b/monitor.c
@@ -5102,7 +5102,7 @@ static QObject *get_qmp_greeting(void)
 {
     QObject *ver = NULL;
 
-    qmp_marshal_input_query_version(NULL, &ver, NULL);
+    qmp_marshal_query_version(NULL, &ver, NULL);
     return qobject_from_jsonf("{'QMP':{'version': %p,'capabilities': []}}",ver);
 }
 
diff --git a/qmp-commands.hx b/qmp-commands.hx
index a05d25f..6760ddb 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -63,7 +63,7 @@ EQMP
     {
         .name       = "quit",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_quit,
+        .mhandler.cmd_new = qmp_marshal_quit,
     },
 
 SQMP
@@ -84,7 +84,7 @@ EQMP
     {
         .name       = "eject",
         .args_type  = "force:-f,device:B",
-        .mhandler.cmd_new = qmp_marshal_input_eject,
+        .mhandler.cmd_new = qmp_marshal_eject,
     },
 
 SQMP
@@ -110,7 +110,7 @@ EQMP
     {
         .name       = "change",
         .args_type  = "device:B,target:F,arg:s?",
-        .mhandler.cmd_new = qmp_marshal_input_change,
+        .mhandler.cmd_new = qmp_marshal_change,
     },
 
 SQMP
@@ -146,7 +146,7 @@ EQMP
     {
         .name       = "screendump",
         .args_type  = "filename:F",
-        .mhandler.cmd_new = qmp_marshal_input_screendump,
+        .mhandler.cmd_new = qmp_marshal_screendump,
     },
 
 SQMP
@@ -169,7 +169,7 @@ EQMP
     {
         .name       = "stop",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_stop,
+        .mhandler.cmd_new = qmp_marshal_stop,
     },
 
 SQMP
@@ -190,7 +190,7 @@ EQMP
     {
         .name       = "cont",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_cont,
+        .mhandler.cmd_new = qmp_marshal_cont,
     },
 
 SQMP
@@ -211,7 +211,7 @@ EQMP
     {
         .name       = "system_wakeup",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_system_wakeup,
+        .mhandler.cmd_new = qmp_marshal_system_wakeup,
     },
 
 SQMP
@@ -232,7 +232,7 @@ EQMP
     {
         .name       = "system_reset",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_system_reset,
+        .mhandler.cmd_new = qmp_marshal_system_reset,
     },
 
 SQMP
@@ -253,7 +253,7 @@ EQMP
     {
         .name       = "system_powerdown",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_system_powerdown,
+        .mhandler.cmd_new = qmp_marshal_system_powerdown,
     },
 
 SQMP
@@ -310,7 +310,7 @@ EQMP
     {
         .name       = "device_del",
         .args_type  = "id:s",
-        .mhandler.cmd_new = qmp_marshal_input_device_del,
+        .mhandler.cmd_new = qmp_marshal_device_del,
     },
 
 SQMP
@@ -333,7 +333,7 @@ EQMP
     {
         .name       = "send-key",
         .args_type  = "keys:q,hold-time:i?",
-        .mhandler.cmd_new = qmp_marshal_input_send_key,
+        .mhandler.cmd_new = qmp_marshal_send_key,
     },
 
 SQMP
@@ -364,7 +364,7 @@ EQMP
     {
         .name       = "cpu",
         .args_type  = "index:i",
-        .mhandler.cmd_new = qmp_marshal_input_cpu,
+        .mhandler.cmd_new = qmp_marshal_cpu,
     },
 
 SQMP
@@ -389,7 +389,7 @@ EQMP
     {
         .name       = "cpu-add",
         .args_type  = "id:i",
-        .mhandler.cmd_new = qmp_marshal_input_cpu_add,
+        .mhandler.cmd_new = qmp_marshal_cpu_add,
     },
 
 SQMP
@@ -412,7 +412,7 @@ EQMP
     {
         .name       = "memsave",
         .args_type  = "val:l,size:i,filename:s,cpu:i?",
-        .mhandler.cmd_new = qmp_marshal_input_memsave,
+        .mhandler.cmd_new = qmp_marshal_memsave,
     },
 
 SQMP
@@ -441,7 +441,7 @@ EQMP
     {
         .name       = "pmemsave",
         .args_type  = "val:l,size:i,filename:s",
-        .mhandler.cmd_new = qmp_marshal_input_pmemsave,
+        .mhandler.cmd_new = qmp_marshal_pmemsave,
     },
 
 SQMP
@@ -469,7 +469,7 @@ EQMP
     {
         .name       = "inject-nmi",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_inject_nmi,
+        .mhandler.cmd_new = qmp_marshal_inject_nmi,
     },
 
 SQMP
@@ -492,7 +492,7 @@ EQMP
     {
         .name       = "ringbuf-write",
         .args_type  = "device:s,data:s,format:s?",
-        .mhandler.cmd_new = qmp_marshal_input_ringbuf_write,
+        .mhandler.cmd_new = qmp_marshal_ringbuf_write,
     },
 
 SQMP
@@ -523,7 +523,7 @@ EQMP
     {
         .name       = "ringbuf-read",
         .args_type  = "device:s,size:i,format:s?",
-        .mhandler.cmd_new = qmp_marshal_input_ringbuf_read,
+        .mhandler.cmd_new = qmp_marshal_ringbuf_read,
     },
 
 SQMP
@@ -559,7 +559,7 @@ EQMP
     {
         .name       = "xen-save-devices-state",
         .args_type  = "filename:F",
-    .mhandler.cmd_new = qmp_marshal_input_xen_save_devices_state,
+    .mhandler.cmd_new = qmp_marshal_xen_save_devices_state,
     },
 
 SQMP
@@ -586,7 +586,7 @@ EQMP
     {
         .name       = "xen-set-global-dirty-log",
         .args_type  = "enable:b",
-        .mhandler.cmd_new = qmp_marshal_input_xen_set_global_dirty_log,
+        .mhandler.cmd_new = qmp_marshal_xen_set_global_dirty_log,
     },
 
 SQMP
@@ -610,7 +610,7 @@ EQMP
     {
         .name       = "migrate",
         .args_type  = "detach:-d,blk:-b,inc:-i,uri:s",
-        .mhandler.cmd_new = qmp_marshal_input_migrate,
+        .mhandler.cmd_new = qmp_marshal_migrate,
     },
 
 SQMP
@@ -643,7 +643,7 @@ EQMP
     {
         .name       = "migrate_cancel",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_migrate_cancel,
+        .mhandler.cmd_new = qmp_marshal_migrate_cancel,
     },
 
 SQMP
@@ -664,7 +664,7 @@ EQMP
     {
         .name       = "migrate-incoming",
         .args_type  = "uri:s",
-        .mhandler.cmd_new = qmp_marshal_input_migrate_incoming,
+        .mhandler.cmd_new = qmp_marshal_migrate_incoming,
     },
 
 SQMP
@@ -692,7 +692,7 @@ EQMP
     {
         .name       = "migrate-set-cache-size",
         .args_type  = "value:o",
-        .mhandler.cmd_new = qmp_marshal_input_migrate_set_cache_size,
+        .mhandler.cmd_new = qmp_marshal_migrate_set_cache_size,
     },
 
 SQMP
@@ -715,7 +715,7 @@ EQMP
     {
         .name       = "query-migrate-cache-size",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_migrate_cache_size,
+        .mhandler.cmd_new = qmp_marshal_query_migrate_cache_size,
     },
 
 SQMP
@@ -737,7 +737,7 @@ EQMP
     {
         .name       = "migrate_set_speed",
         .args_type  = "value:o",
-        .mhandler.cmd_new = qmp_marshal_input_migrate_set_speed,
+        .mhandler.cmd_new = qmp_marshal_migrate_set_speed,
     },
 
 SQMP
@@ -760,7 +760,7 @@ EQMP
     {
         .name       = "migrate_set_downtime",
         .args_type  = "value:T",
-        .mhandler.cmd_new = qmp_marshal_input_migrate_set_downtime,
+        .mhandler.cmd_new = qmp_marshal_migrate_set_downtime,
     },
 
 SQMP
@@ -785,7 +785,7 @@ EQMP
         .args_type  = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
         .params     = "protocol hostname port tls-port cert-subject",
         .help       = "set migration information for remote display",
-        .mhandler.cmd_new = qmp_marshal_input_client_migrate_info,
+        .mhandler.cmd_new = qmp_marshal_client_migrate_info,
     },
 
 SQMP
@@ -819,7 +819,7 @@ EQMP
         .args_type  = "paging:b,protocol:s,begin:i?,end:i?,format:s?",
         .params     = "-p protocol [begin] [length] [format]",
         .help       = "dump guest memory to file",
-        .mhandler.cmd_new = qmp_marshal_input_dump_guest_memory,
+        .mhandler.cmd_new = qmp_marshal_dump_guest_memory,
     },
 
 SQMP
@@ -855,7 +855,7 @@ EQMP
     {
         .name       = "query-dump-guest-memory-capability",
         .args_type  = "",
-    .mhandler.cmd_new = qmp_marshal_input_query_dump_guest_memory_capability,
+    .mhandler.cmd_new = qmp_marshal_query_dump_guest_memory_capability,
     },
 
 SQMP
@@ -904,7 +904,7 @@ EQMP
     {
         .name       = "netdev_del",
         .args_type  = "id:s",
-        .mhandler.cmd_new = qmp_marshal_input_netdev_del,
+        .mhandler.cmd_new = qmp_marshal_netdev_del,
     },
 
 SQMP
@@ -954,7 +954,7 @@ EQMP
     {
         .name       = "object-del",
         .args_type  = "id:s",
-        .mhandler.cmd_new = qmp_marshal_input_object_del,
+        .mhandler.cmd_new = qmp_marshal_object_del,
     },
 
 SQMP
@@ -979,7 +979,7 @@ EQMP
     {
         .name       = "block_resize",
         .args_type  = "device:s?,node-name:s?,size:o",
-        .mhandler.cmd_new = qmp_marshal_input_block_resize,
+        .mhandler.cmd_new = qmp_marshal_block_resize,
     },
 
 SQMP
@@ -1004,7 +1004,7 @@ EQMP
     {
         .name       = "block-stream",
         .args_type  = "device:B,base:s?,speed:o?,backing-file:s?,on-error:s?",
-        .mhandler.cmd_new = qmp_marshal_input_block_stream,
+        .mhandler.cmd_new = qmp_marshal_block_stream,
     },
 
 SQMP
@@ -1047,7 +1047,7 @@ EQMP
     {
         .name       = "block-commit",
         .args_type  = "device:B,base:s?,top:s?,backing-file:s?,speed:o?",
-        .mhandler.cmd_new = qmp_marshal_input_block_commit,
+        .mhandler.cmd_new = qmp_marshal_block_commit,
     },
 
 SQMP
@@ -1111,7 +1111,7 @@ EQMP
         .name       = "drive-backup",
         .args_type  = "sync:s,device:B,target:s,speed:i?,mode:s?,format:s?,"
                       "bitmap:s?,on-source-error:s?,on-target-error:s?",
-        .mhandler.cmd_new = qmp_marshal_input_drive_backup,
+        .mhandler.cmd_new = qmp_marshal_drive_backup,
     },
 
 SQMP
@@ -1165,7 +1165,7 @@ EQMP
         .name       = "blockdev-backup",
         .args_type  = "sync:s,device:B,target:B,speed:i?,"
                       "on-source-error:s?,on-target-error:s?",
-        .mhandler.cmd_new = qmp_marshal_input_blockdev_backup,
+        .mhandler.cmd_new = qmp_marshal_blockdev_backup,
     },
 
 SQMP
@@ -1205,33 +1205,33 @@ EQMP
     {
         .name       = "block-job-set-speed",
         .args_type  = "device:B,speed:o",
-        .mhandler.cmd_new = qmp_marshal_input_block_job_set_speed,
+        .mhandler.cmd_new = qmp_marshal_block_job_set_speed,
     },
 
     {
         .name       = "block-job-cancel",
         .args_type  = "device:B,force:b?",
-        .mhandler.cmd_new = qmp_marshal_input_block_job_cancel,
+        .mhandler.cmd_new = qmp_marshal_block_job_cancel,
     },
     {
         .name       = "block-job-pause",
         .args_type  = "device:B",
-        .mhandler.cmd_new = qmp_marshal_input_block_job_pause,
+        .mhandler.cmd_new = qmp_marshal_block_job_pause,
     },
     {
         .name       = "block-job-resume",
         .args_type  = "device:B",
-        .mhandler.cmd_new = qmp_marshal_input_block_job_resume,
+        .mhandler.cmd_new = qmp_marshal_block_job_resume,
     },
     {
         .name       = "block-job-complete",
         .args_type  = "device:B",
-        .mhandler.cmd_new = qmp_marshal_input_block_job_complete,
+        .mhandler.cmd_new = qmp_marshal_block_job_complete,
     },
     {
         .name       = "transaction",
         .args_type  = "actions:q",
-        .mhandler.cmd_new = qmp_marshal_input_transaction,
+        .mhandler.cmd_new = qmp_marshal_transaction,
     },
 
 SQMP
@@ -1310,7 +1310,7 @@ EQMP
     {
         .name       = "block-dirty-bitmap-add",
         .args_type  = "node:B,name:s,granularity:i?",
-        .mhandler.cmd_new = qmp_marshal_input_block_dirty_bitmap_add,
+        .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add,
     },
 
 SQMP
@@ -1338,7 +1338,7 @@ EQMP
     {
         .name       = "block-dirty-bitmap-remove",
         .args_type  = "node:B,name:s",
-        .mhandler.cmd_new = qmp_marshal_input_block_dirty_bitmap_remove,
+        .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_remove,
     },
 
 SQMP
@@ -1366,7 +1366,7 @@ EQMP
     {
         .name       = "block-dirty-bitmap-clear",
         .args_type  = "node:B,name:s",
-        .mhandler.cmd_new = qmp_marshal_input_block_dirty_bitmap_clear,
+        .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_clear,
     },
 
 SQMP
@@ -1395,7 +1395,7 @@ EQMP
     {
         .name       = "blockdev-snapshot-sync",
         .args_type  = "device:s?,node-name:s?,snapshot-file:s,snapshot-node-name:s?,format:s?,mode:s?",
-        .mhandler.cmd_new = qmp_marshal_input_blockdev_snapshot_sync,
+        .mhandler.cmd_new = qmp_marshal_blockdev_snapshot_sync,
     },
 
 SQMP
@@ -1431,7 +1431,7 @@ EQMP
     {
         .name       = "blockdev-snapshot-internal-sync",
         .args_type  = "device:B,name:s",
-        .mhandler.cmd_new = qmp_marshal_input_blockdev_snapshot_internal_sync,
+        .mhandler.cmd_new = qmp_marshal_blockdev_snapshot_internal_sync,
     },
 
 SQMP
@@ -1461,7 +1461,7 @@ EQMP
         .name       = "blockdev-snapshot-delete-internal-sync",
         .args_type  = "device:B,id:s?,name:s?",
         .mhandler.cmd_new =
-                      qmp_marshal_input_blockdev_snapshot_delete_internal_sync,
+                      qmp_marshal_blockdev_snapshot_delete_internal_sync,
     },
 
 SQMP
@@ -1504,7 +1504,7 @@ EQMP
                       "node-name:s?,replaces:s?,"
                       "on-source-error:s?,on-target-error:s?,"
                       "granularity:i?,buf-size:i?",
-        .mhandler.cmd_new = qmp_marshal_input_drive_mirror,
+        .mhandler.cmd_new = qmp_marshal_drive_mirror,
     },
 
 SQMP
@@ -1562,7 +1562,7 @@ EQMP
     {
         .name       = "change-backing-file",
         .args_type  = "device:s,image-node-name:s,backing-file:s",
-        .mhandler.cmd_new = qmp_marshal_input_change_backing_file,
+        .mhandler.cmd_new = qmp_marshal_change_backing_file,
     },
 
 SQMP
@@ -1601,7 +1601,7 @@ EQMP
     {
         .name       = "balloon",
         .args_type  = "value:M",
-        .mhandler.cmd_new = qmp_marshal_input_balloon,
+        .mhandler.cmd_new = qmp_marshal_balloon,
     },
 
 SQMP
@@ -1624,7 +1624,7 @@ EQMP
     {
         .name       = "set_link",
         .args_type  = "name:s,up:b",
-        .mhandler.cmd_new = qmp_marshal_input_set_link,
+        .mhandler.cmd_new = qmp_marshal_set_link,
     },
 
 SQMP
@@ -1650,7 +1650,7 @@ EQMP
         .args_type  = "fdname:s",
         .params     = "getfd name",
         .help       = "receive a file descriptor via SCM rights and assign it a name",
-        .mhandler.cmd_new = qmp_marshal_input_getfd,
+        .mhandler.cmd_new = qmp_marshal_getfd,
     },
 
 SQMP
@@ -1683,7 +1683,7 @@ EQMP
         .args_type  = "fdname:s",
         .params     = "closefd name",
         .help       = "close a file descriptor previously passed via SCM rights",
-        .mhandler.cmd_new = qmp_marshal_input_closefd,
+        .mhandler.cmd_new = qmp_marshal_closefd,
     },
 
 SQMP
@@ -1708,7 +1708,7 @@ EQMP
         .args_type  = "fdset-id:i?,opaque:s?",
         .params     = "add-fd fdset-id opaque",
         .help       = "Add a file descriptor, that was passed via SCM rights, to an fd set",
-        .mhandler.cmd_new = qmp_marshal_input_add_fd,
+        .mhandler.cmd_new = qmp_marshal_add_fd,
     },
 
 SQMP
@@ -1747,7 +1747,7 @@ EQMP
         .args_type  = "fdset-id:i,fd:i?",
         .params     = "remove-fd fdset-id fd",
         .help       = "Remove a file descriptor from an fd set",
-        .mhandler.cmd_new = qmp_marshal_input_remove_fd,
+        .mhandler.cmd_new = qmp_marshal_remove_fd,
     },
 
 SQMP
@@ -1779,7 +1779,7 @@ EQMP
         .name       = "query-fdsets",
         .args_type  = "",
         .help       = "Return information describing all fd sets",
-        .mhandler.cmd_new = qmp_marshal_input_query_fdsets,
+        .mhandler.cmd_new = qmp_marshal_query_fdsets,
     },
 
 SQMP
@@ -1828,7 +1828,7 @@ EQMP
     {
         .name       = "block_passwd",
         .args_type  = "device:s?,node-name:s?,password:s",
-        .mhandler.cmd_new = qmp_marshal_input_block_passwd,
+        .mhandler.cmd_new = qmp_marshal_block_passwd,
     },
 
 SQMP
@@ -1854,7 +1854,7 @@ EQMP
     {
         .name       = "block_set_io_throttle",
         .args_type  = "device:B,bps:l,bps_rd:l,bps_wr:l,iops:l,iops_rd:l,iops_wr:l,bps_max:l?,bps_rd_max:l?,bps_wr_max:l?,iops_max:l?,iops_rd_max:l?,iops_wr_max:l?,iops_size:l?,group:s?",
-        .mhandler.cmd_new = qmp_marshal_input_block_set_io_throttle,
+        .mhandler.cmd_new = qmp_marshal_block_set_io_throttle,
     },
 
 SQMP
@@ -1904,7 +1904,7 @@ EQMP
     {
         .name       = "set_password",
         .args_type  = "protocol:s,password:s,connected:s?",
-        .mhandler.cmd_new = qmp_marshal_input_set_password,
+        .mhandler.cmd_new = qmp_marshal_set_password,
     },
 
 SQMP
@@ -1930,7 +1930,7 @@ EQMP
     {
         .name       = "expire_password",
         .args_type  = "protocol:s,time:s",
-        .mhandler.cmd_new = qmp_marshal_input_expire_password,
+        .mhandler.cmd_new = qmp_marshal_expire_password,
     },
 
 SQMP
@@ -1955,7 +1955,7 @@ EQMP
     {
         .name       = "add_client",
         .args_type  = "protocol:s,fdname:s,skipauth:b?,tls:b?",
-        .mhandler.cmd_new = qmp_marshal_input_add_client,
+        .mhandler.cmd_new = qmp_marshal_add_client,
     },
 
 SQMP
@@ -2006,7 +2006,7 @@ EQMP
     {
         .name       = "human-monitor-command",
         .args_type  = "command-line:s,cpu-index:i?",
-        .mhandler.cmd_new = qmp_marshal_input_human_monitor_command,
+        .mhandler.cmd_new = qmp_marshal_human_monitor_command,
     },
 
 SQMP
@@ -2085,7 +2085,7 @@ EQMP
     {
         .name       = "query-version",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_version,
+        .mhandler.cmd_new = qmp_marshal_query_version,
     },
 
 SQMP
@@ -2122,7 +2122,7 @@ EQMP
     {
         .name       = "query-commands",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_commands,
+        .mhandler.cmd_new = qmp_marshal_query_commands,
     },
 
 SQMP
@@ -2159,7 +2159,7 @@ EQMP
     {
         .name       = "query-events",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_events,
+        .mhandler.cmd_new = qmp_marshal_query_events,
     },
 
 SQMP
@@ -2204,7 +2204,7 @@ EQMP
     {
         .name       = "query-chardev",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_chardev,
+        .mhandler.cmd_new = qmp_marshal_query_chardev,
     },
 
 SQMP
@@ -2245,7 +2245,7 @@ EQMP
     {
         .name       = "query-chardev-backends",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_chardev_backends,
+        .mhandler.cmd_new = qmp_marshal_query_chardev_backends,
     },
 
 SQMP
@@ -2429,7 +2429,7 @@ EQMP
     {
         .name       = "query-block",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_block,
+        .mhandler.cmd_new = qmp_marshal_query_block,
     },
 
 SQMP
@@ -2556,7 +2556,7 @@ EQMP
     {
         .name       = "query-blockstats",
         .args_type  = "query-nodes:b?",
-        .mhandler.cmd_new = qmp_marshal_input_query_blockstats,
+        .mhandler.cmd_new = qmp_marshal_query_blockstats,
     },
 
 SQMP
@@ -2607,7 +2607,7 @@ EQMP
     {
         .name       = "query-cpus",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_cpus,
+        .mhandler.cmd_new = qmp_marshal_query_cpus,
     },
 
 SQMP
@@ -2646,7 +2646,7 @@ EQMP
     {
         .name       = "query-iothreads",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_iothreads,
+        .mhandler.cmd_new = qmp_marshal_query_iothreads,
     },
 
 SQMP
@@ -2863,7 +2863,7 @@ EQMP
     {
         .name       = "query-pci",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_pci,
+        .mhandler.cmd_new = qmp_marshal_query_pci,
     },
 
 SQMP
@@ -2887,7 +2887,7 @@ EQMP
     {
         .name       = "query-kvm",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_kvm,
+        .mhandler.cmd_new = qmp_marshal_query_kvm,
     },
 
 SQMP
@@ -2927,7 +2927,7 @@ EQMP
     {
         .name       = "query-status",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_status,
+        .mhandler.cmd_new = qmp_marshal_query_status,
     },
 
 SQMP
@@ -2971,7 +2971,7 @@ EQMP
     {
         .name       = "query-mice",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_mice,
+        .mhandler.cmd_new = qmp_marshal_query_mice,
     },
 
 SQMP
@@ -3034,12 +3034,12 @@ EQMP
     {
         .name       = "query-vnc",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_vnc,
+        .mhandler.cmd_new = qmp_marshal_query_vnc,
     },
     {
         .name       = "query-vnc-servers",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_vnc_servers,
+        .mhandler.cmd_new = qmp_marshal_query_vnc_servers,
     },
 
 SQMP
@@ -3116,7 +3116,7 @@ EQMP
     {
         .name       = "query-spice",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_spice,
+        .mhandler.cmd_new = qmp_marshal_query_spice,
     },
 #endif
 
@@ -3140,7 +3140,7 @@ EQMP
     {
         .name       = "query-name",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_name,
+        .mhandler.cmd_new = qmp_marshal_query_name,
     },
 
 SQMP
@@ -3163,7 +3163,7 @@ EQMP
     {
         .name       = "query-uuid",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_uuid,
+        .mhandler.cmd_new = qmp_marshal_query_uuid,
     },
 
 SQMP
@@ -3212,7 +3212,7 @@ EQMP
     {
         .name       = "query-command-line-options",
         .args_type  = "option:s?",
-        .mhandler.cmd_new = qmp_marshal_input_query_command_line_options,
+        .mhandler.cmd_new = qmp_marshal_query_command_line_options,
     },
 
 SQMP
@@ -3390,7 +3390,7 @@ EQMP
     {
         .name       = "query-migrate",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_migrate,
+        .mhandler.cmd_new = qmp_marshal_query_migrate,
     },
 
 SQMP
@@ -3417,7 +3417,7 @@ EQMP
         .name       = "migrate-set-capabilities",
         .args_type  = "capabilities:q",
         .params     = "capability:s,state:b",
-	.mhandler.cmd_new = qmp_marshal_input_migrate_set_capabilities,
+	.mhandler.cmd_new = qmp_marshal_migrate_set_capabilities,
     },
 SQMP
 query-migrate-capabilities
@@ -3443,7 +3443,7 @@ EQMP
     {
         .name       = "query-migrate-capabilities",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_migrate_capabilities,
+        .mhandler.cmd_new = qmp_marshal_query_migrate_capabilities,
     },
 
 SQMP
@@ -3469,7 +3469,7 @@ EQMP
         .name       = "migrate-set-parameters",
         .args_type  =
             "compress-level:i?,compress-threads:i?,decompress-threads:i?",
-	.mhandler.cmd_new = qmp_marshal_input_migrate_set_parameters,
+	.mhandler.cmd_new = qmp_marshal_migrate_set_parameters,
     },
 SQMP
 query-migrate-parameters
@@ -3500,7 +3500,7 @@ EQMP
     {
         .name       = "query-migrate-parameters",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_migrate_parameters,
+        .mhandler.cmd_new = qmp_marshal_query_migrate_parameters,
     },
 
 SQMP
@@ -3528,19 +3528,19 @@ EQMP
     {
         .name       = "query-balloon",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_balloon,
+        .mhandler.cmd_new = qmp_marshal_query_balloon,
     },
 
     {
         .name       = "query-block-jobs",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_block_jobs,
+        .mhandler.cmd_new = qmp_marshal_query_block_jobs,
     },
 
     {
         .name       = "qom-list",
         .args_type  = "path:s",
-        .mhandler.cmd_new = qmp_marshal_input_qom_list,
+        .mhandler.cmd_new = qmp_marshal_qom_list,
     },
 
     {
@@ -3558,58 +3558,58 @@ EQMP
     {
         .name       = "nbd-server-start",
         .args_type  = "addr:q",
-        .mhandler.cmd_new = qmp_marshal_input_nbd_server_start,
+        .mhandler.cmd_new = qmp_marshal_nbd_server_start,
     },
     {
         .name       = "nbd-server-add",
         .args_type  = "device:B,writable:b?",
-        .mhandler.cmd_new = qmp_marshal_input_nbd_server_add,
+        .mhandler.cmd_new = qmp_marshal_nbd_server_add,
     },
     {
         .name       = "nbd-server-stop",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_nbd_server_stop,
+        .mhandler.cmd_new = qmp_marshal_nbd_server_stop,
     },
 
     {
         .name       = "change-vnc-password",
         .args_type  = "password:s",
-        .mhandler.cmd_new = qmp_marshal_input_change_vnc_password,
+        .mhandler.cmd_new = qmp_marshal_change_vnc_password,
     },
     {
         .name       = "qom-list-types",
         .args_type  = "implements:s?,abstract:b?",
-        .mhandler.cmd_new = qmp_marshal_input_qom_list_types,
+        .mhandler.cmd_new = qmp_marshal_qom_list_types,
     },
 
     {
         .name       = "device-list-properties",
         .args_type  = "typename:s",
-        .mhandler.cmd_new = qmp_marshal_input_device_list_properties,
+        .mhandler.cmd_new = qmp_marshal_device_list_properties,
     },
 
     {
         .name       = "query-machines",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_machines,
+        .mhandler.cmd_new = qmp_marshal_query_machines,
     },
 
     {
         .name       = "query-cpu-definitions",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_cpu_definitions,
+        .mhandler.cmd_new = qmp_marshal_query_cpu_definitions,
     },
 
     {
         .name       = "query-target",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_target,
+        .mhandler.cmd_new = qmp_marshal_query_target,
     },
 
     {
         .name       = "query-tpm",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_tpm,
+        .mhandler.cmd_new = qmp_marshal_query_tpm,
     },
 
 SQMP
@@ -3643,7 +3643,7 @@ EQMP
     {
         .name       = "query-tpm-models",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_tpm_models,
+        .mhandler.cmd_new = qmp_marshal_query_tpm_models,
     },
 
 SQMP
@@ -3664,7 +3664,7 @@ EQMP
     {
         .name       = "query-tpm-types",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_tpm_types,
+        .mhandler.cmd_new = qmp_marshal_query_tpm_types,
     },
 
 SQMP
@@ -3685,7 +3685,7 @@ EQMP
     {
         .name       = "chardev-add",
         .args_type  = "id:s,backend:q",
-        .mhandler.cmd_new = qmp_marshal_input_chardev_add,
+        .mhandler.cmd_new = qmp_marshal_chardev_add,
     },
 
 SQMP
@@ -3722,7 +3722,7 @@ EQMP
     {
         .name       = "chardev-remove",
         .args_type  = "id:s",
-        .mhandler.cmd_new = qmp_marshal_input_chardev_remove,
+        .mhandler.cmd_new = qmp_marshal_chardev_remove,
     },
 
 
@@ -3745,7 +3745,7 @@ EQMP
     {
         .name       = "query-rx-filter",
         .args_type  = "name:s?",
-        .mhandler.cmd_new = qmp_marshal_input_query_rx_filter,
+        .mhandler.cmd_new = qmp_marshal_query_rx_filter,
     },
 
 SQMP
@@ -3811,7 +3811,7 @@ EQMP
     {
         .name       = "blockdev-add",
         .args_type  = "options:q",
-        .mhandler.cmd_new = qmp_marshal_input_blockdev_add,
+        .mhandler.cmd_new = qmp_marshal_blockdev_add,
     },
 
 SQMP
@@ -3870,7 +3870,7 @@ EQMP
     {
         .name       = "query-named-block-nodes",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_named_block_nodes,
+        .mhandler.cmd_new = qmp_marshal_query_named_block_nodes,
     },
 
 SQMP
@@ -3932,7 +3932,7 @@ EQMP
     {
         .name       = "query-memdev",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_memdev,
+        .mhandler.cmd_new = qmp_marshal_query_memdev,
     },
 
 SQMP
@@ -3970,7 +3970,7 @@ EQMP
     {
         .name       = "query-memory-devices",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_memory_devices,
+        .mhandler.cmd_new = qmp_marshal_query_memory_devices,
     },
 
 SQMP
@@ -3997,7 +3997,7 @@ EQMP
     {
         .name       = "query-acpi-ospm-status",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_acpi_ospm_status,
+        .mhandler.cmd_new = qmp_marshal_query_acpi_ospm_status,
     },
 
 SQMP
@@ -4020,7 +4020,7 @@ EQMP
     {
         .name       = "rtc-reset-reinjection",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_rtc_reset_reinjection,
+        .mhandler.cmd_new = qmp_marshal_rtc_reset_reinjection,
     },
 #endif
 
@@ -4041,7 +4041,7 @@ EQMP
     {
         .name       = "trace-event-get-state",
         .args_type  = "name:s",
-        .mhandler.cmd_new = qmp_marshal_input_trace_event_get_state,
+        .mhandler.cmd_new = qmp_marshal_trace_event_get_state,
     },
 
 SQMP
@@ -4059,7 +4059,7 @@ EQMP
     {
         .name       = "trace-event-set-state",
         .args_type  = "name:s,enable:b,ignore-unavailable:b?",
-        .mhandler.cmd_new = qmp_marshal_input_trace_event_set_state,
+        .mhandler.cmd_new = qmp_marshal_trace_event_set_state,
     },
 
 SQMP
@@ -4077,7 +4077,7 @@ EQMP
     {
         .name       = "x-input-send-event",
         .args_type  = "console:i?,events:q",
-        .mhandler.cmd_new = qmp_marshal_input_x_input_send_event,
+        .mhandler.cmd_new = qmp_marshal_x_input_send_event,
     },
 
 SQMP
@@ -4142,7 +4142,7 @@ EQMP
     {
         .name       = "block-set-write-threshold",
         .args_type  = "node-name:s,write-threshold:l",
-        .mhandler.cmd_new = qmp_marshal_input_block_set_write_threshold,
+        .mhandler.cmd_new = qmp_marshal_block_set_write_threshold,
     },
 
 SQMP
@@ -4170,7 +4170,7 @@ EQMP
     {
         .name       = "query-rocker",
         .args_type  = "name:s",
-        .mhandler.cmd_new = qmp_marshal_input_query_rocker,
+        .mhandler.cmd_new = qmp_marshal_query_rocker,
     },
 
 SQMP
@@ -4191,7 +4191,7 @@ EQMP
     {
         .name       = "query-rocker-ports",
         .args_type  = "name:s",
-        .mhandler.cmd_new = qmp_marshal_input_query_rocker_ports,
+        .mhandler.cmd_new = qmp_marshal_query_rocker_ports,
     },
 
 SQMP
@@ -4216,7 +4216,7 @@ EQMP
     {
         .name       = "query-rocker-of-dpa-flows",
         .args_type  = "name:s,tbl-id:i?",
-        .mhandler.cmd_new = qmp_marshal_input_query_rocker_of_dpa_flows,
+        .mhandler.cmd_new = qmp_marshal_query_rocker_of_dpa_flows,
     },
 
 SQMP
@@ -4245,7 +4245,7 @@ EQMP
     {
         .name       = "query-rocker-of-dpa-groups",
         .args_type  = "name:s,type:i?",
-        .mhandler.cmd_new = qmp_marshal_input_query_rocker_of_dpa_groups,
+        .mhandler.cmd_new = qmp_marshal_query_rocker_of_dpa_groups,
     },
 
 SQMP
diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 728f15b..d57f8d4 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -202,7 +202,7 @@ out:
                  c_name=rets.c_name())
 
 def gen_marshal_proto(name):
-    ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
+    ret = 'void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
     if not middle_mode:
         ret = 'static ' + ret
     return ret
@@ -246,7 +246,7 @@ def gen_register_command(name, success_response):
         options = 'QCO_NO_SUCCESS_RESP'
 
     ret = mcgen('''
-qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
+qmp_register_command("%(name)s", qmp_marshal_%(c_name)s, %(opts)s);
 ''',
                 name=name, c_name=c_name(name),
                 opts=options)
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 37/47] qapi: De-duplicate parameter list generation

[Markus Armbruster]

Generated qapi-event.[ch] lose line breaks.  No change otherwise.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 11 ++---------
 scripts/qapi-event.py    | 18 +++---------------
 scripts/qapi.py          | 16 ++++++++++++++++
 3 files changed, 21 insertions(+), 24 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index d57f8d4..2dae425 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -16,19 +16,12 @@ from qapi import *
 import re
 
 def gen_command_decl(name, args, rets):
-    argstr = ''
-    if args:
-        for memb in args.members:
-            if memb.optional:
-                argstr += 'bool has_%s, ' % c_name(memb.name)
-            argstr += '%s %s, ' % (memb.type.c_type(is_param=True),
-                                   c_name(memb.name))
     return mcgen('''
-%(c_type)s qmp_%(c_name)s(%(args)sError **errp);
+%(c_type)s qmp_%(c_name)s(%(params)s);
 ''',
                  c_type=(rets and rets.c_type()) or 'void',
                  c_name=c_name(name),
-                 args=argstr)
+                 params=gen_params(args, 'Error **errp'))
 
 def gen_err_check(err):
     if not err:
diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 03bb1ec..184a81f 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -14,21 +14,9 @@
 from qapi import *
 
 def gen_event_send_proto(name, data):
-    api_name = "void qapi_event_send_%s(" % c_name(name).lower()
-    l = len(api_name)
-
-    if data:
-        for m in data.members:
-            if m.optional:
-                api_name += "bool has_%s,\n" % c_name(m.name)
-                api_name += "".ljust(l)
-
-            api_name += "%s %s,\n" % (m.type.c_type(is_param=True),
-                                      c_name(m.name))
-            api_name += "".ljust(l)
-
-    api_name += "Error **errp)"
-    return api_name
+    return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
+        'c_name': c_name(name.lower()),
+        'param': gen_params(data, 'Error **errp')}
 
 def gen_event_send_decl(name, data):
     return mcgen('''
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 4d47214..c6a5ddc 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1384,6 +1384,22 @@ extern const char *const %(c_name)s_lookup[];
                  c_name=c_name(name))
     return ret
 
+def gen_params(args, extra):
+    if not args:
+        return extra
+    assert not args.variants
+    ret = ""
+    sep = ""
+    for memb in args.members:
+        ret += sep
+        sep = ", "
+        if memb.optional:
+            ret += "bool has_%s, " % c_name(memb.name)
+        ret += "%s %s" % (memb.type.c_type(is_param=True), c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret
+
 #
 # Common command line parsing
 #
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 38/47] qapi-commands: De-duplicate output marshaling functions

[Markus Armbruster]

gen_marshal_output() uses its parameter name only for name of the
generated function.  Name it after the type being marshaled instead of
its caller, and drop duplicates.

Saves 7 copies of qmp_marshal_output_int() in qemu-ga, and one copy of
qmp_marshal_output_str() in qemu-system-*.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-commands.py | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 2dae425..fcbb7d0 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -59,7 +59,7 @@ def gen_call(name, args, rets):
 
 qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
 ''',
-                     c_name=c_name(name))
+                     c_name=rets.c_name())
     pop_indent()
     return ret
 
@@ -165,10 +165,10 @@ qapi_dealloc_visitor_cleanup(md);
     pop_indent()
     return ret
 
-def gen_marshal_output(name, rets):
+def gen_marshal_output(rets):
     return mcgen('''
 
-static void qmp_marshal_output_%(c_cmd_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
     QmpOutputVisitor *mo = qmp_output_visitor_new();
@@ -191,8 +191,7 @@ out:
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                 c_type=rets.c_type(), c_cmd_name=c_name(name),
-                 c_name=rets.c_name())
+                 c_type=rets.c_type(), c_name=rets.c_name())
 
 def gen_marshal_proto(name):
     ret = 'void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
@@ -265,10 +264,12 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self.decl = None
         self.defn = None
         self.regy = None
+        self.visited_rets = None
     def visit_begin(self):
         self.decl = ''
         self.defn = ''
         self.regy = ''
+        self.visited_rets = set()
     def visit_end(self):
         if not middle_mode:
             self.defn += gen_registry(self.regy)
@@ -277,8 +278,9 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         if not gen:
             return
         self.decl += gen_command_decl(name, args, rets)
-        if rets:
-            self.defn += gen_marshal_output(name, rets)
+        if rets and rets not in self.visited_rets:
+            self.visited_rets.add(rets)
+            self.defn += gen_marshal_output(rets)
         if middle_mode:
             self.decl += gen_marshal_decl(name)
         self.defn += gen_marshal(name, args, rets)
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 39/47] qapi: Improve built-in type documentation

[Markus Armbruster]

Clarify how they map to JSON.  Add how they map to C.  Fix the
reference to StringInputVisitor.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 docs/qapi-code-gen.txt | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index 83af531..cb0fe75 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -140,17 +140,24 @@ must have a value that forms a struct name.
 
 === Built-in Types ===
 
-The following types are built-in to the parser:
-  'str' - arbitrary UTF-8 string
-  'int' - 64-bit signed integer (although the C code may place further
-          restrictions on acceptable range)
-  'number' - floating point number
-  'bool' - JSON value of true or false
-  'int8', 'int16', 'int32', 'int64' - like 'int', but enforce maximum
-                                      bit size
-  'uint8', 'uint16', 'uint32', 'uint64' - unsigned counterparts
-  'size' - like 'uint64', but allows scaled suffix from command line
-           visitor
+The following types are predefined, and map to C as follows:
+
+  Schema    C          JSON
+  str       char *     any JSON string, UTF-8
+  number    double     any JSON number
+  int       int64_t    a JSON number without fractional part
+                       that fits into the C integer type
+  int8      int8_t     likewise
+  int16     int16_t    likewise
+  int32     int32_t    likewise
+  int64     int64_t    likewise
+  uint8     uint8_t    likewise
+  uint16    uint16_t   likewise
+  uint32    uint32_t   likewise
+  uint64    uint64_t   likewise
+  size      uint64_t   like uint64_t, except StringInputVisitor
+                       accepts size suffixes
+  bool      bool       JSON true or false
 
 
 === Includes ===
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 40/47] qapi: Introduce a first class 'any' type

[Markus Armbruster]

It's first class, because unlike '**', it actually works, i.e. doesn't
require 'gen': false.

'**' will go away next.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 docs/qapi-code-gen.txt            |  1 +
 include/qapi/visitor-impl.h       |  2 ++
 include/qapi/visitor.h            |  1 +
 qapi/qapi-dealloc-visitor.c       |  9 +++++++
 qapi/qapi-visit-core.c            |  6 +++++
 qapi/qmp-input-visitor.c          | 11 ++++++++
 qapi/qmp-output-visitor.c         |  9 +++++++
 scripts/qapi-types.py             |  1 +
 scripts/qapi.py                   |  9 ++++---
 tests/qapi-schema/type-bypass.out |  4 +--
 tests/test-qmp-input-visitor.c    | 45 +++++++++++++++++++++++++++++++++
 tests/test-qmp-output-visitor.c   | 53 +++++++++++++++++++++++++++++++++++++++
 12 files changed, 146 insertions(+), 5 deletions(-)

diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index cb0fe75..bf9e854 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -158,6 +158,7 @@ The following types are predefined, and map to C as follows:
   size      uint64_t   like uint64_t, except StringInputVisitor
                        accepts size suffixes
   bool      bool       JSON true or false
+  any       QObject *  any JSON value
 
 
 === Includes ===
diff --git a/include/qapi/visitor-impl.h b/include/qapi/visitor-impl.h
index f4a2f74..8c0ba57 100644
--- a/include/qapi/visitor-impl.h
+++ b/include/qapi/visitor-impl.h
@@ -40,6 +40,8 @@ struct Visitor
     void (*type_str)(Visitor *v, char **obj, const char *name, Error **errp);
     void (*type_number)(Visitor *v, double *obj, const char *name,
                         Error **errp);
+    void (*type_any)(Visitor *v, QObject **obj, const char *name,
+                     Error **errp);
 
     /* May be NULL */
     void (*optional)(Visitor *v, bool *present, const char *name,
diff --git a/include/qapi/visitor.h b/include/qapi/visitor.h
index 00ba104..cfc19a6 100644
--- a/include/qapi/visitor.h
+++ b/include/qapi/visitor.h
@@ -58,6 +58,7 @@ void visit_type_size(Visitor *v, uint64_t *obj, const char *name, Error **errp);
 void visit_type_bool(Visitor *v, bool *obj, const char *name, Error **errp);
 void visit_type_str(Visitor *v, char **obj, const char *name, Error **errp);
 void visit_type_number(Visitor *v, double *obj, const char *name, Error **errp);
+void visit_type_any(Visitor *v, QObject **obj, const char *name, Error **errp);
 bool visit_start_union(Visitor *v, bool data_present, Error **errp);
 void visit_end_union(Visitor *v, bool data_present, Error **errp);
 
diff --git a/qapi/qapi-dealloc-visitor.c b/qapi/qapi-dealloc-visitor.c
index d7f92c5..737deab 100644
--- a/qapi/qapi-dealloc-visitor.c
+++ b/qapi/qapi-dealloc-visitor.c
@@ -151,6 +151,14 @@ static void qapi_dealloc_type_number(Visitor *v, double *obj, const char *name,
 {
 }
 
+static void qapi_dealloc_type_anything(Visitor *v, QObject **obj,
+                                       const char *name, Error **errp)
+{
+    if (obj) {
+        qobject_decref(*obj);
+    }
+}
+
 static void qapi_dealloc_type_size(Visitor *v, uint64_t *obj, const char *name,
                                    Error **errp)
 {
@@ -216,6 +224,7 @@ QapiDeallocVisitor *qapi_dealloc_visitor_new(void)
     v->visitor.type_bool = qapi_dealloc_type_bool;
     v->visitor.type_str = qapi_dealloc_type_str;
     v->visitor.type_number = qapi_dealloc_type_number;
+    v->visitor.type_any = qapi_dealloc_type_anything;
     v->visitor.type_size = qapi_dealloc_type_size;
     v->visitor.start_union = qapi_dealloc_start_union;
 
diff --git a/qapi/qapi-visit-core.c b/qapi/qapi-visit-core.c
index 5a7c900..230d4b2 100644
--- a/qapi/qapi-visit-core.c
+++ b/qapi/qapi-visit-core.c
@@ -260,6 +260,12 @@ void visit_type_number(Visitor *v, double *obj, const char *name, Error **errp)
     v->type_number(v, obj, name, errp);
 }
 
+void visit_type_any(Visitor *v, QObject **obj, const char *name,
+                         Error **errp)
+{
+    v->type_any(v, obj, name, errp);
+}
+
 void output_type_enum(Visitor *v, int *obj, const char * const strings[],
                       const char *kind, const char *name,
                       Error **errp)
diff --git a/qapi/qmp-input-visitor.c b/qapi/qmp-input-visitor.c
index e97b8a4..5dd9ed5 100644
--- a/qapi/qmp-input-visitor.c
+++ b/qapi/qmp-input-visitor.c
@@ -286,6 +286,16 @@ static void qmp_input_type_number(Visitor *v, double *obj, const char *name,
     }
 }
 
+static void qmp_input_type_any(Visitor *v, QObject **obj, const char *name,
+                               Error **errp)
+{
+    QmpInputVisitor *qiv = to_qiv(v);
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+
+    qobject_incref(qobj);
+    *obj = qobj;
+}
+
 static void qmp_input_optional(Visitor *v, bool *present, const char *name,
                                Error **errp)
 {
@@ -329,6 +339,7 @@ QmpInputVisitor *qmp_input_visitor_new(QObject *obj)
     v->visitor.type_bool = qmp_input_type_bool;
     v->visitor.type_str = qmp_input_type_str;
     v->visitor.type_number = qmp_input_type_number;
+    v->visitor.type_any = qmp_input_type_any;
     v->visitor.optional = qmp_input_optional;
     v->visitor.get_next_type = qmp_input_get_next_type;
 
diff --git a/qapi/qmp-output-visitor.c b/qapi/qmp-output-visitor.c
index efc19d5..1e36a7a 100644
--- a/qapi/qmp-output-visitor.c
+++ b/qapi/qmp-output-visitor.c
@@ -186,6 +186,14 @@ static void qmp_output_type_number(Visitor *v, double *obj, const char *name,
     qmp_output_add(qov, name, qfloat_from_double(*obj));
 }
 
+static void qmp_output_type_any(Visitor *v, QObject **obj, const char *name,
+                                Error **errp)
+{
+    QmpOutputVisitor *qov = to_qov(v);
+    qobject_incref(*obj);
+    qmp_output_add_obj(qov, name, *obj);
+}
+
 QObject *qmp_output_get_qobject(QmpOutputVisitor *qov)
 {
     QObject *obj = qmp_output_first(qov);
@@ -233,6 +241,7 @@ QmpOutputVisitor *qmp_output_visitor_new(void)
     v->visitor.type_bool = qmp_output_type_bool;
     v->visitor.type_str = qmp_output_type_str;
     v->visitor.type_number = qmp_output_type_number;
+    v->visitor.type_any = qmp_output_type_any;
 
     QTAILQ_INIT(&v->stack);
 
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index b6990a5..7c403c0 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -289,6 +289,7 @@ fdef.write(mcgen('''
 fdecl.write(mcgen('''
 #include <stdbool.h>
 #include <stdint.h>
+#include "qapi/qmp/qobject.h"
 '''))
 
 schema = QAPISchema(input_file)
diff --git a/scripts/qapi.py b/scripts/qapi.py
index c6a5ddc..2ff087f 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -33,6 +33,7 @@ builtin_types = {
     'uint32':   'QTYPE_QINT',
     'uint64':   'QTYPE_QINT',
     'size':     'QTYPE_QINT',
+    'any':      None,           # any qtype_code possible, actually
 }
 
 # Whitelist of commands allowed to return a non-dictionary
@@ -1031,8 +1032,7 @@ class QAPISchema(object):
 
     def _def_builtin_type(self, name, json_type, c_type, c_null):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
-        if name != '**':
-            self._make_array_type(name) # TODO really needed?
+        self._make_array_type(name) # TODO really needed?
 
     def _def_predefineds(self):
         for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
@@ -1048,8 +1048,9 @@ class QAPISchema(object):
                   ('uint64', 'int',     'uint64_t', '0'),
                   ('size',   'int',     'uint64_t', '0'),
                   ('bool',   'boolean', 'bool',     'false'),
-                  ('**',     'value',   None,       None)]:
+                  ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
             self._def_builtin_type(*t)
+        self.entity_dict['**'] = self.lookup_type('any') # TODO drop this alias
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
@@ -1190,6 +1191,8 @@ class QAPISchema(object):
     def visit(self, visitor):
         visitor.visit_begin()
         for name in sorted(self.entity_dict.keys()):
+            if self.entity_dict[name].name != name:
+                continue        # ignore alias TODO drop alias and remove
             self.entity_dict[name].visit(visitor)
         visitor.visit_end()
 
diff --git a/tests/qapi-schema/type-bypass.out b/tests/qapi-schema/type-bypass.out
index b147864..2dc0c09 100644
--- a/tests/qapi-schema/type-bypass.out
+++ b/tests/qapi-schema/type-bypass.out
@@ -1,4 +1,4 @@
 object :obj-unsafe-args
-    member arg: ** optional=False
-command unsafe :obj-unsafe-args -> **
+    member arg: any optional=False
+command unsafe :obj-unsafe-args -> any
    gen=False success_response=True
diff --git a/tests/test-qmp-input-visitor.c b/tests/test-qmp-input-visitor.c
index 56da3c6..9c4bfdf 100644
--- a/tests/test-qmp-input-visitor.c
+++ b/tests/test-qmp-input-visitor.c
@@ -298,6 +298,49 @@ static void test_visitor_in_list(TestInputVisitorData *data,
     qapi_free_UserDefOneList(head);
 }
 
+static void test_visitor_in_any(TestInputVisitorData *data,
+                                const void *unused)
+{
+    QObject *res = NULL;
+    Error *err = NULL;
+    Visitor *v;
+    QInt *qint;
+    QBool *qbool;
+    QString *qstring;
+    QDict *qdict;
+    QObject *qobj;
+
+    v = visitor_input_test_init(data, "-42");
+    visit_type_any(v, &res, NULL, &err);
+    g_assert(!err);
+    qint = qobject_to_qint(res);
+    g_assert(qint);
+    g_assert_cmpint(qint_get_int(qint), ==, -42);
+    qobject_decref(res);
+
+    v = visitor_input_test_init(data, "{ 'integer': -42, 'boolean': true, 'string': 'foo' }");
+    visit_type_any(v, &res, NULL, &err);
+    g_assert(!err);
+    qdict = qobject_to_qdict(res);
+    g_assert(qdict);
+    qobj = qdict_get(qdict, "integer");
+    g_assert(qobj);
+    qint = qobject_to_qint(qobj);
+    g_assert(qint);
+    g_assert_cmpint(qint_get_int(qint), ==, -42);
+    qobj = qdict_get(qdict, "boolean");
+    g_assert(qobj);
+    qbool = qobject_to_qbool(qobj);
+    g_assert(qbool);
+    g_assert(qbool_get_bool(qbool) == true);
+    qobj = qdict_get(qdict, "string");
+    g_assert(qobj);
+    qstring = qobject_to_qstring(qobj);
+    g_assert(qstring);
+    g_assert_cmpstr(qstring_get_str(qstring), ==, "foo");
+    qobject_decref(res);
+}
+
 static void test_visitor_in_union_flat(TestInputVisitorData *data,
                                        const void *unused)
 {
@@ -667,6 +710,8 @@ int main(int argc, char **argv)
                            &in_visitor_data, test_visitor_in_struct_nested);
     input_visitor_test_add("/visitor/input/list",
                            &in_visitor_data, test_visitor_in_list);
+    input_visitor_test_add("/visitor/input/any",
+                           &in_visitor_data, test_visitor_in_any);
     input_visitor_test_add("/visitor/input/union-flat",
                            &in_visitor_data, test_visitor_in_union_flat);
     input_visitor_test_add("/visitor/input/alternate",
diff --git a/tests/test-qmp-output-visitor.c b/tests/test-qmp-output-visitor.c
index 338ada0..1a28dc2 100644
--- a/tests/test-qmp-output-visitor.c
+++ b/tests/test-qmp-output-visitor.c
@@ -428,6 +428,57 @@ static void test_visitor_out_list_qapi_free(TestOutputVisitorData *data,
     qapi_free_UserDefTwoList(head);
 }
 
+static void test_visitor_out_any(TestOutputVisitorData *data,
+                                 const void *unused)
+{
+    QObject *qobj;
+    Error *err = NULL;
+    QInt *qint;
+    QBool *qbool;
+    QString *qstring;
+    QDict *qdict;
+    QObject *obj;
+
+    qobj = QOBJECT(qint_from_int(-42));
+    visit_type_any(data->ov, &qobj, NULL, &err);
+    g_assert(!err);
+    obj = qmp_output_get_qobject(data->qov);
+    g_assert(obj != NULL);
+    g_assert(qobject_type(obj) == QTYPE_QINT);
+    g_assert_cmpint(qint_get_int(qobject_to_qint(obj)), ==, -42);
+    qobject_decref(obj);
+    qobject_decref(qobj);
+
+    qdict = qdict_new();
+    qdict_put(qdict, "integer", qint_from_int(-42));
+    qdict_put(qdict, "boolean", qbool_from_bool(true));
+    qdict_put(qdict, "string", qstring_from_str("foo"));
+    qobj = QOBJECT(qdict);
+    visit_type_any(data->ov, &qobj, NULL, &err);
+    g_assert(!err);
+    obj = qmp_output_get_qobject(data->qov);
+    g_assert(obj != NULL);
+    qdict = qobject_to_qdict(obj);
+    g_assert(qdict);
+    qobj = qdict_get(qdict, "integer");
+    g_assert(qobj);
+    qint = qobject_to_qint(qobj);
+    g_assert(qint);
+    g_assert_cmpint(qint_get_int(qint), ==, -42);
+    qobj = qdict_get(qdict, "boolean");
+    g_assert(qobj);
+    qbool = qobject_to_qbool(qobj);
+    g_assert(qbool);
+    g_assert(qbool_get_bool(qbool) == true);
+    qobj = qdict_get(qdict, "string");
+    g_assert(qobj);
+    qstring = qobject_to_qstring(qobj);
+    g_assert(qstring);
+    g_assert_cmpstr(qstring_get_str(qstring), ==, "foo");
+    qobject_decref(obj);
+    qobject_decref(qobj);
+}
+
 static void test_visitor_out_union_flat(TestOutputVisitorData *data,
                                         const void *unused)
 {
@@ -832,6 +883,8 @@ int main(int argc, char **argv)
                             &out_visitor_data, test_visitor_out_struct_errors);
     output_visitor_test_add("/visitor/output/list",
                             &out_visitor_data, test_visitor_out_list);
+    output_visitor_test_add("/visitor/output/any",
+                            &out_visitor_data, test_visitor_out_any);
     output_visitor_test_add("/visitor/output/list-qapi-free",
                             &out_visitor_data, test_visitor_out_list_qapi_free);
     output_visitor_test_add("/visitor/output/union-flat",
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 41/47] qom: Don't use 'gen': false for qom-get, qom-set, object-add

[Markus Armbruster]

With the previous commit, the generated marshalers just work, and save
us a bit of handwritten code.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 include/monitor/monitor.h |  3 ---
 qapi-schema.json          |  9 +++------
 qmp-commands.hx           |  6 +++---
 qmp.c                     | 20 +++++++-------------
 scripts/qapi.py           |  1 +
 5 files changed, 14 insertions(+), 25 deletions(-)

diff --git a/include/monitor/monitor.h b/include/monitor/monitor.h
index 9aff47e..bc6cb6d 100644
--- a/include/monitor/monitor.h
+++ b/include/monitor/monitor.h
@@ -42,9 +42,6 @@ void monitor_read_command(Monitor *mon, int show_prompt);
 int monitor_read_password(Monitor *mon, ReadLineFunc *readline_func,
                           void *opaque);
 
-void qmp_qom_set(QDict *qdict, QObject **ret, Error **errp);
-void qmp_qom_get(QDict *qdict, QObject **ret, Error **errp);
-void qmp_object_add(QDict *qdict, QObject **ret, Error **errp);
 void object_add(const char *type, const char *id, const QDict *qdict,
                 Visitor *v, Error **errp);
 
diff --git a/qapi-schema.json b/qapi-schema.json
index 106008c..815689a 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1695,8 +1695,7 @@
 ##
 { 'command': 'qom-get',
   'data': { 'path': 'str', 'property': 'str' },
-  'returns': '**',
-  'gen': false }
+  'returns': 'any' }
 
 ##
 # @qom-set:
@@ -1713,8 +1712,7 @@
 # Since: 1.2
 ##
 { 'command': 'qom-set',
-  'data': { 'path': 'str', 'property': 'str', 'value': '**' },
-  'gen': false }
+  'data': { 'path': 'str', 'property': 'str', 'value': 'any' } }
 
 ##
 # @set_password:
@@ -2110,8 +2108,7 @@
 # Since: 2.0
 ##
 { 'command': 'object-add',
-  'data': {'qom-type': 'str', 'id': 'str', '*props': '**'},
-  'gen': false }
+  'data': {'qom-type': 'str', 'id': 'str', '*props': 'any'} }
 
 ##
 # @object-del:
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 6760ddb..874da31 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -928,7 +928,7 @@ EQMP
     {
         .name       = "object-add",
         .args_type  = "qom-type:s,id:s,props:q?",
-        .mhandler.cmd_new = qmp_object_add,
+        .mhandler.cmd_new = qmp_marshal_object_add,
     },
 
 SQMP
@@ -3546,13 +3546,13 @@ EQMP
     {
         .name       = "qom-set",
 	.args_type  = "path:s,property:s,value:q",
-	.mhandler.cmd_new = qmp_qom_set,
+	.mhandler.cmd_new = qmp_marshal_qom_set,
     },
 
     {
         .name       = "qom-get",
 	.args_type  = "path:s,property:s",
-	.mhandler.cmd_new = qmp_qom_get,
+	.mhandler.cmd_new = qmp_marshal_qom_get,
     },
 
     {
diff --git a/qmp.c b/qmp.c
index 403805a..564a414 100644
--- a/qmp.c
+++ b/qmp.c
@@ -229,11 +229,9 @@ ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
 }
 
 /* FIXME: teach qapi about how to pass through Visitors */
-void qmp_qom_set(QDict *qdict, QObject **ret, Error **errp)
+void qmp_qom_set(const char *path, const char *property, QObject *value,
+                 Error **errp)
 {
-    const char *path = qdict_get_str(qdict, "path");
-    const char *property = qdict_get_str(qdict, "property");
-    QObject *value = qdict_get(qdict, "value");
     Object *obj;
 
     obj = object_resolve_path(path, NULL);
@@ -246,20 +244,18 @@ void qmp_qom_set(QDict *qdict, QObject **ret, Error **errp)
     object_property_set_qobject(obj, value, property, errp);
 }
 
-void qmp_qom_get(QDict *qdict, QObject **ret, Error **errp)
+QObject *qmp_qom_get(const char *path, const char *property, Error **errp)
 {
-    const char *path = qdict_get_str(qdict, "path");
-    const char *property = qdict_get_str(qdict, "property");
     Object *obj;
 
     obj = object_resolve_path(path, NULL);
     if (!obj) {
         error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
                   "Device '%s' not found", path);
-        return;
+        return NULL;
     }
 
-    *ret = object_property_get_qobject(obj, property, errp);
+    return object_property_get_qobject(obj, property, errp);
 }
 
 void qmp_set_password(const char *protocol, const char *password,
@@ -655,11 +651,9 @@ out:
     object_unref(obj);
 }
 
-void qmp_object_add(QDict *qdict, QObject **ret, Error **errp)
+void qmp_object_add(const char *type, const char *id,
+                    bool has_props, QObject *props, Error **errp)
 {
-    const char *type = qdict_get_str(qdict, "qom-type");
-    const char *id = qdict_get_str(qdict, "id");
-    QObject *props = qdict_get(qdict, "props");
     const QDict *pdict = NULL;
     QmpInputVisitor *qiv;
 
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 2ff087f..df63130 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -40,6 +40,7 @@ builtin_types = {
 returns_whitelist = [
     # From QMP:
     'human-monitor-command',
+    'qom-get',
     'query-migrate-cache-size',
     'query-tpm-models',
     'query-tpm-types',
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 42/47] qapi-schema: Fix up misleading specification of netdev_add

[Markus Armbruster]

It doesn't take a 'props' argument, let alone one in the format
"NAME=VALUE,..."

The bogus arguments specification doesn't matter due to 'gen': false.
Clean it up to be incomplete rather than wrong, and document the
incompleteness.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 docs/qapi-code-gen.txt |  2 +-
 qapi-schema.json       | 13 +++++++------
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index bf9e854..2367c66 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -466,7 +466,7 @@ try to avoid adding new commands that rely on this, and instead use
 type-safe unions.  For an example of bypass usage:
 
  { 'command': 'netdev_add',
-   'data': {'type': 'str', 'id': 'str', '*props': '**'},
+   'data': {'type': 'str', 'id': 'str'},
    'gen': false }
 
 Normally, the QAPI schema is used to describe synchronous exchanges,
diff --git a/qapi-schema.json b/qapi-schema.json
index 815689a..e246acb 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -2062,11 +2062,12 @@
 #
 # @id: the name of the new network backend
 #
-# @props: #optional a list of properties to be passed to the backend in
-#         the format 'name=value', like 'ifname=tap0,script=no'
+# Additional arguments depend on the type.
 #
-# Notes: The semantics of @props is not well defined.  Future commands will be
-#        introduced that provide stronger typing for backend creation.
+# TODO This command effectively bypasses QAPI completely due to its
+# "additional arguments" business.  It shouldn't have been added to
+# the schema in this form.  It should be qapified properly, or
+# replaced by a properly qapified command.
 #
 # Since: 0.14.0
 #
@@ -2074,8 +2075,8 @@
 #          If @type is not a valid network backend, DeviceNotFound
 ##
 { 'command': 'netdev_add',
-  'data': {'type': 'str', 'id': 'str', '*props': '**'},
-  'gen': false }
+  'data': {'type': 'str', 'id': 'str'},
+  'gen': false }                # so we can get the additional arguments
 
 ##
 # @netdev_del:
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 43/47] qmp: Improve netdev_add usage example in the manual

[Markus Armbruster]

Add a device option to show how it's done.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 qmp-commands.hx | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/qmp-commands.hx b/qmp-commands.hx
index 874da31..351173e 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -892,7 +892,9 @@ Arguments:
 
 Example:
 
--> { "execute": "netdev_add", "arguments": { "type": "user", "id": "netdev1" } }
+-> { "execute": "netdev_add",
+     "arguments": { "type": "user", "id": "netdev1",
+                    "dnssearch": "example.org" } }
 <- { "return": {} }
 
 Note: The supported device options are the same ones supported by the '-netdev'
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 44/47] qapi: Pseudo-type '**' is now unused, drop it

[Markus Armbruster]

'gen': false needs to stay for now, because netdev_add is still using
it.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 docs/qapi-code-gen.txt                               | 15 ++++-----------
 scripts/qapi.py                                      | 20 ++++----------------
 tests/Makefile                                       |  4 ++--
 tests/qapi-schema/args-returns-any.err               |  1 +
 ...type-bypass-no-gen.exit => args-returns-any.exit} |  0
 tests/qapi-schema/args-returns-any.json              |  6 ++++++
 .../{type-bypass-no-gen.out => args-returns-any.out} |  0
 tests/qapi-schema/flat-union-base-star.err           |  2 +-
 tests/qapi-schema/flat-union-base-star.json          |  2 +-
 tests/qapi-schema/type-bypass-no-gen.err             |  1 -
 tests/qapi-schema/type-bypass-no-gen.json            |  2 --
 tests/qapi-schema/type-bypass.err                    |  0
 tests/qapi-schema/type-bypass.exit                   |  1 -
 tests/qapi-schema/type-bypass.json                   |  2 --
 tests/qapi-schema/type-bypass.out                    |  4 ----
 15 files changed, 19 insertions(+), 41 deletions(-)
 create mode 100644 tests/qapi-schema/args-returns-any.err
 rename tests/qapi-schema/{type-bypass-no-gen.exit => args-returns-any.exit} (100%)
 create mode 100644 tests/qapi-schema/args-returns-any.json
 rename tests/qapi-schema/{type-bypass-no-gen.out => args-returns-any.out} (100%)
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.err
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.json
 delete mode 100644 tests/qapi-schema/type-bypass.err
 delete mode 100644 tests/qapi-schema/type-bypass.exit
 delete mode 100644 tests/qapi-schema/type-bypass.json
 delete mode 100644 tests/qapi-schema/type-bypass.out

diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index 2367c66..ca578dd 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -111,10 +111,7 @@ and field names within a type, should be all lower case with words
 separated by a hyphen.  However, some existing older commands and
 complex types use underscore; when extending such expressions,
 consistency is preferred over blindly avoiding underscore.  Event
-names should be ALL_CAPS with words separated by underscore.  The
-special string '**' appears for some commands that manually perform
-their own type checking rather than relying on the type-safe code
-produced by the qapi code generators.
+names should be ALL_CAPS with words separated by underscore.
 
 Any name (command, event, type, field, or enum value) beginning with
 "x-" is marked experimental, and may be withdrawn or changed
@@ -457,13 +454,9 @@ which would validate this Client JSON Protocol transaction:
 
 In rare cases, QAPI cannot express a type-safe representation of a
 corresponding Client JSON Protocol command.  In these cases, if the
-command expression includes the key 'gen' with boolean value false,
-then the 'data' or 'returns' member that intends to bypass generated
-type-safety and do its own manual validation should use an inline
-dictionary definition, with a value of '**' rather than a valid type
-name for the keys that the generated code will not validate.  Please
-try to avoid adding new commands that rely on this, and instead use
-type-safe unions.  For an example of bypass usage:
+command expression includes the key 'gen' with boolean value false.
+Please try to avoid adding new commands that rely on this, and instead
+use type-safe unions.  For an example of bypass usage:
 
  { 'command': 'netdev_add',
    'data': {'type': 'str', 'id': 'str'},
diff --git a/scripts/qapi.py b/scripts/qapi.py
index df63130..6ddb33e 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -423,16 +423,13 @@ def is_enum(name):
 
 def check_type(expr_info, source, value, allow_array = False,
                allow_dict = False, allow_optional = False,
-               allow_star = False, allow_metas = []):
+               allow_metas = []):
     global all_names
     orig_value = value
 
     if value is None:
         return
 
-    if allow_star and value == '**':
-        return
-
     # Check if array type for value is okay
     if isinstance(value, list):
         if not allow_array:
@@ -447,10 +444,6 @@ def check_type(expr_info, source, value, allow_array = False,
 
     # Check if type name for value is okay
     if isinstance(value, str):
-        if value == '**':
-            raise QAPIExprError(expr_info,
-                                "%s uses '**' but did not request 'gen':false"
-                                % source)
         if not value in all_names:
             raise QAPIExprError(expr_info,
                                 "%s uses unknown type '%s'"
@@ -474,7 +467,7 @@ def check_type(expr_info, source, value, allow_array = False,
         # Todo: allow dictionaries to represent default values of
         # an optional argument.
         check_type(expr_info, "Member '%s' of %s" % (key, source), arg,
-                   allow_array=True, allow_star=allow_star,
+                   allow_array=True,
                    allow_metas=['built-in', 'union', 'alternate', 'struct',
                                 'enum'])
 
@@ -494,18 +487,16 @@ def check_member_clash(expr_info, base_name, data, source = ""):
 
 def check_command(expr, expr_info):
     name = expr['command']
-    allow_star = expr.has_key('gen')
 
     check_type(expr_info, "'data' for command '%s'" % name,
                expr.get('data'), allow_dict=True, allow_optional=True,
-               allow_metas=['struct'], allow_star=allow_star)
+               allow_metas=['struct'])
     returns_meta = ['union', 'struct']
     if name in returns_whitelist:
         returns_meta += ['built-in', 'alternate', 'enum']
     check_type(expr_info, "'returns' for command '%s'" % name,
                expr.get('returns'), allow_array=True, allow_dict=True,
-               allow_optional=True, allow_metas=returns_meta,
-               allow_star=allow_star)
+               allow_optional=True, allow_metas=returns_meta)
 
 def check_event(expr, expr_info):
     global events
@@ -1051,7 +1042,6 @@ class QAPISchema(object):
                   ('bool',   'boolean', 'bool',     'false'),
                   ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
             self._def_builtin_type(*t)
-        self.entity_dict['**'] = self.lookup_type('any') # TODO drop this alias
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
@@ -1192,8 +1182,6 @@ class QAPISchema(object):
     def visit(self, visitor):
         visitor.visit_begin()
         for name in sorted(self.entity_dict.keys()):
-            if self.entity_dict[name].name != name:
-                continue        # ignore alias TODO drop alias and remove
             self.entity_dict[name].visit(visitor)
         visitor.visit_end()
 
diff --git a/tests/Makefile b/tests/Makefile
index 5a38748..60b82e2 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -222,11 +222,11 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	bad-type-dict.json double-data.json unknown-expr-key.json \
 	redefined-type.json redefined-command.json redefined-builtin.json \
 	redefined-event.json command-int.json bad-data.json event-max.json \
-	type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \
+	type-bypass-bad-gen.json \
 	data-array-empty.json data-array-unknown.json data-int.json \
 	data-unknown.json data-member-unknown.json data-member-array.json \
 	data-member-array-bad.json args-union.json args-alternate.json \
-	returns-array-bad.json returns-int.json \
+	args-returns-any.json returns-array-bad.json returns-int.json \
 	returns-unknown.json returns-alternate.json returns-whitelist.json \
 	missing-colon.json missing-comma-list.json missing-comma-object.json \
 	nested-struct-data.json nested-struct-returns.json non-objects.json \
diff --git a/tests/qapi-schema/args-returns-any.err b/tests/qapi-schema/args-returns-any.err
new file mode 100644
index 0000000..f7ce727
--- /dev/null
+++ b/tests/qapi-schema/args-returns-any.err
@@ -0,0 +1 @@
+tests/qapi-schema/args-returns-any.json:6: 'data' for command 'bar' cannot use built-in type 'any'
diff --git a/tests/qapi-schema/type-bypass-no-gen.exit b/tests/qapi-schema/args-returns-any.exit
similarity index 100%
rename from tests/qapi-schema/type-bypass-no-gen.exit
rename to tests/qapi-schema/args-returns-any.exit
diff --git a/tests/qapi-schema/args-returns-any.json b/tests/qapi-schema/args-returns-any.json
new file mode 100644
index 0000000..a74fc6f
--- /dev/null
+++ b/tests/qapi-schema/args-returns-any.json
@@ -0,0 +1,6 @@
+# built-in type 'any' in arguments and returns
+# works except for 'data': 'any', because that has to be a struct
+# note: command name 'qom-get' chosen to avoid "cannot use built-in" error
+{ 'command': 'qom-get', 'data': { 'arg': 'any' }, 'returns': 'any' }
+{ 'command': 'foo', 'returns': { 'arg': 'any' } }
+{ 'command': 'bar', 'data': 'any' }
diff --git a/tests/qapi-schema/type-bypass-no-gen.out b/tests/qapi-schema/args-returns-any.out
similarity index 100%
rename from tests/qapi-schema/type-bypass-no-gen.out
rename to tests/qapi-schema/args-returns-any.out
diff --git a/tests/qapi-schema/flat-union-base-star.err b/tests/qapi-schema/flat-union-base-star.err
index b7748f0..43e5aa4 100644
--- a/tests/qapi-schema/flat-union-base-star.err
+++ b/tests/qapi-schema/flat-union-base-star.err
@@ -1 +1 @@
-tests/qapi-schema/flat-union-base-star.json:8: Base '**' is not a valid struct
+tests/qapi-schema/flat-union-base-star.json:8: Base 'any' is not a valid struct
diff --git a/tests/qapi-schema/flat-union-base-star.json b/tests/qapi-schema/flat-union-base-star.json
index 5099439..fe66b71 100644
--- a/tests/qapi-schema/flat-union-base-star.json
+++ b/tests/qapi-schema/flat-union-base-star.json
@@ -6,7 +6,7 @@
 { 'struct': 'TestTypeB',
   'data': { 'integer': 'int' } }
 { 'union': 'TestUnion',
-  'base': '**',
+  'base': 'any',
   'discriminator': 'enum1',
   'data': { 'value1': 'TestTypeA',
             'value2': 'TestTypeB' } }
diff --git a/tests/qapi-schema/type-bypass-no-gen.err b/tests/qapi-schema/type-bypass-no-gen.err
deleted file mode 100644
index 20cef0a..0000000
--- a/tests/qapi-schema/type-bypass-no-gen.err
+++ /dev/null
@@ -1 +0,0 @@
-tests/qapi-schema/type-bypass-no-gen.json:2: Member 'arg' of 'data' for command 'unsafe' uses '**' but did not request 'gen':false
diff --git a/tests/qapi-schema/type-bypass-no-gen.json b/tests/qapi-schema/type-bypass-no-gen.json
deleted file mode 100644
index 4feae37..0000000
--- a/tests/qapi-schema/type-bypass-no-gen.json
+++ /dev/null
@@ -1,2 +0,0 @@
-# type bypass only works with 'gen':false
-{ 'command': 'unsafe', 'data': { 'arg': '**' }, 'returns': '**' }
diff --git a/tests/qapi-schema/type-bypass.err b/tests/qapi-schema/type-bypass.err
deleted file mode 100644
index e69de29..0000000
diff --git a/tests/qapi-schema/type-bypass.exit b/tests/qapi-schema/type-bypass.exit
deleted file mode 100644
index 573541a..0000000
--- a/tests/qapi-schema/type-bypass.exit
+++ /dev/null
@@ -1 +0,0 @@
-0
diff --git a/tests/qapi-schema/type-bypass.json b/tests/qapi-schema/type-bypass.json
deleted file mode 100644
index 48b2137..0000000
--- a/tests/qapi-schema/type-bypass.json
+++ /dev/null
@@ -1,2 +0,0 @@
-# Use of 'gen':false allows bypassing type system
-{ 'command': 'unsafe', 'data': { 'arg': '**' }, 'returns': '**', 'gen': false }
diff --git a/tests/qapi-schema/type-bypass.out b/tests/qapi-schema/type-bypass.out
deleted file mode 100644
index 2dc0c09..0000000
--- a/tests/qapi-schema/type-bypass.out
+++ /dev/null
@@ -1,4 +0,0 @@
-object :obj-unsafe-args
-    member arg: any optional=False
-command unsafe :obj-unsafe-args -> any
-   gen=False success_response=True
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 45/47] qapi: New QMP command query-schema for QMP schema introspection

[Markus Armbruster]

Caution, rough edges.

qapi/introspect.json defines the introspection schema.  It should do
for uses other than QMP.
FIXME it's almost entirely devoid of comments.

The introspection schema does not reflect all the rules and
restrictions that apply to QAPI schemata.  A valid QAPI schema has an
introspection value conforming to the introspection schema, but the
converse is not true.

Introspection lowers away a number of schema details:

* The built-in types are declared with their JSON type.

  TODO Should we map all the integer types to just int?

* Implicit type definitions are made explicit, and given
  auto-generated names.  These names start with ':' so they don't
  clash with the user's names.

  Example: a simple union implicitly defines an enumeration type for
  its discriminator.

* All type references are by name.

* Base types are flattened.

* The top type (named 'any') can hold any value.

* The struct and union types are generalized into an object type.

* Commands take a single argument and return a single result.

  Dictionary argument/result or list result is an implicit type
  definition.

  The empty object type is used when a command takes no arguments or
  produces no results.

  The argument is always of object type, but the introspection schema
  doesn't reflect that.

  The 'gen': false and 'success-response': false directives are
  omitted as implementation detail.

* Events carry a single data value.

  Implicit type definition and empty object type use, just like for
  commands.

  The value is of object type, but the introspection schema doesn't
  reflect that.

* Types not used by commands or events are omitted.

  Indirect use counts as use.

* Optional members have a default, which can only be null right now

  Instead of a mandatory "optional" flag, we have an optional default.
  No default means mandatory, default null means optional without
  default value.  Non-null is available for optional with default.

  Alternate members can't have defaults, but the introspection schema
  doesn't reflect that.

* Clients should *not* look up types by name, because type names are
  not ABI.  Look up the command or event you're interested in, then
  follow the references.

  TODO Should we hide the type names to eliminate the temptation?

* Likewise, the names of alternate members are not ABI, and should not
  be examined.

  TODO Should we hide them, too?

TODO much of the above should go into docs.

New generator scripts/qapi-introspect.py computes an introspection
value for its input, and generates a C variable holding it.

FIXME it can generate awfully long lines

A new test-qmp-input-visitor test case feeds its result for both
tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a
QmpInputVisitor to verify it actually conforms to the schema.

New QMP command query-schema takes its return value from that
variable.  Command documentation is incomplete, and marked FIXME.  Its
reply is some 80KiBytes for me right now.

If this turns out to be too much, we have a couple of options:

* We can use shorter names in the JSON.  Not the QMP style.

* Optionally return the sub-schema for commands and events given as
  arguments.

  Right now qmp_query_schema() sends the string literal computed by
  qmp-introspect.py.  To compute sub-schema at run time, we'd have to
  duplicate parts of qapi-introspect.py in C.  Unattractive.

* Let clients cache the output of query-schema.

  It changes only on QEMU upgrades, i.e. rarely.  Provide a command
  query-schema-hash.  Clients can have a cache indexed by hash, and
  re-query the schema only when they don't have it cached.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 .gitignore                                      |   1 +
 Makefile                                        |   9 +-
 Makefile.objs                                   |   4 +-
 monitor.c                                       |  15 +++
 qapi-schema.json                                |   3 +
 qapi/introspect.json                            |  69 ++++++++++
 qmp-commands.hx                                 |  16 +++
 scripts/qapi-commands.py                        |   5 +-
 scripts/qapi-event.py                           |   3 +-
 scripts/qapi-introspect.py                      | 159 ++++++++++++++++++++++++
 scripts/qapi-types.py                           |   1 -
 scripts/qapi-visit.py                           |   2 +-
 scripts/qapi.py                                 |  14 ++-
 tests/.gitignore                                |   1 +
 tests/Makefile                                  |  10 +-
 tests/qapi-schema/alternate-good.out            |   1 +
 tests/qapi-schema/comments.out                  |   1 +
 tests/qapi-schema/data-member-array.out         |   1 +
 tests/qapi-schema/empty.out                     |   1 +
 tests/qapi-schema/enum-empty.out                |   1 +
 tests/qapi-schema/event-case.out                |   1 +
 tests/qapi-schema/flat-union-reverse-define.out |   1 +
 tests/qapi-schema/ident-with-escape.out         |   1 +
 tests/qapi-schema/include-relpath.out           |   1 +
 tests/qapi-schema/include-repetition.out        |   1 +
 tests/qapi-schema/include-simple.out            |   1 +
 tests/qapi-schema/indented-expr.out             |   1 +
 tests/qapi-schema/qapi-schema-test.out          |   1 +
 tests/qapi-schema/returns-int.out               |   1 +
 tests/test-qmp-input-visitor.c                  |  31 +++++
 30 files changed, 345 insertions(+), 12 deletions(-)
 create mode 100644 qapi/introspect.json
 create mode 100644 scripts/qapi-introspect.py

diff --git a/.gitignore b/.gitignore
index aed0e1f..a6a02db 100644
--- a/.gitignore
+++ b/.gitignore
@@ -32,6 +32,7 @@
 /qapi-visit.[ch]
 /qapi-event.[ch]
 /qmp-commands.h
+/qmp-introspect.[ch]
 /qmp-marshal.c
 /qemu-doc.html
 /qemu-tech.html
diff --git a/Makefile b/Makefile
index c9be643..a91afd0 100644
--- a/Makefile
+++ b/Makefile
@@ -52,6 +52,8 @@ endif
 GENERATED_HEADERS = config-host.h qemu-options.def
 GENERATED_HEADERS += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
 GENERATED_SOURCES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
+GENERATED_HEADERS += qmp-introspect.h
+GENERATED_SOURCES += qmp-introspect.c
 
 GENERATED_HEADERS += trace/generated-events.h
 GENERATED_SOURCES += trace/generated-events.c
@@ -263,7 +265,7 @@ $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
 
 qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \
-               $(SRC_PATH)/qapi/event.json
+               $(SRC_PATH)/qapi/event.json $(SRC_PATH)/qapi/introspect.json
 
 qapi-types.c qapi-types.h :\
 $(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
@@ -285,6 +287,11 @@ $(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
 	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
 		$(gen-out-type) -o "." -m $<, \
 		"  GEN   $@")
+qmp-introspect.h qmp-introspect.c :\
+$(qapi-modules) $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
+		$(gen-out-type) -o "." $<, \
+		"  GEN   $@")
 
 QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qmp-commands.h)
 $(qga-obj-y) qemu-ga.o: $(QGALIB_GEN)
diff --git a/Makefile.objs b/Makefile.objs
index 4881d2c..1214a5e 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -1,7 +1,8 @@
 #######################################################################
 # Common libraries for tools and emulators
 stub-obj-y = stubs/
-util-obj-y = util/ qobject/ qapi/ qapi-types.o qapi-visit.o qapi-event.o
+util-obj-y = util/ qobject/ qapi/
+util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
 
 #######################################################################
 # block-obj-y is code used by both qemu system emulation and qemu-img
@@ -82,6 +83,7 @@ common-obj-$(CONFIG_FDT) += device_tree.o
 # qapi
 
 common-obj-y += qmp-marshal.o
+common-obj-y += qmp-introspect.o
 common-obj-y += qmp.o hmp.o
 endif
 
diff --git a/monitor.c b/monitor.c
index 16672f1..0c71526 100644
--- a/monitor.c
+++ b/monitor.c
@@ -74,6 +74,7 @@
 #include "block/qapi.h"
 #include "qapi/qmp-event.h"
 #include "qapi-event.h"
+#include "qmp-introspect.h"
 #include "sysemu/block-backend.h"
 
 /* for hmp_info_irq/pic */
@@ -924,6 +925,20 @@ EventInfoList *qmp_query_events(Error **errp)
     return ev_list;
 }
 
+/*
+ * Minor hack: generated marshalling suppressed for this command
+ * ('gen': false in the schema) so we can simply send the JSON string
+ * instead of first parsing it with visit_type_SchemaInfoList() into a
+ * SchemaInfoList, then unparse it right back in the generated output
+ * marshaller, every time.
+ * Instead, we do it in test-qmp-input-visitor.c, just to make sure
+ * qapi-introspect.py's output actually conforms to the schema.
+ */
+static void qmp_query_schema(QDict *qdict, QObject **ret_data, Error **errp)
+{
+    *ret_data = qobject_from_json(qmp_schema_json);
+}
+
 /* set the current CPU defined by the user */
 int monitor_set_cpu(int cpu_index)
 {
diff --git a/qapi-schema.json b/qapi-schema.json
index e246acb..7a3b8a0 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -14,6 +14,9 @@
 # Tracing commands
 { 'include': 'qapi/trace.json' }
 
+# QAPI introspection
+{ 'include': 'qapi/introspect.json' }
+
 ##
 # LostTickPolicy:
 #
diff --git a/qapi/introspect.json b/qapi/introspect.json
new file mode 100644
index 0000000..673df56
--- /dev/null
+++ b/qapi/introspect.json
@@ -0,0 +1,69 @@
+# -*- Mode: Python -*-
+#
+# QAPI introspection
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# Authors:
+#  Markus Armbruster <armbru@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+{ 'enum': 'SchemaMetaType',
+  'data': [ 'builtin', 'enum', 'array', 'object', 'alternate',
+            'command', 'event' ] }
+
+{ 'struct': 'SchemaInfoBase',
+  'data': { 'name': 'str', 'meta-type': 'SchemaMetaType' } }
+
+{ 'enum': 'JSONType',
+  'data': [ 'string', 'number', 'int', 'boolean', 'null',
+            'object', 'array', 'value' ] }
+
+{ 'struct': 'SchemaInfoBuiltin',
+  'data': { 'json-type': 'JSONType' } }
+
+{ 'struct': 'SchemaInfoEnum',
+  'data': { 'values': ['str'] } }
+
+{ 'struct': 'SchemaInfoArray',
+  'data': { 'element-type': 'str' } }
+
+{ 'struct': 'SchemaInfoObjectMember',
+  'data': { 'name': 'str', 'type': 'str', '*default': 'any' } }
+# @default's type must match @type
+
+{ 'struct': 'SchemaInfoObjectVariant',
+  'data': { 'case': 'str',
+            'members': [ 'SchemaInfoObjectMember' ] } }
+
+{ 'struct': 'SchemaInfoObject',
+  'data': { 'members': [ 'SchemaInfoObjectMember' ],
+            '*tag': 'str',
+            '*variants': [ 'SchemaInfoObjectVariant' ] } }
+
+{ 'struct': 'SchemaInfoAlternate',
+  'data': { 'members': [ 'SchemaInfoObjectMember' ] } }
+
+{ 'struct': 'SchemaInfoCommand',
+  'data': { 'args': 'str', 'returns': 'str' } }
+
+{ 'struct': 'SchemaInfoEvent',
+  'data': { 'data': 'str' } }
+
+{ 'union': 'SchemaInfo',
+  'base': 'SchemaInfoBase',
+  'discriminator': 'meta-type',
+  'data': {
+      'builtin': 'SchemaInfoBuiltin',
+      'enum': 'SchemaInfoEnum',
+      'array': 'SchemaInfoArray',
+      'object': 'SchemaInfoObject',
+      'alternate': 'SchemaInfoAlternate',
+      'command': 'SchemaInfoCommand',
+      'event': 'SchemaInfoEvent' } }
+
+{ 'command': 'query-schema',
+  'returns': [ 'SchemaInfo' ],
+  'gen': false }                # just to simplify qmp_query_json()
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 351173e..31a3f57 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -2165,6 +2165,22 @@ EQMP
     },
 
 SQMP
+query-schema
+------------
+
+Return the QMP schema.
+
+FIXME explain
+
+EQMP
+
+    {
+        .name       = "query-schema",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_query_schema,
+    },
+
+SQMP
 query-chardev
 -------------
 
diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index fcbb7d0..eef4401 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -265,7 +265,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self.defn = None
         self.regy = None
         self.visited_rets = None
-    def visit_begin(self):
+    def visit_begin(self, schema):
         self.decl = ''
         self.defn = ''
         self.regy = ''
@@ -273,7 +273,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
     def visit_end(self):
         if not middle_mode:
             self.defn += gen_registry(self.regy)
-            self.regy = None
+        self.regy = None
+        self.visited_rets = None
     def visit_command(self, name, info, args, rets, gen, success_response):
         if not gen:
             return
diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 184a81f..71da7a9 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -139,13 +139,14 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self.decl = None
         self.defn = None
         self.event_names = None
-    def visit_begin(self):
+    def visit_begin(self, schema):
         self.decl = ''
         self.defn = ''
         self.event_names = []
     def visit_end(self):
         self.decl += gen_enum(event_enum_name, self.event_names)
         self.defn += gen_enum_lookup(event_enum_name, self.event_names)
+        self.event_names = None
     def visit_event(self, name, info, data):
         self.decl += gen_event_send_decl(name, data)
         self.defn += gen_event_send(name, data)
diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
new file mode 100644
index 0000000..e7efc4a
--- /dev/null
+++ b/scripts/qapi-introspect.py
@@ -0,0 +1,159 @@
+#
+# QAPI introspection generator
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# Authors:
+#  Markus Armbruster <armbru@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from qapi import *
+
+class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.schema = None
+        self.jsons = None
+        self.used_types = None
+        self.defn = None
+        self.decl = None
+
+    def visit_begin(self, schema):
+        self.schema = schema
+        self.jsons = []
+        self.used_types = []
+        return QAPISchemaType   # don't visit types for now
+
+    def visit_end(self):
+        # visit the types that are actually used
+        for typ in self.used_types:
+            typ.visit(self)
+        self.jsons.sort()
+        name = prefix + 'qmp_schema_json'
+        self.decl = mcgen('''
+extern char %(c_name)s[];
+''',
+                          c_name=c_name(name))
+        self.defn = mcgen('''
+char %(c_name)s[] = "["
+    "%(c_jsons)s]";
+''',
+                          c_name=c_name(name),
+                          c_jsons=', "\n    "'.join(self.jsons))
+        self.schema = None
+        self.jsons = None
+        self.used_types = None
+
+    def _use_type(self, typ):
+        if typ not in self.used_types:
+            self.used_types.append(typ)
+        return typ.name
+
+    def _gen_json(self, name, mtype, extra):
+        self.jsons.append("{ 'name': '%s', 'meta-type': '%s', %s }"
+                          % (name, mtype, extra))
+
+    def _gen_members(self, members):
+        return ("'members': [ "
+                + ", ".join([self._gen_member(m) for m in members])
+                + " ]")
+
+    def _gen_member(self, member):
+        default = ''
+        if member.optional:
+            default = ", 'default': null"
+        return "{ 'name': '%s', 'type': '%s'%s }" \
+            % (member.name, self._use_type(member.type), default)
+
+    def _gen_variants(self, tag_name, variants):
+        return ("'tag': '%s'" % tag_name
+                + ", 'variants': [ "
+                + ", ".join([self._gen_variant(v) for v in variants])
+                + " ]")
+
+    def _gen_variant(self, variant):
+        if variant.flat:
+            members = self._gen_members(variant.type.members)
+        else:
+            members = "'members': [ { 'name': 'data', 'type': '%s' } ]" \
+                      % self._use_type(variant.type)
+        return "{ 'case': '%s', %s }" % (variant.name, members)
+
+    def visit_builtin_type(self, name, info, json_type):
+        self._gen_json(name, 'builtin',
+                       "'json-type': '%s'" % json_type)
+
+    def visit_enum_type(self, name, info, values):
+        self._gen_json(name, 'enum',
+                       "'values': [ %s ]" % ", ".join(["'%s'" % v
+                                                       for v in values]))
+
+    def visit_array_type(self, name, info, element_type):
+        self._gen_json(name, 'array',
+                       "'element-type': '%s'" % self._use_type(element_type))
+
+    def visit_object_type_flat(self, name, info, members, variants):
+        extra = self._gen_members(members)
+        if variants:
+            extra += ", " + self._gen_variants(variants.tag_name or "type",
+                                               variants.variants)
+        self._gen_json(name, 'object', extra)
+
+    def visit_alternate_type(self, name, info, variants):
+        self._gen_json(name, 'alternate',
+                       self._gen_members(variants.variants))
+
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        args = args or self.schema.the_empty_object_type
+        rets = rets or self.schema.the_empty_object_type
+        self._gen_json(name, 'command',
+                       "'args': '%s', 'returns': '%s'" \
+                       % (self._use_type(args), self._use_type(rets)))
+
+    def visit_event(self, name, info, data):
+        data = data or self.schema.the_empty_object_type
+        self._gen_json(name, 'event', "'data': '%s'" % self._use_type(data))
+
+(input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
+
+c_comment = '''
+/*
+ * QAPI/QMP schema introspection
+ *
+ * Copyright (C) 2015 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+'''
+h_comment = '''
+/*
+ * QAPI/QMP schema introspection
+ *
+ * Copyright (C) 2015 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+'''
+
+(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
+                            'qmp-introspect.c', 'qmp-introspect.h',
+                            c_comment, h_comment)
+
+fdef.write(mcgen('''
+#include "%(prefix)sqmp-introspect.h"
+
+''',
+                 prefix=prefix))
+
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenIntrospectVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
+
+close_output(fdef, fdecl)
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index 7c403c0..12f3767 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -184,7 +184,6 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.fwdecl = None
         self.fwdefn = None
         self.btin = None
-    def visit_begin(self):
         self.decl = ''
         self.defn = ''
         self.fwdecl = ''
diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 2813bb3..7a03292 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -326,7 +326,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         self.decl = None
         self.defn = None
         self.btin = None
-    def visit_begin(self):
+    def visit_begin(self, schema):
         self.decl = ''
         self.defn = ''
         self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 6ddb33e..7f9a159 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -764,7 +764,7 @@ class QAPISchemaEntity(object):
         pass
 
 class QAPISchemaVisitor(object):
-    def visit_begin(self):
+    def visit_begin(self, schema):
         pass
     def visit_end(self):
         pass
@@ -776,6 +776,8 @@ class QAPISchemaVisitor(object):
         pass
     def visit_object_type(self, name, info, base, members, variants):
         pass
+    def visit_object_type_flat(self, name, info, members, variants):
+        pass
     def visit_alternate_type(self, name, info, variants):
         pass
     def visit_command(self, name, info, args, rets, gen, success_response):
@@ -892,6 +894,8 @@ class QAPISchemaObjectType(QAPISchemaType):
     def visit(self, visitor):
         visitor.visit_object_type(self.name, self.info,
                                   self.base, self.local_members, self.variants)
+        visitor.visit_object_type_flat(self.name, self.info,
+                                       self.members, self.variants)
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -1042,6 +1046,9 @@ class QAPISchema(object):
                   ('bool',   'boolean', 'bool',     'false'),
                   ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
             self._def_builtin_type(*t)
+        self.the_empty_object_type = QAPISchemaObjectType(':empty', None, None,
+                                                          [], None)
+        self._def_entity(self.the_empty_object_type)
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
@@ -1180,9 +1187,10 @@ class QAPISchema(object):
             ent.check(self)
 
     def visit(self, visitor):
-        visitor.visit_begin()
+        ignore = visitor.visit_begin(self)
         for name in sorted(self.entity_dict.keys()):
-            self.entity_dict[name].visit(visitor)
+            if not ignore or not isinstance(self.entity_dict[name], ignore):
+                self.entity_dict[name].visit(visitor)
         visitor.visit_end()
 
 #
diff --git a/tests/.gitignore b/tests/.gitignore
index dc813c2..dda86cc 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -19,6 +19,7 @@ test-opts-visitor
 test-qapi-event.[ch]
 test-qapi-types.[ch]
 test-qapi-visit.[ch]
+test-qapi-introspect.[ch]
 test-qdev-global-props
 test-qemu-opts
 test-qmp-commands
diff --git a/tests/Makefile b/tests/Makefile
index 60b82e2..7b1bf92 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -253,7 +253,8 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	struct-base-clash.json struct-base-clash-deep.json )
 
 GENERATED_HEADERS += tests/test-qapi-types.h tests/test-qapi-visit.h \
-		     tests/test-qmp-commands.h tests/test-qapi-event.h
+	tests/test-qmp-commands.h tests/test-qapi-event.h \
+	tests/test-qmp-introspect.h
 
 test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
 	tests/check-qlist.o tests/check-qfloat.o tests/check-qjson.o \
@@ -266,7 +267,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
 	tests/rcutorture.o tests/test-rcu-list.o
 
 test-qapi-obj-y = tests/test-qapi-visit.o tests/test-qapi-types.o \
-		  tests/test-qapi-event.o
+		  tests/test-qapi-event.o tests/test-qmp-introspect.o
 
 $(test-obj-y): QEMU_INCLUDES += -Itests
 QEMU_CFLAGS += -I$(SRC_PATH)/tests
@@ -327,6 +328,11 @@ $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-eve
 	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
 		$(gen-out-type) -o tests -p "test-" $<, \
 		"  GEN   $@")
+tests/test-qmp-introspect.c tests/test-qmp-introspect.h :\
+$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
+		$(gen-out-type) -o tests -p "test-" $<, \
+		"  GEN   $@")
 
 tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y) libqemuutil.a libqemustub.a
 tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y) libqemuutil.a libqemustub.a
diff --git a/tests/qapi-schema/alternate-good.out b/tests/qapi-schema/alternate-good.out
index 0cbdfa1..aede1ae 100644
--- a/tests/qapi-schema/alternate-good.out
+++ b/tests/qapi-schema/alternate-good.out
@@ -1,3 +1,4 @@
+object :empty
 alternate Alt
     case value: int flat=False
     case string: Enum flat=False
diff --git a/tests/qapi-schema/comments.out b/tests/qapi-schema/comments.out
index 6161b90..9e2c656 100644
--- a/tests/qapi-schema/comments.out
+++ b/tests/qapi-schema/comments.out
@@ -1 +1,2 @@
+object :empty
 enum Status ['good', 'bad', 'ugly']
diff --git a/tests/qapi-schema/data-member-array.out b/tests/qapi-schema/data-member-array.out
index 8911179..d72ce48 100644
--- a/tests/qapi-schema/data-member-array.out
+++ b/tests/qapi-schema/data-member-array.out
@@ -1,3 +1,4 @@
+object :empty
 object :obj-okay-args
     member member1: intList optional=False
     member member2: defList optional=False
diff --git a/tests/qapi-schema/empty.out b/tests/qapi-schema/empty.out
index e69de29..272b161 100644
--- a/tests/qapi-schema/empty.out
+++ b/tests/qapi-schema/empty.out
@@ -0,0 +1 @@
+object :empty
diff --git a/tests/qapi-schema/enum-empty.out b/tests/qapi-schema/enum-empty.out
index e09b00f..a449d45 100644
--- a/tests/qapi-schema/enum-empty.out
+++ b/tests/qapi-schema/enum-empty.out
@@ -1 +1,2 @@
+object :empty
 enum MyEnum []
diff --git a/tests/qapi-schema/event-case.out b/tests/qapi-schema/event-case.out
index b5ae4c2..cdfd264 100644
--- a/tests/qapi-schema/event-case.out
+++ b/tests/qapi-schema/event-case.out
@@ -1 +1,2 @@
+object :empty
 event oops None
diff --git a/tests/qapi-schema/flat-union-reverse-define.out b/tests/qapi-schema/flat-union-reverse-define.out
index e156202..1c30867 100644
--- a/tests/qapi-schema/flat-union-reverse-define.out
+++ b/tests/qapi-schema/flat-union-reverse-define.out
@@ -1,3 +1,4 @@
+object :empty
 object TestBase
     member enum1: TestEnum optional=False
 enum TestEnum ['value1', 'value2']
diff --git a/tests/qapi-schema/ident-with-escape.out b/tests/qapi-schema/ident-with-escape.out
index 24abf5c..cbb16d6 100644
--- a/tests/qapi-schema/ident-with-escape.out
+++ b/tests/qapi-schema/ident-with-escape.out
@@ -1,3 +1,4 @@
+object :empty
 object :obj-fooA-args
     member bar1: str optional=False
 command fooA :obj-fooA-args -> None
diff --git a/tests/qapi-schema/include-relpath.out b/tests/qapi-schema/include-relpath.out
index 6161b90..9e2c656 100644
--- a/tests/qapi-schema/include-relpath.out
+++ b/tests/qapi-schema/include-relpath.out
@@ -1 +1,2 @@
+object :empty
 enum Status ['good', 'bad', 'ugly']
diff --git a/tests/qapi-schema/include-repetition.out b/tests/qapi-schema/include-repetition.out
index 6161b90..9e2c656 100644
--- a/tests/qapi-schema/include-repetition.out
+++ b/tests/qapi-schema/include-repetition.out
@@ -1 +1,2 @@
+object :empty
 enum Status ['good', 'bad', 'ugly']
diff --git a/tests/qapi-schema/include-simple.out b/tests/qapi-schema/include-simple.out
index 6161b90..9e2c656 100644
--- a/tests/qapi-schema/include-simple.out
+++ b/tests/qapi-schema/include-simple.out
@@ -1 +1,2 @@
+object :empty
 enum Status ['good', 'bad', 'ugly']
diff --git a/tests/qapi-schema/indented-expr.out b/tests/qapi-schema/indented-expr.out
index c5af55a..226d300 100644
--- a/tests/qapi-schema/indented-expr.out
+++ b/tests/qapi-schema/indented-expr.out
@@ -1,3 +1,4 @@
+object :empty
 command eins None -> None
    gen=True success_response=True
 command zwei None -> None
diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
index 921d7fb..8a12d51 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -1,3 +1,4 @@
+object :empty
 object :obj-EVENT_C-data
     member a: int optional=True
     member b: UserDefOne optional=True
diff --git a/tests/qapi-schema/returns-int.out b/tests/qapi-schema/returns-int.out
index 1ac3e1e..a2da259 100644
--- a/tests/qapi-schema/returns-int.out
+++ b/tests/qapi-schema/returns-int.out
@@ -1,2 +1,3 @@
+object :empty
 command guest-get-time None -> int
    gen=True success_response=True
diff --git a/tests/test-qmp-input-visitor.c b/tests/test-qmp-input-visitor.c
index 9c4bfdf..0f9978b 100644
--- a/tests/test-qmp-input-visitor.c
+++ b/tests/test-qmp-input-visitor.c
@@ -17,6 +17,9 @@
 #include "qapi/qmp-input-visitor.h"
 #include "test-qapi-types.h"
 #include "test-qapi-visit.h"
+#include "test-qmp-introspect.h"
+#include "qmp-introspect.h"
+#include "qapi-visit.h"
 #include "qapi/qmp/types.h"
 
 typedef struct TestInputVisitorData {
@@ -660,6 +663,31 @@ static void test_visitor_in_native_list_number(TestInputVisitorData *data,
     qapi_free_UserDefNativeListUnion(cvalue);
 }
 
+static void do_test_visitor_in_qmp_introspect(TestInputVisitorData *data,
+                                              const char *schema_json)
+{
+    SchemaInfoList *schema = NULL;
+    Error *err = NULL;
+    Visitor *v;
+
+    v = visitor_input_test_init_raw(data, schema_json);
+
+    visit_type_SchemaInfoList(v, &schema, NULL, &err);
+    if (err)
+        fprintf(stderr, "%s", error_get_pretty(err));
+    g_assert(!err);
+    g_assert(schema);
+
+    qapi_free_SchemaInfoList(schema);
+}
+
+static void test_visitor_in_qmp_introspect(TestInputVisitorData *data,
+                                           const void *unused)
+{
+    do_test_visitor_in_qmp_introspect(data, test_qmp_schema_json);
+    do_test_visitor_in_qmp_introspect(data, qmp_schema_json);
+}
+
 static void input_visitor_test_add(const char *testpath,
                                    TestInputVisitorData *data,
                                    void (*test_func)(TestInputVisitorData *data, const void *user_data))
@@ -753,6 +781,9 @@ int main(int argc, char **argv)
     input_visitor_test_add("/visitor/input/native_list/number",
                            &in_visitor_data,
                            test_visitor_in_native_list_number);
+    input_visitor_test_add("/visitor/input/qmp_introspect",
+                           &in_visitor_data,
+                           test_visitor_in_qmp_introspect);
 
     g_test_run();
 
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 46/47] qapi-introspect: Map all integer types to 'int'

[Markus Armbruster]

How many bits we use internally is an implementation detail.  It could
be pressed into external interface service as a very approximate range
information, but that's probably a bad idea.  If we need range
information, we better do it properly.

Reduces output of query-schema by a negligible 0.5 out of 80KiB.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-introspect.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index e7efc4a..961fe88 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -46,6 +46,13 @@ char %(c_name)s[] = "["
         self.used_types = None
 
     def _use_type(self, typ):
+        # Map the various integer types to plain int
+        if typ.json_type() == 'int':
+            typ = self.schema.lookup_type('int')
+        elif isinstance(typ, QAPISchemaArrayType) \
+             and typ.element_type.json_type() == 'int':
+            typ = self.schema.lookup_type('intList')
+        # Add type to work queue if new
         if typ not in self.used_types:
             self.used_types.append(typ)
         return typ.name
-- 
1.9.3

[Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Markus Armbruster]

To eliminate the temptation for clients to look up types by name
(which are not ABI), replace all type names by meaningless strings.

Reduces output of query-schema by 9 out of 80KiB.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi-introspect.py | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 961fe88..efb34ff 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -9,13 +9,18 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+import string
 from qapi import *
 
+def _b32digit(num):
+    return (string.lowercase + string.digits[2:])[num]
+
 class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
     def __init__(self):
         self.schema = None
         self.jsons = None
         self.used_types = None
+        self.name_map = None
         self.defn = None
         self.decl = None
 
@@ -23,13 +28,17 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
         self.schema = schema
         self.jsons = []
         self.used_types = []
+        self.name_map = {}
         return QAPISchemaType   # don't visit types for now
 
     def visit_end(self):
         # visit the types that are actually used
+        jsons = self.jsons
+        self.jsons = []
         for typ in self.used_types:
             typ.visit(self)
         self.jsons.sort()
+        jsons.extend(self.jsons)
         name = prefix + 'qmp_schema_json'
         self.decl = mcgen('''
 extern char %(c_name)s[];
@@ -40,10 +49,19 @@ char %(c_name)s[] = "["
     "%(c_jsons)s]";
 ''',
                           c_name=c_name(name),
-                          c_jsons=', "\n    "'.join(self.jsons))
+                          c_jsons=', "\n    "'.join(jsons))
         self.schema = None
         self.jsons = None
         self.used_types = None
+        self.name_map = None
+
+    def _name(self, name):
+        if name not in self.name_map:
+            n = len(self.name_map)
+            self.name_map[name] = ':' + _b32digit(n / 32 / 32) \
+                                  + _b32digit(n / 32 % 32) \
+                                  + _b32digit(n % 32)
+        return self.name_map[name]
 
     def _use_type(self, typ):
         # Map the various integer types to plain int
@@ -55,9 +73,14 @@ char %(c_name)s[] = "["
         # Add type to work queue if new
         if typ not in self.used_types:
             self.used_types.append(typ)
-        return typ.name
+        # Clients should examine commands and events, not types.  Hide
+        # type names to reduce the temptation.  Also saves a few
+        # characters.
+        return self._name(typ.name)
 
     def _gen_json(self, name, mtype, extra):
+        if mtype != 'command' and mtype != 'event':
+            name = self._name(name)
         self.jsons.append("{ 'name': '%s', 'meta-type': '%s', %s }"
                           % (name, mtype, extra))
 
-- 
1.9.3

Re: [Qemu-devel] [PATCH RFC v2 01/47] qapi: Clarify docs on including the same file multiple times

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> It's idempotent.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  docs/qapi-code-gen.txt | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 02/47] qapi: Clean up cgen() and mcgen()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> Commit 05dfb26 added eatspace stripping to mcgen().  Move it to
> cgen(), just in case somebody gets tempted to use cgen() directly
> instead of via mcgen().
> 
> cgen() indents blank lines.  No such lines get generated right now,
> but fix it anyway.
> 
> We use triple-quoted strings for program text, like this:
> 
>     '''
>     Program text
>     any number of lines
>     '''
> 
> Keeps the program text relatively readable, but puts an extra newline
> at either end.  mcgen() "fixes" that by dropping the first and last
> line outright.  Drop only the newlines.
> 
> This unmasks a bug in qapi-commands.py: four quotes instead of three.
> Fix it up.
> 
> Output doesn't change

Confirmed.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py |  2 +-
>  scripts/qapi.py          | 19 ++++++++++++-------
>  2 files changed, 13 insertions(+), 8 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 03/47] qapi: Simplify guardname()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> The guards around built-in declarations lose their _H.  It never made
> much sense anyway.

Correct - the _H tail on the guard for the overall file makes sense
(when the overall file ends in .h), but for an unrelated use in the
middle of the file, it has no bearing on any file name and does not need
a _H tail.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi.py | 10 +++-------
>  1 file changed, 3 insertions(+), 7 deletions(-)

Reviewed-by: Eric Blake <eblake@redhat.com>

For an example of what it changes, qapi-types.h sees changes like this
(while the original #ifndef QAPI_TYPES_H guard at the front of the file
remains unchanged, which is good):

--- qapi-types.h.old	2015-07-20 10:40:19.513650697 -0600
+++ qapi-types.h	2015-07-20 11:27:15.481152848 -0600
@@ -20,8 +20,8 @@
 #include <stdint.h>


-#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL_H
-#define QAPI_TYPES_BUILTIN_STRUCT_DECL_H
+#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL
+#define QAPI_TYPES_BUILTIN_STRUCT_DECL


 typedef struct int32List
@@ -141,7 +141,7 @@
     struct uint32List *next;
 } uint32List;

-#endif /* QAPI_TYPES_BUILTIN_STRUCT_DECL_H */
+#endif /* QAPI_TYPES_BUILTIN_STRUCT_DECL */


 extern const char * const ErrorClass_lookup[];
...

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 04/47] qapi-event: Clean up how name of enum QAPIEvent is made

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 845 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> Use c_name() instead of ad hoc code.  Doesn't upcase the -p prefix,
> which is an improvement in my book.  Unbreaks prefix containing '.',
> but other funny characters remain broken.  To be fixed next.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-event.py  | 2 +-
>  tests/test-qmp-event.c | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)

No change to the generated qapi-event.[ch], so it looks like only the
testsuite is affected.  [In fact, as far as I can tell, only
docs/qapi-code-gen.txt and tests/Makefile even take advantage of the '-p
prefix' argument.]  Fine by me.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1779 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> qapi-event.py breaks when you ask for a funny prefix like '@'.
> Protect it.

Only possible from the command line (not triggered by our makefiles);
but doesn't hurt.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi.py | 6 ++++++
>  1 file changed, 6 insertions(+)


> 
> diff --git a/scripts/qapi.py b/scripts/qapi.py
> index 2bbc8ff..ea94ce5 100644
> --- a/scripts/qapi.py
> +++ b/scripts/qapi.py
> @@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []):
>      for oa in opts:
>          o, a = oa
>          if o in ("-p", "--prefix"):
> +            match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)

I can understand allowing a leading _, but why bother allowing a leading
'.' or '-'?  Those will get normalized to _, but in all honesty, no one
should ever be doing that.

I'd be just as happy with the shorter:

match = re.match('([A-Za-z_][A-Za-z0-9_.-]*)?', a)

> +            if match.end() != len(a):
> +                print >>sys.stderr, \
> +                    "%s: 'funny character '%s' in argument of -prefix" \

'qemu' is unusual for accepting -single-dash-long-opts; I don't think
python getopts does the same by default.  Please spell this error
message --prefix.

> +                    % (sys.argv[0], a[match.end()])
> +                sys.exit(1)
>              prefix = a
>          elif o in ("-o", "--output-dir"):
>              output_dir = a + "/"
> 

With the second spelling fix, and optionally with the shorter regex,

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

On 07/20/2015 11:57 AM, Eric Blake wrote:
> 'qemu' is unusual for accepting -single-dash-long-opts; I don't think
> python getopts does the same by default.

Or to rephrase, qemu acts as though it uses GNU getopt_long_only() (some
programs like gcc do likewise), but MOST programs that take long opts
behave like they use getopt_long().  It is only the getopt_long_only()
variant that accepts long options with a single dash, when it is
unambiguous with POSIX getopt() parsing of multiple smashed-together
short options.

[well, qemu rolls its own option parser instead of using
getopt_long_only(), but you get the picture]

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 06/47] qapi: Drop unused and useless parameters and variables

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3601 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it
> only as optional argument for push_indent() and pop_indent(), their
> default is four, and gen_sync_call()'s only caller passes four.
> 
> gen_visitor_input_containers_decl()'s parameter obj is always
> "QOBJECT(args)".

It might be nice to call out that several other functions are also
stripped of unused arguments.  I was assuming that only two functions
(and their callers) would be modified, but the patch touched more:

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 27 +++++++++++++--------------
>  scripts/qapi-event.py    |  1 -
>  scripts/qapi-types.py    |  1 -
>  scripts/qapi-visit.py    | 47 ++++++++++++++++++++++-------------------------
>  scripts/qapi.py          |  1 -
>  5 files changed, 35 insertions(+), 42 deletions(-)

> @@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md);
>      pop_indent()
>      return ret.rstrip()
>  
> -def gen_marshal_output(name, args, ret_type, middle_mode):
> +def gen_marshal_output(name, ret_type):
>      if not ret_type:
>          return ""

For example, gen_marshal_output() was not mentioned in the commit message.

>  
> @@ -194,14 +193,14 @@ out:
>  
>      return ret
>  
> -def gen_marshal_input_decl(name, args, ret_type, middle_mode):
> +def gen_marshal_input_decl(name, middle_mode):

Or gen_marshal_input_decl()

> +++ b/scripts/qapi-event.py
> @@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = {
>  ''',
>                  event_enum_name = event_enum_name)
>  
> -    i = 0
>      for string in event_enum_strings:

I guess the subject line covered deletion of unused internal variables.

> +++ b/scripts/qapi-visit.py

> @@ -441,44 +440,42 @@ fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))

>      elif expr.has_key('union'):
>          ret = generate_visit_union(expr)
> -        ret += generate_visit_list(expr['union'], expr['data'])
> +        ret += generate_visit_list(expr['union'])
>          fdef.write(ret)
>  
>          enum_define = discriminator_find_enum_define(expr)
>          ret = ""
>          if not enum_define:
> -            ret = generate_decl_enum('%sKind' % expr['union'],
> -                                     expr['data'].keys())
> -        ret += generate_declaration(expr['union'], expr['data'])
> +            ret = generate_decl_enum('%sKind' % expr['union'])
> +        ret += generate_declaration(expr['union'])

As long as you are touching this, generate_decl_enum(expr['union'] +
'Kind') would read nicer.

>          fdecl.write(ret)
>      elif expr.has_key('alternate'):
>          ret = generate_visit_alternate(expr['alternate'], expr['data'])
> -        ret += generate_visit_list(expr['alternate'], expr['data'])
> +        ret += generate_visit_list(expr['alternate'])
>          fdef.write(ret)
>  
> -        ret = generate_decl_enum('%sKind' % expr['alternate'],
> -                                 expr['data'].keys())
> -        ret += generate_declaration(expr['alternate'], expr['data'])
> +        ret = generate_decl_enum('%sKind' % expr['alternate'])

Same here.

At any rate, no change to generated output.  So assuming you are happy
with the commit message, or take my advice to improve it, the code
cleanup itself is fine.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2807 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> The struct generated for a flat union is weird: the members of its
> base are at the end, except for the union tag, which is renamed to
> 'kind' and put at the beginning.
> 

> Change to put all base members at the beginning, unadulterated.  Not
> only is this easier to understand, it also permits casting the flat
> union to its base, if that should become useful.
> 
> We now generate:
> 
>     struct UserDefFlatUnion
>     {

It might be worth tweaking the generator to output a C comment either
here (at the start of the larger struct)...

>         char *string;
>         EnumOne enum1;

...or here (at the end of the base struct) mentioning that
UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to
make it easier when reading the generated code to trace back to that
"inheritance" relationship.  Right now, there is nothing in the
generated UserDefFlatUnion that points you back to the qapi relationship
of a base class.  But it's not a show-stopper if you don't like my
suggestion.

>         /* union tag is EnumOne enum1 */
>         union {
>             void *data;
>             UserDefA *value1;
>             UserDefB *value2;
>             UserDefB *value3;
>         };
>     };

Only impact in files generated for qemu was to struct BlockdevOptions,
in the files qapi-types.[ch], and I agree that it is an improvement.
Oddly enough, it seems none of the C code cared about the field being
renamed from 'kind' to 'driver' (I guess that's because a lot of our use
of the blockdev code is not directly through the generated C structs,
but through QObject dictionary queries).

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-types.py           | 32 ++++++++++++++++++--------------
>  scripts/qapi-visit.py           |  7 +++++--
>  tests/test-qmp-input-visitor.c  |  2 +-
>  tests/test-qmp-output-visitor.c |  2 +-
>  4 files changed, 25 insertions(+), 18 deletions(-)
> 

> +''',
> +                name=name)
> +    if base:
> +        base_fields = find_struct(base)['data']
> +        ret += generate_struct_fields(base_fields)

> -    if base:
> -        assert discriminator
> -        base_fields = find_struct(base)['data'].copy()
> -        del base_fields[discriminator]
> -        ret += generate_struct_fields(base_fields)

I also like the fact that you no longer modify the base_fields array
(because you are no longer floating a single renamed element
out-of-order compared to the rest of the base), and therefore avoid the
need for a .copy().

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 08/47] qapi-visit: Fix generated code when schema has forward refs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2050 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> The visit_type_implicit_FOO() are generated on demand, right before
> their first use.  Used by visit_type_STRUCT_fields() when STRUCT has
> base FOO, and by visit_type_UNION() when flat UNION has member a FOO.
> 
> If the schema defines FOO after its first use as struct base or flat
> union member, visit_type_implicit_FOO() calls
> visit_type_implicit_FOO() before its definition, which doesn't
> compile.

None of our public qapi .json files currently do this, so no difference
to the generated code used in qemu proper.

> 
> Rearrange qapi-schema-test.json to demonstrate the bug.

Indeed, without testsuite exposure, nothing was relying on this fix.

> 
> Fix by generating the necessary forward declaration.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-visit.py                   | 15 ++++++++++++++-
>  tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++-------------
>  tests/qapi-schema/qapi-schema-test.out  | 10 +++++-----
>  3 files changed, 36 insertions(+), 19 deletions(-)
> 

> +++ b/scripts/qapi-visit.py
> @@ -17,13 +17,23 @@ from qapi import *
>  import re
>  
>  implicit_structs = []
> +struct_fields_seen = set()
>  
>  def generate_visit_implicit_struct(type):
>      global implicit_structs
>      if type in implicit_structs:
>          return ''
>      implicit_structs.append(type)
> -    return mcgen('''
...
> +    ret += mcgen('''

Oddly enough, the ''' is at the same indentation,...

>  
>  static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp)
>  {
> @@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
>  }
>  ''',
>                   c_type=type_name(type))
> +    return ret

so nothing needed to be reindented here :)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 09/47] qapi-visit: Replace list implicit_structs by set

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> Use set because that's what it is.  While there, rename to
> implicit_structs_seen.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-visit.py | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

(I have no idea if that makes python more efficient, but it is cleaner)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 10/47] qapi-visit: Fix two name arguments passed to visitors

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> The generated code passes mangled schema names to visit_type_enum()
> and union's visit_start_struct().  Fix it to pass the names
> unadulterated, like we do everywhere else.
> 
> Only qapi-schema-test.json actually has names where this makes a
> difference: enum __org.qemu_x-Enum, flat union __org.qemu_x-Union2,
> simple union __org.qemu_x-Union1 and its implicit enum
> __org.qemu_x-Union1Kind.

I'm glad I added that test - it covers things that will affect
downstream extensions, but which don't affect mainline.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-visit.py | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 11/47] tests/qapi-schema: Document alternate's enum lacks visit function

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 578 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> We generate a declaration, but no definition.
> 
> The QMP schema has two: Qcow2OverlapChecks and BlockdevRef.  Neither
> visit_type_Qcow2OverlapChecksKind() nor visit_type_BlockdevRefKind()
> is actually used.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  tests/qapi-schema/qapi-schema-test.json | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 12/47] tests/qapi-schema: Document events with with base don't work

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> When event FOO's 'data' is a struct with a base, we consider only the
> struct's direct members, and ignore its base.  The generated
> qapi_event_send_foo() doesn't take arguments for base members.
> 
> No such events currently exist in the QMP schema.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  tests/qapi-schema/qapi-schema-test.json | 3 +++
>  1 file changed, 3 insertions(+)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 13/47] tests/qapi-schema: Restore test case for flat union base bug

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Test case added in commit 2fc0043, and messed up in commit 5223070.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  tests/qapi-schema/qapi-schema-test.json | 5 +++--
>  tests/qapi-schema/qapi-schema-test.out  | 8 ++++----
>  2 files changed, 7 insertions(+), 6 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2442 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> A command's 'data' must be a struct type, given either as a
> dictionary, or as struct type name.
> 
> Existing test case data-int.json covers simple type 'int'.  Add test
> cases for type names referring to union and alternate types.

We could probably relax things to allow a union (which is always a
dictionary on the wire), but I agree that allowing an alternate type is
not appropriate (the goal here is that we require a dictionary).  But
it's also easier to be conservative now and relax later.

> 
> The latter is caught (good), but the former is not (bug).
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  tests/Makefile                        | 3 ++-
>  tests/qapi-schema/args-alternate.err  | 1 +
>  tests/qapi-schema/args-alternate.exit | 1 +
>  tests/qapi-schema/args-alternate.json | 4 ++++
>  tests/qapi-schema/args-alternate.out  | 0
>  tests/qapi-schema/args-union.err      | 0
>  tests/qapi-schema/args-union.exit     | 1 +
>  tests/qapi-schema/args-union.json     | 4 ++++
>  tests/qapi-schema/args-union.out      | 4 ++++
>  9 files changed, 17 insertions(+), 1 deletion(-)

> +++ b/tests/qapi-schema/args-alternate.json
> @@ -0,0 +1,4 @@
> +# we do not allow alternate arguments
> +# TODO should we support this?

I see no reason to allow a non-dictionary in QMP, so this TODO could be
dropped.

> +++ b/tests/qapi-schema/args-union.json
> @@ -0,0 +1,4 @@
> +# FIXME we should reject union arguments
> +# TODO should we support this?
> +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
> +{ 'command': 'oops', 'data': 'Uni' }

This, on the other hand, seems valid from the wire format (it will
always be a dictionary).  I guess the problem is that we generate a C
function signature based by calling out each member of the dictionary -
but how do you do that for a union?  So I see what you are doing:
marking that this test currently passes the parser, but then causes
problems for generating C code, so we should either reject it up front,
or fix the generator.  The FIXME documents what you will do later in the
series (reject it up front) and the TODO documents what we can do down
the road (fix the generator to allow it).

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 15/47] qapi: Fix to reject union command arguments

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1665 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> A command's 'data' must be a struct type, given either as a
> dictionary, or as struct type name.
> 
> Commit dd883c6 tightened the checking there, but not enough: we still
> accept 'union'.  Fix to reject it.

I was thinking that 'all dictionaries are okay' when I wrote that
commit, and obviously missed thinking about 'how would we translate
unions into a C prototype'.

> 
> We may want to support union types there, but we'll have to extend
> qapi-commands.py for it.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi.py                   | 2 +-
>  tests/qapi-schema/args-union.err  | 1 +
>  tests/qapi-schema/args-union.exit | 2 +-
>  tests/qapi-schema/args-union.json | 2 +-
>  tests/qapi-schema/args-union.out  | 4 ----
>  5 files changed, 4 insertions(+), 7 deletions(-)

Reviewed-by: Eric Blake <eblake@redhat.com>

> 
> diff --git a/scripts/qapi.py b/scripts/qapi.py
> index 99e04f6..bebaecc 100644
> --- a/scripts/qapi.py
> +++ b/scripts/qapi.py
> @@ -495,7 +495,7 @@ def check_command(expr, expr_info):
>  
>      check_type(expr_info, "'data' for command '%s'" % name,
>                 expr.get('data'), allow_dict=True, allow_optional=True,
> -               allow_metas=['union', 'struct'], allow_star=allow_star)
> +               allow_metas=['struct'], allow_star=allow_star)

I'm still quite pleased at how my earlier work made this a one-liner fix
- the check_type() interface turned out to be rather pleasant.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 16/47] qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> gen_err_check() hard-codes 'local_err' instead of substituting the
> argument.  Currently harmless, since all callers pass either None or
> 'local_err'.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 17/47] qapi-commands: Inline gen_marshal_output_call()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 374 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 12 ++++--------
>  1 file changed, 4 insertions(+), 8 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 18/47] qapi-commands: Don't feed output of mcgen() to mcgen() again

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Multiple passes through mcgen() is prone to produce unwanted blank
> lines, which we then combat by sprinkling .rstrip() on top.  Just
> don't do it.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 52 +++++++++++++++++++-----------------------------
>  1 file changed, 21 insertions(+), 31 deletions(-)

Reviewed-by: Eric Blake <eblake@redhat.com>

Causes the generated files to be slightly shorter (good), with diffs
like the following:

--- qga/qapi-generated/qga-qmp-marshal.c.old    2015-07-20
17:13:06.056297347 -0600
+++ qga/qapi-generated/qga-qmp-marshal.c        2015-07-21
11:16:09.887228043 -0600
@@ -140,7 +140,6 @@
     (void)args;
     qmp_guest_ping(&local_err);
     error_propagate(errp, local_err);
-
 }

 static void qmp_marshal_output_guest_get_time(int64_t ret_in, QObject
**ret_out, Error **errp)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 4446 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Clean up white-space, brace placement, and superfluous

Incomplete sentence. I bet it's because your editor line-wrapped, and
the rest of your sentence was something like '#ifndef in a .c file.'
(see [1] below), then git ate the line thinking it was a comment.  I
think I'm okay with cramming all of these cleanups into one patch rather
than trying to do one style of cleanup per patch (blank lines, {
placement, and guard cleanup), but the commit message could do a better
job of explaining things.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py |  1 +
>  scripts/qapi-event.py    |  3 +--
>  scripts/qapi-types.py    | 66 +++++++++++++++++++++++-------------------------
>  scripts/qapi-visit.py    |  1 +
>  4 files changed, 34 insertions(+), 37 deletions(-)

Missing changes to docs/qapi-code-gen.txt to reflect the improvements.
Since having docs that are stale compared to implementation can be
misleading, I'd like to wait for v2 before giving my R-b; but see also
[2] below.

Overall, I'm a fan of the cleanups.

I'm going to intersperse diffs of the generated files that were caused
by some of these changes:

> 
> diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
> index cfbd59c..223216d 100644
> --- a/scripts/qapi-commands.py
> +++ b/scripts/qapi-commands.py
> @@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
>          ret += mcgen('''
>  
>      (void)args;
> +
>  ''')

This and similar hunts avoids extra blank lines.  Not worth showing a
diff to the generated files.

>  
>      ret += gen_sync_call(name, args, ret_type)
> diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
> index 88b0620..7f238df 100644
> --- a/scripts/qapi-event.py
> +++ b/scripts/qapi-event.py
> @@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[];
>                          event_enum_name = event_enum_name)
>  
>      enum_decl = mcgen('''
> -typedef enum %(event_enum_name)s
> -{
> +typedef enum %(event_enum_name)s {

Several hunks like this; gets us generally closer to qemu style; with
generated diffs like:

qapi-types.h:
-typedef struct int32List
-{
+typedef struct int32List {
     union {
         int32_t value;
         uint64_t padding;

> @@ -105,7 +103,8 @@ struct %(name)s
>  
>  def generate_enum_lookup(name, values):
>      ret = mcgen('''
> -const char * const %(name)s_lookup[] = {
> +
> +const char *const %(name)s_lookup[] = {

[2] generated diffs like this:

qapi-types.c:
-const char * const OnOffAuto_lookup[] = {
+const char *const OnOffAuto_lookup[] = {

Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we
added a const in commit 2e4450ff that is missing from the documentation.

> @@ -335,22 +333,22 @@ for typename in builtin_types.keys():
>  fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
>  
>  for expr in exprs:
> -    ret = "\n"
> +    ret = ""
>      if expr.has_key('struct'):
>          ret += generate_fwd_struct(expr['struct'])
>      elif expr.has_key('enum'):
> -        ret += generate_enum(expr['enum'], expr['data']) + "\n"
> +        ret += generate_enum(expr['enum'], expr['data'])
>          ret += generate_fwd_enum_struct(expr['enum'])
>          fdef.write(generate_enum_lookup(expr['enum'], expr['data']))

More work at avoiding extra blank lines.

> @@ -370,34 +368,32 @@ fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
>  # have the functions defined, so we use -b option to provide control
>  # over these cases
>  if do_builtins:
> -    fdef.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))
>      for typename in builtin_types.keys():
>          fdef.write(generate_type_cleanup(typename + "List"))
> -    fdef.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))

[1] this is the hunk whose description got corrupted in your commit
message. It gives a diff like this:

qapi-types.c:

-
-#ifndef QAPI_TYPES_BUILTIN_CLEANUP_DEF
-#define QAPI_TYPES_BUILTIN_CLEANUP_DEF
-
-
 void qapi_free_int32List(int32List *obj)

We conditionally declare the functions in the .h, but the .c is not
compiled multiple times, so there is no need for a header-style guard.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 20/47] qapi: Rename class QAPISchema to QAPISchemaParser

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> I want to name a new class QAPISchema.
> 
> While there, make it a new-style class.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi.py | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 20179 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> The QAPI code generators work with a syntax tree (nested dictionaries)
> plus a few symbol tables (also dictionaties) on the side.

s/dictionaties/dictionaries/

> 
> They have clearly outgrown these simple data structures.  There's lots
> of rummaging around in dictionaries, and information is recomputed on
> the fly.  For the work I'm going to do, I want more clearly defined
> and more convenient interfaces.
> 
> Going forward, I also want less coupling between the back-ends and the
> syntax tree, to make messing with the syntax easier.

Indeed - should be a lot easier to add new qapi .json syntax sugar on
the front-end, or to tweak generated C layout on the backend, without
close coupling between the two.  Particularly when adding front-end
sugar extensions that still normalize into the same backend structs.

> 
> Create a bunch of classes to represent QAPI schemata.
> 
> Have the QAPISchema initializer call the parser, then walk the syntax
> tree to create the new internal representation, and finally perform
> semantic analysis.
> 
> Shortcut: the semantic analysis still relies on existing check_exprs()
> to do the actual semantic checking.  All this code needs to move into
> the classes.  Mark as TODO.
> 
> We generate array types eagerly, even though most of them aren't used.
> Mark as TODO.
> 

No change to generated files at this stage in the game (this is mostly
additive, then later patches shed their old ways of doing things by
using this).  Good division of labor between patches.

> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py       |   2 +-
>  scripts/qapi-event.py          |   2 +-
>  scripts/qapi-types.py          |   2 +-
>  scripts/qapi-visit.py          |   2 +-
>  scripts/qapi.py                | 351 +++++++++++++++++++++++++++++++++++++++--
>  tests/qapi-schema/test-qapi.py |   2 +-
>  6 files changed, 347 insertions(+), 14 deletions(-)
> 

> +++ b/scripts/qapi.py

> +
> +#
> +# Schema compiler frontend
> +#
> +
> +class QAPISchemaEntity(object):
> +    def __init__(self, name, info):
> +        assert isinstance(name, str)
> +        self.name = name
> +        self.info = info
> +    def check(self, schema):
> +        pass
> +
> +class QAPISchemaType(QAPISchemaEntity):
> +    pass
> +
> +class QAPISchemaBuiltinType(QAPISchemaType):
> +    def __init__(self, name):
> +        QAPISchemaType.__init__(self, name, None)
> +
> +class QAPISchemaEnumType(QAPISchemaType):
> +    def __init__(self, name, info, values):
> +        QAPISchemaType.__init__(self, name, info)
> +        for v in values:
> +            assert isinstance(v, str)
> +        self.values = values

worth a check() method to ensure values don't collide?  Especially since
you already do that in QAPISchemaObjectTypeMember.check()

> +
> +class QAPISchemaArrayType(QAPISchemaType):
> +    def __init__(self, name, info, element_type):
> +        QAPISchemaType.__init__(self, name, info)
> +        assert isinstance(element_type, str)
> +        self.element_type_name = element_type
> +        self.element_type = None
> +    def check(self, schema):
> +        self.element_type = schema.lookup_type(self.element_type_name)
> +        assert self.element_type
> +
> +class QAPISchemaObjectType(QAPISchemaType):
> +    def __init__(self, name, info, base, local_members, variants):
> +        QAPISchemaType.__init__(self, name, info)
> +        assert base == None or isinstance(base, str)
> +        for m in local_members:
> +            assert isinstance(m, QAPISchemaObjectTypeMember)
> +        if variants != None:
> +            assert isinstance(variants, QAPISchemaObjectTypeVariants)
> +        self.base_name = base
> +        self.base = None
> +        self.local_members = local_members
> +        self.variants = variants
> +        self.members = None
> +    def check(self, schema):
> +        if self.members:
> +            return                      # already checked
> +        assert self.members == None     # not running in cycles
> +        self.members = False            # mark as being checked

Interesting, but makes sense (since we allow self-referential nesting,
populating our own members may require revisiting the same type).  Cute
that python allows both None and False as distinct non-true settings, so
that you end up using the variable as a tri-state.

> +        if self.base_name:
> +            self.base = schema.lookup_type(self.base_name)
> +            self.base.check(schema)

Do you need 'assert self.base' here, similar to how you did it in
QAPISchemaArrayType.check()?  I guess you'll still get a python
exception that None.check() doesn't exist if the lookup_type failed, so
it just depends on what error message you want.

> +            members = list(self.base.members)

For that matter, do you want to assert isinstance(self.base,
QAPISchemaObjectType), since lookup_type can return other types, but
other child classes of QAPISchemaType do not have a .members?

> +        else:
> +            members = []
> +        seen = {}
> +        for m in members:
> +            seen[m.name] = m
> +        for m in self.local_members:
> +            m.check(schema, members, seen)
> +        if self.variants:
> +            self.variants.check(schema, members, seen)
> +        self.members = members
> +
> +class QAPISchemaObjectTypeMember(object):

Is 'object' the right base class, or should it be QAPISchemaEntity?

> +    def __init__(self, name, typ, optional):
> +        assert isinstance(name, str)
> +        assert isinstance(typ, str)

I guess that means all callers flatten array types into an unambiguous
string first.

> +        assert isinstance(optional, bool)
> +        self.name = name
> +        self.type_name = typ
> +        self.type = None
> +        self.optional = optional
> +    def check(self, schema, all_members, seen):
> +        assert self.name not in seen
> +        self.type = schema.lookup_type(self.type_name)
> +        assert self.type
> +        all_members.append(self)
> +        seen[self.name] = self
> +
> +class QAPISchemaObjectTypeVariants(object):

Is 'object' the right base class, or should it be QAPISchemaEntity?

> +    def __init__(self, tag_name, tag_enum, variants):
> +        assert tag_name == None or isinstance(tag_name, str)
> +        assert tag_enum == None or isinstance(tag_enum, str)
> +        for v in variants:
> +            assert isinstance(v, QAPISchemaObjectTypeVariant)
> +        self.tag_name = tag_name
> +        if tag_name:
> +            assert not tag_enum
> +            self.tag_member = None
> +        else:
> +            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
> +                                                         False)

Nice - the tag is either part of the base struct (flat union) or
implicitly created (plain union).  By the time we are done with check(),
we then have both its type (possibly generated implicitly) and its name.

> +        self.variants = variants
> +    def check(self, schema, members, seen):
> +        if self.tag_name:
> +            self.tag_member = seen[self.tag_name]
> +        else:
> +            self.tag_member.check(schema, members, seen)

A little bit tricky here in that flat unions (tag_name was set) must
find the tag from the members already loaded from the base class, while
plain unions (tag_name is None) add their own implicit member into the
union.  But it works.

> +        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
> +        for v in self.variants:
> +            vseen = dict(seen)
> +            v.check(schema, self.tag_member.type, vseen)
> +
> +class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
> +    def __init__(self, name, typ, flat):
> +        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
> +        assert isinstance(flat, bool)
> +        self.flat = flat
> +    def check(self, schema, tag_type, seen):
> +        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
> +        assert self.name in tag_type.values
> +        if self.flat:
> +            self.type.check(schema)
> +            assert isinstance(self.type, QAPISchemaObjectType)

Do we really want to be tracking self.flat for each variant?  After all,
docs/qapi-code-gen.txt already describes the mapping from simple union
into flat union (as if the flat union had a base class with single
member 'kind' of the right type, then each branch of the union composed
of an implicit object with a lone member 'data' of the correct type).
In other words, is it any better to just normalize into that form now,
such that each QAPISchemaObjectTypeVariant is merely a (often
one-element) list of name:type members being added to the overall
QAPISchemaObject?  But I guess it remains to be seen how you use
self.flat before knowing if it is worth normalizing away from it.

> +
> +class QAPISchemaAlternateType(QAPISchemaType):
> +    def __init__(self, name, info, variants):
> +        QAPISchemaType.__init__(self, name, info)
> +        assert isinstance(variants, QAPISchemaObjectTypeVariants)
> +        assert not variants.tag_name
> +        for v in variants.variants:
> +            assert not v.flat

Hmm. You are using .flat after all.

> +        self.variants = variants
> +    def check(self, schema):
> +        self.variants.check(schema, [], {})
> +
> +class QAPISchemaCommand(QAPISchemaEntity):
> +    def __init__(self, name, info, args, rets, gen, success_response):
> +        QAPISchemaEntity.__init__(self, name, info)
> +        assert not args or isinstance(args, str)
> +        assert not rets or isinstance(rets, str)

Again, this means that the caller has already had to convert qapi
dictionaries into an implicit type and pass in that type name here.
Works for me.

> +        self.args_name = args
> +        self.args = None
> +        self.rets_name = rets
> +        self.rets = None
> +        self.gen = gen
> +        self.success_response = success_response
> +    def check(self, schema):
> +        if self.args_name:
> +            self.args = schema.lookup_type(self.args_name)
> +            assert isinstance(self.args, QAPISchemaObjectType)
> +            assert not self.args.variants # not implemented
> +            self.args.check(schema)
> +        if self.rets_name:
> +            self.rets = schema.lookup_type(self.rets_name)
> +            assert isinstance(self.rets, QAPISchemaType)
> +            self.rets.check(schema)

Matches our existing difference in requiring a dictionary for arguments,
but allowing the return of any type (although we now have a whitelist to
prevent too many additions of non-dictionaries).

> +
> +class QAPISchemaEvent(QAPISchemaEntity):
> +    def __init__(self, name, info, data):
> +        QAPISchemaEntity.__init__(self, name, info)
> +        assert not data or isinstance(data, str)
> +        self.data_name = data
> +        self.data = None
> +    def check(self, schema):
> +        if self.data_name:
> +            self.data = schema.lookup_type(self.data_name)
> +            assert isinstance(self.data, QAPISchemaObjectType)
> +            self.data.check(schema)
> +
> +class QAPISchema(object):
> +    def __init__(self, fname):
> +        try:
> +            self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
> +        except (QAPISchemaError, QAPIExprError), err:
> +            print >>sys.stderr, err
> +            exit(1)
> +        self.entity_dict = {}
> +        self._def_predefineds()
> +        self._def_exprs()
> +        self.check()
> +
> +    def get_exprs(self):
> +        return [expr_elem['expr'] for expr_elem in self.exprs]
> +
> +    def _def_entity(self, ent):
> +        assert ent.name not in self.entity_dict
> +        self.entity_dict[ent.name] = ent
> +
> +    def lookup_entity(self, name, typ=None):
> +        ent = self.entity_dict.get(name)
> +        if typ and not isinstance(ent, typ):
> +            return None
> +        return ent

Ah, so you enforce a shared namespace between types, commands, and
events (all three are in self.entity_dict), but can use the typ
parameter to allow limiting a lookup to just types.

> +
> +    def lookup_type(self, name):
> +        return self.lookup_entity(name, QAPISchemaType)
> +
> +    def _def_builtin_type(self, name):
> +        self._def_entity(QAPISchemaBuiltinType(name))
> +        if name != '**':
> +            self._make_array_type(name) # TODO really needed?
> +
> +    def _def_predefineds(self):
> +        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
> +                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
> +            self._def_builtin_type(t)
> +
> +    def _make_implicit_enum_type(self, name, values):
> +        name = name + 'Kind'
> +        self._def_entity(QAPISchemaEnumType(name, None, values))
> +        return name
> +
> +    def _make_array_type(self, element_type):
> +        name = element_type + 'List'
> +        if not self.lookup_type(name):
> +            self._def_entity(QAPISchemaArrayType(name, None, element_type))
> +        return name
> +
> +    def _make_implicit_object_type(self, name, role, members):
> +        name = ':obj-%s-%s' % (name, role)
> +        self._def_entity(QAPISchemaObjectType(name, None, None, members, None))
> +        return name

We may extend the qapi .json syntax to allow anonymous types in more
places, but so far, this matches that all existing use of anonymous
types are for structs, not unions.

> +
> +    def _def_enum_type(self, expr, info):
> +        name = expr['enum']
> +        data = expr['data']
> +        self._def_entity(QAPISchemaEnumType(name, info, data))
> +        self._make_array_type(name) # TODO really needed?
> +
> +    def _make_member(self, name, typ):
> +        optional = False
> +        if name.startswith('*'):
> +            name = name[1:]
> +            optional = True
> +        if isinstance(typ, list):
> +            assert len(typ) == 1
> +            typ = self._make_array_type(typ[0])
> +        return QAPISchemaObjectTypeMember(name, typ, optional)
> +
> +    def _make_members(self, data):
> +        return [self._make_member(key, data[key]) for key in data.keys()]
> +
> +    def _def_struct_type(self, expr, info):
> +        name = expr['struct']
> +        base = expr.get('base')
> +        data = expr['data']
> +        self._def_entity(QAPISchemaObjectType(name, info, base,
> +                                              self._make_members(data),
> +                                              None))
> +        self._make_array_type(name) # TODO really needed?
> +
> +    def _make_flat_variant(self, case, typ):
> +        return QAPISchemaObjectTypeVariant(case, typ, True)
> +
> +    def _make_flat_variants(self, tag_name, data):
> +        variants = [self._make_flat_variant(key, data[key])
> +                    for key in data.keys()]
> +        return QAPISchemaObjectTypeVariants(tag_name, None, variants)
> +
> +    def _make_simple_variant(self, case, typ):
> +        if isinstance(typ, list):
> +            assert len(typ) == 1
> +            typ = self._make_array_type(typ[0])
> +        return QAPISchemaObjectTypeVariant(case, typ, False)
> +
> +    def _make_simple_variants(self, type_name, data):
> +        variants = [self._make_simple_variant(key, data[key])
> +                    for key in data.keys()]
> +        enum = self._make_implicit_enum_type(type_name,
> +                                             [v.name for v in variants])
> +        return QAPISchemaObjectTypeVariants(None, enum, variants)

Again, I wonder if normalizing simple unions into flat unions (to get
rid of the need for tracking .flat) would make later use any easier, or
if it would just make introspection QMP output more verbose.

> +
> +    def _def_union_type(self, expr, info):
> +        name = expr['union']
> +        data = expr['data']
> +        tag_name = expr.get('discriminator')
> +        if tag_name:
> +            base = expr['base']
> +            variants = self._make_flat_variants(tag_name, data)
> +        else:
> +            base = None
> +            variants = self._make_simple_variants(name, data)
> +        self._def_entity(QAPISchemaObjectType(name, info, base,
> +                                              self._make_members(OrderedDict()),
> +                                              variants))
> +        self._make_array_type(name) # TODO really needed?
> +
> +    def _def_alternate_type(self, expr, info):
> +        name = expr['alternate']
> +        data = expr['data']
> +        self._def_entity(QAPISchemaAlternateType(name, info,
> +                                    self._make_simple_variants(name, data)))
> +        self._make_array_type(name) # TODO really needed?
> +
> +    def _def_command(self, expr, info):
> +        name = expr['command']
> +        args = expr.get('data')
> +        rets = expr.get('returns')
> +        gen = expr.get('gen', True)
> +        success_response = expr.get('success-response', True)
> +        if args and not isinstance(args, str):
> +            args = self._make_implicit_object_type(name, 'args',
> +                                                   self._make_members(args))

If I write { 'command':'Foo', 'data':{} }, does that try and create an
implicit type, with no members?  I'm wondering if it is any simpler to
write args = expr.get('data', {}), and have _make_implicit_object_type()
do the right thing when presented with an empty list of members.

> +        if rets and isinstance(rets, list):
> +            assert len(rets) == 1
> +            rets = self._make_array_type(rets[0])
> +        elif rets and not isinstance(rets, str):
> +            rets = self._make_implicit_object_type(name, 'rets',
> +                                                   self._make_members(rets))
> +        self._def_entity(QAPISchemaCommand(name, info, args, rets, gen,
> +                                           success_response))
> +
> +    def _def_event(self, expr, info):
> +        name = expr['event']
> +        data = expr.get('data')
> +        if data and not isinstance(data, str):
> +            data = self._make_implicit_object_type(name, 'data',
> +                                                   self._make_members(data))
> +        self._def_entity(QAPISchemaEvent(name, info, data))
> +
> +    def _def_exprs(self):
> +        for expr_elem in self.exprs:
> +            expr = expr_elem['expr']
> +            info = expr_elem['info']
> +            if 'enum' in expr:
> +                self._def_enum_type(expr, info)
> +            elif 'struct' in expr:
> +                self._def_struct_type(expr, info)
> +            elif 'union' in expr:
> +                self._def_union_type(expr, info)
> +            elif 'alternate' in expr:
> +                self._def_alternate_type(expr, info)
> +            elif 'command' in expr:
> +                self._def_command(expr, info)
> +            elif 'event' in expr:
> +                self._def_event(expr, info)
> +            else:
> +                assert False
> +
> +    def check(self):
> +        for ent in self.entity_dict.values():
> +            ent.check(self)

Overall seems like a very nice layout.

I'll reserve R-b until I see how the rest of the series uses this object
hierarchy, but it looks like you won't have to change very much of this
patch when you do your next spin of the series.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 6706 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> New methods c_name(), c_type(), c_null(), json_type(),
> alternate_qtype().
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------
>  1 file changed, 65 insertions(+), 7 deletions(-)

Still unused in the rest of the code, so no change to generated code
yet; and thanks for separating this apart from the creation of the new
object hierarchy.

>  class QAPISchemaType(QAPISchemaEntity):
> -    pass
> +    def c_type(self, is_param=False):
> +        return c_name(self.name) + pointer_suffix

At first, I thought is_param was unused; but reviewing the rest of the
patch shows that subclasses are overriding it, so declaring it here is
necessary for polymorphic calls to supply the argument and have it
properly acted on.

> +    def c_null(self):
> +        return 'NULL'
> +    def json_type(self):
> +        return None
> +    def alternate_qtype(self):
> +        json2qtype = {
> +            'string':  'QTYPE_QSTRING',
> +            'number':  'QTYPE_QFLOAT',
> +            'int':     'QTYPE_QINT',
> +            'boolean': 'QTYPE_QBOOL',
> +            'object':  'QTYPE_QDICT'
> +        }

Are there any style rules on whether to include or omit a trailing comma?

> +        return json2qtype.get(self.json_type())
>  
>  class QAPISchemaBuiltinType(QAPISchemaType):
> -    def __init__(self, name):
> +    def __init__(self, name, json_type, c_type, c_null):
>          QAPISchemaType.__init__(self, name, None)
> +        assert not c_type or isinstance(c_type, str)
> +        assert json_type in ('string', 'number', 'int', 'boolean', 'null',
> +                             'value')
> +        self.json_type_name = json_type
> +        self.c_type_name = c_type
> +        self.c_null_val = c_null
> +    def c_name(self):
> +        return self.name

At first, I thought this was redundant with QAPISchemaEntity.c_name(),
until I realized that you are intentionally overriding things here to
NOT call global c_name() (so that 'int' does not get munged the 'q_int'
in the generated C code).  Nice, once I figured it out.

> +    def c_type(self, is_param=False):
> +        if is_param and self.name == 'str':
> +            return 'const ' + self.c_type_name
> +        return self.c_type_name
> +    def c_null(self):
> +        return self.c_null_val
> +    def json_type(self):
> +        return self.json_type_name
>  
>  class QAPISchemaEnumType(QAPISchemaType):
>      def __init__(self, name, info, values):
> @@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
>          for v in values:
>              assert isinstance(v, str)
>          self.values = values
> +    def c_type(self, is_param=False):
> +        return c_name(self.name)

Probably showing my unfamiliarity with python polymorphism/overriding
rules - is it necessary to declare an otherwise-unused optional
parameter here if we are overriding a base method where the parameter
has a sane default for our needs?  That is, would c_type(self) here
properly override c_type(self, is_param=False) in the parent class?

> +    def c_null(self):
> +        return c_enum_const(self.name, self.values[0])
> +    def json_type(self):
> +        return 'string'
>  
>  class QAPISchemaArrayType(QAPISchemaType):
>      def __init__(self, name, info, element_type):
> @@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType):
>          if self.variants:
>              self.variants.check(schema, members, seen)
>          self.members = members
> +    def c_name(self):
> +        assert self.info
> +        return QAPISchemaType.c_name(self)
> +    def c_type(self, is_param=False):
> +        assert self.info
> +        return QAPISchemaType.c_type(self)

So, to make sure I understand correctly, implicit types (those generated
from anonymous dictionaries in a command or event) have info=None, and
the assertions here are making sure we never try to learn the c_name or
c_type of those implicit types in generated code, but rather use them
solely in generated code that just enumerates the members (C struct
declarations, parameter list to functions that implement commands).

> +    def json_type(self):
> +        return 'object'
>  
>  class QAPISchemaObjectTypeMember(object):
>      def __init__(self, name, typ, optional):
> @@ -948,15 +994,27 @@ class QAPISchema(object):
>      def lookup_type(self, name):
>          return self.lookup_entity(name, QAPISchemaType)
>  
> -    def _def_builtin_type(self, name):
> -        self._def_entity(QAPISchemaBuiltinType(name))
> +    def _def_builtin_type(self, name, json_type, c_type, c_null):
> +        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
>          if name != '**':
>              self._make_array_type(name) # TODO really needed?
>  
>      def _def_predefineds(self):
> -        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
> -                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
> -            self._def_builtin_type(t)
> +        for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
> +                  ('number', 'number',  'double',   '0'),

Do you want '0.0' as the default here (yes, promoting an int 0 to double
does the right thing, but I like making it obvious that I know I'm using
floating point).

> +                  ('int',    'int',     'int64_t',  '0'),
> +                  ('int8',   'int',     'int8_t',   '0'),
> +                  ('int16',  'int',     'int16_t',  '0'),
> +                  ('int32',  'int',     'int32_t',  '0'),
> +                  ('int64',  'int',     'int64_t',  '0'),
> +                  ('uint8',  'int',     'uint8_t',  '0'),
> +                  ('uint16', 'int',     'uint16_t', '0'),
> +                  ('uint32', 'int',     'uint32_t', '0'),
> +                  ('uint64', 'int',     'uint64_t', '0'),
> +                  ('size',   'int',     'uint64_t', '0'),
> +                  ('bool',   'boolean', 'bool',     'false'),
> +                  ('**',     'value',   None,       None)]:
> +            self._def_builtin_type(*t)
>  
>      def _make_implicit_enum_type(self, name, values):
>          name = name + 'Kind'
> 

Tweaks you make (if any) based on the above comments should be minor, so
I'm fine with adding:

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 23/47] qapi: New QAPISchemaVisitor

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 581 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> The visitor will help keeping the code generation code simple and
> reasonably separated from QAPISchema details.

Still no change to generated code until later patches start using this.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi.py | 44 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 24/47] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 7526 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> The old code prints the result of parsing (list of expression
> dictionaries), and partial results of semantic analysis (list of enum
> dictionaries, list of struct dictionaries).
> 
> The new code prints a trace of a schema visit, i.e. what the back-ends
> are going to use.  Built-in and array types are omitted, because
> they're boring.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  tests/qapi-schema/alternate-good.out            |  15 +-
>  tests/qapi-schema/comments.out                  |   4 +-
>  tests/qapi-schema/data-member-array.out         |  13 +-
>  tests/qapi-schema/empty.out                     |   3 -
>  tests/qapi-schema/enum-empty.out                |   4 +-
>  tests/qapi-schema/event-case.out                |   4 +-
>  tests/qapi-schema/flat-union-reverse-define.out |  21 +--
>  tests/qapi-schema/ident-with-escape.out         |   7 +-
>  tests/qapi-schema/include-relpath.out           |   4 +-
>  tests/qapi-schema/include-repetition.out        |   4 +-
>  tests/qapi-schema/include-simple.out            |   4 +-
>  tests/qapi-schema/indented-expr.out             |   7 +-
>  tests/qapi-schema/qapi-schema-test.out          | 186 +++++++++++++++++-------
>  tests/qapi-schema/returns-int.out               |   5 +-
>  tests/qapi-schema/test-qapi.py                  |  37 ++++-
>  tests/qapi-schema/type-bypass.out               |   7 +-
>  16 files changed, 210 insertions(+), 115 deletions(-)

We have a lot more negative than positive tests of the parser (good
thing, because that meant fewer .out files to update to the new format).

No change to actual qemu code, and proves that the previous three
patches have set up enough of a framework to accurately cover our testsuite.

> 
> diff --git a/tests/qapi-schema/alternate-good.out b/tests/qapi-schema/alternate-good.out
> index 99848ee..0cbdfa1 100644
> --- a/tests/qapi-schema/alternate-good.out
> +++ b/tests/qapi-schema/alternate-good.out
> @@ -1,6 +1,9 @@
> -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))]),
> - OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]),
> - OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value', 'int'), ('string', 'Enum'), ('struct', 'Data')]))])]
> -[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']},
> - {'enum_name': 'AltKind', 'enum_values': None}]
> -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))])]
> +alternate Alt
> +    case value: int flat=False
> +    case string: Enum flat=False
> +    case struct: Data flat=False

I'm still not convinced whether we need .flat exposed through this much
detail, or if we should just normalize plain unions into flat unions
with implicit structs for each branch.  Changing your design will have
obvious ripple effects here.

> +++ b/tests/qapi-schema/data-member-array.out
> @@ -1,5 +1,8 @@
> -[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]),
> - OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))]),
> - OrderedDict([('command', 'okay'), ('data', OrderedDict([('member1', ['int']), ('member2', ['def'])]))])]
> -[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}]
> -[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))])]
> +object :obj-okay-args
> +    member member1: intList optional=False

Took me a moment to realize the object is an implicit one (named
':obj-okay-args') and not a typo for 'object: obj-okay-args' consistent
with members being listed 'name: type'.  But not worth changing things,
as it is sufficiently unambiguous to serve as a valid test.

> +object UserDefFlatUnion
> +    base UserDefUnionBase
> +    tag enum1
> +    case value1: UserDefA flat=True
> +    case value2: UserDefB flat=True
> +    case value3: UserDefB flat=True
> +object UserDefFlatUnion2
> +    base UserDefUnionBase
> +    tag enum1
> +    case value1: UserDefC flat=True
> +    case value2: UserDefB flat=True
> +    case value3: UserDefA flat=True
> +object UserDefNativeListUnion
> +    case integer: intList flat=False
> +    case s8: int8List flat=False
> +    case s16: int16List flat=False
> +    case s32: int32List flat=False
> +    case s64: int64List flat=False
> +    case u8: uint8List flat=False
> +    case u16: uint16List flat=False
> +    case u32: uint32List flat=False
> +    case u64: uint64List flat=False
> +    case number: numberList flat=False
> +    case boolean: boolList flat=False
> +    case string: strList flat=False
> +    case sizes: sizeList flat=False
> +enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes']

Hmm. You are dumping the tag name and type of flat unions, but not of
simple unions.  I would have expected:

object UserDefNativeListUnion
    member kind: UserDefNativeListUnionKind
    tag kind
    case integer: intList flat=False
...

The above was fallout, while below is the meat of the new visitor.

> +++ b/tests/qapi-schema/test-qapi.py
> @@ -15,11 +15,34 @@ from pprint import pprint
>  import os
>  import sys
>  
> -try:
> -    exprs = QAPISchema(sys.argv[1]).get_exprs()
> -except SystemExit:
> -    raise
> +class QAPISchemaTestVisitor(QAPISchemaVisitor):
> +    def visit_enum_type(self, name, info, values):
> +        print 'enum %s %s' % (name, values)
> +    def visit_object_type(self, name, info, base, members, variants):
> +        print 'object %s' % name
> +        if base:
> +            print '    base %s' % base.name
> +        for m in members:
> +            print '    member %s: %s optional=%s' % (m.name, m.type.name,
> +                                                     m.optional)
> +        self._print_variants(variants)

So here is where information was missing when visiting a simple union.
Why didn't 'kind' get injected into members?  And...

> +    def visit_alternate_type(self, name, info, variants):
> +        print 'alternate %s' % name
> +        self._print_variants(variants)
> +    def visit_command(self, name, info, args, rets, gen, success_response):
> +        print 'command %s %s -> %s' % (name, (args and args.name),
> +                                       (rets and rets.name))
> +        print '   gen=%s success_response=%s' % (gen, success_response)
> +    def visit_event(self, name, info, data):
> +        print 'event %s %s' % (name, data and data.name)
>  
> -pprint(exprs)
> -pprint(enum_types)
> -pprint(struct_types)
> +    @staticmethod
> +    def _print_variants(variants):
> +        if variants:
> +            if variants.tag_name:
> +                print '    tag %s' % variants.tag_name

...shouldn't a simple union report 'kind' as its tag_name?

> +            for v in variants.variants:
> +                print '    case %s: %s flat=%s' % (v.name, v.type.name, v.flat)
> +
> +schema = QAPISchema(sys.argv[1])
> +schema.visit(QAPISchemaTestVisitor())
> diff --git a/tests/qapi-schema/type-bypass.out b/tests/qapi-schema/type-bypass.out

Overall, fairly slick, but I have enough questions about the
representation of a simple union that I'll wait for the non-RFC respin
to see if you change any design.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 25/47] qapi: Make generators work on sorted schema expressions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3203 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Order of expressions doesn't matter.  QAPISchema().get_exprs() returns
> expressions in the order they're parsed.
> 
> I'm going to refactor this code.  To check refactorings, I want to
> diff the generated code, and for that I need to preserve its order.
> 
> Since I don't feel like preserving parse order, I'm changing
> get_expr() to return the expressions sorted by name.  This order will
> be trivial to preserve.  It also makes the generated code slightly
> easier to navigate.

Huge change to the generated files, but that's to be expected.  Diffstat
shows it is not a straight 1:1 reshuffle:

 qapi-event.c                          |  890 +--
 qapi-event.h                          |  190
 qapi-types.c                          | 1996 +++----
 qapi-types.h                          | 5420 ++++++++++----------
 qapi-visit.c                          | 9088
+++++++++++++++++-----------------
 qapi-visit.h                          |  746 +-
 qga/qapi-generated/qga-qapi-types.c   |  148
 qga/qapi-generated/qga-qapi-types.h   |  340 -
 qga/qapi-generated/qga-qapi-visit.c   |  446 -
 qga/qapi-generated/qga-qapi-visit.h   |   54
 qga/qapi-generated/qga-qmp-commands.h |   38
 qga/qapi-generated/qga-qmp-marshal.c  |  864 +--
 qmp-commands.h                        |  376 -
 13 files changed, 10308 insertions(+), 10288 deletions(-)

but that's probably because you now emit forward declarations for things
that occur alphabetically after their first use (since 8/47 in this
series touched forward declarations), whereas before they happened to
occurr in topological order from the parse.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi.py | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/qapi.py b/scripts/qapi.py
> index cac7ab5..20ffdaf 100644
> --- a/scripts/qapi.py
> +++ b/scripts/qapi.py
> @@ -1017,7 +1017,14 @@ class QAPISchema(object):
>          self.check()
>  
>      def get_exprs(self):
> -        return [expr_elem['expr'] for expr_elem in self.exprs]
> +        def expr_name(expr):
> +            name = expr.get('enum') or expr.get('union') \
> +                   or expr.get('alternate') or expr.get('struct') \
> +                   or expr.get('command') or expr.get('event')
> +            assert name
> +            return name

When I was working on this file earlier, I half toyed with the idea of
adding expr_elem['name'] holding the entity name, and expr_elem['meta']
holding what meta-type the entity represents; it might have made some of
our later 6-way switches simpler.  But with your hierarchy of actual
objects, I'm not sure my idea helps any more.

> +        return sorted([expr_elem['expr'] for expr_elem in self.exprs],
> +                      key=expr_name)

Python has some nice compact syntactical gems - I'd hate the amount of
boilerplate required to write this same filter using straight C code :)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 13158 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Fixes flat unions to get the base's base members.  Test case is from
> commit 2fc0043, in qapi-schema-test.json:
> 

> 
> Flat union visitors remain broken.  They'll be fixed next.

Sadly, the generated files had a huge diffstat, making it very hard to
determine if this change is sane:

 qapi-types.c                        | 1412 ++++++-------
 qapi-types.h                        | 3705
++++++++++++++++++++----------------
 qga/qapi-generated/qga-qapi-types.c |  110 -
 qga/qapi-generated/qga-qapi-types.h |  542 +++--
 4 files changed, 3208 insertions(+), 2561 deletions(-)

At first, it looks easy, as in:

qapi-types.c:
+const int BlockdevRef_qtypes[QTYPE_MAX] = {
...
 const char *const BlockdevRefKind_lookup[] = {
...
-const int BlockdevRef_qtypes[QTYPE_MAX] = {
...

(that is, the new visitor outputs the two arrays in a different order -
I can live with that).

Then it gets a bit crazy when using normal diff algorithms:

-void qapi_free_int32List(int32List *obj)
+void qapi_free_ACPIOSTInfo(ACPIOSTInfo *obj)

where declarations are rearranged (so the previous commit, which
attempted to sort declarations to make this one easier, did not quite
succeed).

Even in qapi-types.h things were rearranged:


-#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL
-#define QAPI_TYPES_BUILTIN_STRUCT_DECL
+#ifndef QAPI_TYPES_BUILTIN
+#define QAPI_TYPES_BUILTIN


-typedef struct int32List {
+typedef struct boolList boolList;
+
+struct boolList {

I can understand the minor change in #ifdef name, and even the
separation between typedef and struct declaration (even though the old
approach of doing it all at once works).  But the overall diff in the
generated files would be easier to review if done in stages, and either
make 25/47 sort closer to what this patch does, or add yet more
prerequisite patches that further tweak sorting.  Ideally, this patch
will be a lot easier to review if the generated code is much closer to
the pre-patch version (that is, separating sorting changes from other
changes).  On the other hand, yes, it's kind of a pain to patch old code
to do things differently just before throwing it away with the new code
in its place, so it becomes a judgment call of how confident we want to
be that the new implementation isn't breaking things.

I was able to make a bit more sense of things by using git's 'diff
patience' algorithm, which showed things more as code motion rather than
one-or-two-line changes to lots of common boilerplate, but even that
diffstat is still big:

 qapi-types1.c | 2030 +++++++++++++++----------------
 qapi-types1.h | 3707
+++++++++++++++++++++++++++++++++------------------------
 2 files changed, 3148 insertions(+), 2589 deletions(-)

That said, I'll still at least review this code by inspection, and
things still compile fine, so although I'm reluctant to give R-b while
the patch is in RFC stage (because I didn't want to take the time to be
certain the results are the same amidst so much churn), they are mostly
sane.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-types.py                   | 268 +++++++++++++-------------------
>  tests/qapi-schema/qapi-schema-test.json |   4 +-
>  2 files changed, 114 insertions(+), 158 deletions(-)
> 
> diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
> index a48ad9c..d6185c6 100644
> --- a/scripts/qapi-types.py
> +++ b/scripts/qapi-types.py
> @@ -2,88 +2,67 @@
>  # QAPI types generator
>  #
>  # Copyright IBM, Corp. 2011
> +# Copyright (c) 2013-2015 Red Hat Inc.
>  #
>  # Authors:
>  #  Anthony Liguori <aliguori@us.ibm.com>
> +#  Markus Armbruster <armbru@redhat.com>
>  #
>  # This work is licensed under the terms of the GNU GPL, version 2.
>  # See the COPYING file in the top-level directory.
>  
> -from ordereddict import OrderedDict
>  from qapi import *
>  
> -def generate_fwd_builtin(name):
> -    return mcgen('''
> -
> -typedef struct %(name)sList {

For example, the change of splitting this into:

typedef struct %(name)sList;

struct %(name)sList {

done as a separate patch would make the generated diff of this patch
smaller.


>  
> -def generate_struct_fields(members):
> +def gen_struct_field(name, typ, optional):
>      ret = ''
>  
> -    for argname, argentry, optional in parse_args(members):
> -        if optional:
> -            ret += mcgen('''
> +    if optional:
> +        ret += mcgen('''

(sometimes it's annoying that python requires indentation changes when
changing control flow; for C files, sometimes I split a patch that
merely adds {} and indentation from another patch that changes logic, so
that the logic change isn't drowned by the reindentation)

> +class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
> +    def __init__(self):
> +        self.decl = None
> +        self.defn = None
> +        self.fwdecl = None
> +        self.fwdefn = None
> +        self.btin = None
> +    def visit_begin(self):
> +        self.decl = ''
> +        self.defn = ''
> +        self.fwdecl = ''
> +        self.fwdefn = ''
> +        self.btin = guardstart('QAPI_TYPES_BUILTIN')
> +    def visit_end(self):
> +        self.decl = self.fwdecl + self.decl
> +        self.fwdecl = None
> +        self.defn = self.fwdefn + self.defn
> +        self.fwdefn = None
> +        # To avoid header dependency hell, we always generate
> +        # declarations for built-in types in our header files and
> +        # simply guard them.
> +        self.btin += guardend('QAPI_TYPES_BUILTIN')
> +        self.decl = self.btin + self.decl
> +        self.btin = None
> +        # Doesn't work for cases where we link in multiple objects
> +        # that have the functions defined, so generate them only with
> +        # option -b (do_builtins).

Does this comment...

> +    def _gen_type_cleanup(self, name):
> +        self.decl += generate_type_cleanup_decl(name)
> +        self.defn += generate_type_cleanup(name)
> +    def visit_enum_type(self, name, info, values):
> +        self.fwdecl += generate_enum(name, values)
> +        self.fwdefn += generate_enum_lookup(name, values)
> +    def visit_array_type(self, name, info, element_type):
> +        if isinstance(element_type, QAPISchemaBuiltinType):
> +            self.btin += gen_fwd_object_or_array(name)
> +            self.btin += gen_array(name, element_type)
> +            self.btin += generate_type_cleanup_decl(name)
> +            if do_builtins:
> +                self.defn += generate_type_cleanup(name)

...fit better here?

> +        else:
> +            self.fwdecl += gen_fwd_object_or_array(name)
> +            self.decl += gen_array(name, element_type)
> +            self._gen_type_cleanup(name)
> +    def visit_object_type(self, name, info, base, members, variants):
> +        if info:

So to make sure I understand, this ensures that implicit types (those
handled merely by parameters in a function or as members of a struct)
have no declarations required in the -types files.

> +            self.fwdecl += gen_fwd_object_or_array(name)
> +            if variants:
> +                self.decl += gen_union(name, base, variants)

Is it worth 'assert not members' at this point?

> +            else:
> +                self.decl += gen_struct(name, base, members)
> +            self._gen_type_cleanup(name)
> +    def visit_alternate_type(self, name, info, variants):
> +        self.fwdecl += gen_fwd_object_or_array(name)
> +        self.fwdefn += gen_alternate_qtypes(name, variants)
> +        self.decl += gen_union(name, None, variants)
> +        self.decl += gen_alternate_qtypes_decl(name)
> +        self._gen_type_cleanup(name)
> +

The visitor looks reasonable from inspection review.

>  do_builtins = False
>  
>  (input_file, output_dir, do_c, do_h, prefix, opts) = \
> @@ -325,77 +348,10 @@ fdecl.write(mcgen('''
>  #include <stdint.h>
>  '''))
>  
> -exprs = QAPISchema(input_file).get_exprs()
> -
> -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
> -for typename in builtin_types.keys():
> -    fdecl.write(generate_fwd_builtin(typename))
> -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))

This is a place where adding sorting in 25/47 (or a separate patch)
would make the impact on generated code from this patch smaller.

> -
> -for expr in exprs:
> -    ret = ""
> -    if expr.has_key('struct'):
> -        ret += generate_fwd_struct(expr['struct'])
> -    elif expr.has_key('enum'):
> -        ret += generate_enum(expr['enum'], expr['data'])
> -        ret += generate_fwd_enum_struct(expr['enum'])
> -        fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
> -    elif expr.has_key('union'):
> -        ret += generate_fwd_struct(expr['union'])
> -        enum_define = discriminator_find_enum_define(expr)
> -        if not enum_define:
> -            ret += generate_enum('%sKind' % expr['union'], expr['data'].keys())
> -            fdef.write(generate_enum_lookup('%sKind' % expr['union'],
> -                                            expr['data'].keys()))
> -    elif expr.has_key('alternate'):
> -        ret += generate_fwd_struct(expr['alternate'])
> -        ret += generate_enum('%sKind' % expr['alternate'], expr['data'].keys())
> -        fdef.write(generate_enum_lookup('%sKind' % expr['alternate'],
> -                                        expr['data'].keys()))

Tweaking the order of _qtypes[] vs. _lookup[] here in a pre-req patch
would also help review of the diff generated by this patch.

> -        fdef.write(generate_alternate_qtypes(expr))
> -    else:
> -        continue
> -    fdecl.write(ret)
> -
> -# to avoid header dependency hell, we always generate declarations
> -# for built-in types in our header files and simply guard them
> -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
> -for typename in builtin_types.keys():
> -    fdecl.write(generate_type_cleanup_decl(typename + "List"))
> -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
> -
> -# ...this doesn't work for cases where we link in multiple objects that
> -# have the functions defined, so we use -b option to provide control
> -# over these cases
> -if do_builtins:
> -    for typename in builtin_types.keys():
> -        fdef.write(generate_type_cleanup(typename + "List"))

Another iteration worth sorting in an earlier patch.

> -
> -for expr in exprs:
> -    ret = ""
> -    if expr.has_key('struct'):
> -        ret += generate_struct(expr) + "\n"
> -        ret += generate_type_cleanup_decl(expr['struct'] + "List")
> -        fdef.write(generate_type_cleanup(expr['struct'] + "List"))
> -        ret += generate_type_cleanup_decl(expr['struct'])
> -        fdef.write(generate_type_cleanup(expr['struct']))

I think part of the large size of the generated diff comes from whether
array fooList types are emitted in a different order via the visitor.

> -    elif expr.has_key('union'):
> -        ret += generate_union(expr, 'union') + "\n"
> -        ret += generate_type_cleanup_decl(expr['union'] + "List")
> -        fdef.write(generate_type_cleanup(expr['union'] + "List"))
> -        ret += generate_type_cleanup_decl(expr['union'])
> -        fdef.write(generate_type_cleanup(expr['union']))
> -    elif expr.has_key('alternate'):
> -        ret += generate_union(expr, 'alternate') + "\n"
> -        ret += generate_type_cleanup_decl(expr['alternate'] + "List")
> -        fdef.write(generate_type_cleanup(expr['alternate'] + "List"))
> -        ret += generate_type_cleanup_decl(expr['alternate'])
> -        fdef.write(generate_type_cleanup(expr['alternate']))
> -    elif expr.has_key('enum'):
> -        ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List")
> -        fdef.write(generate_type_cleanup(expr['enum'] + "List"))
> -    else:
> -        continue
> -    fdecl.write(ret)
> +schema = QAPISchema(input_file)
> +gen = QAPISchemaGenTypeVisitor()
> +schema.visit(gen)
> +fdef.write(gen.defn)
> +fdecl.write(gen.decl)
>  
>  close_output(fdef, fdecl)
> diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
> index a9e5aab..257b4d4 100644
> --- a/tests/qapi-schema/qapi-schema-test.json
> +++ b/tests/qapi-schema/qapi-schema-test.json
> @@ -39,8 +39,8 @@
>    'data': { 'value1' : 'UserDefA',
>              'value2' : 'UserDefB',
>              'value3' : 'UserDefB' } }
> -# FIXME generated struct UserDefFlatUnion has members for direct base
> -# UserDefUnionBase, but lacks members for indirect base UserDefZero
> +# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
> +# members of indirect base UserDefZero
>  
>  { 'struct': 'UserDefUnionBase',
>    'base': 'UserDefZero',
> 

Overall, moving in the right direction.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

On 07/22/2015 11:34 AM, Eric Blake wrote:

> 
> -typedef struct int32List {
> +typedef struct boolList boolList;
> +
> +struct boolList {
> 
> I can understand the minor change in #ifdef name, and even the
> separation between typedef and struct declaration (even though the old
> approach of doing it all at once works).

Actually, I don't see any clear-cut policy on the matter (all HACKING
says is that "Typedefs are used to eliminate the redundant 'struct'
keyword", but does not say how aggressively they must be used).  In
fact, it looks like most of qemu.git prefers the more compact
representation, while your patch goes to the less-common one:

$ git grep 'typedef struct .* {' | wc
   1370    5549   71105
$ git grep 'typedef struct .*;' | wc
    624    2515   41449

However, one thing that your style has going for it, that was not
possible with the compact version, is to use:

 typedef struct intList intList;
 struct intList {
     union {
         int64_t value;
         uint64_t padding;
     };
-    struct intList *next;
+    intList *next;
 };

that is, because your typedef is separate, you have the option of being
able to drop the use of 'struct' from within the actual intList struct.
 Again, if we want that, it would be better to do that as a separate commit.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 4255 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Fixes flat unions to get the base's base members.  Test case is from
> commit 2fc0043, in qapi-schema-test.json:
> 

Okay, I see a cause for part of my confusion.

>  
> +class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):

> +    def visit_end(self):
> +        self.decl = self.fwdecl + self.decl
> +        self.fwdecl = None
> +        self.defn = self.fwdefn + self.defn
> +        self.fwdefn = None
> +        # To avoid header dependency hell, we always generate
> +        # declarations for built-in types in our header files and
> +        # simply guard them.
> +        self.btin += guardend('QAPI_TYPES_BUILTIN')
> +        self.decl = self.btin + self.decl
> +        self.btin = None

The new code goes to great lengths to separate all builtin decls up
front.  But...

> +        # Doesn't work for cases where we link in multiple objects
> +        # that have the functions defined, so generate them only with
> +        # option -b (do_builtins).
> +    def _gen_type_cleanup(self, name):
> +        self.decl += generate_type_cleanup_decl(name)
> +        self.defn += generate_type_cleanup(name)
> +    def visit_enum_type(self, name, info, values):
> +        self.fwdecl += generate_enum(name, values)
> +        self.fwdefn += generate_enum_lookup(name, values)
> +    def visit_array_type(self, name, info, element_type):
> +        if isinstance(element_type, QAPISchemaBuiltinType):
> +            self.btin += gen_fwd_object_or_array(name)
> +            self.btin += gen_array(name, element_type)
> +            self.btin += generate_type_cleanup_decl(name)
> +            if do_builtins:
> +                self.defn += generate_type_cleanup(name)

...it is still interleaving builtin defns with everything else.


> -exprs = QAPISchema(input_file).get_exprs()
> -
> -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
> -for typename in builtin_types.keys():
> -    fdecl.write(generate_fwd_builtin(typename))
> -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))

Meanwhile, the old code did builtin intList definitions up front...

> -
> -for expr in exprs:
> -    ret = ""
> -    if expr.has_key('struct'):

> -# to avoid header dependency hell, we always generate declarations
> -# for built-in types in our header files and simply guard them
> -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
> -for typename in builtin_types.keys():
> -    fdecl.write(generate_type_cleanup_decl(typename + "List"))
> -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))

...but declared the cleanup functions at the end (so code motion 1 is
that the cleanup functions are now interleaved with the intList
declarations)...

> -
> -# ...this doesn't work for cases where we link in multiple objects that
> -# have the functions defined, so we use -b option to provide control
> -# over these cases
> -if do_builtins:
> -    for typename in builtin_types.keys():
> -        fdef.write(generate_type_cleanup(typename + "List"))

...and wrote the definitions at the end (so code motion 2 is that the
definitions are now interleaved with all other definitions).

Attached a few quick hacks that reduce (but not eliminate) some of the
visitor's churn to the generated files, by consolidating the location of
the builtins and cleaning up struct declarations as separate cleanups
(to be applied prior to 26/47), then updating the visitor to use the
same builtin order (to be applied after or squashed with 26/47).

It still doesn't solve everything, but with those hacks, I was able to
get to a slightly more manageable:

 qapi-types.c                        |  864 ++++----
 qapi-types.h                        | 3602
++++++++++++++++++------------------
 qga/qapi-generated/qga-qapi-types.c |  110 -
 qga/qapi-generated/qga-qapi-types.h |  405 ++--
 4 files changed, 2585 insertions(+), 2396 deletions(-)

I'm sure there are further things that could be done, but at this point,
I hope you get my picture, and I'll quit focusing on this particular patch.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-qapi-types-sort-and-consolidate-builtins.patch --]
[-- Type: text/x-patch; name="0001-qapi-types-sort-and-consolidate-builtins.patch", Size: 2098 bytes --]

From 00df19b3b0d497e0a4670308f971ca95383b20cd Mon Sep 17 00:00:00 2001
From: Eric Blake <eblake@redhat.com>
Date: Wed, 22 Jul 2015 13:59:20 -0600
Subject: [PATCH 1/4] qapi-types: sort and consolidate builtins

Just as sorting generated code based on user-defined structs
will help analysis of later patches, we also want to sort
builtin types rather than letting python randomly dump a
hash table.  We also want to consolidate the two conditional
areas of the generated .h file into one.
---
 scripts/qapi-types.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)

diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index a48ad9c..b7028f6 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -327,11 +327,6 @@ fdecl.write(mcgen('''

 exprs = QAPISchema(input_file).get_exprs()

-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_fwd_builtin(typename))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-
 for expr in exprs:
     ret = ""
     if expr.has_key('struct'):
@@ -359,16 +354,17 @@ for expr in exprs:

 # to avoid header dependency hell, we always generate declarations
 # for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-for typename in builtin_types.keys():
+fdecl.write(guardstart("QAPI_TYPES_BUILTIN"))
+for typename in sorted(builtin_types.keys()):
+    fdecl.write(generate_fwd_builtin(typename))
     fdecl.write(generate_type_cleanup_decl(typename + "List"))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
+fdecl.write(guardend("QAPI_TYPES_BUILTIN"))

 # ...this doesn't work for cases where we link in multiple objects that
 # have the functions defined, so we use -b option to provide control
 # over these cases
 if do_builtins:
-    for typename in builtin_types.keys():
+    for typename in sorted(builtin_types.keys()):
         fdef.write(generate_type_cleanup(typename + "List"))

 for expr in exprs:
-- 
2.4.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.3: 0002-qapi-update-typedef-usage.patch --]
[-- Type: text/x-patch; name="0002-qapi-update-typedef-usage.patch", Size: 2075 bytes --]

From eac8ce42799ecb17271a70aa52185b54173de399 Mon Sep 17 00:00:00 2001
From: Eric Blake <eblake@redhat.com>
Date: Wed, 22 Jul 2015 14:00:06 -0600
Subject: [PATCH 2/4] qapi: update typedef usage

Change from:

 typedef struct fooList {
     ...
     struct fooList *next;
 } fooList;

to:

 typedef struct fooList fooList;

 struct fooList {
     ...
     struct fooList *next;
 };

If desired, we could further eliminate the now-spurious 'struct'
from the 'next' member of the list.

Note that qemu does not have any standard on which of the two
forms is preferred, and that more uses tend to go with the
typedef and declaration smashed into one.
---
 scripts/qapi-types.py | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index b7028f6..d21381c 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -15,13 +15,15 @@ from qapi import *
 def generate_fwd_builtin(name):
     return mcgen('''

-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(type)s value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  type=c_type(name),
                  name=name)
@@ -31,26 +33,30 @@ def generate_fwd_struct(name):

 typedef struct %(name)s %(name)s;

-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(name)s *value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  name=c_name(name))

 def generate_fwd_enum_struct(name):
     return mcgen('''

-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(name)s value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  name=c_name(name))

-- 
2.4.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.4: 0003-qapi-squash-to-visitor.patch --]
[-- Type: text/x-patch; name="0003-qapi-squash-to-visitor.patch", Size: 3244 bytes --]

From 348f3d3d3a35afe7dca4b695972677ce03da91ca Mon Sep 17 00:00:00 2001
From: Eric Blake <eblake@redhat.com>
Date: Wed, 22 Jul 2015 14:58:25 -0600
Subject: [PATCH 4/4] qapi: squash to visitor

Prior to conversion to a visitor, all code related to lists of
builtins were lumped together. Doing so in the visitor code
reduces the churn in comparing the generated files, making it
easier to prove the conversion to the visitor is correct.
---
 scripts/qapi-types.py | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)

diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index d6185c6..0817628 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -240,27 +240,26 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.defn = None
         self.fwdecl = None
         self.fwdefn = None
-        self.btin = None
+        self.btindecl = None
+        self.btindefn = None
     def visit_begin(self):
         self.decl = ''
         self.defn = ''
         self.fwdecl = ''
         self.fwdefn = ''
-        self.btin = guardstart('QAPI_TYPES_BUILTIN')
+        self.btindecl = guardstart('QAPI_TYPES_BUILTIN')
+        self.btindefn = ''
     def visit_end(self):
-        self.decl = self.fwdecl + self.decl
-        self.fwdecl = None
-        self.defn = self.fwdefn + self.defn
-        self.fwdefn = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
         # simply guard them.
-        self.btin += guardend('QAPI_TYPES_BUILTIN')
-        self.decl = self.btin + self.decl
-        self.btin = None
-        # Doesn't work for cases where we link in multiple objects
-        # that have the functions defined, so generate them only with
-        # option -b (do_builtins).
+        self.btindecl += guardend('QAPI_TYPES_BUILTIN')
+        self.decl = self.fwdecl + self.btindecl + self.decl
+        self.fwdecl = None
+        self.btindecl = None
+        self.defn = self.fwdefn + self.btindefn + self.defn
+        self.fwdefn = None
+        self.btindefn = None
     def _gen_type_cleanup(self, name):
         self.decl += generate_type_cleanup_decl(name)
         self.defn += generate_type_cleanup(name)
@@ -269,11 +268,11 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.fwdefn += generate_enum_lookup(name, values)
     def visit_array_type(self, name, info, element_type):
         if isinstance(element_type, QAPISchemaBuiltinType):
-            self.btin += gen_fwd_object_or_array(name)
-            self.btin += gen_array(name, element_type)
-            self.btin += generate_type_cleanup_decl(name)
+            self.btindecl += gen_fwd_object_or_array(name)
+            self.btindecl += gen_array(name, element_type)
+            self.btindecl += generate_type_cleanup_decl(name)
             if do_builtins:
-                self.defn += generate_type_cleanup(name)
+                self.btindefn += generate_type_cleanup(name)
         else:
             self.fwdecl += gen_fwd_object_or_array(name)
             self.decl += gen_array(name, element_type)
-- 
2.4.3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 13959 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Fixes flat unions to visit the base's base members (the previous
> commit merely added them to the struct).  Same test case.
> 
> Patch's effect on visit_type_UserDefFlatUnion():
> 
>      static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp)
>      {
>          Error *err = NULL;
> 
>     +    visit_type_int(m, &(*obj)->integer, "integer", &err);
>     +    if (err) {
>     +        goto out;
>     +    }
>          visit_type_str(m, &(*obj)->string, "string", &err);
>          if (err) {
>              goto out;
> 
> Test cases updated for the bug fix.
> 
> Fixes alternates to generate a visitor for their implicit enumeration
> type.  None of them are currently used, obviously.  Example:
> block-core.json's BlockdevRef now generates
> visit_type_BlockdevRefKind().
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-visit.py                   | 254 ++++++++++++--------------------
>  tests/qapi-schema/qapi-schema-test.json |   3 -
>  tests/test-qmp-input-strict.c           |   2 +-
>  tests/test-qmp-input-visitor.c          |   4 +-
>  4 files changed, 100 insertions(+), 163 deletions(-)

Another conversion that results in a fairly large diffstat to the
generated files:

 qapi-visit.c                        | 4542
++++++++++++++++++------------------
 qapi-visit.h                        |  256 --
 qga/qapi-generated/qga-qapi-visit.c |   88
 qga/qapi-generated/qga-qapi-visit.h |   36
 4 files changed, 2355 insertions(+), 2567 deletions(-)

Same complaints as in 26/47, where splitting some of the cleanups into
separate patches would make it easier to validate that the final
conversion is correct.

Here, a very common thing in the generated .c file is that you end up
swapping the order of visit_type_foo and visit_type_fooList, for any
time when foo is an enum. [1]

> 
> diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
> index a52a572..135e7c1 100644
> --- a/scripts/qapi-visit.py
> +++ b/scripts/qapi-visit.py
> @@ -12,7 +12,6 @@
>  # This work is licensed under the terms of the GNU GPL, version 2.
>  # See the COPYING file in the top-level directory.
>  
> -from ordereddict import OrderedDict
>  from qapi import *
>  import re
>  
> @@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type):
>          return ''
>      implicit_structs_seen.add(type)
>      ret = ''
> -    if type not in struct_fields_seen:
> +    if type.name not in struct_fields_seen:
>          # Need a forward declaration
>          ret += mcgen('''
>  
>  static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
>  ''',
> -                     c_type=type_name(type))
> +                     c_type=type.c_name())

This looks a little fishy on first read; why are we calling
type.c_name() (and not type.c_type()) when assigning to a placeholder
named %(c_type)s?  But on second thought, it looks correct: since we
really do want the name of the type (and not the magic '*' pointer suffix).

Still, it might be nicer to name things %(c_name)s here; and in that
case, it's more of a pre-existing cleanup that might be better floating
into one of your earlier patches.

>  
>      ret += mcgen('''
>  
> @@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
>      error_propagate(errp, err);
>  }
>  ''',
> -                 c_type=type_name(type))
> +                 c_type=type.c_name())

same here.

>      return ret
>  
>  def generate_visit_struct_fields(name, members, base = None):
> @@ -74,24 +73,24 @@ if (err) {
>      goto out;
>  }
>  ''',
> -                     type=type_name(base), c_name=c_name('base'))
> +                     type=base.c_name(), c_name=c_name('base'))

And this one's pointless: c_name('base') == 'base'. Pointless since
commit 622f557 introduced type inheritance.  Why do we even need
%(c_name)s if we are always passing a constant string?

Oh, and that means our generator has a collision bug that none of my
added tests have exposed yet: you cannot have a base class and
simultaneously add a member named 'base':

{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }

because the generated C code is trying to use the name 'base' for its
own purposes.  I guess that means more pre-req patches to the series to
expose the bug, and either tighten the parser to reject things for now
(easiest) or update the generator to not collide (harder, and fine for a
later series).

By the way, now that we are emitting flat unions in such a way that you
can cast to the base class, why don't we change our C code to do
likewise?  That is, where we now have this generated C:

struct BlockdevOptionsGenericFormat {
    BlockdevRef *file;
};

struct BlockdevOptionsGenericCOWFormat {
    BlockdevOptionsGenericFormat *base;
    bool has_backing;
    BlockdevRef *backing;
};

why can't we instead have an unboxed representation:

struct BlockdevOptionsGenericFormat {
    BlockdevRef *file;
};

/* This struct can be cast to BlockdevOptionsGenericFormat */
struct BlockdevOptionsGenericCOWFormat {
    BlockdevRef *file;
    /* end of fields from base class BlockdevOptionsGenericFormat */
    bool has_backing;
    BlockdevRef *backing;
};

where client code that was referring to o->base->file now refers to o->file.

> +def gen_visit_union(name, base, variants):
> +    ret = ''
>  
>      if base:
> -        assert discriminator
> -        base_fields = find_struct(base)['data'].copy()
> -        del base_fields[discriminator]
> -        ret += generate_visit_struct_fields(name, base_fields)
> +        members = [m for m in base.members if m != variants.tag_member]
> +        ret += generate_visit_struct_fields(name, members)

Why not just visit ALL base class members, unconditionally?

>  
> -    if discriminator:
> -        for key in members:
> -            ret += generate_visit_implicit_struct(members[key])
> +    for var in variants.variants:
> +        if var.flat:
> +            ret += generate_visit_implicit_struct(var.type)

Okay, I see where you are using .flat from the initial parse.  I still
think it is a bit odd that you are defining '.flat' for each 'variant'
within 'variants', even though, for a given 'variants', all members will
have the same setting of '.flat'.  That makes me wonder if '.flat'
should belong instead to the top-level 'variants' struct rather than to
each 'variant' member.

But again I wonder what would happen if you had instead normalized the
input of simple unions into always having an implicit struct (with
single member 'data'), so that by the time you get here, you only have
to deal with a single representation of unions instead of having to
still emit different things for flat vs. simple (since on the wire, we
already proved simple is shorthand that can be duplicated by a flat union).

>  
>      ret += mcgen('''
>  
> @@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
>  ''',
>                       name=c_name(name))
>  
> -    if not discriminator:
> -        tag = 'kind'
> -        disc_key = "type"
> -    else:
> -        tag = discriminator
> -        disc_key = discriminator
> +    disc_key = variants.tag_member.name
> +    if not variants.tag_name:
> +        # we pointlessly use a different key for simple unions

We could fix that (as a separate patch); wonder how much C code it would
affect.  A lot of these things that we can alter in generated code are
certainly easier to see now that we have a clean generator :)

> +def gen_visit_decl(name, scalar=False):
> +    c_type = c_name(name) + ' *'
> +    if not scalar:
> +        c_type += '*'
>      return mcgen('''
> -
> -void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
> +void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
>  ''',
> -                 name=c_name(name))
> +                 c_name=c_name(name), c_type=c_type)

Nice way to consolidate several near-identical copies.

> +
> +class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
> +    def __init__(self):
> +        self.decl = None
> +        self.defn = None
> +        self.btin = None
> +    def visit_begin(self):
> +        self.decl = ''
> +        self.defn = ''
> +        self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
> +    def visit_end(self):
> +        # to avoid header dependency hell, we always generate
> +        # declarations for built-in types in our header files and
> +        # simply guard them
> +        self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL')
> +        self.decl = self.btin + self.decl
> +        self.btin = None
> +        # ...this doesn't work for cases where we link in multiple
> +        # objects that have the functions defined, so we use
> +        # do_builtins (option -b) to provide control

And once again, as in 26/47, this floats the .h file to have all builtin
representations in one chunk (for continuity with pre-patch), but fails
to do the same for the .c code...

> +    def visit_enum_type(self, name, info, values):
> +        self.decl += gen_visit_decl(name, scalar=True)
> +        self.defn += generate_visit_enum(name)
> +    def visit_array_type(self, name, info, element_type):
> +        decl = gen_visit_decl(name)
> +        defn = gen_visit_list(name, element_type)
> +        if isinstance(element_type, QAPISchemaBuiltinType):
> +            self.btin += decl
> +            if do_builtins:
> +                self.defn += defn

...where the builtins are now interleaved with everything else instead
of bunched together, making the generated diff larger and more confusing
than necessary.

> +        else:
> +            self.decl += decl
> +            self.defn += defn
> +    def visit_object_type(self, name, info, base, members, variants):
> +        if info:
> +            self.decl += gen_visit_decl(name)
> +            if variants:
> +                self.defn += gen_visit_union(name, base, variants)

Worth adding 'assert not members'?

> +            else:
> +                self.defn += gen_visit_struct(name, base, members)

Or maybe we can someday consolidate these two into a single
gen_visit_object, that handles all members and variants in a uniform
manner, instead of our current differences. I wonder how much C code
would be impacted?

> +    def visit_alternate_type(self, name, info, variants):
> +        self.decl += gen_visit_decl(name)
> +        self.defn += gen_visit_alternate(name, variants)
>  
>  do_builtins = False
>  
> @@ -442,56 +428,10 @@ fdecl.write(mcgen('''
>  ''',
>                    prefix=prefix))
>  
> -exprs = QAPISchema(input_file).get_exprs()
> -
> -# to avoid header dependency hell, we always generate declarations
> -# for built-in types in our header files and simply guard them
> -fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
> -for typename in builtin_types.keys():
> -    fdecl.write(generate_declaration(typename, builtin_type=True))
> -fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
> -
> -# ...this doesn't work for cases where we link in multiple objects that
> -# have the functions defined, so we use -b option to provide control
> -# over these cases
> -if do_builtins:
> -    for typename in builtin_types.keys():
> -        fdef.write(generate_visit_list(typename))

Again, a well-placed sorted() over these two loops in a pre-req patch
will minimize the churn on the builtins.

> -
> -for expr in exprs:
> -    if expr.has_key('struct'):
> -        ret = generate_visit_struct(expr)
> -        ret += generate_visit_list(expr['struct'])
> -        fdef.write(ret)
> -
> -        ret = generate_declaration(expr['struct'])
> -        fdecl.write(ret)
> -    elif expr.has_key('union'):
> -        ret = generate_visit_union(expr)
> -        ret += generate_visit_list(expr['union'])
> -        fdef.write(ret)
> -
> -        enum_define = discriminator_find_enum_define(expr)
> -        ret = ""
> -        if not enum_define:
> -            ret = generate_decl_enum('%sKind' % expr['union'])

Nice that the new visitor automatically visits any implicit enum,
without us having to special case it.

> -        ret += generate_declaration(expr['union'])
> -        fdecl.write(ret)
> -    elif expr.has_key('alternate'):
> -        ret = generate_visit_alternate(expr['alternate'], expr['data'])
> -        ret += generate_visit_list(expr['alternate'])
> -        fdef.write(ret)
> -
> -        ret = generate_decl_enum('%sKind' % expr['alternate'])
> -        ret += generate_declaration(expr['alternate'])
> -        fdecl.write(ret)
> -    elif expr.has_key('enum'):
> -        ret = generate_visit_list(expr['enum'])
> -        ret += generate_visit_enum(expr['enum'])

[1] swapping these two lines in a pre-req patch will minimize the churn
of this conversion.

> -        fdef.write(ret)
> -
> -        ret = generate_decl_enum(expr['enum'])
> -        ret += generate_enum_declaration(expr['enum'])
> -        fdecl.write(ret)
> +schema = QAPISchema(input_file)
> +gen = QAPISchemaGenVisitVisitor()
> +schema.visit(gen)
> +fdef.write(gen.defn)
> +fdecl.write(gen.decl)

Again, overall impression is that your series is headed in the right
direction.

And nice that the TODOs in the testsuite pointed out what this fixes,
for visiting indirect bases.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2404 bytes --]

On 07/22/2015 03:21 PM, Eric Blake wrote:
> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Fixes flat unions to get the base's base members.  Test case is from
>> commit 2fc0043, in qapi-schema-test.json:
>>
> 
> Okay, I see a cause for part of my confusion.
> 
>>  

> ...and wrote the definitions at the end (so code motion 2 is that the
> definitions are now interleaved with all other definitions).
> 
> Attached a few quick hacks that reduce (but not eliminate) some of the
> visitor's churn to the generated files, by consolidating the location of
> the builtins and cleaning up struct declarations as separate cleanups
> (to be applied prior to 26/47), then updating the visitor to use the
> same builtin order (to be applied after or squashed with 26/47).

By the way, I'm perfectly fine if the initial conversion to the visitor
keeps .c output separate into two groups to make review easier, then a
later patch simplifies things back to a single sorting over the whole
file (and undoing the grouping after conversion to visitor will be
easier than figuring out how to interleave the builtin array visitors
into the right sorted order prior to use of the visitor).  I like the
final result of everything sorted, it's just that I think it is easier
to validate the final result when it is broken into stages with no one
step doing too much, even if some of the intermediate stages do more
work than needed and get undone by the final stage.

Or put another way, your next spin of the series could include your
original 26 + my hack 003 squashed together as one patch (conversion to
visitor with minimized generated diff noise), followed by a revert of my
hack 003 as the next (no need to group builtins together in the long run).

[and for the record, I mention this because that's the approach I took
in my earlier cleanups: commit fd41dd4e included a hack to rename 'type'
to 'struct' in the generator, so that commit 895a2a80 could be quite
mindless without touching the generator, and then commit 3e391d35 undid
my hack.  When I originally wrote the conversion in an earlier version
of the series, I did not have that hack in place and the resulting
single patch was too big; breaking it up made my submission and the
review easier.]

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 28/47] qapi-commands: Convert to QAPISchemaVisitor

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Output unchanged except for white-space.

Indeed, and the diffstat shows it was only one blank line:

 qga-qmp-marshal.c |    1 +
 1 file changed, 1 insertion(+)

MUCH friendlier to review :)

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 157 ++++++++++++++++++++++++++---------------------
>  scripts/qapi.py          |   2 +-
>  2 files changed, 87 insertions(+), 72 deletions(-)
> 

Here, I can confidently say:

Reviewed-by: Eric Blake <eblake@redhat.com>

> +++ b/scripts/qapi.py
> @@ -1366,7 +1366,7 @@ def c_type(value, is_param=False):
>          return c_name(value) + pointer_suffix
>  
>  def is_c_ptr(value):
> -    return c_type(value).endswith(pointer_suffix)
> +    return value.endswith(pointer_suffix)

Perhaps this cleanup could be floated earlier in the series?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2578 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> is_c_ptr() looks whether the end of the C text for the type looks like
> a pointer.  Works, but is fragile.
> 
> We now have a better tool: use QAPISchemaType method c_null().  The
> initializers for non-pointers become prettier: 0, false or the
> enumeration constant with the value 0 instead of {0}.
> 
> One place looks suspicious: we initialize pointers, but not
> non-pointers.  Either the initialization is superfluous and should be
> deleted, or the non-pointers need it as well, or something subtle is
> going on and needs a comment.  Since I lack the time to figure it out
> now, mark it FIXME.

Commenting on just this part for now (out of time to review this patch
proper for today):

> @@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
>                  header=hdr)
>  
>      if ret_type:
> -        if is_c_ptr(ret_type.c_type()):
> +        # FIXME fishy: only pointers are initialized
> +        if ret_type.c_null() == 'NULL':
>              retval = "    %s retval = NULL;" % ret_type.c_type()
>          else:
>              retval = "    %s retval;" % ret_type.c_type()

Here's an example function impacted by this:

static void qmp_marshal_input_guest_file_open(QDict *args, QObject
**ret, Error
**errp)
{
    Error *local_err = NULL;
    int64_t retval;
    QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
    QapiDeallocVisitor *md;
    Visitor *v;
    char *path = NULL;
    bool has_mode = false;
    char *mode = NULL;
...
    retval = qmp_guest_file_open(path, has_mode, mode, &local_err);
    if (local_err) {
        goto out;
    }

    qmp_marshal_output_guest_file_open(retval, ret, &local_err);

But compare that to any other function that returns a pointer, and
you'll see the same pattern (the only use of retval is in the final call
to qmp_marshal_output..., right after assigning retval); that is,
initializing to NULL is dead code, and you could get away with just
always declaring it instead of worrying about c_null() in this code.

Or maybe we have a leak - if the 'if (local_err)' can ever trigger even
when a function returned non-NULL, then our out: label is missing a
free(retval), but only when retval is a pointer.  Or maybe we change the
code to assert that retval is NULL if local_err is set after calling the
user's function, to prove we don't have a leak.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> is_c_ptr() looks whether the end of the C text for the type looks like
> a pointer.  Works, but is fragile.
> 
> We now have a better tool: use QAPISchemaType method c_null().  The
> initializers for non-pointers become prettier: 0, false or the
> enumeration constant with the value 0 instead of {0}.
> 
> One place looks suspicious: we initialize pointers, but not
> non-pointers.  Either the initialization is superfluous and should be
> deleted, or the non-pointers need it as well, or something subtle is
> going on and needs a comment.  Since I lack the time to figure it out
> now, mark it FIXME.

I commented on that in another mail - either we have a leak on failure
(and need the initializer only for pointers), or we should add an assert
(and don't need the initializer at all), depending on what semantics we
want to enforce on all handler functions that set their error parameter.
 Probably worth cleaning that up in a separate pre-req patch.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 19 +++++++------------
>  scripts/qapi.py          |  3 ---
>  2 files changed, 7 insertions(+), 15 deletions(-)
> 

Nice reduction in size.  If the marshal code FIXME disappears due to a
separate cleanup, then the rest of this commit is good to go:
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> New methods c_name(), c_type(), c_null(), json_type(),
> alternate_qtype().
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------
>  1 file changed, 65 insertions(+), 7 deletions(-)
> 

I just noticed:

> @@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
>          for v in values:
>              assert isinstance(v, str)
>          self.values = values
> +    def c_type(self, is_param=False):
> +        return c_name(self.name)
> +    def c_null(self):
> +        return c_enum_const(self.name, self.values[0])

What does this return for an empty enum, as in { 'enum':'Empty',
'data':[] }?  Our testsuite proves we can do that, even if our normal
.json code doesn't use it.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 30/47] qapi: De-duplicate enum code generation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Duplicated in commit 21cd70d.  Yes, we can't import qapi-types, but
> that's no excuse.  Move the helpers from qapi-types.py to qapi.py, and
> replace the duplicates in qapi-event.py.
> 
> The generated event enumeration type gains a _MAX member,

Not quite; the _MAX member was already there, but done by hand [1]
instead of by common code.

> and its
> lookup table becomes const-correct (see commit 2e4450f).

Maybe what you meant instead of gaining _MAX was that it also gains C99
initializers:

-const char *QAPIEvent_lookup[] = {
-    "ACPI_DEVICE_OST",
-    "BALLOON_CHANGE",
...
-    "WATCHDOG",
-    NULL,
+const char *const QAPIEvent_lookup[] = {
+    [QAPI_EVENT_ACPI_DEVICE_OST] = "ACPI_DEVICE_OST",
+    [QAPI_EVENT_BALLOON_CHANGE] = "BALLOON_CHANGE",

Overall, a good change.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-event.py | 67 +++------------------------------------------------
>  scripts/qapi-types.py | 55 ------------------------------------------
>  scripts/qapi.py       | 55 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 59 insertions(+), 118 deletions(-)

Since only the commit message needs tweaking,
Reviewed-by: Eric Blake <eblake@redhat.com>

> 
> diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
> index aec2d32..aed45d6 100644
> --- a/scripts/qapi-event.py
> +++ b/scripts/qapi-event.py
> @@ -153,63 +153,6 @@ def generate_event_implement(api_name, event_name, params):

> -    # append automatically generated _MAX value
> -    enum_max_value = c_enum_const(event_enum_name, "MAX")
> -    enum_values = event_enum_values + [ enum_max_value ]

[1]


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 31/47] qapi-event: Eliminate global variable event_enum_value

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 360 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-event.py | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2352 bytes --]

On 07/21/2015 06:43 AM, Eric Blake wrote:
> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> A command's 'data' must be a struct type, given either as a
>> dictionary, or as struct type name.
>>
>> Existing test case data-int.json covers simple type 'int'.  Add test
>> cases for type names referring to union and alternate types.
> 
> We could probably relax things to allow a union (which is always a
> dictionary on the wire), but I agree that allowing an alternate type is
> not appropriate (the goal here is that we require a dictionary).  But
> it's also easier to be conservative now and relax later.
> 

>> +++ b/tests/qapi-schema/args-alternate.json
>> @@ -0,0 +1,4 @@
>> +# we do not allow alternate arguments
>> +# TODO should we support this?
> 
> I see no reason to allow a non-dictionary in QMP, so this TODO could be
> dropped.

Or, to be clear, we document that arguments is always a dictionary, for:
{ "execute":"command", "arguments":{} }. Allowing an alternate would
break that, so it is a different level of change to allow an alternate
(change the QMP protocol) than what it would be to allow a union (the
QMP protocol is unchanged).  Not that we can't do it if we ever have a
reason, it's just that I don't see it being worth a TODO statement.

> 
>> +++ b/tests/qapi-schema/args-union.json
>> @@ -0,0 +1,4 @@
>> +# FIXME we should reject union arguments
>> +# TODO should we support this?
>> +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
>> +{ 'command': 'oops', 'data': 'Uni' }
> 
> This, on the other hand, seems valid from the wire format (it will
> always be a dictionary).  I guess the problem is that we generate a C
> function signature based by calling out each member of the dictionary -
> but how do you do that for a union?  So I see what you are doing:
> marking that this test currently passes the parser, but then causes
> problems for generating C code, so we should either reject it up front,
> or fix the generator.  The FIXME documents what you will do later in the
> series (reject it up front) and the TODO documents what we can do down
> the road (fix the generator to allow it).

See also 32/47 - events have the same problem.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 32/47] qapi-event: Convert to QAPISchemaVisitor, fixing data with base

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3327 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Fixes events whose data is struct with base to include the struct's
> base members.  Test case is qapi-schema-test.json's event
> __org.qemu_x-command:
> 
>     { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
> 
>     { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
>       'data': { '__org.qemu_x-member2': 'str' } }
> 
>     { 'struct': '__org.qemu_x-Base',
>       'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }

No change to generated code in qemu proper, so this is a corner case we
are not yet exploiting. But good to have it fixed :)

> 
> Patch's effect on generated qapi_event_send___org_qemu_x_event():
> 
>     -void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2,
>     +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
>     +                                        const char *__org_qemu_x_member2,
>                                              Error **errp)

Ouch - I think we have a bug in qapi.py:check_event().  There, we call
check_type(... allow_metas=['union', 'struct']) - but it looks like the
generated signature requires that we have no variants, which means we
cannot have:

{ 'union': 'Un', 'data': ... }
{ 'event': 'EV', 'data': 'Un' }

because it would fail C generation. Sounds like you should add a check
to that in 14/47, and a fix for it in 15/47.


> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-event.py                   | 87 ++++++++++++++++-----------------
>  tests/qapi-schema/qapi-schema-test.json |  3 --
>  2 files changed, 43 insertions(+), 47 deletions(-)
> 
> diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
> index 537da17..456e590 100644
> --- a/scripts/qapi-event.py
> +++ b/scripts/qapi-event.py
> @@ -2,28 +2,29 @@
>  # QAPI event generator
>  #
>  # Copyright (c) 2014 Wenchao Xia
> +# Copyright (c) 2013-2015 Red Hat Inc.

Umm - the file didn't exist until 2014; and to my knowledge, you aren't
adding anything to it that was copied from some other file dating back
to 2013.  Using the range 2014-2015 might be better.

But that's minor.  And assuming that you reject union types for events
in an earlier patch, the rest of this is fine:
Reviewed-by: Eric Blake <eblake@redhat.com>

> @@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params):
>  """,
>                  event_name = event_name)
>  
> -        for argname, argentry, optional in parse_args(params):
> -            if optional:
> +        for memb in params.members:
> +            if memb.optional:
>                  ret += mcgen("""
>      if (has_%(var)s) {
>  """,
> -                             var = c_name(argname))
> +                             var=c_name(memb.name))
>                  push_indent()
>  
> -            if argentry == "str":
> +            if memb.type.name == "str":
>                  var_type = "(char **)"

Our visitors require us to cast away const.  Is that something we should
consider reworking, so that we don't need to do that?  But it's a
question for another day.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 33/47] qapi: Clean up after recent conversions to QAPISchemaVisitor

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 4012 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Drop helper functions that are now unused.
> 
> Make pylint reasonably happy.
> 
> Rename generate_FOO() functions to gen_FOO() for consistency.
> 
> Use more consistent and sensible variable names.
> 
> Consistently use c_ for mapping keys when their value is a C
> identifier or type.
> 
> Simplify gen_enum() and gen_visit_union()
> 
> Consistently use single quotes for C text string literals.

Not sure if it is worth splitting this into pieces.  Fortunately, there
are no changes to generated output.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 109 +++++++++++++++++++------------------
>  scripts/qapi-event.py    | 117 +++++++++++++++++++---------------------
>  scripts/qapi-types.py    |  68 ++++++++++++-----------
>  scripts/qapi-visit.py    | 121 ++++++++++++++++++++---------------------
>  scripts/qapi.py          | 138 +++++++++--------------------------------------
>  5 files changed, 229 insertions(+), 324 deletions(-)
> 
> diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
> index d3bddb6..5d11032 100644
> --- a/scripts/qapi-commands.py
> +++ b/scripts/qapi-commands.py
> @@ -15,20 +15,20 @@
>  from qapi import *
>  import re
>  
> -def generate_command_decl(name, args, ret_type):
> -    arglist=""
> +def gen_command_decl(name, args, rets):

I can see how 'args' is plural (even if it is a single string for the
name of a type containing the args), but should it be 'ret' instead of
'rets'?

> @@ -40,34 +40,37 @@ if (%(err)s) {
>  ''',
>                   err=err)
>  
> -def gen_sync_call(name, args, ret_type):
> -    ret = ""
> -    arglist=""
> -    retval=""
> -    if ret_type:
> -        retval = "retval = "
> +def gen_call(name, args, rets):

At least you're consistent on naming it 'rets',

> +    ret = ''

and the naming lets you distinguish between the parameter (the type
describing returned fields) and the local string (the generated C code
holding the return information).

> @@ -82,45 +76,46 @@ def generate_event_implement(api_name, event_name, params):

>  
> +            # Ugly: need to cast away the const
>              if memb.type.name == "str":
> -                var_type = "(char **)"
> +                cast = "(char **)"

And to think I called it out in a previous patch. So you noticed it too :)

Don't you want to use '(char **)' here, since it is a literal string
destined for generated C?

>              else:
> -                var_type = ""
> +                cast = ""

and '' here?

> +++ b/scripts/qapi-types.py
> @@ -16,22 +16,22 @@ from qapi import *
>  def gen_fwd_object_or_array(name):
>      return mcgen('''
>  
> -typedef struct %(name)s %(name)s;
> +typedef struct %(c_name)s %(c_name)s;
>  ''',
> -                 name=c_name(name))
> +                 c_name=c_name(name))
>  
>  def gen_array(name, element_type):
>      return mcgen('''
>  
> -struct %(name)s {
> +struct %(c_name)s {
>      union {
>          %(c_type)s value;
>          uint64_t padding;
>      };
> -    struct %(name)s *next;
> +    struct %(c_name)s *next;

May be some churn here if you like my comment earlier in the series that
this 'struct' is redundant.

> +++ b/scripts/qapi-visit.py

> -def generate_visit_struct_fields(name, members, base = None):
> +def gen_visit_struct_fields(name, base, members):
>      struct_fields_seen.add(name)

> -                     type=base.c_name(), c_name=c_name('base'))
> +                     c_type=base.c_name(), c_name=c_name('base'))

Possible churn here based on my earlier comments about c_name(constant)
being constant.

Fairly big, but aside from some minor '' quoting issues, I didn't see
anything wrong.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 34/47] qapi-visit: Rearrange code a bit

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 492 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Move gen_visit_decl() to a better place.  Inline
> generate_visit_struct_body().
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-visit.py | 42 ++++++++++++++++--------------------------
>  1 file changed, 16 insertions(+), 26 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 35/47] qapi-commands: Rearrange code

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1367 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Rename gen_marshal_input() to gen_marshal(), because the generated
> function marshals both arguments and results.
> 
> Rename gen_visitor_input_containers_decl() to gen_marshal_vars(), and
> move the other variable declarations there, too.
> 
> Rename gen_visitor_input_block()() to gen_marshal_input_visit(), and

Double ().

> rearrange its code slightly.
> 
> Rename gen_marshal_input_decl() to gen_marshal_proto(), because the
> result isn't a full declaration, unlike gen_command_decl()'s.
> 
> New gen_marshal_decl() actually returns a full declaration.

No change to generated code; good.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 95 ++++++++++++++++++++++--------------------------
>  1 file changed, 43 insertions(+), 52 deletions(-)
> 

>      push_indent()
> +
> +    if rets:
> +        # FIXME fishy: only pointers are initialized
> +        if rets.c_null() == 'NULL':
> +            retval = '%s retval = NULL;' % rets.c_type()
> +        else:
> +            retval = '%s retval;' % rets.c_type()

May still need tweaking based on my earlier reviews.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 36/47] qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> These functions marshal both input and output.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  docs/qapi-code-gen.txt        |   4 +-
>  docs/writing-qmp-commands.txt |   8 +-
>  monitor.c                     |   2 +-
>  qmp-commands.hx               | 240 +++++++++++++++++++++---------------------
>  scripts/qapi-commands.py      |   4 +-
>  5 files changed, 129 insertions(+), 129 deletions(-)

Diffstat here says it is a straight rename; likewise the diff on the
generated code:

 qga/qapi-generated/qga-qmp-marshal.c |  116 ++++++++--------
 qmp-commands.h                       |  240
+++++++++++++++++------------------
 qmp-marshal.c                        |  240
+++++++++++++++++------------------
 3 files changed, 298 insertions(+), 298 deletions(-)

Missing: you overlooked a comment in qmp.c line 154:

#ifndef CONFIG_SPICE
/*
 * qmp-commands.hx ensures that QMP command query-spice exists only
 * #ifdef CONFIG_SPICE.  Necessary for an accurate query-commands
 * result.  However, the QAPI schema is blissfully unaware of that,
 * and the QAPI code generator happily generates a dead
 * qmp_marshal_input_query_spice() that calls qmp_query_spice().
 * Provide it one, or else linking fails.
 * FIXME Educate the QAPI schema on CONFIG_SPICE.
 */


With that fixed,
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 37/47] qapi: De-duplicate parameter list generation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 4319 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Generated qapi-event.[ch] lose line breaks.  No change otherwise.

For example,

-void qapi_event_send_block_image_corrupted(const char *device,
-                                           bool has_node_name,
-                                           const char *node_name,
-                                           const char *msg,
-                                           bool has_offset,
-                                           int64_t offset,
-                                           bool has_size,
-                                           int64_t size,
-                                           bool fatal,
-                                           Error **errp)
+void qapi_event_send_block_image_corrupted(const char *device, bool
has_node_name, const char *node_name, const char *msg, bool has_offset,
int64_t offset, bool has_size, int64_t size, bool fatal, Error **errp)

You know, I'd find it a bit more appealing if you had merged the
duplicate code in the _other_ direction. That is, qapi-event's wrapped
lines (usually) fit in 80 columns, and it would be nice if qapi-visit's
did the same.

Yeah, avoiding line wraps consumes fewer source bytes (fewer runs of
spaces), but the space isn't being wasted by storing generated files in
git, nor does the C compiler care which layout we use.  And honestly,
it's easier to spot changes in a vertical list than it is on a long
horizontal line, if a parameter gets added (or removed, although adding
is the more likely action with qapi).

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 11 ++---------
>  scripts/qapi-event.py    | 18 +++---------------
>  scripts/qapi.py          | 16 ++++++++++++++++
>  3 files changed, 21 insertions(+), 24 deletions(-)

I'm a fan of de-duplication, so I'll review this on its merits; but I'm
omitting R-b on this round in hopes that you buy my argument to merge in
the other direction (make qapi-event's implementation the common one).

> 
> diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
> index d57f8d4..2dae425 100644
> --- a/scripts/qapi-commands.py
> +++ b/scripts/qapi-commands.py

> -                 args=argstr)
> +                 params=gen_params(args, 'Error **errp'))

Caller 1.

> +++ b/scripts/qapi-event.py

> +    return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
> +        'c_name': c_name(name.lower()),
> +        'param': gen_params(data, 'Error **errp')}

Caller 2.

>  
>  def gen_event_send_decl(name, data):
>      return mcgen('''
> diff --git a/scripts/qapi.py b/scripts/qapi.py
> index 4d47214..c6a5ddc 100644
> --- a/scripts/qapi.py
> +++ b/scripts/qapi.py
> @@ -1384,6 +1384,22 @@ extern const char *const %(c_name)s_lookup[];
>                   c_name=c_name(name))
>      return ret
>  
> +def gen_params(args, extra):
> +    if not args:
> +        return extra

Both callers pass the same 'extra' - do you need it to be parameterized,
or can it just be generated as a constant here?  (I guess it depends on
what happens with the later introspection patch, which may become caller 3).

> +    assert not args.variants

This assert will trip if you don't fix events to reject 'data':'Union' :)

> +    ret = ""
> +    sep = ""
> +    for memb in args.members:
> +        ret += sep
> +        sep = ", "
> +        if memb.optional:
> +            ret += "bool has_%s, " % c_name(memb.name)

Didn't you just provide a patch that used '' rather than "" for all
generated C constructs?  This violates that paradigm.

> +        ret += "%s %s" % (memb.type.c_type(is_param=True), c_name(memb.name))
> +    if extra:
> +        ret += sep + extra
> +    return ret
> +

To produce line breaks, you could have to add a parameter so that
callers can pass in the starting column for each wrapped argument, and
then you'd have sep = ',\n' + ''.ljust(len).  Or even have the caller
choose its own separator (", " vs. ",\n    "), if you don't want to have
a diff in the generated output (but I think consistent generated output
is nicer).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 38/47] qapi-commands: De-duplicate output marshaling functions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1994 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> gen_marshal_output() uses its parameter name only for name of the
> generated function.  Name it after the type being marshaled instead of
> its caller, and drop duplicates.
> 
> Saves 7 copies of qmp_marshal_output_int() in qemu-ga, and one copy of
> qmp_marshal_output_str() in qemu-system-*.

 qga/qapi-generated/qga-qmp-marshal.c |  227
+++++------------------------------
 qmp-marshal.c                        |  225
+++++++++++++++-------------------
 2 files changed, 134 insertions(+), 318 deletions(-)

Most changes look like:

-static void qmp_marshal_output_add_fd(AddfdInfo *ret_in, QObject
**ret_out, Error **errp)
+static void qmp_marshal_output_AddfdInfo(AddfdInfo *ret_in, QObject
**ret_out, Error **errp)
 {
     Error *local_err = NULL;
     QmpOutputVisitor *mo = qmp_output_visitor_new();
@@ -88,7 +88,7 @@ void qmp_marshal_add_fd(QDict *args, QOb
         goto out;
     }

-    qmp_marshal_output_add_fd(retval, ret, &local_err);
+    qmp_marshal_output_AddfdInfo(retval, ret, &local_err);

coupled with wholesale deletions of functions that previously had
identical bodies.  Nice.

[I suspect there might be ways to trim a LOT more code size, by
rewriting a generic marshaller helper that is passed a varargs or
array-of-struct list of operations to perform in order to visit an
arbitrary object, then having each command's marshaller generated with
the appropriate list of arguments for the generic helpers rather than
the current approach of calling out to one marshaller helper per type -
but exploring ideas like that is work for another series]

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-commands.py | 16 +++++++++-------
>  1 file changed, 9 insertions(+), 7 deletions(-)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 39/47] qapi: Improve built-in type documentation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Clarify how they map to JSON.  Add how they map to C.  Fix the
> reference to StringInputVisitor.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  docs/qapi-code-gen.txt | 29 ++++++++++++++++++-----------
>  1 file changed, 18 insertions(+), 11 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 40/47] qapi: Introduce a first class 'any' type

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 4930 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> It's first class, because unlike '**', it actually works, i.e. doesn't
> require 'gen': false.

Generated code grows in order to accommodate visiting the new anyList type:

 qapi-types.c                        |   15 +++++++++++++++
 qapi-types.h                        |   13 +++++++++++++
 qapi-visit.c                        |   24 ++++++++++++++++++++++++
 qapi-visit.h                        |    1 +
 qga/qapi-generated/qga-qapi-types.h |   13 +++++++++++++
 qga/qapi-generated/qga-qapi-visit.h |    1 +
 6 files changed, 67 insertions(+)

But do we really need anyList as a representation of qapi ['any'] ?  Or
will a later pass that minimizes array creation only to places where it
is needed help?

> 
> '**' will go away next.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  docs/qapi-code-gen.txt            |  1 +
>  include/qapi/visitor-impl.h       |  2 ++
>  include/qapi/visitor.h            |  1 +
>  qapi/qapi-dealloc-visitor.c       |  9 +++++++
>  qapi/qapi-visit-core.c            |  6 +++++
>  qapi/qmp-input-visitor.c          | 11 ++++++++
>  qapi/qmp-output-visitor.c         |  9 +++++++
>  scripts/qapi-types.py             |  1 +
>  scripts/qapi.py                   |  9 ++++---
>  tests/qapi-schema/type-bypass.out |  4 +--
>  tests/test-qmp-input-visitor.c    | 45 +++++++++++++++++++++++++++++++++
>  tests/test-qmp-output-visitor.c   | 53 +++++++++++++++++++++++++++++++++++++++
>  12 files changed, 146 insertions(+), 5 deletions(-)

Touches a bit more this time, but that's okay.

> 
> diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
> index cb0fe75..bf9e854 100644
> --- a/docs/qapi-code-gen.txt
> +++ b/docs/qapi-code-gen.txt
> @@ -158,6 +158,7 @@ The following types are predefined, and map to C as follows:
>    size      uint64_t   like uint64_t, except StringInputVisitor
>                         accepts size suffixes
>    bool      bool       JSON true or false
> +  any       QObject *  any JSON value

And with this, an 'alternate' type is just a special subset of 'any'
that limits the set of valid JSON values according to the qapi description.

> +++ b/qapi/qapi-visit-core.c
> @@ -260,6 +260,12 @@ void visit_type_number(Visitor *v, double *obj, const char *name, Error **errp)
>      v->type_number(v, obj, name, errp);
>  }
>  
> +void visit_type_any(Visitor *v, QObject **obj, const char *name,
> +                         Error **errp)

Indentation looks off.

> @@ -1048,8 +1048,9 @@ class QAPISchema(object):
>                    ('uint64', 'int',     'uint64_t', '0'),
>                    ('size',   'int',     'uint64_t', '0'),
>                    ('bool',   'boolean', 'bool',     'false'),
> -                  ('**',     'value',   None,       None)]:
> +                  ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
>              self._def_builtin_type(*t)
> +        self.entity_dict['**'] = self.lookup_type('any') # TODO drop this alias
>  
>      def _make_implicit_enum_type(self, name, values):
>          name = name + 'Kind'
> @@ -1190,6 +1191,8 @@ class QAPISchema(object):
>      def visit(self, visitor):
>          visitor.visit_begin()
>          for name in sorted(self.entity_dict.keys()):
> +            if self.entity_dict[name].name != name:
> +                continue        # ignore alias TODO drop alias and remove

Nice back-compat hacks while you transition into using it.

> +++ b/tests/test-qmp-input-visitor.c
> @@ -298,6 +298,49 @@ static void test_visitor_in_list(TestInputVisitorData *data,
>      qapi_free_UserDefOneList(head);
>  }
>  
> +static void test_visitor_in_any(TestInputVisitorData *data,
> +                                const void *unused)
> +{

> +
> +    v = visitor_input_test_init(data, "-42");

So we prove it accepts ints,...

> +    v = visitor_input_test_init(data, "{ 'integer': -42, 'boolean': true, 'string': 'foo' }");

objects,...

> +    visit_type_any(v, &res, NULL, &err);
> +    g_assert(!err);
> +    qdict = qobject_to_qdict(res);
> +    g_assert(qdict);

worth asserting the dictionary has 3 keys?

> +}

...but no test of string, boolean, or array?

> +++ b/tests/test-qmp-output-visitor.c
> @@ -428,6 +428,57 @@ static void test_visitor_out_list_qapi_free(TestOutputVisitorData *data,
>      qapi_free_UserDefTwoList(head);
>  }
>  
> +static void test_visitor_out_any(TestOutputVisitorData *data,
> +                                 const void *unused)
> +{

> +}

Same tests of 'int' and 'object' but not of other JSON types.

Incomplete tests are still better than no tests, so what you have is a
good start.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 41/47] qom: Don't use 'gen': false for qom-get, qom-set, object-add

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2468 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> With the previous commit, the generated marshalers just work, and save
> us a bit of handwritten code.
> 

Generated code grows, because you are now generating the hook :)

 qmp-commands.h |    6 ++
 qmp-marshal.c  |  144
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 150 insertions(+)


> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  include/monitor/monitor.h |  3 ---
>  qapi-schema.json          |  9 +++------
>  qmp-commands.hx           |  6 +++---
>  qmp.c                     | 20 +++++++-------------
>  scripts/qapi.py           |  1 +
>  5 files changed, 14 insertions(+), 25 deletions(-)
> 

> +++ b/qmp.c
> @@ -229,11 +229,9 @@ ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
>  }
>  
>  /* FIXME: teach qapi about how to pass through Visitors */
> -void qmp_qom_set(QDict *qdict, QObject **ret, Error **errp)
> +void qmp_qom_set(const char *path, const char *property, QObject *value,
> +                 Error **errp)

The FIXME seems stale now.

>  
> -void qmp_object_add(QDict *qdict, QObject **ret, Error **errp)
> +void qmp_object_add(const char *type, const char *id,
> +                    bool has_props, QObject *props, Error **errp)
>  {
> -    const char *type = qdict_get_str(qdict, "qom-type");
> -    const char *id = qdict_get_str(qdict, "id");
> -    QObject *props = qdict_get(qdict, "props");
>      const QDict *pdict = NULL;
>      QmpInputVisitor *qiv;
>  

Continuing:

    if (props) {
        pdict = qobject_to_qdict(props);
        if (!pdict) {
            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
            return;
        }
    }

I know that we guarantee that all pointers are initialized to NULL in
our marshaler, so this is correct; but wouldn't it be more idiomatic to
write 'if (has_props)' as the condition?

(And it would make a nice followup project for someone to figure out how
to get rid of have_FOO arguments for strings and objects where NULL is a
nice witness; it is only integers and booleans that require them - but
doing that will touch a lot of the tree, and is a series of its own)

What I pointed out is minor, so you can fix it and add:
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 42/47] qapi-schema: Fix up misleading specification of netdev_add

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3359 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> It doesn't take a 'props' argument, let alone one in the format
> "NAME=VALUE,..."
> 
> The bogus arguments specification doesn't matter due to 'gen': false.
> Clean it up to be incomplete rather than wrong, and document the
> incompleteness.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  docs/qapi-code-gen.txt |  2 +-
>  qapi-schema.json       | 13 +++++++------
>  2 files changed, 8 insertions(+), 7 deletions(-)
> 

> +++ b/qapi-schema.json
> @@ -2062,11 +2062,12 @@
>  #
>  # @id: the name of the new network backend
>  #
> -# @props: #optional a list of properties to be passed to the backend in
> -#         the format 'name=value', like 'ifname=tap0,script=no'
> +# Additional arguments depend on the type.

In other words, this would be a perfect candidate to write as taking a
flat union as the argument, where 'type' is the discriminator, and where
the additional arguments are branches of the union.  Except that we
don't yet support passing a union as the direct "arguments":{...} of a
QMP call, so we still have some design work to do if we want to go
there.  Later, of course.

>  #
> -# Notes: The semantics of @props is not well defined.  Future commands will be
> -#        introduced that provide stronger typing for backend creation.
> +# TODO This command effectively bypasses QAPI completely due to its
> +# "additional arguments" business.  It shouldn't have been added to
> +# the schema in this form.  It should be qapified properly, or
> +# replaced by a properly qapified command.

I like that you are getting rid of the sentence about strong typing
(since flat unions ARE strongly typed) and instead couching it in terms
of an incomplete qapi description.  I think we may even be able to come
up with a way to express it without needing a new command, and without
breaking back-compat, once we figure out how to pass unions as a command
argument.

In fact, let's suppose we do have a new union type, NetdevInfo, as well
as a way to flag that a given command will be called by the marshaler
using JUST a pointer to the struct, rather than as a list of parameters
to members of the struct.

{ 'enum': 'NetdevTypes', [ ... ] }
{ 'struct': 'NetdevInfoBase', 'data':
   { 'type': 'NetdevTypes', 'id': 'str' } }
{ 'union': 'NetdevInfo', 'base': 'NetdevInfoBase',
  'discriminator': 'type', 'data': { ... } }
{ 'command': 'netdev_add', 'boxed': true,
  'data': 'NetdevInfo' }

resulting in the marshaller calling

void qmp_netdev_add(NetdevInfo *data, Error **errp)


Here, 'boxed' is an optional key to each 'command', defaulting to false;
but when true, 'data' is passed in a box to the command called by the
marshaler, instead of broken out into fields. 'boxed' must be true for
unions, but may be true elsewhere.

And if we have that, then OTHER commands that have complained about
super-long parameter lists when their struct is exploded into a
parameter list could also use this mechanism, to write their handler in
terms of a single pointer to the struct.

Enough rambling about stuff for the future. This patch is fine as-is;
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 43/47] qmp: Improve netdev_add usage example in the manual

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Add a device option to show how it's done.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  qmp-commands.hx | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Perhaps could be squashed with the previous patch.  Or not - up to you.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 44/47] qapi: Pseudo-type '**' is now unused, drop it

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3902 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> 'gen': false needs to stay for now, because netdev_add is still using
> it.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  docs/qapi-code-gen.txt                               | 15 ++++-----------
>  scripts/qapi.py                                      | 20 ++++----------------
>  tests/Makefile                                       |  4 ++--
>  tests/qapi-schema/args-returns-any.err               |  1 +
>  ...type-bypass-no-gen.exit => args-returns-any.exit} |  0

> 
> diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
> index 2367c66..ca578dd 100644
> --- a/docs/qapi-code-gen.txt
> +++ b/docs/qapi-code-gen.txt

> @@ -457,13 +454,9 @@ which would validate this Client JSON Protocol transaction:
>  
>  In rare cases, QAPI cannot express a type-safe representation of a
>  corresponding Client JSON Protocol command.  In these cases, if the
> -command expression includes the key 'gen' with boolean value false,
> -then the 'data' or 'returns' member that intends to bypass generated
> -type-safety and do its own manual validation should use an inline
> -dictionary definition, with a value of '**' rather than a valid type
> -name for the keys that the generated code will not validate.  Please
> -try to avoid adding new commands that rely on this, and instead use
> -type-safe unions.  For an example of bypass usage:
> +command expression includes the key 'gen' with boolean value false.

Incomplete sentence.  "if the command expression includes... false"
needs "then something...".

> +Please try to avoid adding new commands that rely on this, and instead
> +use type-safe unions.  For an example of bypass usage:

And given my ideas, we may be able to make netdev_add typesafe and
eliminate 'gen' altogether in the future :)

> +++ b/tests/Makefile
> @@ -222,11 +222,11 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
>  	bad-type-dict.json double-data.json unknown-expr-key.json \
>  	redefined-type.json redefined-command.json redefined-builtin.json \
>  	redefined-event.json command-int.json bad-data.json event-max.json \
> -	type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \
> +	type-bypass-bad-gen.json \

If I understand correctly (git didn't quite guess right),
type-bypass-no-gen.json disappears completely (we no longer have working
'**'), and type-bypass.json...

>  	data-array-empty.json data-array-unknown.json data-int.json \
>  	data-unknown.json data-member-unknown.json data-member-array.json \
>  	data-member-array-bad.json args-union.json args-alternate.json \
> -	returns-array-bad.json returns-int.json \
> +	args-returns-any.json returns-array-bad.json returns-int.json \

...gets renamed args-returns-any.json (because type bypass is now
expressed via 'any'.

> +# built-in type 'any' in arguments and returns
> +# works except for 'data': 'any', because that has to be a struct
> +# note: command name 'qom-get' chosen to avoid "cannot use built-in" error
> +{ 'command': 'qom-get', 'data': { 'arg': 'any' }, 'returns': 'any' }
> +{ 'command': 'foo', 'returns': { 'arg': 'any' } }
> +{ 'command': 'bar', 'data': 'any' }

This shows what the parser accepts and rejects, but doesn't prove it
compiles to C; and the previous patch that touched
test-qmp-input-visitor.c was testing that the low-level interactions
worked but did not map it all the way back to qapi.  I wonder if you
should also enhance qapi-schema-test to use 'any', since that tends to
be the one test that most fully exercises interactions with generated
code based on a valid qapi schema.


On the right track, but I want to make sure the botched documentation is
correct before giving R-b.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 45/47] qapi: New QMP command query-schema for QMP schema introspection

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 31281 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Caution, rough edges.

No joke. It doesn't even compile without this fixup to a rebase snafu
(see [0] below):

diff --git i/scripts/qapi-types.py w/scripts/qapi-types.py
index 79e8d24..12f3767 100644
--- i/scripts/qapi-types.py
+++ w/scripts/qapi-types.py
@@ -184,6 +184,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.fwdecl = None
         self.fwdefn = None
         self.btin = None
+    def visit_begin(self, schema):
         self.decl = ''
         self.defn = ''
         self.fwdecl = ''

I already know you'll be editing this further, but here's some things to
look for.

> 
> qapi/introspect.json defines the introspection schema.  It should do
> for uses other than QMP.
> FIXME it's almost entirely devoid of comments.

It generates quite a bit of code to support itself :)

 qapi-types.c |  383 ++++++++++++++++++++++++++++
 qapi-types.h |  293 +++++++++++++++++++++
 qapi-visit.c |  797
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 qapi-visit.h |   26 +
 4 files changed, 1499 insertions(+)


> 
> The introspection schema does not reflect all the rules and
> restrictions that apply to QAPI schemata.  A valid QAPI schema has an
> introspection value conforming to the introspection schema, but the
> converse is not true.
> 
> Introspection lowers away a number of schema details:
> 
> * The built-in types are declared with their JSON type.
> 
>   TODO Should we map all the integer types to just int?

Just int for now seems fine.  It's easier to add refinement later when
we have a proven need, than it is to be stuck advertising too much
information now and being stuck with it.

> 
> * Implicit type definitions are made explicit, and given
>   auto-generated names.  These names start with ':' so they don't
>   clash with the user's names.
> 
>   Example: a simple union implicitly defines an enumeration type for
>   its discriminator.
> 
> * All type references are by name.
> 
> * Base types are flattened.
> 
> * The top type (named 'any') can hold any value.
> 
> * The struct and union types are generalized into an object type.

You've mentioned that idea on list quite some time ago, and it appears
to be working well.

> 
> * Commands take a single argument and return a single result.
> 
>   Dictionary argument/result or list result is an implicit type
>   definition.
> 
>   The empty object type is used when a command takes no arguments or
>   produces no results.
> 
>   The argument is always of object type, but the introspection schema
>   doesn't reflect that.

And if we ever change QMP to allow non-objects for the arguments,
introspection will still work.

> 
>   The 'gen': false and 'success-response': false directives are
>   omitted as implementation detail.

I understand omitting 'gen' (hmm, I though we could get rid of it by
making netdev_add typesafe, but now this patch adds another 'gen':false
and for good reason).  But 'success-response' is part of the ABI; we
already advertise it via qga's 'guest-info' command exposing
'success-response':'bool' for each command.  Adding it would allow a
single introspection command to learn everything, instead of having to
pair introspection with 'guest-info'.  On the other hand, as above, it's
easier to start thin and add more later if needed, than it is to start
full and then have to support it forever.

> 
> * Events carry a single data value.
> 
>   Implicit type definition and empty object type use, just like for
>   commands.
> 
>   The value is of object type, but the introspection schema doesn't
>   reflect that.
> 
> * Types not used by commands or events are omitted.
> 
>   Indirect use counts as use.
> 
> * Optional members have a default, which can only be null right now
> 
>   Instead of a mandatory "optional" flag, we have an optional default.
>   No default means mandatory, default null means optional without
>   default value.  Non-null is available for optional with default.
> 
>   Alternate members can't have defaults, but the introspection schema
>   doesn't reflect that.
> 
> * Clients should *not* look up types by name, because type names are
>   not ABI.  Look up the command or event you're interested in, then
>   follow the references.
> 
>   TODO Should we hide the type names to eliminate the temptation?

Might be worth doing; I see you have a later patch to experiment with it.

> 
> * Likewise, the names of alternate members are not ABI, and should not
>   be examined.
> 
>   TODO Should we hide them, too?

How, by making the name of an object member optional?

> 
> TODO much of the above should go into docs.

Indeed.  Hence why this is RFC still.

> 
> New generator scripts/qapi-introspect.py computes an introspection
> value for its input, and generates a C variable holding it.
> 
> FIXME it can generate awfully long lines

We already have long lines in generated output, but I agree that finding
ways to break it up might be nice.  Actually,

> 
> A new test-qmp-input-visitor test case feeds its result for both
> tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a
> QmpInputVisitor to verify it actually conforms to the schema.
> 
> New QMP command query-schema takes its return value from that
> variable.  Command documentation is incomplete, and marked FIXME.  Its
> reply is some 80KiBytes for me right now.

$ ll qmp-introspect.[ch]
-rw-rw-r--. 1 eblake eblake 89655 Jul 23 17:20 qmp-introspect.c
-rw-rw-r--. 1 eblake eblake   358 Jul 23 17:20 qmp-introspect.h

> 
> If this turns out to be too much, we have a couple of options:
> 
> * We can use shorter names in the JSON.  Not the QMP style.
> 
> * Optionally return the sub-schema for commands and events given as
>   arguments.

Filtering is probably worth having, but I'm not sure how easy or hard it
gets...
> 
>   Right now qmp_query_schema() sends the string literal computed by
>   qmp-introspect.py.  To compute sub-schema at run time, we'd have to
>   duplicate parts of qapi-introspect.py in C.  Unattractive.

...might be simpler than that. You can do some pre-processing, so that
the C code can take shortcuts of having pre-computed dependency
information. Instead of one giant string, you instead have an array of
structs, one array entry per schema entity:

struct foo {
    const char *name;  /* example: ":empty" */
    int metatype;      /*   SCHEMA_META_TYPE_OBJECT */
    const char *json;  /*   "{ 'name': ':empty', 'meta-type': 'object',
'members': [  ] }, " */

    int ndepends;      /* length of depends array */
    int *depends;      /* indices of other entities this depends on */
    bool visited;      /* helper for filtering */
};

If the user requests filtering, you then do a pre-pass that sets
array[i]->visited = filter_match(), then an iterative pass that says for
every array[i]->visited, you also want to set
array[array[i]->depends[j]]->visited for each j up to
array[i]->ndepends; then finally produce output of "[" + each
array[i]->json where visited is true + "]".

Of course, adding filtering as a follow-on patch is the only way to go;
initial implementation is fine as global information.

> 
> * Let clients cache the output of query-schema.
> 
>   It changes only on QEMU upgrades, i.e. rarely.  Provide a command
>   query-schema-hash.  Clients can have a cache indexed by hash, and
>   re-query the schema only when they don't have it cached.

Libvirt already caches things.  I don't know if having qemu output a
hash helps libvirt, or if libvirt won't mind just eating the cost of
reading the entire array every time anything else causes it to
repopulate the cache (such as timestamp on qemu binary changing).  We
can play with it later.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---

Missing a change to docs/qapi-code-gen.txt describing the new script and
output.

>  30 files changed, 345 insertions(+), 12 deletions(-)
>  create mode 100644 qapi/introspect.json
>  create mode 100644 scripts/qapi-introspect.py
> 
> diff --git a/.gitignore b/.gitignore
> index aed0e1f..a6a02db 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -32,6 +32,7 @@
>  /qapi-visit.[ch]
>  /qapi-event.[ch]
>  /qmp-commands.h
> +/qmp-introspect.[ch]
>  /qmp-marshal.c
>  /qemu-doc.html
>  /qemu-tech.html

Missing an addition for the new tests/test-qmp-introspect.h file
leftover after 'make'.  On second thought, that belongs at [2] below.

> +++ b/monitor.c
> @@ -74,6 +74,7 @@
>  #include "block/qapi.h"
>  #include "qapi/qmp-event.h"
>  #include "qapi-event.h"
> +#include "qmp-introspect.h"
>  #include "sysemu/block-backend.h"
>  
>  /* for hmp_info_irq/pic */
> @@ -924,6 +925,20 @@ EventInfoList *qmp_query_events(Error **errp)
>      return ev_list;
>  }
>  
> +/*
> + * Minor hack: generated marshalling suppressed for this command
> + * ('gen': false in the schema) so we can simply send the JSON string
> + * instead of first parsing it with visit_type_SchemaInfoList() into a
> + * SchemaInfoList, then unparse it right back in the generated output
> + * marshaller, every time.
> + * Instead, we do it in test-qmp-input-visitor.c, just to make sure
> + * qapi-introspect.py's output actually conforms to the schema.
> + */
> +static void qmp_query_schema(QDict *qdict, QObject **ret_data, Error **errp)
> +{
> +    *ret_data = qobject_from_json(qmp_schema_json);

Works for me :)

> +++ b/qapi/introspect.json
> @@ -0,0 +1,69 @@
> +# -*- Mode: Python -*-
> +#
> +# QAPI introspection
> +#
> +# Copyright (C) 2015 Red Hat, Inc.
> +#
> +# Authors:
> +#  Markus Armbruster <armbru@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or later.
> +# See the COPYING file in the top-level directory.
> +
> +{ 'enum': 'SchemaMetaType',
> +  'data': [ 'builtin', 'enum', 'array', 'object', 'alternate',
> +            'command', 'event' ] }
> +
> +{ 'struct': 'SchemaInfoBase',
> +  'data': { 'name': 'str', 'meta-type': 'SchemaMetaType' } }
> +
> +{ 'enum': 'JSONType',
> +  'data': [ 'string', 'number', 'int', 'boolean', 'null',
> +            'object', 'array', 'value' ] }
> +
> +{ 'struct': 'SchemaInfoBuiltin',
> +  'data': { 'json-type': 'JSONType' } }
> +
> +{ 'struct': 'SchemaInfoEnum',
> +  'data': { 'values': ['str'] } }

Do we want to document anything about sort ordering of this list?  Is it
worth sorting the array by name, to allow clients to bsearch for whether
a particular enum value is supported, rather than having to linear search?

> +
> +{ 'struct': 'SchemaInfoArray',
> +  'data': { 'element-type': 'str' } }
> +
> +{ 'struct': 'SchemaInfoObjectMember',
> +  'data': { 'name': 'str', 'type': 'str', '*default': 'any' } }
> +# @default's type must match @type

or be null; as you mentioned above, this sets up a tri-state of
mandatory, optional with no default, and (for future extension) optional
with default.

> +
> +{ 'struct': 'SchemaInfoObjectVariant',
> +  'data': { 'case': 'str',
> +            'members': [ 'SchemaInfoObjectMember' ] } }

Would it be simpler to just have:

'data': { 'case': 'str', 'type': 'str' }

and make the user refer recursively to the (possibly-implicit) type for
the members?

In particular, if we ever decide to allow a flat union to have another
union as a branch, rather than the current restriction that all branches
must be structs, then referring to the type of a branch may be easier
than breaking out all members of a struct.  And if that's the case, it
may have knock-on simplifications to your earlier patches for tracking
variants. See [1] below for more thoughts...

Do we want to guarantee anything about the sort ordering in this list?

> +
> +{ 'struct': 'SchemaInfoObject',
> +  'data': { 'members': [ 'SchemaInfoObjectMember' ],
> +            '*tag': 'str',
> +            '*variants': [ 'SchemaInfoObjectVariant' ] } }

or these?

> +
> +{ 'struct': 'SchemaInfoAlternate',
> +  'data': { 'members': [ 'SchemaInfoObjectMember' ] } }

Here's an example of what you generated:
    "{ 'name': 'BlockdevRef', 'meta-type': 'alternate', 'members': [ {
'name': 'definition', 'type': 'BlockdevOptions' }, { 'name':
'reference', 'type': 'str' } ] }, "

I think you could get away with something simpler:

 'data': { 'types': [ 'str' ] }

as in:
 "{ 'name': 'BlockdevRef', 'meta-type': 'alternate', 'types': [
'BlockdevOptions', 'str' ] }, "

the only worry is whether we might want future extensions, where we'd
want additional information per element of that array, vs. being forced
to return two arrays in parallel (arrays of structs are more extensible
than arrays of strings).
Seems like this would be just a

> +
> +{ 'struct': 'SchemaInfoCommand',
> +  'data': { 'args': 'str', 'returns': 'str' } }

Again, we can add 'success-return':'bool' later, if desired.

> +
> +{ 'struct': 'SchemaInfoEvent',
> +  'data': { 'data': 'str' } }
> +
> +{ 'union': 'SchemaInfo',
> +  'base': 'SchemaInfoBase',
> +  'discriminator': 'meta-type',
> +  'data': {
> +      'builtin': 'SchemaInfoBuiltin',
> +      'enum': 'SchemaInfoEnum',
> +      'array': 'SchemaInfoArray',
> +      'object': 'SchemaInfoObject',
> +      'alternate': 'SchemaInfoAlternate',
> +      'command': 'SchemaInfoCommand',
> +      'event': 'SchemaInfoEvent' } }
> +
> +{ 'command': 'query-schema',
> +  'returns': [ 'SchemaInfo' ],
> +  'gen': false }                # just to simplify qmp_query_json()

with an optional 'data':{'*filter':['str']} as a possible future
extension.  Looks nice, in spite of being under-documented!


> +++ b/scripts/qapi-commands.py
> @@ -265,7 +265,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
>          self.defn = None
>          self.regy = None
>          self.visited_rets = None
> -    def visit_begin(self):
> +    def visit_begin(self, schema):

And again my python object-oriented newness is showing through; where I
guess all children have to update signatures to still be polymorphic to
a parent adding a parameter.

Might be worth separating the patch to add the schema parameter so that
there is less churn to the existing scripts - or even rebase the series
up-front to always use the schema parameter and not change the signature
here.

>          self.decl = ''
>          self.defn = ''
>          self.regy = ''
> @@ -273,7 +273,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
>      def visit_end(self):
>          if not middle_mode:
>              self.defn += gen_registry(self.regy)
> -            self.regy = None
> +        self.regy = None
> +        self.visited_rets = None

Is it worth squashing this reset into the patch where the visitor was
first written?

>      def visit_command(self, name, info, args, rets, gen, success_response):
>          if not gen:
>              return
> diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
> index 184a81f..71da7a9 100644
> --- a/scripts/qapi-event.py
> +++ b/scripts/qapi-event.py
> @@ -139,13 +139,14 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
>          self.decl = None
>          self.defn = None
>          self.event_names = None
> -    def visit_begin(self):
> +    def visit_begin(self, schema):
>          self.decl = ''
>          self.defn = ''
>          self.event_names = []
>      def visit_end(self):
>          self.decl += gen_enum(event_enum_name, self.event_names)
>          self.defn += gen_enum_lookup(event_enum_name, self.event_names)
> +        self.event_names = None

and again

>      def visit_event(self, name, info, data):
>          self.decl += gen_event_send_decl(name, data)
>          self.defn += gen_event_send(name, data)
> diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
> new file mode 100644
> index 0000000..e7efc4a
> --- /dev/null
> +++ b/scripts/qapi-introspect.py
> @@ -0,0 +1,159 @@
> +#
> +# QAPI introspection generator
> +#
> +# Copyright (C) 2015 Red Hat, Inc.
> +#
> +# Authors:
> +#  Markus Armbruster <armbru@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2.
> +# See the COPYING file in the top-level directory.
> +
> +from qapi import *
> +
> +class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
> +    def __init__(self):
> +        self.schema = None
> +        self.jsons = None
> +        self.used_types = None
> +        self.defn = None
> +        self.decl = None
> +
> +    def visit_begin(self, schema):
> +        self.schema = schema
> +        self.jsons = []
> +        self.used_types = []
> +        return QAPISchemaType   # don't visit types for now
> +
> +    def visit_end(self):
> +        # visit the types that are actually used
> +        for typ in self.used_types:
> +            typ.visit(self)
> +        self.jsons.sort()
> +        name = prefix + 'qmp_schema_json'
> +        self.decl = mcgen('''
> +extern char %(c_name)s[];

Missing 'const'

> +''',
> +                          c_name=c_name(name))
> +        self.defn = mcgen('''
> +char %(c_name)s[] = "["
> +    "%(c_jsons)s]";

And again. Also, I'd consider putting the "]" on its own line, like the
"[" was, so that you can more easily cut and paste individual lines of
generated output (but since JSON doesn't allow trailing comma, I guess
the last line is still always going to be special).

> +''',
> +                          c_name=c_name(name),
> +                          c_jsons=', "\n    "'.join(self.jsons))

Cool syntax :)

> +        self.schema = None
> +        self.jsons = None
> +        self.used_types = None
> +
> +    def _use_type(self, typ):
> +        if typ not in self.used_types:
> +            self.used_types.append(typ)
> +        return typ.name
> +
> +    def _gen_json(self, name, mtype, extra):
> +        self.jsons.append("{ 'name': '%s', 'meta-type': '%s', %s }"
> +                          % (name, mtype, extra))

Problem. We document that our QMP engine accepts 'single-quoted' input
as an extension to JSON, but that our output will always be
"double-quoted".  That means you _must_ write \" everywhere in the
generated C code. (Also, you set the style guide earlier of using ''
quoting around any text destined for generated C code)

self.jsons.append('''{ \"name\": \"%s\", \"meta-type\": \"%s\", %s }'''
                  % (name, mtype, extra))

(at least, I assume '''...\"...''' is nicer than '...\\\"...')

> +
> +    def _gen_members(self, members):
> +        return ("'members': [ "
> +                + ", ".join([self._gen_member(m) for m in members])
> +                + " ]")

and more choice-of-quotes issues.  Lots more below; I'll quit pointing
them out on this round of review.

> +
> +    def _gen_member(self, member):
> +        default = ''
> +        if member.optional:
> +            default = ", 'default': null"
> +        return "{ 'name': '%s', 'type': '%s'%s }" \
> +            % (member.name, self._use_type(member.type), default)
> +
> +    def _gen_variants(self, tag_name, variants):
> +        return ("'tag': '%s'" % tag_name
> +                + ", 'variants': [ "
> +                + ", ".join([self._gen_variant(v) for v in variants])
> +                + " ]")
> +
> +    def _gen_variant(self, variant):
> +        if variant.flat:
> +            members = self._gen_members(variant.type.members)
> +        else:
> +            members = "'members': [ { 'name': 'data', 'type': '%s' } ]" \
> +                      % self._use_type(variant.type)
> +        return "{ 'case': '%s', %s }" % (variant.name, members)

[1] Ah, so .flat is still in use here, to avoid having to create
implicit types everywhere.  But if we create implicit types for simple
unions, and just track variants by their case name/type instead of case
name/[members], it will allow us to have a union as a case branch (I
don't know that we need that much flexibility), and not have to worry
about exposing .flat everywhere.  It may even result in a smaller JSON
string (you'd have to play with it to know for sure).

> +
> +    def visit_builtin_type(self, name, info, json_type):
> +        self._gen_json(name, 'builtin',
> +                       "'json-type': '%s'" % json_type)
> +
> +    def visit_enum_type(self, name, info, values):
> +        self._gen_json(name, 'enum',
> +                       "'values': [ %s ]" % ", ".join(["'%s'" % v
> +                                                       for v in values]))
> +
> +    def visit_array_type(self, name, info, element_type):
> +        self._gen_json(name, 'array',
> +                       "'element-type': '%s'" % self._use_type(element_type))
> +
> +    def visit_object_type_flat(self, name, info, members, variants):
> +        extra = self._gen_members(members)
> +        if variants:
> +            extra += ", " + self._gen_variants(variants.tag_name or "type",
> +                                               variants.variants)
> +        self._gen_json(name, 'object', extra)
> +

Do we really need two types of object visitors, or can you reuse the
signature all the other visitors have?

> +    def visit_alternate_type(self, name, info, variants):
> +        self._gen_json(name, 'alternate',
> +                       self._gen_members(variants.variants))
> +
> +    def visit_command(self, name, info, args, rets, gen, success_response):
> +        args = args or self.schema.the_empty_object_type
> +        rets = rets or self.schema.the_empty_object_type
> +        self._gen_json(name, 'command',
> +                       "'args': '%s', 'returns': '%s'" \
> +                       % (self._use_type(args), self._use_type(rets)))
> +
> +    def visit_event(self, name, info, data):
> +        data = data or self.schema.the_empty_object_type
> +        self._gen_json(name, 'event', "'data': '%s'" % self._use_type(data))
> +
> +(input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
> +

> +++ b/scripts/qapi-types.py
> @@ -184,7 +184,6 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
>          self.fwdecl = None
>          self.fwdefn = None
>          self.btin = None
> -    def visit_begin(self):
>          self.decl = ''

Rebase botch mentioned above [0]

>          self.defn = ''
>          self.fwdecl = ''
> diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
> index 2813bb3..7a03292 100644
> --- a/scripts/qapi-visit.py
> +++ b/scripts/qapi-visit.py
> @@ -326,7 +326,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
>          self.decl = None
>          self.defn = None
>          self.btin = None
> -    def visit_begin(self):
> +    def visit_begin(self, schema):
>          self.decl = ''
>          self.defn = ''
>          self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
> diff --git a/scripts/qapi.py b/scripts/qapi.py
> index 6ddb33e..7f9a159 100644
> --- a/scripts/qapi.py
> +++ b/scripts/qapi.py
> @@ -764,7 +764,7 @@ class QAPISchemaEntity(object):
>          pass
>  
>  class QAPISchemaVisitor(object):
> -    def visit_begin(self):
> +    def visit_begin(self, schema):

I don't know enough python to know if making schema optional in the
parent class affects what the child class is allowed to implement while
still overriding things.

>          pass
>      def visit_end(self):
>          pass
> @@ -776,6 +776,8 @@ class QAPISchemaVisitor(object):
>          pass
>      def visit_object_type(self, name, info, base, members, variants):
>          pass
> +    def visit_object_type_flat(self, name, info, members, variants):
> +        pass
>      def visit_alternate_type(self, name, info, variants):
>          pass
>      def visit_command(self, name, info, args, rets, gen, success_response):
> @@ -892,6 +894,8 @@ class QAPISchemaObjectType(QAPISchemaType):
>      def visit(self, visitor):
>          visitor.visit_object_type(self.name, self.info,
>                                    self.base, self.local_members, self.variants)
> +        visitor.visit_object_type_flat(self.name, self.info,
> +                                       self.members, self.variants)

Instead of having two object visitor signatures, would it be worth
having a boolean parameter to QAPISchema.visit() that says whether to
pass base, members as (base type, local members) vs. (None, all members)?

>  
>  class QAPISchemaObjectTypeMember(object):
>      def __init__(self, name, typ, optional):
> @@ -1042,6 +1046,9 @@ class QAPISchema(object):
>                    ('bool',   'boolean', 'bool',     'false'),
>                    ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
>              self._def_builtin_type(*t)
> +        self.the_empty_object_type = QAPISchemaObjectType(':empty', None, None,
> +                                                          [], None)

Cool global. Worth adding in a separate patch?

> +        self._def_entity(self.the_empty_object_type)
>  
>      def _make_implicit_enum_type(self, name, values):
>          name = name + 'Kind'
> @@ -1180,9 +1187,10 @@ class QAPISchema(object):
>              ent.check(self)
>  
>      def visit(self, visitor):
> -        visitor.visit_begin()
> +        ignore = visitor.visit_begin(self)
>          for name in sorted(self.entity_dict.keys()):
> -            self.entity_dict[name].visit(visitor)
> +            if not ignore or not isinstance(self.entity_dict[name], ignore):
> +                self.entity_dict[name].visit(visitor)

So this lets introspection bypass visiting types on the first pass, and
then collect used types during visit_end().  It means you can only
bypass a single metatype, but that is sufficient for your use; I don't
know if there is any better idiom for this paradigm.

>          visitor.visit_end()
>  
>  #
> diff --git a/tests/.gitignore b/tests/.gitignore
> index dc813c2..dda86cc 100644
> --- a/tests/.gitignore
> +++ b/tests/.gitignore
> @@ -19,6 +19,7 @@ test-opts-visitor
>  test-qapi-event.[ch]
>  test-qapi-types.[ch]
>  test-qapi-visit.[ch]
> +test-qapi-introspect.[ch]

[2] Ah, maybe this is the file that wasn't quite right.

>  test-qdev-global-props
>  test-qemu-opts
>  test-qmp-commands
> diff --git a/tests/Makefile b/tests/Makefile
> index 60b82e2..7b1bf92 100644
> --- a/tests/Makefile
> +++ b/tests/Makefile
> @@ -253,7 +253,8 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
>  	struct-base-clash.json struct-base-clash-deep.json )
>  
>  GENERATED_HEADERS += tests/test-qapi-types.h tests/test-qapi-visit.h \
> -		     tests/test-qmp-commands.h tests/test-qapi-event.h
> +	tests/test-qmp-commands.h tests/test-qapi-event.h \
> +	tests/test-qmp-introspect.h
>  
>  test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
>  	tests/check-qlist.o tests/check-qfloat.o tests/check-qjson.o \
> @@ -266,7 +267,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
>  	tests/rcutorture.o tests/test-rcu-list.o
>  
>  test-qapi-obj-y = tests/test-qapi-visit.o tests/test-qapi-types.o \
> -		  tests/test-qapi-event.o
> +		  tests/test-qapi-event.o tests/test-qmp-introspect.o
>  
>  $(test-obj-y): QEMU_INCLUDES += -Itests
>  QEMU_CFLAGS += -I$(SRC_PATH)/tests
> @@ -327,6 +328,11 @@ $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-eve
>  	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
>  		$(gen-out-type) -o tests -p "test-" $<, \
>  		"  GEN   $@")
> +tests/test-qmp-introspect.c tests/test-qmp-introspect.h :\
> +$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
> +	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
> +		$(gen-out-type) -o tests -p "test-" $<, \
> +		"  GEN   $@")
>  
>  tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y) libqemuutil.a libqemustub.a
>  tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y) libqemuutil.a libqemustub.a
> diff --git a/tests/qapi-schema/alternate-good.out b/tests/qapi-schema/alternate-good.out
> index 0cbdfa1..aede1ae 100644
> --- a/tests/qapi-schema/alternate-good.out
> +++ b/tests/qapi-schema/alternate-good.out
> @@ -1,3 +1,4 @@
> +object :empty
>  alternate Alt

Again, adding the magic :empty object in a separate patch from the new
introspection code may help minimize the number of files being touched
with the new code.

> +++ b/tests/test-qmp-input-visitor.c
> @@ -17,6 +17,9 @@
>  #include "qapi/qmp-input-visitor.h"
>  #include "test-qapi-types.h"
>  #include "test-qapi-visit.h"
> +#include "test-qmp-introspect.h"
> +#include "qmp-introspect.h"
> +#include "qapi-visit.h"
>  #include "qapi/qmp/types.h"
>  
>  typedef struct TestInputVisitorData {
> @@ -660,6 +663,31 @@ static void test_visitor_in_native_list_number(TestInputVisitorData *data,
>      qapi_free_UserDefNativeListUnion(cvalue);
>  }
>  
> +static void do_test_visitor_in_qmp_introspect(TestInputVisitorData *data,
> +                                              const char *schema_json)
> +{
> +    SchemaInfoList *schema = NULL;
> +    Error *err = NULL;
> +    Visitor *v;
> +
> +    v = visitor_input_test_init_raw(data, schema_json);
> +
> +    visit_type_SchemaInfoList(v, &schema, NULL, &err);
> +    if (err)
> +        fprintf(stderr, "%s", error_get_pretty(err));
> +    g_assert(!err);

Don't you want:
visit_type_SchemaInfoList(..., &error_abort);

> +    g_assert(schema);
> +
> +    qapi_free_SchemaInfoList(schema);
> +}
> +
> +static void test_visitor_in_qmp_introspect(TestInputVisitorData *data,
> +                                           const void *unused)
> +{
> +    do_test_visitor_in_qmp_introspect(data, test_qmp_schema_json);
> +    do_test_visitor_in_qmp_introspect(data, qmp_schema_json);
> +}
> +
>  static void input_visitor_test_add(const char *testpath,
>                                     TestInputVisitorData *data,
>                                     void (*test_func)(TestInputVisitorData *data, const void *user_data))
> @@ -753,6 +781,9 @@ int main(int argc, char **argv)
>      input_visitor_test_add("/visitor/input/native_list/number",
>                             &in_visitor_data,
>                             test_visitor_in_native_list_number);
> +    input_visitor_test_add("/visitor/input/qmp_introspect",
> +                           &in_visitor_data,
> +                           test_visitor_in_qmp_introspect);
>  
>      g_test_run();
>  
> 

Starting to shape up nicely.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 46/47] qapi-introspect: Map all integer types to 'int'

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> How many bits we use internally is an implementation detail.  It could
> be pressed into external interface service as a very approximate range
> information, but that's probably a bad idea.  If we need range
> information, we better do it properly.
> 
> Reduces output of query-schema by a negligible 0.5 out of 80KiB.

But also makes us more conservative - it's easier to add things later
than to be forced to support forever; and mapping everything to 'int' in
introspection lets us change from 'int8' to 'uint8' internally without
affecting introspection-visible API.  I'm in favor of this for more than
just the string size savings.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-introspect.py | 7 +++++++
>  1 file changed, 7 insertions(+)

Reviewed-by: Eric Blake <eblake@redhat.com>

> 
> diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
> index e7efc4a..961fe88 100644
> --- a/scripts/qapi-introspect.py
> +++ b/scripts/qapi-introspect.py
> @@ -46,6 +46,13 @@ char %(c_name)s[] = "["
>          self.used_types = None
>  
>      def _use_type(self, typ):
> +        # Map the various integer types to plain int
> +        if typ.json_type() == 'int':
> +            typ = self.schema.lookup_type('int')
> +        elif isinstance(typ, QAPISchemaArrayType) \
> +             and typ.element_type.json_type() == 'int':
> +            typ = self.schema.lookup_type('intList')
> +        # Add type to work queue if new
>          if typ not in self.used_types:
>              self.used_types.append(typ)
>          return typ.name
> 

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3900 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> To eliminate the temptation for clients to look up types by name
> (which are not ABI), replace all type names by meaningless strings.
> 
> Reduces output of query-schema by 9 out of 80KiB.

I'm not sure whether I like this or not.  It does make sense from the
perspective of forcing clients to stick to ABI queries, but makes it a
bit harder to navigate things except by automated scripts.

> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>
> ---
>  scripts/qapi-introspect.py | 27 +++++++++++++++++++++++++--
>  1 file changed, 25 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
> index 961fe88..efb34ff 100644
> --- a/scripts/qapi-introspect.py
> +++ b/scripts/qapi-introspect.py
> @@ -9,13 +9,18 @@
>  # This work is licensed under the terms of the GNU GPL, version 2.
>  # See the COPYING file in the top-level directory.
>  
> +import string
>  from qapi import *
>  
> +def _b32digit(num):
> +    return (string.lowercase + string.digits[2:])[num]

This feels a bit too magic for me to decipher late at night.

> +
>  class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
>      def __init__(self):
>          self.schema = None
>          self.jsons = None
>          self.used_types = None
> +        self.name_map = None
>          self.defn = None
>          self.decl = None
>  
> @@ -23,13 +28,17 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
>          self.schema = schema
>          self.jsons = []
>          self.used_types = []
> +        self.name_map = {}
>          return QAPISchemaType   # don't visit types for now
>  
>      def visit_end(self):
>          # visit the types that are actually used
> +        jsons = self.jsons
> +        self.jsons = []
>          for typ in self.used_types:
>              typ.visit(self)
>          self.jsons.sort()
> +        jsons.extend(self.jsons)

I'm not sure I follow the use of .extends() here.

>          name = prefix + 'qmp_schema_json'
>          self.decl = mcgen('''
>  extern char %(c_name)s[];
> @@ -40,10 +49,19 @@ char %(c_name)s[] = "["
>      "%(c_jsons)s]";
>  ''',
>                            c_name=c_name(name),
> -                          c_jsons=', "\n    "'.join(self.jsons))
> +                          c_jsons=', "\n    "'.join(jsons))
>          self.schema = None
>          self.jsons = None
>          self.used_types = None
> +        self.name_map = None
> +
> +    def _name(self, name):
> +        if name not in self.name_map:
> +            n = len(self.name_map)
> +            self.name_map[name] = ':' + _b32digit(n / 32 / 32) \
> +                                  + _b32digit(n / 32 % 32) \
> +                                  + _b32digit(n % 32)

Caps our qapi to 32k types (including implicit ones); will that be an issue?

> +        return self.name_map[name]
>  
>      def _use_type(self, typ):
>          # Map the various integer types to plain int
> @@ -55,9 +73,14 @@ char %(c_name)s[] = "["
>          # Add type to work queue if new
>          if typ not in self.used_types:
>              self.used_types.append(typ)
> -        return typ.name
> +        # Clients should examine commands and events, not types.  Hide
> +        # type names to reduce the temptation.  Also saves a few
> +        # characters.
> +        return self._name(typ.name)
>  
>      def _gen_json(self, name, mtype, extra):
> +        if mtype != 'command' and mtype != 'event':
> +            name = self._name(name)
>          self.jsons.append("{ 'name': '%s', 'meta-type': '%s', %s }"
>                            % (name, mtype, extra))
>  
> 

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>> qapi-event.py breaks when you ask for a funny prefix like '@'.
>> Protect it.
>
> Only possible from the command line (not triggered by our makefiles);
> but doesn't hurt.
>
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi.py | 6 ++++++
>>  1 file changed, 6 insertions(+)
>
>
>> 
>> diff --git a/scripts/qapi.py b/scripts/qapi.py
>> index 2bbc8ff..ea94ce5 100644
>> --- a/scripts/qapi.py
>> +++ b/scripts/qapi.py
>> @@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []):
>>      for oa in opts:
>>          o, a = oa
>>          if o in ("-p", "--prefix"):
>> +            match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)
>
> I can understand allowing a leading _, but why bother allowing a leading
> '.' or '-'?  Those will get normalized to _, but in all honesty, no one
> should ever be doing that.

My patch rejects exactly the prefixes that won't work.

> I'd be just as happy with the shorter:
>
> match = re.match('([A-Za-z_][A-Za-z0-9_.-]*)?', a)

This additionally rejects a few rather foolish ones.

I have a slight preference for the tool staying out of policing foolish
prefixes.

>> +            if match.end() != len(a):
>> +                print >>sys.stderr, \
>> +                    "%s: 'funny character '%s' in argument of -prefix" \
>
> 'qemu' is unusual for accepting -single-dash-long-opts; I don't think
> python getopts does the same by default.  Please spell this error
> message --prefix.

Typo, will fix.

>> +                    % (sys.argv[0], a[match.end()])
>> +                sys.exit(1)
>>              prefix = a
>>          elif o in ("-o", "--output-dir"):
>>              output_dir = a + "/"
>> 
>
> With the second spelling fix, and optionally with the shorter regex,
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 06/47] qapi: Drop unused and useless parameters and variables

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>> gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it
>> only as optional argument for push_indent() and pop_indent(), their
>> default is four, and gen_sync_call()'s only caller passes four.
>> 
>> gen_visitor_input_containers_decl()'s parameter obj is always
>> "QOBJECT(args)".
>
> It might be nice to call out that several other functions are also
> stripped of unused arguments.  I was assuming that only two functions
> (and their callers) would be modified, but the patch touched more:

I'll rephrase the commit message to make this more obvious.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py | 27 +++++++++++++--------------
>>  scripts/qapi-event.py    |  1 -
>>  scripts/qapi-types.py    |  1 -
>>  scripts/qapi-visit.py    | 47 ++++++++++++++++++++++-------------------------
>>  scripts/qapi.py          |  1 -
>>  5 files changed, 35 insertions(+), 42 deletions(-)
>
>> @@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md);
>>      pop_indent()
>>      return ret.rstrip()
>>  
>> -def gen_marshal_output(name, args, ret_type, middle_mode):
>> +def gen_marshal_output(name, ret_type):
>>      if not ret_type:
>>          return ""
>
> For example, gen_marshal_output() was not mentioned in the commit message.
>
>>  
>> @@ -194,14 +193,14 @@ out:
>>  
>>      return ret
>>  
>> -def gen_marshal_input_decl(name, args, ret_type, middle_mode):
>> +def gen_marshal_input_decl(name, middle_mode):
>
> Or gen_marshal_input_decl()
>
>> +++ b/scripts/qapi-event.py
>> @@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = {
>>  ''',
>>                  event_enum_name = event_enum_name)
>>  
>> -    i = 0
>>      for string in event_enum_strings:
>
> I guess the subject line covered deletion of unused internal variables.
>
>> +++ b/scripts/qapi-visit.py
>
>> @@ -441,44 +440,42 @@ fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
>
>>      elif expr.has_key('union'):
>>          ret = generate_visit_union(expr)
>> -        ret += generate_visit_list(expr['union'], expr['data'])
>> +        ret += generate_visit_list(expr['union'])
>>          fdef.write(ret)
>>  
>>          enum_define = discriminator_find_enum_define(expr)
>>          ret = ""
>>          if not enum_define:
>> -            ret = generate_decl_enum('%sKind' % expr['union'],
>> -                                     expr['data'].keys())
>> -        ret += generate_declaration(expr['union'], expr['data'])
>> +            ret = generate_decl_enum('%sKind' % expr['union'])
>> +        ret += generate_declaration(expr['union'])
>
> As long as you are touching this, generate_decl_enum(expr['union'] +
> 'Kind') would read nicer.

It'll go away in PATCH 27.  Let's keep this patch simple.

>>          fdecl.write(ret)
>>      elif expr.has_key('alternate'):
>>          ret = generate_visit_alternate(expr['alternate'], expr['data'])
>> -        ret += generate_visit_list(expr['alternate'], expr['data'])
>> +        ret += generate_visit_list(expr['alternate'])
>>          fdef.write(ret)
>>  
>> -        ret = generate_decl_enum('%sKind' % expr['alternate'],
>> -                                 expr['data'].keys())
>> -        ret += generate_declaration(expr['alternate'], expr['data'])
>> +        ret = generate_decl_enum('%sKind' % expr['alternate'])
>
> Same here.

Likewise.

> At any rate, no change to generated output.  So assuming you are happy
> with the commit message, or take my advice to improve it, the code
> cleanup itself is fine.
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>> The struct generated for a flat union is weird: the members of its
>> base are at the end, except for the union tag, which is renamed to
>> 'kind' and put at the beginning.
>> 
>
>> Change to put all base members at the beginning, unadulterated.  Not
>> only is this easier to understand, it also permits casting the flat
>> union to its base, if that should become useful.
>> 
>> We now generate:
>> 
>>     struct UserDefFlatUnion
>>     {
>
> It might be worth tweaking the generator to output a C comment either
> here (at the start of the larger struct)...
>
>>         char *string;
>>         EnumOne enum1;
>
> ...or here (at the end of the base struct) mentioning that
> UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to
> make it easier when reading the generated code to trace back to that
> "inheritance" relationship.  Right now, there is nothing in the
> generated UserDefFlatUnion that points you back to the qapi relationship
> of a base class.  But it's not a show-stopper if you don't like my
> suggestion.

I do like it.  Perhaps something like

    struct UserDefFlatUnion
    {
        /* Members inherited from UserDefUnionBase: */
        char *string;
        EnumOne enum1;
        /* Own members: */
        ...
    };

>>         /* union tag is EnumOne enum1 */
>>         union {
>>             void *data;
>>             UserDefA *value1;
>>             UserDefB *value2;
>>             UserDefB *value3;
>>         };
>>     };
>
> Only impact in files generated for qemu was to struct BlockdevOptions,
> in the files qapi-types.[ch], and I agree that it is an improvement.
> Oddly enough, it seems none of the C code cared about the field being
> renamed from 'kind' to 'driver' (I guess that's because a lot of our use
> of the blockdev code is not directly through the generated C structs,
> but through QObject dictionary queries).

It surprised me, too.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-types.py           | 32 ++++++++++++++++++--------------
>>  scripts/qapi-visit.py           |  7 +++++--
>>  tests/test-qmp-input-visitor.c  |  2 +-
>>  tests/test-qmp-output-visitor.c |  2 +-
>>  4 files changed, 25 insertions(+), 18 deletions(-)
>> 
>
>> +''',
>> +                name=name)
>> +    if base:
>> +        base_fields = find_struct(base)['data']
>> +        ret += generate_struct_fields(base_fields)
>
>> -    if base:
>> -        assert discriminator
>> -        base_fields = find_struct(base)['data'].copy()
>> -        del base_fields[discriminator]
>> -        ret += generate_struct_fields(base_fields)
>
> I also like the fact that you no longer modify the base_fields array
> (because you are no longer floating a single renamed element
> out-of-order compared to the rest of the base), and therefore avoid the
> need for a .copy().
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 08/47] qapi-visit: Fix generated code when schema has forward refs

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>> The visit_type_implicit_FOO() are generated on demand, right before
>> their first use.  Used by visit_type_STRUCT_fields() when STRUCT has
>> base FOO, and by visit_type_UNION() when flat UNION has member a FOO.
>> 
>> If the schema defines FOO after its first use as struct base or flat
>> union member, visit_type_implicit_FOO() calls
>> visit_type_implicit_FOO() before its definition, which doesn't
>> compile.
>
> None of our public qapi .json files currently do this, so no difference
> to the generated code used in qemu proper.

Yes, we've intentionally avoided forward references.

Nevertheless, the generators either have to catch and reject them, or
make them work.  Making them work is easy, so I did that.

>> Rearrange qapi-schema-test.json to demonstrate the bug.
>
> Indeed, without testsuite exposure, nothing was relying on this fix.
>
>> 
>> Fix by generating the necessary forward declaration.
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-visit.py                   | 15 ++++++++++++++-
>>  tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++-------------
>>  tests/qapi-schema/qapi-schema-test.out  | 10 +++++-----
>>  3 files changed, 36 insertions(+), 19 deletions(-)
>> 
>
>> +++ b/scripts/qapi-visit.py
>> @@ -17,13 +17,23 @@ from qapi import *
>>  import re
>>  
>>  implicit_structs = []
>> +struct_fields_seen = set()
>>  
>>  def generate_visit_implicit_struct(type):
>>      global implicit_structs
>>      if type in implicit_structs:
>>          return ''
>>      implicit_structs.append(type)
>> -    return mcgen('''
> ...
>> +    ret += mcgen('''
>
> Oddly enough, the ''' is at the same indentation,...
>
>>  
>>  static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp)
>>  {
>> @@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
>>  }
>>  ''',
>>                   c_type=type_name(type))
>> +    return ret
>
> so nothing needed to be reindented here :)
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/21/2015 06:43 AM, Eric Blake wrote:
>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>> A command's 'data' must be a struct type, given either as a
>>> dictionary, or as struct type name.
>>>
>>> Existing test case data-int.json covers simple type 'int'.  Add test
>>> cases for type names referring to union and alternate types.
>> 
>> We could probably relax things to allow a union (which is always a
>> dictionary on the wire), but I agree that allowing an alternate type is
>> not appropriate (the goal here is that we require a dictionary).  But
>> it's also easier to be conservative now and relax later.

Yes.

>>> +++ b/tests/qapi-schema/args-alternate.json
>>> @@ -0,0 +1,4 @@
>>> +# we do not allow alternate arguments
>>> +# TODO should we support this?
>> 
>> I see no reason to allow a non-dictionary in QMP, so this TODO could be
>> dropped.
>
> Or, to be clear, we document that arguments is always a dictionary, for:
> { "execute":"command", "arguments":{} }. Allowing an alternate would
> break that, so it is a different level of change to allow an alternate
> (change the QMP protocol) than what it would be to allow a union (the
> QMP protocol is unchanged).  Not that we can't do it if we ever have a
> reason, it's just that I don't see it being worth a TODO statement.

I'll drop it.

>>> +++ b/tests/qapi-schema/args-union.json
>>> @@ -0,0 +1,4 @@
>>> +# FIXME we should reject union arguments
>>> +# TODO should we support this?
>>> +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
>>> +{ 'command': 'oops', 'data': 'Uni' }
>> 
>> This, on the other hand, seems valid from the wire format (it will
>> always be a dictionary).  I guess the problem is that we generate a C
>> function signature based by calling out each member of the dictionary -
>> but how do you do that for a union?

Exactly: the problem is neither conceptual nor the wire API, it's the C
API we generate.

>>                                      So I see what you are doing:
>> marking that this test currently passes the parser, but then causes
>> problems for generating C code, so we should either reject it up front,
>> or fix the generator.  The FIXME documents what you will do later in the
>> series (reject it up front)

Yes, in PATCH 15.

>>                             and the TODO documents what we can do down
>> the road (fix the generator to allow it).

I figure we'd change the C API not to explode the data type into
multiple parameters.  We can consider that when we have a use for it.

> See also 32/47 - events have the same problem.

I'm afraid I don't see the connection to PATCH 32.

>> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Clean up white-space, brace placement, and superfluous
>
> Incomplete sentence. I bet it's because your editor line-wrapped, and
> the rest of your sentence was something like '#ifndef in a .c file.'
> (see [1] below), then git ate the line thinking it was a comment.  I
> think I'm okay with cramming all of these cleanups into one patch rather
> than trying to do one style of cleanup per patch (blank lines, {
> placement, and guard cleanup), but the commit message could do a better
> job of explaining things.

Not throwing away half of it by accident would be a start :)

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py |  1 +
>>  scripts/qapi-event.py    |  3 +--
>>  scripts/qapi-types.py | 66
>> +++++++++++++++++++++++-------------------------
>>  scripts/qapi-visit.py    |  1 +
>>  4 files changed, 34 insertions(+), 37 deletions(-)
>
> Missing changes to docs/qapi-code-gen.txt to reflect the improvements.
> Since having docs that are stale compared to implementation can be
> misleading, I'd like to wait for v2 before giving my R-b; but see also
> [2] below.

I agree we should keep docs/qapi-code-gen.txt up-to-date.

Since updating it is manual, updating it just once for a complete series
can make sense.

I'll double check it's accurate at the end of the series, then decide
how to do necessary updates, if any.

> Overall, I'm a fan of the cleanups.
>
> I'm going to intersperse diffs of the generated files that were caused
> by some of these changes:
>
>> 
>> diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
>> index cfbd59c..223216d 100644
>> --- a/scripts/qapi-commands.py
>> +++ b/scripts/qapi-commands.py
>> @@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
>>          ret += mcgen('''
>>  
>>      (void)args;
>> +
>>  ''')
>
> This and similar hunts avoids extra blank lines.  Not worth showing a
> diff to the generated files.
>
>>  
>>      ret += gen_sync_call(name, args, ret_type)
>> diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
>> index 88b0620..7f238df 100644
>> --- a/scripts/qapi-event.py
>> +++ b/scripts/qapi-event.py
>> @@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[];
>>                          event_enum_name = event_enum_name)
>>  
>>      enum_decl = mcgen('''
>> -typedef enum %(event_enum_name)s
>> -{
>> +typedef enum %(event_enum_name)s {
>
> Several hunks like this; gets us generally closer to qemu style; with
> generated diffs like:
>
> qapi-types.h:
> -typedef struct int32List
> -{
> +typedef struct int32List {
>      union {
>          int32_t value;
>          uint64_t padding;
>
>> @@ -105,7 +103,8 @@ struct %(name)s
>>  
>>  def generate_enum_lookup(name, values):
>>      ret = mcgen('''
>> -const char * const %(name)s_lookup[] = {
>> +
>> +const char *const %(name)s_lookup[] = {
>
> [2] generated diffs like this:
>
> qapi-types.c:
> -const char * const OnOffAuto_lookup[] = {
> +const char *const OnOffAuto_lookup[] = {
>
> Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we
> added a const in commit 2e4450ff that is missing from the documentation.

Minor review fail.  Not the first time.

>> @@ -335,22 +333,22 @@ for typename in builtin_types.keys():
>>  fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
>>  
>>  for expr in exprs:
>> -    ret = "\n"
>> +    ret = ""
>>      if expr.has_key('struct'):
>>          ret += generate_fwd_struct(expr['struct'])
>>      elif expr.has_key('enum'):
>> -        ret += generate_enum(expr['enum'], expr['data']) + "\n"
>> +        ret += generate_enum(expr['enum'], expr['data'])
>>          ret += generate_fwd_enum_struct(expr['enum'])
>>          fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
>
> More work at avoiding extra blank lines.
>
>> @@ -370,34 +368,32 @@ fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
>>  # have the functions defined, so we use -b option to provide control
>>  # over these cases
>>  if do_builtins:
>> -    fdef.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))
>>      for typename in builtin_types.keys():
>>          fdef.write(generate_type_cleanup(typename + "List"))
>> -    fdef.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))
>
> [1] this is the hunk whose description got corrupted in your commit
> message. It gives a diff like this:
>
> qapi-types.c:
>
> -
> -#ifndef QAPI_TYPES_BUILTIN_CLEANUP_DEF
> -#define QAPI_TYPES_BUILTIN_CLEANUP_DEF
> -
> -
>  void qapi_free_int32List(int32List *obj)
>
> We conditionally declare the functions in the .h, but the .c is not
> compiled multiple times, so there is no need for a header-style guard.

This is going to help me reconstruct my crippled commit message, thanks!

Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> The QAPI code generators work with a syntax tree (nested dictionaries)
>> plus a few symbol tables (also dictionaties) on the side.
>
> s/dictionaties/dictionaries/

Will fix.

>> They have clearly outgrown these simple data structures.  There's lots
>> of rummaging around in dictionaries, and information is recomputed on
>> the fly.  For the work I'm going to do, I want more clearly defined
>> and more convenient interfaces.
>> 
>> Going forward, I also want less coupling between the back-ends and the
>> syntax tree, to make messing with the syntax easier.
>
> Indeed - should be a lot easier to add new qapi .json syntax sugar on
> the front-end, or to tweak generated C layout on the backend, without
> close coupling between the two.  Particularly when adding front-end
> sugar extensions that still normalize into the same backend structs.
>
>> 
>> Create a bunch of classes to represent QAPI schemata.
>> 
>> Have the QAPISchema initializer call the parser, then walk the syntax
>> tree to create the new internal representation, and finally perform
>> semantic analysis.
>> 
>> Shortcut: the semantic analysis still relies on existing check_exprs()
>> to do the actual semantic checking.  All this code needs to move into
>> the classes.  Mark as TODO.
>> 
>> We generate array types eagerly, even though most of them aren't used.
>> Mark as TODO.
>> 
>
> No change to generated files at this stage in the game (this is mostly
> additive, then later patches shed their old ways of doing things by
> using this).  Good division of labor between patches.

I'll steal from this paragraph for my commit message.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py       |   2 +-
>>  scripts/qapi-event.py          |   2 +-
>>  scripts/qapi-types.py          |   2 +-
>>  scripts/qapi-visit.py          |   2 +-
>>  scripts/qapi.py                | 351 +++++++++++++++++++++++++++++++++++++++--
>>  tests/qapi-schema/test-qapi.py |   2 +-
>>  6 files changed, 347 insertions(+), 14 deletions(-)
>> 
>
>> +++ b/scripts/qapi.py
>
>> +
>> +#
>> +# Schema compiler frontend
>> +#
>> +
>> +class QAPISchemaEntity(object):
>> +    def __init__(self, name, info):
>> +        assert isinstance(name, str)
>> +        self.name = name
>> +        self.info = info
>> +    def check(self, schema):
>> +        pass
>> +
>> +class QAPISchemaType(QAPISchemaEntity):
>> +    pass
>> +
>> +class QAPISchemaBuiltinType(QAPISchemaType):
>> +    def __init__(self, name):
>> +        QAPISchemaType.__init__(self, name, None)
>> +
>> +class QAPISchemaEnumType(QAPISchemaType):
>> +    def __init__(self, name, info, values):
>> +        QAPISchemaType.__init__(self, name, info)
>> +        for v in values:
>> +            assert isinstance(v, str)
>> +        self.values = values
>
> worth a check() method to ensure values don't collide?  Especially since
> you already do that in QAPISchemaObjectTypeMember.check()

Can do.

Aside: everything the check() methods assert must be established by
semantic analysis.  Moving semantic analysis into the classes as
outlined in the commit message will hopefully make that obvious enough
to permit dropping the assertions.

>> +
>> +class QAPISchemaArrayType(QAPISchemaType):
>> +    def __init__(self, name, info, element_type):
>> +        QAPISchemaType.__init__(self, name, info)
>> +        assert isinstance(element_type, str)
>> +        self.element_type_name = element_type
>> +        self.element_type = None
>> +    def check(self, schema):
>> +        self.element_type = schema.lookup_type(self.element_type_name)
>> +        assert self.element_type
>> +
>> +class QAPISchemaObjectType(QAPISchemaType):
>> +    def __init__(self, name, info, base, local_members, variants):
>> +        QAPISchemaType.__init__(self, name, info)
>> +        assert base == None or isinstance(base, str)
>> +        for m in local_members:
>> +            assert isinstance(m, QAPISchemaObjectTypeMember)
>> +        if variants != None:
>> +            assert isinstance(variants, QAPISchemaObjectTypeVariants)
>> +        self.base_name = base
>> +        self.base = None
>> +        self.local_members = local_members
>> +        self.variants = variants
>> +        self.members = None
>> +    def check(self, schema):
>> +        if self.members:
>> +            return                      # already checked
>> +        assert self.members == None     # not running in cycles
>> +        self.members = False            # mark as being checked
>
> Interesting, but makes sense (since we allow self-referential nesting,
> populating our own members may require revisiting the same type).

Yes.  Another way to put it: for a recursive type like this, we better
check the recursion terminates.

>                                                                    Cute
> that python allows both None and False as distinct non-true settings, so
> that you end up using the variable as a tri-state.

Yup :)

>> +        if self.base_name:
>> +            self.base = schema.lookup_type(self.base_name)
>> +            self.base.check(schema)
>
> Do you need 'assert self.base' here, similar to how you did it in
> QAPISchemaArrayType.check()?  I guess you'll still get a python
> exception that None.check() doesn't exist if the lookup_type failed, so
> it just depends on what error message you want.

I don't see a hard need, but consistency is nice.  Since I generally
assert elsewhere, I'll add an assertion here.

>> +            members = list(self.base.members)
>
> For that matter, do you want to assert isinstance(self.base,
> QAPISchemaObjectType), since lookup_type can return other types, but
> other child classes of QAPISchemaType do not have a .members?

Yes, that's the appropriate assertion.

>> +        else:
>> +            members = []
>> +        seen = {}
>> +        for m in members:
>> +            seen[m.name] = m
>> +        for m in self.local_members:
>> +            m.check(schema, members, seen)
>> +        if self.variants:
>> +            self.variants.check(schema, members, seen)
>> +        self.members = members
>> +
>> +class QAPISchemaObjectTypeMember(object):
>
> Is 'object' the right base class, or should it be QAPISchemaEntity?

No, it's really object.  A schema's entities can be object types, other
types, commands and events, but not members.  A schema doesn't have
"members", only an object type has.

>> +    def __init__(self, name, typ, optional):
>> +        assert isinstance(name, str)
>> +        assert isinstance(typ, str)
>
> I guess that means all callers flatten array types into an unambiguous
> string first.

Yes.

Conscious design decision: every type in the internal representation has
a unique name.  If the concrete syntax lets you define an anonymous
type, we make up a name in the internal representation.  One less
special case to worry about.

>> +        assert isinstance(optional, bool)
>> +        self.name = name
>> +        self.type_name = typ
>> +        self.type = None
>> +        self.optional = optional
>> +    def check(self, schema, all_members, seen):
>> +        assert self.name not in seen
>> +        self.type = schema.lookup_type(self.type_name)
>> +        assert self.type
>> +        all_members.append(self)
>> +        seen[self.name] = self
>> +
>> +class QAPISchemaObjectTypeVariants(object):
>
> Is 'object' the right base class, or should it be QAPISchemaEntity?

As for QAPISchemaObjectTypeMember above, object is appropriate.

>> +    def __init__(self, tag_name, tag_enum, variants):
>> +        assert tag_name == None or isinstance(tag_name, str)
>> +        assert tag_enum == None or isinstance(tag_enum, str)
>> +        for v in variants:
>> +            assert isinstance(v, QAPISchemaObjectTypeVariant)
>> +        self.tag_name = tag_name
>> +        if tag_name:
>> +            assert not tag_enum
>> +            self.tag_member = None
>> +        else:
>> +            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
>> +                                                         False)
>
> Nice - the tag is either part of the base struct (flat union) or
> implicitly created (plain union).  By the time we are done with check(),
> we then have both its type (possibly generated implicitly) and its name.

... and we don't have to know where they came from anymore.

>> +        self.variants = variants
>> +    def check(self, schema, members, seen):
>> +        if self.tag_name:
>> +            self.tag_member = seen[self.tag_name]
>> +        else:
>> +            self.tag_member.check(schema, members, seen)
>
> A little bit tricky here in that flat unions (tag_name was set) must
> find the tag from the members already loaded from the base class, while
> plain unions (tag_name is None) add their own implicit member into the
> union.  But it works.
>
>> +        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
>> +        for v in self.variants:
>> +            vseen = dict(seen)
>> +            v.check(schema, self.tag_member.type, vseen)
>> +
>> +class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
>> +    def __init__(self, name, typ, flat):
>> +        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
>> +        assert isinstance(flat, bool)
>> +        self.flat = flat
>> +    def check(self, schema, tag_type, seen):
>> +        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
>> +        assert self.name in tag_type.values
>> +        if self.flat:
>> +            self.type.check(schema)
>> +            assert isinstance(self.type, QAPISchemaObjectType)
>
> Do we really want to be tracking self.flat for each variant?  After all,
> docs/qapi-code-gen.txt already describes the mapping from simple union
> into flat union (as if the flat union had a base class with single
> member 'kind' of the right type, then each branch of the union composed
> of an implicit object with a lone member 'data' of the correct type).
> In other words, is it any better to just normalize into that form now,
> such that each QAPISchemaObjectTypeVariant is merely a (often
> one-element) list of name:type members being added to the overall
> QAPISchemaObject?

I tried to do exactly that, but got bogged down in special cases and
copped out.  Then I went on vacation, and now I don't remember the exact
problems anymore %-}

I guess / hope it's just relatively pointless differences in the
generated C code I didn't want to get rid of at this time.  The series
is long and hairy enough as it is...

>                    But I guess it remains to be seen how you use
> self.flat before knowing if it is worth normalizing away from it.

At least introspect.json is oblivious of it.

>> +
>> +class QAPISchemaAlternateType(QAPISchemaType):
>> +    def __init__(self, name, info, variants):
>> +        QAPISchemaType.__init__(self, name, info)
>> +        assert isinstance(variants, QAPISchemaObjectTypeVariants)
>> +        assert not variants.tag_name
>> +        for v in variants.variants:
>> +            assert not v.flat
>
> Hmm. You are using .flat after all.
>
>> +        self.variants = variants
>> +    def check(self, schema):
>> +        self.variants.check(schema, [], {})
>> +
>> +class QAPISchemaCommand(QAPISchemaEntity):
>> +    def __init__(self, name, info, args, rets, gen, success_response):
>> +        QAPISchemaEntity.__init__(self, name, info)
>> +        assert not args or isinstance(args, str)
>> +        assert not rets or isinstance(rets, str)
>
> Again, this means that the caller has already had to convert qapi
> dictionaries into an implicit type and pass in that type name here.
> Works for me.
>
>> +        self.args_name = args
>> +        self.args = None
>> +        self.rets_name = rets
>> +        self.rets = None
>> +        self.gen = gen
>> +        self.success_response = success_response
>> +    def check(self, schema):
>> +        if self.args_name:
>> +            self.args = schema.lookup_type(self.args_name)
>> +            assert isinstance(self.args, QAPISchemaObjectType)
>> +            assert not self.args.variants # not implemented
>> +            self.args.check(schema)
>> +        if self.rets_name:
>> +            self.rets = schema.lookup_type(self.rets_name)
>> +            assert isinstance(self.rets, QAPISchemaType)
>> +            self.rets.check(schema)
>
> Matches our existing difference in requiring a dictionary for arguments,
> but allowing the return of any type (although we now have a whitelist to
> prevent too many additions of non-dictionaries).

Yes.  In my opinion, the difference between arguments and returns is a
design blemish (a function maps from domain to codomain, and both are
just sets, so any differences are likely pointless baggage), but we need
to work with what we've got.

>> +
>> +class QAPISchemaEvent(QAPISchemaEntity):
>> +    def __init__(self, name, info, data):
>> +        QAPISchemaEntity.__init__(self, name, info)
>> +        assert not data or isinstance(data, str)
>> +        self.data_name = data
>> +        self.data = None
>> +    def check(self, schema):
>> +        if self.data_name:
>> +            self.data = schema.lookup_type(self.data_name)
>> +            assert isinstance(self.data, QAPISchemaObjectType)
>> +            self.data.check(schema)
>> +
>> +class QAPISchema(object):
>> +    def __init__(self, fname):
>> +        try:
>> + self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
>> +        except (QAPISchemaError, QAPIExprError), err:
>> +            print >>sys.stderr, err
>> +            exit(1)
>> +        self.entity_dict = {}
>> +        self._def_predefineds()
>> +        self._def_exprs()
>> +        self.check()
>> +
>> +    def get_exprs(self):
>> +        return [expr_elem['expr'] for expr_elem in self.exprs]
>> +
>> +    def _def_entity(self, ent):
>> +        assert ent.name not in self.entity_dict
>> +        self.entity_dict[ent.name] = ent
>> +
>> +    def lookup_entity(self, name, typ=None):
>> +        ent = self.entity_dict.get(name)
>> +        if typ and not isinstance(ent, typ):
>> +            return None
>> +        return ent
>
> Ah, so you enforce a shared namespace between types, commands, and
> events (all three are in self.entity_dict), but can use the typ
> parameter to allow limiting a lookup to just types.

Yes.  It's a convenience feature.

>> +
>> +    def lookup_type(self, name):
>> +        return self.lookup_entity(name, QAPISchemaType)
>> +
>> +    def _def_builtin_type(self, name):
>> +        self._def_entity(QAPISchemaBuiltinType(name))
>> +        if name != '**':
>> +            self._make_array_type(name) # TODO really needed?
>> +
>> +    def _def_predefineds(self):
>> +        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
>> + 'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
>> +            self._def_builtin_type(t)
>> +
>> +    def _make_implicit_enum_type(self, name, values):
>> +        name = name + 'Kind'
>> +        self._def_entity(QAPISchemaEnumType(name, None, values))
>> +        return name
>> +
>> +    def _make_array_type(self, element_type):
>> +        name = element_type + 'List'
>> +        if not self.lookup_type(name):
>> +            self._def_entity(QAPISchemaArrayType(name, None, element_type))
>> +        return name
>> +
>> +    def _make_implicit_object_type(self, name, role, members):
>> +        name = ':obj-%s-%s' % (name, role)
>> +        self._def_entity(QAPISchemaObjectType(name, None, None, members, None))
>> +        return name
>
> We may extend the qapi .json syntax to allow anonymous types in more
> places, but so far, this matches that all existing use of anonymous
> types are for structs, not unions.

Yes.

If we unify struct and union in the concrete syntax like we do in this
internal representation, then anonymous unions (really: anonymous
objects with variants) may well fall out naturally.

>> +
>> +    def _def_enum_type(self, expr, info):
>> +        name = expr['enum']
>> +        data = expr['data']
>> +        self._def_entity(QAPISchemaEnumType(name, info, data))
>> +        self._make_array_type(name) # TODO really needed?
>> +
>> +    def _make_member(self, name, typ):
>> +        optional = False
>> +        if name.startswith('*'):
>> +            name = name[1:]
>> +            optional = True
>> +        if isinstance(typ, list):
>> +            assert len(typ) == 1
>> +            typ = self._make_array_type(typ[0])
>> +        return QAPISchemaObjectTypeMember(name, typ, optional)
>> +
>> +    def _make_members(self, data):
>> +        return [self._make_member(key, data[key]) for key in data.keys()]
>> +
>> +    def _def_struct_type(self, expr, info):
>> +        name = expr['struct']
>> +        base = expr.get('base')
>> +        data = expr['data']
>> +        self._def_entity(QAPISchemaObjectType(name, info, base,
>> +                                              self._make_members(data),
>> +                                              None))
>> +        self._make_array_type(name) # TODO really needed?
>> +
>> +    def _make_flat_variant(self, case, typ):
>> +        return QAPISchemaObjectTypeVariant(case, typ, True)
>> +
>> +    def _make_flat_variants(self, tag_name, data):
>> +        variants = [self._make_flat_variant(key, data[key])
>> +                    for key in data.keys()]
>> +        return QAPISchemaObjectTypeVariants(tag_name, None, variants)
>> +
>> +    def _make_simple_variant(self, case, typ):
>> +        if isinstance(typ, list):
>> +            assert len(typ) == 1
>> +            typ = self._make_array_type(typ[0])
>> +        return QAPISchemaObjectTypeVariant(case, typ, False)
>> +
>> +    def _make_simple_variants(self, type_name, data):
>> +        variants = [self._make_simple_variant(key, data[key])
>> +                    for key in data.keys()]
>> +        enum = self._make_implicit_enum_type(type_name,
>> +                                             [v.name for v in variants])
>> +        return QAPISchemaObjectTypeVariants(None, enum, variants)
>
> Again, I wonder if normalizing simple unions into flat unions (to get
> rid of the need for tracking .flat) would make later use any easier, or
> if it would just make introspection QMP output more verbose.

They *are* normalized in introspection output!

>> +
>> +    def _def_union_type(self, expr, info):
>> +        name = expr['union']
>> +        data = expr['data']
>> +        tag_name = expr.get('discriminator')
>> +        if tag_name:
>> +            base = expr['base']
>> +            variants = self._make_flat_variants(tag_name, data)
>> +        else:
>> +            base = None
>> +            variants = self._make_simple_variants(name, data)
>> +        self._def_entity(QAPISchemaObjectType(name, info, base,
>> + self._make_members(OrderedDict()),
>> +                                              variants))
>> +        self._make_array_type(name) # TODO really needed?
>> +
>> +    def _def_alternate_type(self, expr, info):
>> +        name = expr['alternate']
>> +        data = expr['data']
>> +        self._def_entity(QAPISchemaAlternateType(name, info,
>> +                                    self._make_simple_variants(name, data)))
>> +        self._make_array_type(name) # TODO really needed?
>> +
>> +    def _def_command(self, expr, info):
>> +        name = expr['command']
>> +        args = expr.get('data')
>> +        rets = expr.get('returns')
>> +        gen = expr.get('gen', True)
>> +        success_response = expr.get('success-response', True)
>> +        if args and not isinstance(args, str):
>> +            args = self._make_implicit_object_type(name, 'args',
>> +                                                   self._make_members(args))
>
> If I write { 'command':'Foo', 'data':{} }, does that try and create an
> implicit type, with no members?

tests/qapi-schema/qapi-schema-test.json tests this:

    { 'command': 'user_def_cmd', 'data': {} }

args = expr.get('data') gets an empty OrderedDict().

Since an empty dictionary counts as false, we skip calling
_make_implicit_object_type(), and pass the empty dictionary to
QAPISchemaCommand().  Works, because the class consistently treats any
false args the same as None.  tests/qapi-schema/qapi-schema-test.out:

    command user_def_cmd None -> None
       gen=True success_response=True

I'll consider cleaning this up a bit.

>                                  I'm wondering if it is any simpler to
> write args = expr.get('data', {}), and have _make_implicit_object_type()
> do the right thing when presented with an empty list of members.
>
>> +        if rets and isinstance(rets, list):
>> +            assert len(rets) == 1
>> +            rets = self._make_array_type(rets[0])
>> +        elif rets and not isinstance(rets, str):
>> +            rets = self._make_implicit_object_type(name, 'rets',
>> +                                                   self._make_members(rets))
>> +        self._def_entity(QAPISchemaCommand(name, info, args, rets, gen,
>> +                                           success_response))
>> +
>> +    def _def_event(self, expr, info):
>> +        name = expr['event']
>> +        data = expr.get('data')
>> +        if data and not isinstance(data, str):
>> +            data = self._make_implicit_object_type(name, 'data',
>> +                                                   self._make_members(data))
>> +        self._def_entity(QAPISchemaEvent(name, info, data))
>> +
>> +    def _def_exprs(self):
>> +        for expr_elem in self.exprs:
>> +            expr = expr_elem['expr']
>> +            info = expr_elem['info']
>> +            if 'enum' in expr:
>> +                self._def_enum_type(expr, info)
>> +            elif 'struct' in expr:
>> +                self._def_struct_type(expr, info)
>> +            elif 'union' in expr:
>> +                self._def_union_type(expr, info)
>> +            elif 'alternate' in expr:
>> +                self._def_alternate_type(expr, info)
>> +            elif 'command' in expr:
>> +                self._def_command(expr, info)
>> +            elif 'event' in expr:
>> +                self._def_event(expr, info)
>> +            else:
>> +                assert False
>> +
>> +    def check(self):
>> +        for ent in self.entity_dict.values():
>> +            ent.check(self)
>
> Overall seems like a very nice layout.
>
> I'll reserve R-b until I see how the rest of the series uses this object
> hierarchy, but it looks like you won't have to change very much of this
> patch when you do your next spin of the series.

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> New methods c_name(), c_type(), c_null(), json_type(),
>> alternate_qtype().
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi.py | 72
>> +++++++++++++++++++++++++++++++++++++++++++++++++++------
>>  1 file changed, 65 insertions(+), 7 deletions(-)
>
> Still unused in the rest of the code, so no change to generated code
> yet; and thanks for separating this apart from the creation of the new
> object hierarchy.
>
>>  class QAPISchemaType(QAPISchemaEntity):
>> -    pass
>> +    def c_type(self, is_param=False):
>> +        return c_name(self.name) + pointer_suffix
>
> At first, I thought is_param was unused; but reviewing the rest of the
> patch shows that subclasses are overriding it, so declaring it here is
> necessary for polymorphic calls to supply the argument and have it
> properly acted on.

Yup.

>> +    def c_null(self):
>> +        return 'NULL'
>> +    def json_type(self):
>> +        return None
>> +    def alternate_qtype(self):
>> +        json2qtype = {
>> +            'string':  'QTYPE_QSTRING',
>> +            'number':  'QTYPE_QFLOAT',
>> +            'int':     'QTYPE_QINT',
>> +            'boolean': 'QTYPE_QBOOL',
>> +            'object':  'QTYPE_QDICT'
>> +        }
>
> Are there any style rules on whether to include or omit a trailing comma?

PEP8 appears to be silent on it.

>> +        return json2qtype.get(self.json_type())
>>  
>>  class QAPISchemaBuiltinType(QAPISchemaType):
>> -    def __init__(self, name):
>> +    def __init__(self, name, json_type, c_type, c_null):
>>          QAPISchemaType.__init__(self, name, None)
>> +        assert not c_type or isinstance(c_type, str)
>> +        assert json_type in ('string', 'number', 'int', 'boolean', 'null',
>> +                             'value')
>> +        self.json_type_name = json_type
>> +        self.c_type_name = c_type
>> +        self.c_null_val = c_null
>> +    def c_name(self):
>> +        return self.name
>
> At first, I thought this was redundant with QAPISchemaEntity.c_name(),
> until I realized that you are intentionally overriding things here to
> NOT call global c_name() (so that 'int' does not get munged the 'q_int'
> in the generated C code).  Nice, once I figured it out.

I'll consider adding comments to help future readers figure it out.

>> +    def c_type(self, is_param=False):
>> +        if is_param and self.name == 'str':
>> +            return 'const ' + self.c_type_name
>> +        return self.c_type_name
>> +    def c_null(self):
>> +        return self.c_null_val
>> +    def json_type(self):
>> +        return self.json_type_name
>>  
>>  class QAPISchemaEnumType(QAPISchemaType):
>>      def __init__(self, name, info, values):
>> @@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
>>          for v in values:
>>              assert isinstance(v, str)
>>          self.values = values
>> +    def c_type(self, is_param=False):
>> +        return c_name(self.name)
>
> Probably showing my unfamiliarity with python polymorphism/overriding
> rules - is it necessary to declare an otherwise-unused optional
> parameter here if we are overriding a base method where the parameter
> has a sane default for our needs?  That is, would c_type(self) here
> properly override c_type(self, is_param=False) in the parent class?

I don't know.  But the way I wrote it, we don't need to know :)

>> +    def c_null(self):
>> +        return c_enum_const(self.name, self.values[0])
>> +    def json_type(self):
>> +        return 'string'
>>  
>>  class QAPISchemaArrayType(QAPISchemaType):
>>      def __init__(self, name, info, element_type):
>> @@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType):
>>          if self.variants:
>>              self.variants.check(schema, members, seen)
>>          self.members = members
>> +    def c_name(self):
>> +        assert self.info
>> +        return QAPISchemaType.c_name(self)
>> +    def c_type(self, is_param=False):
>> +        assert self.info
>> +        return QAPISchemaType.c_type(self)
>
> So, to make sure I understand correctly, implicit types (those generated
> from anonymous dictionaries in a command or event) have info=None,

Correct.

>                                                                    and
> the assertions here are making sure we never try to learn the c_name or
> c_type of those implicit types in generated code, but rather use them
> solely in generated code that just enumerates the members (C struct
> declarations, parameter list to functions that implement commands).

Correct again.

>> +    def json_type(self):
>> +        return 'object'
>>  
>>  class QAPISchemaObjectTypeMember(object):
>>      def __init__(self, name, typ, optional):
>> @@ -948,15 +994,27 @@ class QAPISchema(object):
>>      def lookup_type(self, name):
>>          return self.lookup_entity(name, QAPISchemaType)
>>  
>> -    def _def_builtin_type(self, name):
>> -        self._def_entity(QAPISchemaBuiltinType(name))
>> +    def _def_builtin_type(self, name, json_type, c_type, c_null):
>> +        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
>>          if name != '**':
>>              self._make_array_type(name) # TODO really needed?
>>  
>>      def _def_predefineds(self):
>> -        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
>> -                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
>> -            self._def_builtin_type(t)
>> +        for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
>> +                  ('number', 'number',  'double',   '0'),
>
> Do you want '0.0' as the default here (yes, promoting an int 0 to double
> does the right thing, but I like making it obvious that I know I'm using
> floating point).

The '0' here is used as initializer, i.e. we generate things like

    double d = 0;

Changing it to '0.0' will instead generate

    double d = 0.0;

No strong preference on my part.

>> +                  ('int',    'int',     'int64_t',  '0'),
>> +                  ('int8',   'int',     'int8_t',   '0'),
>> +                  ('int16',  'int',     'int16_t',  '0'),
>> +                  ('int32',  'int',     'int32_t',  '0'),
>> +                  ('int64',  'int',     'int64_t',  '0'),
>> +                  ('uint8',  'int',     'uint8_t',  '0'),
>> +                  ('uint16', 'int',     'uint16_t', '0'),
>> +                  ('uint32', 'int',     'uint32_t', '0'),
>> +                  ('uint64', 'int',     'uint64_t', '0'),
>> +                  ('size',   'int',     'uint64_t', '0'),
>> +                  ('bool',   'boolean', 'bool',     'false'),
>> +                  ('**',     'value',   None,       None)]:
>> +            self._def_builtin_type(*t)
>>  
>>      def _make_implicit_enum_type(self, name, values):
>>          name = name + 'Kind'
>> 
>
> Tweaks you make (if any) based on the above comments should be minor, so
> I'm fine with adding:
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> New methods c_name(), c_type(), c_null(), json_type(),
>> alternate_qtype().
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi.py | 72
>> +++++++++++++++++++++++++++++++++++++++++++++++++++------
>>  1 file changed, 65 insertions(+), 7 deletions(-)
>> 
>
> I just noticed:
>
>> @@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
>>          for v in values:
>>              assert isinstance(v, str)
>>          self.values = values
>> +    def c_type(self, is_param=False):
>> +        return c_name(self.name)
>> +    def c_null(self):
>> +        return c_enum_const(self.name, self.values[0])
>
> What does this return for an empty enum, as in { 'enum':'Empty',
> 'data':[] }?

I suspect self.values will be [] then, and self.values[0] will bomb.

Possible fixes:

* Outlaw empty enums

* Add the implicit MAX member to self.values[] (other code may have to
  skip it)

* Catch the special case here, and return the implicit MAX member.

>               Our testsuite proves we can do that, even if our normal
> .json code doesn't use it.

tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1719 bytes --]

On 07/27/2015 01:50 AM, Markus Armbruster wrote:

>>> This, on the other hand, seems valid from the wire format (it will
>>> always be a dictionary).  I guess the problem is that we generate a C
>>> function signature based by calling out each member of the dictionary -
>>> but how do you do that for a union?
> 
> Exactly: the problem is neither conceptual nor the wire API, it's the C
> API we generate.
> 
>>>                                      So I see what you are doing:
>>> marking that this test currently passes the parser, but then causes
>>> problems for generating C code, so we should either reject it up front,
>>> or fix the generator.  The FIXME documents what you will do later in the
>>> series (reject it up front)
> 
> Yes, in PATCH 15.
> 
>>>                             and the TODO documents what we can do down
>>> the road (fix the generator to allow it).
> 
> I figure we'd change the C API not to explode the data type into
> multiple parameters.  We can consider that when we have a use for it.
> 
>> See also 32/47 - events have the same problem.
> 
> I'm afraid I don't see the connection to PATCH 32.

Patch 32 was where I figured out that we have the same problem where the
C code we generate for an event will break if an event tries to use a
union type as its data dictionary; and therefore, this patch would be an
ideal time to add a test for that, and patch 15 would be an ideal time
to tweak events to not allow unions (as the simple fix, where someday
down the road we may relax things to allow unions in both commands and
events).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

On 07/27/2015 03:23 AM, Markus Armbruster wrote:

>>>
>>> Create a bunch of classes to represent QAPI schemata.
>>>
>>> Have the QAPISchema initializer call the parser, then walk the syntax
>>> tree to create the new internal representation, and finally perform
>>> semantic analysis.
>>>

>>> +class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
>>> +    def __init__(self, name, typ, flat):
>>> +        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
>>> +        assert isinstance(flat, bool)
>>> +        self.flat = flat
>>> +    def check(self, schema, tag_type, seen):
>>> +        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
>>> +        assert self.name in tag_type.values
>>> +        if self.flat:
>>> +            self.type.check(schema)
>>> +            assert isinstance(self.type, QAPISchemaObjectType)
>>
>> Do we really want to be tracking self.flat for each variant?  After all,
>> docs/qapi-code-gen.txt already describes the mapping from simple union
>> into flat union (as if the flat union had a base class with single
>> member 'kind' of the right type, then each branch of the union composed
>> of an implicit object with a lone member 'data' of the correct type).
>> In other words, is it any better to just normalize into that form now,
>> such that each QAPISchemaObjectTypeVariant is merely a (often
>> one-element) list of name:type members being added to the overall
>> QAPISchemaObject?
> 
> I tried to do exactly that, but got bogged down in special cases and
> copped out.  Then I went on vacation, and now I don't remember the exact
> problems anymore %-}
> 
> I guess / hope it's just relatively pointless differences in the
> generated C code I didn't want to get rid of at this time.  The series
> is long and hairy enough as it is...
> 
>>                    But I guess it remains to be seen how you use
>> self.flat before knowing if it is worth normalizing away from it.
> 
> At least introspect.json is oblivious of it.

Yeah, that was my conclusion by the end of the series - we still had
enough special cases where we generate different code for simple unions
than for flat unions. It would be possible to merge the generator and
simplify things further, but at this point, it is best done as a
follow-up series because it will touch lots of C code (anything that
uses a simple union would have to convert to the common representation).

And you actually did reference .flat in the patch that first added
qapi-introspect.py (good that it did not leak to the introspection
output, but it did have to be considered when figuring out what to
output); again, something you may want to rework before polishing this
into v3, or something you may end up just documenting as a possible
cleanup for a future series. But after having reviewed the whole series
now, I'm able to live with your use of .flat, if only because it makes
the initial conversion faster.

>>> +
>>> +    def lookup_entity(self, name, typ=None):
>>> +        ent = self.entity_dict.get(name)
>>> +        if typ and not isinstance(ent, typ):
>>> +            return None
>>> +        return ent
>>
>> Ah, so you enforce a shared namespace between types, commands, and
>> events (all three are in self.entity_dict), but can use the typ
>> parameter to allow limiting a lookup to just types.
> 
> Yes.  It's a convenience feature.

Documentation comments never hurt :) (You did mention in the cover
letter that part of the reason this was still RFC was that it was
lacking documentation)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 24/47] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> The old code prints the result of parsing (list of expression
>> dictionaries), and partial results of semantic analysis (list of enum
>> dictionaries, list of struct dictionaries).
>> 
>> The new code prints a trace of a schema visit, i.e. what the back-ends
>> are going to use.  Built-in and array types are omitted, because
>> they're boring.
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  tests/qapi-schema/alternate-good.out            |  15 +-
>>  tests/qapi-schema/comments.out                  |   4 +-
>>  tests/qapi-schema/data-member-array.out         |  13 +-
>>  tests/qapi-schema/empty.out                     |   3 -
>>  tests/qapi-schema/enum-empty.out                |   4 +-
>>  tests/qapi-schema/event-case.out                |   4 +-
>>  tests/qapi-schema/flat-union-reverse-define.out |  21 +--
>>  tests/qapi-schema/ident-with-escape.out         |   7 +-
>>  tests/qapi-schema/include-relpath.out           |   4 +-
>>  tests/qapi-schema/include-repetition.out        |   4 +-
>>  tests/qapi-schema/include-simple.out            |   4 +-
>>  tests/qapi-schema/indented-expr.out             |   7 +-
>>  tests/qapi-schema/qapi-schema-test.out | 186
>> +++++++++++++++++-------
>>  tests/qapi-schema/returns-int.out               |   5 +-
>>  tests/qapi-schema/test-qapi.py                  |  37 ++++-
>>  tests/qapi-schema/type-bypass.out               |   7 +-
>>  16 files changed, 210 insertions(+), 115 deletions(-)
>
> We have a lot more negative than positive tests of the parser (good
> thing, because that meant fewer .out files to update to the new format).
>
> No change to actual qemu code, and proves that the previous three
> patches have set up enough of a framework to accurately cover our testsuite.
>
>> 
>> diff --git a/tests/qapi-schema/alternate-good.out
>> b/tests/qapi-schema/alternate-good.out
>> index 99848ee..0cbdfa1 100644
>> --- a/tests/qapi-schema/alternate-good.out
>> +++ b/tests/qapi-schema/alternate-good.out
>> @@ -1,6 +1,9 @@
>> -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number',
>> 'int'), ('*name', 'str')]))]),
>> - OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]),
>> - OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value',
>> 'int'), ('string', 'Enum'), ('struct', 'Data')]))])]
>> -[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']},
>> - {'enum_name': 'AltKind', 'enum_values': None}]
>> -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number',
>> 'int'), ('*name', 'str')]))])]
>> +alternate Alt
>> +    case value: int flat=False
>> +    case string: Enum flat=False
>> +    case struct: Data flat=False
>
> I'm still not convinced whether we need .flat exposed through this much
> detail, or if we should just normalize plain unions into flat unions
> with implicit structs for each branch.  Changing your design will have
> obvious ripple effects here.

Yes.  I think it's okay as long as we keep it out of external
interfaces.

>> +++ b/tests/qapi-schema/data-member-array.out
>> @@ -1,5 +1,8 @@
>> -[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]),
>> - OrderedDict([('struct', 'def'), ('data', OrderedDict([('array',
>> ['abc'])]))]),
>> - OrderedDict([('command', 'okay'), ('data',
>> OrderedDict([('member1', ['int']), ('member2', ['def'])]))])]
>> -[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}]
>> -[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array',
>> ['abc'])]))])]
>> +object :obj-okay-args
>> +    member member1: intList optional=False
>
> Took me a moment to realize the object is an implicit one (named
> ':obj-okay-args') and not a typo for 'object: obj-okay-args' consistent
> with members being listed 'name: type'.  But not worth changing things,
> as it is sufficiently unambiguous to serve as a valid test.

Perhaps omitting the ':' after member names would be less confusing.

>> +object UserDefFlatUnion
>> +    base UserDefUnionBase
>> +    tag enum1
>> +    case value1: UserDefA flat=True
>> +    case value2: UserDefB flat=True
>> +    case value3: UserDefB flat=True
>> +object UserDefFlatUnion2
>> +    base UserDefUnionBase
>> +    tag enum1
>> +    case value1: UserDefC flat=True
>> +    case value2: UserDefB flat=True
>> +    case value3: UserDefA flat=True
>> +object UserDefNativeListUnion
>> +    case integer: intList flat=False
>> +    case s8: int8List flat=False
>> +    case s16: int16List flat=False
>> +    case s32: int32List flat=False
>> +    case s64: int64List flat=False
>> +    case u8: uint8List flat=False
>> +    case u16: uint16List flat=False
>> +    case u32: uint32List flat=False
>> +    case u64: uint64List flat=False
>> +    case number: numberList flat=False
>> +    case boolean: boolList flat=False
>> +    case string: strList flat=False
>> +    case sizes: sizeList flat=False
>> +enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32',
>> 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string',
>> 'sizes']
>
> Hmm. You are dumping the tag name and type of flat unions, but not of
> simple unions.  I would have expected:
>
> object UserDefNativeListUnion
>     member kind: UserDefNativeListUnionKind
>     tag kind
>     case integer: intList flat=False
> ...

See below.

> The above was fallout, while below is the meat of the new visitor.
>
>> +++ b/tests/qapi-schema/test-qapi.py
>> @@ -15,11 +15,34 @@ from pprint import pprint
>>  import os
>>  import sys
>>  
>> -try:
>> -    exprs = QAPISchema(sys.argv[1]).get_exprs()
>> -except SystemExit:
>> -    raise
>> +class QAPISchemaTestVisitor(QAPISchemaVisitor):
>> +    def visit_enum_type(self, name, info, values):
>> +        print 'enum %s %s' % (name, values)
>> +    def visit_object_type(self, name, info, base, members, variants):
>> +        print 'object %s' % name
>> +        if base:
>> +            print '    base %s' % base.name
>> +        for m in members:
>> +            print '    member %s: %s optional=%s' % (m.name, m.type.name,
>> +                                                     m.optional)
>> +        self._print_variants(variants)
>
> So here is where information was missing when visiting a simple union.
> Why didn't 'kind' get injected into members?  And...

Because members are the explicit members.  A simple union's implicit tag
member is in QAPISchemaObjectTypeVariants.  See further below.

>> +    def visit_alternate_type(self, name, info, variants):
>> +        print 'alternate %s' % name
>> +        self._print_variants(variants)
>> +    def visit_command(self, name, info, args, rets, gen, success_response):
>> +        print 'command %s %s -> %s' % (name, (args and args.name),
>> +                                       (rets and rets.name))
>> +        print '   gen=%s success_response=%s' % (gen, success_response)
>> +    def visit_event(self, name, info, data):
>> +        print 'event %s %s' % (name, data and data.name)
>>  
>> -pprint(exprs)
>> -pprint(enum_types)
>> -pprint(struct_types)
>> +    @staticmethod
>> +    def _print_variants(variants):
>> +        if variants:
>> +            if variants.tag_name:
>> +                print '    tag %s' % variants.tag_name
>
> ...shouldn't a simple union report 'kind' as its tag_name?

It could.  Here's why it currently isn't.

QAPISchemaObjectType has members .base and .local_members, which come
straight from the type definition.

These get flattened into .members.

We do create a simple union's implicit member 'kind', and store it in
.variants.tag_member.  But we don't add it to .local_members, only to
.members, similar to how .base's members are only in .members.

We don't print .members here.  Instead, we print .base, .local_members
and .variants.  The part printing .variants doesn't print
.variants.tag_member.

We need to draw the line on what to print *somewhere*.  Where exactly is
of course debatable.  My working idea on what to print here is "print
everything that isn't derived from other stuff".

>> +            for v in variants.variants:
>> + print ' case %s: %s flat=%s' % (v.name, v.type.name, v.flat)
>> +
>> +schema = QAPISchema(sys.argv[1])
>> +schema.visit(QAPISchemaTestVisitor())
>> diff --git a/tests/qapi-schema/type-bypass.out
>> b/tests/qapi-schema/type-bypass.out
>
> Overall, fairly slick, but I have enough questions about the
> representation of a simple union that I'll wait for the non-RFC respin
> to see if you change any design.

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1984 bytes --]

On 07/27/2015 03:54 AM, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>> New methods c_name(), c_type(), c_null(), json_type(),
>>> alternate_qtype().
>>>
>>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>>> ---
>>>  scripts/qapi.py | 72
>>> +++++++++++++++++++++++++++++++++++++++++++++++++++------
>>>  1 file changed, 65 insertions(+), 7 deletions(-)
>>>
>>
>> I just noticed:
>>
>>> @@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
>>>          for v in values:
>>>              assert isinstance(v, str)
>>>          self.values = values
>>> +    def c_type(self, is_param=False):
>>> +        return c_name(self.name)
>>> +    def c_null(self):
>>> +        return c_enum_const(self.name, self.values[0])
>>
>> What does this return for an empty enum, as in { 'enum':'Empty',
>> 'data':[] }?
> 
> I suspect self.values will be [] then, and self.values[0] will bomb.
> 
> Possible fixes:
> 
> * Outlaw empty enums
> 
> * Add the implicit MAX member to self.values[] (other code may have to
>   skip it)
> 
> * Catch the special case here, and return the implicit MAX member.

I'm leaning toward the third option here.

> 
>>               Our testsuite proves we can do that, even if our normal
>> .json code doesn't use it.
> 
> tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }

As I've mentioned elsewhere, most of our tests/qapi-schema/*.json merely
cover whether the parser is okay with the input, while
tests/qapi-schema/qapi-schema-test.json is the one that also tests that
the generated C code works; so sounds like that test should be enhanced
to cover some of these corner cases we have been considering in this
series (empty enum, command with no arguments and no returns, and so forth).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 25/47] qapi: Make generators work on sorted schema expressions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Order of expressions doesn't matter.  QAPISchema().get_exprs() returns
>> expressions in the order they're parsed.
>> 
>> I'm going to refactor this code.  To check refactorings, I want to
>> diff the generated code, and for that I need to preserve its order.
>> 
>> Since I don't feel like preserving parse order, I'm changing
>> get_expr() to return the expressions sorted by name.  This order will
>> be trivial to preserve.  It also makes the generated code slightly
>> easier to navigate.
>
> Huge change to the generated files, but that's to be expected.  Diffstat
> shows it is not a straight 1:1 reshuffle:
>
>  qapi-event.c                          |  890 +--
>  qapi-event.h                          |  190
>  qapi-types.c                          | 1996 +++----
>  qapi-types.h                          | 5420 ++++++++++----------
>  qapi-visit.c                          | 9088 +++++++++++++++++-----------------
>  qapi-visit.h                          |  746 +-
>  qga/qapi-generated/qga-qapi-types.c   |  148
>  qga/qapi-generated/qga-qapi-types.h   |  340 -
>  qga/qapi-generated/qga-qapi-visit.c   |  446 -
>  qga/qapi-generated/qga-qapi-visit.h   |   54
>  qga/qapi-generated/qga-qmp-commands.h |   38
>  qga/qapi-generated/qga-qmp-marshal.c  |  864 +--
>  qmp-commands.h                        |  376 -
>  13 files changed, 10308 insertions(+), 10288 deletions(-)
>
> but that's probably because you now emit forward declarations for things
> that occur alphabetically after their first use (since 8/47 in this
> series touched forward declarations), whereas before they happened to
> occurr in topological order from the parse.

Here's my cheap trick for sanity checking this commit: compare before
and after *sorted*.  Results:

* qapi-event.h
  - Members of enum QAPIEvent reordered, values changed accordingly (but
    they shouldn't matter)

* qapi-visit.c
  - A bunch of new forward declarations

* test-qapi-visit.c
  - One forward declaration gone.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi.py | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
>> 
>> diff --git a/scripts/qapi.py b/scripts/qapi.py
>> index cac7ab5..20ffdaf 100644
>> --- a/scripts/qapi.py
>> +++ b/scripts/qapi.py
>> @@ -1017,7 +1017,14 @@ class QAPISchema(object):
>>          self.check()
>>  
>>      def get_exprs(self):
>> -        return [expr_elem['expr'] for expr_elem in self.exprs]
>> +        def expr_name(expr):
>> +            name = expr.get('enum') or expr.get('union') \
>> +                   or expr.get('alternate') or expr.get('struct') \
>> +                   or expr.get('command') or expr.get('event')
>> +            assert name
>> +            return name
>
> When I was working on this file earlier, I half toyed with the idea of
> adding expr_elem['name'] holding the entity name, and expr_elem['meta']
> holding what meta-type the entity represents; it might have made some of
> our later 6-way switches simpler.  But with your hierarchy of actual
> objects, I'm not sure my idea helps any more.

The more work we do with the new internal representation instead of the
syntax tree, the less it'll help.

>> +        return sorted([expr_elem['expr'] for expr_elem in self.exprs],
>> +                      key=expr_name)
>
> Python has some nice compact syntactical gems - I'd hate the amount of
> boilerplate required to write this same filter using straight C code :)
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Fixes flat unions to get the base's base members.  Test case is from
>> commit 2fc0043, in qapi-schema-test.json:
>> 
>
>> 
>> Flat union visitors remain broken.  They'll be fixed next.
>
> Sadly, the generated files had a huge diffstat, making it very hard to
> determine if this change is sane:
>
>  qapi-types.c                        | 1412 ++++++-------
>  qapi-types.h                        | 3705 ++++++++++++++++++++----------------
>  qga/qapi-generated/qga-qapi-types.c |  110 -
>  qga/qapi-generated/qga-qapi-types.h |  542 +++--
>  4 files changed, 3208 insertions(+), 2561 deletions(-)
>
> At first, it looks easy, as in:
>
> qapi-types.c:
> +const int BlockdevRef_qtypes[QTYPE_MAX] = {
> ...
>  const char *const BlockdevRefKind_lookup[] = {
> ...
> -const int BlockdevRef_qtypes[QTYPE_MAX] = {
> ...
>
> (that is, the new visitor outputs the two arrays in a different order -
> I can live with that).

The old code generates enum lookup tables for implicit union or
alternate enumerations from within the union or alternate code.

The new code doesn't, because it makes all these enums full-fledged
types, so the ordinary enum generator does everything we need.

And that's why generating stuff in alphabetical doesn't help here: the
new types get inserted in the sorted sequence, but that's not where the
old code generated the implicit enum stuff.

> Then it gets a bit crazy when using normal diff algorithms:
>
> -void qapi_free_int32List(int32List *obj)
> +void qapi_free_ACPIOSTInfo(ACPIOSTInfo *obj)
>
> where declarations are rearranged (so the previous commit, which
> attempted to sort declarations to make this one easier, did not quite
> succeed).

I found that quite disappointing myself, but I didn't have better ideas
at the time.

> Even in qapi-types.h things were rearranged:
>
>
> -#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL
> -#define QAPI_TYPES_BUILTIN_STRUCT_DECL
> +#ifndef QAPI_TYPES_BUILTIN
> +#define QAPI_TYPES_BUILTIN
>
>
> -typedef struct int32List {
> +typedef struct boolList boolList;
> +
> +struct boolList {
>
> I can understand the minor change in #ifdef name, and even the
> separation between typedef and struct declaration (even though the old
> approach of doing it all at once works).

Artifact of my attempt to DRY: I sometimes need the typedef in one place
(say .h) and the actual struct in another place (say .c).  I make do
with the functions generating that even when I want both of them in the
same place.

>                                           But the overall diff in the
> generated files would be easier to review if done in stages, and either
> make 25/47 sort closer to what this patch does, or add yet more
> prerequisite patches that further tweak sorting.  Ideally, this patch
> will be a lot easier to review if the generated code is much closer to
> the pre-patch version (that is, separating sorting changes from other
> changes).  On the other hand, yes, it's kind of a pain to patch old code
> to do things differently just before throwing it away with the new code
> in its place,

Especially when you're throwing away the old code because you can't
stand working with it :)

>               so it becomes a judgment call of how confident we want to
> be that the new implementation isn't breaking things.

Different idea: I hack up a script to split file FOO into one part per
top-level syntactical construct (typedef, function, whatever), and store
each part in a file named like NUM-NAME-FOO.  The script will most
likely be hairy, so use diff FOO <(cat *-FOO) to verify it works.  Then
diff old/*-NAME-FOO new/*-NAME-FOO to see what changed.

Writing the script will take non-trivial effort, but I'll try if you
think it's needed.

For what it's worth, I sanity-checked with my "diff sorted" trick
mentioned in my reply to your review of PATCH 25.

> I was able to make a bit more sense of things by using git's 'diff
> patience' algorithm, which showed things more as code motion rather than
> one-or-two-line changes to lots of common boilerplate, but even that
> diffstat is still big:
>
>  qapi-types1.c | 2030 +++++++++++++++----------------
>  qapi-types1.h | 3707 +++++++++++++++++++++++++++++++++------------------------
>  2 files changed, 3148 insertions(+), 2589 deletions(-)
>
> That said, I'll still at least review this code by inspection, and
> things still compile fine, so although I'm reluctant to give R-b while
> the patch is in RFC stage (because I didn't want to take the time to be
> certain the results are the same amidst so much churn), they are mostly
> sane.

Fair enough.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-types.py | 268 +++++++++++++-------------------
>>  tests/qapi-schema/qapi-schema-test.json |   4 +-
>>  2 files changed, 114 insertions(+), 158 deletions(-)
>> 
>> diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
>> index a48ad9c..d6185c6 100644
>> --- a/scripts/qapi-types.py
>> +++ b/scripts/qapi-types.py
>> @@ -2,88 +2,67 @@
>>  # QAPI types generator
>>  #
>>  # Copyright IBM, Corp. 2011
>> +# Copyright (c) 2013-2015 Red Hat Inc.
>>  #
>>  # Authors:
>>  #  Anthony Liguori <aliguori@us.ibm.com>
>> +#  Markus Armbruster <armbru@redhat.com>
>>  #
>>  # This work is licensed under the terms of the GNU GPL, version 2.
>>  # See the COPYING file in the top-level directory.
>>  
>> -from ordereddict import OrderedDict
>>  from qapi import *
>>  
>> -def generate_fwd_builtin(name):
>> -    return mcgen('''
>> -
>> -typedef struct %(name)sList {
>
> For example, the change of splitting this into:
>
> typedef struct %(name)sList;
>
> struct %(name)sList {
>
> done as a separate patch would make the generated diff of this patch
> smaller.

I'm willing to mess with the old code *locally* to reduce the diffs, if
that helps.

>> -def generate_struct_fields(members):
>> +def gen_struct_field(name, typ, optional):
>>      ret = ''
>>  
>> -    for argname, argentry, optional in parse_args(members):
>> -        if optional:
>> -            ret += mcgen('''
>> +    if optional:
>> +        ret += mcgen('''
>
> (sometimes it's annoying that python requires indentation changes when
> changing control flow; for C files, sometimes I split a patch that
> merely adds {} and indentation from another patch that changes logic, so
> that the logic change isn't drowned by the reindentation)

I sometimes use C-c C-w in diff-mode to view a patch hunk without its
whitespace changes.

>> +class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
>> +    def __init__(self):
>> +        self.decl = None
>> +        self.defn = None
>> +        self.fwdecl = None
>> +        self.fwdefn = None
>> +        self.btin = None
>> +    def visit_begin(self):
>> +        self.decl = ''
>> +        self.defn = ''
>> +        self.fwdecl = ''
>> +        self.fwdefn = ''
>> +        self.btin = guardstart('QAPI_TYPES_BUILTIN')
>> +    def visit_end(self):
>> +        self.decl = self.fwdecl + self.decl
>> +        self.fwdecl = None
>> +        self.defn = self.fwdefn + self.defn
>> +        self.fwdefn = None
>> +        # To avoid header dependency hell, we always generate
>> +        # declarations for built-in types in our header files and
>> +        # simply guard them.
>> +        self.btin += guardend('QAPI_TYPES_BUILTIN')
>> +        self.decl = self.btin + self.decl
>> +        self.btin = None
>> +        # Doesn't work for cases where we link in multiple objects
>> +        # that have the functions defined, so generate them only with
>> +        # option -b (do_builtins).
>
> Does this comment...
>
>> +    def _gen_type_cleanup(self, name):
>> +        self.decl += generate_type_cleanup_decl(name)
>> +        self.defn += generate_type_cleanup(name)
>> +    def visit_enum_type(self, name, info, values):
>> +        self.fwdecl += generate_enum(name, values)
>> +        self.fwdefn += generate_enum_lookup(name, values)
>> +    def visit_array_type(self, name, info, element_type):
>> +        if isinstance(element_type, QAPISchemaBuiltinType):
>> +            self.btin += gen_fwd_object_or_array(name)
>> +            self.btin += gen_array(name, element_type)
>> +            self.btin += generate_type_cleanup_decl(name)
>> +            if do_builtins:
>> +                self.defn += generate_type_cleanup(name)
>
> ...fit better here?

If I rephrase it, probably yes.

>> +        else:
>> +            self.fwdecl += gen_fwd_object_or_array(name)
>> +            self.decl += gen_array(name, element_type)
>> +            self._gen_type_cleanup(name)
>> +    def visit_object_type(self, name, info, base, members, variants):
>> +        if info:
>
> So to make sure I understand, this ensures that implicit types (those
> handled merely by parameters in a function or as members of a struct)
> have no declarations required in the -types files.

Yes, (not info) means the object type is implicit.  We create implicit
object types for command parameters, command returns, and event
paramaters that aren't type names.

We don't generate C for these types, they're purely internal.  We
generate *parameter lists* for them.

Hmm, that's actually not true for command returns.  And current master
duly bombs if I add such a command to qapi-schema-test.json:

    { 'command': 'user_def_cmd4', 'returns': { 'a': 'int' } }

      GEN   tests/test-qmp-commands.h
    Traceback (most recent call last):
      File "/work/armbru/qemu/scripts/qapi-commands.py", line 360, in <module>
        ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
      File "/work/armbru/qemu/scripts/qapi-commands.py", line 29, in generate_command_decl
        ret_type=c_type(ret_type), name=c_name(name),
      File "/work/armbru/qemu/scripts/qapi.py", line 924, in c_type
        assert isinstance(value, str) and value != ""
    AssertionError

The easy fix is to outlaw it.

>> +            self.fwdecl += gen_fwd_object_or_array(name)
>> +            if variants:
>> +                self.decl += gen_union(name, base, variants)
>
> Is it worth 'assert not members' at this point?

Yes, with a TODO, because it's an artificial restriction I'd like to
lift some day.

>> +            else:
>> +                self.decl += gen_struct(name, base, members)
>> +            self._gen_type_cleanup(name)
>> +    def visit_alternate_type(self, name, info, variants):
>> +        self.fwdecl += gen_fwd_object_or_array(name)
>> +        self.fwdefn += gen_alternate_qtypes(name, variants)
>> +        self.decl += gen_union(name, None, variants)
>> +        self.decl += gen_alternate_qtypes_decl(name)
>> +        self._gen_type_cleanup(name)
>> +
>
> The visitor looks reasonable from inspection review.
>
>>  do_builtins = False
>>  
>>  (input_file, output_dir, do_c, do_h, prefix, opts) = \
>> @@ -325,77 +348,10 @@ fdecl.write(mcgen('''
>>  #include <stdint.h>
>>  '''))
>>  
>> -exprs = QAPISchema(input_file).get_exprs()
>> -
>> -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
>> -for typename in builtin_types.keys():
>> -    fdecl.write(generate_fwd_builtin(typename))
>> -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
>
> This is a place where adding sorting in 25/47 (or a separate patch)
> would make the impact on generated code from this patch smaller.

You mean iterating over sorted(builtin_types.keys())?  I can try and see
whether it helps.

>> -
>> -for expr in exprs:
>> -    ret = ""
>> -    if expr.has_key('struct'):
>> -        ret += generate_fwd_struct(expr['struct'])
>> -    elif expr.has_key('enum'):
>> -        ret += generate_enum(expr['enum'], expr['data'])
>> -        ret += generate_fwd_enum_struct(expr['enum'])
>> -        fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
>> -    elif expr.has_key('union'):
>> -        ret += generate_fwd_struct(expr['union'])
>> -        enum_define = discriminator_find_enum_define(expr)
>> -        if not enum_define:
>> - ret += generate_enum('%sKind' % expr['union'],
>> expr['data'].keys())
>> -            fdef.write(generate_enum_lookup('%sKind' % expr['union'],
>> -                                            expr['data'].keys()))
>> -    elif expr.has_key('alternate'):
>> -        ret += generate_fwd_struct(expr['alternate'])
>> - ret += generate_enum('%sKind' % expr['alternate'],
>> expr['data'].keys())
>> -        fdef.write(generate_enum_lookup('%sKind' % expr['alternate'],
>> -                                        expr['data'].keys()))
>
> Tweaking the order of _qtypes[] vs. _lookup[] here in a pre-req patch
> would also help review of the diff generated by this patch.

Can try and see.

>> -        fdef.write(generate_alternate_qtypes(expr))
>> -    else:
>> -        continue
>> -    fdecl.write(ret)
>> -
>> -# to avoid header dependency hell, we always generate declarations
>> -# for built-in types in our header files and simply guard them
>> -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
>> -for typename in builtin_types.keys():
>> -    fdecl.write(generate_type_cleanup_decl(typename + "List"))
>> -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
>> -
>> -# ...this doesn't work for cases where we link in multiple objects that
>> -# have the functions defined, so we use -b option to provide control
>> -# over these cases
>> -if do_builtins:
>> -    for typename in builtin_types.keys():
>> -        fdef.write(generate_type_cleanup(typename + "List"))
>
> Another iteration worth sorting in an earlier patch.

Can try and see.

>> -
>> -for expr in exprs:
>> -    ret = ""
>> -    if expr.has_key('struct'):
>> -        ret += generate_struct(expr) + "\n"
>> -        ret += generate_type_cleanup_decl(expr['struct'] + "List")
>> -        fdef.write(generate_type_cleanup(expr['struct'] + "List"))
>> -        ret += generate_type_cleanup_decl(expr['struct'])
>> -        fdef.write(generate_type_cleanup(expr['struct']))
>
> I think part of the large size of the generated diff comes from whether
> array fooList types are emitted in a different order via the visitor.

That's harder to avoid.  The old code generates them from within the
code generating the element type.  The new code doesn't, because it
makes all array types full-fledged types, which get visited like any
other type.

>> -    elif expr.has_key('union'):
>> -        ret += generate_union(expr, 'union') + "\n"
>> -        ret += generate_type_cleanup_decl(expr['union'] + "List")
>> -        fdef.write(generate_type_cleanup(expr['union'] + "List"))
>> -        ret += generate_type_cleanup_decl(expr['union'])
>> -        fdef.write(generate_type_cleanup(expr['union']))
>> -    elif expr.has_key('alternate'):
>> -        ret += generate_union(expr, 'alternate') + "\n"
>> -        ret += generate_type_cleanup_decl(expr['alternate'] + "List")
>> -        fdef.write(generate_type_cleanup(expr['alternate'] + "List"))
>> -        ret += generate_type_cleanup_decl(expr['alternate'])
>> -        fdef.write(generate_type_cleanup(expr['alternate']))
>> -    elif expr.has_key('enum'):
>> -        ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List")
>> -        fdef.write(generate_type_cleanup(expr['enum'] + "List"))
>> -    else:
>> -        continue
>> -    fdecl.write(ret)
>> +schema = QAPISchema(input_file)
>> +gen = QAPISchemaGenTypeVisitor()
>> +schema.visit(gen)
>> +fdef.write(gen.defn)
>> +fdecl.write(gen.decl)
>>  
>>  close_output(fdef, fdecl)
>> diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
>> index a9e5aab..257b4d4 100644
>> --- a/tests/qapi-schema/qapi-schema-test.json
>> +++ b/tests/qapi-schema/qapi-schema-test.json
>> @@ -39,8 +39,8 @@
>>    'data': { 'value1' : 'UserDefA',
>>              'value2' : 'UserDefB',
>>              'value3' : 'UserDefB' } }
>> -# FIXME generated struct UserDefFlatUnion has members for direct base
>> -# UserDefUnionBase, but lacks members for indirect base UserDefZero
>> +# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
>> +# members of indirect base UserDefZero
>>  
>>  { 'struct': 'UserDefUnionBase',
>>    'base': 'UserDefZero',
>> 
>
> Overall, moving in the right direction.

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Fixes flat unions to get the base's base members.  Test case is from
>> commit 2fc0043, in qapi-schema-test.json:
>> 
>
> Okay, I see a cause for part of my confusion.
>
>>  
>> +class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
>
>> +    def visit_end(self):
>> +        self.decl = self.fwdecl + self.decl
>> +        self.fwdecl = None
>> +        self.defn = self.fwdefn + self.defn
>> +        self.fwdefn = None
>> +        # To avoid header dependency hell, we always generate
>> +        # declarations for built-in types in our header files and
>> +        # simply guard them.
>> +        self.btin += guardend('QAPI_TYPES_BUILTIN')
>> +        self.decl = self.btin + self.decl
>> +        self.btin = None
>
> The new code goes to great lengths to separate all builtin decls up
> front.  But...

Because they all need to be under the QAPI_TYPES_BUILTIN guard.  If I
didn't separate them, each of them would need its own guard, which would
be ugly.

>> +        # Doesn't work for cases where we link in multiple objects
>> +        # that have the functions defined, so generate them only with
>> +        # option -b (do_builtins).
>> +    def _gen_type_cleanup(self, name):
>> +        self.decl += generate_type_cleanup_decl(name)
>> +        self.defn += generate_type_cleanup(name)
>> +    def visit_enum_type(self, name, info, values):
>> +        self.fwdecl += generate_enum(name, values)
>> +        self.fwdefn += generate_enum_lookup(name, values)
>> +    def visit_array_type(self, name, info, element_type):
>> +        if isinstance(element_type, QAPISchemaBuiltinType):
>> +            self.btin += gen_fwd_object_or_array(name)
>> +            self.btin += gen_array(name, element_type)
>> +            self.btin += generate_type_cleanup_decl(name)
>> +            if do_builtins:
>> +                self.defn += generate_type_cleanup(name)
>
> ...it is still interleaving builtin defns with everything else.

Because there's no such need for the builtin definitions.

>> -exprs = QAPISchema(input_file).get_exprs()
>> -
>> -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
>> -for typename in builtin_types.keys():
>> -    fdecl.write(generate_fwd_builtin(typename))
>> -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
>
> Meanwhile, the old code did builtin intList definitions up front...
>
>> -
>> -for expr in exprs:
>> -    ret = ""
>> -    if expr.has_key('struct'):
>
>> -# to avoid header dependency hell, we always generate declarations
>> -# for built-in types in our header files and simply guard them
>> -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
>> -for typename in builtin_types.keys():
>> -    fdecl.write(generate_type_cleanup_decl(typename + "List"))
>> -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
>
> ...but declared the cleanup functions at the end (so code motion 1 is
> that the cleanup functions are now interleaved with the intList
> declarations)...
>
>> -
>> -# ...this doesn't work for cases where we link in multiple objects that
>> -# have the functions defined, so we use -b option to provide control
>> -# over these cases
>> -if do_builtins:
>> -    for typename in builtin_types.keys():
>> -        fdef.write(generate_type_cleanup(typename + "List"))
>
> ...and wrote the definitions at the end (so code motion 2 is that the
> definitions are now interleaved with all other definitions).
>
> Attached a few quick hacks that reduce (but not eliminate) some of the
> visitor's churn to the generated files, by consolidating the location of
> the builtins and cleaning up struct declarations as separate cleanups
> (to be applied prior to 26/47), then updating the visitor to use the
> same builtin order (to be applied after or squashed with 26/47).
>
> It still doesn't solve everything, but with those hacks, I was able to
> get to a slightly more manageable:
>
>  qapi-types.c                        |  864 ++++----
>  qapi-types.h                        | 3602 ++++++++++++++++++------------------
>  qga/qapi-generated/qga-qapi-types.c |  110 -
>  qga/qapi-generated/qga-qapi-types.h |  405 ++--
>  4 files changed, 2585 insertions(+), 2396 deletions(-)

Still plenty bad...

> I'm sure there are further things that could be done, but at this point,
> I hope you get my picture, and I'll quit focusing on this particular patch.

We need to decide how much code churn to accept just for making the diff
of the generated code easier to review.

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2886 bytes --]

On 07/23/2015 09:44 PM, Eric Blake wrote:
> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> To eliminate the temptation for clients to look up types by name
>> (which are not ABI), replace all type names by meaningless strings.
>>
>> Reduces output of query-schema by 9 out of 80KiB.
> 
> I'm not sure whether I like this or not.  It does make sense from the
> perspective of forcing clients to stick to ABI queries, but makes it a
> bit harder to navigate things except by automated scripts.
> 

>>  
>> +import string
>>  from qapi import *
>>  
>> +def _b32digit(num):
>> +    return (string.lowercase + string.digits[2:])[num]
> 
> This feels a bit too magic for me to decipher late at night.

Looking at this again after a weekend: you are doing a poor-man's base32
encoding that maps a number between 0 and 31 inclusive to
"abc...xyz234567" (skipping 0 and 1 due to some fonts being unclear with
O and l/I).

Question - if we really want to compress things down to an integer,
would it make any more sense to actually use an integer instead of a
string encoding of an unusual base32 alphabet (that is, output an
integer instead of a string)?  Conversely, if we are going to insist on
an encoding, can we use the RFC4648 base32 definition (upper case, not
lower case)?

More thoughts: I'm probably okay with hiding type names (since they
aren't ABI), but can we come up with an output format that is more
conducive for use by the end user?  Considering that we are returning a
JSON array, what if we return the integer offset of the type as recorded
in the returned array.  That is, if we have
 { 'name':'ACPI_DEVICE_OST', 'meta-type':'event', 'data':100 }
in slot [0] of the return array, then
 { 'name': ':aaa', 'meta-type': 'object', 'members': [ { 'name': 'info',
'type': 250 } ] }
in slot [100], then the definition for type ':ae5' in slot [250], and so
forth.  My goal here is not a topologically sorted return array, so much
as a shorthand where using an integer for a type-name means that we can
quickly locate that offset within the JSON array, instead of having to
do a linear search over the entire array for an array member that has
the matching name.  Or if type names are truly unimportant, then omit
names for type elements (by making name optional in the introspection
qapi description), and using ONLY offsets in the returned JSON array for
referring to types.  Of course, if we do this, life gets a lot trickier
for adding filtering down to a subset of the overall schema (unless you
don't mind populating lots of 'null' entries for parts that get filtered
out so that the parts that are displayed are always at the same array
offset, just with less overall output bulk due to the filtering).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1141 bytes --]

On 07/27/2015 10:09 AM, Markus Armbruster wrote:

> 
>> I'm sure there are further things that could be done, but at this point,
>> I hope you get my picture, and I'll quit focusing on this particular patch.
> 
> We need to decide how much code churn to accept just for making the diff
> of the generated code easier to review.

At this point, I'd be happy with just adding a script or other
high-level instructions in the commit message that says how to divide a
generated file into pieces (pull out all *List types into one piece, all
typedefs into another, etc) and which can be done both pre- and
post-patch.  With pieces in hand, if you can easily compare that each
pair is minimally different, then you have a nice reassurance that the
difference in the overall file is due merely to differences in how the
pieces are interleaved, and not to added or unintentionally dropped
material.  As you pointed out, there comes a point of diminishing
returns in trying to clean up code that will just be discarded.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Fixes flat unions to visit the base's base members (the previous
>> commit merely added them to the struct).  Same test case.
>> 
>> Patch's effect on visit_type_UserDefFlatUnion():
>> 
>>      static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp)
>>      {
>>          Error *err = NULL;
>> 
>>     +    visit_type_int(m, &(*obj)->integer, "integer", &err);
>>     +    if (err) {
>>     +        goto out;
>>     +    }
>>          visit_type_str(m, &(*obj)->string, "string", &err);
>>          if (err) {
>>              goto out;
>> 
>> Test cases updated for the bug fix.
>> 
>> Fixes alternates to generate a visitor for their implicit enumeration
>> type.  None of them are currently used, obviously.  Example:
>> block-core.json's BlockdevRef now generates
>> visit_type_BlockdevRefKind().
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-visit.py                   | 254 ++++++++++++--------------------
>>  tests/qapi-schema/qapi-schema-test.json |   3 -
>>  tests/test-qmp-input-strict.c           |   2 +-
>>  tests/test-qmp-input-visitor.c          |   4 +-
>>  4 files changed, 100 insertions(+), 163 deletions(-)
>
> Another conversion that results in a fairly large diffstat to the
> generated files:
>
>  qapi-visit.c                        | 4542 ++++++++++++++++++------------------
>  qapi-visit.h                        |  256 --
>  qga/qapi-generated/qga-qapi-visit.c |   88
>  qga/qapi-generated/qga-qapi-visit.h |   36
>  4 files changed, 2355 insertions(+), 2567 deletions(-)
>
> Same complaints as in 26/47, where splitting some of the cleanups into
> separate patches would make it easier to validate that the final
> conversion is correct.
>
> Here, a very common thing in the generated .c file is that you end up
> swapping the order of visit_type_foo and visit_type_fooList, for any
> time when foo is an enum. [1]

Again, this is because the old code generates the array visitor from
within the code generating the element type visitor, while the new code
makes all array types full-fledged types, which get visited like any
other type.

>> 
>> diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
>> index a52a572..135e7c1 100644
>> --- a/scripts/qapi-visit.py
>> +++ b/scripts/qapi-visit.py
>> @@ -12,7 +12,6 @@
>>  # This work is licensed under the terms of the GNU GPL, version 2.
>>  # See the COPYING file in the top-level directory.
>>  
>> -from ordereddict import OrderedDict
>>  from qapi import *
>>  import re
>>  
>> @@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type):
>>          return ''
>>      implicit_structs_seen.add(type)
>>      ret = ''
>> -    if type not in struct_fields_seen:
>> +    if type.name not in struct_fields_seen:
>>          # Need a forward declaration
>>          ret += mcgen('''
>>  
>>  static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
>>  ''',
>> -                     c_type=type_name(type))
>> +                     c_type=type.c_name())
>
> This looks a little fishy on first read; why are we calling
> type.c_name() (and not type.c_type()) when assigning to a placeholder
> named %(c_type)s?  But on second thought, it looks correct: since we
> really do want the name of the type (and not the magic '*' pointer suffix).

Most of the time, c_type is an identifier naming a type (something like
typ.c_name()), but sometimes it's a type expression (something like
typ.c_type()).  If that turns out to be too confusing, we can use
different names for the two.  Not exactly looking forward to tracking
them down, though...

> Still, it might be nicer to name things %(c_name)s here; and in that
> case, it's more of a pre-existing cleanup that might be better floating
> into one of your earlier patches.

This patch tries very hard not to touch lines without need.  Cleanup is
left for PATCH 33.

I could do some (but not all) cleanup before converting to
QAPISchemaVisitor.  But I'm afraid that just complicates things even
more.

>>  
>>      ret += mcgen('''
>>  
>> @@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
>>      error_propagate(errp, err);
>>  }
>>  ''',
>> -                 c_type=type_name(type))
>> +                 c_type=type.c_name())
>
> same here.
>
>>      return ret
>>  
>>  def generate_visit_struct_fields(name, members, base = None):
>> @@ -74,24 +73,24 @@ if (err) {
            ret += mcgen('''
    visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);
    if (err) {
>>      goto out;
>>  }
>>  ''',
>> -                     type=type_name(base), c_name=c_name('base'))
>> +                     type=base.c_name(), c_name=c_name('base'))
>
> And this one's pointless: c_name('base') == 'base'. Pointless since
> commit 622f557 introduced type inheritance.  Why do we even need
> %(c_name)s if we are always passing a constant string?

My best guess: to keep the code similar to other code generating
visit_type_FOO() calls?

> Oh, and that means our generator has a collision bug that none of my
> added tests have exposed yet: you cannot have a base class and
> simultaneously add a member named 'base':
>
> { 'struct': 'Base', 'data': { 'i': 'int' } }
> { 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }
>
> because the generated C code is trying to use the name 'base' for its
> own purposes.

*sigh*

>                I guess that means more pre-req patches to the series to
> expose the bug, and either tighten the parser to reject things for now
> (easiest) or update the generator to not collide (harder, and fine for a
> later series).

Yes.

Life would be easier if the original authors had adopted sane naming
conventions from the start.

> By the way, now that we are emitting flat unions in such a way that you
> can cast to the base class, why don't we change our C code to do
> likewise?  That is, where we now have this generated C:
>
> struct BlockdevOptionsGenericFormat {
>     BlockdevRef *file;
> };
>
> struct BlockdevOptionsGenericCOWFormat {
>     BlockdevOptionsGenericFormat *base;
>     bool has_backing;
>     BlockdevRef *backing;
> };
>
> why can't we instead have an unboxed representation:
>
> struct BlockdevOptionsGenericFormat {
>     BlockdevRef *file;
> };
>
> /* This struct can be cast to BlockdevOptionsGenericFormat */
> struct BlockdevOptionsGenericCOWFormat {
>     BlockdevRef *file;
>     /* end of fields from base class BlockdevOptionsGenericFormat */
>     bool has_backing;
>     BlockdevRef *backing;
> };
>
> where client code that was referring to o->base->file now refers to o->file.

That's where I want to go, but not in this series.

>> +def gen_visit_union(name, base, variants):
>> +    ret = ''
>>  
>>      if base:
>> -        assert discriminator
>> -        base_fields = find_struct(base)['data'].copy()
>> -        del base_fields[discriminator]
>> -        ret += generate_visit_struct_fields(name, base_fields)
>> +        members = [m for m in base.members if m != variants.tag_member]
>> +        ret += generate_visit_struct_fields(name, members)
>
> Why not just visit ALL base class members, unconditionally?

This patch converts to QAPISchemaVisitor.  It tries hard to keep the
generated code the same.

The old code generates the discriminator visit into the visit_type_FOO()
and not the the visit_type_FOO_fields() helper (can't tell why), so the
new code does the same.

Simply visiting all members makes more sense, but it needs to be done in
a followup patch.  Since this series is already huge, it'll likely be in
a followup series.

>> -    if discriminator:
>> -        for key in members:
>> -            ret += generate_visit_implicit_struct(members[key])
>> +    for var in variants.variants:
>> +        if var.flat:
>> +            ret += generate_visit_implicit_struct(var.type)
>
> Okay, I see where you are using .flat from the initial parse.  I still
> think it is a bit odd that you are defining '.flat' for each 'variant'
> within 'variants', even though, for a given 'variants', all members will
> have the same setting of '.flat'.  That makes me wonder if '.flat'
> should belong instead to the top-level 'variants' struct rather than to
> each 'variant' member.

Two reasons for putting flat where it is:

* The philosophical one: from the generator's point of view, there's no
  fundamental reason why all variants need to be flat or none.  The
  generator really doesn't care.

* The pragmatic one (a.k.a. the real one): there are places where I use
  v.flat, but don't have the variants owning v handy.

> But again I wonder what would happen if you had instead normalized the
> input of simple unions into always having an implicit struct (with
> single member 'data'), so that by the time you get here, you only have
> to deal with a single representation of unions instead of having to
> still emit different things for flat vs. simple (since on the wire, we
> already proved simple is shorthand that can be duplicated by a flat union).

I hope we can get there!  But at this stage of the conversion, I want to
minimize output change, and .flat makes preserving all its warts much
easier.

>>  
>>      ret += mcgen('''
>>  
>> @@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
>>  ''',
>>                       name=c_name(name))
>>  
>> -    if not discriminator:
>> -        tag = 'kind'
>> -        disc_key = "type"
>> -    else:
>> -        tag = discriminator
>> -        disc_key = discriminator
>> +    disc_key = variants.tag_member.name
>> +    if not variants.tag_name:
>> +        # we pointlessly use a different key for simple unions
>
> We could fix that (as a separate patch); wonder how much C code it would
> affect.  A lot of these things that we can alter in generated code are
> certainly easier to see now that we have a clean generator :)

Yup, the warts stand out now.

>> +def gen_visit_decl(name, scalar=False):
>> +    c_type = c_name(name) + ' *'
>> +    if not scalar:
>> +        c_type += '*'
>>      return mcgen('''
>> -
>> -void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
>> +void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
>>  ''',
>> -                 name=c_name(name))
>> +                 c_name=c_name(name), c_type=c_type)
>
> Nice way to consolidate several near-identical copies.
>
>> +
>> +class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
>> +    def __init__(self):
>> +        self.decl = None
>> +        self.defn = None
>> +        self.btin = None
>> +    def visit_begin(self):
>> +        self.decl = ''
>> +        self.defn = ''
>> +        self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
>> +    def visit_end(self):
>> +        # to avoid header dependency hell, we always generate
>> +        # declarations for built-in types in our header files and
>> +        # simply guard them
>> +        self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL')
>> +        self.decl = self.btin + self.decl
>> +        self.btin = None
>> +        # ...this doesn't work for cases where we link in multiple
>> +        # objects that have the functions defined, so we use
>> +        # do_builtins (option -b) to provide control
>
> And once again, as in 26/47, this floats the .h file to have all builtin
> representations in one chunk (for continuity with pre-patch), but fails
> to do the same for the .c code...

Yes, because they all need to be under the
QAPI_VISIT_BUILTIN_VISITOR_DECL guard.

>> +    def visit_enum_type(self, name, info, values):
>> +        self.decl += gen_visit_decl(name, scalar=True)
>> +        self.defn += generate_visit_enum(name)
>> +    def visit_array_type(self, name, info, element_type):
>> +        decl = gen_visit_decl(name)
>> +        defn = gen_visit_list(name, element_type)
>> +        if isinstance(element_type, QAPISchemaBuiltinType):
>> +            self.btin += decl
>> +            if do_builtins:
>> +                self.defn += defn
>
> ...where the builtins are now interleaved with everything else instead
> of bunched together, making the generated diff larger and more confusing
> than necessary.

Yes, because there's no need to separate them out.

>> +        else:
>> +            self.decl += decl
>> +            self.defn += defn
>> +    def visit_object_type(self, name, info, base, members, variants):
>> +        if info:
>> +            self.decl += gen_visit_decl(name)
>> +            if variants:
>> +                self.defn += gen_visit_union(name, base, variants)
>
> Worth adding 'assert not members'?

Yes, with a TODO, because it's an artificial restriction I'd like to
lift some day.

>> +            else:
>> +                self.defn += gen_visit_struct(name, base, members)
>
> Or maybe we can someday consolidate these two into a single
> gen_visit_object, that handles all members and variants in a uniform
> manner, instead of our current differences. I wonder how much C code
> would be impacted?

Yes, that's how we'd lift the restriction.

>> +    def visit_alternate_type(self, name, info, variants):
>> +        self.decl += gen_visit_decl(name)
>> +        self.defn += gen_visit_alternate(name, variants)
>>  
>>  do_builtins = False
>>  
>> @@ -442,56 +428,10 @@ fdecl.write(mcgen('''
>>  ''',
>>                    prefix=prefix))
>>  
>> -exprs = QAPISchema(input_file).get_exprs()
>> -
>> -# to avoid header dependency hell, we always generate declarations
>> -# for built-in types in our header files and simply guard them
>> -fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
>> -for typename in builtin_types.keys():
>> -    fdecl.write(generate_declaration(typename, builtin_type=True))
>> -fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
>> -
>> -# ...this doesn't work for cases where we link in multiple objects that
>> -# have the functions defined, so we use -b option to provide control
>> -# over these cases
>> -if do_builtins:
>> -    for typename in builtin_types.keys():
>> -        fdef.write(generate_visit_list(typename))
>
> Again, a well-placed sorted() over these two loops in a pre-req patch
> will minimize the churn on the builtins.

Can try and see.

>> -
>> -for expr in exprs:
>> -    if expr.has_key('struct'):
>> -        ret = generate_visit_struct(expr)
>> -        ret += generate_visit_list(expr['struct'])
>> -        fdef.write(ret)
>> -
>> -        ret = generate_declaration(expr['struct'])
>> -        fdecl.write(ret)
>> -    elif expr.has_key('union'):
>> -        ret = generate_visit_union(expr)
>> -        ret += generate_visit_list(expr['union'])
>> -        fdef.write(ret)
>> -
>> -        enum_define = discriminator_find_enum_define(expr)
>> -        ret = ""
>> -        if not enum_define:
>> -            ret = generate_decl_enum('%sKind' % expr['union'])
>
> Nice that the new visitor automatically visits any implicit enum,
> without us having to special case it.

Making implicit stuff explicit in the internal representation kills
special cases and de-duplicates code :)

>> -        ret += generate_declaration(expr['union'])
>> -        fdecl.write(ret)
>> -    elif expr.has_key('alternate'):
>> -        ret = generate_visit_alternate(expr['alternate'], expr['data'])
>> -        ret += generate_visit_list(expr['alternate'])
>> -        fdef.write(ret)
>> -
>> -        ret = generate_decl_enum('%sKind' % expr['alternate'])
>> -        ret += generate_declaration(expr['alternate'])
>> -        fdecl.write(ret)
>> -    elif expr.has_key('enum'):
>> -        ret = generate_visit_list(expr['enum'])
>> -        ret += generate_visit_enum(expr['enum'])
>
> [1] swapping these two lines in a pre-req patch will minimize the churn
> of this conversion.

Can try and see.

>> -        fdef.write(ret)
>> -
>> -        ret = generate_decl_enum(expr['enum'])
>> -        ret += generate_enum_declaration(expr['enum'])
>> -        fdecl.write(ret)
>> +schema = QAPISchema(input_file)
>> +gen = QAPISchemaGenVisitVisitor()
>> +schema.visit(gen)
>> +fdef.write(gen.defn)
>> +fdecl.write(gen.decl)
>
> Again, overall impression is that your series is headed in the right
> direction.
>
> And nice that the TODOs in the testsuite pointed out what this fixes,
> for visiting indirect bases.

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 28/47] qapi-commands: Convert to QAPISchemaVisitor

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Output unchanged except for white-space.
>
> Indeed, and the diffstat shows it was only one blank line:
>
>  qga-qmp-marshal.c |    1 +
>  1 file changed, 1 insertion(+)
>
> MUCH friendlier to review :)
>
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py | 157
>> ++++++++++++++++++++++++++---------------------
>>  scripts/qapi.py          |   2 +-
>>  2 files changed, 87 insertions(+), 72 deletions(-)
>> 
>
> Here, I can confidently say:
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

>> +++ b/scripts/qapi.py
>> @@ -1366,7 +1366,7 @@ def c_type(value, is_param=False):
>>          return c_name(value) + pointer_suffix
>>  
>>  def is_c_ptr(value):
>> -    return c_type(value).endswith(pointer_suffix)
>> +    return value.endswith(pointer_suffix)
>
> Perhaps this cleanup could be floated earlier in the series?

Before this patch, is_c_ptr() is called with an argument suitable for
c_type().  It returns true when c_type() returns a string ending with
pointer_suffix.

This patch replaces uses of function c_type() by QAPISchemaType method
c_type().

Two ways to do that for the is_c_ptr() calls:

1. Change the argument to a QAPISchemaType object, and make is_c_ptr()
call its .c_type().

2. Change the argument to a string, and make the caller compute it by
calling the appropriate object's .c_type().

I picked 2.  It hardly matters, because is_c_ptr() will go away in the
next patch.

I could do a part of 2. in an earlier patch, namely lifting the c_type()
call into the callers.  But I can't do the conversion from function to
method any earlier, because the objects become available only in this
patch.

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3848 bytes --]

On 07/27/2015 11:53 AM, Markus Armbruster wrote:

>> Oh, and that means our generator has a collision bug that none of my
>> added tests have exposed yet: you cannot have a base class and
>> simultaneously add a member named 'base':
>>
>> { 'struct': 'Base', 'data': { 'i': 'int' } }
>> { 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }
>>
>> because the generated C code is trying to use the name 'base' for its
>> own purposes.
> 
> *sigh*
> 
>>                I guess that means more pre-req patches to the series to
>> expose the bug, and either tighten the parser to reject things for now
>> (easiest) or update the generator to not collide (harder, and fine for a
>> later series).
> 
> Yes.
> 
> Life would be easier if the original authors had adopted sane naming
> conventions from the start.

Like reserving ourselves a namespace based on single _ for internal use.
 We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple)
underscore for internally-generated purposes, freeing up 'base',
'*Kind', '*_MAX', and other namespace abuses back to the user.  Well, we
may also need to reserve mid-name double-underscore (that is, the user
can only supply double underscore at the beginning, but not middle, of
an identifier).  Ah well, food for thought for later patches.


>> Okay, I see where you are using .flat from the initial parse.  I still
>> think it is a bit odd that you are defining '.flat' for each 'variant'
>> within 'variants', even though, for a given 'variants', all members will
>> have the same setting of '.flat'.  That makes me wonder if '.flat'
>> should belong instead to the top-level 'variants' struct rather than to
>> each 'variant' member.
> 
> Two reasons for putting flat where it is:
> 
> * The philosophical one: from the generator's point of view, there's no
>   fundamental reason why all variants need to be flat or none.  The
>   generator really doesn't care.

And we may decide to exploit that down the road to allow some sort of
qapi syntax for explicitly designating a union branch as flat or boxed,
rather than the current approach of the type of union determining the
status of all branch members.

> 
> * The pragmatic one (a.k.a. the real one): there are places where I use
>   v.flat, but don't have the variants owning v handy.
> 
>> But again I wonder what would happen if you had instead normalized the
>> input of simple unions into always having an implicit struct (with
>> single member 'data'), so that by the time you get here, you only have
>> to deal with a single representation of unions instead of having to
>> still emit different things for flat vs. simple (since on the wire, we
>> already proved simple is shorthand that can be duplicated by a flat union).
> 
> I hope we can get there!  But at this stage of the conversion, I want to
> minimize output change, and .flat makes preserving all its warts much
> easier.

Agreed.  By the end of the series, I was convinced that the use of
.flat, at least in this series, makes sense.


>>> +    disc_key = variants.tag_member.name
>>> +    if not variants.tag_name:
>>> +        # we pointlessly use a different key for simple unions
>>
>> We could fix that (as a separate patch); wonder how much C code it would
>> affect.  A lot of these things that we can alter in generated code are
>> certainly easier to see now that we have a clean generator :)
> 
> Yup, the warts stand out now.

And I've already demonstrated what sort of cleanups can be done to
attack some of the warts:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

On 07/24/2015 06:01 AM, Markus Armbruster wrote:

>> It might be worth tweaking the generator to output a C comment either
>> here (at the start of the larger struct)...
>>
>>>         char *string;
>>>         EnumOne enum1;
>>
>> ...or here (at the end of the base struct) mentioning that
>> UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to
>> make it easier when reading the generated code to trace back to that
>> "inheritance" relationship.  Right now, there is nothing in the
>> generated UserDefFlatUnion that points you back to the qapi relationship
>> of a base class.  But it's not a show-stopper if you don't like my
>> suggestion.
> 
> I do like it.  Perhaps something like
> 
>     struct UserDefFlatUnion
>     {
>         /* Members inherited from UserDefUnionBase: */
>         char *string;
>         EnumOne enum1;
>         /* Own members: */
>         ...
>     };

What about nested types?  Whatever we pick has to look good when
indirect bases are involved. Suppose we have Base -> Mid -> Derived as a
struct inheritance.  If we generate a comment at both start and end of
visiting a base class (and visit base classes recursively), we'd have
something like:

struct Derived
{
    /* Members inherited from Mid */
    /* Members inherited from Base */
    int one;
    /* End members inherited from Base */
    int two;
    /* End members inherited from Mid */
    /* own members */
    int three;
    ...
};

Don't know if that helps you think about it more, but now I'm just
painting a bikeshed that just adds the tail comment.  Here's the
resulting RFC:

http://thread.gmane.org/gmane.comp.emulators.qemu/353204

and the resulting generated structs:

 struct BlockdevOptionsQcow2 {
-    BlockdevOptionsGenericCOWFormat *base;
+    BlockdevRef *file;
+    /* End fields inherited from BlockdevOptionsGenericFormat. */
+    bool has_backing;
+    BlockdevRef *backing;
+    /* End fields inherited from BlockdevOptionsGenericCOWFormat. */
     bool has_lazy_refcounts;
     bool lazy_refcounts;
...

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2947 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Fixes flat unions to visit the base's base members (the previous
> commit merely added them to the struct).  Same test case.
> 
> Patch's effect on visit_type_UserDefFlatUnion():
> 
>      static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp)
>      {
>          Error *err = NULL;
> 
>     +    visit_type_int(m, &(*obj)->integer, "integer", &err);
>     +    if (err) {
>     +        goto out;
>     +    }
>          visit_type_str(m, &(*obj)->string, "string", &err);
>          if (err) {
>              goto out;
> 

> +def gen_visit_union(name, base, variants):
> +    ret = ''
>  
>      if base:
> -        assert discriminator
> -        base_fields = find_struct(base)['data'].copy()
> -        del base_fields[discriminator]
> -        ret += generate_visit_struct_fields(name, base_fields)
> +        members = [m for m in base.members if m != variants.tag_member]
> +        ret += generate_visit_struct_fields(name, members)

Ouch. This hurts.  If the same class is used as both the base class of a
flat union, and the base class of an ordinary struct, then the struct
tries to visit the base class, but no longer parses the field that the
union was using as its discriminator.

We don't have any code that demonstrates this, but probably should.  I
ran into it while working up my POC of what it would take to unbox
inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)

We want visit_FOO_fields to visit _all_ fields of the struct, no matter
who called generate_visit_struct_fields().  So what you must instead do
here is use the fact that we've already visited the discriminator...

>  
> -    if discriminator:
> -        for key in members:
> -            ret += generate_visit_implicit_struct(members[key])
> +    for var in variants.variants:
> +        if var.flat:
> +            ret += generate_visit_implicit_struct(var.type)
>  
>      ret += mcgen('''
>  
> @@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
>  ''',
>                       name=c_name(name))
>  
> -    if not discriminator:
> -        tag = 'kind'
> -        disc_key = "type"
> -    else:
> -        tag = discriminator
> -        disc_key = discriminator
> +    disc_key = variants.tag_member.name
> +    if not variants.tag_name:
> +        # we pointlessly use a different key for simple unions
> +        disc_key = 'type'
>      ret += mcgen('''
> -        visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err);
> +        visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err);

...and omit this call if the flat union's base class already took care
of it.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/24/2015 06:01 AM, Markus Armbruster wrote:
>
>>> It might be worth tweaking the generator to output a C comment either
>>> here (at the start of the larger struct)...
>>>
>>>>         char *string;
>>>>         EnumOne enum1;
>>>
>>> ...or here (at the end of the base struct) mentioning that
>>> UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to
>>> make it easier when reading the generated code to trace back to that
>>> "inheritance" relationship.  Right now, there is nothing in the
>>> generated UserDefFlatUnion that points you back to the qapi relationship
>>> of a base class.  But it's not a show-stopper if you don't like my
>>> suggestion.
>> 
>> I do like it.  Perhaps something like
>> 
>>     struct UserDefFlatUnion
>>     {
>>         /* Members inherited from UserDefUnionBase: */
>>         char *string;
>>         EnumOne enum1;
>>         /* Own members: */
>>         ...
>>     };
>
> What about nested types?  Whatever we pick has to look good when
> indirect bases are involved. Suppose we have Base -> Mid -> Derived as a
> struct inheritance.  If we generate a comment at both start and end of
> visiting a base class (and visit base classes recursively), we'd have
> something like:
>
> struct Derived
> {
>     /* Members inherited from Mid */
>     /* Members inherited from Base */
>     int one;
>     /* End members inherited from Base */
>     int two;
>     /* End members inherited from Mid */
>     /* own members */
>     int three;
>     ...
> };

Documenting the direct ancestor in the code is more important than the
others, because you can always peruse the direct ancestor to find the
next one.

> Don't know if that helps you think about it more, but now I'm just
> painting a bikeshed that just adds the tail comment.  Here's the
> resulting RFC:
>
> http://thread.gmane.org/gmane.comp.emulators.qemu/353204
>
> and the resulting generated structs:
>
>  struct BlockdevOptionsQcow2 {
> -    BlockdevOptionsGenericCOWFormat *base;
> +    BlockdevRef *file;
> +    /* End fields inherited from BlockdevOptionsGenericFormat. */
> +    bool has_backing;
> +    BlockdevRef *backing;
> +    /* End fields inherited from BlockdevOptionsGenericCOWFormat. */
>      bool has_lazy_refcounts;
>      bool lazy_refcounts;
> ...

This and other followup patches are encouraging.  I'll gladly review
them when I can spare the cycles.  Right now, I need to focus them on
introspection.

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/27/2015 10:09 AM, Markus Armbruster wrote:
>
>> 
>>> I'm sure there are further things that could be done, but at this point,
>>> I hope you get my picture, and I'll quit focusing on this particular patch.
>> 
>> We need to decide how much code churn to accept just for making the diff
>> of the generated code easier to review.
>
> At this point, I'd be happy with just adding a script or other
> high-level instructions in the commit message that says how to divide a
> generated file into pieces (pull out all *List types into one piece, all
> typedefs into another, etc) and which can be done both pre- and
> post-patch.  With pieces in hand, if you can easily compare that each
> pair is minimally different, then you have a nice reassurance that the
> difference in the overall file is due merely to differences in how the
> pieces are interleaved, and not to added or unintentionally dropped
> material.  As you pointed out, there comes a point of diminishing
> returns in trying to clean up code that will just be discarded.

I'll give it a try.

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/27/2015 11:53 AM, Markus Armbruster wrote:
>
>>> Oh, and that means our generator has a collision bug that none of my
>>> added tests have exposed yet: you cannot have a base class and
>>> simultaneously add a member named 'base':
>>>
>>> { 'struct': 'Base', 'data': { 'i': 'int' } }
>>> { 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }
>>>
>>> because the generated C code is trying to use the name 'base' for its
>>> own purposes.
>> 
>> *sigh*
>> 
>>>                I guess that means more pre-req patches to the series to
>>> expose the bug, and either tighten the parser to reject things for now
>>> (easiest) or update the generator to not collide (harder, and fine for a
>>> later series).
>> 
>> Yes.
>> 
>> Life would be easier if the original authors had adopted sane naming
>> conventions from the start.
>
> Like reserving ourselves a namespace based on single _ for internal use.
>  We practically already have that - all user names either start with a
> letter or double underscore, so we could use single (and triple)
> underscore for internally-generated purposes, freeing up 'base',
> '*Kind', '*_MAX', and other namespace abuses back to the user.  Well, we
> may also need to reserve mid-name double-underscore (that is, the user
> can only supply double underscore at the beginning, but not middle, of
> an identifier).  Ah well, food for thought for later patches.

Another concern: we should take care not to generate reserved
identifiers.

* Potential issue with your proposal: identifiers that begin with an
  underscore and either an uppercase letter or another underscore are
  always reserved for any use.

* Existing issue: downstream extensions carry a __RFQDN_ prefix in the
  schema, which map to reserved C identifiers.

  Example: qapi-schema-test.json type '__org.qemu_x-Enum' generates

    typedef enum __org_qemu_x_Enum {
        ORG_QEMU_X_ENUM___ORG_QEMU_X_VALUE = 0,
        ORG_QEMU_X_ENUM_MAX = 1,
    } __org_qemu_x_Enum;

    extern const char *const __org_qemu_x_Enum_lookup[];

>>> Okay, I see where you are using .flat from the initial parse.  I still
>>> think it is a bit odd that you are defining '.flat' for each 'variant'
>>> within 'variants', even though, for a given 'variants', all members will
>>> have the same setting of '.flat'.  That makes me wonder if '.flat'
>>> should belong instead to the top-level 'variants' struct rather than to
>>> each 'variant' member.
>> 
>> Two reasons for putting flat where it is:
>> 
>> * The philosophical one: from the generator's point of view, there's no
>>   fundamental reason why all variants need to be flat or none.  The
>>   generator really doesn't care.
>
> And we may decide to exploit that down the road to allow some sort of
> qapi syntax for explicitly designating a union branch as flat or boxed,
> rather than the current approach of the type of union determining the
> status of all branch members.

I can't see a need now, but if one arises, we could do it.

>> * The pragmatic one (a.k.a. the real one): there are places where I use
>>   v.flat, but don't have the variants owning v handy.
>> 
>>> But again I wonder what would happen if you had instead normalized the
>>> input of simple unions into always having an implicit struct (with
>>> single member 'data'), so that by the time you get here, you only have
>>> to deal with a single representation of unions instead of having to
>>> still emit different things for flat vs. simple (since on the wire, we
>>> already proved simple is shorthand that can be duplicated by a flat union).
>> 
>> I hope we can get there!  But at this stage of the conversion, I want to
>> minimize output change, and .flat makes preserving all its warts much
>> easier.
>
> Agreed.  By the end of the series, I was convinced that the use of
> .flat, at least in this series, makes sense.

Good :)

>>>> +    disc_key = variants.tag_member.name
>>>> +    if not variants.tag_name:
>>>> +        # we pointlessly use a different key for simple unions
>>>
>>> We could fix that (as a separate patch); wonder how much C code it would
>>> affect.  A lot of these things that we can alter in generated code are
>>> certainly easier to see now that we have a clean generator :)
>> 
>> Yup, the warts stand out now.
>
> And I've already demonstrated what sort of cleanups can be done to
> attack some of the warts:
> https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html

I only have time for a quick glance now.  It looks lovely!

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Fixes flat unions to visit the base's base members (the previous
>> commit merely added them to the struct).  Same test case.
>> 
>> Patch's effect on visit_type_UserDefFlatUnion():
>> 
>>      static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp)
>>      {
>>          Error *err = NULL;
>> 
>>     +    visit_type_int(m, &(*obj)->integer, "integer", &err);
>>     +    if (err) {
>>     +        goto out;
>>     +    }
>>          visit_type_str(m, &(*obj)->string, "string", &err);
>>          if (err) {
>>              goto out;
>> 
>
>> +def gen_visit_union(name, base, variants):
>> +    ret = ''
>>  
>>      if base:
>> -        assert discriminator
>> -        base_fields = find_struct(base)['data'].copy()
>> -        del base_fields[discriminator]
>> -        ret += generate_visit_struct_fields(name, base_fields)
>> +        members = [m for m in base.members if m != variants.tag_member]
>> +        ret += generate_visit_struct_fields(name, members)
>
> Ouch. This hurts.  If the same class is used as both the base class of a
> flat union, and the base class of an ordinary struct, then the struct
> tries to visit the base class, but no longer parses the field that the
> union was using as its discriminator.
>
> We don't have any code that demonstrates this, but probably should.  I
> ran into it while working up my POC of what it would take to unbox
> inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)

Is this broken in master, or do my patches break it?

Got a reproducer?

[...]

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> is_c_ptr() looks whether the end of the C text for the type looks like
>> a pointer.  Works, but is fragile.
>> 
>> We now have a better tool: use QAPISchemaType method c_null().  The
>> initializers for non-pointers become prettier: 0, false or the
>> enumeration constant with the value 0 instead of {0}.
>> 
>> One place looks suspicious: we initialize pointers, but not
>> non-pointers.  Either the initialization is superfluous and should be
>> deleted, or the non-pointers need it as well, or something subtle is
>> going on and needs a comment.  Since I lack the time to figure it out
>> now, mark it FIXME.
>
> Commenting on just this part for now (out of time to review this patch
> proper for today):
>
>> @@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
>>                  header=hdr)
>>  
>>      if ret_type:
>> -        if is_c_ptr(ret_type.c_type()):
>> +        # FIXME fishy: only pointers are initialized
>> +        if ret_type.c_null() == 'NULL':
>>              retval = "    %s retval = NULL;" % ret_type.c_type()
>>          else:
>>              retval = "    %s retval;" % ret_type.c_type()
>
> Here's an example function impacted by this:
>
> static void qmp_marshal_input_guest_file_open(QDict *args, QObject
> **ret, Error
> **errp)
> {
>     Error *local_err = NULL;
>     int64_t retval;
>     QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
>     QapiDeallocVisitor *md;
>     Visitor *v;
>     char *path = NULL;
>     bool has_mode = false;
>     char *mode = NULL;
> ...
>     retval = qmp_guest_file_open(path, has_mode, mode, &local_err);
>     if (local_err) {
>         goto out;
>     }
>
>     qmp_marshal_output_guest_file_open(retval, ret, &local_err);
>
> But compare that to any other function that returns a pointer, and
> you'll see the same pattern (the only use of retval is in the final call
> to qmp_marshal_output..., right after assigning retval); that is,

Correct.  Inspection of qapi-commands.py shows retval is only ever read
in the call of qmp_marshal_output_FOO().

> initializing to NULL is dead code, and you could get away with just
> always declaring it instead of worrying about c_null() in this code.
>
> Or maybe we have a leak - if the 'if (local_err)' can ever trigger even
> when a function returned non-NULL, then our out: label is missing a
> free(retval), but only when retval is a pointer.  Or maybe we change the
> code to assert that retval is NULL if local_err is set after calling the
> user's function, to prove we don't have a leak.

Let me rephrase to make sure I understand.

Ignore the (not rets) case, because retval doesn't exist then.

qmp_marshal_output_FOO() visits retval twice.  First, with a QMP output
visitor to do the actual marshalling.  Second, with a QAPI dealloc
visitor to destroy it.

If we execute the assignment to retval, we must go on to call
qmp_marshal_output_FOO(), or else we have a leak.

If we can reach qmp_marshal_output_FOO() without executing the
assignment, we must initialize retval.  If we can't, any initialization
is unused.

gen_call() generates code of the form

        retval = qmp_FOO(... args ..., &local_err);
        if (local_err) {
            goto out;
        }

        qmp_marshal_output_FOO(retval, ret, &local_err);

Its caller then generates

    out:
        error_propagate(errp, local_err);

and so forth.

Observe:

1. The assignment dominates the only use.  Therefore, the initialization
   is unused.  Let's drop it in a separate cleanup patch.

2. We can leak retval only when qmp_FOO() returns non-null and local_err
   is non-null.  This must not happen, because:

   a. local_err must be null before the call, and

   b. the call must not return non-null when it sets local_err.

   We could right after out: assert(!local_err || !retval).  Not sure
   it's worthwhile.

TL;DR: I concur with your analysis.

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> is_c_ptr() looks whether the end of the C text for the type looks like
>> a pointer.  Works, but is fragile.
>> 
>> We now have a better tool: use QAPISchemaType method c_null().  The
>> initializers for non-pointers become prettier: 0, false or the
>> enumeration constant with the value 0 instead of {0}.
>> 
>> One place looks suspicious: we initialize pointers, but not
>> non-pointers.  Either the initialization is superfluous and should be
>> deleted, or the non-pointers need it as well, or something subtle is
>> going on and needs a comment.  Since I lack the time to figure it out
>> now, mark it FIXME.
>
> I commented on that in another mail - either we have a leak on failure
> (and need the initializer only for pointers), or we should add an assert
> (and don't need the initializer at all), depending on what semantics we
> want to enforce on all handler functions that set their error parameter.
>  Probably worth cleaning that up in a separate pre-req patch.

A function call should either fail completely or succeed completely.

On success, it must not set an error (d'oh!).

On failure, it must set an error (d'oh again!), and should not return
stuff for the caller to clean up.

The rules on setting errors together with the rule that a &local_err
argument must contain null before the call ensures that afterwards
local_err is set exactly when the call failed.

The rule on returning stuff lets us keep the error path simple: no need
for cleaning up anything returned by the function.  This is pervasive in
the code.  We generally don't bother to assert() the function actually
obeys the rule.

Perhaps I should fix up my pending patch "error: Revamp interface
documentation"[*] to cover the "should not return stuff for the caller
to clean up" part.

Apply to handler functions returning objects in need of clean up:

        retval = qmp_FOO(... args ..., &local_err);
        if (local_err) {
            goto out;
        }

        qmp_marshal_output_FOO(retval, ret, &local_err);

On success, qmp_FOO() must not touch local_err.

On failure, qmp_FOO() must set local_err, and return null.

Bypassing the cleanup qmp_marshal_output_FOO() is therfore perfectly
fine.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py | 19 +++++++------------
>>  scripts/qapi.py          |  3 ---
>>  2 files changed, 7 insertions(+), 15 deletions(-)
>> 
>
> Nice reduction in size.  If the marshal code FIXME disappears due to a
> separate cleanup, then the rest of this commit is good to go:
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!


[*] https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg04756.html

Re: [Qemu-devel] [PATCH RFC v2 30/47] qapi: De-duplicate enum code generation

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Duplicated in commit 21cd70d.  Yes, we can't import qapi-types, but
>> that's no excuse.  Move the helpers from qapi-types.py to qapi.py, and
>> replace the duplicates in qapi-event.py.
>> 
>> The generated event enumeration type gains a _MAX member,
>
> Not quite; the _MAX member was already there, but done by hand [1]
> instead of by common code.

You're right; I was confused.

>> and its
>> lookup table becomes const-correct (see commit 2e4450f).
>
> Maybe what you meant instead of gaining _MAX was that it also gains C99
> initializers:
>
> -const char *QAPIEvent_lookup[] = {
> -    "ACPI_DEVICE_OST",
> -    "BALLOON_CHANGE",
> ...
> -    "WATCHDOG",
> -    NULL,
> +const char *const QAPIEvent_lookup[] = {
> +    [QAPI_EVENT_ACPI_DEVICE_OST] = "ACPI_DEVICE_OST",
> +    [QAPI_EVENT_BALLOON_CHANGE] = "BALLOON_CHANGE",

That's not what I had in mind when I wrote the sentence about _MAX. but
mentioning the change in the commit message won't hurt.  Note to self:
comes from commit 912ae9c.

> Overall, a good change.
>
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-event.py | 67
>> +++------------------------------------------------
>>  scripts/qapi-types.py | 55 ------------------------------------------
>>  scripts/qapi.py       | 55 ++++++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 59 insertions(+), 118 deletions(-)
>
> Since only the commit message needs tweaking,
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

[...]

Re: [Qemu-devel] [PATCH RFC v2 32/47] qapi-event: Convert to QAPISchemaVisitor, fixing data with base

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Fixes events whose data is struct with base to include the struct's
>> base members.  Test case is qapi-schema-test.json's event
>> __org.qemu_x-command:
>> 
>>     { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
>> 
>>     { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
>>       'data': { '__org.qemu_x-member2': 'str' } }
>> 
>>     { 'struct': '__org.qemu_x-Base',
>>       'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }
>
> No change to generated code in qemu proper, so this is a corner case we
> are not yet exploiting. But good to have it fixed :)
>
>> 
>> Patch's effect on generated qapi_event_send___org_qemu_x_event():
>> 
>>     -void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2,
>>     +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
>>     +                                        const char *__org_qemu_x_member2,
>>                                              Error **errp)
>
> Ouch - I think we have a bug in qapi.py:check_event().  There, we call
> check_type(... allow_metas=['union', 'struct']) - but it looks like the
> generated signature requires that we have no variants, which means we
> cannot have:
>
> { 'union': 'Un', 'data': ... }
> { 'event': 'EV', 'data': 'Un' }
>
> because it would fail C generation. Sounds like you should add a check
> to that in 14/47, and a fix for it in 15/47.

Will do.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-event.py | 87 ++++++++++++++++-----------------
>>  tests/qapi-schema/qapi-schema-test.json |  3 --
>>  2 files changed, 43 insertions(+), 47 deletions(-)
>> 
>> diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
>> index 537da17..456e590 100644
>> --- a/scripts/qapi-event.py
>> +++ b/scripts/qapi-event.py
>> @@ -2,28 +2,29 @@
>>  # QAPI event generator
>>  #
>>  # Copyright (c) 2014 Wenchao Xia
>> +# Copyright (c) 2013-2015 Red Hat Inc.
>
> Umm - the file didn't exist until 2014; and to my knowledge, you aren't
> adding anything to it that was copied from some other file dating back
> to 2013.  Using the range 2014-2015 might be better.

Pasto, will fix.

> But that's minor.  And assuming that you reject union types for events
> in an earlier patch, the rest of this is fine:
> Reviewed-by: Eric Blake <eblake@redhat.com>
>
>> @@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params):
>>  """,
>>                  event_name = event_name)
>>  
>> -        for argname, argentry, optional in parse_args(params):
>> -            if optional:
>> +        for memb in params.members:
>> +            if memb.optional:
>>                  ret += mcgen("""
>>      if (has_%(var)s) {
>>  """,
>> -                             var = c_name(argname))
>> +                             var=c_name(memb.name))
>>                  push_indent()
>>  
>> -            if argentry == "str":
>> +            if memb.type.name == "str":
>>                  var_type = "(char **)"
>
> Our visitors require us to cast away const.  Is that something we should
> consider reworking, so that we don't need to do that?  But it's a
> question for another day.

Yes, it's annoying, and yes, we need to leave it for later.

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 33/47] qapi: Clean up after recent conversions to QAPISchemaVisitor

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Drop helper functions that are now unused.
>> 
>> Make pylint reasonably happy.
>> 
>> Rename generate_FOO() functions to gen_FOO() for consistency.
>> 
>> Use more consistent and sensible variable names.
>> 
>> Consistently use c_ for mapping keys when their value is a C
>> identifier or type.
>> 
>> Simplify gen_enum() and gen_visit_union()
>> 
>> Consistently use single quotes for C text string literals.
>
> Not sure if it is worth splitting this into pieces.  Fortunately, there
> are no changes to generated output.

I'm afraid splitting would increase churn without much gain.  If you
want me to split, I can try.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py | 109 +++++++++++++++++++------------------
>>  scripts/qapi-event.py    | 117 +++++++++++++++++++---------------------
>>  scripts/qapi-types.py    |  68 ++++++++++++-----------
>>  scripts/qapi-visit.py    | 121 ++++++++++++++++++++---------------------
>>  scripts/qapi.py | 138
>> +++++++++--------------------------------------
>>  5 files changed, 229 insertions(+), 324 deletions(-)
>> 
>> diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
>> index d3bddb6..5d11032 100644
>> --- a/scripts/qapi-commands.py
>> +++ b/scripts/qapi-commands.py
>> @@ -15,20 +15,20 @@
>>  from qapi import *
>>  import re
>>  
>> -def generate_command_decl(name, args, ret_type):
>> -    arglist=""
>> +def gen_command_decl(name, args, rets):
>
> I can see how 'args' is plural (even if it is a single string for the
> name of a type containing the args), but should it be 'ret' instead of
> 'rets'?

I now realize readers may find this odd.

The QMP specification talks about command arguments and command
responses.

The QMP wire format uses 'arguments' and 'return'.

The schema language uses 'data' and 'returns'.  Near-perfect score on
terminology inconsistency, as usual.  Anyway, 'data' is plural (and a
rather unhelpful choice of syntax).  'returns' could either be the
plural of the noun "return", or the third person singular of the verb
"to return".

Permit me a philosophical digression.  The common way to do functions in
programming is to have multiple arguments and a single return value.  I
believe this is mostly common machines' calling conventions leaking into
languages.  From a more abstract point of view, there's no structural
difference between function arguments and values: both are simply an
element of an arbitrary set (domain and codomain, respectively).  In
particular, both can be tuples.

It's perfectly sane to have functions take exactly one argument and
yield exactly one value.  Some functional languages work that way.

But when both argument and value are generally tuples anyway, as they
are in QAPI/QMP, it's more natural to talk about arguments and return
values.  I abbreviated to args and rets.  There's method to my madness
;)

I'm open to better ideas on terminology.

>> @@ -40,34 +40,37 @@ if (%(err)s) {
>>  ''',
>>                   err=err)
>>  
>> -def gen_sync_call(name, args, ret_type):
>> -    ret = ""
>> -    arglist=""
>> -    retval=""
>> -    if ret_type:
>> -        retval = "retval = "
>> +def gen_call(name, args, rets):
>
> At least you're consistent on naming it 'rets',
>
>> +    ret = ''
>
> and the naming lets you distinguish between the parameter (the type
> describing returned fields) and the local string (the generated C code
> holding the return information).
>
>> @@ -82,45 +76,46 @@ def generate_event_implement(api_name, event_name, params):
>
>>  
>> +            # Ugly: need to cast away the const
>>              if memb.type.name == "str":
>> -                var_type = "(char **)"
>> +                cast = "(char **)"
>
> And to think I called it out in a previous patch. So you noticed it too :)
>
> Don't you want to use '(char **)' here, since it is a literal string
> destined for generated C?

Yes.

>>              else:
>> -                var_type = ""
>> +                cast = ""
>
> and '' here?

Yes.

>> +++ b/scripts/qapi-types.py
>> @@ -16,22 +16,22 @@ from qapi import *
>>  def gen_fwd_object_or_array(name):
>>      return mcgen('''
>>  
>> -typedef struct %(name)s %(name)s;
>> +typedef struct %(c_name)s %(c_name)s;
>>  ''',
>> -                 name=c_name(name))
>> +                 c_name=c_name(name))
>>  
>>  def gen_array(name, element_type):
>>      return mcgen('''
>>  
>> -struct %(name)s {
>> +struct %(c_name)s {
>>      union {
>>          %(c_type)s value;
>>          uint64_t padding;
>>      };
>> -    struct %(name)s *next;
>> +    struct %(c_name)s *next;
>
> May be some churn here if you like my comment earlier in the series that
> this 'struct' is redundant.

Can drop it in this patch.

I'm not into typedef'ing struct/union away unless the type is opaque,
but it's the QEMU style, so let's stick to it.

>> +++ b/scripts/qapi-visit.py
>
>> -def generate_visit_struct_fields(name, members, base = None):
>> +def gen_visit_struct_fields(name, base, members):
>>      struct_fields_seen.add(name)
>
>> -                     type=base.c_name(), c_name=c_name('base'))
>> +                     c_type=base.c_name(), c_name=c_name('base'))
>
> Possible churn here based on my earlier comments about c_name(constant)
> being constant.

I'm leaning towards leaving it as is just to keep the code similar to
other places generating visit_type_FOO() calls.

> Fairly big, but aside from some minor '' quoting issues, I didn't see
> anything wrong.
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 35/47] qapi-commands: Rearrange code

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Rename gen_marshal_input() to gen_marshal(), because the generated
>> function marshals both arguments and results.
>> 
>> Rename gen_visitor_input_containers_decl() to gen_marshal_vars(), and
>> move the other variable declarations there, too.
>> 
>> Rename gen_visitor_input_block()() to gen_marshal_input_visit(), and
>
> Double ().

Will fix.

>> rearrange its code slightly.
>> 
>> Rename gen_marshal_input_decl() to gen_marshal_proto(), because the
>> result isn't a full declaration, unlike gen_command_decl()'s.
>> 
>> New gen_marshal_decl() actually returns a full declaration.
>
> No change to generated code; good.
>
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py | 95 ++++++++++++++++++++++--------------------------
>>  1 file changed, 43 insertions(+), 52 deletions(-)
>> 
>
>>      push_indent()
>> +
>> +    if rets:
>> +        # FIXME fishy: only pointers are initialized
>> +        if rets.c_null() == 'NULL':
>> +            retval = '%s retval = NULL;' % rets.c_type()
>> +        else:
>> +            retval = '%s retval;' % rets.c_type()
>
> May still need tweaking based on my earlier reviews.

Yes, this part should go away.

> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 36/47] qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> These functions marshal both input and output.
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  docs/qapi-code-gen.txt        |   4 +-
>>  docs/writing-qmp-commands.txt |   8 +-
>>  monitor.c                     |   2 +-
>>  qmp-commands.hx | 240 +++++++++++++++++++++---------------------
>>  scripts/qapi-commands.py      |   4 +-
>>  5 files changed, 129 insertions(+), 129 deletions(-)
>
> Diffstat here says it is a straight rename; likewise the diff on the
> generated code:
>
>  qga/qapi-generated/qga-qmp-marshal.c |  116 ++++++++--------
>  qmp-commands.h                       |  240
> +++++++++++++++++------------------
>  qmp-marshal.c                        |  240
> +++++++++++++++++------------------
>  3 files changed, 298 insertions(+), 298 deletions(-)
>
> Missing: you overlooked a comment in qmp.c line 154:
>
> #ifndef CONFIG_SPICE
> /*
>  * qmp-commands.hx ensures that QMP command query-spice exists only
>  * #ifdef CONFIG_SPICE.  Necessary for an accurate query-commands
>  * result.  However, the QAPI schema is blissfully unaware of that,
>  * and the QAPI code generator happily generates a dead
>  * qmp_marshal_input_query_spice() that calls qmp_query_spice().
>  * Provide it one, or else linking fails.
>  * FIXME Educate the QAPI schema on CONFIG_SPICE.
>  */

Will fix.

> With that fixed,
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 37/47] qapi: De-duplicate parameter list generation

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Generated qapi-event.[ch] lose line breaks.  No change otherwise.
>
> For example,
>
> -void qapi_event_send_block_image_corrupted(const char *device,
> -                                           bool has_node_name,
> -                                           const char *node_name,
> -                                           const char *msg,
> -                                           bool has_offset,
> -                                           int64_t offset,
> -                                           bool has_size,
> -                                           int64_t size,
> -                                           bool fatal,
> -                                           Error **errp)
> +void qapi_event_send_block_image_corrupted(const char *device, bool
> has_node_name, const char *node_name, const char *msg, bool has_offset,
> int64_t offset, bool has_size, int64_t size, bool fatal, Error **errp)
>
> You know, I'd find it a bit more appealing if you had merged the
> duplicate code in the _other_ direction. That is, qapi-event's wrapped
> lines (usually) fit in 80 columns, and it would be nice if qapi-visit's
> did the same.
>
> Yeah, avoiding line wraps consumes fewer source bytes (fewer runs of
> spaces), but the space isn't being wasted by storing generated files in
> git, nor does the C compiler care which layout we use.  And honestly,
> it's easier to spot changes in a vertical list than it is on a long
> horizontal line, if a parameter gets added (or removed, although adding
> is the more likely action with qapi).

Number of source bytes is not an issue.

The generators make no effort to wrap source lines, except in the
qapi_event_send_FOO()'s parameter lists.

We could preserve that one-off.  We could extend it to more places that
can generate long lines, saddling the generation code with indentation
concerns.  I don't want to write such code, and I don't want to maintain
it.

Instead, why not keep the generators straightforward, and feed their
result to indent when "pretty" is wanted?  Requires an indent profile
matching QEMU style.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py | 11 ++---------
>>  scripts/qapi-event.py    | 18 +++---------------
>>  scripts/qapi.py          | 16 ++++++++++++++++
>>  3 files changed, 21 insertions(+), 24 deletions(-)
>
> I'm a fan of de-duplication, so I'll review this on its merits; but I'm
> omitting R-b on this round in hopes that you buy my argument to merge in
> the other direction (make qapi-event's implementation the common one).
>
>> 
>> diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
>> index d57f8d4..2dae425 100644
>> --- a/scripts/qapi-commands.py
>> +++ b/scripts/qapi-commands.py
>
>> -                 args=argstr)
>> +                 params=gen_params(args, 'Error **errp'))
>
> Caller 1.
>
>> +++ b/scripts/qapi-event.py
>
>> +    return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
>> +        'c_name': c_name(name.lower()),
>> +        'param': gen_params(data, 'Error **errp')}
>
> Caller 2.
>
>>  
>>  def gen_event_send_decl(name, data):
>>      return mcgen('''
>> diff --git a/scripts/qapi.py b/scripts/qapi.py
>> index 4d47214..c6a5ddc 100644
>> --- a/scripts/qapi.py
>> +++ b/scripts/qapi.py
>> @@ -1384,6 +1384,22 @@ extern const char *const %(c_name)s_lookup[];
>>                   c_name=c_name(name))
>>      return ret
>>  
>> +def gen_params(args, extra):
>> +    if not args:
>> +        return extra
>
> Both callers pass the same 'extra' - do you need it to be parameterized,
> or can it just be generated as a constant here?  (I guess it depends on
> what happens with the later introspection patch, which may become caller 3).

The series doesn't add callers later on.

I made it a parameter simply because I feel gen_params() shouldn't need
to know what extra parameters its caller may need.  Even when all
callers need the same.

>> +    assert not args.variants
>
> This assert will trip if you don't fix events to reject 'data':'Union' :)

Looks like it :)

>> +    ret = ""
>> +    sep = ""
>> +    for memb in args.members:
>> +        ret += sep
>> +        sep = ", "
>> +        if memb.optional:
>> +            ret += "bool has_%s, " % c_name(memb.name)
>
> Didn't you just provide a patch that used '' rather than "" for all
> generated C constructs?  This violates that paradigm.

Will fix.

>> +        ret += "%s %s" % (memb.type.c_type(is_param=True), c_name(memb.name))
>> +    if extra:
>> +        ret += sep + extra
>> +    return ret
>> +
>
> To produce line breaks, you could have to add a parameter so that
> callers can pass in the starting column for each wrapped argument, and
> then you'd have sep = ',\n' + ''.ljust(len).  Or even have the caller
> choose its own separator (", " vs. ",\n    "), if you don't want to have
> a diff in the generated output (but I think consistent generated output
> is nicer).

Re: [Qemu-devel] [PATCH RFC v2 38/47] qapi-commands: De-duplicate output marshaling functions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> gen_marshal_output() uses its parameter name only for name of the
>> generated function.  Name it after the type being marshaled instead of
>> its caller, and drop duplicates.
>> 
>> Saves 7 copies of qmp_marshal_output_int() in qemu-ga, and one copy of
>> qmp_marshal_output_str() in qemu-system-*.
>
>  qga/qapi-generated/qga-qmp-marshal.c |  227
> +++++------------------------------
>  qmp-marshal.c                        |  225
> +++++++++++++++-------------------
>  2 files changed, 134 insertions(+), 318 deletions(-)
>
> Most changes look like:
>
> -static void qmp_marshal_output_add_fd(AddfdInfo *ret_in, QObject
> **ret_out, Error **errp)
> +static void qmp_marshal_output_AddfdInfo(AddfdInfo *ret_in, QObject
> **ret_out, Error **errp)
>  {
>      Error *local_err = NULL;
>      QmpOutputVisitor *mo = qmp_output_visitor_new();
> @@ -88,7 +88,7 @@ void qmp_marshal_add_fd(QDict *args, QOb
>          goto out;
>      }
>
> -    qmp_marshal_output_add_fd(retval, ret, &local_err);
> +    qmp_marshal_output_AddfdInfo(retval, ret, &local_err);
>
> coupled with wholesale deletions of functions that previously had
> identical bodies.  Nice.

Exactly.

> [I suspect there might be ways to trim a LOT more code size, by
> rewriting a generic marshaller helper that is passed a varargs or
> array-of-struct list of operations to perform in order to visit an
> arbitrary object, then having each command's marshaller generated with
> the appropriate list of arguments for the generic helpers rather than
> the current approach of calling out to one marshaller helper per type -
> but exploring ideas like that is work for another series]

Oh, but generating thousands of line of repetitive code is so much fun!

Seriously: yes, that's a far better way to do it.  Marshalling doesn't
have to be as fast as it could possibly be, space cost be damned.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-commands.py | 16 +++++++++-------
>>  1 file changed, 9 insertions(+), 7 deletions(-)
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 40/47] qapi: Introduce a first class 'any' type

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> It's first class, because unlike '**', it actually works, i.e. doesn't
>> require 'gen': false.
>
> Generated code grows in order to accommodate visiting the new anyList type:
>
>  qapi-types.c                        |   15 +++++++++++++++
>  qapi-types.h                        |   13 +++++++++++++
>  qapi-visit.c                        |   24 ++++++++++++++++++++++++
>  qapi-visit.h                        |    1 +
>  qga/qapi-generated/qga-qapi-types.h |   13 +++++++++++++
>  qga/qapi-generated/qga-qapi-visit.h |    1 +
>  6 files changed, 67 insertions(+)
>
> But do we really need anyList as a representation of qapi ['any'] ?  Or
> will a later pass that minimizes array creation only to places where it
> is needed help?

Like most generated list types, anyList isn't used.  I didn't feel like
making it a special case.  If the unused list types bother us, we can
generate only the ones actually mentioned in the schema.  May require
mentioning a few we use internally just to have them generated.

>> '**' will go away next.
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  docs/qapi-code-gen.txt            |  1 +
>>  include/qapi/visitor-impl.h       |  2 ++
>>  include/qapi/visitor.h            |  1 +
>>  qapi/qapi-dealloc-visitor.c       |  9 +++++++
>>  qapi/qapi-visit-core.c            |  6 +++++
>>  qapi/qmp-input-visitor.c          | 11 ++++++++
>>  qapi/qmp-output-visitor.c         |  9 +++++++
>>  scripts/qapi-types.py             |  1 +
>>  scripts/qapi.py                   |  9 ++++---
>>  tests/qapi-schema/type-bypass.out |  4 +--
>>  tests/test-qmp-input-visitor.c    | 45 +++++++++++++++++++++++++++++++++
>>  tests/test-qmp-output-visitor.c | 53
>> +++++++++++++++++++++++++++++++++++++++
>>  12 files changed, 146 insertions(+), 5 deletions(-)
>
> Touches a bit more this time, but that's okay.
>
>> 
>> diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
>> index cb0fe75..bf9e854 100644
>> --- a/docs/qapi-code-gen.txt
>> +++ b/docs/qapi-code-gen.txt
>> @@ -158,6 +158,7 @@ The following types are predefined, and map to C as follows:
>>    size      uint64_t   like uint64_t, except StringInputVisitor
>>                         accepts size suffixes
>>    bool      bool       JSON true or false
>> +  any       QObject *  any JSON value
>
> And with this, an 'alternate' type is just a special subset of 'any'
> that limits the set of valid JSON values according to the qapi description.
>
>> +++ b/qapi/qapi-visit-core.c
>> @@ -260,6 +260,12 @@ void visit_type_number(Visitor *v, double *obj, const char *name, Error **errp)
>>      v->type_number(v, obj, name, errp);
>>  }
>>  
>> +void visit_type_any(Visitor *v, QObject **obj, const char *name,
>> +                         Error **errp)
>
> Indentation looks off.

Will fix.

>> @@ -1048,8 +1048,9 @@ class QAPISchema(object):
>>                    ('uint64', 'int',     'uint64_t', '0'),
>>                    ('size',   'int',     'uint64_t', '0'),
>>                    ('bool',   'boolean', 'bool',     'false'),
>> -                  ('**',     'value',   None,       None)]:
>> +                  ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
>>              self._def_builtin_type(*t)
>> +        self.entity_dict['**'] = self.lookup_type('any') # TODO drop this alias
>>  
>>      def _make_implicit_enum_type(self, name, values):
>>          name = name + 'Kind'
>> @@ -1190,6 +1191,8 @@ class QAPISchema(object):
>>      def visit(self, visitor):
>>          visitor.visit_begin()
>>          for name in sorted(self.entity_dict.keys()):
>> +            if self.entity_dict[name].name != name:
>> +                continue        # ignore alias TODO drop alias and remove
>
> Nice back-compat hacks while you transition into using it.
>
>> +++ b/tests/test-qmp-input-visitor.c
>> @@ -298,6 +298,49 @@ static void test_visitor_in_list(TestInputVisitorData *data,
>>      qapi_free_UserDefOneList(head);
>>  }
>>  
>> +static void test_visitor_in_any(TestInputVisitorData *data,
>> +                                const void *unused)
>> +{
>
>> +
>> +    v = visitor_input_test_init(data, "-42");
>
> So we prove it accepts ints,...
>
>> + v = visitor_input_test_init(data, "{ 'integer': -42, 'boolean':
>> true, 'string': 'foo' }");
>
> objects,...
>
>> +    visit_type_any(v, &res, NULL, &err);
>> +    g_assert(!err);
>> +    qdict = qobject_to_qdict(res);
>> +    g_assert(qdict);
>
> worth asserting the dictionary has 3 keys?

Can't hurt.

>> +}
>
> ...but no test of string, boolean, or array?

I cut corners to get the thing out: one scalar and one non-scalar type.

We can always add more.

>> +++ b/tests/test-qmp-output-visitor.c
>> @@ -428,6 +428,57 @@ static void test_visitor_out_list_qapi_free(TestOutputVisitorData *data,
>>      qapi_free_UserDefTwoList(head);
>>  }
>>  
>> +static void test_visitor_out_any(TestOutputVisitorData *data,
>> +                                 const void *unused)
>> +{
>
>> +}
>
> Same tests of 'int' and 'object' but not of other JSON types.
>
> Incomplete tests are still better than no tests, so what you have is a
> good start.
>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 41/47] qom: Don't use 'gen': false for qom-get, qom-set, object-add

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> With the previous commit, the generated marshalers just work, and save
>> us a bit of handwritten code.
>> 
>
> Generated code grows, because you are now generating the hook :)
>
>  qmp-commands.h |    6 ++
>  qmp-marshal.c  |  144 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 150 insertions(+)
>
>
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  include/monitor/monitor.h |  3 ---
>>  qapi-schema.json          |  9 +++------
>>  qmp-commands.hx           |  6 +++---
>>  qmp.c                     | 20 +++++++-------------
>>  scripts/qapi.py           |  1 +
>>  5 files changed, 14 insertions(+), 25 deletions(-)
>> 
>
>> +++ b/qmp.c
>> @@ -229,11 +229,9 @@ ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
>>  }
>>  
>>  /* FIXME: teach qapi about how to pass through Visitors */
>> -void qmp_qom_set(QDict *qdict, QObject **ret, Error **errp)
>> +void qmp_qom_set(const char *path, const char *property, QObject *value,
>> +                 Error **errp)
>
> The FIXME seems stale now.

I'm not 100% sure I understand the intent of the FIXME, but my best
guess is "do what this patch does".  I'll drop it.

>> -void qmp_object_add(QDict *qdict, QObject **ret, Error **errp)
>> +void qmp_object_add(const char *type, const char *id,
>> +                    bool has_props, QObject *props, Error **errp)
>>  {
>> -    const char *type = qdict_get_str(qdict, "qom-type");
>> -    const char *id = qdict_get_str(qdict, "id");
>> -    QObject *props = qdict_get(qdict, "props");
>>      const QDict *pdict = NULL;
>>      QmpInputVisitor *qiv;
>>  
>
> Continuing:
>
>     if (props) {
>         pdict = qobject_to_qdict(props);
>         if (!pdict) {
>             error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
>             return;
>         }
>     }
>
> I know that we guarantee that all pointers are initialized to NULL in
> our marshaler, so this is correct; but wouldn't it be more idiomatic to
> write 'if (has_props)' as the condition?

We do similar things elsewhere.  Moreover:

> (And it would make a nice followup project for someone to figure out how
> to get rid of have_FOO arguments for strings and objects where NULL is a
> nice witness; it is only integers and booleans that require them - but
> doing that will touch a lot of the tree, and is a series of its own)

Shouldn't be hard, and I really want it done.

Will change all the places that test has_PTR to just PTR.  More reason
to simply test PTR now.

> What I pointed out is minor, so you can fix it and add:
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 42/47] qapi-schema: Fix up misleading specification of netdev_add

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> It doesn't take a 'props' argument, let alone one in the format
>> "NAME=VALUE,..."
>> 
>> The bogus arguments specification doesn't matter due to 'gen': false.
>> Clean it up to be incomplete rather than wrong, and document the
>> incompleteness.
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  docs/qapi-code-gen.txt |  2 +-
>>  qapi-schema.json       | 13 +++++++------
>>  2 files changed, 8 insertions(+), 7 deletions(-)
>> 
>
>> +++ b/qapi-schema.json
>> @@ -2062,11 +2062,12 @@
>>  #
>>  # @id: the name of the new network backend
>>  #
>> -# @props: #optional a list of properties to be passed to the backend in
>> -#         the format 'name=value', like 'ifname=tap0,script=no'
>> +# Additional arguments depend on the type.
>
> In other words, this would be a perfect candidate to write as taking a
> flat union as the argument, where 'type' is the discriminator, and where
> the additional arguments are branches of the union.  Except that we
> don't yet support passing a union as the direct "arguments":{...} of a
> QMP call, so we still have some design work to do if we want to go
> there.  Later, of course.

Exactly.  I started hacking in that direction, but had to stop to get
this stuff out.

>>  #
>> -# Notes: The semantics of @props is not well defined.  Future
>> commands will be
>> -#        introduced that provide stronger typing for backend creation.
>> +# TODO This command effectively bypasses QAPI completely due to its
>> +# "additional arguments" business.  It shouldn't have been added to
>> +# the schema in this form.  It should be qapified properly, or
>> +# replaced by a properly qapified command.
>
> I like that you are getting rid of the sentence about strong typing
> (since flat unions ARE strongly typed) and instead couching it in terms
> of an incomplete qapi description.  I think we may even be able to come
> up with a way to express it without needing a new command, and without
> breaking back-compat, once we figure out how to pass unions as a command
> argument.
>
> In fact, let's suppose we do have a new union type, NetdevInfo, as well
> as a way to flag that a given command will be called by the marshaler
> using JUST a pointer to the struct, rather than as a list of parameters
> to members of the struct.
>
> { 'enum': 'NetdevTypes', [ ... ] }
> { 'struct': 'NetdevInfoBase', 'data':
>    { 'type': 'NetdevTypes', 'id': 'str' } }
> { 'union': 'NetdevInfo', 'base': 'NetdevInfoBase',
>   'discriminator': 'type', 'data': { ... } }
> { 'command': 'netdev_add', 'boxed': true,
>   'data': 'NetdevInfo' }
>
> resulting in the marshaller calling
>
> void qmp_netdev_add(NetdevInfo *data, Error **errp)
>
>
> Here, 'boxed' is an optional key to each 'command', defaulting to false;
> but when true, 'data' is passed in a box to the command called by the
> marshaler, instead of broken out into fields. 'boxed' must be true for
> unions, but may be true elsewhere.

Matches my plans.

Note existing type NetClientOptions, used internally.  I hope we can
avoid duplicating it and all its Netdev*Options types.

> And if we have that, then OTHER commands that have complained about
> super-long parameter lists when their struct is exploded into a
> parameter list could also use this mechanism, to write their handler in
> terms of a single pointer to the struct.

Yes.

> Enough rambling about stuff for the future. This patch is fine as-is;
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 44/47] qapi: Pseudo-type '**' is now unused, drop it

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> 'gen': false needs to stay for now, because netdev_add is still using
>> it.
>> 
>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  docs/qapi-code-gen.txt                               | 15 ++++-----------
>>  scripts/qapi.py                                      | 20 ++++----------------
>>  tests/Makefile                                       |  4 ++--
>>  tests/qapi-schema/args-returns-any.err               |  1 +
>>  ...type-bypass-no-gen.exit => args-returns-any.exit} |  0
>
>> 
>> diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
>> index 2367c66..ca578dd 100644
>> --- a/docs/qapi-code-gen.txt
>> +++ b/docs/qapi-code-gen.txt
>
>> @@ -457,13 +454,9 @@ which would validate this Client JSON Protocol transaction:
>>  
>>  In rare cases, QAPI cannot express a type-safe representation of a
>>  corresponding Client JSON Protocol command.  In these cases, if the
>> -command expression includes the key 'gen' with boolean value false,
>> -then the 'data' or 'returns' member that intends to bypass generated
>> -type-safety and do its own manual validation should use an inline
>> -dictionary definition, with a value of '**' rather than a valid type
>> -name for the keys that the generated code will not validate.  Please
>> -try to avoid adding new commands that rely on this, and instead use
>> -type-safe unions.  For an example of bypass usage:
>> +command expression includes the key 'gen' with boolean value false.
>
> Incomplete sentence.  "if the command expression includes... false"
> needs "then something...".

Oops.  I'll rephrase.

>> +Please try to avoid adding new commands that rely on this, and instead
>> +use type-safe unions.  For an example of bypass usage:
>
> And given my ideas, we may be able to make netdev_add typesafe and
> eliminate 'gen' altogether in the future :)

The harder problem is qapifying device_add without use of 'gen': false.
It's structurally just like netdev_add, "only" with has many, many more
variants (the device models), which are collected only at run time.

Changing things to define device model properties in the schema seems
unappealing.

We could try to collect them at build time.  Not trivial.

We could collect them at run time, but then the schema becomes dynamic.
Not trivial, either.

The easiest solution is probably something like Python's **kwds
parameter: collect any remaining arguments in a dictionary, and pass
that as argument kwds.  Sadly, QAPI introspection remains useless for
device models then.

>> +++ b/tests/Makefile
>> @@ -222,11 +222,11 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
>>  	bad-type-dict.json double-data.json unknown-expr-key.json \
>>  	redefined-type.json redefined-command.json redefined-builtin.json \
>>  	redefined-event.json command-int.json bad-data.json event-max.json \
>> -	type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \
>> +	type-bypass-bad-gen.json \
>
> If I understand correctly (git didn't quite guess right),
> type-bypass-no-gen.json disappears completely (we no longer have working
> '**'), and type-bypass.json...
>
>>  	data-array-empty.json data-array-unknown.json data-int.json \
>>  	data-unknown.json data-member-unknown.json data-member-array.json \
>>  	data-member-array-bad.json args-union.json args-alternate.json \
>> -	returns-array-bad.json returns-int.json \
>> +	args-returns-any.json returns-array-bad.json returns-int.json \
>
> ...gets renamed args-returns-any.json (because type bypass is now
> expressed via 'any'.

I'd view args-returns-any.json as new.  But viewing it as a replacement
for type-bypass.json is okay.

>> +# built-in type 'any' in arguments and returns
>> +# works except for 'data': 'any', because that has to be a struct
>> +# note: command name 'qom-get' chosen to avoid "cannot use built-in" error
>> +{ 'command': 'qom-get', 'data': { 'arg': 'any' }, 'returns': 'any' }
>> +{ 'command': 'foo', 'returns': { 'arg': 'any' } }
>> +{ 'command': 'bar', 'data': 'any' }
>
> This shows what the parser accepts and rejects, but doesn't prove it
> compiles to C; and the previous patch that touched
> test-qmp-input-visitor.c was testing that the low-level interactions
> worked but did not map it all the way back to qapi.  I wonder if you
> should also enhance qapi-schema-test to use 'any', since that tends to
> be the one test that most fully exercises interactions with generated
> code based on a valid qapi schema.

I guess I'll have to split args-returns-any.json: move the positive test
cases to qapi-schema-test.json, move the negative test case to its own
test.

> On the right track, but I want to make sure the botched documentation is
> correct before giving R-b.

Thanks!

Re: [Qemu-devel] [PATCH RFC v2 45/47] qapi: New QMP command query-schema for QMP schema introspection

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Caution, rough edges.
>
> No joke. It doesn't even compile without this fixup to a rebase snafu
> (see [0] below):

Uh, how did that happen?  I compiled it a million times... (except for
the last time, obviously).

> diff --git i/scripts/qapi-types.py w/scripts/qapi-types.py
> index 79e8d24..12f3767 100644
> --- i/scripts/qapi-types.py
> +++ w/scripts/qapi-types.py
> @@ -184,6 +184,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
>          self.fwdecl = None
>          self.fwdefn = None
>          self.btin = None
> +    def visit_begin(self, schema):
>          self.decl = ''
>          self.defn = ''
>          self.fwdecl = ''
>
> I already know you'll be editing this further, but here's some things to
> look for.
>
>> 
>> qapi/introspect.json defines the introspection schema.  It should do
>> for uses other than QMP.
>> FIXME it's almost entirely devoid of comments.
>
> It generates quite a bit of code to support itself :)
>
>  qapi-types.c |  383 ++++++++++++++++++++++++++++
>  qapi-types.h |  293 +++++++++++++++++++++
>  qapi-visit.c |  797 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  qapi-visit.h |   26 +
>  4 files changed, 1499 insertions(+)

Can't be helped.  We need the types to define query-schema properly.
And then the generators are eager to generate everything that could be
possibly useful for these types.

Due to the hacky way I implement qmp_query_schema(), most of it
shouldn't be needed in qemu proper.  But it *is* needed for the new
test-qmp-input-visitor test case.

>> The introspection schema does not reflect all the rules and
>> restrictions that apply to QAPI schemata.  A valid QAPI schema has an
>> introspection value conforming to the introspection schema, but the
>> converse is not true.
>> 
>> Introspection lowers away a number of schema details:
>> 
>> * The built-in types are declared with their JSON type.
>> 
>>   TODO Should we map all the integer types to just int?
>
> Just int for now seems fine.  It's easier to add refinement later when
> we have a proven need, than it is to be stuck advertising too much
> information now and being stuck with it.

Matches my current thinking.

>> * Implicit type definitions are made explicit, and given
>>   auto-generated names.  These names start with ':' so they don't
>>   clash with the user's names.
>> 
>>   Example: a simple union implicitly defines an enumeration type for
>>   its discriminator.
>> 
>> * All type references are by name.
>> 
>> * Base types are flattened.
>> 
>> * The top type (named 'any') can hold any value.
>> 
>> * The struct and union types are generalized into an object type.
>
> You've mentioned that idea on list quite some time ago, and it appears
> to be working well.
>
>> 
>> * Commands take a single argument and return a single result.
>> 
>>   Dictionary argument/result or list result is an implicit type
>>   definition.
>> 
>>   The empty object type is used when a command takes no arguments or
>>   produces no results.
>> 
>>   The argument is always of object type, but the introspection schema
>>   doesn't reflect that.
>
> And if we ever change QMP to allow non-objects for the arguments,
> introspection will still work.

Yes.

>>   The 'gen': false and 'success-response': false directives are
>>   omitted as implementation detail.
>
> I understand omitting 'gen' (hmm, I though we could get rid of it by
> making netdev_add typesafe, but now this patch adds another 'gen':false
> and for good reason).  But 'success-response' is part of the ABI; we
> already advertise it via qga's 'guest-info' command exposing
> 'success-response':'bool' for each command.  Adding it would allow a
> single introspection command to learn everything, instead of having to
> pair introspection with 'guest-info'.  On the other hand, as above, it's
> easier to start thin and add more later if needed, than it is to start
> full and then have to support it forever.

You're right, 'success-response' is not an implementation detail.  At
the very least, I need to come up with a different reason for omitting
it, say "TODO don't omit it".

>> * Events carry a single data value.
>> 
>>   Implicit type definition and empty object type use, just like for
>>   commands.
>> 
>>   The value is of object type, but the introspection schema doesn't
>>   reflect that.
>> 
>> * Types not used by commands or events are omitted.
>> 
>>   Indirect use counts as use.
>> 
>> * Optional members have a default, which can only be null right now
>> 
>>   Instead of a mandatory "optional" flag, we have an optional default.
>>   No default means mandatory, default null means optional without
>>   default value.  Non-null is available for optional with default.
>> 
>>   Alternate members can't have defaults, but the introspection schema
>>   doesn't reflect that.
>> 
>> * Clients should *not* look up types by name, because type names are
>>   not ABI.  Look up the command or event you're interested in, then
>>   follow the references.
>> 
>>   TODO Should we hide the type names to eliminate the temptation?
>
> Might be worth doing; I see you have a later patch to experiment with it.
>
>> 
>> * Likewise, the names of alternate members are not ABI, and should not
>>   be examined.
>> 
>>   TODO Should we hide them, too?
>
> How, by making the name of an object member optional?

I'd do the same as I do for non-ABI type names in my PATCH 47: replace
them by obviously useless names :)

>> TODO much of the above should go into docs.
>
> Indeed.  Hence why this is RFC still.
>
>> 
>> New generator scripts/qapi-introspect.py computes an introspection
>> value for its input, and generates a C variable holding it.
>> 
>> FIXME it can generate awfully long lines
>
> We already have long lines in generated output, but I agree that finding
> ways to break it up might be nice.  Actually,

This one's different in that the length is due to string literals.
indent is happy to break lines for us, but not string literals.

Longest line is a bit over 4KiB for me.

>> A new test-qmp-input-visitor test case feeds its result for both
>> tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a
>> QmpInputVisitor to verify it actually conforms to the schema.
>> 
>> New QMP command query-schema takes its return value from that
>> variable.  Command documentation is incomplete, and marked FIXME.  Its
>> reply is some 80KiBytes for me right now.
>
> $ ll qmp-introspect.[ch]
> -rw-rw-r--. 1 eblake eblake 89655 Jul 23 17:20 qmp-introspect.c
> -rw-rw-r--. 1 eblake eblake   358 Jul 23 17:20 qmp-introspect.h

Source is a bit longer than the string literal it defines.

>> If this turns out to be too much, we have a couple of options:
>> 
>> * We can use shorter names in the JSON.  Not the QMP style.
>> 
>> * Optionally return the sub-schema for commands and events given as
>>   arguments.
>
> Filtering is probably worth having, but I'm not sure how easy or hard it
> gets...
>> 
>>   Right now qmp_query_schema() sends the string literal computed by
>>   qmp-introspect.py.  To compute sub-schema at run time, we'd have to
>>   duplicate parts of qapi-introspect.py in C.  Unattractive.
>
> ...might be simpler than that. You can do some pre-processing, so that
> the C code can take shortcuts of having pre-computed dependency
> information. Instead of one giant string, you instead have an array of
> structs, one array entry per schema entity:
>
> struct foo {
>     const char *name;  /* example: ":empty" */
>     int metatype;      /*   SCHEMA_META_TYPE_OBJECT */
>     const char *json;  /*   "{ 'name': ':empty', 'meta-type': 'object',
> 'members': [  ] }, " */
>
>     int ndepends;      /* length of depends array */
>     int *depends;      /* indices of other entities this depends on */
>     bool visited;      /* helper for filtering */
> };
>
> If the user requests filtering, you then do a pre-pass that sets
> array[i]->visited = filter_match(), then an iterative pass that says for
> every array[i]->visited, you also want to set
> array[array[i]->depends[j]]->visited for each j up to
> array[i]->ndepends; then finally produce output of "[" + each
> array[i]->json where visited is true + "]".

Not thrilled by the additional complexity...

> Of course, adding filtering as a follow-on patch is the only way to go;
> initial implementation is fine as global information.

Agreed.

>> * Let clients cache the output of query-schema.
>> 
>>   It changes only on QEMU upgrades, i.e. rarely.  Provide a command
>>   query-schema-hash.  Clients can have a cache indexed by hash, and
>>   re-query the schema only when they don't have it cached.
>
> Libvirt already caches things.  I don't know if having qemu output a
> hash helps libvirt, or if libvirt won't mind just eating the cost of
> reading the entire array every time anything else causes it to
> repopulate the cache (such as timestamp on qemu binary changing).  We
> can play with it later.

Agreed.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>
> Missing a change to docs/qapi-code-gen.txt describing the new script and
> output.

Will fix.

>>  30 files changed, 345 insertions(+), 12 deletions(-)
>>  create mode 100644 qapi/introspect.json
>>  create mode 100644 scripts/qapi-introspect.py
>> 
>> diff --git a/.gitignore b/.gitignore
>> index aed0e1f..a6a02db 100644
>> --- a/.gitignore
>> +++ b/.gitignore
>> @@ -32,6 +32,7 @@
>>  /qapi-visit.[ch]
>>  /qapi-event.[ch]
>>  /qmp-commands.h
>> +/qmp-introspect.[ch]
>>  /qmp-marshal.c
>>  /qemu-doc.html
>>  /qemu-tech.html
>
> Missing an addition for the new tests/test-qmp-introspect.h file
> leftover after 'make'.  On second thought, that belongs at [2] below.
>
>> +++ b/monitor.c
>> @@ -74,6 +74,7 @@
>>  #include "block/qapi.h"
>>  #include "qapi/qmp-event.h"
>>  #include "qapi-event.h"
>> +#include "qmp-introspect.h"
>>  #include "sysemu/block-backend.h"
>>  
>>  /* for hmp_info_irq/pic */
>> @@ -924,6 +925,20 @@ EventInfoList *qmp_query_events(Error **errp)
>>      return ev_list;
>>  }
>>  
>> +/*
>> + * Minor hack: generated marshalling suppressed for this command
>> + * ('gen': false in the schema) so we can simply send the JSON string
>> + * instead of first parsing it with visit_type_SchemaInfoList() into a
>> + * SchemaInfoList, then unparse it right back in the generated output
>> + * marshaller, every time.
>> + * Instead, we do it in test-qmp-input-visitor.c, just to make sure
>> + * qapi-introspect.py's output actually conforms to the schema.
>> + */
>> +static void qmp_query_schema(QDict *qdict, QObject **ret_data, Error **errp)
>> +{
>> +    *ret_data = qobject_from_json(qmp_schema_json);
>
> Works for me :)

Should this ever become the last user of 'gen': false, we can still go
the parse - unparse route.

>> +++ b/qapi/introspect.json
>> @@ -0,0 +1,69 @@
>> +# -*- Mode: Python -*-
>> +#
>> +# QAPI introspection
>> +#
>> +# Copyright (C) 2015 Red Hat, Inc.
>> +#
>> +# Authors:
>> +#  Markus Armbruster <armbru@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2 or later.
>> +# See the COPYING file in the top-level directory.
>> +
>> +{ 'enum': 'SchemaMetaType',
>> +  'data': [ 'builtin', 'enum', 'array', 'object', 'alternate',
>> +            'command', 'event' ] }
>> +
>> +{ 'struct': 'SchemaInfoBase',
>> +  'data': { 'name': 'str', 'meta-type': 'SchemaMetaType' } }
>> +
>> +{ 'enum': 'JSONType',
>> +  'data': [ 'string', 'number', 'int', 'boolean', 'null',
>> +            'object', 'array', 'value' ] }
>> +
>> +{ 'struct': 'SchemaInfoBuiltin',
>> +  'data': { 'json-type': 'JSONType' } }
>> +
>> +{ 'struct': 'SchemaInfoEnum',
>> +  'data': { 'values': ['str'] } }
>
> Do we want to document anything about sort ordering of this list?  Is it
> worth sorting the array by name, to allow clients to bsearch for whether
> a particular enum value is supported, rather than having to linear search?

Haven't thought about it.  We currently emit them in definition order,
which is not sorted.  Largest enums:

    Q_KEY_CODE_MAX = 125,
    BLKDEBUG_EVENT_MAX = 43,
    BLOCKDEV_DRIVER_MAX = 28,
    CHARDEV_BACKEND_KIND_MAX = 19,
    RUN_STATE_MAX = 15,
    NET_CLIENT_OPTIONS_KIND_MAX = 12,

Most enums are small: median is 3.

Would libvirt prefer them sorted?

>> +
>> +{ 'struct': 'SchemaInfoArray',
>> +  'data': { 'element-type': 'str' } }
>> +
>> +{ 'struct': 'SchemaInfoObjectMember',
>> +  'data': { 'name': 'str', 'type': 'str', '*default': 'any' } }
>> +# @default's type must match @type
>
> or be null; as you mentioned above, this sets up a tri-state of
> mandatory, optional with no default, and (for future extension) optional
> with default.

Yes.

>> +
>> +{ 'struct': 'SchemaInfoObjectVariant',
>> +  'data': { 'case': 'str',
>> +            'members': [ 'SchemaInfoObjectMember' ] } }
>
> Would it be simpler to just have:
>
> 'data': { 'case': 'str', 'type': 'str' }
>
> and make the user refer recursively to the (possibly-implicit) type for
> the members?

Hmmm...

QAPISchemaObjectTypeVariant has members name, type, flat, and a few more
that don't matter here.

For a non-flat variant with name=N, type=T, my code creates

    { 'case': 'N', 'members': [ { 'name': 'data', 'type': 'T' } ] }

This means when the tag is 'N', we have a member 'data' of type 'T'.

For a flat variant, it creates

    { 'case': 'N', 'members': [ { ... the members of T ... } ] }

This means when the tag is 'N', we have all the members of T.

If I understand you correctly, you're proposing

    { 'case': 'N', 'type': 'T' }

to mean when the tag is 'N', we have all the members of T.  For the flat
variant above, we'd create exactly that.

T must not have variants, but the schema doesn't reflect that.

For the simple variant, we'd then create

    { 'case': 'N', 'type': 'TT' }

where TT is a new implicit object type with a single member with name
data and type T.

Correct?

> In particular, if we ever decide to allow a flat union to have another
> union as a branch, rather than the current restriction that all branches
> must be structs, then referring to the type of a branch may be easier
> than breaking out all members of a struct.

Not sure I completely get you here.  Using an object type instead of a
member list is obviously more flexible, because for any member list we
can make up an object type with the same meaning, but not vice versa.

>                                             And if that's the case, it
> may have knock-on simplifications to your earlier patches for tracking
> variants. See [1] below for more thoughts...
>
> Do we want to guarantee anything about the sort ordering in this list?

Again, haven't thought about it.

Do we expect member lists to get so large binary search is called for?

>> +
>> +{ 'struct': 'SchemaInfoObject',
>> +  'data': { 'members': [ 'SchemaInfoObjectMember' ],
>> +            '*tag': 'str',
>> +            '*variants': [ 'SchemaInfoObjectVariant' ] } }
>
> or these?

Same question.

>> +
>> +{ 'struct': 'SchemaInfoAlternate',
>> +  'data': { 'members': [ 'SchemaInfoObjectMember' ] } }
>
> Here's an example of what you generated:
>     "{ 'name': 'BlockdevRef', 'meta-type': 'alternate', 'members': [ {
> 'name': 'definition', 'type': 'BlockdevOptions' }, { 'name':
> 'reference', 'type': 'str' } ] }, "
>
> I think you could get away with something simpler:
>
>  'data': { 'types': [ 'str' ] }
>
> as in:
>  "{ 'name': 'BlockdevRef', 'meta-type': 'alternate', 'types': [
> 'BlockdevOptions', 'str' ] }, "

I.e. have a list of types instead of a list of members.

Let's see what we'd lose, by enumerating the members of
SchemaInfoObjectMember:

* name: not ABI, should not be examined (see commit message), thus no
  loss.

* type: kept.

* default: never present (see commit message), thus no loss

> the only worry is whether we might want future extensions, where we'd
> want additional information per element of that array, vs. being forced
> to return two arrays in parallel (arrays of structs are more extensible
> than arrays of strings).
> Seems like this would be just a

Yes?

Choices:

* The only piece of information we need on an alternative right now is
  the type, so make members a list of types.  Nice now, awkward if we
  ever need more,

* To provide for future additions, make it a list of
  SchemaInfoAlternateMember, where SchemaInfoAlternateMember has just
  one member type now.

* Reuse existing SchemaInfoObjectMember, because that's close and I'm
  lazy.

Preferences?

>> +
>> +{ 'struct': 'SchemaInfoCommand',
>> +  'data': { 'args': 'str', 'returns': 'str' } }
>
> Again, we can add 'success-return':'bool' later, if desired.

We probably will.

>> +
>> +{ 'struct': 'SchemaInfoEvent',
>> +  'data': { 'data': 'str' } }
>> +
>> +{ 'union': 'SchemaInfo',
>> +  'base': 'SchemaInfoBase',
>> +  'discriminator': 'meta-type',
>> +  'data': {
>> +      'builtin': 'SchemaInfoBuiltin',
>> +      'enum': 'SchemaInfoEnum',
>> +      'array': 'SchemaInfoArray',
>> +      'object': 'SchemaInfoObject',
>> +      'alternate': 'SchemaInfoAlternate',
>> +      'command': 'SchemaInfoCommand',
>> +      'event': 'SchemaInfoEvent' } }
>> +
>> +{ 'command': 'query-schema',
>> +  'returns': [ 'SchemaInfo' ],
>> +  'gen': false }                # just to simplify qmp_query_json()
>
> with an optional 'data':{'*filter':['str']} as a possible future
> extension.  Looks nice, in spite of being under-documented!

Thanks :)

>> +++ b/scripts/qapi-commands.py
>> @@ -265,7 +265,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
>>          self.defn = None
>>          self.regy = None
>>          self.visited_rets = None
>> -    def visit_begin(self):
>> +    def visit_begin(self, schema):
>
> And again my python object-oriented newness is showing through; where I
> guess all children have to update signatures to still be polymorphic to
> a parent adding a parameter.

Maybe they need to be updated, maybe not, but updating them neatly
sidesteps Python polymorphism questions from mere mortals like you and
me.

> Might be worth separating the patch to add the schema parameter so that
> there is less churn to the existing scripts - or even rebase the series
> up-front to always use the schema parameter and not change the signature
> here.

I'll consider it.

>>          self.decl = ''
>>          self.defn = ''
>>          self.regy = ''
>> @@ -273,7 +273,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
>>      def visit_end(self):
>>          if not middle_mode:
>>              self.defn += gen_registry(self.regy)
>> -            self.regy = None
>> +        self.regy = None
>> +        self.visited_rets = None
>
> Is it worth squashing this reset into the patch where the visitor was
> first written?

I'll double-check these resets are added in the correct patch.

>>      def visit_command(self, name, info, args, rets, gen, success_response):
>>          if not gen:
>>              return
>> diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
>> index 184a81f..71da7a9 100644
>> --- a/scripts/qapi-event.py
>> +++ b/scripts/qapi-event.py
>> @@ -139,13 +139,14 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
>>          self.decl = None
>>          self.defn = None
>>          self.event_names = None
>> -    def visit_begin(self):
>> +    def visit_begin(self, schema):
>>          self.decl = ''
>>          self.defn = ''
>>          self.event_names = []
>>      def visit_end(self):
>>          self.decl += gen_enum(event_enum_name, self.event_names)
>>          self.defn += gen_enum_lookup(event_enum_name, self.event_names)
>> +        self.event_names = None
>
> and again

I'll double-check.

>>      def visit_event(self, name, info, data):
>>          self.decl += gen_event_send_decl(name, data)
>>          self.defn += gen_event_send(name, data)
>> diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
>> new file mode 100644
>> index 0000000..e7efc4a
>> --- /dev/null
>> +++ b/scripts/qapi-introspect.py
>> @@ -0,0 +1,159 @@
>> +#
>> +# QAPI introspection generator
>> +#
>> +# Copyright (C) 2015 Red Hat, Inc.
>> +#
>> +# Authors:
>> +#  Markus Armbruster <armbru@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2.
>> +# See the COPYING file in the top-level directory.
>> +
>> +from qapi import *
>> +
>> +class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
>> +    def __init__(self):
>> +        self.schema = None
>> +        self.jsons = None
>> +        self.used_types = None
>> +        self.defn = None
>> +        self.decl = None
>> +
>> +    def visit_begin(self, schema):
>> +        self.schema = schema
>> +        self.jsons = []
>> +        self.used_types = []
>> +        return QAPISchemaType   # don't visit types for now
>> +
>> +    def visit_end(self):
>> +        # visit the types that are actually used
>> +        for typ in self.used_types:
>> +            typ.visit(self)
>> +        self.jsons.sort()
>> +        name = prefix + 'qmp_schema_json'
>> +        self.decl = mcgen('''
>> +extern char %(c_name)s[];
>
> Missing 'const'

Will fix.

>> +''',
>> +                          c_name=c_name(name))
>> +        self.defn = mcgen('''
>> +char %(c_name)s[] = "["
>> +    "%(c_jsons)s]";
>
> And again. Also, I'd consider putting the "]" on its own line, like the
> "[" was, so that you can more easily cut and paste individual lines of
> generated output (but since JSON doesn't allow trailing comma, I guess
> the last line is still always going to be special).

That's my reason for keepint the "]" there.

>> +''',
>> +                          c_name=c_name(name),
>> +                          c_jsons=', "\n    "'.join(self.jsons))
>
> Cool syntax :)

Possibly bordering on too cool :)

>> +        self.schema = None
>> +        self.jsons = None
>> +        self.used_types = None
>> +
>> +    def _use_type(self, typ):
>> +        if typ not in self.used_types:
>> +            self.used_types.append(typ)
>> +        return typ.name
>> +
>> +    def _gen_json(self, name, mtype, extra):
>> +        self.jsons.append("{ 'name': '%s', 'meta-type': '%s', %s }"
>> +                          % (name, mtype, extra))
>
> Problem. We document that our QMP engine accepts 'single-quoted' input
> as an extension to JSON, but that our output will always be
> "double-quoted".  That means you _must_ write \" everywhere in the
> generated C code. (Also, you set the style guide earlier of using ''
> quoting around any text destined for generated C code)
>
> self.jsons.append('''{ \"name\": \"%s\", \"meta-type\": \"%s\", %s }'''
>                   % (name, mtype, extra))
>
> (at least, I assume '''...\"...''' is nicer than '...\\\"...')

You're right, I shouldn't emit single-quoted JSON.

>> +
>> +    def _gen_members(self, members):
>> +        return ("'members': [ "
>> +                + ", ".join([self._gen_member(m) for m in members])
>> +                + " ]")
>
> and more choice-of-quotes issues.  Lots more below; I'll quit pointing
> them out on this round of review.
>
>> +
>> +    def _gen_member(self, member):
>> +        default = ''
>> +        if member.optional:
>> +            default = ", 'default': null"
>> +        return "{ 'name': '%s', 'type': '%s'%s }" \
>> +            % (member.name, self._use_type(member.type), default)
>> +
>> +    def _gen_variants(self, tag_name, variants):
>> +        return ("'tag': '%s'" % tag_name
>> +                + ", 'variants': [ "
>> +                + ", ".join([self._gen_variant(v) for v in variants])
>> +                + " ]")
>> +
>> +    def _gen_variant(self, variant):
>> +        if variant.flat:
>> +            members = self._gen_members(variant.type.members)
>> +        else:
>> +            members = "'members': [ { 'name': 'data', 'type': '%s' } ]" \
>> +                      % self._use_type(variant.type)
>> +        return "{ 'case': '%s', %s }" % (variant.name, members)
>
> [1] Ah, so .flat is still in use here, to avoid having to create
> implicit types everywhere.  But if we create implicit types for simple
> unions, and just track variants by their case name/type instead of case
> name/[members], it will allow us to have a union as a case branch (I
> don't know that we need that much flexibility), and not have to worry
> about exposing .flat everywhere.  It may even result in a smaller JSON
> string (you'd have to play with it to know for sure).

We should try hard to get the introspection schema right from the start.

But the internal representation is malleable.  I know we can implement
non-flat unions completely in terms of flat ones (and that means no
.flat), but I also know I failed at doing it in this series.  I'm not
sure your idea will do the trick completely.  It's fine, we can finish
the job later.

>> +
>> +    def visit_builtin_type(self, name, info, json_type):
>> +        self._gen_json(name, 'builtin',
>> +                       "'json-type': '%s'" % json_type)
>> +
>> +    def visit_enum_type(self, name, info, values):
>> +        self._gen_json(name, 'enum',
>> +                       "'values': [ %s ]" % ", ".join(["'%s'" % v
>> +                                                       for v in values]))
>> +
>> +    def visit_array_type(self, name, info, element_type):
>> +        self._gen_json(name, 'array',
>> +                       "'element-type': '%s'" % self._use_type(element_type))
>> +
>> +    def visit_object_type_flat(self, name, info, members, variants):
>> +        extra = self._gen_members(members)
>> +        if variants:
>> +            extra += ", " + self._gen_variants(variants.tag_name or "type",
>> +                                               variants.variants)
>> +        self._gen_json(name, 'object', extra)
>> +
>
> Do we really need two types of object visitors, or can you reuse the
> signature all the other visitors have?

The other generators all work with high-level information: base, local
members, variants.

qapi-introspect.py wants a different, more low-level view of the
members: flattened members (including inherited members and any implicit
tag member).

I could shoehorn both views into a single visitor function, by passing
both views, .base + .local_members, and .members.  All implementations
will use only one of the views, but it's not immediately obvious which
one.  So I chose to have two visitor functions.  Matter of taste.

>> +    def visit_alternate_type(self, name, info, variants):
>> +        self._gen_json(name, 'alternate',
>> +                       self._gen_members(variants.variants))
>> +
>> +    def visit_command(self, name, info, args, rets, gen, success_response):
>> +        args = args or self.schema.the_empty_object_type
>> +        rets = rets or self.schema.the_empty_object_type
>> +        self._gen_json(name, 'command',
>> +                       "'args': '%s', 'returns': '%s'" \
>> +                       % (self._use_type(args), self._use_type(rets)))
>> +
>> +    def visit_event(self, name, info, data):
>> +        data = data or self.schema.the_empty_object_type
>> +        self._gen_json(name, 'event', "'data': '%s'" % self._use_type(data))
>> +
>> +(input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
>> +
>
>> +++ b/scripts/qapi-types.py
>> @@ -184,7 +184,6 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
>>          self.fwdecl = None
>>          self.fwdefn = None
>>          self.btin = None
>> -    def visit_begin(self):
>>          self.decl = ''
>
> Rebase botch mentioned above [0]

Will fix.

>>          self.defn = ''
>>          self.fwdecl = ''
>> diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
>> index 2813bb3..7a03292 100644
>> --- a/scripts/qapi-visit.py
>> +++ b/scripts/qapi-visit.py
>> @@ -326,7 +326,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
>>          self.decl = None
>>          self.defn = None
>>          self.btin = None
>> -    def visit_begin(self):
>> +    def visit_begin(self, schema):
>>          self.decl = ''
>>          self.defn = ''
>>          self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
>> diff --git a/scripts/qapi.py b/scripts/qapi.py
>> index 6ddb33e..7f9a159 100644
>> --- a/scripts/qapi.py
>> +++ b/scripts/qapi.py
>> @@ -764,7 +764,7 @@ class QAPISchemaEntity(object):
>>          pass
>>  
>>  class QAPISchemaVisitor(object):
>> -    def visit_begin(self):
>> +    def visit_begin(self, schema):
>
> I don't know enough python to know if making schema optional in the
> parent class affects what the child class is allowed to implement while
> still overriding things.

Not sure I get you here.

>>          pass
>>      def visit_end(self):
>>          pass
>> @@ -776,6 +776,8 @@ class QAPISchemaVisitor(object):
>>          pass
>>      def visit_object_type(self, name, info, base, members, variants):
>>          pass
>> +    def visit_object_type_flat(self, name, info, members, variants):
>> +        pass
>>      def visit_alternate_type(self, name, info, variants):
>>          pass
>>      def visit_command(self, name, info, args, rets, gen, success_response):
>> @@ -892,6 +894,8 @@ class QAPISchemaObjectType(QAPISchemaType):
>>      def visit(self, visitor):
>>          visitor.visit_object_type(self.name, self.info,
>>                                    self.base, self.local_members, self.variants)
>> +        visitor.visit_object_type_flat(self.name, self.info,
>> +                                       self.members, self.variants)
>
> Instead of having two object visitor signatures, would it be worth
> having a boolean parameter to QAPISchema.visit() that says whether to
> pass base, members as (base type, local members) vs. (None, all members)?

Then semantics of the single function depends on state set elsewhere.
Doesn't feel simpler to me.

>>  class QAPISchemaObjectTypeMember(object):
>>      def __init__(self, name, typ, optional):
>> @@ -1042,6 +1046,9 @@ class QAPISchema(object):
>>                    ('bool',   'boolean', 'bool',     'false'),
>>                    ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
>>              self._def_builtin_type(*t)
>> +        self.the_empty_object_type = QAPISchemaObjectType(':empty', None, None,
>> +                                                          [], None)
>
> Cool global. Worth adding in a separate patch?

Three-liner patch without a user.  I'd expect reviewers suggesting to
squash it into the patch adding a user :)

>> +        self._def_entity(self.the_empty_object_type)
>>  
>>      def _make_implicit_enum_type(self, name, values):
>>          name = name + 'Kind'
>> @@ -1180,9 +1187,10 @@ class QAPISchema(object):
>>              ent.check(self)
>>  
>>      def visit(self, visitor):
>> -        visitor.visit_begin()
>> +        ignore = visitor.visit_begin(self)
>>          for name in sorted(self.entity_dict.keys()):
>> -            self.entity_dict[name].visit(visitor)
>> +            if not ignore or not isinstance(self.entity_dict[name], ignore):
>> +                self.entity_dict[name].visit(visitor)
>
> So this lets introspection bypass visiting types on the first pass, and
> then collect used types during visit_end().  It means you can only
> bypass a single metatype, but that is sufficient for your use; I don't
> know if there is any better idiom for this paradigm.

I toyed around with several solutions for my problem, and this one came
out simplest.  I don't exactly love it, but I dislike it less than the
other ones.

>>          visitor.visit_end()
>>  
>>  #
>> diff --git a/tests/.gitignore b/tests/.gitignore
>> index dc813c2..dda86cc 100644
>> --- a/tests/.gitignore
>> +++ b/tests/.gitignore
>> @@ -19,6 +19,7 @@ test-opts-visitor
>>  test-qapi-event.[ch]
>>  test-qapi-types.[ch]
>>  test-qapi-visit.[ch]
>> +test-qapi-introspect.[ch]
>
> [2] Ah, maybe this is the file that wasn't quite right.

Anything I need to fix?

>>  test-qdev-global-props
>>  test-qemu-opts
>>  test-qmp-commands
>> diff --git a/tests/Makefile b/tests/Makefile
>> index 60b82e2..7b1bf92 100644
>> --- a/tests/Makefile
>> +++ b/tests/Makefile
>> @@ -253,7 +253,8 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
>>  	struct-base-clash.json struct-base-clash-deep.json )
>>  
>>  GENERATED_HEADERS += tests/test-qapi-types.h tests/test-qapi-visit.h \
>> -		     tests/test-qmp-commands.h tests/test-qapi-event.h
>> +	tests/test-qmp-commands.h tests/test-qapi-event.h \
>> +	tests/test-qmp-introspect.h
>>  
>>  test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
>>  	tests/check-qlist.o tests/check-qfloat.o tests/check-qjson.o \
>> @@ -266,7 +267,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
>>  	tests/rcutorture.o tests/test-rcu-list.o
>>  
>>  test-qapi-obj-y = tests/test-qapi-visit.o tests/test-qapi-types.o \
>> -		  tests/test-qapi-event.o
>> +		  tests/test-qapi-event.o tests/test-qmp-introspect.o
>>  
>>  $(test-obj-y): QEMU_INCLUDES += -Itests
>>  QEMU_CFLAGS += -I$(SRC_PATH)/tests
>> @@ -327,6 +328,11 @@ $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-eve
>>  	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
>>  		$(gen-out-type) -o tests -p "test-" $<, \
>>  		"  GEN   $@")
>> +tests/test-qmp-introspect.c tests/test-qmp-introspect.h :\
>> +$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
>> +	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
>> +		$(gen-out-type) -o tests -p "test-" $<, \
>> +		"  GEN   $@")
>>  
>>  tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y) libqemuutil.a libqemustub.a
>>  tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y) libqemuutil.a libqemustub.a
>> diff --git a/tests/qapi-schema/alternate-good.out b/tests/qapi-schema/alternate-good.out
>> index 0cbdfa1..aede1ae 100644
>> --- a/tests/qapi-schema/alternate-good.out
>> +++ b/tests/qapi-schema/alternate-good.out
>> @@ -1,3 +1,4 @@
>> +object :empty
>>  alternate Alt
>
> Again, adding the magic :empty object in a separate patch from the new
> introspection code may help minimize the number of files being touched
> with the new code.

Okay, it's not a three-liner, it's a three-liner plus a trivial expected
test output churn.  I'll think about it.

>> +++ b/tests/test-qmp-input-visitor.c
>> @@ -17,6 +17,9 @@
>>  #include "qapi/qmp-input-visitor.h"
>>  #include "test-qapi-types.h"
>>  #include "test-qapi-visit.h"
>> +#include "test-qmp-introspect.h"
>> +#include "qmp-introspect.h"
>> +#include "qapi-visit.h"
>>  #include "qapi/qmp/types.h"
>>  
>>  typedef struct TestInputVisitorData {
>> @@ -660,6 +663,31 @@ static void test_visitor_in_native_list_number(TestInputVisitorData *data,
>>      qapi_free_UserDefNativeListUnion(cvalue);
>>  }
>>  
>> +static void do_test_visitor_in_qmp_introspect(TestInputVisitorData *data,
>> +                                              const char *schema_json)
>> +{
>> +    SchemaInfoList *schema = NULL;
>> +    Error *err = NULL;
>> +    Visitor *v;
>> +
>> +    v = visitor_input_test_init_raw(data, schema_json);
>> +
>> +    visit_type_SchemaInfoList(v, &schema, NULL, &err);
>> +    if (err)
>> +        fprintf(stderr, "%s", error_get_pretty(err));
>> +    g_assert(!err);
>
> Don't you want:
> visit_type_SchemaInfoList(..., &error_abort);

I'm mimicking existing usage in this file.

>> +    g_assert(schema);
>> +
>> +    qapi_free_SchemaInfoList(schema);
>> +}
>> +
>> +static void test_visitor_in_qmp_introspect(TestInputVisitorData *data,
>> +                                           const void *unused)
>> +{
>> +    do_test_visitor_in_qmp_introspect(data, test_qmp_schema_json);
>> +    do_test_visitor_in_qmp_introspect(data, qmp_schema_json);
>> +}
>> +
>>  static void input_visitor_test_add(const char *testpath,
>>                                     TestInputVisitorData *data,
>>                                     void (*test_func)(TestInputVisitorData *data, const void *user_data))
>> @@ -753,6 +781,9 @@ int main(int argc, char **argv)
>>      input_visitor_test_add("/visitor/input/native_list/number",
>>                             &in_visitor_data,
>>                             test_visitor_in_native_list_number);
>> +    input_visitor_test_add("/visitor/input/qmp_introspect",
>> +                           &in_visitor_data,
>> +                           test_visitor_in_qmp_introspect);
>>  
>>      g_test_run();
>>  
>> 
>
> Starting to shape up nicely.

Thanks!  I really appreciate your review.  Must have been a big chunk of
work.

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1935 bytes --]

On 07/28/2015 12:41 AM, Markus Armbruster wrote:
>> Like reserving ourselves a namespace based on single _ for internal use.
>>  We practically already have that - all user names either start with a
>> letter or double underscore, so we could use single (and triple)
>> underscore for internally-generated purposes, freeing up 'base',
>> '*Kind', '*_MAX', and other namespace abuses back to the user.  Well, we
>> may also need to reserve mid-name double-underscore (that is, the user
>> can only supply double underscore at the beginning, but not middle, of
>> an identifier).  Ah well, food for thought for later patches.
> 
> Another concern: we should take care not to generate reserved
> identifiers.

And we do that for a number of identifiers already, by renaming
'default' in qapi to 'q_default' in C.

> 
> * Potential issue with your proposal: identifiers that begin with an
>   underscore and either an uppercase letter or another underscore are
>   always reserved for any use.

True, so only underscore and lower is guaranteed safe. But in practice,
we can often get away with more...

> 
> * Existing issue: downstream extensions carry a __RFQDN_ prefix in the
>   schema, which map to reserved C identifiers.

...the whole point of the __RFQDN_ prefix was that it is very unlikely
to conflict with any vendor extensions (unless you happen to be the same
vendor that writes the compiler and als use __RFQDN_ as part of your
compiler implementation).  Yes, it's risky, but a risk that hasn't hurt
any downstream clients yet :)  It's one of those things where if someone
reports an actual problem in their environment, we'll fix it; but until
then, it's not worth the headache of strict compliance to "fix"
something that happens to work in spite of being undefined behavior.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2266 bytes --]

On 07/28/2015 01:34 AM, Markus Armbruster wrote:
> Let me rephrase to make sure I understand.
> 
> Ignore the (not rets) case, because retval doesn't exist then.
> 
> qmp_marshal_output_FOO() visits retval twice.  First, with a QMP output
> visitor to do the actual marshalling.  Second, with a QAPI dealloc
> visitor to destroy it.

And the use of the dealloc visitor is buried inside the
qmp_marshal_output_FOO() call.

> 
> If we execute the assignment to retval, we must go on to call
> qmp_marshal_output_FOO(), or else we have a leak.
> 
> If we can reach qmp_marshal_output_FOO() without executing the
> assignment, we must initialize retval.  If we can't, any initialization
> is unused.
> 
> gen_call() generates code of the form
> 
>         retval = qmp_FOO(... args ..., &local_err);
>         if (local_err) {
>             goto out;
>         }
> 
>         qmp_marshal_output_FOO(retval, ret, &local_err);
> 

> Observe:
> 
> 1. The assignment dominates the only use.  Therefore, the initialization
>    is unused.  Let's drop it in a separate cleanup patch.
> 
> 2. We can leak retval only when qmp_FOO() returns non-null and local_err
>    is non-null.  This must not happen, because:
> 
>    a. local_err must be null before the call, and
> 
>    b. the call must not return non-null when it sets local_err.

We don't state that contract anywhere, but I doubt any of the qmp_FOO()
functions violate it, so it is worth making it part of the contract.

> 
>    We could right after out: assert(!local_err || !retval).  Not sure
>    it's worthwhile.

I think it IS worthwhile, because it would catch buggy callers.  Not
sure if after out: is the right place (then you'd need an initializer to
cover any other code that jumps to out), but this would do the same

retval = qmp_FOO(...);
if (local_err) {
    assert(!retval);
    goto out;
}
qmp_marshal_output_FOO(retval, ...);

> 
> TL;DR: I concur with your analysis.

Is it worth dropping the dead initializer and adding the assert in the
same pre-req cleanup patch?  Do you want me to submit it since I did the
analysis?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 37/47] qapi: De-duplicate parameter list generation

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

On 07/28/2015 05:15 AM, Markus Armbruster wrote:

>> Yeah, avoiding line wraps consumes fewer source bytes (fewer runs of
>> spaces), but the space isn't being wasted by storing generated files in
>> git, nor does the C compiler care which layout we use.  And honestly,
>> it's easier to spot changes in a vertical list than it is on a long
>> horizontal line, if a parameter gets added (or removed, although adding
>> is the more likely action with qapi).
> 
> Number of source bytes is not an issue.
> 
> The generators make no effort to wrap source lines, except in the
> qapi_event_send_FOO()'s parameter lists.
> 
> We could preserve that one-off.  We could extend it to more places that
> can generate long lines, saddling the generation code with indentation
> concerns.  I don't want to write such code, and I don't want to maintain
> it.
> 
> Instead, why not keep the generators straightforward, and feed their
> result to indent when "pretty" is wanted?  Requires an indent profile
> matching QEMU style.

Long lines aren't the end of the world. They may be harder to read when
diffing pre- and post-patch generated output to see if a generator
change makes sense, but you have a point that line wrapping is more
maintenance. So you win; keep the long lines, and if someone wants
wrapping, they can (re-)add it as a later patch series.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> To eliminate the temptation for clients to look up types by name
>> (which are not ABI), replace all type names by meaningless strings.
>> 
>> Reduces output of query-schema by 9 out of 80KiB.
>
> I'm not sure whether I like this or not.  It does make sense from the
> perspective of forcing clients to stick to ABI queries, but makes it a
> bit harder to navigate things except by automated scripts.

Yes.  I'm not sure it's a good idea.  If we decide to hide types this
way, then I'd find an option to generate without type hiding useful.

>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>> ---
>>  scripts/qapi-introspect.py | 27 +++++++++++++++++++++++++--
>>  1 file changed, 25 insertions(+), 2 deletions(-)
>> 
>> diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
>> index 961fe88..efb34ff 100644
>> --- a/scripts/qapi-introspect.py
>> +++ b/scripts/qapi-introspect.py
>> @@ -9,13 +9,18 @@
>>  # This work is licensed under the terms of the GNU GPL, version 2.
>>  # See the COPYING file in the top-level directory.
>>  
>> +import string
>>  from qapi import *
>>  
>> +def _b32digit(num):
>> +    return (string.lowercase + string.digits[2:])[num]
>
> This feels a bit too magic for me to decipher late at night.
>
>> +
>>  class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
>>      def __init__(self):
>>          self.schema = None
>>          self.jsons = None
>>          self.used_types = None
>> +        self.name_map = None
>>          self.defn = None
>>          self.decl = None
>>  
>> @@ -23,13 +28,17 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
>>          self.schema = schema
>>          self.jsons = []
>>          self.used_types = []
>> +        self.name_map = {}
>>          return QAPISchemaType   # don't visit types for now
>>  
>>      def visit_end(self):
>>          # visit the types that are actually used
>> +        jsons = self.jsons
>> +        self.jsons = []
>>          for typ in self.used_types:
>>              typ.visit(self)
>>          self.jsons.sort()
>> +        jsons.extend(self.jsons)
>
> I'm not sure I follow the use of .extends() here.

jsons.extend(self.jsons) appends the elements of array self.jsons to
array jsons.

Before the patch:

At function entry, self.jsons contains introspection information for the
non-types.

The loop visits the types.  This adds introspection information for the
types to self.json.  Sort self.jsons by name.

What the patch adds:

Move the introspection information for the non-types out of the way
before the loop, append the information on types afterwards.  The result
is now in jsons rather than self.jsons (see the next patch hunk).

Why it does that:

With the funny typenames, sorting everything by name results in a mess.
Keeping non-types and types separate is less of a mess.

>>          name = prefix + 'qmp_schema_json'
>>          self.decl = mcgen('''
>>  extern char %(c_name)s[];
>> @@ -40,10 +49,19 @@ char %(c_name)s[] = "["
>>      "%(c_jsons)s]";
>>  ''',
>>                            c_name=c_name(name),
>> -                          c_jsons=', "\n    "'.join(self.jsons))
>> +                          c_jsons=', "\n    "'.join(jsons))
>>          self.schema = None
>>          self.jsons = None
>>          self.used_types = None
>> +        self.name_map = None
>> +
>> +    def _name(self, name):
>> +        if name not in self.name_map:
>> +            n = len(self.name_map)
>> +            self.name_map[name] = ':' + _b32digit(n / 32 / 32) \
>> +                                  + _b32digit(n / 32 % 32) \
>> +                                  + _b32digit(n % 32)
>
> Caps our qapi to 32k types (including implicit ones); will that be an issue?

I doubt it :)

Should we ever hit the limit, we can simply add another base32
character.

>> +        return self.name_map[name]
>>  
>>      def _use_type(self, typ):
>>          # Map the various integer types to plain int
>> @@ -55,9 +73,14 @@ char %(c_name)s[] = "["
>>          # Add type to work queue if new
>>          if typ not in self.used_types:
>>              self.used_types.append(typ)
>> -        return typ.name
>> +        # Clients should examine commands and events, not types.  Hide
>> +        # type names to reduce the temptation.  Also saves a few
>> +        # characters.
>> +        return self._name(typ.name)
>>  
>>      def _gen_json(self, name, mtype, extra):
>> +        if mtype != 'command' and mtype != 'event':
>> +            name = self._name(name)
>>          self.jsons.append("{ 'name': '%s', 'meta-type': '%s', %s }"
>>                            % (name, mtype, extra))
>>  
>> 

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/23/2015 09:44 PM, Eric Blake wrote:
>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>> To eliminate the temptation for clients to look up types by name
>>> (which are not ABI), replace all type names by meaningless strings.
>>>
>>> Reduces output of query-schema by 9 out of 80KiB.
>> 
>> I'm not sure whether I like this or not.  It does make sense from the
>> perspective of forcing clients to stick to ABI queries, but makes it a
>> bit harder to navigate things except by automated scripts.
>> 
>
>>>  
>>> +import string
>>>  from qapi import *
>>>  
>>> +def _b32digit(num):
>>> +    return (string.lowercase + string.digits[2:])[num]
>> 
>> This feels a bit too magic for me to decipher late at night.
>
> Looking at this again after a weekend: you are doing a poor-man's base32
> encoding that maps a number between 0 and 31 inclusive to
> "abc...xyz234567" (skipping 0 and 1 due to some fonts being unclear with
> O and l/I).
>
> Question - if we really want to compress things down to an integer,
> would it make any more sense to actually use an integer instead of a
> string encoding of an unusual base32 alphabet (that is, output an
> integer instead of a string)?  Conversely, if we are going to insist on
> an encoding, can we use the RFC4648 base32 definition (upper case, not
> lower case)?

I don't remember why I did lowercase.  Possibly an accident at the tail
end of a mad hacking spree.  Uppercase is fine with me.

Could do plain integer.  I guess I started down the base32 road to
squeeze out a few more characters, then sabotaged myself by always using
three base32 characters.

> More thoughts: I'm probably okay with hiding type names (since they
> aren't ABI), but can we come up with an output format that is more
> conducive for use by the end user?  Considering that we are returning a
> JSON array, what if we return the integer offset of the type as recorded
> in the returned array.  That is, if we have
>  { 'name':'ACPI_DEVICE_OST', 'meta-type':'event', 'data':100 }
> in slot [0] of the return array, then
>  { 'name': ':aaa', 'meta-type': 'object', 'members': [ { 'name': 'info',
> 'type': 250 } ] }
> in slot [100], then the definition for type ':ae5' in slot [250], and so
> forth.  My goal here is not a topologically sorted return array, so much
> as a shorthand where using an integer for a type-name means that we can
> quickly locate that offset within the JSON array, instead of having to
> do a linear search over the entire array for an array member that has
> the matching name.

In the introspection schema, every 'str' that's really a type name needs
to be replaced by 'int'.

Should we later decide we don't want to hide type names after all, then
backward compatibility will make it very hard to go back.

I wouldn't expect clients to find stuff with a linear search.  Use a
dictionary.  Should be plenty fast enough for processing the schema.

>                     Or if type names are truly unimportant, then omit
> names for type elements (by making name optional in the introspection
> qapi description), and using ONLY offsets in the returned JSON array for
> referring to types.  Of course, if we do this, life gets a lot trickier
> for adding filtering down to a subset of the overall schema (unless you
> don't mind populating lots of 'null' entries for parts that get filtered
> out so that the parts that are displayed are always at the same array
> offset, just with less overall output bulk due to the filtering).

Filtering is a headache I'd prefer to avoid.

Re: [Qemu-devel] [PATCH RFC v2 45/47] qapi: New QMP command query-schema for QMP schema introspection

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 15948 bytes --]

On 07/28/2015 08:33 AM, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>> Caution, rough edges.
>>
>> No joke. It doesn't even compile without this fixup to a rebase snafu
>> (see [0] below):
> 
> Uh, how did that happen?  I compiled it a million times... (except for
> the last time, obviously).

At least you're not alone; I've done dumb things like that, too.  And at
least it was easy to figure out, so I could test it.

>>>
>>> FIXME it can generate awfully long lines
>>
>> We already have long lines in generated output, but I agree that finding
>> ways to break it up might be nice.  Actually,
> 
> This one's different in that the length is due to string literals.
> indent is happy to break lines for us, but not string literals.
> 
> Longest line is a bit over 4KiB for me.
> 

If we break up string literals, at least use some indentation to make it
obvious that multiple lines merge to a single array entry. For example
(after patch 47):

...
    "{ 'name': ':abr', 'meta-type': 'object', "
      "'members': [ "
        "{ 'name': 'device', 'type': ':acg', 'default': null }, "
        "{ 'name': 'node-name', 'type': ':acg', 'default': null }, "
        "{ 'name': 'snapshot-file', 'type': ':acg' }, "
        "{ 'name': 'snapshot-node-name', 'type': ':acg', 'default': null
}, "
        "{ 'name': 'format', 'type': ':acg', 'default': null }, "
        "{ 'name': 'mode', 'type': ':afo', 'default': null } ] }, "
    "{ 'name': ... "

Oh, and just thought of something while typing: although patch 47 masks
type names from the end user, it would be VERY worthwhile if the C code
had strategic comments that pointed back to the qapi type name:

    "{ 'name': ':abr', " /* TypeFoo */
      "'meta-type': 'object'"
...

(and then there's the '' vs. "" issue which impacts the output, as
well).  And if you do the "," between entries as a separate line, that
might impact whether you do a trailing "]" on a line by its own (see [3]
below).

>>> +
>>> +{ 'struct': 'SchemaInfoEnum',
>>> +  'data': { 'values': ['str'] } }
>>
>> Do we want to document anything about sort ordering of this list?  Is it
>> worth sorting the array by name, to allow clients to bsearch for whether
>> a particular enum value is supported, rather than having to linear search?
> 
> Haven't thought about it.  We currently emit them in definition order,
> which is not sorted.  Largest enums:
> 
>     Q_KEY_CODE_MAX = 125,
>     BLKDEBUG_EVENT_MAX = 43,
>     BLOCKDEV_DRIVER_MAX = 28,
>     CHARDEV_BACKEND_KIND_MAX = 19,
>     RUN_STATE_MAX = 15,
>     NET_CLIENT_OPTIONS_KIND_MAX = 12,
> 
> Most enums are small: median is 3.
> 
> Would libvirt prefer them sorted?

Libvirt can probably live without sorting of enum constants (searching
Q_KEY_CODE_MAX isn't going to be that noticeable of an impact on O(n)
vs. O(logn); I predict much more time will be spent searching for type
"xyz" referenced by command "abc" from the overall array, and repeating
that search for multiple feature probes).

But if we do want sorting, we need it up front (it will be easier to
decide to use bsearch down the road if we are guaranteed that output is
sorted, than it would be to try and learn whether whether we are talking
to a new vs. old qemu to learn if sorting is in effect because it was
added as an after-thought).  And if we don't want sorting, documenting
that data is NOT guaranteed to be position-dependent, in spite of being
in a JSON array, is a nice touch.

>>> +
>>> +{ 'struct': 'SchemaInfoObjectVariant',
>>> +  'data': { 'case': 'str',
>>> +            'members': [ 'SchemaInfoObjectMember' ] } }
>>
>> Would it be simpler to just have:
>>
>> 'data': { 'case': 'str', 'type': 'str' }
>>
>> and make the user refer recursively to the (possibly-implicit) type for
>> the members?
> 
> Hmmm...
> 
> QAPISchemaObjectTypeVariant has members name, type, flat, and a few more
> that don't matter here.
> 
> For a non-flat variant with name=N, type=T, my code creates
> 
>     { 'case': 'N', 'members': [ { 'name': 'data', 'type': 'T' } ] }
> 
> This means when the tag is 'N', we have a member 'data' of type 'T'.
> 
> For a flat variant, it creates
> 
>     { 'case': 'N', 'members': [ { ... the members of T ... } ] }
> 
> This means when the tag is 'N', we have all the members of T.
> 
> If I understand you correctly, you're proposing
> 
>     { 'case': 'N', 'type': 'T' }
> 
> to mean when the tag is 'N', we have all the members of T.  For the flat
> variant above, we'd create exactly that.
> 
> T must not have variants, but the schema doesn't reflect that.

That's our current restriction, but it's one we might decide to lift in
the future.  Having a type with two different discriminators could get a
bit weird to think about, but doesn't seem to be technically impossible.

> 
> For the simple variant, we'd then create
> 
>     { 'case': 'N', 'type': 'TT' }
> 
> where TT is a new implicit object type with a single member with name
> data and type T.
> 
> Correct?

Yes.

> 
>> In particular, if we ever decide to allow a flat union to have another
>> union as a branch, rather than the current restriction that all branches
>> must be structs, then referring to the type of a branch may be easier
>> than breaking out all members of a struct.
> 
> Not sure I completely get you here.  Using an object type instead of a
> member list is obviously more flexible, because for any member list we
> can make up an object type with the same meaning, but not vice versa.

Indeed, and that's the restriction that I mention we might someday want
to lift.  Or, in (modified, due to inline {} types) qapi terms, if we
ever want to allow:

{ 'union': 'Helper', 'data': { 'number': 'int', 'string': 'str' } }
{ 'enum': 'MyEnum', 'data': [ 'a', 'b' ] }
{ 'union': 'Main', 'base': { 'switch': 'MyEnum' },
  'discriminator': 'switch', 'data': { 'a': { 'boolean': 'bool' },
                                       'b': 'Helper' } }
{ 'command': 'foo', 'data': { 'data': 'Main' } }

then that would permit QMP invocations of:
{ "execute": "foo", "arguments": { "data": {
  "switch": "a", "boolean": true } } }
{ "execute": "foo", "arguments": { "data": {
  "switch": "b", "type": "number", "data": 1 } } }
{ "execute": "foo", "arguments": { "data": {
  "switch": "b", "type": "string", "data": "hello" } } }

which we can express as a list of case/types for the primary variants of
'Main' (those types in turn refer to the secondary variants of type
Helper), but which we cannot express as a [ 'SchemaInfoObjectMember' ]
list, because the type of the "data" member depends on the secondary
discriminator that is called into use on the "b" case of the primary
discriminator.

So I think we're saying the same thing, that a [
'SchemaInfoObjectMember' ] can always be written as a reference to an
object type name, but not all object type names can be broken back into
an array of SchemaInfoObjectMember (only those types that are pure
structs without variants); and that although we currently do not allow
sub-variants within a union, we should not get in the way of that being
a possible future extension.

> 
>>                                             And if that's the case, it
>> may have knock-on simplifications to your earlier patches for tracking
>> variants. See [1] below for more thoughts...
>>
>> Do we want to guarantee anything about the sort ordering in this list?
> 
> Again, haven't thought about it.
> 
> Do we expect member lists to get so large binary search is called for?

Probably not, since such a list would be unwieldy for both client and
server.  We tend to add boxing and optional sub-structs rather than
direct parameters if we have that much information to pass along (think
about how adding throttling parameters to a new block device was done
with a single top-level parameter pointing to a throttling sub-struct,
rather than adding lots of throttling parameters at top level).

But, as with enum sorting, actually documenting our choice will help
cement the expectations of clients on what they have to do when learning
if a parameter was added.

> 
>>> +
>>> +{ 'struct': 'SchemaInfoObject',
>>> +  'data': { 'members': [ 'SchemaInfoObjectMember' ],
>>> +            '*tag': 'str',
>>> +            '*variants': [ 'SchemaInfoObjectVariant' ] } }
>>
>> or these?
> 
> Same question.
> 

Here, if enums are sorted, then case branches within variants should be
sorted. If enums are unsorted, then I'm fine if case branches are also
unsorted (and possibly in a different order than the enum was), but be
sure to document that.

>>> +
>>> +{ 'struct': 'SchemaInfoAlternate',
>>> +  'data': { 'members': [ 'SchemaInfoObjectMember' ] } }
>>
>> Here's an example of what you generated:
>>     "{ 'name': 'BlockdevRef', 'meta-type': 'alternate', 'members': [ {
>> 'name': 'definition', 'type': 'BlockdevOptions' }, { 'name':
>> 'reference', 'type': 'str' } ] }, "
>>
>> I think you could get away with something simpler:
>>
>>  'data': { 'types': [ 'str' ] }
>>
>> as in:
>>  "{ 'name': 'BlockdevRef', 'meta-type': 'alternate', 'types': [
>> 'BlockdevOptions', 'str' ] }, "
> 
> I.e. have a list of types instead of a list of members.
> 
> Let's see what we'd lose, by enumerating the members of
> SchemaInfoObjectMember:
> 
> * name: not ABI, should not be examined (see commit message), thus no
>   loss.
> 
> * type: kept.
> 
> * default: never present (see commit message), thus no loss
> 
>> the only worry is whether we might want future extensions, where we'd
>> want additional information per element of that array, vs. being forced
>> to return two arrays in parallel (arrays of structs are more extensible
>> than arrays of strings).
>> Seems like this would be just a
> 
> Yes?
> 
> Choices:
> 
> * The only piece of information we need on an alternative right now is
>   the type, so make members a list of types.  Nice now, awkward if we
>   ever need more,
> 
> * To provide for future additions, make it a list of
>   SchemaInfoAlternateMember, where SchemaInfoAlternateMember has just
>   one member type now.
> 
> * Reuse existing SchemaInfoObjectMember, because that's close and I'm
>   lazy.
> 
> Preferences?

At this point, my vote is with a new SchemaInfoAlternateMember class
(SchemaInfoObjectMember may diverge in a different direction, and it
would eliminate the question of how to not expose the branch names as
ABI; but keeping things as a (one-member, for now) dictionary will allow
future extensions).

>>> @@ -265,7 +265,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
>>>          self.defn = None
>>>          self.regy = None
>>>          self.visited_rets = None
>>> -    def visit_begin(self):
>>> +    def visit_begin(self, schema):
>>
>> And again my python object-oriented newness is showing through; where I
>> guess all children have to update signatures to still be polymorphic to
>> a parent adding a parameter.
> 
> Maybe they need to be updated, maybe not, but updating them neatly
> sidesteps Python polymorphism questions from mere mortals like you and
> me.

see reference to [4] below

> 
>>> +''',
>>> +                          c_name=c_name(name))
>>> +        self.defn = mcgen('''
>>> +char %(c_name)s[] = "["
>>> +    "%(c_jsons)s]";
>>
>> And again. Also, I'd consider putting the "]" on its own line, like the
>> "[" was, so that you can more easily cut and paste individual lines of
>> generated output (but since JSON doesn't allow trailing comma, I guess
>> the last line is still always going to be special).
> 
> That's my reason for keepint the "]" there.
> 

[3] and that works for me, unless we want to break long lines into
shorter ones by virtue of putting "," (and thus "]") on their own line
to make it even more obvious where we are breaking between elements of
the overall array.

>>> +''',
>>> +                          c_name=c_name(name),
>>> +                          c_jsons=', "\n    "'.join(self.jsons))
>>
>> Cool syntax :)
> 
> Possibly bordering on too cool :)

Java was the first language I encountered where "".foo() was valid
syntax; sometimes, I wish C had made strings a first class type.  I'm
fine with it as it is.

>> [1] Ah, so .flat is still in use here, to avoid having to create
>> implicit types everywhere.  But if we create implicit types for simple
>> unions, and just track variants by their case name/type instead of case
>> name/[members], it will allow us to have a union as a case branch (I
>> don't know that we need that much flexibility), and not have to worry
>> about exposing .flat everywhere.  It may even result in a smaller JSON
>> string (you'd have to play with it to know for sure).
> 
> We should try hard to get the introspection schema right from the start.
> 
> But the internal representation is malleable.  I know we can implement
> non-flat unions completely in terms of flat ones (and that means no
> .flat), but I also know I failed at doing it in this series.  I'm not
> sure your idea will do the trick completely.  It's fine, we can finish
> the job later.

Yes, I've come to that conclusion myself - doesn't matter what we do on
the first round internally as long as the output is right; cleaning up
the internals can come later.


> I could shoehorn both views into a single visitor function, by passing
> both views, .base + .local_members, and .members.  All implementations
> will use only one of the views, but it's not immediately obvious which
> one.  So I chose to have two visitor functions.  Matter of taste.

I can live with it (documentation that sub-classes should override at
most only one of the two visitors might help the cause, though).

>>> +++ b/scripts/qapi.py
>>> @@ -764,7 +764,7 @@ class QAPISchemaEntity(object):
>>>          pass
>>>  
>>>  class QAPISchemaVisitor(object):
>>> -    def visit_begin(self):
>>> +    def visit_begin(self, schema):
>>
>> I don't know enough python to know if making schema optional in the
>> parent class affects what the child class is allowed to implement while
>> still overriding things.
> 
> Not sure I get you here.

It was an idle musing on whether

class QAPISchemaVisitor(object):
    def visit_begin(self, schema=None):

would permit:
class QAPIIntrospectVisitor(QAPISchemaVisitor):
    def visit_begin(self):

with proper polymorphism.  But you've already come to the conclusion
above [4] that it's easier to not mess with optional parameters (leave
that for the python gurus), and that mere mortals are better off using
something that obviously works instead of requiring knowledge about
language internals.


>>> +++ b/tests/.gitignore
>>> @@ -19,6 +19,7 @@ test-opts-visitor
>>>  test-qapi-event.[ch]
>>>  test-qapi-types.[ch]
>>>  test-qapi-visit.[ch]
>>> +test-qapi-introspect.[ch]
>>
>> [2] Ah, maybe this is the file that wasn't quite right.
> 
> Anything I need to fix?

The generated files created by 'make check' wer named
test-qmp-introspect.[ch] (either s/qapi/qmp/ here, or else fix a
Makefile rule to generate the desired name).

> 
> Thanks!  I really appreciate your review.  Must have been a big chunk of
> work.

Yes, doing a thorough review took me the better part of a week to go
through it all, so I can only imagine how much time you've invested in
it.  Hopefully v3 will be easier to review, because it will be diffs
against this version and mainly focusing on whether review comments were
addressed.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]

On 07/01/2015 02:21 PM, Markus Armbruster wrote:
> The struct generated for a flat union is weird: the members of its
> base are at the end, except for the union tag, which is renamed to
> 'kind' and put at the beginning.

The renaming to 'kind' was a bug waiting to happen.  Consider this
example, which is broken before your series:

diff --git i/tests/qapi-schema/qapi-schema-test.json
w/tests/qapi-schema/qapi-schema-test.json
index c7eaa86..12c09e3 100644
--- i/tests/qapi-schema/qapi-schema-test.json
+++ w/tests/qapi-schema/qapi-schema-test.json
@@ -37,7 +37,7 @@
   'data': { 'string1': 'str', 'string2': 'str' } }

 { 'struct': 'UserDefUnionBase',
-  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
+  'data': { 'kind': 'str', 'enum1': 'EnumOne' } }

 { 'union': 'UserDefFlatUnion',
   'base': 'UserDefUnionBase',


leading to this compilation error during 'make check-unit':

In file included from tests/test-qmp-output-visitor.c:17:0:
tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’
     char *kind;
           ^
tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’
     char *kind;
           ^

Therefore, it might be worth mentioning that avoiding the rename to
'kind' is a bug fix, not just a nicer struct :)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

On 07/28/2015 12:44 AM, Markus Armbruster wrote:

>>
>>> +def gen_visit_union(name, base, variants):
>>> +    ret = ''
>>>  
>>>      if base:
>>> -        assert discriminator
>>> -        base_fields = find_struct(base)['data'].copy()
>>> -        del base_fields[discriminator]
>>> -        ret += generate_visit_struct_fields(name, base_fields)
>>> +        members = [m for m in base.members if m != variants.tag_member]
>>> +        ret += generate_visit_struct_fields(name, members)
>>
>> Ouch. This hurts.  If the same class is used as both the base class of a
>> flat union, and the base class of an ordinary struct, then the struct
>> tries to visit the base class, but no longer parses the field that the
>> union was using as its discriminator.
>>
>> We don't have any code that demonstrates this, but probably should.  I
>> ran into it while working up my POC of what it would take to unbox
>> inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)
> 
> Is this broken in master, or do my patches break it?
> 
> Got a reproducer?

Turns out I'm mistaken; we got lucky.  The call to
generate_visit_struct_fields() creates a function for 'name' (aka the
union name), and not for 'base' (aka the class name that owns the
fields).  So even if we have Base as a common struct between Child and
Union, the code emitted for Child generates visit_Base_fields(), while
the code emitted for Union generates visit_Union_fields().

So there is no reproducer, but _only_ as long as we reject unions as a
base class for any other object.  And there is redundancy: we could
reuse visit_Base_fields() for the sake of the union, then avoid
(re-)visiting the discriminator, and we would no longer need to emit
visit_Union_fields().  But I can do that as part of the followup
cleanups; since I don't see anything broken with your patch, we don't
have to worry about it during this series.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 33/47] qapi: Clean up after recent conversions to QAPISchemaVisitor

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3531 bytes --]

On 07/28/2015 03:18 AM, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>> Drop helper functions that are now unused.
>>>
>>> Make pylint reasonably happy.
>>>
>>> Rename generate_FOO() functions to gen_FOO() for consistency.
>>>
>>> Use more consistent and sensible variable names.
>>>
>>> Consistently use c_ for mapping keys when their value is a C
>>> identifier or type.
>>>
>>> Simplify gen_enum() and gen_visit_union()
>>>
>>> Consistently use single quotes for C text string literals.
>>
>> Not sure if it is worth splitting this into pieces.  Fortunately, there
>> are no changes to generated output.
> 
> I'm afraid splitting would increase churn without much gain.  If you
> want me to split, I can try.

I can live without the split; generated output being unchanged is
already a good sign.

>>> -def generate_command_decl(name, args, ret_type):
>>> -    arglist=""
>>> +def gen_command_decl(name, args, rets):
>>
>> I can see how 'args' is plural (even if it is a single string for the
>> name of a type containing the args), but should it be 'ret' instead of
>> 'rets'?
> 
> I now realize readers may find this odd.
> 
> The QMP specification talks about command arguments and command
> responses.
> 
> The QMP wire format uses 'arguments' and 'return'.
> 
> The schema language uses 'data' and 'returns'.  Near-perfect score on
> terminology inconsistency, as usual.  Anyway, 'data' is plural (and a
> rather unhelpful choice of syntax).  'returns' could either be the
> plural of the noun "return", or the third person singular of the verb
> "to return".
> 
> Permit me a philosophical digression.  The common way to do functions in
> programming is to have multiple arguments and a single return value.  I
> believe this is mostly common machines' calling conventions leaking into
> languages.  From a more abstract point of view, there's no structural
> difference between function arguments and values: both are simply an
> element of an arbitrary set (domain and codomain, respectively).  In
> particular, both can be tuples.
> 
> It's perfectly sane to have functions take exactly one argument and
> yield exactly one value.  Some functional languages work that way.
> 
> But when both argument and value are generally tuples anyway, as they
> are in QAPI/QMP, it's more natural to talk about arguments and return
> values.  I abbreviated to args and rets.  There's method to my madness
> ;)
> 
> I'm open to better ideas on terminology.

Not sure I'm thinking of anything better; so while I found it unusual,
the explanation helps and I certainly won't reject it as wrong.

>>> -def generate_visit_struct_fields(name, members, base = None):
>>> +def gen_visit_struct_fields(name, base, members):
>>>      struct_fields_seen.add(name)
>>
>>> -                     type=base.c_name(), c_name=c_name('base'))
>>> +                     c_type=base.c_name(), c_name=c_name('base'))
>>
>> Possible churn here based on my earlier comments about c_name(constant)
>> being constant.
> 
> I'm leaning towards leaving it as is just to keep the code similar to
> other places generating visit_type_FOO() calls.

And I already easily got rid of it in my followup RFC patches, so no
problem if you leave it as is for the sake of getting this series in.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3438 bytes --]

On 07/28/2015 12:39 PM, Markus Armbruster wrote:

> 
> Could do plain integer.  I guess I started down the base32 road to
> squeeze out a few more characters, then sabotaged myself by always using
> three base32 characters.
> 

> 
> In the introspection schema, every 'str' that's really a type name needs
> to be replaced by 'int'.

Sadly, our introspection union is based on 'name':'str',
'meta-type':'...' as the base members of the union; and that 'name'
would be one of the places where we'd want an integer; so we probably
have to stick to a string.  That said, using a string-ized integer may
be a bit more legible (for some reason, humans are better at reading
decimal numbers than base32 numbers), and since no valid command or
event starts with an integer, we could use the stringized integer
directly without having to use a colon (in fact, for the first 999 types
encountered, a stringized integer is shorter than the base32 compression
+ the leading colon for namespacing) :)

> 
> Should we later decide we don't want to hide type names after all, then
> backward compatibility will make it very hard to go back.

Indeed - changing 'str' to 'int' in the schema is a much stronger
commitment than leaving things 'str' but passing a stringized integer in
that string.

> 
> I wouldn't expect clients to find stuff with a linear search.  Use a
> dictionary.  Should be plenty fast enough for processing the schema.

Requires more memory in the client to create that hash table on the one
pass through the JSON - but as is often the case in computer science,
asking the client to trade space for time is not too onerous.
(Libvirt's current JSON parser does NOT create a hash table of the
dictionaries that it reads, so libvirt would actually have to do two
passes and/or update its use of libyajl into creating hash tables when
appropriate - but that should not be a driving factor in your design.)

> 
>>                     Or if type names are truly unimportant, then omit
>> names for type elements (by making name optional in the introspection
>> qapi description), and using ONLY offsets in the returned JSON array for
>> referring to types.  Of course, if we do this, life gets a lot trickier
>> for adding filtering down to a subset of the overall schema (unless you
>> don't mind populating lots of 'null' entries for parts that get filtered
>> out so that the parts that are displayed are always at the same array
>> offset, just with less overall output bulk due to the filtering).
> 
> Filtering is a headache I'd prefer to avoid.

Well, since I've had most of the ideas about how filtering could even be
done, I'm perfectly okay with you leaving the guts of filtering for me
(or someone else) to do as a followup series; and even then, I first
have to decide if it would help libvirt to have a filtered list.  What's
important for this series is that we haven't precluded the possibility
of adding filtering later, and I think we've succeeded at that.  And as
for using raw integers to represent offsets into the returned JSON
array, I think that is a bit too brittle; so I'm happy with forcing
clients to create their own dictionary/hash lookup of string type names,
even if the strings are munged to avoid leaking qapi types as non-ABI.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2657 bytes --]

On 07/28/2015 12:24 PM, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>> To eliminate the temptation for clients to look up types by name
>>> (which are not ABI), replace all type names by meaningless strings.
>>>
>>> Reduces output of query-schema by 9 out of 80KiB.
>>
>> I'm not sure whether I like this or not.  It does make sense from the
>> perspective of forcing clients to stick to ABI queries, but makes it a
>> bit harder to navigate things except by automated scripts.
> 
> Yes.  I'm not sure it's a good idea.  If we decide to hide types this
> way, then I'd find an option to generate without type hiding useful.

As in, an optional boolean flag to the QMP command that requests whether
to get compact output with hidden names vs. full output with qapi names
exposed (but then, are we storing TWO copies of the introspection
strings)?  Or merely as in the generated qmp-introspect.c file having
strategic comments so that reading _that_ file lets you see the type
names, even if they don't get passed on to the end user?


> What the patch adds:
> 
> Move the introspection information for the non-types out of the way
> before the loop, append the information on types afterwards.  The result
> is now in jsons rather than self.jsons (see the next patch hunk).

I did notice that; pre-patch interleaved commands and types all
according to a global namespace, while post-patch sank all types to the
bottom.  Interestingly enough, if libvirt is going to query what
features a command has, it will first find the command (in the first
half of the returned array), then resolve the types used by that command
until it learns if the member is present (whether new enum value or
added dictionary member) - so there is that slight optimization that if
we guarantee that types are always output last, then libvirt doesn't
have to start generating its own hash table lookup of types until the
first type is seen, after already learning everything it needed from the
earlier command listings.

> 
> Why it does that:
> 
> With the funny typenames, sorting everything by name results in a mess.
> Keeping non-types and types separate is less of a mess.

You do have a point there - as soon as we introduce name aliases, the
aliases are unlikely to sort in the same manner as the original types,
particularly if we generate the aliases based solely on what order we
detected that those particular types were in use.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 33/47] qapi: Clean up after recent conversions to QAPISchemaVisitor

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]

On 07/28/2015 03:13 PM, Eric Blake wrote:

> 
>>>> -def generate_command_decl(name, args, ret_type):
>>>> -    arglist=""
>>>> +def gen_command_decl(name, args, rets):
>>>
>>> I can see how 'args' is plural (even if it is a single string for the
>>> name of a type containing the args), but should it be 'ret' instead of
>>> 'rets'?
>>

>> I'm open to better ideas on terminology.
> 
> Not sure I'm thinking of anything better; so while I found it unusual,
> the explanation helps and I certainly won't reject it as wrong.

Maybe gen_command_decl(name, arg_type, ret_type) since we've already
reduced the set of dictionaries into a single type name (where the type
name is possibly implicit)?  After all, we're not calling the function
with a python list or set, but with a singlar string (where that string
is the name of a type, and it is the type that then has multiple members
to form plural arguments or plural return values).  A bit more typing,
though, so up to you.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> To eliminate the temptation for clients to look up types by name
> (which are not ABI), replace all type names by meaningless strings.
> 
> Reduces output of query-schema by 9 out of 80KiB.

Among other things, it replaced all instances of 'str' with ':acg', and
all instances of 'int' with ':adu' (at least for the qapi-schema.json
files at the point in qemu.git that I tested on); if you were to tweak
things to NOT rename builtin types (limiting the renaming to just object
and array types), you'd save even more space and make libvirt's life
easier by not masking builtin types.  (Yes, the 'meta-type':'builtin'
entry for 'str' should still be present whether or not we hide the name
'str', but knowing the small set of builtin names up front and only
having to chase objects and arrays seems a bit more efficient)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>> The struct generated for a flat union is weird: the members of its
>> base are at the end, except for the union tag, which is renamed to
>> 'kind' and put at the beginning.
>
> The renaming to 'kind' was a bug waiting to happen.  Consider this
> example, which is broken before your series:
>
> diff --git i/tests/qapi-schema/qapi-schema-test.json
> w/tests/qapi-schema/qapi-schema-test.json
> index c7eaa86..12c09e3 100644
> --- i/tests/qapi-schema/qapi-schema-test.json
> +++ w/tests/qapi-schema/qapi-schema-test.json
> @@ -37,7 +37,7 @@
>    'data': { 'string1': 'str', 'string2': 'str' } }
>
>  { 'struct': 'UserDefUnionBase',
> -  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
> +  'data': { 'kind': 'str', 'enum1': 'EnumOne' } }
>
>  { 'union': 'UserDefFlatUnion',
>    'base': 'UserDefUnionBase',
>
>
> leading to this compilation error during 'make check-unit':
>
> In file included from tests/test-qmp-output-visitor.c:17:0:
> tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’
>      char *kind;
>            ^
> tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’
>      char *kind;
>            ^
>
> Therefore, it might be worth mentioning that avoiding the rename to
> 'kind' is a bug fix, not just a nicer struct :)

Cool!  I'll work (a variation of) this test case into my series.

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/28/2015 12:41 AM, Markus Armbruster wrote:
>>> Like reserving ourselves a namespace based on single _ for internal use.
>>>  We practically already have that - all user names either start with a
>>> letter or double underscore, so we could use single (and triple)
>>> underscore for internally-generated purposes, freeing up 'base',
>>> '*Kind', '*_MAX', and other namespace abuses back to the user.  Well, we
>>> may also need to reserve mid-name double-underscore (that is, the user
>>> can only supply double underscore at the beginning, but not middle, of
>>> an identifier).  Ah well, food for thought for later patches.
>> 
>> Another concern: we should take care not to generate reserved
>> identifiers.
>
> And we do that for a number of identifiers already, by renaming
> 'default' in qapi to 'q_default' in C.
>
>> 
>> * Potential issue with your proposal: identifiers that begin with an
>>   underscore and either an uppercase letter or another underscore are
>>   always reserved for any use.
>
> True, so only underscore and lower is guaranteed safe.

And we'd get that for most names.  Only names we want to shout (macros,
enumeration constants) could result in a problematic underscore and
uppercase letter.

Note that we *strip* leading underscores from enumeration constants.
Example (from qapi-schema-test.json):

    { 'enum': '__org.qemu_x-Enum', 'data': [ '__org.qemu_x-value' ] }

generates

    typedef enum __org_qemu_x_Enum
    {
        ORG_QEMU_X_ENUM___ORG_QEMU_X_VALUE = 0,
        ORG_QEMU_X_ENUM_MAX = 1,
    } __org_qemu_x_Enum;

Now add this one:

    { 'enum': 'org_qemu_x-Enum', 'data': [ ] }

Just fine according to our documentation.  But of course it clashes:

    typedef enum org_qemu_x_Enum
    {
        ORG_QEMU_X_ENUM_MAX = 0,
    } org_qemu_x_Enum;

Short term, we should note in qapi-code-gen.txt: different names in the
schema can nevertheless clash in generated C, and when that happens, you
get to pick non-clashing names.

Longer term, we may want to rethink how we map names to C.

> we can often get away with more...
>
>> 
>> * Existing issue: downstream extensions carry a __RFQDN_ prefix in the
>>   schema, which map to reserved C identifiers.
>
> ...the whole point of the __RFQDN_ prefix was that it is very unlikely
> to conflict with any vendor extensions (unless you happen to be the same
> vendor that writes the compiler and als use __RFQDN_ as part of your
> compiler implementation).  Yes, it's risky, but a risk that hasn't hurt
> any downstream clients yet :)  It's one of those things where if someone
> reports an actual problem in their environment, we'll fix it; but until
> then, it's not worth the headache of strict compliance to "fix"
> something that happens to work in spite of being undefined behavior.

I'm not religious about reserved names, but this intrusion into the
reserved name space was entirely avoidable: we could've just as well
picked a non-reserved prefix.

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/28/2015 12:44 AM, Markus Armbruster wrote:
>
>>>
>>>> +def gen_visit_union(name, base, variants):
>>>> +    ret = ''
>>>>  
>>>>      if base:
>>>> -        assert discriminator
>>>> -        base_fields = find_struct(base)['data'].copy()
>>>> -        del base_fields[discriminator]
>>>> -        ret += generate_visit_struct_fields(name, base_fields)
>>>> +        members = [m for m in base.members if m != variants.tag_member]
>>>> +        ret += generate_visit_struct_fields(name, members)
>>>
>>> Ouch. This hurts.  If the same class is used as both the base class of a
>>> flat union, and the base class of an ordinary struct, then the struct
>>> tries to visit the base class, but no longer parses the field that the
>>> union was using as its discriminator.
>>>
>>> We don't have any code that demonstrates this, but probably should.  I
>>> ran into it while working up my POC of what it would take to unbox
>>> inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)
>> 
>> Is this broken in master, or do my patches break it?
>> 
>> Got a reproducer?
>
> Turns out I'm mistaken; we got lucky.  The call to
> generate_visit_struct_fields() creates a function for 'name' (aka the
> union name), and not for 'base' (aka the class name that owns the
> fields).  So even if we have Base as a common struct between Child and
> Union, the code emitted for Child generates visit_Base_fields(), while
> the code emitted for Union generates visit_Union_fields().
>
> So there is no reproducer, but _only_ as long as we reject unions as a
> base class for any other object.  And there is redundancy: we could
> reuse visit_Base_fields() for the sake of the union, then avoid
> (re-)visiting the discriminator, and we would no longer need to emit
> visit_Union_fields().  But I can do that as part of the followup
> cleanups; since I don't see anything broken with your patch, we don't
> have to worry about it during this series.

Good.  Thank you!

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/28/2015 01:34 AM, Markus Armbruster wrote:
>> Let me rephrase to make sure I understand.
>> 
>> Ignore the (not rets) case, because retval doesn't exist then.
>> 
>> qmp_marshal_output_FOO() visits retval twice.  First, with a QMP output
>> visitor to do the actual marshalling.  Second, with a QAPI dealloc
>> visitor to destroy it.
>
> And the use of the dealloc visitor is buried inside the
> qmp_marshal_output_FOO() call.
>
>> 
>> If we execute the assignment to retval, we must go on to call
>> qmp_marshal_output_FOO(), or else we have a leak.
>> 
>> If we can reach qmp_marshal_output_FOO() without executing the
>> assignment, we must initialize retval.  If we can't, any initialization
>> is unused.
>> 
>> gen_call() generates code of the form
>> 
>>         retval = qmp_FOO(... args ..., &local_err);
>>         if (local_err) {
>>             goto out;
>>         }
>> 
>>         qmp_marshal_output_FOO(retval, ret, &local_err);
>> 
>
>> Observe:
>> 
>> 1. The assignment dominates the only use.  Therefore, the initialization
>>    is unused.  Let's drop it in a separate cleanup patch.
>> 
>> 2. We can leak retval only when qmp_FOO() returns non-null and local_err
>>    is non-null.  This must not happen, because:
>> 
>>    a. local_err must be null before the call, and
>> 
>>    b. the call must not return non-null when it sets local_err.
>
> We don't state that contract anywhere, but I doubt any of the qmp_FOO()
> functions violate it, so it is worth making it part of the contract.

It's a general Error API rule: set an error exactly on failure.  It
applies to any function returning errors through an Error **errp
parameter, and we generally don't bother to spell it out for the
individual functions.

The part that needs to be spelling out is what success and failure mean.
A qmp_FOO() returning an object returns null on failure.

>>    We could right after out: assert(!local_err || !retval).  Not sure
>>    it's worthwhile.
>
> I think it IS worthwhile, because it would catch buggy callers.  Not

We use the same assumption all over the place, without asserting it.

The boilerplate around calls using the Error API is bad enough as it is.
I'm reluctant to make it worse by adding assertions.

Less of an issue in generated code, but violations are even less likely
there.

However, I'm reluctant to assert in some places, but not all, because
inconsistency breeds confusion.

In general, I prefer a function's pre- and post-conditions to be
asserted in the function itself, not in its callers.  Asserting selected
parts of the post-condition in callers can be occasionally helpful to
help human readers and static analyzers lacking insight into the
function.

> sure if after out: is the right place (then you'd need an initializer to
> cover any other code that jumps to out), but this would do the same
>
> retval = qmp_FOO(...);
> if (local_err) {
>     assert(!retval);
>     goto out;
> }
> qmp_marshal_output_FOO(retval, ...);

Not quite the same: it doesn't catch the case !local_err && !retval.
But it doesn't matter, because that case is bound to die in
qmp_marshal_output_FOO().

>> TL;DR: I concur with your analysis.
>
> Is it worth dropping the dead initializer and adding the assert in the
> same pre-req cleanup patch?  Do you want me to submit it since I did the
> analysis?

Let's drop the useless initializer.  As explained above, I don't like
the assertion for reasons explained above, but if you want it, I'm
willing to take it anyway, in a separate follow-up patch.

I'd prefer to drop the initializer myself myself (with you credited in
the commit message), because it's certainly less total work, and quite
possibly less work for just for me.

Re: [Qemu-devel] [PATCH RFC v2 33/47] qapi: Clean up after recent conversions to QAPISchemaVisitor

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/28/2015 03:13 PM, Eric Blake wrote:
>
>> 
>>>>> -def generate_command_decl(name, args, ret_type):
>>>>> -    arglist=""
>>>>> +def gen_command_decl(name, args, rets):
>>>>
>>>> I can see how 'args' is plural (even if it is a single string for the
>>>> name of a type containing the args), but should it be 'ret' instead of
>>>> 'rets'?
>>>
>
>>> I'm open to better ideas on terminology.
>> 
>> Not sure I'm thinking of anything better; so while I found it unusual,
>> the explanation helps and I certainly won't reject it as wrong.
>
> Maybe gen_command_decl(name, arg_type, ret_type) since we've already
> reduced the set of dictionaries into a single type name (where the type
> name is possibly implicit)?  After all, we're not calling the function
> with a python list or set, but with a singlar string (where that string
> is the name of a type, and it is the type that then has multiple members
> to form plural arguments or plural return values).  A bit more typing,
> though, so up to you.

Okay, I'll try and see how these names pan out.

Re: [Qemu-devel] [PATCH RFC v2 37/47] qapi: De-duplicate parameter list generation

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/28/2015 05:15 AM, Markus Armbruster wrote:
>
>>> Yeah, avoiding line wraps consumes fewer source bytes (fewer runs of
>>> spaces), but the space isn't being wasted by storing generated files in
>>> git, nor does the C compiler care which layout we use.  And honestly,
>>> it's easier to spot changes in a vertical list than it is on a long
>>> horizontal line, if a parameter gets added (or removed, although adding
>>> is the more likely action with qapi).
>> 
>> Number of source bytes is not an issue.
>> 
>> The generators make no effort to wrap source lines, except in the
>> qapi_event_send_FOO()'s parameter lists.
>> 
>> We could preserve that one-off.  We could extend it to more places that
>> can generate long lines, saddling the generation code with indentation
>> concerns.  I don't want to write such code, and I don't want to maintain
>> it.
>> 
>> Instead, why not keep the generators straightforward, and feed their
>> result to indent when "pretty" is wanted?  Requires an indent profile
>> matching QEMU style.
>
> Long lines aren't the end of the world. They may be harder to read when
> diffing pre- and post-patch generated output to see if a generator
> change makes sense, but you have a point that line wrapping is more
> maintenance. So you win; keep the long lines, and if someone wants
> wrapping, they can (re-)add it as a later patch series.

Settled.

Aside: diff-mode can color the parts of the changed lines that actually
differ.  Indispensable, and not just for long lines.

Re: [Qemu-devel] [PATCH RFC v2 45/47] qapi: New QMP command query-schema for QMP schema introspection

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/28/2015 08:33 AM, Markus Armbruster wrote:
>> Eric Blake <eblake@redhat.com> writes:
>> 
>>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>>> Caution, rough edges.
>>>
>>> No joke. It doesn't even compile without this fixup to a rebase snafu
>>> (see [0] below):
>> 
>> Uh, how did that happen?  I compiled it a million times... (except for
>> the last time, obviously).
>
> At least you're not alone; I've done dumb things like that, too.  And at
> least it was easy to figure out, so I could test it.
>
>>>>
>>>> FIXME it can generate awfully long lines
>>>
>>> We already have long lines in generated output, but I agree that finding
>>> ways to break it up might be nice.  Actually,
>> 
>> This one's different in that the length is due to string literals.
>> indent is happy to break lines for us, but not string literals.
>> 
>> Longest line is a bit over 4KiB for me.
>> 
>
> If we break up string literals, at least use some indentation to make it
> obvious that multiple lines merge to a single array entry. For example
> (after patch 47):
>
> ...
>     "{ 'name': ':abr', 'meta-type': 'object', "
>       "'members': [ "
>         "{ 'name': 'device', 'type': ':acg', 'default': null }, "
>         "{ 'name': 'node-name', 'type': ':acg', 'default': null }, "
>         "{ 'name': 'snapshot-file', 'type': ':acg' }, "
>         "{ 'name': 'snapshot-node-name', 'type': ':acg', 'default': null
> }, "
>         "{ 'name': 'format', 'type': ':acg', 'default': null }, "
>         "{ 'name': 'mode', 'type': ':afo', 'default': null } ] }, "
>     "{ 'name': ... "

Unconventional indentation, but if it helps the reader...

> Oh, and just thought of something while typing: although patch 47 masks
> type names from the end user, it would be VERY worthwhile if the C code
> had strategic comments that pointed back to the qapi type name:
>
>     "{ 'name': ':abr', " /* TypeFoo */
>       "'meta-type': 'object'"
> ...

If we do that for uses and definitions, we don't need a flag to generate
nice typenames.  But doing it for uses might complicate the code too
much.

> (and then there's the '' vs. "" issue which impacts the output, as
> well).  And if you do the "," between entries as a separate line, that
> might impact whether you do a trailing "]" on a line by its own (see [3]
> below).
>
>>>> +
>>>> +{ 'struct': 'SchemaInfoEnum',
>>>> +  'data': { 'values': ['str'] } }
>>>
>>> Do we want to document anything about sort ordering of this list?  Is it
>>> worth sorting the array by name, to allow clients to bsearch for whether
>>> a particular enum value is supported, rather than having to linear search?
>> 
>> Haven't thought about it.  We currently emit them in definition order,
>> which is not sorted.  Largest enums:
>> 
>>     Q_KEY_CODE_MAX = 125,
>>     BLKDEBUG_EVENT_MAX = 43,
>>     BLOCKDEV_DRIVER_MAX = 28,
>>     CHARDEV_BACKEND_KIND_MAX = 19,
>>     RUN_STATE_MAX = 15,
>>     NET_CLIENT_OPTIONS_KIND_MAX = 12,
>> 
>> Most enums are small: median is 3.
>> 
>> Would libvirt prefer them sorted?
>
> Libvirt can probably live without sorting of enum constants (searching
> Q_KEY_CODE_MAX isn't going to be that noticeable of an impact on O(n)
> vs. O(logn); I predict much more time will be spent searching for type
> "xyz" referenced by command "abc" from the overall array, and repeating
> that search for multiple feature probes).
>
> But if we do want sorting, we need it up front (it will be easier to
> decide to use bsearch down the road if we are guaranteed that output is
> sorted, than it would be to try and learn whether whether we are talking
> to a new vs. old qemu to learn if sorting is in effect because it was
> added as an after-thought).

I agree adding sorting later is prone to add another instance of rarely
used, bit-rotting backward-compatibility code to libvirt, along with the
logic whether to use it.  Best avoided.

>                              And if we don't want sorting, documenting
> that data is NOT guaranteed to be position-dependent, in spite of being
> in a JSON array, is a nice touch.

What do you mean by "position-dependent"?

>>>> +
>>>> +{ 'struct': 'SchemaInfoObjectVariant',
>>>> +  'data': { 'case': 'str',
>>>> +            'members': [ 'SchemaInfoObjectMember' ] } }
>>>
>>> Would it be simpler to just have:
>>>
>>> 'data': { 'case': 'str', 'type': 'str' }
>>>
>>> and make the user refer recursively to the (possibly-implicit) type for
>>> the members?
>> 
>> Hmmm...
>> 
>> QAPISchemaObjectTypeVariant has members name, type, flat, and a few more
>> that don't matter here.
>> 
>> For a non-flat variant with name=N, type=T, my code creates
>> 
>>     { 'case': 'N', 'members': [ { 'name': 'data', 'type': 'T' } ] }
>> 
>> This means when the tag is 'N', we have a member 'data' of type 'T'.
>> 
>> For a flat variant, it creates
>> 
>>     { 'case': 'N', 'members': [ { ... the members of T ... } ] }
>> 
>> This means when the tag is 'N', we have all the members of T.
>> 
>> If I understand you correctly, you're proposing
>> 
>>     { 'case': 'N', 'type': 'T' }
>> 
>> to mean when the tag is 'N', we have all the members of T.  For the flat
>> variant above, we'd create exactly that.
>> 
>> T must not have variants, but the schema doesn't reflect that.
>
> That's our current restriction, but it's one we might decide to lift in
> the future.  Having a type with two different discriminators could get a
> bit weird to think about, but doesn't seem to be technically impossible.
>
>> 
>> For the simple variant, we'd then create
>> 
>>     { 'case': 'N', 'type': 'TT' }
>> 
>> where TT is a new implicit object type with a single member with name
>> data and type T.
>> 
>> Correct?
>
> Yes.

I'm starting to like it.

>>> In particular, if we ever decide to allow a flat union to have another
>>> union as a branch, rather than the current restriction that all branches
>>> must be structs, then referring to the type of a branch may be easier
>>> than breaking out all members of a struct.
>> 
>> Not sure I completely get you here.  Using an object type instead of a
>> member list is obviously more flexible, because for any member list we
>> can make up an object type with the same meaning, but not vice versa.
>
> Indeed, and that's the restriction that I mention we might someday want
> to lift.  Or, in (modified, due to inline {} types) qapi terms, if we
> ever want to allow:
>
> { 'union': 'Helper', 'data': { 'number': 'int', 'string': 'str' } }
> { 'enum': 'MyEnum', 'data': [ 'a', 'b' ] }
> { 'union': 'Main', 'base': { 'switch': 'MyEnum' },
>   'discriminator': 'switch', 'data': { 'a': { 'boolean': 'bool' },
>                                        'b': 'Helper' } }
> { 'command': 'foo', 'data': { 'data': 'Main' } }
>
> then that would permit QMP invocations of:
> { "execute": "foo", "arguments": { "data": {
>   "switch": "a", "boolean": true } } }
> { "execute": "foo", "arguments": { "data": {
>   "switch": "b", "type": "number", "data": 1 } } }
> { "execute": "foo", "arguments": { "data": {
>   "switch": "b", "type": "string", "data": "hello" } } }
>
> which we can express as a list of case/types for the primary variants of
> 'Main' (those types in turn refer to the secondary variants of type
> Helper), but which we cannot express as a [ 'SchemaInfoObjectMember' ]
> list, because the type of the "data" member depends on the secondary
> discriminator that is called into use on the "b" case of the primary
> discriminator.
>
> So I think we're saying the same thing, that a [
> 'SchemaInfoObjectMember' ] can always be written as a reference to an
> object type name, but not all object type names can be broken back into
> an array of SchemaInfoObjectMember (only those types that are pure
> structs without variants); and that although we currently do not allow
> sub-variants within a union, we should not get in the way of that being
> a possible future extension.
>
>> 
>>>                                             And if that's the case, it
>>> may have knock-on simplifications to your earlier patches for tracking
>>> variants. See [1] below for more thoughts...
>>>
>>> Do we want to guarantee anything about the sort ordering in this list?
>> 
>> Again, haven't thought about it.
>> 
>> Do we expect member lists to get so large binary search is called for?
>
> Probably not, since such a list would be unwieldy for both client and
> server.  We tend to add boxing and optional sub-structs rather than
> direct parameters if we have that much information to pass along (think
> about how adding throttling parameters to a new block device was done
> with a single top-level parameter pointing to a throttling sub-struct,
> rather than adding lots of throttling parameters at top level).
>
> But, as with enum sorting, actually documenting our choice will help
> cement the expectations of clients on what they have to do when learning
> if a parameter was added.

We may want to adopt a consistent rule on sorting stuff.

>>>> +
>>>> +{ 'struct': 'SchemaInfoObject',
>>>> +  'data': { 'members': [ 'SchemaInfoObjectMember' ],
>>>> +            '*tag': 'str',
>>>> +            '*variants': [ 'SchemaInfoObjectVariant' ] } }
>>>
>>> or these?
>> 
>> Same question.
>> 
>
> Here, if enums are sorted, then case branches within variants should be
> sorted. If enums are unsorted, then I'm fine if case branches are also
> unsorted (and possibly in a different order than the enum was),

Okay.

>                                                                 but be
> sure to document that.

Documentation I produce tends to err on the side of brevity, sometimes
too much.

If something is meant to produce sorted results, I document the
sortedness.  If not, I habitually say nothing.  Since nothing's said,
you better assume nothing.

Please keep calling out instances of excessive brevity :)

>>>> +
>>>> +{ 'struct': 'SchemaInfoAlternate',
>>>> +  'data': { 'members': [ 'SchemaInfoObjectMember' ] } }
>>>
>>> Here's an example of what you generated:
>>>     "{ 'name': 'BlockdevRef', 'meta-type': 'alternate', 'members': [ {
>>> 'name': 'definition', 'type': 'BlockdevOptions' }, { 'name':
>>> 'reference', 'type': 'str' } ] }, "
>>>
>>> I think you could get away with something simpler:
>>>
>>>  'data': { 'types': [ 'str' ] }
>>>
>>> as in:
>>>  "{ 'name': 'BlockdevRef', 'meta-type': 'alternate', 'types': [
>>> 'BlockdevOptions', 'str' ] }, "
>> 
>> I.e. have a list of types instead of a list of members.
>> 
>> Let's see what we'd lose, by enumerating the members of
>> SchemaInfoObjectMember:
>> 
>> * name: not ABI, should not be examined (see commit message), thus no
>>   loss.
>> 
>> * type: kept.
>> 
>> * default: never present (see commit message), thus no loss
>> 
>>> the only worry is whether we might want future extensions, where we'd
>>> want additional information per element of that array, vs. being forced
>>> to return two arrays in parallel (arrays of structs are more extensible
>>> than arrays of strings).
>>> Seems like this would be just a
>> 
>> Yes?
>> 
>> Choices:
>> 
>> * The only piece of information we need on an alternative right now is
>>   the type, so make members a list of types.  Nice now, awkward if we
>>   ever need more,
>> 
>> * To provide for future additions, make it a list of
>>   SchemaInfoAlternateMember, where SchemaInfoAlternateMember has just
>>   one member type now.
>> 
>> * Reuse existing SchemaInfoObjectMember, because that's close and I'm
>>   lazy.
>> 
>> Preferences?
>
> At this point, my vote is with a new SchemaInfoAlternateMember class
> (SchemaInfoObjectMember may diverge in a different direction, and it
> would eliminate the question of how to not expose the branch names as
> ABI; but keeping things as a (one-member, for now) dictionary will allow
> future extensions).

Since Kevin also questioned my reuse of SchemaInfoObjectMember for
alternates in his review of RFC v1, let's go with a separate
SchemaInfoAlternateMember.

>>>> @@ -265,7 +265,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
>>>>          self.defn = None
>>>>          self.regy = None
>>>>          self.visited_rets = None
>>>> -    def visit_begin(self):
>>>> +    def visit_begin(self, schema):
>>>
>>> And again my python object-oriented newness is showing through; where I
>>> guess all children have to update signatures to still be polymorphic to
>>> a parent adding a parameter.
>> 
>> Maybe they need to be updated, maybe not, but updating them neatly
>> sidesteps Python polymorphism questions from mere mortals like you and
>> me.
>
> see reference to [4] below
>
>> 
>>>> +''',
>>>> +                          c_name=c_name(name))
>>>> +        self.defn = mcgen('''
>>>> +char %(c_name)s[] = "["
>>>> +    "%(c_jsons)s]";
>>>
>>> And again. Also, I'd consider putting the "]" on its own line, like the
>>> "[" was, so that you can more easily cut and paste individual lines of
>>> generated output (but since JSON doesn't allow trailing comma, I guess
>>> the last line is still always going to be special).
>> 
>> That's my reason for keepint the "]" there.
>> 
>
> [3] and that works for me, unless we want to break long lines into
> shorter ones by virtue of putting "," (and thus "]") on their own line
> to make it even more obvious where we are breaking between elements of
> the overall array.
>
>>>> +''',
>>>> +                          c_name=c_name(name),
>>>> +                          c_jsons=', "\n    "'.join(self.jsons))
>>>
>>> Cool syntax :)
>> 
>> Possibly bordering on too cool :)
>
> Java was the first language I encountered where "".foo() was valid
> syntax; sometimes, I wish C had made strings a first class type.  I'm
> fine with it as it is.
>
>>> [1] Ah, so .flat is still in use here, to avoid having to create
>>> implicit types everywhere.  But if we create implicit types for simple
>>> unions, and just track variants by their case name/type instead of case
>>> name/[members], it will allow us to have a union as a case branch (I
>>> don't know that we need that much flexibility), and not have to worry
>>> about exposing .flat everywhere.  It may even result in a smaller JSON
>>> string (you'd have to play with it to know for sure).
>> 
>> We should try hard to get the introspection schema right from the start.
>> 
>> But the internal representation is malleable.  I know we can implement
>> non-flat unions completely in terms of flat ones (and that means no
>> .flat), but I also know I failed at doing it in this series.  I'm not
>> sure your idea will do the trick completely.  It's fine, we can finish
>> the job later.
>
> Yes, I've come to that conclusion myself - doesn't matter what we do on
> the first round internally as long as the output is right; cleaning up
> the internals can come later.
>
>
>> I could shoehorn both views into a single visitor function, by passing
>> both views, .base + .local_members, and .members.  All implementations
>> will use only one of the views, but it's not immediately obvious which
>> one.  So I chose to have two visitor functions.  Matter of taste.
>
> I can live with it (documentation that sub-classes should override at
> most only one of the two visitors might help the cause, though).

Actually, overriding both would be just fine.  We just haven't had a use
for that.

>>>> +++ b/scripts/qapi.py
>>>> @@ -764,7 +764,7 @@ class QAPISchemaEntity(object):
>>>>          pass
>>>>  
>>>>  class QAPISchemaVisitor(object):
>>>> -    def visit_begin(self):
>>>> +    def visit_begin(self, schema):
>>>
>>> I don't know enough python to know if making schema optional in the
>>> parent class affects what the child class is allowed to implement while
>>> still overriding things.
>> 
>> Not sure I get you here.
>
> It was an idle musing on whether
>
> class QAPISchemaVisitor(object):
>     def visit_begin(self, schema=None):
>
> would permit:
> class QAPIIntrospectVisitor(QAPISchemaVisitor):
>     def visit_begin(self):
>
> with proper polymorphism.  But you've already come to the conclusion
> above [4] that it's easier to not mess with optional parameters (leave
> that for the python gurus), and that mere mortals are better off using
> something that obviously works instead of requiring knowledge about
> language internals.
>
>
>>>> +++ b/tests/.gitignore
>>>> @@ -19,6 +19,7 @@ test-opts-visitor
>>>>  test-qapi-event.[ch]
>>>>  test-qapi-types.[ch]
>>>>  test-qapi-visit.[ch]
>>>> +test-qapi-introspect.[ch]
>>>
>>> [2] Ah, maybe this is the file that wasn't quite right.
>> 
>> Anything I need to fix?
>
> The generated files created by 'make check' wer named
> test-qmp-introspect.[ch] (either s/qapi/qmp/ here, or else fix a
> Makefile rule to generate the desired name).

Scales falling from my eyes...

>> Thanks!  I really appreciate your review.  Must have been a big chunk of
>> work.
>
> Yes, doing a thorough review took me the better part of a week to go
> through it all, so I can only imagine how much time you've invested in
> it.  Hopefully v3 will be easier to review, because it will be diffs
> against this version and mainly focusing on whether review comments were
> addressed.

I'll do my best to keep the diffs in check.

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/28/2015 12:39 PM, Markus Armbruster wrote:
>
>> 
>> Could do plain integer.  I guess I started down the base32 road to
>> squeeze out a few more characters, then sabotaged myself by always using
>> three base32 characters.
>> 
>
>> 
>> In the introspection schema, every 'str' that's really a type name needs
>> to be replaced by 'int'.
>
> Sadly, our introspection union is based on 'name':'str',
> 'meta-type':'...' as the base members of the union; and that 'name'
> would be one of the places where we'd want an integer; so we probably
> have to stick to a string.

We could use an alternate.  Feels like gratuitous complexity to me,
though.

>                             That said, using a string-ized integer may
> be a bit more legible (for some reason, humans are better at reading
> decimal numbers than base32 numbers), and since no valid command or
> event starts with an integer, we could use the stringized integer
> directly without having to use a colon (in fact, for the first 999 types
> encountered, a stringized integer is shorter than the base32 compression
> + the leading colon for namespacing) :)

I'll try and see how that pans out.

>> Should we later decide we don't want to hide type names after all, then
>> backward compatibility will make it very hard to go back.
>
> Indeed - changing 'str' to 'int' in the schema is a much stronger
> commitment than leaving things 'str' but passing a stringized integer in
> that string.
>
>> 
>> I wouldn't expect clients to find stuff with a linear search.  Use a
>> dictionary.  Should be plenty fast enough for processing the schema.
>
> Requires more memory in the client to create that hash table on the one
> pass through the JSON - but as is often the case in computer science,
> asking the client to trade space for time is not too onerous.
> (Libvirt's current JSON parser does NOT create a hash table of the
> dictionaries that it reads, so libvirt would actually have to do two
> passes and/or update its use of libyajl into creating hash tables when
> appropriate - but that should not be a driving factor in your design.)

For completeness: dictionary needn't be implemented as hash table.

>>>                     Or if type names are truly unimportant, then omit
>>> names for type elements (by making name optional in the introspection
>>> qapi description), and using ONLY offsets in the returned JSON array for
>>> referring to types.  Of course, if we do this, life gets a lot trickier
>>> for adding filtering down to a subset of the overall schema (unless you
>>> don't mind populating lots of 'null' entries for parts that get filtered
>>> out so that the parts that are displayed are always at the same array
>>> offset, just with less overall output bulk due to the filtering).
>> 
>> Filtering is a headache I'd prefer to avoid.
>
> Well, since I've had most of the ideas about how filtering could even be
> done, I'm perfectly okay with you leaving the guts of filtering for me
> (or someone else) to do as a followup series; and even then, I first
> have to decide if it would help libvirt to have a filtered list.  What's
> important for this series is that we haven't precluded the possibility
> of adding filtering later, and I think we've succeeded at that.  And as
> for using raw integers to represent offsets into the returned JSON
> array, I think that is a bit too brittle; so I'm happy with forcing
> clients to create their own dictionary/hash lookup of string type names,
> even if the strings are munged to avoid leaking qapi types as non-ABI.

Okay.  Thanks!

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/28/2015 12:24 PM, Markus Armbruster wrote:
>> Eric Blake <eblake@redhat.com> writes:
>> 
>>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>>> To eliminate the temptation for clients to look up types by name
>>>> (which are not ABI), replace all type names by meaningless strings.
>>>>
>>>> Reduces output of query-schema by 9 out of 80KiB.
>>>
>>> I'm not sure whether I like this or not.  It does make sense from the
>>> perspective of forcing clients to stick to ABI queries, but makes it a
>>> bit harder to navigate things except by automated scripts.
>> 
>> Yes.  I'm not sure it's a good idea.  If we decide to hide types this
>> way, then I'd find an option to generate without type hiding useful.
>
> As in, an optional boolean flag to the QMP command that requests whether
> to get compact output with hidden names vs. full output with qapi names
> exposed (but then, are we storing TWO copies of the introspection
> strings)?  Or merely as in the generated qmp-introspect.c file having
> strategic comments so that reading _that_ file lets you see the type
> names, even if they don't get passed on to the end user?

I was thinking of optionally unhidden type names, for when you messed up
the output, and the stupid hidden type names make it hard to see what
exactly's wrong.  I.e. the option is merely a development aid.

>> What the patch adds:
>> 
>> Move the introspection information for the non-types out of the way
>> before the loop, append the information on types afterwards.  The result
>> is now in jsons rather than self.jsons (see the next patch hunk).
>
> I did notice that; pre-patch interleaved commands and types all
> according to a global namespace, while post-patch sank all types to the
> bottom.  Interestingly enough, if libvirt is going to query what
> features a command has, it will first find the command (in the first
> half of the returned array), then resolve the types used by that command
> until it learns if the member is present (whether new enum value or
> added dictionary member) - so there is that slight optimization that if
> we guarantee that types are always output last, then libvirt doesn't
> have to start generating its own hash table lookup of types until the
> first type is seen, after already learning everything it needed from the
> earlier command listings.

I'm reluctant to complicate the contract with such a promise, though.

I guess I'd simply slurp in the JSON, then do a quick pass over the
resulting data structure to build an index.

>> Why it does that:
>> 
>> With the funny typenames, sorting everything by name results in a mess.
>> Keeping non-types and types separate is less of a mess.
>
> You do have a point there - as soon as we introduce name aliases, the
> aliases are unlikely to sort in the same manner as the original types,

s/are unlikely to/won't/ :)

> particularly if we generate the aliases based solely on what order we
> detected that those particular types were in use.

That's the easy order.  Other orders are possible, but they're extra
work.  Best to keep things simple.

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> To eliminate the temptation for clients to look up types by name
>> (which are not ABI), replace all type names by meaningless strings.
>> 
>> Reduces output of query-schema by 9 out of 80KiB.
>
> Among other things, it replaced all instances of 'str' with ':acg', and
> all instances of 'int' with ':adu' (at least for the qapi-schema.json
> files at the point in qemu.git that I tested on); if you were to tweak
> things to NOT rename builtin types (limiting the renaming to just object
> and array types), you'd save even more space and make libvirt's life
> easier by not masking builtin types.  (Yes, the 'meta-type':'builtin'
> entry for 'str' should still be present whether or not we hide the name
> 'str', but knowing the small set of builtin names up front and only
> having to chase objects and arrays seems a bit more efficient)

You're right, hiding builtin types is pointless, they're ABI.

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

On 07/29/2015 02:32 AM, Markus Armbruster wrote:

>>> 2. We can leak retval only when qmp_FOO() returns non-null and local_err
>>>    is non-null.  This must not happen, because:
>>>
>>>    a. local_err must be null before the call, and
>>>
>>>    b. the call must not return non-null when it sets local_err.
>>
>> We don't state that contract anywhere, but I doubt any of the qmp_FOO()
>> functions violate it, so it is worth making it part of the contract.
> 
> It's a general Error API rule: set an error exactly on failure.  It
> applies to any function returning errors through an Error **errp
> parameter, and we generally don't bother to spell it out for the
> individual functions.
> 
> The part that needs to be spelling out is what success and failure mean.
> A qmp_FOO() returning an object returns null on failure.
> 
>>>    We could right after out: assert(!local_err || !retval).  Not sure
>>>    it's worthwhile.
>>
>> I think it IS worthwhile, because it would catch buggy callers.  Not
> 
> We use the same assumption all over the place, without asserting it.

Okay, you've got a point there.

> 
> Let's drop the useless initializer.  As explained above, I don't like
> the assertion for reasons explained above, but if you want it, I'm
> willing to take it anyway, in a separate follow-up patch.
> 
> I'd prefer to drop the initializer myself myself (with you credited in
> the commit message), because it's certainly less total work, and quite
> possibly less work for just for me.

No need to add assertions.  Maybe worth a patch to add a comment
somewhere, maybe as a new section in docs/qapi-code-gen.txt, documenting
how to write a handler and hook it into qmp-commands.hx, and what
conditions the handler must obey.  And I'm fine with you dropping the
initializer as part of your v3 series.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 45/47] qapi: New QMP command query-schema for QMP schema introspection

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3425 bytes --]

On 07/29/2015 03:19 AM, Markus Armbruster wrote:
>>> Longest line is a bit over 4KiB for me.
>>>
>>
>> If we break up string literals, at least use some indentation to make it
>> obvious that multiple lines merge to a single array entry. For example
>> (after patch 47):
>>
>> ...
>>     "{ 'name': ':abr', 'meta-type': 'object', "
>>       "'members': [ "
>>         "{ 'name': 'device', 'type': ':acg', 'default': null }, "
>>         "{ 'name': 'node-name', 'type': ':acg', 'default': null }, "
>>         "{ 'name': 'snapshot-file', 'type': ':acg' }, "
>>         "{ 'name': 'snapshot-node-name', 'type': ':acg', 'default': null
>> }, "
>>         "{ 'name': 'format', 'type': ':acg', 'default': null }, "
>>         "{ 'name': 'mode', 'type': ':afo', 'default': null } ] }, "
>>     "{ 'name': ... "
> 
> Unconventional indentation, but if it helps the reader...

I'm not a stickler about the particular spacing I used, so much as
demonstrating an idea.  Pick any indentation you like; I was just
demonstrating that some well-chosen line breaks, coupled with visual
clues on what belongs together, can help in reading the string literal
in the generated file.

In fact, doesn't python have a way to pretty-print JSON, and then
post-process the pretty-printed string to add C \" escaping?

>>                              And if we don't want sorting, documenting
>> that data is NOT guaranteed to be position-dependent, in spite of being
>> in a JSON array, is a nice touch.
> 
> What do you mean by "position-dependent"?

If qemu 2.5 has { 'struct':'Foo', 'data': { 'b': 'int', 'c': 'int' } },
then qemu 2.6 adds '*a': 'int' to the end of that list, then either we
guarantee sorting (if you read the members and see 'b' first, then you
know 'a' was not added) or we don't (you must read the entire list to
see if 'a' has been added; and you cannot assume that 'a' will be last
even if it was listed last in the .json file of 2.6); the position in
the .json file need not determine the position in the introspection output.

>> But, as with enum sorting, actually documenting our choice will help
>> cement the expectations of clients on what they have to do when learning
>> if a parameter was added.
> 
> We may want to adopt a consistent rule on sorting stuff.

To be consistent, the rule would be that either everything is sorted, or
nothing is; and if we choose nothing to be sorted, we are unlikely to
ever want to add sorting in the future.

>>> I could shoehorn both views into a single visitor function, by passing
>>> both views, .base + .local_members, and .members.  All implementations
>>> will use only one of the views, but it's not immediately obvious which
>>> one.  So I chose to have two visitor functions.  Matter of taste.
>>
>> I can live with it (documentation that sub-classes should override at
>> most only one of the two visitors might help the cause, though).
> 
> Actually, overriding both would be just fine.  We just haven't had a use
> for that.

I guess it's hard to predict why a visitor would want to have both
callbacks called for every object, so we can't outright state that it is
useless. The whole point of a visitor interface is to provide
flexibility in designing a visitor later down the road :)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 47/47] qapi-introspect: Hide type names

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3436 bytes --]

On 07/29/2015 03:34 AM, Markus Armbruster wrote:

>>>> I'm not sure whether I like this or not.  It does make sense from the
>>>> perspective of forcing clients to stick to ABI queries, but makes it a
>>>> bit harder to navigate things except by automated scripts.
>>>
>>> Yes.  I'm not sure it's a good idea.  If we decide to hide types this
>>> way, then I'd find an option to generate without type hiding useful.
>>
>> As in, an optional boolean flag to the QMP command that requests whether
>> to get compact output with hidden names vs. full output with qapi names
>> exposed (but then, are we storing TWO copies of the introspection
>> strings)?  Or merely as in the generated qmp-introspect.c file having
>> strategic comments so that reading _that_ file lets you see the type
>> names, even if they don't get passed on to the end user?
> 
> I was thinking of optionally unhidden type names, for when you messed up
> the output, and the stupid hidden type names make it hard to see what
> exactly's wrong.  I.e. the option is merely a development aid.

Ah, an option to qapi-introspect.py that says to generate the file
differently, but which is not normally enabled and therefore not exposed
to the QMP end user.  That, or unconditional strategic comments, both do
the trick for me.

> 
>>> What the patch adds:
>>>
>>> Move the introspection information for the non-types out of the way
>>> before the loop, append the information on types afterwards.  The result
>>> is now in jsons rather than self.jsons (see the next patch hunk).
>>
>> I did notice that; pre-patch interleaved commands and types all
>> according to a global namespace, while post-patch sank all types to the
>> bottom.  Interestingly enough, if libvirt is going to query what
>> features a command has, it will first find the command (in the first
>> half of the returned array), then resolve the types used by that command
>> until it learns if the member is present (whether new enum value or
>> added dictionary member) - so there is that slight optimization that if
>> we guarantee that types are always output last, then libvirt doesn't
>> have to start generating its own hash table lookup of types until the
>> first type is seen, after already learning everything it needed from the
>> earlier command listings.
> 
> I'm reluctant to complicate the contract with such a promise, though.

Concur. If we aren't going to guarantee sorting, then we also should not
guarantee that all commands come before any types, but that the overall
output is just an entire lookup dictionary where any entry can hold any
metatype.

> 
> I guess I'd simply slurp in the JSON, then do a quick pass over the
> resulting data structure to build an index.

Yep, that's probably what I'll have to code into libvirt.

> 
> That's the easy order.  Other orders are possible, but they're extra
> work.  Best to keep things simple.

Simplest for qemu is no order at all (and the client has to take on the
burden of sorting things as desired).  Not always the smartest (if every
client wants to sort, we've pushed a lot of duplicated work onto each
client instead of doing the work ourselves), but we have right up until
2.5 is released to commit to our decision of whether to sort in qemu or not.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1529 bytes --]

On 07/29/2015 02:00 AM, Markus Armbruster wrote:
>>>> We don't have any code that demonstrates this, but probably should.  I
>>>> ran into it while working up my POC of what it would take to unbox
>>>> inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)
>>>
>>> Is this broken in master, or do my patches break it?
>>>
>>> Got a reproducer?
>>
>> Turns out I'm mistaken; we got lucky.  The call to
>> generate_visit_struct_fields() creates a function for 'name' (aka the
>> union name), and not for 'base' (aka the class name that owns the
>> fields).  So even if we have Base as a common struct between Child and
>> Union, the code emitted for Child generates visit_Base_fields(), while
>> the code emitted for Union generates visit_Union_fields().
>>
>> So there is no reproducer, but _only_ as long as we reject unions as a
>> base class for any other object.  And there is redundancy: we could
>> reuse visit_Base_fields() for the sake of the union, then avoid
>> (re-)visiting the discriminator, and we would no longer need to emit
>> visit_Union_fields().  But I can do that as part of the followup
>> cleanups; since I don't see anything broken with your patch, we don't
>> have to worry about it during this series.
> 
> Good.  Thank you!

And here's a followup patch that cleans it up:
http://thread.gmane.org/gmane.comp.emulators.qemu/353204/focus=353728

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/29/2015 02:32 AM, Markus Armbruster wrote:
>
>>>> 2. We can leak retval only when qmp_FOO() returns non-null and local_err
>>>>    is non-null.  This must not happen, because:
>>>>
>>>>    a. local_err must be null before the call, and
>>>>
>>>>    b. the call must not return non-null when it sets local_err.
>>>
>>> We don't state that contract anywhere, but I doubt any of the qmp_FOO()
>>> functions violate it, so it is worth making it part of the contract.
>> 
>> It's a general Error API rule: set an error exactly on failure.  It
>> applies to any function returning errors through an Error **errp
>> parameter, and we generally don't bother to spell it out for the
>> individual functions.
>> 
>> The part that needs to be spelling out is what success and failure mean.
>> A qmp_FOO() returning an object returns null on failure.
>> 
>>>>    We could right after out: assert(!local_err || !retval).  Not sure
>>>>    it's worthwhile.
>>>
>>> I think it IS worthwhile, because it would catch buggy callers.  Not
>> 
>> We use the same assumption all over the place, without asserting it.
>
> Okay, you've got a point there.
>
>> 
>> Let's drop the useless initializer.  As explained above, I don't like
>> the assertion for reasons explained above, but if you want it, I'm
>> willing to take it anyway, in a separate follow-up patch.
>> 
>> I'd prefer to drop the initializer myself myself (with you credited in
>> the commit message), because it's certainly less total work, and quite
>> possibly less work for just for me.
>
> No need to add assertions.  Maybe worth a patch to add a comment
> somewhere, maybe as a new section in docs/qapi-code-gen.txt, documenting
> how to write a handler and hook it into qmp-commands.hx, and what
> conditions the handler must obey.  And I'm fine with you dropping the
> initializer as part of your v3 series.

Sounds like a plan.  Thanks!

Re: [Qemu-devel] [PATCH RFC v2 45/47] qapi: New QMP command query-schema for QMP schema introspection

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/29/2015 03:19 AM, Markus Armbruster wrote:
>>>> Longest line is a bit over 4KiB for me.
>>>>
>>>
>>> If we break up string literals, at least use some indentation to make it
>>> obvious that multiple lines merge to a single array entry. For example
>>> (after patch 47):
>>>
>>> ...
>>>     "{ 'name': ':abr', 'meta-type': 'object', "
>>>       "'members': [ "
>>>         "{ 'name': 'device', 'type': ':acg', 'default': null }, "
>>>         "{ 'name': 'node-name', 'type': ':acg', 'default': null }, "
>>>         "{ 'name': 'snapshot-file', 'type': ':acg' }, "
>>>         "{ 'name': 'snapshot-node-name', 'type': ':acg', 'default': null
>>> }, "
>>>         "{ 'name': 'format', 'type': ':acg', 'default': null }, "
>>>         "{ 'name': 'mode', 'type': ':afo', 'default': null } ] }, "
>>>     "{ 'name': ... "
>> 
>> Unconventional indentation, but if it helps the reader...
>
> I'm not a stickler about the particular spacing I used, so much as
> demonstrating an idea.  Pick any indentation you like; I was just
> demonstrating that some well-chosen line breaks, coupled with visual
> clues on what belongs together, can help in reading the string literal
> in the generated file.
>
> In fact, doesn't python have a way to pretty-print JSON, and then
> post-process the pretty-printed string to add C \" escaping?

Interesting idea, definitely worth a doc search.

Prettier output can of course be punted to a followup-patch.

>>>                              And if we don't want sorting, documenting
>>> that data is NOT guaranteed to be position-dependent, in spite of being
>>> in a JSON array, is a nice touch.
>> 
>> What do you mean by "position-dependent"?
>
> If qemu 2.5 has { 'struct':'Foo', 'data': { 'b': 'int', 'c': 'int' } },
> then qemu 2.6 adds '*a': 'int' to the end of that list, then either we
> guarantee sorting (if you read the members and see 'b' first, then you
> know 'a' was not added) or we don't (you must read the entire list to
> see if 'a' has been added; and you cannot assume that 'a' will be last
> even if it was listed last in the .json file of 2.6); the position in
> the .json file need not determine the position in the introspection output.

Got it.

>>> But, as with enum sorting, actually documenting our choice will help
>>> cement the expectations of clients on what they have to do when learning
>>> if a parameter was added.
>> 
>> We may want to adopt a consistent rule on sorting stuff.
>
> To be consistent, the rule would be that either everything is sorted, or
> nothing is; and if we choose nothing to be sorted, we are unlikely to
> ever want to add sorting in the future.

Agree.

If sorting everything turns out to be cheap (in complexity; other
metrics don't matter much), I'm inclined to sort.

>>>> I could shoehorn both views into a single visitor function, by passing
>>>> both views, .base + .local_members, and .members.  All implementations
>>>> will use only one of the views, but it's not immediately obvious which
>>>> one.  So I chose to have two visitor functions.  Matter of taste.
>>>
>>> I can live with it (documentation that sub-classes should override at
>>> most only one of the two visitors might help the cause, though).
>> 
>> Actually, overriding both would be just fine.  We just haven't had a use
>> for that.
>
> I guess it's hard to predict why a visitor would want to have both
> callbacks called for every object, so we can't outright state that it is
> useless. The whole point of a visitor interface is to provide
> flexibility in designing a visitor later down the road :)

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]

On 07/29/2015 01:33 AM, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
>> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>>> The struct generated for a flat union is weird: the members of its
>>> base are at the end, except for the union tag, which is renamed to
>>> 'kind' and put at the beginning.
>>

>> Therefore, it might be worth mentioning that avoiding the rename to
>> 'kind' is a bug fix, not just a nicer struct :)
> 
> Cool!  I'll work (a variation of) this test case into my series.

Another name collision bug: our code generates flat unions as:

struct BlockdevOptions {
    BlockdevDriver driver;
...
    /* End fields inherited from BlockdevOptionsBase. */
    /* union tag is BlockdevDriver driver */
    union {
        void *data;
        BlockdevOptionsArchipelago *archipelago;
...

which means that if we name any of the branches 'data' (that is, if
'data' is a member of the enum discriminator), things fail to compile.
We could probably fix that by naming our dummy branch '_data'.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 5048 bytes --]

On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> Fixes flat unions to get the base's base members.  Test case is from
> commit 2fc0043, in qapi-schema-test.json:
> 

> -def generate_alternate_qtypes(expr):
> +def gen_alternate_qtypes_decl(name):
> +    return mcgen('''
>  
> -    name = expr['alternate']
> -    members = expr['data']
> +extern const int %(c_name)s_qtypes[];
> +''',
> +                 c_name=c_name(name))
>  
> +def gen_alternate_qtypes(name, variants):
>      ret = mcgen('''
>  
>  const int %(name)s_qtypes[QTYPE_MAX] = {
>  ''',
>                  name=c_name(name))
>  
> -    for key in members:
> -        qtype = find_alternate_member_qtype(members[key])
> -        assert qtype, "Invalid alternate member"
> +    for var in variants.variants:
> +        qtype = var.type.alternate_qtype()
> +        assert qtype

I think I found a couple more corner case bugs here. We are using C99
initialization of the array; for example:

const int BlockdevRef_qtypes[QTYPE_MAX] = {
    [QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION,
    [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE,
};

but paired with an enum that starts at 0:

typedef enum BlockdevRefKind {
    BLOCKDEV_REF_KIND_DEFINITION = 0,
    BLOCKDEV_REF_KIND_REFERENCE = 1,
    BLOCKDEV_REF_KIND_MAX = 2,
} BlockdevRefKind;


and that means that every QTYPE_ constant that we don't specify in
_qtypes[] is also assigned the value 0 (aka BLOCKDEV_REF_KIND_DEFINITION
in this example).  In operation, calling something like:

{"execute":"blockdev-add","arguments":{"options":
 {"driver":"raw","id":"a","file":true}}}

which is invalid per the .json description ("file" must be string or
object, not boolean), still manages to get past visit_get_next_type()
with success, and fall through to the 0th branch of the switch.  If that
0th branch happens to be a struct (as it is for BlockdevRef), then we
fortunately catch the error on the very next parse call, where
qmp_input_start_struct() complains:

{"error": {"class": "GenericError", "desc": "Invalid parameter type for
'file', expected: QDict"}}

But what happens if the 0th branch is mapped to a different parser, as
would be the case if one of the alternate's branches is 'number'?  In
particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
 So, if we have this qapi:
 { 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer, visit_get_next_type() will see a qtype of QInt,
but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and
we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the
string parser doesn't like ints) even though the parse should succeed by
using the FOO_KIND_B branch.

Interestingly, this means that if we ever write an alternate type that
accepts both 'int' and 'number' (we have not attempted that so far),
then the number branch will only be taken for inputs that don't also
look like ints (normally, 'number' accepts anything numeric). Maybe that
means we should document and enforce that 'number' and 'int' cannot be
mixed in the same alternate?

So, the bugs are: visit_get_next_type() can't tell the difference
between a *_qtypes[] lookup that was explicitly initialized to 0 from
one that was accidentally left that way, and therefore can't report
failure for an unexpected type (but mostly mitigated by the fact that
always returning 0 means the parser will attempt to parse the first
branch of the alternate and gracefully fail); and that we don't properly
handle QInt for an alternate that accepts 'number' but not 'int'.

I don't think either bug has to be fixed in your series, although you
may want to add tests.

The first bug could be resolved by guaranteeing that the _qtypes[] array
has non-zero values for the explicitly initialized lookups, and teaching
visit_get_next_type() that a lookup that produces 0 meant that an
unexpected type was encountered.  Perhaps by changing the creation of
_qtypes[] in qapi-types.c to list:

const int BlockdevRef_qtypes[QTYPE_MAX] = {
  [QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION + 1,
  [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE + 1,
};

and then having visit_get_next_type() subtract one after verifying a
non-zero value was looked up.  Or perhaps leave _qtypes alone, and
instead change the alternate enum to have a placeholder at 0:

typedef enum BlockdevRefKind {
    BLOCKDEV_REF_KIND_INVALID = 0,
    BLOCKDEV_REF_KIND_DEFINITION = 1,
    BLOCKDEV_REF_KIND_REFERENCE = 2,
    BLOCKDEV_REF_KIND_MAX = 3,
} BlockdevRefKind;

and then teaching the generator for visit_type_BlockdevRef() to emit an
error if branch 0 is hit.

Fixing the second bug probably entails teaching the generator that if an
alternate contains 'number' but not 'int', then we need [QTYPE_QINT] to
map to the same lookup value as [QTYPE_QNUMBER].

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> Fixes flat unions to get the base's base members.  Test case is from
>> commit 2fc0043, in qapi-schema-test.json:
>> 
>
>> -def generate_alternate_qtypes(expr):
>> +def gen_alternate_qtypes_decl(name):
>> +    return mcgen('''
>>  
>> -    name = expr['alternate']
>> -    members = expr['data']
>> +extern const int %(c_name)s_qtypes[];
>> +''',
>> +                 c_name=c_name(name))
>>  
>> +def gen_alternate_qtypes(name, variants):
>>      ret = mcgen('''
>>  
>>  const int %(name)s_qtypes[QTYPE_MAX] = {
>>  ''',
>>                  name=c_name(name))
>>  
>> -    for key in members:
>> -        qtype = find_alternate_member_qtype(members[key])
>> -        assert qtype, "Invalid alternate member"
>> +    for var in variants.variants:
>> +        qtype = var.type.alternate_qtype()
>> +        assert qtype
>
> I think I found a couple more corner case bugs here. We are using C99
> initialization of the array; for example:
>
> const int BlockdevRef_qtypes[QTYPE_MAX] = {
>     [QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION,
>     [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE,
> };
>
> but paired with an enum that starts at 0:
>
> typedef enum BlockdevRefKind {
>     BLOCKDEV_REF_KIND_DEFINITION = 0,
>     BLOCKDEV_REF_KIND_REFERENCE = 1,
>     BLOCKDEV_REF_KIND_MAX = 2,
> } BlockdevRefKind;
>
>
> and that means that every QTYPE_ constant that we don't specify in
> _qtypes[] is also assigned the value 0 (aka BLOCKDEV_REF_KIND_DEFINITION

I see where this is going...

> in this example).  In operation, calling something like:
>
> {"execute":"blockdev-add","arguments":{"options":
>  {"driver":"raw","id":"a","file":true}}}
>
> which is invalid per the .json description ("file" must be string or
> object, not boolean), still manages to get past visit_get_next_type()
> with success, and fall through to the 0th branch of the switch.  If that
> 0th branch happens to be a struct (as it is for BlockdevRef), then we
> fortunately catch the error on the very next parse call, where
> qmp_input_start_struct() complains:
>
> {"error": {"class": "GenericError", "desc": "Invalid parameter type for
> 'file', expected: QDict"}}

A lucky case.

> But what happens if the 0th branch is mapped to a different parser, as
> would be the case if one of the alternate's branches is 'number'?  In
> particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
>  So, if we have this qapi:
>  { 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
> but pass in an integer, visit_get_next_type() will see a qtype of QInt,
> but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and
> we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the
> string parser doesn't like ints) even though the parse should succeed by
> using the FOO_KIND_B branch.

Yup, bug.

> Interestingly, this means that if we ever write an alternate type that
> accepts both 'int' and 'number' (we have not attempted that so far),
> then the number branch will only be taken for inputs that don't also
> look like ints (normally, 'number' accepts anything numeric). Maybe that
> means we should document and enforce that 'number' and 'int' cannot be
> mixed in the same alternate?

Even if we outlaw mixing the two, I'm afraid we still have this bug: an
alternate with a 'number' member rejects input that gets parsed as
QTYPE_QINT.

Let's simply make alternates behave sanely:

    alternate has      case selected for
    'int'  'number'    QTYPE_QINT  QTYPE_QFLOAT
      no        no     error       error
      no       yes     'number'    'number'
     yes        no     'int'       error
     yes       yes     'int'       'number'

> So, the bugs are: visit_get_next_type() can't tell the difference
> between a *_qtypes[] lookup that was explicitly initialized to 0 from
> one that was accidentally left that way, and therefore can't report
> failure for an unexpected type (but mostly mitigated by the fact that
> always returning 0 means the parser will attempt to parse the first
> branch of the alternate and gracefully fail);

Yes.

>                                               and that we don't properly
> handle QInt for an alternate that accepts 'number' but not 'int'.

Yes.

> I don't think either bug has to be fixed in your series, although you
> may want to add tests.

Agree.

> The first bug could be resolved by guaranteeing that the _qtypes[] array
> has non-zero values for the explicitly initialized lookups, and teaching
> visit_get_next_type() that a lookup that produces 0 meant that an
> unexpected type was encountered.  Perhaps by changing the creation of
> _qtypes[] in qapi-types.c to list:
>
> const int BlockdevRef_qtypes[QTYPE_MAX] = {
>   [QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION + 1,
>   [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE + 1,
> };
>
> and then having visit_get_next_type() subtract one after verifying a
> non-zero value was looked up.

+ 1 works, because the element type is int, not BlockdevRefKind.  It's
int so it can serve as argument for visit_get_next_type()'s parameter
const int *qtypes.

The + 1, - 1 business could be mildly confusing.  We could set all
unused elements to -1 instead:

    const int BlockdevRef_qtypes[QTYPE_MAX] = {
        [QTYPE_NONE] = -1,
        [QTYPE_QNULL] = -1,
        [QTYPE_QINT] = -1,
        [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE + 1,
        [QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION + 1,
        [QTYPE_QLIST] = -1,
        [QTYPE_QFLOAT] = -1,
        [QTYPE_QBOOL] = -1,
    };

I wouldn't recommend that in hand-written code, but generating it should
be fine.

>                                Or perhaps leave _qtypes alone, and
> instead change the alternate enum to have a placeholder at 0:
>
> typedef enum BlockdevRefKind {
>     BLOCKDEV_REF_KIND_INVALID = 0,
>     BLOCKDEV_REF_KIND_DEFINITION = 1,
>     BLOCKDEV_REF_KIND_REFERENCE = 2,
>     BLOCKDEV_REF_KIND_MAX = 3,
> } BlockdevRefKind;
>
> and then teaching the generator for visit_type_BlockdevRef() to emit an
> error if branch 0 is hit.

BLOCKDEV_REF_KIND_INVALID could get in the way elsewhere, e.g. with
-Wswitch.

> Fixing the second bug probably entails teaching the generator that if an
> alternate contains 'number' but not 'int', then we need [QTYPE_QINT] to
> map to the same lookup value as [QTYPE_QNUMBER].

Add test eight test cases from my table above, then fix the generator to
make them pass.

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/29/2015 01:33 AM, Markus Armbruster wrote:
>> Eric Blake <eblake@redhat.com> writes:
>> 
>>> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>>>> The struct generated for a flat union is weird: the members of its
>>>> base are at the end, except for the union tag, which is renamed to
>>>> 'kind' and put at the beginning.
>>>
>
>>> Therefore, it might be worth mentioning that avoiding the rename to
>>> 'kind' is a bug fix, not just a nicer struct :)
>> 
>> Cool!  I'll work (a variation of) this test case into my series.
>
> Another name collision bug: our code generates flat unions as:
>
> struct BlockdevOptions {
>     BlockdevDriver driver;
> ...
>     /* End fields inherited from BlockdevOptionsBase. */
>     /* union tag is BlockdevDriver driver */
>     union {
>         void *data;
>         BlockdevOptionsArchipelago *archipelago;
> ...
>
> which means that if we name any of the branches 'data' (that is, if
> 'data' is a member of the enum discriminator), things fail to compile.
> We could probably fix that by naming our dummy branch '_data'.

I wonder whether member data is actually used.  I'll find out.

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2871 bytes --]

On 07/30/2015 12:42 AM, Markus Armbruster wrote:

>> But what happens if the 0th branch is mapped to a different parser, as
>> would be the case if one of the alternate's branches is 'number'?  In
>> particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
>>  So, if we have this qapi:
>>  { 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
>> but pass in an integer, visit_get_next_type() will see a qtype of QInt,
>> but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and
>> we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the
>> string parser doesn't like ints) even though the parse should succeed by
>> using the FOO_KIND_B branch.
> 
> Yup, bug.

And it's an order-dependent bug - merely declaring 'b' first makes it
appear to work correctly.

> 
>> Interestingly, this means that if we ever write an alternate type that
>> accepts both 'int' and 'number' (we have not attempted that so far),
>> then the number branch will only be taken for inputs that don't also
>> look like ints (normally, 'number' accepts anything numeric). Maybe that
>> means we should document and enforce that 'number' and 'int' cannot be
>> mixed in the same alternate?
> 
> Even if we outlaw mixing the two, I'm afraid we still have this bug: an
> alternate with a 'number' member rejects input that gets parsed as
> QTYPE_QINT.
> 
> Let's simply make alternates behave sanely:
> 
>     alternate has      case selected for
>     'int'  'number'    QTYPE_QINT  QTYPE_QFLOAT
>       no        no     error       error
>       no       yes     'number'    'number'
>      yes        no     'int'       error
>      yes       yes     'int'       'number'

Works for me.


> 
> + 1 works, because the element type is int, not BlockdevRefKind.  It's
> int so it can serve as argument for visit_get_next_type()'s parameter
> const int *qtypes.
> 
> The + 1, - 1 business could be mildly confusing.  We could set all
> unused elements to -1 instead:

Or, we could ditch the qtypes lookup altogether, and merely create the
alternate enum as a non-consecutive QTYPE mapping, for one less level of
indirection, as in:

typedef enum BlockdevRefKind {
    BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,
    BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
};

then rewrite visit_get_next_type() to directly return the qtype, as well
as rewrite the generated switch statement in visit_type_BlockdevRef() to
directly inspect the qtypes it cares about.  In fact, that's the
approach I'm currently playing with.

> Add test eight test cases from my table above, then fix the generator to
> make them pass.

I hope to post an RFC followup patch along those lines later today.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1835 bytes --]

On 07/30/2015 01:11 AM, Markus Armbruster wrote:

>> Another name collision bug: our code generates flat unions as:
>>
>> struct BlockdevOptions {
>>     BlockdevDriver driver;
>> ...
>>     /* End fields inherited from BlockdevOptionsBase. */
>>     /* union tag is BlockdevDriver driver */
>>     union {
>>         void *data;
>>         BlockdevOptionsArchipelago *archipelago;
>> ...
>>
>> which means that if we name any of the branches 'data' (that is, if
>> 'data' is a member of the enum discriminator), things fail to compile.
>> We could probably fix that by naming our dummy branch '_data'.
> 
> I wonder whether member data is actually used.  I'll find out.

The dealloc visitor uses 'data' being non-null as a flag on whether to
deallocate the union even if the tag was invalid for some reason; or
more importantly, if parsing consumed the tag but then detected an error
while parsing the union, leaving the union branch partially allocated.
To avoid a leak, we have to deallocate the branch.

But if the tag was invalid, then why did we ever allocate the union in
the first place, and how do we prove we are calling the correct free-ing
function?  And if the tag is valid, why can't we just guarantee that the
union is 0-initialized and that deleting the branch will work through
the correct branch type instead of worrying about 'data'?

We still need a dummy member if it is valid to do { 'union':'Foo',
'data':{} } since C doesn't like empty unions, but an empty union seems
like something we may want to reject, at which point you are probably
right that deleting the data member altogether should work and still let
us recover from bad partial parses without a leak.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

On 07/29/2015 11:22 AM, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
>> On 07/29/2015 02:32 AM, Markus Armbruster wrote:
>>
>>>>> 2. We can leak retval only when qmp_FOO() returns non-null and local_err
>>>>>    is non-null.  This must not happen, because:
>>>>>
>>>>>    a. local_err must be null before the call, and
>>>>>
>>>>>    b. the call must not return non-null when it sets local_err.
>>>>
>>>> We don't state that contract anywhere, but I doubt any of the qmp_FOO()
>>>> functions violate it, so it is worth making it part of the contract.
>>>
>>> It's a general Error API rule: set an error exactly on failure.  It
>>> applies to any function returning errors through an Error **errp
>>> parameter, and we generally don't bother to spell it out for the
>>> individual functions.
>>>
>>> The part that needs to be spelling out is what success and failure mean.
>>> A qmp_FOO() returning an object returns null on failure.

For qmp_FOO(), this is a reasonable contract.  But our very own
generated code does not follow these rules: visit_type_FOO() can assign
into *obj even when setting an error, if it encounters a parse error
halfway through the struct, leaving the caller responsible to still
clean up the mess if it wants to avoid a memory leak.

Maybe that means our generated code needs to be reworked to properly
clean up on a failed parse, such that *obj is guaranteed to be NULL if
an error is returned.  As a separate patch, of course.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/30/2015 01:11 AM, Markus Armbruster wrote:
>
>>> Another name collision bug: our code generates flat unions as:
>>>
>>> struct BlockdevOptions {
>>>     BlockdevDriver driver;
>>> ...
>>>     /* End fields inherited from BlockdevOptionsBase. */
>>>     /* union tag is BlockdevDriver driver */
>>>     union {
>>>         void *data;
>>>         BlockdevOptionsArchipelago *archipelago;
>>> ...
>>>
>>> which means that if we name any of the branches 'data' (that is, if
>>> 'data' is a member of the enum discriminator), things fail to compile.
>>> We could probably fix that by naming our dummy branch '_data'.
>> 
>> I wonder whether member data is actually used.  I'll find out.
>
> The dealloc visitor uses 'data' being non-null as a flag on whether to
> deallocate the union even if the tag was invalid for some reason; or
> more importantly, if parsing consumed the tag but then detected an error
> while parsing the union, leaving the union branch partially allocated.
> To avoid a leak, we have to deallocate the branch.
>
> But if the tag was invalid, then why did we ever allocate the union in
> the first place, and how do we prove we are calling the correct free-ing
> function?  And if the tag is valid, why can't we just guarantee that the
> union is 0-initialized and that deleting the branch will work through
> the correct branch type instead of worrying about 'data'?

Good questions.  Someone will have to review and fix this code.  Let's
add a FIXME so we don't forget.  Care to propose one?

> We still need a dummy member if it is valid to do { 'union':'Foo',
> 'data':{} } since C doesn't like empty unions, but an empty union seems
> like something we may want to reject, at which point you are probably
> right that deleting the data member altogether should work and still let
> us recover from bad partial parses without a leak.

Either we reject empty unions, or we detect them and add a dummy,
similar to what we do for structs since commit 83ecb22.  I figure simply
rejecting them is easier.

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/30/2015 12:42 AM, Markus Armbruster wrote:
>
>>> But what happens if the 0th branch is mapped to a different parser, as
>>> would be the case if one of the alternate's branches is 'number'?  In
>>> particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
>>>  So, if we have this qapi:
>>>  { 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
>>> but pass in an integer, visit_get_next_type() will see a qtype of QInt,
>>> but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and
>>> we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the
>>> string parser doesn't like ints) even though the parse should succeed by
>>> using the FOO_KIND_B branch.
>> 
>> Yup, bug.
>
> And it's an order-dependent bug - merely declaring 'b' first makes it
> appear to work correctly.
>
>> 
>>> Interestingly, this means that if we ever write an alternate type that
>>> accepts both 'int' and 'number' (we have not attempted that so far),
>>> then the number branch will only be taken for inputs that don't also
>>> look like ints (normally, 'number' accepts anything numeric). Maybe that
>>> means we should document and enforce that 'number' and 'int' cannot be
>>> mixed in the same alternate?
>> 
>> Even if we outlaw mixing the two, I'm afraid we still have this bug: an
>> alternate with a 'number' member rejects input that gets parsed as
>> QTYPE_QINT.
>> 
>> Let's simply make alternates behave sanely:
>> 
>>     alternate has      case selected for
>>     'int'  'number'    QTYPE_QINT  QTYPE_QFLOAT
>>       no        no     error       error
>>       no       yes     'number'    'number'
>>      yes        no     'int'       error
>>      yes       yes     'int'       'number'
>
> Works for me.
>
>
>> 
>> + 1 works, because the element type is int, not BlockdevRefKind.  It's
>> int so it can serve as argument for visit_get_next_type()'s parameter
>> const int *qtypes.
>> 
>> The + 1, - 1 business could be mildly confusing.  We could set all
>> unused elements to -1 instead:
>
> Or, we could ditch the qtypes lookup altogether, and merely create the
> alternate enum as a non-consecutive QTYPE mapping, for one less level of
> indirection, as in:
>
> typedef enum BlockdevRefKind {
>     BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,

QTYPE_QDICT, but I get what you mean.

>     BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
> };
>
> then rewrite visit_get_next_type() to directly return the qtype, as well
> as rewrite the generated switch statement in visit_type_BlockdevRef() to
> directly inspect the qtypes it cares about.  In fact, that's the
> approach I'm currently playing with.

Hmm.  The schema currently doesn't let you control enumeration values.
qapi-code-gen.txt specifies:

    The enumeration values [...] are encoded as C enum integral values
    in generated code.  While the C code starts numbering at 0, it is
    better to use explicit comparisons to enum values than implicit
    comparisons to 0; the C code will also include a generated enum
    member ending in _MAX for tracking the size of the enum, useful when
    using common functions for converting between strings and enum
    values.

Strictly speaking, this needn't apply to implicit enums like
BlockdevRefKind.  But is it a good idea to make them different?

Hmm, your new BlockdevRefKind is basically a subset of qtype_code with
the members renamed.  Could we simply use qtype_code directly?

>> Add test eight test cases from my table above, then fix the generator to
>> make them pass.
>
> I hope to post an RFC followup patch along those lines later today.

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/29/2015 11:22 AM, Markus Armbruster wrote:
>> Eric Blake <eblake@redhat.com> writes:
>> 
>>> On 07/29/2015 02:32 AM, Markus Armbruster wrote:
>>>
>>>>>> 2. We can leak retval only when qmp_FOO() returns non-null and local_err
>>>>>>    is non-null.  This must not happen, because:
>>>>>>
>>>>>>    a. local_err must be null before the call, and
>>>>>>
>>>>>>    b. the call must not return non-null when it sets local_err.
>>>>>
>>>>> We don't state that contract anywhere, but I doubt any of the qmp_FOO()
>>>>> functions violate it, so it is worth making it part of the contract.
>>>>
>>>> It's a general Error API rule: set an error exactly on failure.  It
>>>> applies to any function returning errors through an Error **errp
>>>> parameter, and we generally don't bother to spell it out for the
>>>> individual functions.
>>>>
>>>> The part that needs to be spelling out is what success and failure mean.
>>>> A qmp_FOO() returning an object returns null on failure.
>
> For qmp_FOO(), this is a reasonable contract.  But our very own
> generated code does not follow these rules: visit_type_FOO() can assign
> into *obj even when setting an error, if it encounters a parse error
> halfway through the struct, leaving the caller responsible to still
> clean up the mess if it wants to avoid a memory leak.

Assigning to *obj, then fail is tolerable[*].  Relying on the caller to
free it is not.  If we do that, it's a bug.

> Maybe that means our generated code needs to be reworked to properly
> clean up on a failed parse, such that *obj is guaranteed to be NULL if
> an error is returned.  As a separate patch, of course.

Yes.  Would you like to propose a FIXME for me to put into this series?


[*] Leaving *obj alone on failure is nicer, but may not always be
practical.

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2666 bytes --]

On 07/30/2015 09:53 AM, Markus Armbruster wrote:

>> Or, we could ditch the qtypes lookup altogether, and merely create the
>> alternate enum as a non-consecutive QTYPE mapping, for one less level of
>> indirection, as in:
>>
>> typedef enum BlockdevRefKind {
>>     BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,
> 
> QTYPE_QDICT, but I get what you mean.
> 
>>     BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
>> };
>>
>> then rewrite visit_get_next_type() to directly return the qtype, as well
>> as rewrite the generated switch statement in visit_type_BlockdevRef() to
>> directly inspect the qtypes it cares about.  In fact, that's the
>> approach I'm currently playing with.
> 
> Hmm.  The schema currently doesn't let you control enumeration values.

For public enums. But the enum for alternate is never public - you never
send the name of the branch over the wire, so we don't need to stringize
the name anywhere.

> qapi-code-gen.txt specifies:
> 
>     The enumeration values [...] are encoded as C enum integral values
>     in generated code.  While the C code starts numbering at 0, it is
>     better to use explicit comparisons to enum values than implicit
>     comparisons to 0; the C code will also include a generated enum
>     member ending in _MAX for tracking the size of the enum, useful when
>     using common functions for converting between strings and enum
>     values.
> 
> Strictly speaking, this needn't apply to implicit enums like
> BlockdevRefKind.  But is it a good idea to make them different?

I think so, but maybe under a different name (maybe
BLOCKDEV_REF_REFERENCE_QTYPE) to make it more obvious that this is not
the usual generated enum, but special to the alternate.

> 
> Hmm, your new BlockdevRefKind is basically a subset of qtype_code with
> the members renamed.  Could we simply use qtype_code directly?

We could, except that clients that manipulate the generated struct then
have to know the qtype mapping directly; while keeping symbolic names
lets them do 'foo->type = BLOCKDEV_REF_REFERENCE; foo->reference = xyz;'
as a nice visual indicator of which union member within the struct is
being assigned according to the discriminator.

I guess I'll see how much code currently manipulates the generated
structs (I already recall from other patches in this series that
blockdev played a bit loose by  validating that the QMP was okay and
then using QDict for everything else rather than the generated struct)
and make my decision when posting my RFC patch.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

On 07/30/2015 10:36 AM, Eric Blake wrote:
> On 07/30/2015 09:53 AM, Markus Armbruster wrote:
> 
>>> Or, we could ditch the qtypes lookup altogether, and merely create the
>>> alternate enum as a non-consecutive QTYPE mapping, for one less level of
>>> indirection, as in:
>>>
>>> typedef enum BlockdevRefKind {
>>>     BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,
>>
>> QTYPE_QDICT, but I get what you mean.
>>
>>>     BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
>>> };
>>>

>> Hmm, your new BlockdevRefKind is basically a subset of qtype_code with
>> the members renamed.  Could we simply use qtype_code directly?
> 
> We could, except that clients that manipulate the generated struct then
> have to know the qtype mapping directly; while keeping symbolic names
> lets them do 'foo->type = BLOCKDEV_REF_REFERENCE; foo->reference = xyz;'
> as a nice visual indicator of which union member within the struct is
> being assigned according to the discriminator.
> 
> I guess I'll see how much code currently manipulates the generated
> structs (I already recall from other patches in this series that
> blockdev played a bit loose by  validating that the QMP was okay and
> then using QDict for everything else rather than the generated struct)
> and make my decision when posting my RFC patch.

Turns out that using it directly was easier, and less code:
http://thread.gmane.org/gmane.comp.emulators.qemu/353204/focus=354008

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

[Qemu-devel] [RFC PATCH 12.5/47] qapi: Document that input visitor semantics are prone to leaks

[Eric Blake]

Most functions that can return a pointer or set an Error ** value
are decent enough to guarantee a NULL return when reporting an error.
Not so with our generated qapi visitor functions.  If the caller
is not careful to clean up partially-allocated objects on error,
then the caller suffers a memory leak.

Properly fixing it is probably complex enough to save for a later
day, so merely document it for now.

Signed-off-by: Eric Blake <eblake@redhat.com>
---

As mentioned elsewhere in the thread (comments on 29/47
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg06107.html)
it would be worth documenting a FIXME for a future series.
I'm submitting this as 12.5/47 due to its relation to other
similar shortcoming doc patches; and assuming Markus can rebase
the rest of the series on top if he wants to fold it into his
v3 posting at this spot. Otherwise, I can wait for his v3 and
rebase it to be part of my (growing) followup series.

 scripts/qapi-visit.py          | 4 ++++
 tests/test-qmp-input-visitor.c | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 73f136f..eec5f1f 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -115,6 +115,10 @@ out:


 def generate_visit_struct_body(name):
+    # FIXME: if *obj is NULL on entry, and visit_start_struct() assigns to
+    # *obj, but then visit_type_FOO_fields() fails, we should clean up *obj
+    # rather than leaving it non-NULL. As currently written, the caller must
+    # call qapi_free_FOO() to avoid a memory leak of the partial FOO.
     ret = mcgen('''
     Error *err = NULL;

diff --git a/tests/test-qmp-input-visitor.c b/tests/test-qmp-input-visitor.c
index b7a87ee..a5cfefa 100644
--- a/tests/test-qmp-input-visitor.c
+++ b/tests/test-qmp-input-visitor.c
@@ -636,6 +636,8 @@ static void test_visitor_in_errors(TestInputVisitorData *data,

     visit_type_TestStruct(v, &p, NULL, &err);
     g_assert(err);
+    /* FIXME - a failed parse should not leave a partially-allocated p
+     * for us to clean up; this could cause callers to leak memory. */
     g_assert(p->string == NULL);

     error_free(err);
-- 
2.4.3

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

On 07/30/2015 09:57 AM, Markus Armbruster wrote:
>> For qmp_FOO(), this is a reasonable contract.  But our very own
>> generated code does not follow these rules: visit_type_FOO() can assign
>> into *obj even when setting an error, if it encounters a parse error
>> halfway through the struct, leaving the caller responsible to still
>> clean up the mess if it wants to avoid a memory leak.
> 
> Assigning to *obj, then fail is tolerable[*].  Relying on the caller to
> free it is not.  If we do that, it's a bug.
> 
>> Maybe that means our generated code needs to be reworked to properly
>> clean up on a failed parse, such that *obj is guaranteed to be NULL if
>> an error is returned.  As a separate patch, of course.
> 
> Yes.  Would you like to propose a FIXME for me to put into this series?

Done (see 12.5/47).

> 
> 
> [*] Leaving *obj alone on failure is nicer, but may not always be
> practical.

I'm wondering if visit_end_struct() should be changed to accept a bool
parameter that is true if the struct is ended normally, and false if an
error has been detected.  We may also need to alter visit_start_struct
to return a bool (true if an object was allocated), to help feed our
visitor logic on whether to pass true or false to the visit_end_struct().

We did something like that for visit_start_union()/visit_end_union(),
but I suspect those visitor interfaces are also a bit screwy.  Oh well,
I'm not going to try and tweak it today, but am happy with just adding
the FIXME.

Also, it would be really nice if we had docs in visitor.h and/or
visitor-impl.h, to get some sort of feel for how the visitor is supposed
to be used.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

[Qemu-devel] [RFC PATCH 12.6/47] qapi: Document shortcoming with union 'data' branch

[Eric Blake]

Add a FIXME to remind us to fully audit whether removing the
'void *data' branch of each qapi union type can be done safely.

Signed-off-by: Eric Blake <eblake@redhat.com>
---

Another potential doc FIXME patch, this time based on 7/47
(https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg06101.html)

 scripts/qapi-types.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index c6c2786..b3434b9 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -217,6 +217,14 @@ struct %(name)s
 ''',
                      discriminator_type_name=c_name(discriminator_type_name))

+    # FIXME: What purpose does data serve, besides preventing a union that
+    # has a branch named 'data'? We use it in qapi-visit.py to decide
+    # whether to bypass the switch statement if visiting the discriminator
+    # failed; but since we 0-initialize structs, and cannot tell what
+    # branch of the union is in use if the discriminator is invalid, there
+    # should not be any data leaks even without a data pointer.  Or, if
+    # 'data' is merely added to guarantee we don't have an empty union,
+    # shouldn't we enforce that at .json parse time?
     ret += mcgen('''
     union {
         void *data;
-- 
2.4.3

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

On 07/30/2015 09:44 AM, Markus Armbruster wrote:

>>>> which means that if we name any of the branches 'data' (that is, if
>>>> 'data' is a member of the enum discriminator), things fail to compile.
>>>> We could probably fix that by naming our dummy branch '_data'.
>>>
>>> I wonder whether member data is actually used.  I'll find out.
>>
>> The dealloc visitor uses 'data' being non-null as a flag on whether to
>> deallocate the union even if the tag was invalid for some reason; or
>> more importantly, if parsing consumed the tag but then detected an error
>> while parsing the union, leaving the union branch partially allocated.
>> To avoid a leak, we have to deallocate the branch.
>>
>> But if the tag was invalid, then why did we ever allocate the union in
>> the first place, and how do we prove we are calling the correct free-ing
>> function?  And if the tag is valid, why can't we just guarantee that the
>> union is 0-initialized and that deleting the branch will work through
>> the correct branch type instead of worrying about 'data'?
> 
> Good questions.  Someone will have to review and fix this code.  Let's
> add a FIXME so we don't forget.  Care to propose one?

Sure; see 12.6/47 (since that is close to several other patches adding
FIXME comments).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/30/2015 09:57 AM, Markus Armbruster wrote:
>>> For qmp_FOO(), this is a reasonable contract.  But our very own
>>> generated code does not follow these rules: visit_type_FOO() can assign
>>> into *obj even when setting an error, if it encounters a parse error
>>> halfway through the struct, leaving the caller responsible to still
>>> clean up the mess if it wants to avoid a memory leak.
>> 
>> Assigning to *obj, then fail is tolerable[*].  Relying on the caller to
>> free it is not.  If we do that, it's a bug.
>> 
>>> Maybe that means our generated code needs to be reworked to properly
>>> clean up on a failed parse, such that *obj is guaranteed to be NULL if
>>> an error is returned.  As a separate patch, of course.
>> 
>> Yes.  Would you like to propose a FIXME for me to put into this series?
>
> Done (see 12.5/47).
>
>> 
>> 
>> [*] Leaving *obj alone on failure is nicer, but may not always be
>> practical.
>
> I'm wondering if visit_end_struct() should be changed to accept a bool
> parameter that is true if the struct is ended normally, and false if an
> error has been detected.  We may also need to alter visit_start_struct
> to return a bool (true if an object was allocated), to help feed our
> visitor logic on whether to pass true or false to the visit_end_struct().
>
> We did something like that for visit_start_union()/visit_end_union(),
> but I suspect those visitor interfaces are also a bit screwy.  Oh well,
> I'm not going to try and tweak it today, but am happy with just adding
> the FIXME.

We need to keep the introspection series reasonably focused.  I can't
afford a detour into visitor semantics right now, so we'll go with your
FIXMEs.

> Also, it would be really nice if we had docs in visitor.h and/or
> visitor-impl.h, to get some sort of feel for how the visitor is supposed
> to be used.

Indeed.  We got a grand total of three one-liner comments, and one of
them is inaccurate: actual visitors don't obey /* Must be set */.

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Markus Armbruster]

Markus Armbruster <armbru@redhat.com> writes:

> Eric Blake <eblake@redhat.com> writes:
>
>> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>>> The struct generated for a flat union is weird: the members of its
>>> base are at the end, except for the union tag, which is renamed to
>>> 'kind' and put at the beginning.
>>
>> The renaming to 'kind' was a bug waiting to happen.  Consider this
>> example, which is broken before your series:
>>
>> diff --git i/tests/qapi-schema/qapi-schema-test.json
>> w/tests/qapi-schema/qapi-schema-test.json
>> index c7eaa86..12c09e3 100644
>> --- i/tests/qapi-schema/qapi-schema-test.json
>> +++ w/tests/qapi-schema/qapi-schema-test.json
>> @@ -37,7 +37,7 @@
>>    'data': { 'string1': 'str', 'string2': 'str' } }
>>
>>  { 'struct': 'UserDefUnionBase',
>> -  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
>> +  'data': { 'kind': 'str', 'enum1': 'EnumOne' } }
>>
>>  { 'union': 'UserDefFlatUnion',
>>    'base': 'UserDefUnionBase',
>>
>>
>> leading to this compilation error during 'make check-unit':
>>
>> In file included from tests/test-qmp-output-visitor.c:17:0:
>> tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’
>>      char *kind;
>>            ^
>> tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’
>>      char *kind;
>>            ^
>>
>> Therefore, it might be worth mentioning that avoiding the rename to
>> 'kind' is a bug fix, not just a nicer struct :)
>
> Cool!  I'll work (a variation of) this test case into my series.

I split the bug fix off this patch.  I put the test in the commit
message, because I feel it has little value as regression test going
forward.

Re: [Qemu-devel] [RFC PATCH 12.5/47] qapi: Document that input visitor semantics are prone to leaks

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> Most functions that can return a pointer or set an Error ** value
> are decent enough to guarantee a NULL return when reporting an error.
> Not so with our generated qapi visitor functions.  If the caller
> is not careful to clean up partially-allocated objects on error,
> then the caller suffers a memory leak.
>
> Properly fixing it is probably complex enough to save for a later
> day, so merely document it for now.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>

Picked into my series.  Thanks!

Re: [Qemu-devel] [RFC PATCH 12.6/47] qapi: Document shortcoming with union 'data' branch

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> Add a FIXME to remind us to fully audit whether removing the
> 'void *data' branch of each qapi union type can be done safely.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>

Picked into my series.  Thanks!

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/29/2015 01:33 AM, Markus Armbruster wrote:
>> Eric Blake <eblake@redhat.com> writes:
>> 
>>> On 07/01/2015 02:21 PM, Markus Armbruster wrote:
>>>> The struct generated for a flat union is weird: the members of its
>>>> base are at the end, except for the union tag, which is renamed to
>>>> 'kind' and put at the beginning.
>>>
>
>>> Therefore, it might be worth mentioning that avoiding the rename to
>>> 'kind' is a bug fix, not just a nicer struct :)
>> 
>> Cool!  I'll work (a variation of) this test case into my series.
>
> Another name collision bug: our code generates flat unions as:
>
> struct BlockdevOptions {
>     BlockdevDriver driver;
> ...
>     /* End fields inherited from BlockdevOptionsBase. */
>     /* union tag is BlockdevDriver driver */
>     union {
>         void *data;
>         BlockdevOptionsArchipelago *archipelago;
> ...
>
> which means that if we name any of the branches 'data' (that is, if
> 'data' is a member of the enum discriminator), things fail to compile.
> We could probably fix that by naming our dummy branch '_data'.

Works, because schema names should not begin with '_', except for
downstream extensions, which begin with '__RFQDN_'.  We don't enforce
that, however.  I'll include the appended patch.


Subject: [PATCH] qapi: Document flaws in checking of names

We don't actually enforce our "other than downstream extensions [...],
all names should begin with a letter" rule.  Add a FIXME.

We should reject names that differ only in '_' vs. '.'  vs. '-',
because they're liable to clash in generated C.  Add a FIXME.
---
 scripts/qapi.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 4af47ef..e61db30 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -341,6 +341,8 @@ def discriminator_find_enum_define(expr):
 
     return find_enum(discriminator_type)
 
+# FIXME should enforce "other than downstream extensions [...], all
+# names should begin with a letter".
 valid_name = re.compile('^[a-zA-Z_][a-zA-Z0-9_.-]*$')
 def check_name(expr_info, source, name, allow_optional = False,
                enum_member = False):
@@ -367,6 +369,8 @@ def check_name(expr_info, source, name, allow_optional = False,
 def add_name(name, info, meta, implicit = False):
     global all_names
     check_name(info, "'%s'" % meta, name)
+    # FIXME should reject names that differ only in '_' vs. '.'
+    # vs. '-', because they're liable to clash in generated C.
     if name in all_names:
         raise QAPIExprError(info,
                             "%s '%s' is already defined"
-- 
2.4.3

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/21/2015 06:43 AM, Eric Blake wrote:
>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>> A command's 'data' must be a struct type, given either as a
>>> dictionary, or as struct type name.
>>>
>>> Existing test case data-int.json covers simple type 'int'.  Add test
>>> cases for type names referring to union and alternate types.
>> 
>> We could probably relax things to allow a union (which is always a
>> dictionary on the wire), but I agree that allowing an alternate type is
>> not appropriate (the goal here is that we require a dictionary).  But
>> it's also easier to be conservative now and relax later.
>> 
>
>>> +++ b/tests/qapi-schema/args-alternate.json
>>> @@ -0,0 +1,4 @@
>>> +# we do not allow alternate arguments
>>> +# TODO should we support this?
>> 
>> I see no reason to allow a non-dictionary in QMP, so this TODO could be
>> dropped.
>
> Or, to be clear, we document that arguments is always a dictionary, for:
> { "execute":"command", "arguments":{} }. Allowing an alternate would
> break that, so it is a different level of change to allow an alternate
> (change the QMP protocol) than what it would be to allow a union (the
> QMP protocol is unchanged).  Not that we can't do it if we ever have a
> reason, it's just that I don't see it being worth a TODO statement.
>
>> 
>>> +++ b/tests/qapi-schema/args-union.json
>>> @@ -0,0 +1,4 @@
>>> +# FIXME we should reject union arguments
>>> +# TODO should we support this?
>>> +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
>>> +{ 'command': 'oops', 'data': 'Uni' }
>> 
>> This, on the other hand, seems valid from the wire format (it will
>> always be a dictionary).  I guess the problem is that we generate a C
>> function signature based by calling out each member of the dictionary -
>> but how do you do that for a union?  So I see what you are doing:
>> marking that this test currently passes the parser, but then causes
>> problems for generating C code, so we should either reject it up front,
>> or fix the generator.  The FIXME documents what you will do later in the
>> series (reject it up front) and the TODO documents what we can do down
>> the road (fix the generator to allow it).
>
> See also 32/47 - events have the same problem.

I'll amend the next patch to fix them, too.

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 07/27/2015 03:54 AM, Markus Armbruster wrote:
>> Eric Blake <eblake@redhat.com> writes:
>> 
>>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>>>> New methods c_name(), c_type(), c_null(), json_type(),
>>>> alternate_qtype().
>>>>
>>>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>>>> ---
>>>>  scripts/qapi.py | 72
>>>> +++++++++++++++++++++++++++++++++++++++++++++++++++------
>>>>  1 file changed, 65 insertions(+), 7 deletions(-)
>>>>
>>>
>>> I just noticed:
>>>
>>>> @@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
>>>>          for v in values:
>>>>              assert isinstance(v, str)
>>>>          self.values = values
>>>> +    def c_type(self, is_param=False):
>>>> +        return c_name(self.name)
>>>> +    def c_null(self):
>>>> +        return c_enum_const(self.name, self.values[0])
>>>
>>> What does this return for an empty enum, as in { 'enum':'Empty',
>>> 'data':[] }?
>> 
>> I suspect self.values will be [] then, and self.values[0] will bomb.
>> 
>> Possible fixes:
>> 
>> * Outlaw empty enums
>> 
>> * Add the implicit MAX member to self.values[] (other code may have to
>>   skip it)
>> 
>> * Catch the special case here, and return the implicit MAX member.
>
> I'm leaning toward the third option here.

Me too now, because it's easy:

        return c_enum_const(self.name, (self.values + [ 'MAX' ])[0])

>>>               Our testsuite proves we can do that, even if our normal
>>> .json code doesn't use it.
>> 
>> tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }
>
> As I've mentioned elsewhere, most of our tests/qapi-schema/*.json merely
> cover whether the parser is okay with the input, while
> tests/qapi-schema/qapi-schema-test.json is the one that also tests that
> the generated C code works; so sounds like that test should be enhanced
> to cover some of these corner cases we have been considering in this
> series (empty enum, command with no arguments and no returns, and so forth).

Yes, that would be nice, but I feel I have to leave it for another day.

Re: [Qemu-devel] [PATCH RFC v2 45/47] qapi: New QMP command query-schema for QMP schema introspection

[Markus Armbruster]

Markus Armbruster <armbru@redhat.com> writes:

> Eric Blake <eblake@redhat.com> writes:
>
>> On 07/29/2015 03:19 AM, Markus Armbruster wrote:
>>>>> Longest line is a bit over 4KiB for me.
>>>>>
>>>>
>>>> If we break up string literals, at least use some indentation to make it
>>>> obvious that multiple lines merge to a single array entry. For example
>>>> (after patch 47):
>>>>
>>>> ...
>>>>     "{ 'name': ':abr', 'meta-type': 'object', "
>>>>       "'members': [ "
>>>>         "{ 'name': 'device', 'type': ':acg', 'default': null }, "
>>>>         "{ 'name': 'node-name', 'type': ':acg', 'default': null }, "
>>>>         "{ 'name': 'snapshot-file', 'type': ':acg' }, "
>>>>         "{ 'name': 'snapshot-node-name', 'type': ':acg', 'default': null
>>>> }, "
>>>>         "{ 'name': 'format', 'type': ':acg', 'default': null }, "
>>>>         "{ 'name': 'mode', 'type': ':afo', 'default': null } ] }, "
>>>>     "{ 'name': ... "
>>> 
>>> Unconventional indentation, but if it helps the reader...
>>
>> I'm not a stickler about the particular spacing I used, so much as
>> demonstrating an idea.  Pick any indentation you like; I was just
>> demonstrating that some well-chosen line breaks, coupled with visual
>> clues on what belongs together, can help in reading the string literal
>> in the generated file.
>>
>> In fact, doesn't python have a way to pretty-print JSON, and then
>> post-process the pretty-printed string to add C \" escaping?
>
> Interesting idea, definitely worth a doc search.

Module json, new in 2.6 (Oct 2008).  As usual, we're a decade behind:
2.4 (Nov 2004).

> Prettier output can of course be punted to a followup-patch.

I guess it'll have to wait for 2.6.

[...]

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

[Markus Armbruster]

Markus Armbruster <armbru@redhat.com> writes:

> Eric Blake <eblake@redhat.com> writes:
>
>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
[...]
>>> @@ -105,7 +103,8 @@ struct %(name)s
>>>  
>>>  def generate_enum_lookup(name, values):
>>>      ret = mcgen('''
>>> -const char * const %(name)s_lookup[] = {
>>> +
>>> +const char *const %(name)s_lookup[] = {
>>
>> [2] generated diffs like this:
>>
>> qapi-types.c:
>> -const char * const OnOffAuto_lookup[] = {
>> +const char *const OnOffAuto_lookup[] = {
>>
>> Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we
>> added a const in commit 2e4450ff that is missing from the documentation.
>
> Minor review fail.  Not the first time.

I take that back, it's actually not visible in qapi-code-gen.txt.

[...]

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]

On 08/04/2015 03:08 AM, Markus Armbruster wrote:
> Markus Armbruster <armbru@redhat.com> writes:
> 
>> Eric Blake <eblake@redhat.com> writes:
>>
>>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
> [...]
>>>> @@ -105,7 +103,8 @@ struct %(name)s
>>>>  
>>>>  def generate_enum_lookup(name, values):
>>>>      ret = mcgen('''
>>>> -const char * const %(name)s_lookup[] = {
>>>> +
>>>> +const char *const %(name)s_lookup[] = {
>>>
>>> [2] generated diffs like this:
>>>
>>> qapi-types.c:
>>> -const char * const OnOffAuto_lookup[] = {
>>> +const char *const OnOffAuto_lookup[] = {
>>>
>>> Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we
>>> added a const in commit 2e4450ff that is missing from the documentation.
>>
>> Minor review fail.  Not the first time.
> 
> I take that back, it's actually not visible in qapi-code-gen.txt.

Oh, you're right - the only mention of *_lookup[] in the docs file is
the array generated for all events, and that one was output with correct
spacing (until the rest of your series fixes it to share code rather
than duplicate things with slight differences).

Maybe we SHOULD be showing what the generators do for enums, unions, and
alternates (by expanding the example-schema.json that is then fed to all
the example script usage). But that's fine as a project for another day.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 08/04/2015 03:08 AM, Markus Armbruster wrote:
>> Markus Armbruster <armbru@redhat.com> writes:
>> 
>>> Eric Blake <eblake@redhat.com> writes:
>>>
>>>> On 07/01/2015 02:22 PM, Markus Armbruster wrote:
>> [...]
>>>>> @@ -105,7 +103,8 @@ struct %(name)s
>>>>>  
>>>>>  def generate_enum_lookup(name, values):
>>>>>      ret = mcgen('''
>>>>> -const char * const %(name)s_lookup[] = {
>>>>> +
>>>>> +const char *const %(name)s_lookup[] = {
>>>>
>>>> [2] generated diffs like this:
>>>>
>>>> qapi-types.c:
>>>> -const char * const OnOffAuto_lookup[] = {
>>>> +const char *const OnOffAuto_lookup[] = {
>>>>
>>>> Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we
>>>> added a const in commit 2e4450ff that is missing from the documentation.
>>>
>>> Minor review fail.  Not the first time.
>> 
>> I take that back, it's actually not visible in qapi-code-gen.txt.
>
> Oh, you're right - the only mention of *_lookup[] in the docs file is
> the array generated for all events, and that one was output with correct
> spacing (until the rest of your series fixes it to share code rather
> than duplicate things with slight differences).
>
> Maybe we SHOULD be showing what the generators do for enums, unions, and
> alternates (by expanding the example-schema.json that is then fed to all
> the example script usage). But that's fine as a project for another day.

The place where we try to exercise all the schema features is
qapi-schema-test.json.