qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
@ 2019-06-24 17:39 Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values Max Reitz
                   ` (17 more replies)
  0 siblings, 18 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

Hi,

There are two explanations of this cover letter, a relative one (to v3)
and an absolute one.


*** Important note ***

The final patch in this series is an example that converts most of
block-core.json to use default values where possible.  We may decide to
take it or not.  It isn’t important for the main purpose of this series,
so I’d be very much fine with chopping it off.

(It does have a nice diff stat, though.)

*** Important note end ***


Relative explanation:

The actual functional goal of this series is to allow all blockdev
options that can be represented with -drive to have an equivalent with
-blockdev (safe for rbd’s =keyvalue-pairs).

To this end, qcow(2)’s encryption needs an “auto” format which can
automatically deduce the format from the image header.  To make things
nicer, I decided (already in v1) to make this format optional so users
could just specify encrypt.secret and let the format driver figure out
the rest.

Until v3, this was implemented by letting the discriminator of flat
unions be optional, as long as a default-value is provided.  Markus
(rightfully) complained that this is very specific and would be covered
by just having default values for QAPI struct members in general.
So now this version implements this.  This is a bit more complicated
than just implementing a default-variant, mainly because the latter only
needs to accept enum values, whereas a generally usable “default” should
accept values of all QAPI types (to the extent what is reasonable).

So what was (until v3)

  { 'union': 'Foo',
    'base': { '*discr': 'SomeEnum' },
    'discriminator': 'discr',
    'default-variant': 'value1',
    'data': { 'value1': 'Bar', 'value2': 'Baz' } }

becomes

  { 'union': 'Foo',
    'base': { '*discr': { 'type': 'SomeEnum', 'default': 'value1' } },
    'discriminator': 'discr',
    'data': { 'value1': 'Bar', 'value2': 'Baz' } }



Absolute explanation:

When qemu reports json:{} filename, it just uses whatever type you gave
an option in.  With -drive, all options are strings and they do not have
to pass the test of the typing firewall of the QAPI schema, so you just
get strings thrown back at you even if that does not match the schema.
(Also, if you use json:{} yourself, you’re free to give the options as
strings as well.)

Example:

$ ./qemu-img info --image-opts driver=raw,size=512,file.driver=null-co
image: json:{"driver": "raw", "size": "512", "file": {"driver": "null-co"}}

@size is supposed to be an integer, according to the schema, so the
correct result would be (which is what you get after this series):

$ ./qemu-img info --image-opts driver=raw,size=512,file.driver=null-co
image: json:{"driver": "raw", "size": 512, "file": {"driver": "null-co"}}


This is achieved by patch 11, which makes bdrv_refresh_filename() run
the options through the flat-confused input visitor, and then through
the output visitor to get all to the correct type.  If anything fails,
the result is as before (hence the “Try” in the title).

There are cases where this cannot work.  Those are the ones where -drive
accepts something that is not allowed by the QAPI schema.  One of these
cases is rbd’s =keyvalue-pairs, which is just broken altogether, so
let’s simply ignore that.  (I don’t think anybody’s going to complain
that the json:{} filename they get is not correctly typed after they’ve
used that option.)

The other case (I know of) is qcow(2)’s encryption.  In the QAPI schema,
encrypt.format is not optional because it is the discriminator for which
kind of options to use.  However, for -drive, it is optional because the
qcow2 driver can infer the encryption format from the image header.

The solution that’s proposed by this series is to make flat union
discriminators optional and provide a default.  This is accomplished by
generally allowing default values to be provided for QAPI struct
members.

Both AES and LUKS encryption allow only a key-secret option, so we can
add a new pseudo-format “auto” that accepts exactly that option and
makes the qcow2 driver read the real format from the image header.  This
pseudo-format is made the default for encrypt.format, and thus you can
then specify encrypt.key-secret without having to specify
encrypt.format (while still adhering to the QAPI schema).


So, in this series:
- The QAPI code generator is modified to allow default values for
  optional struct members.  This in turn allows flat union
  discriminators be optional, too, but only if a default value is
  provided.
  - Accordingly, documentation, tests, and introspection are adjusted.

- This is used to make qcow’s and qcow2’s encrypt.format parameter
  optional.  It now defaults to “from-image” which is a new
  pseudo-format that allows a key-secret to be given, and otherwise
  leaves it to the format driver to determine the encryption format.

- json:{} filenames are attempted to be typed correctly when they are
  generated, by running bs->full_open_options through a healthy mix of
  qdict_flatten(), the flat-confused input visitor for BlockdevOptions,
  and the output visitor.
  This may not always work but I hope it usually will.  Fingers crossed.
  (At least it won’t make things worse.)

- Tests, tests, tests.


(Yes, I know that “In this series tests, tests, tests.” is not a
 sentence.)


v4:
- Drop the default-variant stuff and replace it by a more general
  concept of allowing default values for all QAPI struct members


git backport-diff against v3:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/14:[down] 'qapi: Parse numeric values'
002/14:[down] 'qapi: Move to_c_string() to common.py'
003/14:[down] 'qapi: Introduce default values for struct members'
004/14:[down] 'qapi: Allow optional discriminators'
005/14:[down] 'qapi: Document default values for struct members'
006/14:[down] 'test-qapi: Print struct members' default values'
007/14:[down] 'tests: Test QAPI default values for struct members'
008/14:[0044] [FC] 'tests: Add QAPI optional discriminator tests'
009/14:[0009] [FC] 'qapi: Formalize qcow2 encryption probing'
010/14:[0005] [FC] 'qapi: Formalize qcow encryption probing'
011/14:[0014] [FC] 'block: Try to create well typed json:{} filenames'
012/14:[----] [--] 'iotests: Test internal option typing'
013/14:[----] [--] 'iotests: qcow2's encrypt.format is now optional'
014/14:[down] 'block: Make use of QAPI defaults'


Max Reitz (14):
  qapi: Parse numeric values
  qapi: Move to_c_string() to common.py
  qapi: Introduce default values for struct members
  qapi: Allow optional discriminators
  qapi: Document default values for struct members
  test-qapi: Print struct members' default values
  tests: Test QAPI default values for struct members
  tests: Add QAPI optional discriminator tests
  qapi: Formalize qcow2 encryption probing
  qapi: Formalize qcow encryption probing
  block: Try to create well typed json:{} filenames
  iotests: Test internal option typing
  iotests: qcow2's encrypt.format is now optional
  block: Make use of QAPI defaults

 docs/devel/qapi-code-gen.txt                  |  81 +++++-
 tests/Makefile.include                        |  17 +-
 qapi/block-core.json                          | 180 +++++++++-----
 qapi/introspect.json                          |   9 +-
 tests/qapi-schema/bad-type-int.json           |   1 -
 tests/qapi-schema/enum-int-member.json        |   1 -
 ...l-discriminator-invalid-specification.json |  11 +
 ...on-optional-discriminator-no-default.json} |   5 +-
 tests/qapi-schema/qapi-schema-test.json       |  38 +++
 .../struct-member-alternate-default.json      |  10 +
 ...struct-member-bool-wrong-default-type.json |   3 +
 .../struct-member-enum-invalid-default.json   |   4 +
 ...struct-member-enum-wrong-default-type.json |   4 +
 .../struct-member-float-invalid-default.json  |   4 +
 ...truct-member-float-wrong-default-type.json |   3 +
 .../struct-member-int-wrong-default-type.json |   3 +
 .../struct-member-int8-erange-default.json    |   3 +
 .../struct-member-list-nonempty-default.json  |   4 +
 .../struct-member-non-optional-default.json   |   3 +
 .../struct-member-null-default.json           |   6 +
 .../struct-member-str-wrong-default-type.json |   3 +
 .../struct-member-uint8-erange-default.json   |   3 +
 .../struct-member-uint8-negative-default.json |   3 +
 block.c                                       |  68 ++++-
 block/file-posix.c                            |   9 -
 block/file-win32.c                            |   8 +-
 block/parallels.c                             |   6 +-
 block/qcow2.c                                 |  39 +--
 block/qed.c                                   |   3 -
 block/sheepdog.c                              |   3 -
 block/vdi.c                                   |   3 -
 block/vhdx.c                                  |  28 +--
 block/vpc.c                                   |   3 -
 blockdev.c                                    | 182 +++-----------
 monitor/hmp-cmds.c                            |  27 +-
 monitor/qmp-cmds.c                            |   3 +-
 scripts/qapi/commands.py                      |   2 +-
 scripts/qapi/common.py                        | 232 ++++++++++++++++--
 scripts/qapi/doc.py                           |  20 +-
 scripts/qapi/introspect.py                    |   8 +-
 scripts/qapi/types.py                         |   2 +-
 scripts/qapi/visit.py                         |  38 ++-
 tests/qapi-schema/bad-type-int.err            |   2 +-
 tests/qapi-schema/enum-int-member.err         |   2 +-
 ...al-discriminator-invalid-specification.err |   1 +
 ...-discriminator-invalid-specification.exit} |   0
 ...l-discriminator-invalid-specification.out} |   0
 ...nion-optional-discriminator-no-default.err |   1 +
 ...ion-optional-discriminator-no-default.exit |   1 +
 ...nion-optional-discriminator-no-default.out |   0
 .../flat-union-optional-discriminator.err     |   1 -
 tests/qapi-schema/leading-comma-list.err      |   2 +-
 tests/qapi-schema/qapi-schema-test.out        |  33 +++
 .../struct-member-alternate-default.err       |   1 +
 .../struct-member-alternate-default.exit      |   1 +
 .../struct-member-alternate-default.out       |   0
 .../struct-member-bool-wrong-default-type.err |   1 +
 ...struct-member-bool-wrong-default-type.exit |   1 +
 .../struct-member-bool-wrong-default-type.out |   0
 .../struct-member-enum-invalid-default.err    |   1 +
 .../struct-member-enum-invalid-default.exit   |   1 +
 .../struct-member-enum-invalid-default.out    |   0
 .../struct-member-enum-wrong-default-type.err |   1 +
 ...struct-member-enum-wrong-default-type.exit |   1 +
 .../struct-member-enum-wrong-default-type.out |   0
 .../struct-member-float-invalid-default.err   |   1 +
 .../struct-member-float-invalid-default.exit  |   1 +
 .../struct-member-float-invalid-default.out   |   0
 ...struct-member-float-wrong-default-type.err |   1 +
 ...truct-member-float-wrong-default-type.exit |   1 +
 ...struct-member-float-wrong-default-type.out |   0
 .../struct-member-int-wrong-default-type.err  |   1 +
 .../struct-member-int-wrong-default-type.exit |   1 +
 .../struct-member-int-wrong-default-type.out  |   0
 .../struct-member-int8-erange-default.err     |   1 +
 .../struct-member-int8-erange-default.exit    |   1 +
 .../struct-member-int8-erange-default.out     |   0
 .../struct-member-list-nonempty-default.err   |   1 +
 .../struct-member-list-nonempty-default.exit  |   1 +
 .../struct-member-list-nonempty-default.out   |   0
 .../struct-member-non-optional-default.err    |   1 +
 .../struct-member-non-optional-default.exit   |   1 +
 .../struct-member-non-optional-default.out    |   0
 .../struct-member-null-default.err            |   1 +
 .../struct-member-null-default.exit           |   1 +
 .../struct-member-null-default.out            |   0
 .../struct-member-str-wrong-default-type.err  |   1 +
 .../struct-member-str-wrong-default-type.exit |   1 +
 .../struct-member-str-wrong-default-type.out  |   0
 .../struct-member-uint8-erange-default.err    |   1 +
 .../struct-member-uint8-erange-default.exit   |   1 +
 .../struct-member-uint8-erange-default.out    |   0
 .../struct-member-uint8-negative-default.err  |   1 +
 .../struct-member-uint8-negative-default.exit |   1 +
 .../struct-member-uint8-negative-default.out  |   0
 tests/qapi-schema/test-qapi.py                |   8 +-
 tests/qemu-iotests/059.out                    |   2 +-
 tests/qemu-iotests/087                        |  65 +++--
 tests/qemu-iotests/087.out                    |  26 +-
 tests/qemu-iotests/089                        |  25 ++
 tests/qemu-iotests/089.out                    |   9 +
 tests/qemu-iotests/099.out                    |   4 +-
 tests/qemu-iotests/110.out                    |   2 +-
 tests/qemu-iotests/198.out                    |   4 +-
 104 files changed, 915 insertions(+), 384 deletions(-)
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.json
 rename tests/qapi-schema/{flat-union-optional-discriminator.json => flat-union-optional-discriminator-no-default.json} (68%)
 create mode 100644 tests/qapi-schema/struct-member-alternate-default.json
 create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.json
 create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.json
 create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.json
 create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.json
 create mode 100644 tests/qapi-schema/struct-member-non-optional-default.json
 create mode 100644 tests/qapi-schema/struct-member-null-default.json
 create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.json
 create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.json
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.err
 rename tests/qapi-schema/{flat-union-optional-discriminator.exit => flat-union-optional-discriminator-invalid-specification.exit} (100%)
 rename tests/qapi-schema/{flat-union-optional-discriminator.out => flat-union-optional-discriminator-invalid-specification.out} (100%)
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out
 delete mode 100644 tests/qapi-schema/flat-union-optional-discriminator.err
 create mode 100644 tests/qapi-schema/struct-member-alternate-default.err
 create mode 100644 tests/qapi-schema/struct-member-alternate-default.exit
 create mode 100644 tests/qapi-schema/struct-member-alternate-default.out
 create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.err
 create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.exit
 create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.out
 create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.err
 create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.exit
 create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.out
 create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.err
 create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.exit
 create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.out
 create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.err
 create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.exit
 create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.out
 create mode 100644 tests/qapi-schema/struct-member-non-optional-default.err
 create mode 100644 tests/qapi-schema/struct-member-non-optional-default.exit
 create mode 100644 tests/qapi-schema/struct-member-non-optional-default.out
 create mode 100644 tests/qapi-schema/struct-member-null-default.err
 create mode 100644 tests/qapi-schema/struct-member-null-default.exit
 create mode 100644 tests/qapi-schema/struct-member-null-default.out
 create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.err
 create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.exit
 create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.out
 create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.err
 create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.exit
 create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.out

-- 
2.21.0



^ permalink raw reply	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-11-14  9:15   ` Markus Armbruster
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 02/14] qapi: Move to_c_string() to common.py Max Reitz
                   ` (16 subsequent siblings)
  17 siblings, 1 reply; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qapi-schema/bad-type-int.json      |  1 -
 tests/qapi-schema/enum-int-member.json   |  1 -
 scripts/qapi/common.py                   | 25 ++++++++++++++++++++----
 scripts/qapi/introspect.py               |  2 ++
 tests/qapi-schema/bad-type-int.err       |  2 +-
 tests/qapi-schema/enum-int-member.err    |  2 +-
 tests/qapi-schema/leading-comma-list.err |  2 +-
 7 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/tests/qapi-schema/bad-type-int.json b/tests/qapi-schema/bad-type-int.json
index 56fc6f8126..81355eb196 100644
--- a/tests/qapi-schema/bad-type-int.json
+++ b/tests/qapi-schema/bad-type-int.json
@@ -1,3 +1,2 @@
 # we reject an expression with a metatype that is not a string
-# FIXME: once the parser understands integer inputs, improve the error message
 { 'struct': 1, 'data': { } }
diff --git a/tests/qapi-schema/enum-int-member.json b/tests/qapi-schema/enum-int-member.json
index 6c9c32e149..6958440c6d 100644
--- a/tests/qapi-schema/enum-int-member.json
+++ b/tests/qapi-schema/enum-int-member.json
@@ -1,3 +1,2 @@
 # we reject any enum member that is not a string
-# FIXME: once the parser understands integer inputs, improve the error message
 { 'enum': 'MyEnum', 'data': [ 1 ] }
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index d61bfdc526..3396ea4a09 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -498,6 +498,8 @@ class QAPISchemaParser(object):
             raise QAPISemError(info, "Unknown pragma '%s'" % name)
 
     def accept(self, skip_comment=True):
+        num_match = re.compile(r'([-+]?inf|nan|[-+0-9.][0-9a-f.ex]*)')
+
         while True:
             self.tok = self.src[self.cursor]
             self.pos = self.cursor
@@ -584,7 +586,22 @@ class QAPISchemaParser(object):
                     return
                 self.line += 1
                 self.line_pos = self.cursor
-            elif not self.tok.isspace():
+            elif self.tok.isspace():
+                pass
+            elif num_match.match(self.src[self.pos:]):
+                match = num_match.match(self.src[self.pos:]).group(0)
+                try:
+                    self.val = int(match, 0)
+                except ValueError:
+                    try:
+                        self.val = float(match)
+                    except ValueError:
+                        raise QAPIParseError(self,
+                                '"%s" is not a valid integer or float' % match)
+
+                self.cursor += len(match) - 1
+                return
+            else:
                 raise QAPIParseError(self, 'Stray "%s"' % self.tok)
 
     def get_members(self):
@@ -617,9 +634,9 @@ class QAPISchemaParser(object):
         if self.tok == ']':
             self.accept()
             return expr
-        if self.tok not in "{['tfn":
+        if self.tok not in "{['tfn-+0123456789.i":
             raise QAPIParseError(self, 'Expected "{", "[", "]", string, '
-                                 'boolean or "null"')
+                                 'boolean, number or "null"')
         while True:
             expr.append(self.get_expr(True))
             if self.tok == ']':
@@ -638,7 +655,7 @@ class QAPISchemaParser(object):
         elif self.tok == '[':
             self.accept()
             expr = self.get_values()
-        elif self.tok in "'tfn":
+        elif self.tok in "'tfn-+0123456789.i":
             expr = self.val
             self.accept()
         else:
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index f62cf0a2e1..6a61dd831f 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -57,6 +57,8 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
         ret += indent(level) + '}))'
     elif isinstance(obj, bool):
         ret += 'QLIT_QBOOL(%s)' % ('true' if obj else 'false')
+    elif isinstance(obj, int) and obj >= -(2 ** 63) and obj < 2 ** 63:
+        ret += 'QLIT_QNUM(%i)' % obj
     else:
         assert False                # not implemented
     if level > 0:
diff --git a/tests/qapi-schema/bad-type-int.err b/tests/qapi-schema/bad-type-int.err
index da89895404..e22fb4f655 100644
--- a/tests/qapi-schema/bad-type-int.err
+++ b/tests/qapi-schema/bad-type-int.err
@@ -1 +1 @@
-tests/qapi-schema/bad-type-int.json:3:13: Stray "1"
+tests/qapi-schema/bad-type-int.json:2: 'struct' key must have a string value
diff --git a/tests/qapi-schema/enum-int-member.err b/tests/qapi-schema/enum-int-member.err
index 071c5213d8..112175f79d 100644
--- a/tests/qapi-schema/enum-int-member.err
+++ b/tests/qapi-schema/enum-int-member.err
@@ -1 +1 @@
-tests/qapi-schema/enum-int-member.json:3:31: Stray "1"
+tests/qapi-schema/enum-int-member.json:2: Member of enum 'MyEnum' requires a string name
diff --git a/tests/qapi-schema/leading-comma-list.err b/tests/qapi-schema/leading-comma-list.err
index f5c870bb9c..fa9c80aa57 100644
--- a/tests/qapi-schema/leading-comma-list.err
+++ b/tests/qapi-schema/leading-comma-list.err
@@ -1 +1 @@
-tests/qapi-schema/leading-comma-list.json:2:13: Expected "{", "[", "]", string, boolean or "null"
+tests/qapi-schema/leading-comma-list.json:2:13: Expected "{", "[", "]", string, boolean, number or "null"
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 02/14] qapi: Move to_c_string() to common.py
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-11-14  9:20   ` Markus Armbruster
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members Max Reitz
                   ` (15 subsequent siblings)
  17 siblings, 1 reply; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

This function will be useful for code generation once we allow default
values, so move it to the other "C helper functions".  In the process,
rewrite it so it supports all nonprintable and non-ASCII characters.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 scripts/qapi/common.py     | 26 ++++++++++++++++++++++++++
 scripts/qapi/introspect.py |  4 ----
 2 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 3396ea4a09..c6754a5856 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -2208,6 +2208,32 @@ def c_fname(filename):
     return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
+# Translates a string to a valid C constant
+def to_c_string(string):
+    result = '"'
+
+    python2 = isinstance(string, bytes)
+    if not python2:
+        # Will return integers when iterated over
+        string = string.encode()
+
+    for c in string:
+        value = ord(c) if python2 else c
+        if value < 0x20 or value > 0x7e:
+            result += '\\%03o' % value
+        else:
+            c = chr(value)
+            if c == '"':
+                result += '\\"'
+            elif c == '\\':
+                result += '\\\\'
+            else:
+                result += c
+
+    result += '"'
+    return result
+
+
 def guardstart(name):
     return mcgen('''
 #ifndef %(name)s
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 6a61dd831f..572e0b8331 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -66,10 +66,6 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
     return ret
 
 
-def to_c_string(string):
-    return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"'
-
-
 class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor):
 
     def __init__(self, prefix, unmask):
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 02/14] qapi: Move to_c_string() to common.py Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-11-14 15:53   ` Markus Armbruster
  2019-11-21 15:07   ` Markus Armbruster
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 04/14] qapi: Allow optional discriminators Max Reitz
                   ` (14 subsequent siblings)
  17 siblings, 2 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

With this change, it is possible to give default values for struct
members, as follows:

  What you had to do so far:

    # @member: Some description, defaults to 42.
    { 'struct': 'Foo',
      'data': { '*member': 'int' } }

  What you can do now:

    { 'struct': 'Foo',
      'data': { '*member': { 'type': 'int', 'default': 42 } }

On the C side, this change would remove Foo.has_member, because
Foo.member is always valid now.  The input visitor deals with setting
it.  (Naturally, this means that such defaults are useful only for input
parameters.)

At least three things are left unimplemented:

First, support for alternate data types.  This is because supporting
them would mean having to allocate the object in the input visitor, and
then potentially going through multiple levels of nested types.  In any
case, it would have been difficult and I do not think there is need for
such support at this point.

Second, support for null.  The most important reason for this is that
introspection already uses "'default': null" for "no default, but this
field is optional".  The second reason is that without support for
alternate data types, there is not really a point in supporting null.

Third, full support for default lists.  This has a similar reason to the
lack of support for alternate data types: Allocating a default list is
not trivial -- unless the list is empty, which is exactly what we have
support for.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/introspect.json       |   9 +-
 scripts/qapi/commands.py   |   2 +-
 scripts/qapi/common.py     | 167 +++++++++++++++++++++++++++++++++++--
 scripts/qapi/doc.py        |  20 ++++-
 scripts/qapi/introspect.py |   2 +-
 scripts/qapi/types.py      |   2 +-
 scripts/qapi/visit.py      |  38 ++++++++-
 7 files changed, 217 insertions(+), 23 deletions(-)

diff --git a/qapi/introspect.json b/qapi/introspect.json
index 1843c1cb17..db703135f9 100644
--- a/qapi/introspect.json
+++ b/qapi/introspect.json
@@ -198,11 +198,10 @@
 #
 # @default: default when used as command parameter.
 #           If absent, the parameter is mandatory.
-#           If present, the value must be null.  The parameter is
-#           optional, and behavior when it's missing is not specified
-#           here.
-#           Future extension: if present and non-null, the parameter
-#           is optional, and defaults to this value.
+#           If present and null, the parameter is optional, and
+#           behavior when it's missing is not specified here.
+#           If present and non-null, the parameter is optional, and
+#           defaults to this value.
 #
 # Since: 2.5
 ##
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index b929e07be4..6c407cd4ba 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -35,7 +35,7 @@ def gen_call(name, arg_type, boxed, ret_type):
     elif arg_type:
         assert not arg_type.variants
         for memb in arg_type.members:
-            if memb.optional:
+            if memb.optional and memb.default is None:
                 argstr += 'arg.has_%s, ' % c_name(memb.name)
             argstr += 'arg.%s, ' % c_name(memb.name)
 
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index c6754a5856..8c57d0c67a 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -14,6 +14,7 @@
 from __future__ import print_function
 from contextlib import contextmanager
 import errno
+import math
 import os
 import re
 import string
@@ -800,6 +801,136 @@ def check_if(expr, info):
         check_if_str(ifcond, info)
 
 
+def check_value_str(info, value):
+    return 'g_strdup(%s)' % to_c_string(value) if type(value) is str else False
+
+def check_value_number(info, value):
+    if type(value) is not float:
+        return False
+    if math.isinf(value):
+        return 'INFINITY' if value > 0 else '-INFINITY'
+    elif math.isnan(value):
+        return 'NAN'
+    else:
+        return '%.16e' % value
+
+def check_value_bool(info, value):
+    if type(value) is not bool:
+        return False
+    return 'true' if value else 'false'
+
+def is_int_type(value):
+    if type(value) is int:
+        return True
+    # 'long' does not exist in Python 3
+    try:
+        if type(value) is long:
+            return True
+    except NameError:
+        pass
+
+    return False
+
+def gen_check_value_int(bits):
+    def check_value_int(info, value):
+        if not is_int_type(value) or \
+           value < -(2 ** (bits - 1)) or value >= 2 ** (bits - 1):
+            return False
+        if bits > 32:
+            return '%ill' % value
+        else:
+            return '%i' % value
+
+    return check_value_int
+
+def gen_check_value_uint(bits):
+    def check_value_uint(info, value):
+        if not is_int_type(value) or value < 0 or value >= 2 ** bits:
+            return False
+        if bits > 32:
+            return '%uull' % value
+        elif bits > 16:
+            return '%uu' % value
+        else:
+            return '%u' % value
+
+    return check_value_uint
+
+# Check whether the given value fits the given QAPI type.
+# If so, return a C representation of the value (pointers point to
+# newly allocated objects).
+# Otherwise, raise an exception.
+def check_value(info, qapi_type, value):
+    builtin_type_checks = {
+        'str':      check_value_str,
+        'int':      gen_check_value_int(64),
+        'number':   check_value_number,
+        'bool':     check_value_bool,
+        'int8':     gen_check_value_int(8),
+        'int16':    gen_check_value_int(16),
+        'int32':    gen_check_value_int(32),
+        'int64':    gen_check_value_int(64),
+        'uint8':    gen_check_value_uint(8),
+        'uint16':   gen_check_value_uint(16),
+        'uint32':   gen_check_value_uint(32),
+        'uint64':   gen_check_value_uint(64),
+        'size':     gen_check_value_uint(64),
+    }
+
+    # Cannot support null because that would require a value of "None"
+    # (which is reserved for no default)
+    unsupported_builtin_types = ['null', 'any', 'QType']
+
+    if type(qapi_type) is list:
+        has_list = True
+        qapi_type = qapi_type[0]
+    elif qapi_type.endswith('List'):
+        has_list = True
+        qapi_type = qapi_type[:-4]
+    else:
+        has_list = False
+
+    if has_list:
+        if value == []:
+            return 'NULL'
+        else:
+            raise QAPISemError(info,
+                "Support for non-empty lists as default values has not been " \
+                "implemented yet: '{}'".format(value))
+
+    if qapi_type in builtin_type_checks:
+        c_val = builtin_type_checks[qapi_type](info, value)
+        if not c_val:
+            raise QAPISemError(info,
+                "Value '{}' does not match type {}".format(value, qapi_type))
+        return c_val
+
+    if qapi_type in unsupported_builtin_types:
+        raise QAPISemError(info,
+                           "Cannot specify values for type %s" % qapi_type)
+
+    if qapi_type in enum_types:
+        if not check_value_str(info, value):
+            raise QAPISemError(info,
+                "Enum values must be strings, but '{}' is no string" \
+                        .format(value))
+
+        enum_values = enum_types[qapi_type]['data']
+        for ev in enum_values:
+            if ev['name'] == value:
+                return c_enum_const(qapi_type, value,
+                                    enum_types[qapi_type].get('prefix'))
+
+        raise QAPISemError(info,
+            "Value '{}' does not occur in enum {}".format(value, qapi_type))
+
+    # TODO: Support alternates
+
+    raise QAPISemError(info,
+        "Cannot specify values for type %s (not built-in or an enum)" %
+        qapi_type)
+
+
 def check_type(info, source, value, allow_array=False,
                allow_dict=False, allow_optional=False,
                allow_metas=[]):
@@ -842,15 +973,22 @@ def check_type(info, source, value, allow_array=False,
         if c_name(key, False) == 'u' or c_name(key, False).startswith('has_'):
             raise QAPISemError(info, "Member of %s uses reserved name '%s'"
                                % (source, key))
-        # Todo: allow dictionaries to represent default values of
-        # an optional argument.
+
         check_known_keys(info, "member '%s' of %s" % (key, source),
-                         arg, ['type'], ['if'])
+                         arg, ['type'], ['if', 'default'])
         check_type(info, "Member '%s' of %s" % (key, source),
                    arg['type'], allow_array=True,
                    allow_metas=['built-in', 'union', 'alternate', 'struct',
                                 'enum'])
 
+        if 'default' in arg:
+            if key[0] != '*':
+                raise QAPISemError(info,
+                    "'%s' is not optional, so it cannot have a default value" %
+                    key)
+
+            check_value(info, arg['type'], arg['default'])
+
 
 def check_command(expr, info):
     name = expr['command']
@@ -1601,13 +1739,14 @@ class QAPISchemaFeature(QAPISchemaMember):
 
 
 class QAPISchemaObjectTypeMember(QAPISchemaMember):
-    def __init__(self, name, typ, optional, ifcond=None):
+    def __init__(self, name, typ, optional, ifcond=None, default=None):
         QAPISchemaMember.__init__(self, name, ifcond)
         assert isinstance(typ, str)
         assert isinstance(optional, bool)
         self._type_name = typ
         self.type = None
         self.optional = optional
+        self.default = default
 
     def check(self, schema):
         assert self.owner
@@ -1917,7 +2056,7 @@ class QAPISchema(object):
             name, info, doc, ifcond,
             self._make_enum_members(data), prefix))
 
-    def _make_member(self, name, typ, ifcond, info):
+    def _make_member(self, name, typ, ifcond, default, info):
         optional = False
         if name.startswith('*'):
             name = name[1:]
@@ -1925,10 +2064,11 @@ class QAPISchema(object):
         if isinstance(typ, list):
             assert len(typ) == 1
             typ = self._make_array_type(typ[0], info)
-        return QAPISchemaObjectTypeMember(name, typ, optional, ifcond)
+        return QAPISchemaObjectTypeMember(name, typ, optional, ifcond, default)
 
     def _make_members(self, data, info):
-        return [self._make_member(key, value['type'], value.get('if'), info)
+        return [self._make_member(key, value['type'], value.get('if'),
+                                  value.get('default'), info)
                 for (key, value) in data.items()]
 
     def _def_struct_type(self, expr, info, doc):
@@ -1951,7 +2091,7 @@ class QAPISchema(object):
             typ = self._make_array_type(typ[0], info)
         typ = self._make_implicit_object_type(
             typ, info, None, self.lookup_type(typ),
-            'wrapper', [self._make_member('data', typ, None, info)])
+            'wrapper', [self._make_member('data', typ, None, None, info)])
         return QAPISchemaObjectTypeVariant(case, typ, ifcond)
 
     def _def_union_type(self, expr, info, doc):
@@ -2234,6 +2374,15 @@ def to_c_string(string):
     return result
 
 
+# Translates a value for the given QAPI type to its C representation.
+# The caller must have called check_value() during parsing to be sure
+# that the given value fits the type.
+def c_value(qapi_type, value):
+    pseudo_info = {'file': '(generator bug)', 'line': 0, 'parent': None}
+    # The caller guarantees this does not raise an exception
+    return check_value(pseudo_info, qapi_type, value)
+
+
 def guardstart(name):
     return mcgen('''
 #ifndef %(name)s
@@ -2356,7 +2505,7 @@ def build_params(arg_type, boxed, extra=None):
         for memb in arg_type.members:
             ret += sep
             sep = ', '
-            if memb.optional:
+            if memb.optional and memb.default is None:
                 ret += 'bool has_%s, ' % c_name(memb.name)
             ret += '%s %s' % (memb.type.c_param_type(),
                               c_name(memb.name))
diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
index 5fc0fc7e06..78a9052738 100755
--- a/scripts/qapi/doc.py
+++ b/scripts/qapi/doc.py
@@ -139,13 +139,29 @@ def texi_enum_value(value, desc, suffix):
         value.name, desc, texi_if(value.ifcond, prefix='@*'))
 
 
+def doc_value(value):
+    if value is True:
+        return 'true'
+    elif value is False:
+        return 'false'
+    elif value is None:
+        return 'null'
+    else:
+        return '{}'.format(value)
+
 def texi_member(member, desc, suffix):
     """Format a table of members item for an object type member"""
     typ = member.type.doc_type()
     membertype = ': ' + typ if typ else ''
+
+    optional_info = ''
+    if member.default is not None:
+        optional_info = ' (optional, default: %s)' % doc_value(member.default)
+    elif member.optional:
+        optional_info = ' (optional)'
+
     return '@item @code{%s%s}%s%s\n%s%s' % (
-        member.name, membertype,
-        ' (optional)' if member.optional else '',
+        member.name, membertype, optional_info,
         suffix, desc, texi_if(member.ifcond, prefix='@*'))
 
 
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 572e0b8331..7d73020a42 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -159,7 +159,7 @@ const QLitObject %(c_name)s = %(c_string)s;
     def _gen_member(self, member):
         ret = {'name': member.name, 'type': self._use_type(member.type)}
         if member.optional:
-            ret['default'] = None
+            ret['default'] = member.default
         if member.ifcond:
             ret = (ret, {'if': member.ifcond})
         return ret
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 3edd9374aa..46a6d33379 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -44,7 +44,7 @@ def gen_struct_members(members):
     ret = ''
     for memb in members:
         ret += gen_if(memb.ifcond)
-        if memb.optional:
+        if memb.optional and memb.default is None:
             ret += mcgen('''
     bool has_%(c_name)s;
 ''',
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 484ebb66ad..0960e25a25 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -40,10 +40,14 @@ def gen_visit_object_members(name, base, members, variants):
 void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
 {
     Error *err = NULL;
-
 ''',
                 c_name=c_name(name))
 
+    if len([m for m in members if m.default is not None]) > 0:
+        ret += mcgen('''
+    bool has_optional;
+''')
+
     if base:
         ret += mcgen('''
     visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, &err);
@@ -53,13 +57,28 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
 ''',
                      c_type=base.c_name())
 
+    ret += mcgen('''
+
+''')
+
     for memb in members:
         ret += gen_if(memb.ifcond)
         if memb.optional:
+            if memb.default is not None:
+                optional_target = 'has_optional'
+                # Visitors other than the input visitor do not have to implement
+                # .optional().  Therefore, we have to initialize has_optional.
+                # Initialize it to true, because the field's value is always
+                # present when using any visitor but the input visitor.
+                ret += mcgen('''
+    has_optional = true;
+''')
+            else:
+                optional_target = 'obj->has_' + c_name(memb.name)
             ret += mcgen('''
-    if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
+    if (visit_optional(v, "%(name)s", &%(opt_target)s)) {
 ''',
-                         name=memb.name, c_name=c_name(memb.name))
+                         name=memb.name, opt_target=optional_target)
             push_indent()
         ret += mcgen('''
     visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, &err);
@@ -69,7 +88,16 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
 ''',
                      c_type=memb.type.c_name(), name=memb.name,
                      c_name=c_name(memb.name))
-        if memb.optional:
+        if memb.default is not None:
+            pop_indent()
+            ret += mcgen('''
+    } else {
+        obj->%(c_name)s = %(c_value)s;
+    }
+''',
+                         c_name=c_name(memb.name),
+                         c_value=c_value(memb._type_name, memb.default))
+        elif memb.optional:
             pop_indent()
             ret += mcgen('''
     }
@@ -287,6 +315,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
         self._add_system_module(None, ' * Built-in QAPI visitors')
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
+#include <math.h>
 #include "qapi/error.h"
 #include "qapi/qapi-builtin-visit.h"
 '''))
@@ -302,6 +331,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
         visit = self._module_basename('qapi-visit', name)
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
+#include <math.h>
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
 #include "%(visit)s.h"
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 04/14] qapi: Allow optional discriminators
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (2 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-11-21 15:13   ` Markus Armbruster
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 05/14] qapi: Document default values for struct members Max Reitz
                   ` (13 subsequent siblings)
  17 siblings, 1 reply; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

Optional discriminators are fine, as long as there is a default value.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 scripts/qapi/common.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 8c57d0c67a..203623795b 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -1052,11 +1052,21 @@ def check_union(expr, info):
         base_members = find_base_members(base)
         assert base_members is not None
 
-        # The value of member 'discriminator' must name a non-optional
-        # member of the base struct.
+        # The value of member 'discriminator' must name a member of
+        # the base struct.  (Optional members are allowed, but the
+        # discriminator name must not start with '*', so keep
+        # allow_optional=False.)
         check_name(info, "Discriminator of flat union '%s'" % name,
                    discriminator)
+
         discriminator_value = base_members.get(discriminator)
+        if not discriminator_value:
+            discriminator_value = base_members.get('*' + discriminator)
+            if discriminator_value and 'default' not in discriminator_value:
+                raise QAPISemError(info,
+                    "Optional discriminator '%s' has no default value" %
+                    discriminator)
+
         if not discriminator_value:
             raise QAPISemError(info,
                                "Discriminator '%s' is not a member of base "
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 05/14] qapi: Document default values for struct members
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (3 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 04/14] qapi: Allow optional discriminators Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 06/14] test-qapi: Print struct members' default values Max Reitz
                   ` (12 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 docs/devel/qapi-code-gen.txt | 81 ++++++++++++++++++++++++++++++------
 1 file changed, 69 insertions(+), 12 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index e8ec8ac1de..9dd7816701 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -334,11 +334,15 @@ Usage: { 'struct': STRING, 'data': DICT, '*base': STRUCT-NAME }
 A struct is a dictionary containing a single 'data' key whose value is
 a dictionary; the dictionary may be empty.  This corresponds to a
 struct in C or an Object in JSON. Each value of the 'data' dictionary
-must be the name of a type, or a one-element array containing a type
-name.  An example of a struct is:
+must be the name of a type, a one-element array containing a type
+name, or a dictionary whose 'type' key gives a type name.  An example
+of a struct is:
 
  { 'struct': 'MyType',
-   'data': { 'member1': 'str', 'member2': 'int', '*member3': 'str' } }
+   'data': { 'member1': 'str',
+             'member2': 'int',
+             '*member3': 'str',
+             '*member4': { 'type': 'bool' } } }
 
 The use of '*' as a prefix to the name means the member is optional in
 the corresponding JSON protocol usage.
@@ -371,6 +375,21 @@ A structure that is used in both input and output of various commands
 must consider the backwards compatibility constraints of both directions
 of use.
 
+Instead of describing the default values for input structures' members
+in the documentation, it is possible to specify it explicitly in the
+struct definition.  In the following example, we let the optional
+'member4' of the above 'MyType' struct default to true:
+
+{ 'struct': 'MyType',
+   'data': { 'member1': 'str',
+             'member2': 'int',
+             '*member3': 'str',
+             '*member4': { 'type': 'bool', 'default': true } } }
+
+In the resulting C interface in QEMU 'member4' will then appear as
+non-optional.  If the client does not specify it, it will be
+automatically set to true.
+
 A struct definition can specify another struct as its base.
 In this case, the members of the base type are included as top-level members
 of the new struct's dictionary in the Client JSON Protocol wire
@@ -472,8 +491,9 @@ All branches of the union must be complex types, and the top-level
 members of the union dictionary on the wire will be combination of
 members from both the base type and the appropriate branch type (when
 merging two dictionaries, there must be no keys in common).  The
-'discriminator' member must be the name of a non-optional enum-typed
-member of the base struct.
+'discriminator' member must be the name of an enum-typed member of the
+base struct.  If that member is optional, a default value must be
+given.
 
 The following example enhances the above simple union example by
 adding an optional common member 'read-only', renaming the
@@ -504,6 +524,24 @@ In the resulting generated C data types, a flat union is
 represented as a struct with the base members included directly, and
 then a union of structures for each branch of the struct.
 
+In the following example, the above BlockdevOptions struct is changed
+so it defaults to the 'file' driver is that field is omitted on the
+wire:
+
+ { 'union': 'BlockdevOptions',
+   'base': {
+       '*driver': { 'type': 'BlockdevDriver', 'default': 'file' },
+       '*read-only': 'bool'
+   },
+   'discriminator': 'driver',
+   'data': { 'file': 'BlockdevOptionsFile',
+             'qcow2': 'BlockdevOptionsQcow2' } }
+
+Now the 'file' JSON object can be abbreviated to:
+
+ { "read-only": true,
+   "filename": "/some/place/my-image" }
+
 A simple union can always be re-written as a flat union where the base
 class has a single member named 'type', and where each branch of the
 union has a struct with a single member named 'data'.  That is,
@@ -922,11 +960,11 @@ and "variants".
 "members" is a JSON array describing the object's common members, if
 any.  Each element is a JSON object with members "name" (the member's
 name), "type" (the name of its type), and optionally "default".  The
-member is optional if "default" is present.  Currently, "default" can
-only have value null.  Other values are reserved for future
-extensions.  The "members" array is in no particular order; clients
-must search the entire object when learning whether a particular
-member is supported.
+member is optional if "default" is present.  If "default" has any
+value but null, that value will be used as the default if the member
+is not specified.  The "members" array is in no particular order;
+clients must search the entire object when learning whether a
+particular member is supported.
 
 Example: the SchemaInfo for MyType from section Struct types
 
@@ -934,7 +972,8 @@ Example: the SchemaInfo for MyType from section Struct types
       "members": [
           { "name": "member1", "type": "str" },
           { "name": "member2", "type": "int" },
-          { "name": "member3", "type": "str", "default": null } ] }
+          { "name": "member3", "type": "str", "default": null },
+          { "name": "member4", "type": "bool", "default": true } ] }
 
 "tag" is the name of the common member serving as type tag.
 "variants" is a JSON array describing the object's variant members.
@@ -1052,7 +1091,9 @@ qmp_my_command(); everything else is produced by the generator.
 
     $ cat example-schema.json
     { 'struct': 'UserDefOne',
-      'data': { 'integer': 'int', '*string': 'str' } }
+      'data': { 'integer': 'int',
+                '*string': 'str',
+                '*defaultbool': { 'type': 'bool', 'default': true } } }
 
     { 'command': 'my-command',
       'data': { 'arg1': ['UserDefOne'] },
@@ -1104,6 +1145,7 @@ Example:
         int64_t integer;
         bool has_string;
         char *string;
+        bool defaultbool;
     };
 
     void qapi_free_UserDefOne(UserDefOne *obj);
@@ -1207,6 +1249,7 @@ Example:
     void visit_type_UserDefOne_members(Visitor *v, UserDefOne *obj, Error **errp)
     {
         Error *err = NULL;
+        bool has_optional;
 
         visit_type_int(v, "integer", &obj->integer, &err);
         if (err) {
@@ -1218,6 +1261,14 @@ Example:
                 goto out;
             }
         }
+        if (visit_optional(v, "defaultbool", &has_optional)) {
+            visit_type_bool(v, "defaultbool", &obj->defaultbool, &err);
+            if (err) {
+                goto out;
+            }
+        } else {
+            obj->defaultbool = true;
+        }
 
     out:
         error_propagate(errp, err);
@@ -1563,6 +1614,12 @@ Example:
                     { "type", QLIT_QSTR("str"), },
                     {}
                 })),
+                QLIT_QDICT(((QLitDictEntry[]) {
+                    { "default", QLIT_QBOOL(true), },
+                    { "name", QLIT_QSTR("defaultbool"), },
+                    { "type", QLIT_QSTR("bool"), },
+                    {}
+                })),
                 {}
             })), },
             { "meta-type", QLIT_QSTR("object"), },
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 06/14] test-qapi: Print struct members' default values
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (4 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 05/14] qapi: Document default values for struct members Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 07/14] tests: Test QAPI default values for struct members Max Reitz
                   ` (11 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qapi-schema/test-qapi.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index b0f770b9bd..320e027d28 100644
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -44,8 +44,12 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
         if base:
             print('    base %s' % base.name)
         for m in members:
-            print('    member %s: %s optional=%s'
-                  % (m.name, m.type.name, m.optional))
+            if m.default is not None:
+                default = ' default={}'.format(m.default)
+            else:
+                default = ''
+            print('    member %s: %s optional=%s%s'
+                  % (m.name, m.type.name, m.optional, default))
             self._print_if(m.ifcond, 8)
         self._print_variants(variants)
         self._print_if(ifcond)
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 07/14] tests: Test QAPI default values for struct members
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (5 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 06/14] test-qapi: Print struct members' default values Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 08/14] tests: Add QAPI optional discriminator tests Max Reitz
                   ` (10 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

This patch adds a number of tests for how not to specify a default for
an optional struct member, and one rather large test on how to do it
right (in qapi-schema-test.json).

As a side effect, this patch tests the QAPI code generator's integer and
float parsing capability.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/Makefile.include                        | 14 ++++++++++
 tests/qapi-schema/qapi-schema-test.json       | 28 +++++++++++++++++++
 .../struct-member-alternate-default.json      | 10 +++++++
 ...struct-member-bool-wrong-default-type.json |  3 ++
 .../struct-member-enum-invalid-default.json   |  4 +++
 ...struct-member-enum-wrong-default-type.json |  4 +++
 .../struct-member-float-invalid-default.json  |  4 +++
 ...truct-member-float-wrong-default-type.json |  3 ++
 .../struct-member-int-wrong-default-type.json |  3 ++
 .../struct-member-int8-erange-default.json    |  3 ++
 .../struct-member-list-nonempty-default.json  |  4 +++
 .../struct-member-non-optional-default.json   |  3 ++
 .../struct-member-null-default.json           |  6 ++++
 .../struct-member-str-wrong-default-type.json |  3 ++
 .../struct-member-uint8-erange-default.json   |  3 ++
 .../struct-member-uint8-negative-default.json |  3 ++
 tests/qapi-schema/qapi-schema-test.out        | 24 ++++++++++++++++
 .../struct-member-alternate-default.err       |  1 +
 .../struct-member-alternate-default.exit      |  1 +
 .../struct-member-alternate-default.out       |  0
 .../struct-member-bool-wrong-default-type.err |  1 +
 ...struct-member-bool-wrong-default-type.exit |  1 +
 .../struct-member-bool-wrong-default-type.out |  0
 .../struct-member-enum-invalid-default.err    |  1 +
 .../struct-member-enum-invalid-default.exit   |  1 +
 .../struct-member-enum-invalid-default.out    |  0
 .../struct-member-enum-wrong-default-type.err |  1 +
 ...struct-member-enum-wrong-default-type.exit |  1 +
 .../struct-member-enum-wrong-default-type.out |  0
 .../struct-member-float-invalid-default.err   |  1 +
 .../struct-member-float-invalid-default.exit  |  1 +
 .../struct-member-float-invalid-default.out   |  0
 ...struct-member-float-wrong-default-type.err |  1 +
 ...truct-member-float-wrong-default-type.exit |  1 +
 ...struct-member-float-wrong-default-type.out |  0
 .../struct-member-int-wrong-default-type.err  |  1 +
 .../struct-member-int-wrong-default-type.exit |  1 +
 .../struct-member-int-wrong-default-type.out  |  0
 .../struct-member-int8-erange-default.err     |  1 +
 .../struct-member-int8-erange-default.exit    |  1 +
 .../struct-member-int8-erange-default.out     |  0
 .../struct-member-list-nonempty-default.err   |  1 +
 .../struct-member-list-nonempty-default.exit  |  1 +
 .../struct-member-list-nonempty-default.out   |  0
 .../struct-member-non-optional-default.err    |  1 +
 .../struct-member-non-optional-default.exit   |  1 +
 .../struct-member-non-optional-default.out    |  0
 .../struct-member-null-default.err            |  1 +
 .../struct-member-null-default.exit           |  1 +
 .../struct-member-null-default.out            |  0
 .../struct-member-str-wrong-default-type.err  |  1 +
 .../struct-member-str-wrong-default-type.exit |  1 +
 .../struct-member-str-wrong-default-type.out  |  0
 .../struct-member-uint8-erange-default.err    |  1 +
 .../struct-member-uint8-erange-default.exit   |  1 +
 .../struct-member-uint8-erange-default.out    |  0
 .../struct-member-uint8-negative-default.err  |  1 +
 .../struct-member-uint8-negative-default.exit |  1 +
 .../struct-member-uint8-negative-default.out  |  0
 59 files changed, 150 insertions(+)
 create mode 100644 tests/qapi-schema/struct-member-alternate-default.json
 create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.json
 create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.json
 create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.json
 create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.json
 create mode 100644 tests/qapi-schema/struct-member-non-optional-default.json
 create mode 100644 tests/qapi-schema/struct-member-null-default.json
 create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.json
 create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.json
 create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.json
 create mode 100644 tests/qapi-schema/struct-member-alternate-default.err
 create mode 100644 tests/qapi-schema/struct-member-alternate-default.exit
 create mode 100644 tests/qapi-schema/struct-member-alternate-default.out
 create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.err
 create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.exit
 create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.out
 create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.err
 create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.exit
 create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.out
 create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.err
 create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.exit
 create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.out
 create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.err
 create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.exit
 create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.out
 create mode 100644 tests/qapi-schema/struct-member-non-optional-default.err
 create mode 100644 tests/qapi-schema/struct-member-non-optional-default.exit
 create mode 100644 tests/qapi-schema/struct-member-non-optional-default.out
 create mode 100644 tests/qapi-schema/struct-member-null-default.err
 create mode 100644 tests/qapi-schema/struct-member-null-default.exit
 create mode 100644 tests/qapi-schema/struct-member-null-default.out
 create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.err
 create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.exit
 create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.out
 create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.err
 create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.exit
 create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.out
 create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.err
 create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.exit
 create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.out

diff --git a/tests/Makefile.include b/tests/Makefile.include
index db750dd6d0..76dc581096 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -451,6 +451,20 @@ qapi-schema += returns-whitelist.json
 qapi-schema += struct-base-clash-deep.json
 qapi-schema += struct-base-clash.json
 qapi-schema += struct-data-invalid.json
+qapi-schema += struct-member-alternate-default.json
+qapi-schema += struct-member-bool-wrong-default-type.json
+qapi-schema += struct-member-enum-invalid-default.json
+qapi-schema += struct-member-enum-wrong-default-type.json
+qapi-schema += struct-member-float-invalid-default.json
+qapi-schema += struct-member-float-wrong-default-type.json
+qapi-schema += struct-member-int-wrong-default-type.json
+qapi-schema += struct-member-int8-erange-default.json
+qapi-schema += struct-member-list-nonempty-default.json
+qapi-schema += struct-member-non-optional-default.json
+qapi-schema += struct-member-null-default.json
+qapi-schema += struct-member-str-wrong-default-type.json
+qapi-schema += struct-member-uint8-erange-default.json
+qapi-schema += struct-member-uint8-negative-default.json
 qapi-schema += struct-member-invalid-dict.json
 qapi-schema += struct-member-invalid.json
 qapi-schema += trailing-comma-list.json
diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index c6d59acc3e..12ae387d46 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -281,3 +281,31 @@
             'cfs1': 'CondFeatureStruct1',
             'cfs2': 'CondFeatureStruct2',
             'cfs3': 'CondFeatureStruct3' } }
+
+# test defaults for optional struct members
+
+{ 'enum': 'TestDefaultEnum',
+  'data': [ 'value1', 'value2', 'value3' ] }
+
+{ 'struct': 'TestDefaultSubStruct',
+  'data': { 'member': 'int' } }
+
+{ 'struct': 'TestDefaultStruct',
+  'data': {
+      '*bool_false': { 'type': 'bool', 'default': false },
+      '*bool_true':  { 'type': 'bool', 'default': true },
+      '*integer':    { 'type': 'int',   'default': -42 },
+      '*int8_low':   { 'type': 'int8',  'default': -0x80 },
+      '*int8_high':  { 'type': 'int8',  'default':  0x7f },
+      '*uint8_high': { 'type': 'uint8', 'default':  0xff },
+      '*size':       { 'type': 'size',  'default':  0xffffffffffffffff },
+      '*fpneg':    { 'type': 'number', 'default': -.375e3 },
+      '*fppos':    { 'type': 'number', 'default': +42. },
+      '*fpposinf': { 'type': 'number', 'default':  inf },
+      '*fpneginf': { 'type': 'number', 'default': -inf },
+      '*fpnan':    { 'type': 'number', 'default': nan },
+      '*str': { 'type': 'str', 'default': 'foo \\鹿"\u007f\b"\'' },
+      '*enum_value2': { 'type': 'TestDefaultEnum', 'default': 'value2' },
+      '*list_scalar': { 'type': ['int'], 'default': [] },
+      '*list_struct': { 'type': ['TestDefaultSubStruct'], 'default': [] }
+  } }
diff --git a/tests/qapi-schema/struct-member-alternate-default.json b/tests/qapi-schema/struct-member-alternate-default.json
new file mode 100644
index 0000000000..0441b3af80
--- /dev/null
+++ b/tests/qapi-schema/struct-member-alternate-default.json
@@ -0,0 +1,10 @@
+# Check defaults for unsupported types (alternates, here)
+# TODO: Implement support for alternates, and then add a test where
+#       default=null so we can see that that is another thing that
+#       should be implemented
+{ 'enum': 'TestEnum', 'data': [ 'value1', 'value2' ] }
+{ 'alternate': 'TestAlt',
+  'data': { 'alt1': 'TestEnum',
+            'alt2': 'null' } }
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'TestAlt', 'default': 'value1' } } }
diff --git a/tests/qapi-schema/struct-member-bool-wrong-default-type.json b/tests/qapi-schema/struct-member-bool-wrong-default-type.json
new file mode 100644
index 0000000000..c9f9c45a87
--- /dev/null
+++ b/tests/qapi-schema/struct-member-bool-wrong-default-type.json
@@ -0,0 +1,3 @@
+# Check that booleans do not accept non-booleans
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'bool', 'default': 0 } } }
diff --git a/tests/qapi-schema/struct-member-enum-invalid-default.json b/tests/qapi-schema/struct-member-enum-invalid-default.json
new file mode 100644
index 0000000000..d5724970be
--- /dev/null
+++ b/tests/qapi-schema/struct-member-enum-invalid-default.json
@@ -0,0 +1,4 @@
+# Check that enum values do not accept anything that is not part of the enum
+{ 'enum': 'TestEnum', 'data': [ 'value1', 'value2' ] }
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'TestEnum', 'default': 'value3' } } }
diff --git a/tests/qapi-schema/struct-member-enum-wrong-default-type.json b/tests/qapi-schema/struct-member-enum-wrong-default-type.json
new file mode 100644
index 0000000000..98f18b462c
--- /dev/null
+++ b/tests/qapi-schema/struct-member-enum-wrong-default-type.json
@@ -0,0 +1,4 @@
+# Check that enum values do not accept non-strings
+{ 'enum': 'TestEnum', 'data': [ 'value1', 'value2' ] }
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'TestEnum', 'default': 0 } } }
diff --git a/tests/qapi-schema/struct-member-float-invalid-default.json b/tests/qapi-schema/struct-member-float-invalid-default.json
new file mode 100644
index 0000000000..ba351a91c8
--- /dev/null
+++ b/tests/qapi-schema/struct-member-float-invalid-default.json
@@ -0,0 +1,4 @@
+# This matches our number regex, but it is an invalid float still --
+# test that it is rejected
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'number', 'default': 4.42. } } }
diff --git a/tests/qapi-schema/struct-member-float-wrong-default-type.json b/tests/qapi-schema/struct-member-float-wrong-default-type.json
new file mode 100644
index 0000000000..89b6863307
--- /dev/null
+++ b/tests/qapi-schema/struct-member-float-wrong-default-type.json
@@ -0,0 +1,3 @@
+# Check that floats do not accept non-floats
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'number', 'default': 42 } } }
diff --git a/tests/qapi-schema/struct-member-int-wrong-default-type.json b/tests/qapi-schema/struct-member-int-wrong-default-type.json
new file mode 100644
index 0000000000..6c6c752176
--- /dev/null
+++ b/tests/qapi-schema/struct-member-int-wrong-default-type.json
@@ -0,0 +1,3 @@
+# Check that integers do not accept non-integers
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'int', 'default': 0.0 } } }
diff --git a/tests/qapi-schema/struct-member-int8-erange-default.json b/tests/qapi-schema/struct-member-int8-erange-default.json
new file mode 100644
index 0000000000..a0093bc4b9
--- /dev/null
+++ b/tests/qapi-schema/struct-member-int8-erange-default.json
@@ -0,0 +1,3 @@
+# int8 does not accept out-of-range values
+{ 'struct': 'Test',
+  'data': { '*s8': { 'type': 'int8', 'default': -0x81 } } }
diff --git a/tests/qapi-schema/struct-member-list-nonempty-default.json b/tests/qapi-schema/struct-member-list-nonempty-default.json
new file mode 100644
index 0000000000..604692631b
--- /dev/null
+++ b/tests/qapi-schema/struct-member-list-nonempty-default.json
@@ -0,0 +1,4 @@
+# Currently, we do not support nonempty defaults for lists
+# TODO: Implement support
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': ['int'], 'default': [42] } } }
diff --git a/tests/qapi-schema/struct-member-non-optional-default.json b/tests/qapi-schema/struct-member-non-optional-default.json
new file mode 100644
index 0000000000..354b34167d
--- /dev/null
+++ b/tests/qapi-schema/struct-member-non-optional-default.json
@@ -0,0 +1,3 @@
+# Only optional members can have defaults
+{ 'struct': 'Test',
+  'data': { 'member': { 'type': 'int', 'default': 42 } } }
diff --git a/tests/qapi-schema/struct-member-null-default.json b/tests/qapi-schema/struct-member-null-default.json
new file mode 100644
index 0000000000..c0cf70b984
--- /dev/null
+++ b/tests/qapi-schema/struct-member-null-default.json
@@ -0,0 +1,6 @@
+# Currently, you cannot give a default for null members, because
+# 'None' is reserved for 'no default'.
+# TODO: Fix that, though it would only really make sense for
+#       alternates (which currently cannot receive defaults either)
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'null', 'default': null } } }
diff --git a/tests/qapi-schema/struct-member-str-wrong-default-type.json b/tests/qapi-schema/struct-member-str-wrong-default-type.json
new file mode 100644
index 0000000000..d460d3a9b4
--- /dev/null
+++ b/tests/qapi-schema/struct-member-str-wrong-default-type.json
@@ -0,0 +1,3 @@
+# Check that strings do not accept non-strings
+{ 'struct': 'Test',
+  'data': { '*member': { 'type': 'str', 'default': null } } }
diff --git a/tests/qapi-schema/struct-member-uint8-erange-default.json b/tests/qapi-schema/struct-member-uint8-erange-default.json
new file mode 100644
index 0000000000..a4fe7fd918
--- /dev/null
+++ b/tests/qapi-schema/struct-member-uint8-erange-default.json
@@ -0,0 +1,3 @@
+# uint8 does not accept out-of-range values
+{ 'struct': 'Test',
+  'data': { '*u8': { 'type': 'uint8', 'default': 0x100 } } }
diff --git a/tests/qapi-schema/struct-member-uint8-negative-default.json b/tests/qapi-schema/struct-member-uint8-negative-default.json
new file mode 100644
index 0000000000..62bbf86af9
--- /dev/null
+++ b/tests/qapi-schema/struct-member-uint8-negative-default.json
@@ -0,0 +1,3 @@
+# uint8 does not accept negative values
+{ 'struct': 'Test',
+  'data': { '*u8': { 'type': 'uint8', 'default': -1 } } }
diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
index 85d510bc00..724276f5de 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -397,3 +397,27 @@ object q_obj_test-features-arg
     member cfs3: CondFeatureStruct3 optional=False
 command test-features q_obj_test-features-arg -> None
    gen=True success_response=True boxed=False oob=False preconfig=False
+enum TestDefaultEnum
+    member value1
+    member value2
+    member value3
+object TestDefaultSubStruct
+    member member: int optional=False
+array TestDefaultSubStructList TestDefaultSubStruct
+object TestDefaultStruct
+    member bool_false: bool optional=True default=False
+    member bool_true: bool optional=True default=True
+    member integer: int optional=True default=-42
+    member int8_low: int8 optional=True default=-128
+    member int8_high: int8 optional=True default=127
+    member uint8_high: uint8 optional=True default=255
+    member size: size optional=True default=18446744073709551615
+    member fpneg: number optional=True default=-375.0
+    member fppos: number optional=True default=42.0
+    member fpposinf: number optional=True default=inf
+    member fpneginf: number optional=True default=-inf
+    member fpnan: number optional=True default=nan
+    member str: str optional=True default=foo \鹿"\x7f\b"'
+    member enum_value2: TestDefaultEnum optional=True default=value2
+    member list_scalar: intList optional=True default=[]
+    member list_struct: TestDefaultSubStructList optional=True default=[]
diff --git a/tests/qapi-schema/struct-member-alternate-default.err b/tests/qapi-schema/struct-member-alternate-default.err
new file mode 100644
index 0000000000..48314475f9
--- /dev/null
+++ b/tests/qapi-schema/struct-member-alternate-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-alternate-default.json:9: Cannot specify values for type TestAlt (not built-in or an enum)
diff --git a/tests/qapi-schema/struct-member-alternate-default.exit b/tests/qapi-schema/struct-member-alternate-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-alternate-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-alternate-default.out b/tests/qapi-schema/struct-member-alternate-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-bool-wrong-default-type.err b/tests/qapi-schema/struct-member-bool-wrong-default-type.err
new file mode 100644
index 0000000000..a0095e5c83
--- /dev/null
+++ b/tests/qapi-schema/struct-member-bool-wrong-default-type.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-bool-wrong-default-type.json:2: Value '0' does not match type bool
diff --git a/tests/qapi-schema/struct-member-bool-wrong-default-type.exit b/tests/qapi-schema/struct-member-bool-wrong-default-type.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-bool-wrong-default-type.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-bool-wrong-default-type.out b/tests/qapi-schema/struct-member-bool-wrong-default-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-enum-invalid-default.err b/tests/qapi-schema/struct-member-enum-invalid-default.err
new file mode 100644
index 0000000000..6aa3684a28
--- /dev/null
+++ b/tests/qapi-schema/struct-member-enum-invalid-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-enum-invalid-default.json:3: Value 'value3' does not occur in enum TestEnum
diff --git a/tests/qapi-schema/struct-member-enum-invalid-default.exit b/tests/qapi-schema/struct-member-enum-invalid-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-enum-invalid-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-enum-invalid-default.out b/tests/qapi-schema/struct-member-enum-invalid-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-enum-wrong-default-type.err b/tests/qapi-schema/struct-member-enum-wrong-default-type.err
new file mode 100644
index 0000000000..846dad0316
--- /dev/null
+++ b/tests/qapi-schema/struct-member-enum-wrong-default-type.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-enum-wrong-default-type.json:3: Enum values must be strings, but '0' is no string
diff --git a/tests/qapi-schema/struct-member-enum-wrong-default-type.exit b/tests/qapi-schema/struct-member-enum-wrong-default-type.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-enum-wrong-default-type.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-enum-wrong-default-type.out b/tests/qapi-schema/struct-member-enum-wrong-default-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-float-invalid-default.err b/tests/qapi-schema/struct-member-float-invalid-default.err
new file mode 100644
index 0000000000..1ae241641f
--- /dev/null
+++ b/tests/qapi-schema/struct-member-float-invalid-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-float-invalid-default.json:4:55: "4.42." is not a valid integer or float
diff --git a/tests/qapi-schema/struct-member-float-invalid-default.exit b/tests/qapi-schema/struct-member-float-invalid-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-float-invalid-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-float-invalid-default.out b/tests/qapi-schema/struct-member-float-invalid-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-float-wrong-default-type.err b/tests/qapi-schema/struct-member-float-wrong-default-type.err
new file mode 100644
index 0000000000..8f8bf68d23
--- /dev/null
+++ b/tests/qapi-schema/struct-member-float-wrong-default-type.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-float-wrong-default-type.json:2: Value '42' does not match type number
diff --git a/tests/qapi-schema/struct-member-float-wrong-default-type.exit b/tests/qapi-schema/struct-member-float-wrong-default-type.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-float-wrong-default-type.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-float-wrong-default-type.out b/tests/qapi-schema/struct-member-float-wrong-default-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-int-wrong-default-type.err b/tests/qapi-schema/struct-member-int-wrong-default-type.err
new file mode 100644
index 0000000000..a6b93e8403
--- /dev/null
+++ b/tests/qapi-schema/struct-member-int-wrong-default-type.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-int-wrong-default-type.json:2: Value '0.0' does not match type int
diff --git a/tests/qapi-schema/struct-member-int-wrong-default-type.exit b/tests/qapi-schema/struct-member-int-wrong-default-type.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-int-wrong-default-type.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-int-wrong-default-type.out b/tests/qapi-schema/struct-member-int-wrong-default-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-int8-erange-default.err b/tests/qapi-schema/struct-member-int8-erange-default.err
new file mode 100644
index 0000000000..e58bf7279c
--- /dev/null
+++ b/tests/qapi-schema/struct-member-int8-erange-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-int8-erange-default.json:2: Value '-129' does not match type int8
diff --git a/tests/qapi-schema/struct-member-int8-erange-default.exit b/tests/qapi-schema/struct-member-int8-erange-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-int8-erange-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-int8-erange-default.out b/tests/qapi-schema/struct-member-int8-erange-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-list-nonempty-default.err b/tests/qapi-schema/struct-member-list-nonempty-default.err
new file mode 100644
index 0000000000..48b6fd61a6
--- /dev/null
+++ b/tests/qapi-schema/struct-member-list-nonempty-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-list-nonempty-default.json:3: Support for non-empty lists as default values has not been implemented yet: '[42]'
diff --git a/tests/qapi-schema/struct-member-list-nonempty-default.exit b/tests/qapi-schema/struct-member-list-nonempty-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-list-nonempty-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-list-nonempty-default.out b/tests/qapi-schema/struct-member-list-nonempty-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-non-optional-default.err b/tests/qapi-schema/struct-member-non-optional-default.err
new file mode 100644
index 0000000000..ba79cb43fa
--- /dev/null
+++ b/tests/qapi-schema/struct-member-non-optional-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-non-optional-default.json:2: 'member' is not optional, so it cannot have a default value
diff --git a/tests/qapi-schema/struct-member-non-optional-default.exit b/tests/qapi-schema/struct-member-non-optional-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-non-optional-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-non-optional-default.out b/tests/qapi-schema/struct-member-non-optional-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-null-default.err b/tests/qapi-schema/struct-member-null-default.err
new file mode 100644
index 0000000000..6a37655f3d
--- /dev/null
+++ b/tests/qapi-schema/struct-member-null-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-null-default.json:5: Cannot specify values for type null
diff --git a/tests/qapi-schema/struct-member-null-default.exit b/tests/qapi-schema/struct-member-null-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-null-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-null-default.out b/tests/qapi-schema/struct-member-null-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-str-wrong-default-type.err b/tests/qapi-schema/struct-member-str-wrong-default-type.err
new file mode 100644
index 0000000000..597165e22a
--- /dev/null
+++ b/tests/qapi-schema/struct-member-str-wrong-default-type.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-str-wrong-default-type.json:2: Value 'None' does not match type str
diff --git a/tests/qapi-schema/struct-member-str-wrong-default-type.exit b/tests/qapi-schema/struct-member-str-wrong-default-type.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-str-wrong-default-type.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-str-wrong-default-type.out b/tests/qapi-schema/struct-member-str-wrong-default-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-uint8-erange-default.err b/tests/qapi-schema/struct-member-uint8-erange-default.err
new file mode 100644
index 0000000000..a870c7074e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-uint8-erange-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-uint8-erange-default.json:2: Value '256' does not match type uint8
diff --git a/tests/qapi-schema/struct-member-uint8-erange-default.exit b/tests/qapi-schema/struct-member-uint8-erange-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-uint8-erange-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-uint8-erange-default.out b/tests/qapi-schema/struct-member-uint8-erange-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/struct-member-uint8-negative-default.err b/tests/qapi-schema/struct-member-uint8-negative-default.err
new file mode 100644
index 0000000000..e17ef38e15
--- /dev/null
+++ b/tests/qapi-schema/struct-member-uint8-negative-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/struct-member-uint8-negative-default.json:2: Value '-1' does not match type uint8
diff --git a/tests/qapi-schema/struct-member-uint8-negative-default.exit b/tests/qapi-schema/struct-member-uint8-negative-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/struct-member-uint8-negative-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/struct-member-uint8-negative-default.out b/tests/qapi-schema/struct-member-uint8-negative-default.out
new file mode 100644
index 0000000000..e69de29bb2
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 08/14] tests: Add QAPI optional discriminator tests
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (6 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 07/14] tests: Test QAPI default values for struct members Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 09/14] qapi: Formalize qcow2 encryption probing Max Reitz
                   ` (9 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

There already is an optional discriminator test, although it also noted
the discriminator name itself as optional.  This already gives us one
error test case, to which this patch adds one other, namely whether that
using an optional discriminator requires the respective field to have a
default value.

Furthermore, a passing test case is added to qapi-schema-test.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/Makefile.include                                |  3 ++-
 ...-optional-discriminator-invalid-specification.json | 11 +++++++++++
 ...flat-union-optional-discriminator-no-default.json} |  5 +++--
 tests/qapi-schema/qapi-schema-test.json               | 10 ++++++++++
 ...n-optional-discriminator-invalid-specification.err |  1 +
 ...optional-discriminator-invalid-specification.exit} |  0
 ...-optional-discriminator-invalid-specification.out} |  0
 .../flat-union-optional-discriminator-no-default.err  |  1 +
 .../flat-union-optional-discriminator-no-default.exit |  1 +
 .../flat-union-optional-discriminator-no-default.out  |  0
 .../qapi-schema/flat-union-optional-discriminator.err |  1 -
 tests/qapi-schema/qapi-schema-test.out                |  9 +++++++++
 12 files changed, 38 insertions(+), 4 deletions(-)
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.json
 rename tests/qapi-schema/{flat-union-optional-discriminator.json => flat-union-optional-discriminator-no-default.json} (68%)
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.err
 rename tests/qapi-schema/{flat-union-optional-discriminator.exit => flat-union-optional-discriminator-invalid-specification.exit} (100%)
 rename tests/qapi-schema/{flat-union-optional-discriminator.out => flat-union-optional-discriminator-invalid-specification.out} (100%)
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out
 delete mode 100644 tests/qapi-schema/flat-union-optional-discriminator.err

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 76dc581096..3202ddab10 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -398,7 +398,8 @@ qapi-schema += flat-union-invalid-branch-key.json
 qapi-schema += flat-union-invalid-discriminator.json
 qapi-schema += flat-union-invalid-if-discriminator.json
 qapi-schema += flat-union-no-base.json
-qapi-schema += flat-union-optional-discriminator.json
+qapi-schema += flat-union-optional-discriminator-invalid-specification.json
+qapi-schema += flat-union-optional-discriminator-no-default.json
 qapi-schema += flat-union-string-discriminator.json
 qapi-schema += funny-char.json
 qapi-schema += ident-with-escape.json
diff --git a/tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.json b/tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.json
new file mode 100644
index 0000000000..d20a2cd295
--- /dev/null
+++ b/tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.json
@@ -0,0 +1,11 @@
+# For using optional discriminators, only the field in the base struct
+# must be marked optional, not the discriminator name itself
+{ 'enum': 'Enum', 'data': [ 'one', 'two' ] }
+{ 'struct': 'Base',
+  'data': { '*switch': { 'type': 'Enum', 'default': 'one' } } }
+{ 'struct': 'Branch', 'data': { 'name': 'str' } }
+{ 'union': 'MyUnion',
+  'base': 'Base',
+  'discriminator': '*switch',
+  'data': { 'one': 'Branch',
+            'two': 'Branch' } }
diff --git a/tests/qapi-schema/flat-union-optional-discriminator.json b/tests/qapi-schema/flat-union-optional-discriminator-no-default.json
similarity index 68%
rename from tests/qapi-schema/flat-union-optional-discriminator.json
rename to tests/qapi-schema/flat-union-optional-discriminator-no-default.json
index 08a8f7ef8b..31ebb85afb 100644
--- a/tests/qapi-schema/flat-union-optional-discriminator.json
+++ b/tests/qapi-schema/flat-union-optional-discriminator-no-default.json
@@ -1,10 +1,11 @@
-# we require the discriminator to be non-optional
+# Using an optional discriminator requires the respective field to
+# have a default
 { 'enum': 'Enum', 'data': [ 'one', 'two' ] }
 { 'struct': 'Base',
   'data': { '*switch': 'Enum' } }
 { 'struct': 'Branch', 'data': { 'name': 'str' } }
 { 'union': 'MyUnion',
   'base': 'Base',
-  'discriminator': '*switch',
+  'discriminator': 'switch',
   'data': { 'one': 'Branch',
             'two': 'Branch' } }
diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index 12ae387d46..0f4b123a82 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -104,6 +104,16 @@
 { 'struct': 'UserDefC',
   'data': { 'string1': 'str', 'string2': 'str' } }
 
+# for testing unions with an optional discriminator
+{ 'union': 'UserDefFlatUnion3',
+  'base': { '*enum1': { 'type': 'EnumOne', 'default': 'value1' } },
+  'discriminator': 'enum1',
+  'data': { 'value1' : 'UserDefA',
+            'value2' : 'UserDefB',
+            'value3' : 'UserDefB'
+            # 'value4' defaults to empty
+  } }
+
 # for testing use of 'number' within alternates
 { 'alternate': 'AltEnumBool', 'data': { 'e': 'EnumOne', 'b': 'bool' } }
 { 'alternate': 'AltEnumNum', 'data': { 'e': 'EnumOne', 'n': 'number' } }
diff --git a/tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.err b/tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.err
new file mode 100644
index 0000000000..cbf154e726
--- /dev/null
+++ b/tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.err
@@ -0,0 +1 @@
+tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.json:7: Discriminator of flat union 'MyUnion' does not allow optional name '*switch'
diff --git a/tests/qapi-schema/flat-union-optional-discriminator.exit b/tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.exit
similarity index 100%
rename from tests/qapi-schema/flat-union-optional-discriminator.exit
rename to tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.exit
diff --git a/tests/qapi-schema/flat-union-optional-discriminator.out b/tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.out
similarity index 100%
rename from tests/qapi-schema/flat-union-optional-discriminator.out
rename to tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.out
diff --git a/tests/qapi-schema/flat-union-optional-discriminator-no-default.err b/tests/qapi-schema/flat-union-optional-discriminator-no-default.err
new file mode 100644
index 0000000000..5a022a0bc7
--- /dev/null
+++ b/tests/qapi-schema/flat-union-optional-discriminator-no-default.err
@@ -0,0 +1 @@
+tests/qapi-schema/flat-union-optional-discriminator-no-default.json:7: Optional discriminator 'switch' has no default value
diff --git a/tests/qapi-schema/flat-union-optional-discriminator-no-default.exit b/tests/qapi-schema/flat-union-optional-discriminator-no-default.exit
new file mode 100644
index 0000000000..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/flat-union-optional-discriminator-no-default.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/flat-union-optional-discriminator-no-default.out b/tests/qapi-schema/flat-union-optional-discriminator-no-default.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/flat-union-optional-discriminator.err b/tests/qapi-schema/flat-union-optional-discriminator.err
deleted file mode 100644
index aaabedb3bd..0000000000
--- a/tests/qapi-schema/flat-union-optional-discriminator.err
+++ /dev/null
@@ -1 +0,0 @@
-tests/qapi-schema/flat-union-optional-discriminator.json:6: Discriminator of flat union 'MyUnion' does not allow optional name '*switch'
diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
index 724276f5de..3729736747 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -96,6 +96,15 @@ alternate UserDefAlternate
 object UserDefC
     member string1: str optional=False
     member string2: str optional=False
+object q_obj_UserDefFlatUnion3-base
+    member enum1: EnumOne optional=True default=value1
+object UserDefFlatUnion3
+    base q_obj_UserDefFlatUnion3-base
+    tag enum1
+    case value1: UserDefA
+    case value2: UserDefB
+    case value3: UserDefB
+    case value4: q_empty
 alternate AltEnumBool
     tag type
     case e: EnumOne
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 09/14] qapi: Formalize qcow2 encryption probing
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (7 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 08/14] tests: Add QAPI optional discriminator tests Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 10/14] qapi: Formalize qcow " Max Reitz
                   ` (8 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

Currently, you can give no encryption format for a qcow2 file while
still passing a key-secret.  That does not conform to the schema, so
this patch changes the schema to allow it.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 32 +++++++++++++++++++++++++++++---
 block/qcow2.c        |  3 +++
 2 files changed, 32 insertions(+), 3 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 0d43d4f37c..9df3fc8bd7 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -47,6 +47,9 @@
 ##
 # @ImageInfoSpecificQCow2Encryption:
 #
+# @format will never be "auto", as this pseudo-format just tells the
+# qcow2 driver to read the actual format from the image header.
+#
 # Since: 2.10
 ##
 { 'union': 'ImageInfoSpecificQCow2Encryption',
@@ -3081,10 +3084,30 @@
 # @BlockdevQcow2EncryptionFormat:
 # @aes: AES-CBC with plain64 initialization vectors
 #
+# @auto:            Determine the encryption format from the image
+#                   header.  This only allows the use of the
+#                   key-secret option.  (Since: 4.1)
+#
 # Since: 2.10
 ##
 { 'enum': 'BlockdevQcow2EncryptionFormat',
-  'data': [ 'aes', 'luks' ] }
+  'data': [ 'aes', 'luks', 'auto' ] }
+
+##
+# @BlockdevQcow2EncryptionSecret:
+#
+# Allows specifying a key-secret without specifying the exact
+# encryption format, which is determined automatically from the image
+# header.
+#
+# @key-secret:      The ID of a QCryptoSecret object providing the
+#                   decryption key.  Mandatory except when probing
+#                   image for metadata only.
+#
+# Since: 4.1
+##
+{ 'struct': 'BlockdevQcow2EncryptionSecret',
+  'data': { '*key-secret': 'str' } }
 
 ##
 # @BlockdevQcow2Encryption:
@@ -3092,10 +3115,13 @@
 # Since: 2.10
 ##
 { 'union': 'BlockdevQcow2Encryption',
-  'base': { 'format': 'BlockdevQcow2EncryptionFormat' },
+  'base': {
+      '*format': { 'type': 'BlockdevQcow2EncryptionFormat', 'default': 'auto' }
+  },
   'discriminator': 'format',
   'data': { 'aes': 'QCryptoBlockOptionsQCow',
-            'luks': 'QCryptoBlockOptionsLUKS'} }
+            'luks': 'QCryptoBlockOptionsLUKS',
+            'auto': 'BlockdevQcow2EncryptionSecret' } }
 
 ##
 # @BlockdevOptionsQcow2:
diff --git a/block/qcow2.c b/block/qcow2.c
index 9396d490d5..95de19d906 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -930,6 +930,9 @@ static int qcow2_update_options_prepare(BlockDriverState *bs,
 
     qdict_extract_subqdict(options, &encryptopts, "encrypt.");
     encryptfmt = qdict_get_try_str(encryptopts, "format");
+    if (!g_strcmp0(encryptfmt, "auto")) {
+        encryptfmt = NULL;
+    }
 
     opts = qemu_opts_create(&qcow2_runtime_opts, NULL, 0, &error_abort);
     qemu_opts_absorb_qdict(opts, options, &local_err);
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 10/14] qapi: Formalize qcow encryption probing
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (8 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 09/14] qapi: Formalize qcow2 encryption probing Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 11/14] block: Try to create well typed json:{} filenames Max Reitz
                   ` (7 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

qcow only supports a single encryption (and there is no reason why that
would change in the future), so we can make it the default.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 9df3fc8bd7..b30a19bf8e 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -3059,7 +3059,9 @@
 # Since: 2.10
 ##
 { 'union': 'BlockdevQcowEncryption',
-  'base': { 'format': 'BlockdevQcowEncryptionFormat' },
+  'base': {
+      '*format': { 'type': 'BlockdevQcowEncryptionFormat', 'default': 'aes' }
+  },
   'discriminator': 'format',
   'data': { 'aes': 'QCryptoBlockOptionsQCow' } }
 
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 11/14] block: Try to create well typed json:{} filenames
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (9 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 10/14] qapi: Formalize qcow " Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 20:06   ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 12/14] iotests: Test internal option typing Max Reitz
                   ` (6 subsequent siblings)
  17 siblings, 1 reply; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

By applying qdict_flatten(), the flat-confused input visitor, and the
output visitor, we can at least try to bring bs->full_open_options into
accordance with the QAPI schema.  This may not always work (there are
some options left that have not been QAPI-fied yet), but in practice it
usually will.

In any case, sometimes emitting wrongly typed json:{} filenames is
better than doing it effectively half the time.

This affects some iotests because json:{} filenames are now usually
crumpled.  In 198, "format": "auto" now appears in the qcow2 encryption
options because going through a visitor makes optional members' default
values explicit.

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1534396
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c                    | 68 +++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/059.out |  2 +-
 tests/qemu-iotests/099.out |  4 +--
 tests/qemu-iotests/110.out |  2 +-
 tests/qemu-iotests/198.out |  4 +--
 5 files changed, 73 insertions(+), 7 deletions(-)

diff --git a/block.c b/block.c
index c139540f2b..d3c1041087 100644
--- a/block.c
+++ b/block.c
@@ -36,6 +36,7 @@
 #include "qapi/qmp/qjson.h"
 #include "qapi/qmp/qnull.h"
 #include "qapi/qmp/qstring.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qobject-output-visitor.h"
 #include "qapi/qapi-visit-block-core.h"
 #include "sysemu/block-backend.h"
@@ -6283,6 +6284,56 @@ static bool bdrv_backing_overridden(BlockDriverState *bs)
     }
 }
 
+/**
+ * Take a blockdev @options QDict and convert its values to the
+ * correct type.
+ *
+ * Fail if @options does not match the QAPI schema of BlockdevOptions.
+ *
+ * In case of failure, return NULL and set @errp.
+ *
+ * In case of success, return a correctly typed new QDict.
+ */
+static QDict *bdrv_type_blockdev_opts(const QDict *options, Error **errp)
+{
+    Visitor *v;
+    BlockdevOptions *blockdev_options;
+    QObject *typed_opts;
+    QDict *string_options;
+    Error *local_err = NULL;
+
+    string_options = qdict_clone_shallow(options);
+
+    qdict_flatten(string_options);
+    v = qobject_input_visitor_new_flat_confused(string_options, errp);
+    if (!v) {
+        error_prepend(errp, "Failed to prepare options: ");
+        return NULL;
+    }
+
+    visit_type_BlockdevOptions(v, NULL, &blockdev_options, &local_err);
+    visit_free(v);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        error_prepend(errp, "Not a valid BlockdevOptions object: ");
+        return NULL;
+    }
+
+    v = qobject_output_visitor_new(&typed_opts);
+    visit_type_BlockdevOptions(v, NULL, &blockdev_options, &local_err);
+    if (!local_err) {
+        visit_complete(v, &typed_opts);
+    }
+    visit_free(v);
+    qapi_free_BlockdevOptions(blockdev_options);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return NULL;
+    }
+
+    return qobject_to(QDict, typed_opts);
+}
+
 /* Updates the following BDS fields:
  *  - exact_filename: A filename which may be used for opening a block device
  *                    which (mostly) equals the given BDS (even without any
@@ -6400,10 +6451,25 @@ void bdrv_refresh_filename(BlockDriverState *bs)
     if (bs->exact_filename[0]) {
         pstrcpy(bs->filename, sizeof(bs->filename), bs->exact_filename);
     } else {
-        QString *json = qobject_to_json(QOBJECT(bs->full_open_options));
+        QString *json;
+        QDict *typed_opts, *json_opts;
+
+        typed_opts = bdrv_type_blockdev_opts(bs->full_open_options, NULL);
+
+        /*
+         * We cannot be certain that bs->full_open_options matches
+         * BlockdevOptions, so bdrv_type_blockdev_opts() may fail.
+         * That is not fatal, we can just emit bs->full_open_options
+         * directly -- qemu will accept that, even if it does not
+         * match the schema.
+         */
+        json_opts = typed_opts ?: bs->full_open_options;
+
+        json = qobject_to_json(QOBJECT(json_opts));
         snprintf(bs->filename, sizeof(bs->filename), "json:%s",
                  qstring_get_str(json));
         qobject_unref(json);
+        qobject_unref(typed_opts);
     }
 }
 
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index 4fab42a28c..53109b2d49 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -2050,7 +2050,7 @@ wrote 512/512 bytes at offset 10240
 
 === Testing monolithicFlat with internally generated JSON file name ===
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
-qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
+qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"inject-error": [{"event": "read_aio"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}'
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
diff --git a/tests/qemu-iotests/099.out b/tests/qemu-iotests/099.out
index 8cce627529..0a9c434148 100644
--- a/tests/qemu-iotests/099.out
+++ b/tests/qemu-iotests/099.out
@@ -12,11 +12,11 @@ blkverify:TEST_DIR/t.IMGFMT.compare:TEST_DIR/t.IMGFMT
 
 === Testing JSON filename for blkdebug ===
 
-json:{"driver": "IMGFMT", "file": {"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "l1_update"}}
+json:{"driver": "IMGFMT", "file": {"inject-error": [{"event": "l1_update"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}
 
 === Testing indirectly enforced JSON filename ===
 
-json:{"driver": "raw", "file": {"test": {"driver": "IMGFMT", "file": {"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "l1_update"}}, "driver": "blkverify", "raw": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.compare"}}}
+json:{"driver": "raw", "file": {"test": {"driver": "IMGFMT", "file": {"inject-error": [{"event": "l1_update"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}, "driver": "blkverify", "raw": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.compare"}}}
 
 === Testing plain filename for blkdebug ===
 
diff --git a/tests/qemu-iotests/110.out b/tests/qemu-iotests/110.out
index f60b26390e..d95b92bee7 100644
--- a/tests/qemu-iotests/110.out
+++ b/tests/qemu-iotests/110.out
@@ -11,7 +11,7 @@ backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base)
 
 === Non-reconstructable filename ===
 
-image: json:{"driver": "IMGFMT", "file": {"set-state.0.event": "read_aio", "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "set-state.0.new_state": 42}}
+image: json:{"driver": "IMGFMT", "file": {"set-state": [{"new_state": 42, "event": "read_aio"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}
 file format: IMGFMT
 virtual size: 64 MiB (67108864 bytes)
 backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base)
diff --git a/tests/qemu-iotests/198.out b/tests/qemu-iotests/198.out
index e86b175e39..eef8af3cdc 100644
--- a/tests/qemu-iotests/198.out
+++ b/tests/qemu-iotests/198.out
@@ -32,7 +32,7 @@ read 16777216/16777216 bytes at offset 0
 16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 == checking image base ==
-image: json:{"encrypt.key-secret": "sec0", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.base"}}
+image: json:{"driver": "IMGFMT", "encrypt": {"format": "auto", "key-secret": "sec0"}, "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.base"}}
 file format: IMGFMT
 virtual size: 16 MiB (16777216 bytes)
 Format specific information:
@@ -74,7 +74,7 @@ Format specific information:
         master key iters: 1024
 
 == checking image layer ==
-image: json:{"encrypt.key-secret": "sec1", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}}
+image: json:{"driver": "IMGFMT", "encrypt": {"format": "auto", "key-secret": "sec1"}, "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}}
 file format: IMGFMT
 virtual size: 16 MiB (16777216 bytes)
 backing file: TEST_DIR/t.IMGFMT.base
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 12/14] iotests: Test internal option typing
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (10 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 11/14] block: Try to create well typed json:{} filenames Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 13/14] iotests: qcow2's encrypt.format is now optional Max Reitz
                   ` (5 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

It would be nice if qemu used the correct types for blockdev options
internally, even if the user specified string values (either through
-drive or by being not so nice and using json:{} with string values).
This patch adds a test verifying that fact.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/089     | 25 +++++++++++++++++++++++++
 tests/qemu-iotests/089.out |  9 +++++++++
 2 files changed, 34 insertions(+)

diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089
index ad029f1f09..156a8af85c 100755
--- a/tests/qemu-iotests/089
+++ b/tests/qemu-iotests/089
@@ -35,6 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
+. ./common.qemu
 
 _supported_fmt qcow2
 _supported_proto file
@@ -143,6 +144,30 @@ $QEMU_IO -c "open -o driver=qcow2 json:{\"file.filename\":\"$TEST_IMG\"}" \
 $QEMU_IO -c "open -o driver=qcow2 json:{\"driver\":\"raw\",\"file.filename\":\"$TEST_IMG\"}" \
          -c "info" 2>&1 | _filter_img_info
 
+echo
+echo "=== Testing option typing ==="
+echo
+
+# json:{} accepts both strings and correctly typed values (even mixed,
+# although we probably do not want to support that...), but when
+# creating a json:{} filename, it should be correctly typed.
+# Therefore, both of these should make the "size" value an integer.
+
+TEST_IMG="json:{'driver': 'null-co', 'size':  42 }" _img_info | grep '^image'
+TEST_IMG="json:{'driver': 'null-co', 'size': '42'}" _img_info | grep '^image'
+
+echo
+
+# This should even work when some driver abuses bs->options to store
+# non-QAPI options (and the given -drive options are not complete)
+# (Watch for whether file.align appears as an int or a string)
+qemu_comm_method=monitor _launch_qemu \
+    -drive if=none,id=drv0,node-name=node0,format=raw,file=blkdebug::null-co://,file.align=512
+
+_send_qemu_cmd $QEMU_HANDLE 'info block' 'json:'
+
+_cleanup_qemu
+
 
 # success, all done
 echo "*** done"
diff --git a/tests/qemu-iotests/089.out b/tests/qemu-iotests/089.out
index 20c8ce8f0e..a43dfd3cf8 100644
--- a/tests/qemu-iotests/089.out
+++ b/tests/qemu-iotests/089.out
@@ -49,4 +49,13 @@ vm state offset: 512 MiB
 format name: IMGFMT
 cluster size: 64 KiB
 vm state offset: 512 MiB
+
+=== Testing option typing ===
+
+image: json:{"driver": "null-co", "size": 42}
+image: json:{"driver": "null-co", "size": 42}
+
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) info block
+drv0 (node0): json:{"driver": "raw", "file": {"image": {"driver": "null-co"}, "driver": "blkdebug", "align": 512}} (raw)
 *** done
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 13/14] iotests: qcow2's encrypt.format is now optional
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (11 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 12/14] iotests: Test internal option typing Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 14/14] block: Make use of QAPI defaults Max Reitz
                   ` (4 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

Remove the encrypt.format option from the two blockdev-add test cases
for encrypted qcow2 images in 087 to explicitly test that this parameter
is now optional.

Additionally, show that explicitly specifying encrypt.format=auto works
just as well, the same for specifying the correct format
(encrypt.format=luks here), and that specifying the wrong format
(encrypt.format=aes) results in an error.

While touching this test case, reduce the LUKS iteration time to 10 so
the test stays reasonably fast.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/087     | 65 +++++++++++++++++++++++---------------
 tests/qemu-iotests/087.out | 26 ++++++++++++++-
 2 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087
index d6c8613419..0e52dec483 100755
--- a/tests/qemu-iotests/087
+++ b/tests/qemu-iotests/087
@@ -149,7 +149,6 @@ run_qemu <<EOF
           "filename": "$TEST_IMG"
       },
       "encrypt": {
-          "format": "aes",
           "key-secret": "sec0"
       }
     }
@@ -161,34 +160,48 @@ echo
 echo === Encrypted image LUKS ===
 echo
 
-_make_test_img --object secret,id=sec0,data=123456 -o encrypt.format=luks,encrypt.key-secret=sec0 $size
-run_qemu <<EOF
-{ "execute": "qmp_capabilities" }
-{ "execute": "object-add",
-  "arguments": {
-      "qom-type": "secret",
-      "id": "sec0",
-      "props": {
-          "data": "123456"
-      }
-  }
-}
-{ "execute": "blockdev-add",
-  "arguments": {
-      "driver": "$IMGFMT",
-      "node-name": "disk",
-      "file": {
-          "driver": "file",
-          "filename": "$TEST_IMG"
-      },
-      "encrypt": {
-        "format": "luks",
-        "key-secret": "sec0"
+_make_test_img \
+    --object secret,id=sec0,data=123456 \
+    -o encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10 \
+    $size
+
+# Adds the qcow2+LUKS image via blockdev-add.
+# First parameter: Optional entry for the 'encrypt' option dict
+function luks_test()
+{
+    run_qemu <<EOF
+    { "execute": "qmp_capabilities" }
+    { "execute": "object-add",
+      "arguments": {
+          "qom-type": "secret",
+          "id": "sec0",
+          "props": {
+              "data": "123456"
+          }
       }
     }
-  }
-{ "execute": "quit" }
+    { "execute": "blockdev-add",
+      "arguments": {
+          "driver": "$IMGFMT",
+          "node-name": "disk",
+          "file": {
+              "driver": "file",
+              "filename": "$TEST_IMG"
+          },
+          "encrypt": {
+            $1
+            "key-secret": "sec0"
+          }
+        }
+      }
+    { "execute": "quit" }
 EOF
+}
+
+luks_test ''                    # Implicit encrypt.format=auto
+luks_test '"format": "auto",'   # Explicit encrypt.format=auto
+luks_test '"format": "luks",'   # Explicit encrypt.format=luks
+luks_test '"format": "aes",'    # Explicit encrypt.format=aes (wrong)
 
 echo
 echo === Missing driver ===
diff --git a/tests/qemu-iotests/087.out b/tests/qemu-iotests/087.out
index 2d92ea847b..6c1d83519b 100644
--- a/tests/qemu-iotests/087.out
+++ b/tests/qemu-iotests/087.out
@@ -46,7 +46,7 @@ QMP_VERSION
 
 === Encrypted image LUKS ===
 
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encrypt.format=luks encrypt.key-secret=sec0
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10
 Testing:
 QMP_VERSION
 {"return": {}}
@@ -55,6 +55,30 @@ QMP_VERSION
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
 
+Testing:
+QMP_VERSION
+{"return": {}}
+{"return": {}}
+{"return": {}}
+{"return": {}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+
+Testing:
+QMP_VERSION
+{"return": {}}
+{"return": {}}
+{"return": {}}
+{"return": {}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+
+Testing:
+QMP_VERSION
+{"return": {}}
+{"return": {}}
+{"error": {"class": "GenericError", "desc": "Header reported 'luks' encryption format but options specify 'aes'"}}
+{"return": {}}
+{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
+
 
 === Missing driver ===
 
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* [Qemu-devel] [PATCH v4 14/14] block: Make use of QAPI defaults
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (12 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 13/14] iotests: qcow2's encrypt.format is now optional Max Reitz
@ 2019-06-24 17:39 ` Max Reitz
  2019-06-24 18:35 ` [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames no-reply
                   ` (3 subsequent siblings)
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 17:39 UTC (permalink / raw)
  To: qemu-block
  Cc: Kevin Wolf, qemu-devel, Michael Roth, Markus Armbruster, Max Reitz

This changes most of block-core.json to use QAPI defaults where
possible.

Notably omitted is everything around BlockdevOptions*, because that
would change json:{} filenames (which is technically not an incompatible
change, it would just break many iotests).

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 144 +++++++++++++++++++++-------------
 block/file-posix.c   |   9 ---
 block/file-win32.c   |   8 +-
 block/parallels.c    |   6 +-
 block/qcow2.c        |  36 +++------
 block/qed.c          |   3 -
 block/sheepdog.c     |   3 -
 block/vdi.c          |   3 -
 block/vhdx.c         |  28 ++-----
 block/vpc.c          |   3 -
 blockdev.c           | 182 +++++++++----------------------------------
 monitor/hmp-cmds.c   |  27 ++++---
 monitor/qmp-cmds.c   |   3 +-
 13 files changed, 165 insertions(+), 290 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index b30a19bf8e..5d5c0c4b81 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1300,7 +1300,8 @@
 { 'struct': 'BlockdevSnapshotSync',
   'data': { '*device': 'str', '*node-name': 'str',
             'snapshot-file': 'str', '*snapshot-node-name': 'str',
-            '*format': 'str', '*mode': 'NewImageMode' } }
+            '*format': { 'type': 'str', 'default': 'qcow2' },
+            '*mode': { 'type': 'NewImageMode', 'default': 'absolute-paths' } } }
 
 ##
 # @BlockdevSnapshot:
@@ -1378,11 +1379,16 @@
 { 'struct': 'DriveBackup',
   'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
             '*format': 'str', 'sync': 'MirrorSyncMode',
-            '*mode': 'NewImageMode', '*speed': 'int',
-            '*bitmap': 'str', '*compress': 'bool',
-            '*on-source-error': 'BlockdevOnError',
-            '*on-target-error': 'BlockdevOnError',
-            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+            '*mode': { 'type': 'NewImageMode', 'default': 'absolute-paths' },
+            '*speed': { 'type': 'int', 'default': 0 },
+            '*bitmap': 'str',
+            '*compress': { 'type': 'bool', 'default': false },
+            '*on-source-error': { 'type': 'BlockdevOnError',
+                                  'default': 'report' },
+            '*on-target-error': { 'type': 'BlockdevOnError',
+                                  'default': 'report' },
+            '*auto-finalize': { 'type': 'bool', 'default': true },
+            '*auto-dismiss': { 'type': 'bool', 'default': true } } }
 
 ##
 # @BlockdevBackup:
@@ -1437,11 +1443,16 @@
 ##
 { 'struct': 'BlockdevBackup',
   'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
-            'sync': 'MirrorSyncMode', '*speed': 'int',
-            '*bitmap': 'str', '*compress': 'bool',
-            '*on-source-error': 'BlockdevOnError',
-            '*on-target-error': 'BlockdevOnError',
-            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+            'sync': 'MirrorSyncMode',
+            '*speed': { 'type': 'int', 'default': 0 },
+            '*bitmap': 'str',
+            '*compress': { 'type': 'bool', 'default': false },
+            '*on-source-error': { 'type': 'BlockdevOnError',
+                                  'default': 'report' },
+            '*on-target-error': { 'type': 'BlockdevOnError',
+                                  'default': 'report' },
+            '*auto-finalize': { 'type': 'bool', 'default': true },
+            '*auto-dismiss': { 'type': 'bool', 'default': true } } }
 
 ##
 # @blockdev-snapshot-sync:
@@ -1631,9 +1642,11 @@
 { 'command': 'block-commit',
   'data': { '*job-id': 'str', 'device': 'str', '*base-node': 'str',
             '*base': 'str', '*top-node': 'str', '*top': 'str',
-            '*backing-file': 'str', '*speed': 'int',
+            '*backing-file': 'str',
+            '*speed': { 'type': 'int', 'default': 0 },
             '*filter-node-name': 'str',
-            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+            '*auto-finalize': { 'type': 'bool', 'default': true },
+            '*auto-dismiss': { 'type': 'bool', 'default': true } } }
 
 ##
 # @drive-backup:
@@ -1958,12 +1971,19 @@
 { 'struct': 'DriveMirror',
   'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
             '*format': 'str', '*node-name': 'str', '*replaces': 'str',
-            'sync': 'MirrorSyncMode', '*mode': 'NewImageMode',
-            '*speed': 'int', '*granularity': 'uint32',
-            '*buf-size': 'int', '*on-source-error': 'BlockdevOnError',
-            '*on-target-error': 'BlockdevOnError',
-            '*unmap': 'bool', '*copy-mode': 'MirrorCopyMode',
-            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+            'sync': 'MirrorSyncMode',
+            '*mode': { 'type': 'NewImageMode', 'default': 'absolute-paths' },
+            '*speed': { 'type': 'int', 'default': 0 },
+            '*granularity': 'uint32',
+            '*buf-size': 'int',
+            '*on-source-error': { 'type': 'BlockdevOnError',
+                                  'default': 'report' },
+            '*on-target-error': { 'type': 'BlockdevOnError',
+                                  'default': 'report' },
+            '*unmap': { 'type': 'bool', 'default': true },
+            '*copy-mode': { 'type': 'MirrorCopyMode', 'default': 'background' },
+            '*auto-finalize': { 'type': 'bool', 'default': true },
+            '*auto-dismiss': { 'type': 'bool', 'default': true } } }
 
 ##
 # @BlockDirtyBitmap:
@@ -1984,8 +2004,8 @@
 #
 # @name: name of the dirty bitmap
 #
-# @granularity: the bitmap granularity, default is 64k for
-#               block-dirty-bitmap-add
+# @granularity: the bitmap granularity, default is the node's cluster size
+#               (clamped to [4k, 64k]), or 64k
 #
 # @persistent: the bitmap is persistent, i.e. it will be saved to the
 #              corresponding block device image file on its close. For now only
@@ -2004,7 +2024,9 @@
 ##
 { 'struct': 'BlockDirtyBitmapAdd',
   'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32',
-            '*persistent': 'bool', '*autoload': 'bool', '*disabled': 'bool' } }
+            '*persistent': { 'type': 'bool', 'default': false },
+            '*autoload': 'bool',
+            '*disabled': { 'type': 'bool', 'default': false } } }
 
 ##
 # @BlockDirtyBitmapMergeSource:
@@ -2282,12 +2304,17 @@
   'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
             '*replaces': 'str',
             'sync': 'MirrorSyncMode',
-            '*speed': 'int', '*granularity': 'uint32',
-            '*buf-size': 'int', '*on-source-error': 'BlockdevOnError',
-            '*on-target-error': 'BlockdevOnError',
+            '*speed': { 'type': 'int', 'default': 0 },
+            '*granularity': 'uint32',
+            '*buf-size': 'int',
+            '*on-source-error': { 'type': 'BlockdevOnError',
+                                  'default': 'report' },
+            '*on-target-error': { 'type': 'BlockdevOnError',
+                                  'default': 'report' },
             '*filter-node-name': 'str',
-            '*copy-mode': 'MirrorCopyMode',
-            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+            '*copy-mode': { 'type': 'MirrorCopyMode', 'default': 'background' },
+            '*auto-finalize': { 'type': 'bool', 'default': true },
+            '*auto-dismiss': { 'type': 'bool', 'default': true } } }
 
 ##
 # @block_set_io_throttle:
@@ -2580,9 +2607,11 @@
 ##
 { 'command': 'block-stream',
   'data': { '*job-id': 'str', 'device': 'str', '*base': 'str',
-            '*base-node': 'str', '*backing-file': 'str', '*speed': 'int',
-            '*on-error': 'BlockdevOnError',
-            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+            '*base-node': 'str', '*backing-file': 'str',
+            '*speed': { 'type': 'int', 'default': 0 },
+            '*on-error': { 'type': 'BlockdevOnError', 'default': 'report' },
+            '*auto-finalize': { 'type': 'bool', 'default': true },
+            '*auto-dismiss': { 'type': 'bool', 'default': true } } }
 
 ##
 # @block-job-set-speed:
@@ -4204,8 +4233,8 @@
 { 'struct': 'BlockdevCreateOptionsFile',
   'data': { 'filename':         'str',
             'size':             'size',
-            '*preallocation':   'PreallocMode',
-            '*nocow':           'bool' } }
+            '*preallocation':   { 'type': 'PreallocMode', 'default': 'off' },
+            '*nocow':           { 'type': 'bool', 'default': false } } }
 
 ##
 # @BlockdevCreateOptionsGluster:
@@ -4224,7 +4253,7 @@
 { 'struct': 'BlockdevCreateOptionsGluster',
   'data': { 'location':         'BlockdevOptionsGluster',
             'size':             'size',
-            '*preallocation':   'PreallocMode' } }
+            '*preallocation':   { 'type': 'PreallocMode', 'default': 'off' } } }
 
 ##
 # @BlockdevCreateOptionsLUKS:
@@ -4269,7 +4298,7 @@
 { 'struct': 'BlockdevCreateOptionsParallels',
   'data': { 'file':             'BlockdevRef',
             'size':             'size',
-            '*cluster-size':    'size' } }
+            '*cluster-size':    { 'type': 'size', 'default': 1048576 } } }
 
 ##
 # @BlockdevCreateOptionsQcow:
@@ -4331,16 +4360,16 @@
 { 'struct': 'BlockdevCreateOptionsQcow2',
   'data': { 'file':             'BlockdevRef',
             '*data-file':       'BlockdevRef',
-            '*data-file-raw':   'bool',
+            '*data-file-raw':   { 'type': 'bool', 'default': false },
             'size':             'size',
-            '*version':         'BlockdevQcow2Version',
+            '*version':         { 'type': 'BlockdevQcow2Version', 'default': 'v3' },
             '*backing-file':    'str',
             '*backing-fmt':     'BlockdevDriver',
             '*encrypt':         'QCryptoBlockCreateOptions',
-            '*cluster-size':    'size',
-            '*preallocation':   'PreallocMode',
-            '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*cluster-size':    { 'type': 'size', 'default': 65536 },
+            '*preallocation':   { 'type': 'PreallocMode', 'default': 'off' },
+            '*lazy-refcounts':  { 'type': 'bool', 'default': false },
+            '*refcount-bits':   { 'type': 'int', 'default': 16 } } }
 
 ##
 # @BlockdevCreateOptionsQed:
@@ -4362,7 +4391,7 @@
             'size':             'size',
             '*backing-file':    'str',
             '*backing-fmt':     'BlockdevDriver',
-            '*cluster-size':    'size',
+            '*cluster-size':    { 'type': 'size', 'default': 65536 },
             '*table-size':      'int' } }
 
 ##
@@ -4445,11 +4474,13 @@
   'data': { 'file':             'BlockdevRef',
             'size':             'size',
             '*extents':          ['BlockdevRef'],
-            '*subformat':       'BlockdevVmdkSubformat',
+            '*subformat':       { 'type': 'BlockdevVmdkSubformat',
+                                  'default': 'monolithicSparse' },
             '*backing-file':    'str',
-            '*adapter-type':    'BlockdevVmdkAdapterType',
-            '*hwversion':       'str',
-            '*zeroed-grain':    'bool' } }
+            '*adapter-type':    { 'type': 'BlockdevVmdkAdapterType',
+                                  'default': 'ide' },
+            '*hwversion':       { 'type': 'str', 'default': '4' },
+            '*zeroed-grain':    { 'type': 'bool', 'default': false } } }
 
 
 ##
@@ -4516,7 +4547,7 @@
   'data': { 'location':         'BlockdevOptionsSheepdog',
             'size':             'size',
             '*backing-file':    'str',
-            '*preallocation':   'PreallocMode',
+            '*preallocation':   { 'type': 'PreallocMode', 'default': 'off' },
             '*redundancy':      'SheepdogRedundancy',
             '*object-size':     'size' } }
 
@@ -4549,7 +4580,7 @@
 { 'struct': 'BlockdevCreateOptionsVdi',
   'data': { 'file':             'BlockdevRef',
             'size':             'size',
-            '*preallocation':   'PreallocMode' } }
+            '*preallocation':   { 'type': 'PreallocMode', 'default': 'off' } } }
 
 ##
 # @BlockdevVhdxSubformat:
@@ -4584,10 +4615,11 @@
 { 'struct': 'BlockdevCreateOptionsVhdx',
   'data': { 'file':                 'BlockdevRef',
             'size':                 'size',
-            '*log-size':            'size',
+            '*log-size':            { 'type': 'size', 'default': 1048576 },
             '*block-size':          'size',
-            '*subformat':           'BlockdevVhdxSubformat',
-            '*block-state-zero':    'bool' } }
+            '*subformat':           { 'type': 'BlockdevVhdxSubformat',
+                                      'default': 'dynamic' },
+            '*block-state-zero':    { 'type': 'bool', 'default': true } } }
 
 ##
 # @BlockdevVpcSubformat:
@@ -4607,7 +4639,7 @@
 #
 # @file             Node to create the image format on
 # @size             Size of the virtual disk in bytes
-# @subformat        vhdx subformat (default: dynamic)
+# @subformat        vpc subformat (default: dynamic)
 # @force-size       Force use of the exact byte size instead of rounding to the
 #                   next size that can be represented in CHS geometry
 #                   (default: false)
@@ -4617,8 +4649,9 @@
 { 'struct': 'BlockdevCreateOptionsVpc',
   'data': { 'file':                 'BlockdevRef',
             'size':                 'size',
-            '*subformat':           'BlockdevVpcSubformat',
-            '*force-size':          'bool' } }
+            '*subformat':           { 'type': 'BlockdevVpcSubformat',
+                                      'default': 'dynamic' },
+            '*force-size':          { 'type': 'bool', 'default': false } } }
 
 ##
 # @BlockdevCreateOptions:
@@ -4714,7 +4747,7 @@
 { 'command': 'blockdev-open-tray',
   'data': { '*device': 'str',
             '*id': 'str',
-            '*force': 'bool' } }
+            '*force': { 'type': 'bool', 'default': false } } }
 
 ##
 # @blockdev-close-tray:
@@ -4905,7 +4938,8 @@
             '*id': 'str',
             'filename': 'str',
             '*format': 'str',
-            '*read-only-mode': 'BlockdevChangeReadOnlyMode' } }
+            '*read-only-mode': { 'type': 'BlockdevChangeReadOnlyMode',
+                                 'default': 'retain' } } }
 
 
 ##
diff --git a/block/file-posix.c b/block/file-posix.c
index ab05b51a66..88ac031e9f 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -2243,13 +2243,6 @@ raw_co_create(BlockdevCreateOptions *options, Error **errp)
     assert(options->driver == BLOCKDEV_DRIVER_FILE);
     file_opts = &options->u.file;
 
-    if (!file_opts->has_nocow) {
-        file_opts->nocow = false;
-    }
-    if (!file_opts->has_preallocation) {
-        file_opts->preallocation = PREALLOC_MODE_OFF;
-    }
-
     /* Create file */
     fd = qemu_open(file_opts->filename, O_RDWR | O_CREAT | O_BINARY, 0644);
     if (fd < 0) {
@@ -2365,9 +2358,7 @@ static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts,
         .u.file     = {
             .filename           = (char *) filename,
             .size               = total_size,
-            .has_preallocation  = true,
             .preallocation      = prealloc,
-            .has_nocow          = true,
             .nocow              = nocow,
         },
     };
diff --git a/block/file-win32.c b/block/file-win32.c
index 6b2d67b239..9164b3d77c 100644
--- a/block/file-win32.c
+++ b/block/file-win32.c
@@ -565,11 +565,11 @@ static int raw_co_create(BlockdevCreateOptions *options, Error **errp)
     assert(options->driver == BLOCKDEV_DRIVER_FILE);
     file_opts = &options->u.file;
 
-    if (file_opts->has_preallocation) {
+    if (file_opts->preallocation) {
         error_setg(errp, "Preallocation is not supported on Windows");
         return -EINVAL;
     }
-    if (file_opts->has_nocow) {
+    if (file_opts->nocow) {
         error_setg(errp, "nocow is not supported on Windows");
         return -EINVAL;
     }
@@ -604,8 +604,8 @@ static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts,
         .u.file     = {
             .filename           = (char *) filename,
             .size               = total_size,
-            .has_preallocation  = false,
-            .has_nocow          = false,
+            .preallocation      = false,
+            .nocow              = false,
         },
     };
     return raw_co_create(&options, errp);
diff --git a/block/parallels.c b/block/parallels.c
index 00fae125d1..67de90bd51 100644
--- a/block/parallels.c
+++ b/block/parallels.c
@@ -522,11 +522,7 @@ static int coroutine_fn parallels_co_create(BlockdevCreateOptions* opts,
     /* Sanity checks */
     total_size = parallels_opts->size;
 
-    if (parallels_opts->has_cluster_size) {
-        cl_size = parallels_opts->cluster_size;
-    } else {
-        cl_size = DEFAULT_CLUSTER_SIZE;
-    }
+    cl_size = parallels_opts->cluster_size;
 
     /* XXX What is the real limit here? This is an insanely large maximum. */
     if (cl_size >= INT64_MAX / MAX_PARALLELS_IMAGE_FACTOR) {
diff --git a/block/qcow2.c b/block/qcow2.c
index 95de19d906..0136be1a14 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -3076,35 +3076,23 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
         goto out;
     }
 
-    if (qcow2_opts->has_version) {
-        switch (qcow2_opts->version) {
-        case BLOCKDEV_QCOW2_VERSION_V2:
-            version = 2;
-            break;
-        case BLOCKDEV_QCOW2_VERSION_V3:
-            version = 3;
-            break;
-        default:
-            g_assert_not_reached();
-        }
-    } else {
+    switch (qcow2_opts->version) {
+    case BLOCKDEV_QCOW2_VERSION_V2:
+        version = 2;
+        break;
+    case BLOCKDEV_QCOW2_VERSION_V3:
         version = 3;
+        break;
+    default:
+        g_assert_not_reached();
     }
 
-    if (qcow2_opts->has_cluster_size) {
-        cluster_size = qcow2_opts->cluster_size;
-    } else {
-        cluster_size = DEFAULT_CLUSTER_SIZE;
-    }
-
+    cluster_size = qcow2_opts->cluster_size;
     if (!validate_cluster_size(cluster_size, errp)) {
         ret = -EINVAL;
         goto out;
     }
 
-    if (!qcow2_opts->has_preallocation) {
-        qcow2_opts->preallocation = PREALLOC_MODE_OFF;
-    }
     if (qcow2_opts->has_backing_file &&
         qcow2_opts->preallocation != PREALLOC_MODE_OFF)
     {
@@ -3119,9 +3107,6 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
         goto out;
     }
 
-    if (!qcow2_opts->has_lazy_refcounts) {
-        qcow2_opts->lazy_refcounts = false;
-    }
     if (version < 3 && qcow2_opts->lazy_refcounts) {
         error_setg(errp, "Lazy refcounts only supported with compatibility "
                    "level 1.1 and above (use version=v3 or greater)");
@@ -3129,9 +3114,6 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
         goto out;
     }
 
-    if (!qcow2_opts->has_refcount_bits) {
-        qcow2_opts->refcount_bits = 16;
-    }
     if (qcow2_opts->refcount_bits > 64 ||
         !is_power_of_2(qcow2_opts->refcount_bits))
     {
diff --git a/block/qed.c b/block/qed.c
index 77c7cef175..7046eb63ae 100644
--- a/block/qed.c
+++ b/block/qed.c
@@ -615,9 +615,6 @@ static int coroutine_fn bdrv_qed_co_create(BlockdevCreateOptions *opts,
     qed_opts = &opts->u.qed;
 
     /* Validate options and set default values */
-    if (!qed_opts->has_cluster_size) {
-        qed_opts->cluster_size = QED_DEFAULT_CLUSTER_SIZE;
-    }
     if (!qed_opts->has_table_size) {
         qed_opts->table_size = QED_DEFAULT_TABLE_SIZE;
     }
diff --git a/block/sheepdog.c b/block/sheepdog.c
index 6f402e5d4d..7d74e76c34 100644
--- a/block/sheepdog.c
+++ b/block/sheepdog.c
@@ -2034,9 +2034,6 @@ static int sd_co_create(BlockdevCreateOptions *options, Error **errp)
     s->inode.vdi_size = opts->size;
     backing_file = opts->backing_file;
 
-    if (!opts->has_preallocation) {
-        opts->preallocation = PREALLOC_MODE_OFF;
-    }
     switch (opts->preallocation) {
     case PREALLOC_MODE_OFF:
         prealloc = false;
diff --git a/block/vdi.c b/block/vdi.c
index b9845a4cbd..9dd22ee1d0 100644
--- a/block/vdi.c
+++ b/block/vdi.c
@@ -756,9 +756,6 @@ static int coroutine_fn vdi_co_do_create(BlockdevCreateOptions *create_options,
     /* Validate options and set default values */
     bytes = vdi_opts->size;
 
-    if (!vdi_opts->has_preallocation) {
-        vdi_opts->preallocation = PREALLOC_MODE_OFF;
-    }
     switch (vdi_opts->preallocation) {
     case PREALLOC_MODE_OFF:
         image_type = VDI_TYPE_DYNAMIC;
diff --git a/block/vhdx.c b/block/vhdx.c
index d6070b6fa8..b238438c1d 100644
--- a/block/vhdx.c
+++ b/block/vhdx.c
@@ -1829,29 +1829,19 @@ static int coroutine_fn vhdx_co_create(BlockdevCreateOptions *opts,
         return -EINVAL;
     }
 
-    if (!vhdx_opts->has_log_size) {
-        log_size = DEFAULT_LOG_SIZE;
-    } else {
-        if (vhdx_opts->log_size > UINT32_MAX) {
-            error_setg(errp, "Log size must be smaller than 4 GB");
-            return -EINVAL;
-        }
-        log_size = vhdx_opts->log_size;
+    if (vhdx_opts->log_size > UINT32_MAX) {
+        error_setg(errp, "Log size must be smaller than 4 GB");
+        return -EINVAL;
     }
+    log_size = vhdx_opts->log_size;
     if (log_size < MiB || (log_size % MiB) != 0) {
         error_setg(errp, "Log size must be a multiple of 1 MB");
         return -EINVAL;
     }
 
-    if (!vhdx_opts->has_block_state_zero) {
-        use_zero_blocks = true;
-    } else {
-        use_zero_blocks = vhdx_opts->block_state_zero;
-    }
+    use_zero_blocks = vhdx_opts->block_state_zero;
 
-    if (!vhdx_opts->has_subformat) {
-        vhdx_opts->subformat = BLOCKDEV_VHDX_SUBFORMAT_DYNAMIC;
-    }
+    vhdx_opts->subformat = BLOCKDEV_VHDX_SUBFORMAT_DYNAMIC;
 
     switch (vhdx_opts->subformat) {
     case BLOCKDEV_VHDX_SUBFORMAT_DYNAMIC:
@@ -2030,10 +2020,8 @@ static int coroutine_fn vhdx_co_create_opts(const char *filename,
     create_options->u.vhdx.size =
         ROUND_UP(create_options->u.vhdx.size, BDRV_SECTOR_SIZE);
 
-    if (create_options->u.vhdx.has_log_size) {
-        create_options->u.vhdx.log_size =
-            ROUND_UP(create_options->u.vhdx.log_size, MiB);
-    }
+    create_options->u.vhdx.log_size =
+        ROUND_UP(create_options->u.vhdx.log_size, MiB);
     if (create_options->u.vhdx.has_block_size) {
         create_options->u.vhdx.block_size =
             ROUND_UP(create_options->u.vhdx.block_size, MiB);
diff --git a/block/vpc.c b/block/vpc.c
index d4776ee8a5..82cd8d4081 100644
--- a/block/vpc.c
+++ b/block/vpc.c
@@ -991,9 +991,6 @@ static int coroutine_fn vpc_co_create(BlockdevCreateOptions *opts,
     /* Validate options and set default values */
     total_size = vpc_opts->size;
 
-    if (!vpc_opts->has_subformat) {
-        vpc_opts->subformat = BLOCKDEV_VPC_SUBFORMAT_DYNAMIC;
-    }
     switch (vpc_opts->subformat) {
     case BLOCKDEV_VPC_SUBFORMAT_DYNAMIC:
         disk_type = VHD_DYNAMIC;
diff --git a/blockdev.c b/blockdev.c
index 4d141e9a1f..f3828697a8 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1122,8 +1122,8 @@ void qmp_blockdev_snapshot_sync(bool has_device, const char *device,
                                 const char *snapshot_file,
                                 bool has_snapshot_node_name,
                                 const char *snapshot_node_name,
-                                bool has_format, const char *format,
-                                bool has_mode, NewImageMode mode, Error **errp)
+                                const char *format, NewImageMode mode,
+                                Error **errp)
 {
     BlockdevSnapshotSync snapshot = {
         .has_device = has_device,
@@ -1133,9 +1133,7 @@ void qmp_blockdev_snapshot_sync(bool has_device, const char *device,
         .snapshot_file = (char *) snapshot_file,
         .has_snapshot_node_name = has_snapshot_node_name,
         .snapshot_node_name = (char *) snapshot_node_name,
-        .has_format = has_format,
         .format = (char *) format,
-        .has_mode = has_mode,
         .mode = mode,
     };
     TransactionAction action = {
@@ -1601,8 +1599,6 @@ static void external_snapshot_prepare(BlkActionState *common,
 
     if (action->type == TRANSACTION_ACTION_KIND_BLOCKDEV_SNAPSHOT_SYNC) {
         BlockdevSnapshotSync *s = action->u.blockdev_snapshot_sync.data;
-        const char *format = s->has_format ? s->format : "qcow2";
-        enum NewImageMode mode;
         const char *snapshot_node_name =
             s->has_snapshot_node_name ? s->snapshot_node_name : NULL;
 
@@ -1622,15 +1618,14 @@ static void external_snapshot_prepare(BlkActionState *common,
         flags |= BDRV_O_NO_BACKING;
 
         /* create new image w/backing file */
-        mode = s->has_mode ? s->mode : NEW_IMAGE_MODE_ABSOLUTE_PATHS;
-        if (mode != NEW_IMAGE_MODE_EXISTING) {
+        if (s->mode != NEW_IMAGE_MODE_EXISTING) {
             int64_t size = bdrv_getlength(state->old_bs);
             if (size < 0) {
                 error_setg_errno(errp, -size, "bdrv_getlength failed");
                 goto out;
             }
             bdrv_refresh_filename(state->old_bs);
-            bdrv_img_create(new_image_file, format,
+            bdrv_img_create(new_image_file, s->format,
                             state->old_bs->filename,
                             state->old_bs->drv->format_name,
                             NULL, size, flags, false, &local_err);
@@ -1644,7 +1639,7 @@ static void external_snapshot_prepare(BlkActionState *common,
         if (snapshot_node_name) {
             qdict_put_str(options, "node-name", snapshot_node_name);
         }
-        qdict_put_str(options, "driver", format);
+        qdict_put_str(options, "driver", s->format);
     }
 
     state->new_bs = bdrv_open(new_image_file, snapshot_ref, options, flags,
@@ -1963,9 +1958,9 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,
     /* AIO context taken and released within qmp_block_dirty_bitmap_add */
     qmp_block_dirty_bitmap_add(action->node, action->name,
                                action->has_granularity, action->granularity,
-                               action->has_persistent, action->persistent,
+                               action->persistent,
                                action->has_autoload, action->autoload,
-                               action->has_disabled, action->disabled,
+                               action->disabled,
                                &local_err);
 
     if (!local_err) {
@@ -2410,15 +2405,11 @@ static int do_open_tray(const char *blk_name, const char *qdev_id,
 
 void qmp_blockdev_open_tray(bool has_device, const char *device,
                             bool has_id, const char *id,
-                            bool has_force, bool force,
-                            Error **errp)
+                            bool force, Error **errp)
 {
     Error *local_err = NULL;
     int rc;
 
-    if (!has_force) {
-        force = false;
-    }
     rc = do_open_tray(has_device ? device : NULL,
                       has_id ? id : NULL,
                       force, &local_err);
@@ -2610,7 +2601,6 @@ void qmp_blockdev_change_medium(bool has_device, const char *device,
                                 bool has_id, const char *id,
                                 const char *filename,
                                 bool has_format, const char *format,
-                                bool has_read_only,
                                 BlockdevChangeReadOnlyMode read_only,
                                 Error **errp)
 {
@@ -2637,10 +2627,6 @@ void qmp_blockdev_change_medium(bool has_device, const char *device,
     bdrv_flags &= ~(BDRV_O_TEMPORARY | BDRV_O_SNAPSHOT | BDRV_O_NO_BACKING |
         BDRV_O_PROTOCOL | BDRV_O_AUTO_RDONLY);
 
-    if (!has_read_only) {
-        read_only = BLOCKDEV_CHANGE_READ_ONLY_MODE_RETAIN;
-    }
-
     switch (read_only) {
     case BLOCKDEV_CHANGE_READ_ONLY_MODE_RETAIN:
         break;
@@ -2804,10 +2790,8 @@ out:
 
 void qmp_block_dirty_bitmap_add(const char *node, const char *name,
                                 bool has_granularity, uint32_t granularity,
-                                bool has_persistent, bool persistent,
-                                bool has_autoload, bool autoload,
-                                bool has_disabled, bool disabled,
-                                Error **errp)
+                                bool persistent, bool has_autoload,
+                                bool autoload, bool disabled, Error **errp)
 {
     BlockDriverState *bs;
     BdrvDirtyBitmap *bitmap;
@@ -2834,18 +2818,10 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
         granularity = bdrv_get_default_bitmap_granularity(bs);
     }
 
-    if (!has_persistent) {
-        persistent = false;
-    }
-
     if (has_autoload) {
         warn_report("Autoload option is deprecated and its value is ignored");
     }
 
-    if (!has_disabled) {
-        disabled = false;
-    }
-
     if (persistent) {
         aio_context = bdrv_get_aio_context(bs);
         aio_context_acquire(aio_context);
@@ -3172,10 +3148,8 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device,
                       bool has_base, const char *base,
                       bool has_base_node, const char *base_node,
                       bool has_backing_file, const char *backing_file,
-                      bool has_speed, int64_t speed,
-                      bool has_on_error, BlockdevOnError on_error,
-                      bool has_auto_finalize, bool auto_finalize,
-                      bool has_auto_dismiss, bool auto_dismiss,
+                      int64_t speed, BlockdevOnError on_error,
+                      bool auto_finalize, bool auto_dismiss,
                       Error **errp)
 {
     BlockDriverState *bs, *iter;
@@ -3185,10 +3159,6 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device,
     const char *base_name = NULL;
     int job_flags = JOB_DEFAULT;
 
-    if (!has_on_error) {
-        on_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
-
     bs = bdrv_lookup_bs(device, device, errp);
     if (!bs) {
         return;
@@ -3246,15 +3216,15 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device,
     /* backing_file string overrides base bs filename */
     base_name = has_backing_file ? backing_file : base_name;
 
-    if (has_auto_finalize && !auto_finalize) {
+    if (!auto_finalize) {
         job_flags |= JOB_MANUAL_FINALIZE;
     }
-    if (has_auto_dismiss && !auto_dismiss) {
+    if (!auto_dismiss) {
         job_flags |= JOB_MANUAL_DISMISS;
     }
 
     stream_start(has_job_id ? job_id : NULL, bs, base_bs, base_name,
-                 job_flags, has_speed ? speed : 0, on_error, &local_err);
+                 job_flags, speed, on_error, &local_err);
     if (local_err) {
         error_propagate(errp, local_err);
         goto out;
@@ -3272,11 +3242,9 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
                       bool has_top_node, const char *top_node,
                       bool has_top, const char *top,
                       bool has_backing_file, const char *backing_file,
-                      bool has_speed, int64_t speed,
+                      int64_t speed,
                       bool has_filter_node_name, const char *filter_node_name,
-                      bool has_auto_finalize, bool auto_finalize,
-                      bool has_auto_dismiss, bool auto_dismiss,
-                      Error **errp)
+                      bool auto_finalize, bool auto_dismiss, Error **errp)
 {
     BlockDriverState *bs;
     BlockDriverState *iter;
@@ -3289,16 +3257,13 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
     BlockdevOnError on_error = BLOCKDEV_ON_ERROR_REPORT;
     int job_flags = JOB_DEFAULT;
 
-    if (!has_speed) {
-        speed = 0;
-    }
     if (!has_filter_node_name) {
         filter_node_name = NULL;
     }
-    if (has_auto_finalize && !auto_finalize) {
+    if (!auto_finalize) {
         job_flags |= JOB_MANUAL_FINALIZE;
     }
-    if (has_auto_dismiss && !auto_dismiss) {
+    if (!auto_dismiss) {
         job_flags |= JOB_MANUAL_DISMISS;
     }
 
@@ -3441,30 +3406,9 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,
     bool set_backing_hd = false;
     int ret;
 
-    if (!backup->has_speed) {
-        backup->speed = 0;
-    }
-    if (!backup->has_on_source_error) {
-        backup->on_source_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
-    if (!backup->has_on_target_error) {
-        backup->on_target_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
-    if (!backup->has_mode) {
-        backup->mode = NEW_IMAGE_MODE_ABSOLUTE_PATHS;
-    }
     if (!backup->has_job_id) {
         backup->job_id = NULL;
     }
-    if (!backup->has_auto_finalize) {
-        backup->auto_finalize = true;
-    }
-    if (!backup->has_auto_dismiss) {
-        backup->auto_dismiss = true;
-    }
-    if (!backup->has_compress) {
-        backup->compress = false;
-    }
 
     bs = bdrv_lookup_bs(backup->device, backup->device, errp);
     if (!bs) {
@@ -3619,27 +3563,9 @@ BlockJob *do_blockdev_backup(BlockdevBackup *backup, JobTxn *txn,
     int job_flags = JOB_DEFAULT;
     int ret;
 
-    if (!backup->has_speed) {
-        backup->speed = 0;
-    }
-    if (!backup->has_on_source_error) {
-        backup->on_source_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
-    if (!backup->has_on_target_error) {
-        backup->on_target_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
     if (!backup->has_job_id) {
         backup->job_id = NULL;
     }
-    if (!backup->has_auto_finalize) {
-        backup->auto_finalize = true;
-    }
-    if (!backup->has_auto_dismiss) {
-        backup->auto_dismiss = true;
-    }
-    if (!backup->has_compress) {
-        backup->compress = false;
-    }
 
     bs = bdrv_lookup_bs(backup->device, backup->device, errp);
     if (!bs) {
@@ -3705,51 +3631,33 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
                                    bool has_replaces, const char *replaces,
                                    enum MirrorSyncMode sync,
                                    BlockMirrorBackingMode backing_mode,
-                                   bool has_speed, int64_t speed,
+                                   int64_t speed,
                                    bool has_granularity, uint32_t granularity,
                                    bool has_buf_size, int64_t buf_size,
-                                   bool has_on_source_error,
                                    BlockdevOnError on_source_error,
-                                   bool has_on_target_error,
                                    BlockdevOnError on_target_error,
-                                   bool has_unmap, bool unmap,
+                                   bool unmap,
                                    bool has_filter_node_name,
                                    const char *filter_node_name,
-                                   bool has_copy_mode, MirrorCopyMode copy_mode,
-                                   bool has_auto_finalize, bool auto_finalize,
-                                   bool has_auto_dismiss, bool auto_dismiss,
+                                   MirrorCopyMode copy_mode,
+                                   bool auto_finalize, bool auto_dismiss,
                                    Error **errp)
 {
     int job_flags = JOB_DEFAULT;
 
-    if (!has_speed) {
-        speed = 0;
-    }
-    if (!has_on_source_error) {
-        on_source_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
-    if (!has_on_target_error) {
-        on_target_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
     if (!has_granularity) {
         granularity = 0;
     }
     if (!has_buf_size) {
         buf_size = 0;
     }
-    if (!has_unmap) {
-        unmap = true;
-    }
     if (!has_filter_node_name) {
         filter_node_name = NULL;
     }
-    if (!has_copy_mode) {
-        copy_mode = MIRROR_COPY_MODE_BACKGROUND;
-    }
-    if (has_auto_finalize && !auto_finalize) {
+    if (!auto_finalize) {
         job_flags |= JOB_MANUAL_FINALIZE;
     }
-    if (has_auto_dismiss && !auto_dismiss) {
+    if (!auto_dismiss) {
         job_flags |= JOB_MANUAL_DISMISS;
     }
 
@@ -3844,10 +3752,6 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     aio_context = bdrv_get_aio_context(bs);
     aio_context_acquire(aio_context);
 
-    if (!arg->has_mode) {
-        arg->mode = NEW_IMAGE_MODE_ABSOLUTE_PATHS;
-    }
-
     if (!arg->has_format) {
         format = (arg->mode == NEW_IMAGE_MODE_EXISTING
                   ? NULL : bs->drv->format_name);
@@ -3938,17 +3842,12 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
 
     blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs,
                            arg->has_replaces, arg->replaces, arg->sync,
-                           backing_mode, arg->has_speed, arg->speed,
+                           backing_mode, arg->speed,
                            arg->has_granularity, arg->granularity,
                            arg->has_buf_size, arg->buf_size,
-                           arg->has_on_source_error, arg->on_source_error,
-                           arg->has_on_target_error, arg->on_target_error,
-                           arg->has_unmap, arg->unmap,
-                           false, NULL,
-                           arg->has_copy_mode, arg->copy_mode,
-                           arg->has_auto_finalize, arg->auto_finalize,
-                           arg->has_auto_dismiss, arg->auto_dismiss,
-                           &local_err);
+                           arg->on_source_error, arg->on_target_error,
+                           arg->unmap, false, NULL, arg->copy_mode,
+                           arg->auto_finalize, arg->auto_dismiss, &local_err);
     bdrv_unref(target_bs);
     error_propagate(errp, local_err);
 out:
@@ -3958,20 +3857,15 @@ out:
 void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
                          const char *device, const char *target,
                          bool has_replaces, const char *replaces,
-                         MirrorSyncMode sync,
-                         bool has_speed, int64_t speed,
+                         MirrorSyncMode sync, int64_t speed,
                          bool has_granularity, uint32_t granularity,
                          bool has_buf_size, int64_t buf_size,
-                         bool has_on_source_error,
                          BlockdevOnError on_source_error,
-                         bool has_on_target_error,
                          BlockdevOnError on_target_error,
                          bool has_filter_node_name,
                          const char *filter_node_name,
-                         bool has_copy_mode, MirrorCopyMode copy_mode,
-                         bool has_auto_finalize, bool auto_finalize,
-                         bool has_auto_dismiss, bool auto_dismiss,
-                         Error **errp)
+                         MirrorCopyMode copy_mode,
+                         bool auto_finalize, bool auto_dismiss, Error **errp)
 {
     BlockDriverState *bs;
     BlockDriverState *target_bs;
@@ -4000,17 +3894,11 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
 
     blockdev_mirror_common(has_job_id ? job_id : NULL, bs, target_bs,
                            has_replaces, replaces, sync, backing_mode,
-                           has_speed, speed,
-                           has_granularity, granularity,
+                           speed, has_granularity, granularity,
                            has_buf_size, buf_size,
-                           has_on_source_error, on_source_error,
-                           has_on_target_error, on_target_error,
-                           true, true,
+                           on_source_error, on_target_error, true,
                            has_filter_node_name, filter_node_name,
-                           has_copy_mode, copy_mode,
-                           has_auto_finalize, auto_finalize,
-                           has_auto_dismiss, auto_dismiss,
-                           &local_err);
+                           copy_mode, auto_finalize, auto_dismiss, &local_err);
     error_propagate(errp, local_err);
 out:
     aio_context_release(aio_context);
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index c283dde0e9..486feccdc5 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -1359,9 +1359,14 @@ void hmp_drive_mirror(Monitor *mon, const QDict *qdict)
         .has_format = !!format,
         .format = (char *)format,
         .sync = full ? MIRROR_SYNC_MODE_FULL : MIRROR_SYNC_MODE_TOP,
-        .has_mode = true,
         .mode = reuse ? NEW_IMAGE_MODE_EXISTING : NEW_IMAGE_MODE_ABSOLUTE_PATHS,
+        .speed = 0,
+        .on_source_error = BLOCKDEV_ON_ERROR_REPORT,
+        .on_target_error = BLOCKDEV_ON_ERROR_REPORT,
         .unmap = true,
+        .copy_mode = MIRROR_COPY_MODE_BACKGROUND,
+        .auto_finalize = true,
+        .auto_dismiss = true,
     };
 
     if (!filename) {
@@ -1388,10 +1393,13 @@ void hmp_drive_backup(Monitor *mon, const QDict *qdict)
         .has_format = !!format,
         .format = (char *)format,
         .sync = full ? MIRROR_SYNC_MODE_FULL : MIRROR_SYNC_MODE_TOP,
-        .has_mode = true,
         .mode = reuse ? NEW_IMAGE_MODE_EXISTING : NEW_IMAGE_MODE_ABSOLUTE_PATHS,
-        .has_compress = !!compress,
+        .speed = 0,
         .compress = compress,
+        .on_source_error = BLOCKDEV_ON_ERROR_REPORT,
+        .on_target_error = BLOCKDEV_ON_ERROR_REPORT,
+        .auto_finalize = true,
+        .auto_dismiss = true,
     };
 
     if (!filename) {
@@ -1408,7 +1416,7 @@ void hmp_snapshot_blkdev(Monitor *mon, const QDict *qdict)
 {
     const char *device = qdict_get_str(qdict, "device");
     const char *filename = qdict_get_try_str(qdict, "snapshot-file");
-    const char *format = qdict_get_try_str(qdict, "format");
+    const char *format = qdict_get_try_str(qdict, "format") ?: "qcow2";
     bool reuse = qdict_get_try_bool(qdict, "reuse", false);
     enum NewImageMode mode;
     Error *err = NULL;
@@ -1424,8 +1432,7 @@ void hmp_snapshot_blkdev(Monitor *mon, const QDict *qdict)
     mode = reuse ? NEW_IMAGE_MODE_EXISTING : NEW_IMAGE_MODE_ABSOLUTE_PATHS;
     qmp_blockdev_snapshot_sync(true, device, false, NULL,
                                filename, false, NULL,
-                               !!format, format,
-                               true, mode, &err);
+                               format, mode, &err);
     hmp_handle_error(mon, &err);
 }
 
@@ -1978,11 +1985,12 @@ void hmp_change(Monitor *mon, const QDict *qdict)
                 hmp_handle_error(mon, &err);
                 return;
             }
+        } else {
+            read_only_mode = BLOCKDEV_CHANGE_READ_ONLY_MODE_RETAIN;
         }
 
         qmp_blockdev_change_medium(true, device, false, NULL, target,
-                                   !!arg, arg, !!read_only, read_only_mode,
-                                   &err);
+                                   !!arg, arg, read_only_mode, &err);
     }
 
     hmp_handle_error(mon, &err);
@@ -2024,8 +2032,7 @@ void hmp_block_stream(Monitor *mon, const QDict *qdict)
     int64_t speed = qdict_get_try_int(qdict, "speed", 0);
 
     qmp_block_stream(true, device, device, base != NULL, base, false, NULL,
-                     false, NULL, qdict_haskey(qdict, "speed"), speed, true,
-                     BLOCKDEV_ON_ERROR_REPORT, false, false, false, false,
+                     false, NULL, speed, BLOCKDEV_ON_ERROR_REPORT, true, true,
                      &error);
 
     hmp_handle_error(mon, &error);
diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index 01ce77e129..55eaf010b5 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -408,7 +408,8 @@ void qmp_change(const char *device, const char *target,
 #endif
     } else {
         qmp_blockdev_change_medium(true, device, false, NULL, target,
-                                   has_arg, arg, false, 0, errp);
+                                   has_arg, arg,
+                                   BLOCKDEV_CHANGE_READ_ONLY_MODE_RETAIN, errp);
     }
 }
 
-- 
2.21.0



^ permalink raw reply related	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (13 preceding siblings ...)
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 14/14] block: Make use of QAPI defaults Max Reitz
@ 2019-06-24 18:35 ` no-reply
  2019-06-24 19:04   ` Max Reitz
  2019-06-24 19:00 ` no-reply
                   ` (2 subsequent siblings)
  17 siblings, 1 reply; 41+ messages in thread
From: no-reply @ 2019-06-24 18:35 UTC (permalink / raw)
  To: mreitz; +Cc: kwolf, mdroth, qemu-block, qemu-devel, armbru, mreitz

Patchew URL: https://patchew.org/QEMU/20190624173935.25747-1-mreitz@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20190624173935.25747-1-mreitz@redhat.com
Type: series
Subject: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
ddd669a block: Make use of QAPI defaults
3bcf922 iotests: qcow2's encrypt.format is now optional
155c1db iotests: Test internal option typing
364eac1 block: Try to create well typed json:{} filenames
11726d2 qapi: Formalize qcow encryption probing
567644f qapi: Formalize qcow2 encryption probing
4eb3ca4 tests: Add QAPI optional discriminator tests
f3d9f53 tests: Test QAPI default values for struct members
2862824 test-qapi: Print struct members' default values
ff7a7c5 qapi: Document default values for struct members
faef37f qapi: Allow optional discriminators
64ae73c qapi: Introduce default values for struct members
f485950 qapi: Move to_c_string() to common.py
8711bb4 qapi: Parse numeric values

=== OUTPUT BEGIN ===
1/14 Checking commit 8711bb4c30f3 (qapi: Parse numeric values)
2/14 Checking commit f485950c4595 (qapi: Move to_c_string() to common.py)
3/14 Checking commit 64ae73cfedd6 (qapi: Introduce default values for struct members)
4/14 Checking commit faef37f6cd45 (qapi: Allow optional discriminators)
5/14 Checking commit ff7a7c5b6024 (qapi: Document default values for struct members)
6/14 Checking commit 28628249dcf1 (test-qapi: Print struct members' default values)
7/14 Checking commit f3d9f5343a99 (tests: Test QAPI default values for struct members)
ERROR: Invalid UTF-8, patch and commit message should be encoded in UTF-8
#106: FILE: tests/qapi-schema/qapi-schema-test.out:420:
+    member str: str optional=True default=foo \鹿""'
                                                    ^

WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#111: 
new file mode 100644

total: 1 errors, 1 warnings, 162 lines checked

Patch 7/14 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

8/14 Checking commit 4eb3ca457ca3 (tests: Add QAPI optional discriminator tests)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#33: 
new file mode 100644

total: 0 errors, 1 warnings, 67 lines checked

Patch 8/14 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
9/14 Checking commit 567644f6aaa8 (qapi: Formalize qcow2 encryption probing)
10/14 Checking commit 11726d25c367 (qapi: Formalize qcow encryption probing)
11/14 Checking commit 364eac1f31fc (block: Try to create well typed json:{} filenames)
12/14 Checking commit 155c1dbdf3f0 (iotests: Test internal option typing)
13/14 Checking commit 3bcf92270e86 (iotests: qcow2's encrypt.format is now optional)
14/14 Checking commit ddd669ad96a0 (block: Make use of QAPI defaults)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190624173935.25747-1-mreitz@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (14 preceding siblings ...)
  2019-06-24 18:35 ` [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames no-reply
@ 2019-06-24 19:00 ` no-reply
  2019-06-24 20:06   ` Max Reitz
  2019-08-19 16:49 ` Max Reitz
  2019-09-13 11:49 ` Max Reitz
  17 siblings, 1 reply; 41+ messages in thread
From: no-reply @ 2019-06-24 19:00 UTC (permalink / raw)
  To: mreitz; +Cc: kwolf, mdroth, qemu-block, qemu-devel, armbru, mreitz

Patchew URL: https://patchew.org/QEMU/20190624173935.25747-1-mreitz@redhat.com/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

PASS 1 fdc-test /x86_64/fdc/cmos
PASS 2 fdc-test /x86_64/fdc/no_media_on_start
PASS 3 fdc-test /x86_64/fdc/read_without_media
==11234==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 fdc-test /x86_64/fdc/media_change
PASS 5 fdc-test /x86_64/fdc/sense_interrupt
PASS 6 fdc-test /x86_64/fdc/relative_seek
---
PASS 32 test-opts-visitor /visitor/opts/range/beyond
PASS 33 test-opts-visitor /visitor/opts/dict/unvisited
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-coroutine -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-coroutine" 
==11269==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-coroutine /basic/no-dangling-access
==11269==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd309dc000; bottom 0x7f41b90f8000; size: 0x00bb778e4000 (805164695552)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 2 test-coroutine /basic/lifecycle
---
PASS 11 test-aio /aio/event/wait
PASS 12 test-aio /aio/event/flush
PASS 13 test-aio /aio/event/wait/no-flush-cb
==11284==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 test-aio /aio/timer/schedule
PASS 15 test-aio /aio/coroutine/queue-chaining
PASS 16 test-aio /aio-gsource/flush
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-aio-multithread -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-aio-multithread" 
PASS 12 fdc-test /x86_64/fdc/read_no_dma_19
PASS 13 fdc-test /x86_64/fdc/fuzz-registers
==11290==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-aio-multithread /aio/multi/lifecycle
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/ide-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="ide-test" 
PASS 2 test-aio-multithread /aio/multi/schedule
==11308==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 test-aio-multithread /aio/multi/mutex/contended
PASS 1 ide-test /x86_64/ide/identify
==11324==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 ide-test /x86_64/ide/flush
==11330==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 ide-test /x86_64/ide/bmdma/simple_rw
PASS 4 test-aio-multithread /aio/multi/mutex/handoff
==11336==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 ide-test /x86_64/ide/bmdma/trim
PASS 5 test-aio-multithread /aio/multi/mutex/mcs
==11347==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 ide-test /x86_64/ide/bmdma/short_prdt
PASS 6 test-aio-multithread /aio/multi/mutex/pthread
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-throttle -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-throttle" 
==11358==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11362==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-throttle /throttle/leak_bucket
PASS 2 test-throttle /throttle/compute_wait
PASS 3 test-throttle /throttle/init
---
PASS 15 test-throttle /throttle/config/iops_size
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-thread-pool -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-thread-pool" 
PASS 6 ide-test /x86_64/ide/bmdma/one_sector_short_prdt
==11371==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-thread-pool /thread-pool/submit
PASS 2 test-thread-pool /thread-pool/submit-aio
PASS 3 test-thread-pool /thread-pool/submit-co
PASS 4 test-thread-pool /thread-pool/submit-many
==11373==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 ide-test /x86_64/ide/bmdma/long_prdt
==11445==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 test-thread-pool /thread-pool/cancel
==11445==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe141a1000; bottom 0x7fd799ffe000; size: 0x00267a1a3000 (165257293824)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 8 ide-test /x86_64/ide/bmdma/no_busmaster
PASS 9 ide-test /x86_64/ide/flush/nodev
PASS 6 test-thread-pool /thread-pool/cancel-async
==11456==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-hbitmap -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-hbitmap" 
PASS 10 ide-test /x86_64/ide/flush/empty_drive
PASS 1 test-hbitmap /hbitmap/granularity
PASS 2 test-hbitmap /hbitmap/size/0
PASS 3 test-hbitmap /hbitmap/size/unaligned
PASS 4 test-hbitmap /hbitmap/iter/empty
==11466==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 test-hbitmap /hbitmap/iter/partial
PASS 11 ide-test /x86_64/ide/flush/retry_pci
PASS 6 test-hbitmap /hbitmap/iter/granularity
---
PASS 10 test-hbitmap /hbitmap/set/all
PASS 11 test-hbitmap /hbitmap/set/one
PASS 12 test-hbitmap /hbitmap/set/two-elem
==11472==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 test-hbitmap /hbitmap/set/general
PASS 14 test-hbitmap /hbitmap/set/twice
PASS 12 ide-test /x86_64/ide/flush/retry_isa
PASS 15 test-hbitmap /hbitmap/set/overlap
PASS 16 test-hbitmap /hbitmap/reset/empty
==11478==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 17 test-hbitmap /hbitmap/reset/general
PASS 13 ide-test /x86_64/ide/cdrom/pio
PASS 18 test-hbitmap /hbitmap/reset/all
---
PASS 28 test-hbitmap /hbitmap/truncate/shrink/medium
PASS 29 test-hbitmap /hbitmap/truncate/shrink/large
PASS 30 test-hbitmap /hbitmap/meta/zero
==11484==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 ide-test /x86_64/ide/cdrom/pio_large
==11490==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 15 ide-test /x86_64/ide/cdrom/dma
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/ahci-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="ahci-test" 
==11504==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 ahci-test /x86_64/ahci/sanity
==11510==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 31 test-hbitmap /hbitmap/meta/one
PASS 32 test-hbitmap /hbitmap/meta/byte
PASS 33 test-hbitmap /hbitmap/meta/word
PASS 2 ahci-test /x86_64/ahci/pci_spec
==11516==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 34 test-hbitmap /hbitmap/meta/sector
PASS 35 test-hbitmap /hbitmap/serialize/align
PASS 3 ahci-test /x86_64/ahci/pci_enable
==11522==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 ahci-test /x86_64/ahci/hba_spec
==11528==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 36 test-hbitmap /hbitmap/serialize/basic
PASS 37 test-hbitmap /hbitmap/serialize/part
PASS 38 test-hbitmap /hbitmap/serialize/zeroes
---
PASS 43 test-hbitmap /hbitmap/next_dirty_area/next_dirty_area_4
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bdrv-drain -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bdrv-drain" 
PASS 5 ahci-test /x86_64/ahci/hba_enable
==11536==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-bdrv-drain /bdrv-drain/nested
PASS 2 test-bdrv-drain /bdrv-drain/multiparent
PASS 3 test-bdrv-drain /bdrv-drain/set_aio_context
---
PASS 29 test-bdrv-drain /bdrv-drain/blockjob/iothread/drain_subtree
PASS 30 test-bdrv-drain /bdrv-drain/blockjob/iothread/error/drain_all
PASS 31 test-bdrv-drain /bdrv-drain/blockjob/iothread/error/drain
==11539==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 32 test-bdrv-drain /bdrv-drain/blockjob/iothread/error/drain_subtree
PASS 33 test-bdrv-drain /bdrv-drain/deletion/drain
PASS 34 test-bdrv-drain /bdrv-drain/detach/drain_all
---
PASS 38 test-bdrv-drain /bdrv-drain/detach/driver_cb
PASS 39 test-bdrv-drain /bdrv-drain/attach/drain
PASS 6 ahci-test /x86_64/ahci/identify
==11582==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!

=================================================================
==11536==ERROR: LeakSanitizer: detected memory leaks

Indirect leak of 4120 byte(s) in 1 object(s) allocated from:
    #0 0x56245fee9d3e in calloc (/tmp/qemu-test/build/tests/test-bdrv-drain+0x52fd3e)
---

SUMMARY: AddressSanitizer: 441420 byte(s) leaked in 525 allocation(s).
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bdrv-graph-mod -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bdrv-graph-mod" 
==11590==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-bdrv-graph-mod /bdrv-graph-mod/update-perm-tree
PASS 2 test-bdrv-graph-mod /bdrv-graph-mod/should-update-child
==11588==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!

=================================================================
==11590==ERROR: LeakSanitizer: detected memory leaks

Indirect leak of 4120 byte(s) in 1 object(s) allocated from:
    #0 0x562c894fdd2e in calloc (/tmp/qemu-test/build/tests/test-bdrv-graph-mod+0x523d2e)
---
SUMMARY: AddressSanitizer: 12621 byte(s) leaked in 15 allocation(s).
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-blockjob -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-blockjob" 
PASS 8 ahci-test /x86_64/ahci/reset
==11601==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-blockjob /blockjob/ids
PASS 2 test-blockjob /blockjob/cancel/created
PASS 3 test-blockjob /blockjob/cancel/running
---
PASS 7 test-blockjob /blockjob/cancel/pending
PASS 8 test-blockjob /blockjob/cancel/concluded
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-blockjob-txn -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-blockjob-txn" 
==11603==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11607==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-blockjob-txn /single/success
PASS 2 test-blockjob-txn /single/failure
PASS 3 test-blockjob-txn /single/cancel
---
PASS 5 test-blockjob-txn /pair/failure
PASS 6 test-blockjob-txn /pair/cancel
PASS 7 test-blockjob-txn /pair/fail-cancel-race
==11603==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc1625a000; bottom 0x7f3ddf3fe000; size: 0x00be36e5c000 (816964812800)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-block-backend -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-block-backend" 
==11617==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-block-backend /block-backend/drain_aio_error
PASS 2 test-block-backend /block-backend/drain_all_aio_error
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-block-iothread -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-block-iothread" 
PASS 9 ahci-test /x86_64/ahci/io/pio/lba28/simple/zero
==11622==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-block-iothread /sync-op/pread
PASS 2 test-block-iothread /sync-op/pwrite
PASS 3 test-block-iothread /sync-op/load_vmstate
---
PASS 14 test-block-iothread /propagate/basic
PASS 15 test-block-iothread /propagate/diamond
PASS 16 test-block-iothread /propagate/mirror
==11624==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11624==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe39ac7000; bottom 0x7f35d1bfe000; size: 0x00c867ec9000 (860737015808)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189

=================================================================
==11622==ERROR: LeakSanitizer: detected memory leaks

Indirect leak of 4120 byte(s) in 1 object(s) allocated from:
    #0 0x5649ccc53d3e in calloc (/tmp/qemu-test/build/tests/test-block-iothread+0x527d3e)
---
SUMMARY: AddressSanitizer: 16816 byte(s) leaked in 20 allocation(s).
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-image-locking -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-image-locking" 
PASS 10 ahci-test /x86_64/ahci/io/pio/lba28/simple/low
==11650==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-image-locking /image-locking/basic
PASS 2 test-image-locking /image-locking/set-perm-abort
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-x86-cpuid -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-x86-cpuid" 
==11652==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-x86-cpuid /cpuid/topology/basic
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-xbzrle -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-xbzrle" 
==11652==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe6cc1c000; bottom 0x7f6e0c3fe000; size: 0x00906081e000 (620094414848)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-xbzrle /xbzrle/uleb
---
PASS 11 ahci-test /x86_64/ahci/io/pio/lba28/simple/high
PASS 6 test-xbzrle /xbzrle/encode_decode
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-vmstate -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-vmstate" 
==11667==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-vmstate /vmstate/tmp_struct
PASS 2 test-vmstate /vmstate/simple/primitive
PASS 3 test-vmstate /vmstate/simple/array
---
PASS 133 test-cutils /cutils/strtosz/erange
PASS 134 test-cutils /cutils/strtosz/metric
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-shift128 -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-shift128" 
==11667==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe963dc000; bottom 0x7f2d855fe000; size: 0x00d110dde000 (897931141120)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-shift128 /host-utils/test_lshift
---
PASS 9 test-int128 /int128/int128_gt
PASS 10 test-int128 /int128/int128_rshift
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/rcutorture -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="rcutorture" 
==11694==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11694==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd4dbc6000; bottom 0x7f00be1fe000; size: 0x00fc8f9c8000 (1084741156864)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 13 ahci-test /x86_64/ahci/io/pio/lba28/double/low
PASS 1 rcutorture /rcu/torture/1reader
==11712==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11712==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffcfbce000; bottom 0x7f5020ffe000; size: 0x00afaebd0000 (754550898688)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 14 ahci-test /x86_64/ahci/io/pio/lba28/double/high
PASS 2 rcutorture /rcu/torture/10readers
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-rcu-list -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-list" 
==11734==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11734==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffbd688000; bottom 0x7f6508d7c000; size: 0x009ab490c000 (664454348800)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-rcu-list /rcu/qlist/single-threaded
PASS 15 ahci-test /x86_64/ahci/io/pio/lba28/long/zero
==11753==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11753==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffcf618000; bottom 0x7fcc7ed7c000; size: 0x00335089c000 (220394536960)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 2 test-rcu-list /rcu/qlist/short-few
PASS 16 ahci-test /x86_64/ahci/io/pio/lba28/long/low
==11780==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11780==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd38611000; bottom 0x7eff7297c000; size: 0x00fdc5c95000 (1089945030656)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 3 test-rcu-list /rcu/qlist/long-many
PASS 17 ahci-test /x86_64/ahci/io/pio/lba28/long/high
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-rcu-simpleq -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-simpleq" 
==11787==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-rcu-simpleq /rcu/qsimpleq/single-threaded
PASS 18 ahci-test /x86_64/ahci/io/pio/lba28/short/zero
==11805==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 test-rcu-simpleq /rcu/qsimpleq/short-few
PASS 19 ahci-test /x86_64/ahci/io/pio/lba28/short/low
==11832==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 20 ahci-test /x86_64/ahci/io/pio/lba28/short/high
==11838==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 test-rcu-simpleq /rcu/qsimpleq/long-many
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-rcu-tailq -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-tailq" 
==11838==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe89f51000; bottom 0x7f7e405fe000; size: 0x008049953000 (550990327808)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 21 ahci-test /x86_64/ahci/io/pio/lba48/simple/zero
==11851==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-rcu-tailq /rcu/qtailq/single-threaded
==11851==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffebaf5b000; bottom 0x7fc9cc5fe000; size: 0x0034ee95d000 (227341094912)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 22 ahci-test /x86_64/ahci/io/pio/lba48/simple/low
PASS 2 test-rcu-tailq /rcu/qtailq/short-few
==11863==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11863==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffda879000; bottom 0x7f6de61fe000; size: 0x0091f467b000 (626870693888)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 23 ahci-test /x86_64/ahci/io/pio/lba48/simple/high
==11890==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11890==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffe52e2000; bottom 0x7f895effe000; size: 0x0076862e4000 (509057318912)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 3 test-rcu-tailq /rcu/qtailq/long-many
---
PASS 7 test-qdist /qdist/binning/expand
PASS 8 test-qdist /qdist/binning/shrink
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-qht -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qht" 
==11901==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11901==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc41b31000; bottom 0x7f420dffe000; size: 0x00ba33b33000 (799731298304)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 25 ahci-test /x86_64/ahci/io/pio/lba48/double/low
==11911==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11911==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd2c1a6000; bottom 0x7f4a5c1fe000; size: 0x00b2cffa8000 (767993479168)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 26 ahci-test /x86_64/ahci/io/pio/lba48/double/high
==11917==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11917==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd08cef000; bottom 0x7f6550ffe000; size: 0x0097b7cf1000 (651623862272)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 27 ahci-test /x86_64/ahci/io/pio/lba48/long/zero
==11923==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11923==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffdc8a16000; bottom 0x7f61e9dfe000; size: 0x009bdec18000 (669457154048)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 28 ahci-test /x86_64/ahci/io/pio/lba48/long/low
==11929==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11929==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff8e660000; bottom 0x7f3c131fe000; size: 0x00c37b462000 (839586816000)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 29 ahci-test /x86_64/ahci/io/pio/lba48/long/high
==11935==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 30 ahci-test /x86_64/ahci/io/pio/lba48/short/zero
==11941==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 31 ahci-test /x86_64/ahci/io/pio/lba48/short/low
==11947==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 32 ahci-test /x86_64/ahci/io/pio/lba48/short/high
==11953==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 33 ahci-test /x86_64/ahci/io/dma/lba28/fragmented
==11959==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-qht /qht/mode/default
PASS 34 ahci-test /x86_64/ahci/io/dma/lba28/retry
PASS 2 test-qht /qht/mode/resize
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-qht-par -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qht-par" 
==11965==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 35 ahci-test /x86_64/ahci/io/dma/lba28/simple/zero
PASS 1 test-qht-par /qht/parallel/2threads-0%updates-1s
==11981==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 36 ahci-test /x86_64/ahci/io/dma/lba28/simple/low
PASS 2 test-qht-par /qht/parallel/2threads-20%updates-1s
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bitops -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bitops" 
---
PASS 5 test-bitops /bitops/half_unshuffle32
PASS 6 test-bitops /bitops/half_unshuffle64
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bitcnt -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bitcnt" 
==11994==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-bitcnt /bitcnt/ctpop8
PASS 2 test-bitcnt /bitcnt/ctpop16
PASS 3 test-bitcnt /bitcnt/ctpop32
---
PASS 18 test-qemu-opts /qemu-opts/to_qdict/filtered
PASS 19 test-qemu-opts /qemu-opts/to_qdict/duplicates
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-keyval -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-keyval" 
==12025==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-keyval /keyval/keyval_parse
PASS 2 test-keyval /keyval/keyval_parse/list
PASS 3 test-keyval /keyval/visit/bool
---
PASS 4 test-crypto-hash /crypto/hash/digest
PASS 5 test-crypto-hash /crypto/hash/base64
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-crypto-hmac -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-hmac" 
==12048==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-crypto-hmac /crypto/hmac/iov
PASS 2 test-crypto-hmac /crypto/hmac/alloc
PASS 3 test-crypto-hmac /crypto/hmac/prealloc
---
PASS 16 test-crypto-secret /crypto/secret/crypt/badiv
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-crypto-tlscredsx509 -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-tlscredsx509" 
PASS 39 ahci-test /x86_64/ahci/io/dma/lba28/double/low
==12077==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/perfectserver
PASS 2 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/perfectclient
PASS 3 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodca1
PASS 40 ahci-test /x86_64/ahci/io/dma/lba28/double/high
PASS 4 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodca2
==12084==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodca3
PASS 6 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/badca1
PASS 7 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/badca2
PASS 8 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/badca3
PASS 41 ahci-test /x86_64/ahci/io/dma/lba28/long/zero
==12090==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver1
PASS 10 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver2
PASS 42 ahci-test /x86_64/ahci/io/dma/lba28/long/low
PASS 11 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver3
PASS 12 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver4
==12096==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver5
PASS 14 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver6
PASS 15 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver7
---
PASS 37 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/missingca
PASS 38 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/missingserver
PASS 39 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/missingclient
==12102==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-crypto-tlssession -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-tlssession" 
PASS 44 ahci-test /x86_64/ahci/io/dma/lba28/short/zero
==12113==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-crypto-tlssession /qcrypto/tlssession/psk
PASS 2 test-crypto-tlssession /qcrypto/tlssession/basicca
PASS 45 ahci-test /x86_64/ahci/io/dma/lba28/short/low
PASS 3 test-crypto-tlssession /qcrypto/tlssession/differentca
==12119==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 46 ahci-test /x86_64/ahci/io/dma/lba28/short/high
PASS 4 test-crypto-tlssession /qcrypto/tlssession/altname1
PASS 5 test-crypto-tlssession /qcrypto/tlssession/altname2
==12125==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 47 ahci-test /x86_64/ahci/io/dma/lba48/simple/zero
==12131==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 test-crypto-tlssession /qcrypto/tlssession/altname3
PASS 7 test-crypto-tlssession /qcrypto/tlssession/altname4
PASS 48 ahci-test /x86_64/ahci/io/dma/lba48/simple/low
PASS 8 test-crypto-tlssession /qcrypto/tlssession/altname5
==12137==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 test-crypto-tlssession /qcrypto/tlssession/altname6
PASS 49 ahci-test /x86_64/ahci/io/dma/lba48/simple/high
==12143==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 50 ahci-test /x86_64/ahci/io/dma/lba48/double/zero
PASS 10 test-crypto-tlssession /qcrypto/tlssession/wildcard1
==12149==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 11 test-crypto-tlssession /qcrypto/tlssession/wildcard2
PASS 51 ahci-test /x86_64/ahci/io/dma/lba48/double/low
PASS 12 test-crypto-tlssession /qcrypto/tlssession/wildcard3
==12155==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 test-crypto-tlssession /qcrypto/tlssession/wildcard4
PASS 14 test-crypto-tlssession /qcrypto/tlssession/wildcard5
PASS 52 ahci-test /x86_64/ahci/io/dma/lba48/double/high
PASS 15 test-crypto-tlssession /qcrypto/tlssession/wildcard6
PASS 16 test-crypto-tlssession /qcrypto/tlssession/cachain
==12162==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-qga -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qga" 
PASS 53 ahci-test /x86_64/ahci/io/dma/lba48/long/zero
PASS 1 test-qga /qga/sync-delimited
---
PASS 15 test-qga /qga/invalid-cmd
PASS 16 test-qga /qga/invalid-args
PASS 17 test-qga /qga/fsfreeze-status
==12174==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 54 ahci-test /x86_64/ahci/io/dma/lba48/long/low
PASS 18 test-qga /qga/blacklist
PASS 19 test-qga /qga/config
PASS 20 test-qga /qga/guest-exec
PASS 21 test-qga /qga/guest-exec-invalid
==12181==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 55 ahci-test /x86_64/ahci/io/dma/lba48/long/high
PASS 22 test-qga /qga/guest-get-osinfo
PASS 23 test-qga /qga/guest-get-host-name
PASS 24 test-qga /qga/guest-get-timezone
PASS 25 test-qga /qga/guest-get-users
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-timed-average -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-timed-average" 
==12194==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-timed-average /timed-average/average
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-util-filemonitor -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-util-filemonitor" 
PASS 1 test-util-filemonitor /util/filemonitor
---
PASS 4 test-io-task /crypto/task/thread_complete
PASS 5 test-io-task /crypto/task/thread_failure
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-io-channel-socket -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-io-channel-socket" 
==12228==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-io-channel-socket /io/channel/socket/ipv4-sync
PASS 2 test-io-channel-socket /io/channel/socket/ipv4-async
PASS 3 test-io-channel-socket /io/channel/socket/ipv4-fd
---
PASS 57 ahci-test /x86_64/ahci/io/dma/lba48/short/low
PASS 1 test-io-channel-tls /qio/channel/tls/basic
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-io-channel-command -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-io-channel-command" 
==12300==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-io-channel-command /io/channel/command/fifo/sync
PASS 2 test-io-channel-command /io/channel/command/fifo/async
PASS 3 test-io-channel-command /io/channel/command/echo/sync
---
PASS 8 test-crypto-ivgen /crypto/ivgen/essiv/1f2e3d4c
PASS 9 test-crypto-ivgen /crypto/ivgen/essiv/1f2e3d4c5b6a7988
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-crypto-afsplit -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-afsplit" 
==12328==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-crypto-afsplit /crypto/afsplit/sha256/5
PASS 2 test-crypto-afsplit /crypto/afsplit/sha256/5000
PASS 3 test-crypto-afsplit /crypto/afsplit/sha256/big
---
PASS 1 test-logging /logging/parse_range
PASS 2 test-logging /logging/parse_path
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-replication -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-replication" 
==12356==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==12362==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-replication /replication/primary/read
PASS 2 test-replication /replication/primary/write
PASS 60 ahci-test /x86_64/ahci/io/ncq/retry
==12371==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 test-replication /replication/primary/start
PASS 4 test-replication /replication/primary/stop
PASS 5 test-replication /replication/primary/do_checkpoint
PASS 6 test-replication /replication/primary/get_error_all
PASS 61 ahci-test /x86_64/ahci/flush/simple
==12377==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 test-replication /replication/secondary/read
PASS 62 ahci-test /x86_64/ahci/flush/retry
==12383==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 test-replication /replication/secondary/write
==12388==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==12362==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd4139e000; bottom 0x7f9bdfffc000; size: 0x0061613a2000 (418243026944)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 63 ahci-test /x86_64/ahci/flush/migrate
PASS 9 test-replication /replication/secondary/start
==12417==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==12422==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 64 ahci-test /x86_64/ahci/migrate/sanity
==12431==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 test-replication /replication/secondary/stop
==12436==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 11 test-replication /replication/secondary/do_checkpoint
PASS 12 test-replication /replication/secondary/get_error_all
PASS 65 ahci-test /x86_64/ahci/migrate/dma/simple
==12447==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!

=================================================================
==12362==ERROR: LeakSanitizer: detected memory leaks

Indirect leak of 4120 byte(s) in 1 object(s) allocated from:
    #0 0x55dccc091d1e in calloc (/tmp/qemu-test/build/tests/test-replication+0x525d1e)
---
    #12 0x7f9bedfdef9d  (/lib64/libglib-2.0.so.0+0x77f9d)

SUMMARY: AddressSanitizer: 119042 byte(s) leaked in 506 allocation(s).
==12453==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bufferiszero -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bufferiszero" 
PASS 66 ahci-test /x86_64/ahci/migrate/dma/halted
==12465==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==12470==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 67 ahci-test /x86_64/ahci/migrate/ncq/simple
==12479==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==12484==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 68 ahci-test /x86_64/ahci/migrate/ncq/halted
==12493==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 69 ahci-test /x86_64/ahci/cdrom/eject
==12498==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 70 ahci-test /x86_64/ahci/cdrom/dma/single
==12504==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 71 ahci-test /x86_64/ahci/cdrom/dma/multi
==12510==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 72 ahci-test /x86_64/ahci/cdrom/pio/single
==12516==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==12516==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe47d5a000; bottom 0x7f68275dc000; size: 0x00962077e000 (644789821440)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 73 ahci-test /x86_64/ahci/cdrom/pio/multi
==12522==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 74 ahci-test /x86_64/ahci/cdrom/pio/bcl
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/hd-geo-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="hd-geo-test" 
PASS 1 hd-geo-test /x86_64/hd-geo/ide/none
==12536==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 hd-geo-test /x86_64/hd-geo/ide/drive/cd_0
==12542==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/blank
==12548==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/lba
==12554==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/chs
==12560==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 hd-geo-test /x86_64/hd-geo/ide/device/mbr/blank
==12566==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 hd-geo-test /x86_64/hd-geo/ide/device/mbr/lba
==12572==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 hd-geo-test /x86_64/hd-geo/ide/device/mbr/chs
==12578==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 hd-geo-test /x86_64/hd-geo/ide/device/user/chs
==12583==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 hd-geo-test /x86_64/hd-geo/ide/device/user/chst
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/boot-order-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="boot-order-test" 
PASS 1 test-bufferiszero /cutils/bufferiszero
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12668==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 bios-tables-test /x86_64/acpi/piix4
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12674==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 bios-tables-test /x86_64/acpi/q35
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12680==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 bios-tables-test /x86_64/acpi/piix4/bridge
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12686==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 bios-tables-test /x86_64/acpi/piix4/ipmi
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12692==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 bios-tables-test /x86_64/acpi/piix4/cpuhp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12699==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 bios-tables-test /x86_64/acpi/piix4/memhp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12705==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 bios-tables-test /x86_64/acpi/piix4/numamem
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12711==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 bios-tables-test /x86_64/acpi/piix4/dimmpxm
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12720==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 bios-tables-test /x86_64/acpi/q35/bridge
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12726==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 bios-tables-test /x86_64/acpi/q35/mmio64
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12732==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 11 bios-tables-test /x86_64/acpi/q35/ipmi
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12738==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 12 bios-tables-test /x86_64/acpi/q35/cpuhp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12745==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 bios-tables-test /x86_64/acpi/q35/memhp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12751==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 bios-tables-test /x86_64/acpi/q35/numamem
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12757==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 15 bios-tables-test /x86_64/acpi/q35/dimmpxm
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/boot-serial-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="boot-serial-test" 
PASS 1 boot-serial-test /x86_64/boot-serial/isapc
---
PASS 1 i440fx-test /x86_64/i440fx/defaults
PASS 2 i440fx-test /x86_64/i440fx/pam
PASS 3 i440fx-test /x86_64/i440fx/firmware/bios
==12841==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 i440fx-test /x86_64/i440fx/firmware/pflash
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/fw_cfg-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="fw_cfg-test" 
PASS 1 fw_cfg-test /x86_64/fw_cfg/signature
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/drive_del-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="drive_del-test" 
PASS 1 drive_del-test /x86_64/drive_del/without-dev
PASS 2 drive_del-test /x86_64/drive_del/after_failed_device_add
==12929==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 drive_del-test /x86_64/blockdev/drive_del_device_del
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/wdt_ib700-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="wdt_ib700-test" 
PASS 1 wdt_ib700-test /x86_64/wdt_ib700/pause
---
PASS 1 usb-hcd-uhci-test /x86_64/uhci/pci/init
PASS 2 usb-hcd-uhci-test /x86_64/uhci/pci/port1
PASS 3 usb-hcd-uhci-test /x86_64/uhci/pci/hotplug
==13124==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 usb-hcd-uhci-test /x86_64/uhci/pci/hotplug/usb-storage
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/usb-hcd-xhci-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="usb-hcd-xhci-test" 
PASS 1 usb-hcd-xhci-test /x86_64/xhci/pci/init
PASS 2 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug
==13133==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug/usb-uas
PASS 4 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug/usb-ccid
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/cpu-plug-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="cpu-plug-test" 
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13239==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 vmgenid-test /x86_64/vmgenid/vmgenid/set-guid
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13245==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 vmgenid-test /x86_64/vmgenid/vmgenid/set-guid-auto
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13251==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 vmgenid-test /x86_64/vmgenid/vmgenid/query-monitor
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/tpm-crb-swtpm-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="tpm-crb-swtpm-test" 
SKIP 1 tpm-crb-swtpm-test /x86_64/tpm/crb-swtpm/test # SKIP swtpm not in PATH or missing --tpm2 support
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13356==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13361==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 migration-test /x86_64/migration/fd_proto
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13369==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13374==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 migration-test /x86_64/migration/postcopy/unix
PASS 5 migration-test /x86_64/migration/postcopy/recovery
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13404==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13409==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 migration-test /x86_64/migration/precopy/unix
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13418==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13423==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 migration-test /x86_64/migration/precopy/tcp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13432==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==13437==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 migration-test /x86_64/migration/xbzrle/unix
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/test-x86-cpuid-compat -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-x86-cpuid-compat" 
PASS 1 test-x86-cpuid-compat /x86/cpuid/parsing-plus-minus
---
PASS 6 numa-test /x86_64/numa/pc/dynamic/cpu
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qmp-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="qmp-test" 
PASS 1 qmp-test /x86_64/qmp/protocol
==13766==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 qmp-test /x86_64/qmp/oob
PASS 3 qmp-test /x86_64/qmp/preconfig
PASS 4 qmp-test /x86_64/qmp/missing-any-arg
---
PASS 6 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/sdhci-pci/sdhci/sdhci-tests/registers
PASS 7 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/tpci200/ipack/ipoctal232/ipoctal232-tests/nop
PASS 8 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/tpci200/pci-device/pci-device-tests/nop
==14175==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/pci-device/pci-device-tests/nop
PASS 10 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio/virtio-tests/nop
PASS 11 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-9p/virtio-9p-tests/config
---
PASS 20 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-9p/virtio-9p-tests/fs/flush/ignored
PASS 21 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-balloon-pci/pci-device/pci-device-tests/nop
PASS 22 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-balloon-pci/virtio/virtio-tests/nop
==14188==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 23 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk/virtio-blk-tests/indirect
==14195==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 24 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk/virtio-blk-tests/config
==14202==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 25 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk/virtio-blk-tests/basic
==14209==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 26 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk/virtio-blk-tests/resize
==14216==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 27 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk-pci-tests/msix
==14223==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 28 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk-pci-tests/idx
==14230==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 29 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk-pci-tests/nxvirtq
==14237==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 30 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk-pci-tests/hotplug
PASS 31 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/basic
PASS 32 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/rx_stop_cont
---
PASS 40 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-rng-pci/pci-device/pci-device-tests/nop
PASS 41 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-rng-pci/virtio/virtio-tests/nop
PASS 42 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-rng-pci/virtio-rng-pci-tests/hotplug
==14348==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 43 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-scsi-pci/pci-device/pci-device-tests/nop
==14354==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 44 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-scsi-pci/virtio-scsi/virtio-scsi-tests/hotplug
==14360==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 45 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-scsi-pci/virtio-scsi/virtio-scsi-tests/unaligned-write-same

=================================================================
==14360==ERROR: LeakSanitizer: detected memory leaks

Indirect leak of 4120 byte(s) in 1 object(s) allocated from:
    #0 0x55a747851abe in calloc (/tmp/qemu-test/build/x86_64-softmmu/qemu-system-x86_64+0x19e9abe)
---

SUMMARY: AddressSanitizer: 4382 byte(s) leaked in 13 allocation(s).
/tmp/qemu-test/src/tests/libqtest.c:137: kill_qemu() tried to terminate QEMU process but encountered exit status 1
ERROR - too few tests run (expected 78, got 45)
make: *** [/tmp/qemu-test/src/tests/Makefile.include:909: check-qtest-x86_64] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):


The full log is available at
http://patchew.org/logs/20190624173935.25747-1-mreitz@redhat.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-06-24 18:35 ` [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames no-reply
@ 2019-06-24 19:04   ` Max Reitz
  2019-06-24 19:06     ` Max Reitz
  0 siblings, 1 reply; 41+ messages in thread
From: Max Reitz @ 2019-06-24 19:04 UTC (permalink / raw)
  To: qemu-devel; +Cc: kwolf, mdroth, qemu-block, armbru


[-- Attachment #1.1: Type: text/plain, Size: 2388 bytes --]

On 24.06.19 20:35, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20190624173935.25747-1-mreitz@redhat.com/
> 
> 
> 
> Hi,
> 
> This series seems to have some coding style problems. See output below for
> more information:
> 
> Message-id: 20190624173935.25747-1-mreitz@redhat.com
> Type: series
> Subject: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> git rev-parse base > /dev/null || exit 0
> git config --local diff.renamelimit 0
> git config --local diff.renames True
> git config --local diff.algorithm histogram
> ./scripts/checkpatch.pl --mailback base..
> === TEST SCRIPT END ===
> 
> Switched to a new branch 'test'
> ddd669a block: Make use of QAPI defaults
> 3bcf922 iotests: qcow2's encrypt.format is now optional
> 155c1db iotests: Test internal option typing
> 364eac1 block: Try to create well typed json:{} filenames
> 11726d2 qapi: Formalize qcow encryption probing
> 567644f qapi: Formalize qcow2 encryption probing
> 4eb3ca4 tests: Add QAPI optional discriminator tests
> f3d9f53 tests: Test QAPI default values for struct members
> 2862824 test-qapi: Print struct members' default values
> ff7a7c5 qapi: Document default values for struct members
> faef37f qapi: Allow optional discriminators
> 64ae73c qapi: Introduce default values for struct members
> f485950 qapi: Move to_c_string() to common.py
> 8711bb4 qapi: Parse numeric values
> 
> === OUTPUT BEGIN ===
> 1/14 Checking commit 8711bb4c30f3 (qapi: Parse numeric values)
> 2/14 Checking commit f485950c4595 (qapi: Move to_c_string() to common.py)
> 3/14 Checking commit 64ae73cfedd6 (qapi: Introduce default values for struct members)
> 4/14 Checking commit faef37f6cd45 (qapi: Allow optional discriminators)
> 5/14 Checking commit ff7a7c5b6024 (qapi: Document default values for struct members)
> 6/14 Checking commit 28628249dcf1 (test-qapi: Print struct members' default values)
> 7/14 Checking commit f3d9f5343a99 (tests: Test QAPI default values for struct members)
> ERROR: Invalid UTF-8, patch and commit message should be encoded in UTF-8
> #106: FILE: tests/qapi-schema/qapi-schema-test.out:420:
> +    member str: str optional=True default=foo \鹿""'
>                                                     ^

Already noted in patch 7.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-06-24 19:04   ` Max Reitz
@ 2019-06-24 19:06     ` Max Reitz
  0 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 19:06 UTC (permalink / raw)
  To: qemu-devel; +Cc: kwolf, mdroth, qemu-block, armbru


[-- Attachment #1.1: Type: text/plain, Size: 2810 bytes --]

On 24.06.19 21:04, Max Reitz wrote:
> On 24.06.19 20:35, no-reply@patchew.org wrote:
>> Patchew URL: https://patchew.org/QEMU/20190624173935.25747-1-mreitz@redhat.com/
>>
>>
>>
>> Hi,
>>
>> This series seems to have some coding style problems. See output below for
>> more information:
>>
>> Message-id: 20190624173935.25747-1-mreitz@redhat.com
>> Type: series
>> Subject: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
>>
>> === TEST SCRIPT BEGIN ===
>> #!/bin/bash
>> git rev-parse base > /dev/null || exit 0
>> git config --local diff.renamelimit 0
>> git config --local diff.renames True
>> git config --local diff.algorithm histogram
>> ./scripts/checkpatch.pl --mailback base..
>> === TEST SCRIPT END ===
>>
>> Switched to a new branch 'test'
>> ddd669a block: Make use of QAPI defaults
>> 3bcf922 iotests: qcow2's encrypt.format is now optional
>> 155c1db iotests: Test internal option typing
>> 364eac1 block: Try to create well typed json:{} filenames
>> 11726d2 qapi: Formalize qcow encryption probing
>> 567644f qapi: Formalize qcow2 encryption probing
>> 4eb3ca4 tests: Add QAPI optional discriminator tests
>> f3d9f53 tests: Test QAPI default values for struct members
>> 2862824 test-qapi: Print struct members' default values
>> ff7a7c5 qapi: Document default values for struct members
>> faef37f qapi: Allow optional discriminators
>> 64ae73c qapi: Introduce default values for struct members
>> f485950 qapi: Move to_c_string() to common.py
>> 8711bb4 qapi: Parse numeric values
>>
>> === OUTPUT BEGIN ===
>> 1/14 Checking commit 8711bb4c30f3 (qapi: Parse numeric values)
>> 2/14 Checking commit f485950c4595 (qapi: Move to_c_string() to common.py)
>> 3/14 Checking commit 64ae73cfedd6 (qapi: Introduce default values for struct members)
>> 4/14 Checking commit faef37f6cd45 (qapi: Allow optional discriminators)
>> 5/14 Checking commit ff7a7c5b6024 (qapi: Document default values for struct members)
>> 6/14 Checking commit 28628249dcf1 (test-qapi: Print struct members' default values)
>> 7/14 Checking commit f3d9f5343a99 (tests: Test QAPI default values for struct members)
>> ERROR: Invalid UTF-8, patch and commit message should be encoded in UTF-8
>> #106: FILE: tests/qapi-schema/qapi-schema-test.out:420:
>> +    member str: str optional=True default=foo \鹿""'
>>                                                     ^
> 
> Already noted in patch 7.

Oops.  Didn’t note it in patch 7 because I reformatted the patches
afterwards and thus deleted my note. m(

Yes, it’s a test output and I’d like to test weird control characters,
mostly that the generated C code doesn’t break.  Any ideas how to fix
this?  I could just use \t instead of \b\x7f, but that would be boring.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-06-24 19:00 ` no-reply
@ 2019-06-24 20:06   ` Max Reitz
  0 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 20:06 UTC (permalink / raw)
  To: qemu-devel; +Cc: kwolf, mdroth, qemu-block, armbru


[-- Attachment #1.1: Type: text/plain, Size: 863 bytes --]

On 24.06.19 21:00, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20190624173935.25747-1-mreitz@redhat.com/
> 
> 
> 
> Hi,
> 
> This series failed the asan build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.

Many ASan leaks of the form:

> Indirect leak of 4120 byte(s) in 1 object(s) allocated from:
>     #0 0x56245fee9d3e in calloc (/tmp/qemu-test/build/tests/test-bdrv-drain+0x52fd3e)
>     #1 0x7f285054bcf0 in g_malloc0 (/lib64/libglib-2.0.so.0+0x55cf0)
>     #2 0x562460610cf3 in qdict_clone_shallow /tmp/qemu-test/src/qobject/qdict.c:367:12
>     #3 0x56245ff8487b in bdrv_type_blockdev_opts /tmp/qemu-test/src/block.c:6305:22

Oops, I forgot to free string_options in bdrv_type_blockdev_opts().
Thanks, Patchew!

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 11/14] block: Try to create well typed json:{} filenames
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 11/14] block: Try to create well typed json:{} filenames Max Reitz
@ 2019-06-24 20:06   ` Max Reitz
  0 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-06-24 20:06 UTC (permalink / raw)
  To: qemu-block; +Cc: Michael Roth, Kevin Wolf, qemu-devel, Markus Armbruster


[-- Attachment #1.1: Type: text/plain, Size: 8970 bytes --]

On 24.06.19 19:39, Max Reitz wrote:
> By applying qdict_flatten(), the flat-confused input visitor, and the
> output visitor, we can at least try to bring bs->full_open_options into
> accordance with the QAPI schema.  This may not always work (there are
> some options left that have not been QAPI-fied yet), but in practice it
> usually will.
> 
> In any case, sometimes emitting wrongly typed json:{} filenames is
> better than doing it effectively half the time.
> 
> This affects some iotests because json:{} filenames are now usually
> crumpled.  In 198, "format": "auto" now appears in the qcow2 encryption
> options because going through a visitor makes optional members' default
> values explicit.
> 
> Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1534396
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  block.c                    | 68 +++++++++++++++++++++++++++++++++++++-
>  tests/qemu-iotests/059.out |  2 +-
>  tests/qemu-iotests/099.out |  4 +--
>  tests/qemu-iotests/110.out |  2 +-
>  tests/qemu-iotests/198.out |  4 +--
>  5 files changed, 73 insertions(+), 7 deletions(-)
> 
> diff --git a/block.c b/block.c
> index c139540f2b..d3c1041087 100644
> --- a/block.c
> +++ b/block.c
> @@ -36,6 +36,7 @@
>  #include "qapi/qmp/qjson.h"
>  #include "qapi/qmp/qnull.h"
>  #include "qapi/qmp/qstring.h"
> +#include "qapi/qobject-input-visitor.h"
>  #include "qapi/qobject-output-visitor.h"
>  #include "qapi/qapi-visit-block-core.h"
>  #include "sysemu/block-backend.h"
> @@ -6283,6 +6284,56 @@ static bool bdrv_backing_overridden(BlockDriverState *bs)
>      }
>  }
>  
> +/**
> + * Take a blockdev @options QDict and convert its values to the
> + * correct type.
> + *
> + * Fail if @options does not match the QAPI schema of BlockdevOptions.
> + *
> + * In case of failure, return NULL and set @errp.
> + *
> + * In case of success, return a correctly typed new QDict.
> + */
> +static QDict *bdrv_type_blockdev_opts(const QDict *options, Error **errp)
> +{
> +    Visitor *v;
> +    BlockdevOptions *blockdev_options;
> +    QObject *typed_opts;
> +    QDict *string_options;
> +    Error *local_err = NULL;
> +
> +    string_options = qdict_clone_shallow(options);
> +
> +    qdict_flatten(string_options);
> +    v = qobject_input_visitor_new_flat_confused(string_options, errp);

Imagine a

+    qobject_unref(string_options);

here.

Max

> +    if (!v) {
> +        error_prepend(errp, "Failed to prepare options: ");
> +        return NULL;
> +    }
> +
> +    visit_type_BlockdevOptions(v, NULL, &blockdev_options, &local_err);
> +    visit_free(v);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        error_prepend(errp, "Not a valid BlockdevOptions object: ");
> +        return NULL;
> +    }
> +
> +    v = qobject_output_visitor_new(&typed_opts);
> +    visit_type_BlockdevOptions(v, NULL, &blockdev_options, &local_err);
> +    if (!local_err) {
> +        visit_complete(v, &typed_opts);
> +    }
> +    visit_free(v);
> +    qapi_free_BlockdevOptions(blockdev_options);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return NULL;
> +    }
> +
> +    return qobject_to(QDict, typed_opts);
> +}
> +
>  /* Updates the following BDS fields:
>   *  - exact_filename: A filename which may be used for opening a block device
>   *                    which (mostly) equals the given BDS (even without any
> @@ -6400,10 +6451,25 @@ void bdrv_refresh_filename(BlockDriverState *bs)
>      if (bs->exact_filename[0]) {
>          pstrcpy(bs->filename, sizeof(bs->filename), bs->exact_filename);
>      } else {
> -        QString *json = qobject_to_json(QOBJECT(bs->full_open_options));
> +        QString *json;
> +        QDict *typed_opts, *json_opts;
> +
> +        typed_opts = bdrv_type_blockdev_opts(bs->full_open_options, NULL);
> +
> +        /*
> +         * We cannot be certain that bs->full_open_options matches
> +         * BlockdevOptions, so bdrv_type_blockdev_opts() may fail.
> +         * That is not fatal, we can just emit bs->full_open_options
> +         * directly -- qemu will accept that, even if it does not
> +         * match the schema.
> +         */
> +        json_opts = typed_opts ?: bs->full_open_options;
> +
> +        json = qobject_to_json(QOBJECT(json_opts));
>          snprintf(bs->filename, sizeof(bs->filename), "json:%s",
>                   qstring_get_str(json));
>          qobject_unref(json);
> +        qobject_unref(typed_opts);
>      }
>  }
>  
> diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
> index 4fab42a28c..53109b2d49 100644
> --- a/tests/qemu-iotests/059.out
> +++ b/tests/qemu-iotests/059.out
> @@ -2050,7 +2050,7 @@ wrote 512/512 bytes at offset 10240
>  
>  === Testing monolithicFlat with internally generated JSON file name ===
>  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
> -qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
> +qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"inject-error": [{"event": "read_aio"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}'
>  
>  === Testing version 3 ===
>  image: TEST_DIR/iotest-version3.IMGFMT
> diff --git a/tests/qemu-iotests/099.out b/tests/qemu-iotests/099.out
> index 8cce627529..0a9c434148 100644
> --- a/tests/qemu-iotests/099.out
> +++ b/tests/qemu-iotests/099.out
> @@ -12,11 +12,11 @@ blkverify:TEST_DIR/t.IMGFMT.compare:TEST_DIR/t.IMGFMT
>  
>  === Testing JSON filename for blkdebug ===
>  
> -json:{"driver": "IMGFMT", "file": {"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "l1_update"}}
> +json:{"driver": "IMGFMT", "file": {"inject-error": [{"event": "l1_update"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}
>  
>  === Testing indirectly enforced JSON filename ===
>  
> -json:{"driver": "raw", "file": {"test": {"driver": "IMGFMT", "file": {"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "l1_update"}}, "driver": "blkverify", "raw": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.compare"}}}
> +json:{"driver": "raw", "file": {"test": {"driver": "IMGFMT", "file": {"inject-error": [{"event": "l1_update"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}, "driver": "blkverify", "raw": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.compare"}}}
>  
>  === Testing plain filename for blkdebug ===
>  
> diff --git a/tests/qemu-iotests/110.out b/tests/qemu-iotests/110.out
> index f60b26390e..d95b92bee7 100644
> --- a/tests/qemu-iotests/110.out
> +++ b/tests/qemu-iotests/110.out
> @@ -11,7 +11,7 @@ backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base)
>  
>  === Non-reconstructable filename ===
>  
> -image: json:{"driver": "IMGFMT", "file": {"set-state.0.event": "read_aio", "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "set-state.0.new_state": 42}}
> +image: json:{"driver": "IMGFMT", "file": {"set-state": [{"new_state": 42, "event": "read_aio"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}
>  file format: IMGFMT
>  virtual size: 64 MiB (67108864 bytes)
>  backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base)
> diff --git a/tests/qemu-iotests/198.out b/tests/qemu-iotests/198.out
> index e86b175e39..eef8af3cdc 100644
> --- a/tests/qemu-iotests/198.out
> +++ b/tests/qemu-iotests/198.out
> @@ -32,7 +32,7 @@ read 16777216/16777216 bytes at offset 0
>  16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>  
>  == checking image base ==
> -image: json:{"encrypt.key-secret": "sec0", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.base"}}
> +image: json:{"driver": "IMGFMT", "encrypt": {"format": "auto", "key-secret": "sec0"}, "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.base"}}
>  file format: IMGFMT
>  virtual size: 16 MiB (16777216 bytes)
>  Format specific information:
> @@ -74,7 +74,7 @@ Format specific information:
>          master key iters: 1024
>  
>  == checking image layer ==
> -image: json:{"encrypt.key-secret": "sec1", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}}
> +image: json:{"driver": "IMGFMT", "encrypt": {"format": "auto", "key-secret": "sec1"}, "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}}
>  file format: IMGFMT
>  virtual size: 16 MiB (16777216 bytes)
>  backing file: TEST_DIR/t.IMGFMT.base
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (15 preceding siblings ...)
  2019-06-24 19:00 ` no-reply
@ 2019-08-19 16:49 ` Max Reitz
  2019-09-13 11:49 ` Max Reitz
  17 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-08-19 16:49 UTC (permalink / raw)
  To: qemu-block; +Cc: Michael Roth, Kevin Wolf, qemu-devel, Markus Armbruster


[-- Attachment #1.1: Type: text/plain, Size: 19457 bytes --]

Ping

On 24.06.19 19:39, Max Reitz wrote:
> Hi,
> 
> There are two explanations of this cover letter, a relative one (to v3)
> and an absolute one.
> 
> 
> *** Important note ***
> 
> The final patch in this series is an example that converts most of
> block-core.json to use default values where possible.  We may decide to
> take it or not.  It isn’t important for the main purpose of this series,
> so I’d be very much fine with chopping it off.
> 
> (It does have a nice diff stat, though.)
> 
> *** Important note end ***
> 
> 
> Relative explanation:
> 
> The actual functional goal of this series is to allow all blockdev
> options that can be represented with -drive to have an equivalent with
> -blockdev (safe for rbd’s =keyvalue-pairs).
> 
> To this end, qcow(2)’s encryption needs an “auto” format which can
> automatically deduce the format from the image header.  To make things
> nicer, I decided (already in v1) to make this format optional so users
> could just specify encrypt.secret and let the format driver figure out
> the rest.
> 
> Until v3, this was implemented by letting the discriminator of flat
> unions be optional, as long as a default-value is provided.  Markus
> (rightfully) complained that this is very specific and would be covered
> by just having default values for QAPI struct members in general.
> So now this version implements this.  This is a bit more complicated
> than just implementing a default-variant, mainly because the latter only
> needs to accept enum values, whereas a generally usable “default” should
> accept values of all QAPI types (to the extent what is reasonable).
> 
> So what was (until v3)
> 
>   { 'union': 'Foo',
>     'base': { '*discr': 'SomeEnum' },
>     'discriminator': 'discr',
>     'default-variant': 'value1',
>     'data': { 'value1': 'Bar', 'value2': 'Baz' } }
> 
> becomes
> 
>   { 'union': 'Foo',
>     'base': { '*discr': { 'type': 'SomeEnum', 'default': 'value1' } },
>     'discriminator': 'discr',
>     'data': { 'value1': 'Bar', 'value2': 'Baz' } }
> 
> 
> 
> Absolute explanation:
> 
> When qemu reports json:{} filename, it just uses whatever type you gave
> an option in.  With -drive, all options are strings and they do not have
> to pass the test of the typing firewall of the QAPI schema, so you just
> get strings thrown back at you even if that does not match the schema.
> (Also, if you use json:{} yourself, you’re free to give the options as
> strings as well.)
> 
> Example:
> 
> $ ./qemu-img info --image-opts driver=raw,size=512,file.driver=null-co
> image: json:{"driver": "raw", "size": "512", "file": {"driver": "null-co"}}
> 
> @size is supposed to be an integer, according to the schema, so the
> correct result would be (which is what you get after this series):
> 
> $ ./qemu-img info --image-opts driver=raw,size=512,file.driver=null-co
> image: json:{"driver": "raw", "size": 512, "file": {"driver": "null-co"}}
> 
> 
> This is achieved by patch 11, which makes bdrv_refresh_filename() run
> the options through the flat-confused input visitor, and then through
> the output visitor to get all to the correct type.  If anything fails,
> the result is as before (hence the “Try” in the title).
> 
> There are cases where this cannot work.  Those are the ones where -drive
> accepts something that is not allowed by the QAPI schema.  One of these
> cases is rbd’s =keyvalue-pairs, which is just broken altogether, so
> let’s simply ignore that.  (I don’t think anybody’s going to complain
> that the json:{} filename they get is not correctly typed after they’ve
> used that option.)
> 
> The other case (I know of) is qcow(2)’s encryption.  In the QAPI schema,
> encrypt.format is not optional because it is the discriminator for which
> kind of options to use.  However, for -drive, it is optional because the
> qcow2 driver can infer the encryption format from the image header.
> 
> The solution that’s proposed by this series is to make flat union
> discriminators optional and provide a default.  This is accomplished by
> generally allowing default values to be provided for QAPI struct
> members.
> 
> Both AES and LUKS encryption allow only a key-secret option, so we can
> add a new pseudo-format “auto” that accepts exactly that option and
> makes the qcow2 driver read the real format from the image header.  This
> pseudo-format is made the default for encrypt.format, and thus you can
> then specify encrypt.key-secret without having to specify
> encrypt.format (while still adhering to the QAPI schema).
> 
> 
> So, in this series:
> - The QAPI code generator is modified to allow default values for
>   optional struct members.  This in turn allows flat union
>   discriminators be optional, too, but only if a default value is
>   provided.
>   - Accordingly, documentation, tests, and introspection are adjusted.
> 
> - This is used to make qcow’s and qcow2’s encrypt.format parameter
>   optional.  It now defaults to “from-image” which is a new
>   pseudo-format that allows a key-secret to be given, and otherwise
>   leaves it to the format driver to determine the encryption format.
> 
> - json:{} filenames are attempted to be typed correctly when they are
>   generated, by running bs->full_open_options through a healthy mix of
>   qdict_flatten(), the flat-confused input visitor for BlockdevOptions,
>   and the output visitor.
>   This may not always work but I hope it usually will.  Fingers crossed.
>   (At least it won’t make things worse.)
> 
> - Tests, tests, tests.
> 
> 
> (Yes, I know that “In this series tests, tests, tests.” is not a
>  sentence.)
> 
> 
> v4:
> - Drop the default-variant stuff and replace it by a more general
>   concept of allowing default values for all QAPI struct members
> 
> 
> git backport-diff against v3:
> 
> Key:
> [----] : patches are identical
> [####] : number of functional differences between upstream/downstream patch
> [down] : patch is downstream-only
> The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
> 
> 001/14:[down] 'qapi: Parse numeric values'
> 002/14:[down] 'qapi: Move to_c_string() to common.py'
> 003/14:[down] 'qapi: Introduce default values for struct members'
> 004/14:[down] 'qapi: Allow optional discriminators'
> 005/14:[down] 'qapi: Document default values for struct members'
> 006/14:[down] 'test-qapi: Print struct members' default values'
> 007/14:[down] 'tests: Test QAPI default values for struct members'
> 008/14:[0044] [FC] 'tests: Add QAPI optional discriminator tests'
> 009/14:[0009] [FC] 'qapi: Formalize qcow2 encryption probing'
> 010/14:[0005] [FC] 'qapi: Formalize qcow encryption probing'
> 011/14:[0014] [FC] 'block: Try to create well typed json:{} filenames'
> 012/14:[----] [--] 'iotests: Test internal option typing'
> 013/14:[----] [--] 'iotests: qcow2's encrypt.format is now optional'
> 014/14:[down] 'block: Make use of QAPI defaults'
> 
> 
> Max Reitz (14):
>   qapi: Parse numeric values
>   qapi: Move to_c_string() to common.py
>   qapi: Introduce default values for struct members
>   qapi: Allow optional discriminators
>   qapi: Document default values for struct members
>   test-qapi: Print struct members' default values
>   tests: Test QAPI default values for struct members
>   tests: Add QAPI optional discriminator tests
>   qapi: Formalize qcow2 encryption probing
>   qapi: Formalize qcow encryption probing
>   block: Try to create well typed json:{} filenames
>   iotests: Test internal option typing
>   iotests: qcow2's encrypt.format is now optional
>   block: Make use of QAPI defaults
> 
>  docs/devel/qapi-code-gen.txt                  |  81 +++++-
>  tests/Makefile.include                        |  17 +-
>  qapi/block-core.json                          | 180 +++++++++-----
>  qapi/introspect.json                          |   9 +-
>  tests/qapi-schema/bad-type-int.json           |   1 -
>  tests/qapi-schema/enum-int-member.json        |   1 -
>  ...l-discriminator-invalid-specification.json |  11 +
>  ...on-optional-discriminator-no-default.json} |   5 +-
>  tests/qapi-schema/qapi-schema-test.json       |  38 +++
>  .../struct-member-alternate-default.json      |  10 +
>  ...struct-member-bool-wrong-default-type.json |   3 +
>  .../struct-member-enum-invalid-default.json   |   4 +
>  ...struct-member-enum-wrong-default-type.json |   4 +
>  .../struct-member-float-invalid-default.json  |   4 +
>  ...truct-member-float-wrong-default-type.json |   3 +
>  .../struct-member-int-wrong-default-type.json |   3 +
>  .../struct-member-int8-erange-default.json    |   3 +
>  .../struct-member-list-nonempty-default.json  |   4 +
>  .../struct-member-non-optional-default.json   |   3 +
>  .../struct-member-null-default.json           |   6 +
>  .../struct-member-str-wrong-default-type.json |   3 +
>  .../struct-member-uint8-erange-default.json   |   3 +
>  .../struct-member-uint8-negative-default.json |   3 +
>  block.c                                       |  68 ++++-
>  block/file-posix.c                            |   9 -
>  block/file-win32.c                            |   8 +-
>  block/parallels.c                             |   6 +-
>  block/qcow2.c                                 |  39 +--
>  block/qed.c                                   |   3 -
>  block/sheepdog.c                              |   3 -
>  block/vdi.c                                   |   3 -
>  block/vhdx.c                                  |  28 +--
>  block/vpc.c                                   |   3 -
>  blockdev.c                                    | 182 +++-----------
>  monitor/hmp-cmds.c                            |  27 +-
>  monitor/qmp-cmds.c                            |   3 +-
>  scripts/qapi/commands.py                      |   2 +-
>  scripts/qapi/common.py                        | 232 ++++++++++++++++--
>  scripts/qapi/doc.py                           |  20 +-
>  scripts/qapi/introspect.py                    |   8 +-
>  scripts/qapi/types.py                         |   2 +-
>  scripts/qapi/visit.py                         |  38 ++-
>  tests/qapi-schema/bad-type-int.err            |   2 +-
>  tests/qapi-schema/enum-int-member.err         |   2 +-
>  ...al-discriminator-invalid-specification.err |   1 +
>  ...-discriminator-invalid-specification.exit} |   0
>  ...l-discriminator-invalid-specification.out} |   0
>  ...nion-optional-discriminator-no-default.err |   1 +
>  ...ion-optional-discriminator-no-default.exit |   1 +
>  ...nion-optional-discriminator-no-default.out |   0
>  .../flat-union-optional-discriminator.err     |   1 -
>  tests/qapi-schema/leading-comma-list.err      |   2 +-
>  tests/qapi-schema/qapi-schema-test.out        |  33 +++
>  .../struct-member-alternate-default.err       |   1 +
>  .../struct-member-alternate-default.exit      |   1 +
>  .../struct-member-alternate-default.out       |   0
>  .../struct-member-bool-wrong-default-type.err |   1 +
>  ...struct-member-bool-wrong-default-type.exit |   1 +
>  .../struct-member-bool-wrong-default-type.out |   0
>  .../struct-member-enum-invalid-default.err    |   1 +
>  .../struct-member-enum-invalid-default.exit   |   1 +
>  .../struct-member-enum-invalid-default.out    |   0
>  .../struct-member-enum-wrong-default-type.err |   1 +
>  ...struct-member-enum-wrong-default-type.exit |   1 +
>  .../struct-member-enum-wrong-default-type.out |   0
>  .../struct-member-float-invalid-default.err   |   1 +
>  .../struct-member-float-invalid-default.exit  |   1 +
>  .../struct-member-float-invalid-default.out   |   0
>  ...struct-member-float-wrong-default-type.err |   1 +
>  ...truct-member-float-wrong-default-type.exit |   1 +
>  ...struct-member-float-wrong-default-type.out |   0
>  .../struct-member-int-wrong-default-type.err  |   1 +
>  .../struct-member-int-wrong-default-type.exit |   1 +
>  .../struct-member-int-wrong-default-type.out  |   0
>  .../struct-member-int8-erange-default.err     |   1 +
>  .../struct-member-int8-erange-default.exit    |   1 +
>  .../struct-member-int8-erange-default.out     |   0
>  .../struct-member-list-nonempty-default.err   |   1 +
>  .../struct-member-list-nonempty-default.exit  |   1 +
>  .../struct-member-list-nonempty-default.out   |   0
>  .../struct-member-non-optional-default.err    |   1 +
>  .../struct-member-non-optional-default.exit   |   1 +
>  .../struct-member-non-optional-default.out    |   0
>  .../struct-member-null-default.err            |   1 +
>  .../struct-member-null-default.exit           |   1 +
>  .../struct-member-null-default.out            |   0
>  .../struct-member-str-wrong-default-type.err  |   1 +
>  .../struct-member-str-wrong-default-type.exit |   1 +
>  .../struct-member-str-wrong-default-type.out  |   0
>  .../struct-member-uint8-erange-default.err    |   1 +
>  .../struct-member-uint8-erange-default.exit   |   1 +
>  .../struct-member-uint8-erange-default.out    |   0
>  .../struct-member-uint8-negative-default.err  |   1 +
>  .../struct-member-uint8-negative-default.exit |   1 +
>  .../struct-member-uint8-negative-default.out  |   0
>  tests/qapi-schema/test-qapi.py                |   8 +-
>  tests/qemu-iotests/059.out                    |   2 +-
>  tests/qemu-iotests/087                        |  65 +++--
>  tests/qemu-iotests/087.out                    |  26 +-
>  tests/qemu-iotests/089                        |  25 ++
>  tests/qemu-iotests/089.out                    |   9 +
>  tests/qemu-iotests/099.out                    |   4 +-
>  tests/qemu-iotests/110.out                    |   2 +-
>  tests/qemu-iotests/198.out                    |   4 +-
>  104 files changed, 915 insertions(+), 384 deletions(-)
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.json
>  rename tests/qapi-schema/{flat-union-optional-discriminator.json => flat-union-optional-discriminator-no-default.json} (68%)
>  create mode 100644 tests/qapi-schema/struct-member-alternate-default.json
>  create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.json
>  create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.json
>  create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.json
>  create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.json
>  create mode 100644 tests/qapi-schema/struct-member-non-optional-default.json
>  create mode 100644 tests/qapi-schema/struct-member-null-default.json
>  create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.json
>  create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.json
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.err
>  rename tests/qapi-schema/{flat-union-optional-discriminator.exit => flat-union-optional-discriminator-invalid-specification.exit} (100%)
>  rename tests/qapi-schema/{flat-union-optional-discriminator.out => flat-union-optional-discriminator-invalid-specification.out} (100%)
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out
>  delete mode 100644 tests/qapi-schema/flat-union-optional-discriminator.err
>  create mode 100644 tests/qapi-schema/struct-member-alternate-default.err
>  create mode 100644 tests/qapi-schema/struct-member-alternate-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-alternate-default.out
>  create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.err
>  create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.out
>  create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.err
>  create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.out
>  create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.err
>  create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.out
>  create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.err
>  create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.out
>  create mode 100644 tests/qapi-schema/struct-member-non-optional-default.err
>  create mode 100644 tests/qapi-schema/struct-member-non-optional-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-non-optional-default.out
>  create mode 100644 tests/qapi-schema/struct-member-null-default.err
>  create mode 100644 tests/qapi-schema/struct-member-null-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-null-default.out
>  create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.err
>  create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.out
>  create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.err
>  create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.out
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
                   ` (16 preceding siblings ...)
  2019-08-19 16:49 ` Max Reitz
@ 2019-09-13 11:49 ` Max Reitz
  2019-11-06 13:01   ` Max Reitz
  17 siblings, 1 reply; 41+ messages in thread
From: Max Reitz @ 2019-09-13 11:49 UTC (permalink / raw)
  To: qemu-block; +Cc: Michael Roth, Kevin Wolf, qemu-devel, Markus Armbruster


[-- Attachment #1.1: Type: text/plain, Size: 19482 bytes --]

Another gentle ping.

Max


On 24.06.19 19:39, Max Reitz wrote:
> Hi,
> 
> There are two explanations of this cover letter, a relative one (to v3)
> and an absolute one.
> 
> 
> *** Important note ***
> 
> The final patch in this series is an example that converts most of
> block-core.json to use default values where possible.  We may decide to
> take it or not.  It isn’t important for the main purpose of this series,
> so I’d be very much fine with chopping it off.
> 
> (It does have a nice diff stat, though.)
> 
> *** Important note end ***
> 
> 
> Relative explanation:
> 
> The actual functional goal of this series is to allow all blockdev
> options that can be represented with -drive to have an equivalent with
> -blockdev (safe for rbd’s =keyvalue-pairs).
> 
> To this end, qcow(2)’s encryption needs an “auto” format which can
> automatically deduce the format from the image header.  To make things
> nicer, I decided (already in v1) to make this format optional so users
> could just specify encrypt.secret and let the format driver figure out
> the rest.
> 
> Until v3, this was implemented by letting the discriminator of flat
> unions be optional, as long as a default-value is provided.  Markus
> (rightfully) complained that this is very specific and would be covered
> by just having default values for QAPI struct members in general.
> So now this version implements this.  This is a bit more complicated
> than just implementing a default-variant, mainly because the latter only
> needs to accept enum values, whereas a generally usable “default” should
> accept values of all QAPI types (to the extent what is reasonable).
> 
> So what was (until v3)
> 
>   { 'union': 'Foo',
>     'base': { '*discr': 'SomeEnum' },
>     'discriminator': 'discr',
>     'default-variant': 'value1',
>     'data': { 'value1': 'Bar', 'value2': 'Baz' } }
> 
> becomes
> 
>   { 'union': 'Foo',
>     'base': { '*discr': { 'type': 'SomeEnum', 'default': 'value1' } },
>     'discriminator': 'discr',
>     'data': { 'value1': 'Bar', 'value2': 'Baz' } }
> 
> 
> 
> Absolute explanation:
> 
> When qemu reports json:{} filename, it just uses whatever type you gave
> an option in.  With -drive, all options are strings and they do not have
> to pass the test of the typing firewall of the QAPI schema, so you just
> get strings thrown back at you even if that does not match the schema.
> (Also, if you use json:{} yourself, you’re free to give the options as
> strings as well.)
> 
> Example:
> 
> $ ./qemu-img info --image-opts driver=raw,size=512,file.driver=null-co
> image: json:{"driver": "raw", "size": "512", "file": {"driver": "null-co"}}
> 
> @size is supposed to be an integer, according to the schema, so the
> correct result would be (which is what you get after this series):
> 
> $ ./qemu-img info --image-opts driver=raw,size=512,file.driver=null-co
> image: json:{"driver": "raw", "size": 512, "file": {"driver": "null-co"}}
> 
> 
> This is achieved by patch 11, which makes bdrv_refresh_filename() run
> the options through the flat-confused input visitor, and then through
> the output visitor to get all to the correct type.  If anything fails,
> the result is as before (hence the “Try” in the title).
> 
> There are cases where this cannot work.  Those are the ones where -drive
> accepts something that is not allowed by the QAPI schema.  One of these
> cases is rbd’s =keyvalue-pairs, which is just broken altogether, so
> let’s simply ignore that.  (I don’t think anybody’s going to complain
> that the json:{} filename they get is not correctly typed after they’ve
> used that option.)
> 
> The other case (I know of) is qcow(2)’s encryption.  In the QAPI schema,
> encrypt.format is not optional because it is the discriminator for which
> kind of options to use.  However, for -drive, it is optional because the
> qcow2 driver can infer the encryption format from the image header.
> 
> The solution that’s proposed by this series is to make flat union
> discriminators optional and provide a default.  This is accomplished by
> generally allowing default values to be provided for QAPI struct
> members.
> 
> Both AES and LUKS encryption allow only a key-secret option, so we can
> add a new pseudo-format “auto” that accepts exactly that option and
> makes the qcow2 driver read the real format from the image header.  This
> pseudo-format is made the default for encrypt.format, and thus you can
> then specify encrypt.key-secret without having to specify
> encrypt.format (while still adhering to the QAPI schema).
> 
> 
> So, in this series:
> - The QAPI code generator is modified to allow default values for
>   optional struct members.  This in turn allows flat union
>   discriminators be optional, too, but only if a default value is
>   provided.
>   - Accordingly, documentation, tests, and introspection are adjusted.
> 
> - This is used to make qcow’s and qcow2’s encrypt.format parameter
>   optional.  It now defaults to “from-image” which is a new
>   pseudo-format that allows a key-secret to be given, and otherwise
>   leaves it to the format driver to determine the encryption format.
> 
> - json:{} filenames are attempted to be typed correctly when they are
>   generated, by running bs->full_open_options through a healthy mix of
>   qdict_flatten(), the flat-confused input visitor for BlockdevOptions,
>   and the output visitor.
>   This may not always work but I hope it usually will.  Fingers crossed.
>   (At least it won’t make things worse.)
> 
> - Tests, tests, tests.
> 
> 
> (Yes, I know that “In this series tests, tests, tests.” is not a
>  sentence.)
> 
> 
> v4:
> - Drop the default-variant stuff and replace it by a more general
>   concept of allowing default values for all QAPI struct members
> 
> 
> git backport-diff against v3:
> 
> Key:
> [----] : patches are identical
> [####] : number of functional differences between upstream/downstream patch
> [down] : patch is downstream-only
> The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
> 
> 001/14:[down] 'qapi: Parse numeric values'
> 002/14:[down] 'qapi: Move to_c_string() to common.py'
> 003/14:[down] 'qapi: Introduce default values for struct members'
> 004/14:[down] 'qapi: Allow optional discriminators'
> 005/14:[down] 'qapi: Document default values for struct members'
> 006/14:[down] 'test-qapi: Print struct members' default values'
> 007/14:[down] 'tests: Test QAPI default values for struct members'
> 008/14:[0044] [FC] 'tests: Add QAPI optional discriminator tests'
> 009/14:[0009] [FC] 'qapi: Formalize qcow2 encryption probing'
> 010/14:[0005] [FC] 'qapi: Formalize qcow encryption probing'
> 011/14:[0014] [FC] 'block: Try to create well typed json:{} filenames'
> 012/14:[----] [--] 'iotests: Test internal option typing'
> 013/14:[----] [--] 'iotests: qcow2's encrypt.format is now optional'
> 014/14:[down] 'block: Make use of QAPI defaults'
> 
> 
> Max Reitz (14):
>   qapi: Parse numeric values
>   qapi: Move to_c_string() to common.py
>   qapi: Introduce default values for struct members
>   qapi: Allow optional discriminators
>   qapi: Document default values for struct members
>   test-qapi: Print struct members' default values
>   tests: Test QAPI default values for struct members
>   tests: Add QAPI optional discriminator tests
>   qapi: Formalize qcow2 encryption probing
>   qapi: Formalize qcow encryption probing
>   block: Try to create well typed json:{} filenames
>   iotests: Test internal option typing
>   iotests: qcow2's encrypt.format is now optional
>   block: Make use of QAPI defaults
> 
>  docs/devel/qapi-code-gen.txt                  |  81 +++++-
>  tests/Makefile.include                        |  17 +-
>  qapi/block-core.json                          | 180 +++++++++-----
>  qapi/introspect.json                          |   9 +-
>  tests/qapi-schema/bad-type-int.json           |   1 -
>  tests/qapi-schema/enum-int-member.json        |   1 -
>  ...l-discriminator-invalid-specification.json |  11 +
>  ...on-optional-discriminator-no-default.json} |   5 +-
>  tests/qapi-schema/qapi-schema-test.json       |  38 +++
>  .../struct-member-alternate-default.json      |  10 +
>  ...struct-member-bool-wrong-default-type.json |   3 +
>  .../struct-member-enum-invalid-default.json   |   4 +
>  ...struct-member-enum-wrong-default-type.json |   4 +
>  .../struct-member-float-invalid-default.json  |   4 +
>  ...truct-member-float-wrong-default-type.json |   3 +
>  .../struct-member-int-wrong-default-type.json |   3 +
>  .../struct-member-int8-erange-default.json    |   3 +
>  .../struct-member-list-nonempty-default.json  |   4 +
>  .../struct-member-non-optional-default.json   |   3 +
>  .../struct-member-null-default.json           |   6 +
>  .../struct-member-str-wrong-default-type.json |   3 +
>  .../struct-member-uint8-erange-default.json   |   3 +
>  .../struct-member-uint8-negative-default.json |   3 +
>  block.c                                       |  68 ++++-
>  block/file-posix.c                            |   9 -
>  block/file-win32.c                            |   8 +-
>  block/parallels.c                             |   6 +-
>  block/qcow2.c                                 |  39 +--
>  block/qed.c                                   |   3 -
>  block/sheepdog.c                              |   3 -
>  block/vdi.c                                   |   3 -
>  block/vhdx.c                                  |  28 +--
>  block/vpc.c                                   |   3 -
>  blockdev.c                                    | 182 +++-----------
>  monitor/hmp-cmds.c                            |  27 +-
>  monitor/qmp-cmds.c                            |   3 +-
>  scripts/qapi/commands.py                      |   2 +-
>  scripts/qapi/common.py                        | 232 ++++++++++++++++--
>  scripts/qapi/doc.py                           |  20 +-
>  scripts/qapi/introspect.py                    |   8 +-
>  scripts/qapi/types.py                         |   2 +-
>  scripts/qapi/visit.py                         |  38 ++-
>  tests/qapi-schema/bad-type-int.err            |   2 +-
>  tests/qapi-schema/enum-int-member.err         |   2 +-
>  ...al-discriminator-invalid-specification.err |   1 +
>  ...-discriminator-invalid-specification.exit} |   0
>  ...l-discriminator-invalid-specification.out} |   0
>  ...nion-optional-discriminator-no-default.err |   1 +
>  ...ion-optional-discriminator-no-default.exit |   1 +
>  ...nion-optional-discriminator-no-default.out |   0
>  .../flat-union-optional-discriminator.err     |   1 -
>  tests/qapi-schema/leading-comma-list.err      |   2 +-
>  tests/qapi-schema/qapi-schema-test.out        |  33 +++
>  .../struct-member-alternate-default.err       |   1 +
>  .../struct-member-alternate-default.exit      |   1 +
>  .../struct-member-alternate-default.out       |   0
>  .../struct-member-bool-wrong-default-type.err |   1 +
>  ...struct-member-bool-wrong-default-type.exit |   1 +
>  .../struct-member-bool-wrong-default-type.out |   0
>  .../struct-member-enum-invalid-default.err    |   1 +
>  .../struct-member-enum-invalid-default.exit   |   1 +
>  .../struct-member-enum-invalid-default.out    |   0
>  .../struct-member-enum-wrong-default-type.err |   1 +
>  ...struct-member-enum-wrong-default-type.exit |   1 +
>  .../struct-member-enum-wrong-default-type.out |   0
>  .../struct-member-float-invalid-default.err   |   1 +
>  .../struct-member-float-invalid-default.exit  |   1 +
>  .../struct-member-float-invalid-default.out   |   0
>  ...struct-member-float-wrong-default-type.err |   1 +
>  ...truct-member-float-wrong-default-type.exit |   1 +
>  ...struct-member-float-wrong-default-type.out |   0
>  .../struct-member-int-wrong-default-type.err  |   1 +
>  .../struct-member-int-wrong-default-type.exit |   1 +
>  .../struct-member-int-wrong-default-type.out  |   0
>  .../struct-member-int8-erange-default.err     |   1 +
>  .../struct-member-int8-erange-default.exit    |   1 +
>  .../struct-member-int8-erange-default.out     |   0
>  .../struct-member-list-nonempty-default.err   |   1 +
>  .../struct-member-list-nonempty-default.exit  |   1 +
>  .../struct-member-list-nonempty-default.out   |   0
>  .../struct-member-non-optional-default.err    |   1 +
>  .../struct-member-non-optional-default.exit   |   1 +
>  .../struct-member-non-optional-default.out    |   0
>  .../struct-member-null-default.err            |   1 +
>  .../struct-member-null-default.exit           |   1 +
>  .../struct-member-null-default.out            |   0
>  .../struct-member-str-wrong-default-type.err  |   1 +
>  .../struct-member-str-wrong-default-type.exit |   1 +
>  .../struct-member-str-wrong-default-type.out  |   0
>  .../struct-member-uint8-erange-default.err    |   1 +
>  .../struct-member-uint8-erange-default.exit   |   1 +
>  .../struct-member-uint8-erange-default.out    |   0
>  .../struct-member-uint8-negative-default.err  |   1 +
>  .../struct-member-uint8-negative-default.exit |   1 +
>  .../struct-member-uint8-negative-default.out  |   0
>  tests/qapi-schema/test-qapi.py                |   8 +-
>  tests/qemu-iotests/059.out                    |   2 +-
>  tests/qemu-iotests/087                        |  65 +++--
>  tests/qemu-iotests/087.out                    |  26 +-
>  tests/qemu-iotests/089                        |  25 ++
>  tests/qemu-iotests/089.out                    |   9 +
>  tests/qemu-iotests/099.out                    |   4 +-
>  tests/qemu-iotests/110.out                    |   2 +-
>  tests/qemu-iotests/198.out                    |   4 +-
>  104 files changed, 915 insertions(+), 384 deletions(-)
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.json
>  rename tests/qapi-schema/{flat-union-optional-discriminator.json => flat-union-optional-discriminator-no-default.json} (68%)
>  create mode 100644 tests/qapi-schema/struct-member-alternate-default.json
>  create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.json
>  create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.json
>  create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.json
>  create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.json
>  create mode 100644 tests/qapi-schema/struct-member-non-optional-default.json
>  create mode 100644 tests/qapi-schema/struct-member-null-default.json
>  create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.json
>  create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.json
>  create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.json
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-specification.err
>  rename tests/qapi-schema/{flat-union-optional-discriminator.exit => flat-union-optional-discriminator-invalid-specification.exit} (100%)
>  rename tests/qapi-schema/{flat-union-optional-discriminator.out => flat-union-optional-discriminator-invalid-specification.out} (100%)
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit
>  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out
>  delete mode 100644 tests/qapi-schema/flat-union-optional-discriminator.err
>  create mode 100644 tests/qapi-schema/struct-member-alternate-default.err
>  create mode 100644 tests/qapi-schema/struct-member-alternate-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-alternate-default.out
>  create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-bool-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.err
>  create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-enum-invalid-default.out
>  create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-enum-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.err
>  create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-float-invalid-default.out
>  create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-float-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-int-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.err
>  create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-int8-erange-default.out
>  create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.err
>  create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-list-nonempty-default.out
>  create mode 100644 tests/qapi-schema/struct-member-non-optional-default.err
>  create mode 100644 tests/qapi-schema/struct-member-non-optional-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-non-optional-default.out
>  create mode 100644 tests/qapi-schema/struct-member-null-default.err
>  create mode 100644 tests/qapi-schema/struct-member-null-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-null-default.out
>  create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.err
>  create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.exit
>  create mode 100644 tests/qapi-schema/struct-member-str-wrong-default-type.out
>  create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.err
>  create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-uint8-erange-default.out
>  create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.err
>  create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.exit
>  create mode 100644 tests/qapi-schema/struct-member-uint8-negative-default.out
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-09-13 11:49 ` Max Reitz
@ 2019-11-06 13:01   ` Max Reitz
  2019-11-14  8:54     ` Markus Armbruster
  0 siblings, 1 reply; 41+ messages in thread
From: Max Reitz @ 2019-11-06 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: Michael Roth, Kevin Wolf, qemu-devel, Markus Armbruster


[-- Attachment #1.1: Type: text/plain, Size: 86 bytes --]

On 13.09.19 13:49, Max Reitz wrote:
> Another gentle ping.

And another.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-11-06 13:01   ` Max Reitz
@ 2019-11-14  8:54     ` Markus Armbruster
  2019-11-21 15:17       ` Markus Armbruster
  0 siblings, 1 reply; 41+ messages in thread
From: Markus Armbruster @ 2019-11-14  8:54 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, Michael Roth, qemu-block, qemu-devel

Max Reitz <mreitz@redhat.com> writes:

> On 13.09.19 13:49, Max Reitz wrote:
>> Another gentle ping.
>
> And another.

Conflicts with the refactoring merged in commit 69717d0f890.  Please
accept my apologies for the inconvenience caused by the excessive delay.

I'll try to review anyway.



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values Max Reitz
@ 2019-11-14  9:15   ` Markus Armbruster
  2019-11-14  9:50     ` Max Reitz
  0 siblings, 1 reply; 41+ messages in thread
From: Markus Armbruster @ 2019-11-14  9:15 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth

Max Reitz <mreitz@redhat.com> writes:

> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qapi-schema/bad-type-int.json      |  1 -
>  tests/qapi-schema/enum-int-member.json   |  1 -
>  scripts/qapi/common.py                   | 25 ++++++++++++++++++++----
>  scripts/qapi/introspect.py               |  2 ++
>  tests/qapi-schema/bad-type-int.err       |  2 +-
>  tests/qapi-schema/enum-int-member.err    |  2 +-
>  tests/qapi-schema/leading-comma-list.err |  2 +-
>  7 files changed, 26 insertions(+), 9 deletions(-)
>
> diff --git a/tests/qapi-schema/bad-type-int.json b/tests/qapi-schema/bad-type-int.json
> index 56fc6f8126..81355eb196 100644
> --- a/tests/qapi-schema/bad-type-int.json
> +++ b/tests/qapi-schema/bad-type-int.json
> @@ -1,3 +1,2 @@
>  # we reject an expression with a metatype that is not a string
> -# FIXME: once the parser understands integer inputs, improve the error message
>  { 'struct': 1, 'data': { } }
> diff --git a/tests/qapi-schema/enum-int-member.json b/tests/qapi-schema/enum-int-member.json
> index 6c9c32e149..6958440c6d 100644
> --- a/tests/qapi-schema/enum-int-member.json
> +++ b/tests/qapi-schema/enum-int-member.json
> @@ -1,3 +1,2 @@
>  # we reject any enum member that is not a string
> -# FIXME: once the parser understands integer inputs, improve the error message
>  { 'enum': 'MyEnum', 'data': [ 1 ] }
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index d61bfdc526..3396ea4a09 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -498,6 +498,8 @@ class QAPISchemaParser(object):
>              raise QAPISemError(info, "Unknown pragma '%s'" % name)
>  
>      def accept(self, skip_comment=True):
> +        num_match = re.compile(r'([-+]?inf|nan|[-+0-9.][0-9a-f.ex]*)')
> +
>          while True:
>              self.tok = self.src[self.cursor]
>              self.pos = self.cursor

This is yet another extension over plain JSON.  RFC 8259:

      number = [ minus ] int [ frac ] [ exp ]
      decimal-point = %x2E       ; .
      digit1-9 = %x31-39         ; 1-9
      e = %x65 / %x45            ; e E
      exp = e [ minus / plus ] 1*DIGIT
      frac = decimal-point 1*DIGIT
      int = zero / ( digit1-9 *DIGIT )
      minus = %x2D               ; -
      plus = %x2B                ; +
      zero = %x30                ; 0

Extensions are acceptable when we have an actual use for it, and we
document them properly.

Isn't the parenthesis in your regular expression redundant?

What use do you have in mind for 'inf' and 'nan'?

Why accept leading '+' as in '+123'?

Why accept empty integral part as in '.123'?

Why accept '.xe.'?  Kidding you, that must be a bug in your regexp.
Please decide what number syntax you'd like to accept, then specify it
in docs/devel/qapi-code-gen.txt, so we can first discuss the
specification, and then check the regexp implements it.

docs/devel/qapi-code-gen.txt update goes here:

    === Schema syntax ===

    Syntax is loosely based on JSON (http://www.ietf.org/rfc/rfc8259.txt).
    Differences:

    * Comments: start with a hash character (#) that is not part of a
      string, and extend to the end of the line.

    * Strings are enclosed in 'single quotes', not "double quotes".

    * Strings are restricted to printable ASCII, and escape sequences to
      just '\\'.

--> * Numbers and null are not supported.

Hrmm, commit 9d55380b5a "qapi: Remove null from schema language" left
two instances in error messages behind.  I'll fix them.

> @@ -584,7 +586,22 @@ class QAPISchemaParser(object):
>                      return
>                  self.line += 1
>                  self.line_pos = self.cursor
> -            elif not self.tok.isspace():
> +            elif self.tok.isspace():
> +                pass
> +            elif num_match.match(self.src[self.pos:]):
> +                match = num_match.match(self.src[self.pos:]).group(0)

Sadly, the walrus operator is Python 3.8.

> +                try:
> +                    self.val = int(match, 0)
> +                except ValueError:
> +                    try:
> +                        self.val = float(match)
> +                    except ValueError:
> +                        raise QAPIParseError(self,
> +                                '"%s" is not a valid integer or float' % match)
> +
> +                self.cursor += len(match) - 1
> +                return
> +            else:
>                  raise QAPIParseError(self, 'Stray "%s"' % self.tok)

Any particular reason for putting the number case last?

>  
>      def get_members(self):
> @@ -617,9 +634,9 @@ class QAPISchemaParser(object):
>          if self.tok == ']':
>              self.accept()
>              return expr
> -        if self.tok not in "{['tfn":
> +        if self.tok not in "{['tfn-+0123456789.i":

This is getting a bit ugly.  Let's not worry about it now.

>              raise QAPIParseError(self, 'Expected "{", "[", "]", string, '
> -                                 'boolean or "null"')
> +                                 'boolean, number or "null"')
>          while True:
>              expr.append(self.get_expr(True))
>              if self.tok == ']':
> @@ -638,7 +655,7 @@ class QAPISchemaParser(object):
>          elif self.tok == '[':
>              self.accept()
>              expr = self.get_values()
> -        elif self.tok in "'tfn":
> +        elif self.tok in "'tfn-+0123456789.i":
>              expr = self.val
>              self.accept()
>          else:
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index f62cf0a2e1..6a61dd831f 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -57,6 +57,8 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
>          ret += indent(level) + '}))'
>      elif isinstance(obj, bool):
>          ret += 'QLIT_QBOOL(%s)' % ('true' if obj else 'false')
> +    elif isinstance(obj, int) and obj >= -(2 ** 63) and obj < 2 ** 63:
> +        ret += 'QLIT_QNUM(%i)' % obj

Please explain the range check.

>      else:
>          assert False                # not implemented
>      if level > 0:
> diff --git a/tests/qapi-schema/bad-type-int.err b/tests/qapi-schema/bad-type-int.err
> index da89895404..e22fb4f655 100644
> --- a/tests/qapi-schema/bad-type-int.err
> +++ b/tests/qapi-schema/bad-type-int.err
> @@ -1 +1 @@
> -tests/qapi-schema/bad-type-int.json:3:13: Stray "1"
> +tests/qapi-schema/bad-type-int.json:2: 'struct' key must have a string value

Test needs a rename, assuming it's not redundant now.

> diff --git a/tests/qapi-schema/enum-int-member.err b/tests/qapi-schema/enum-int-member.err
> index 071c5213d8..112175f79d 100644
> --- a/tests/qapi-schema/enum-int-member.err
> +++ b/tests/qapi-schema/enum-int-member.err
> @@ -1 +1 @@
> -tests/qapi-schema/enum-int-member.json:3:31: Stray "1"
> +tests/qapi-schema/enum-int-member.json:2: Member of enum 'MyEnum' requires a string name

This one's name is still good.

> diff --git a/tests/qapi-schema/leading-comma-list.err b/tests/qapi-schema/leading-comma-list.err
> index f5c870bb9c..fa9c80aa57 100644
> --- a/tests/qapi-schema/leading-comma-list.err
> +++ b/tests/qapi-schema/leading-comma-list.err
> @@ -1 +1 @@
> -tests/qapi-schema/leading-comma-list.json:2:13: Expected "{", "[", "]", string, boolean or "null"
> +tests/qapi-schema/leading-comma-list.json:2:13: Expected "{", "[", "]", string, boolean, number or "null"



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 02/14] qapi: Move to_c_string() to common.py
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 02/14] qapi: Move to_c_string() to common.py Max Reitz
@ 2019-11-14  9:20   ` Markus Armbruster
  2019-11-14  9:58     ` Max Reitz
  0 siblings, 1 reply; 41+ messages in thread
From: Markus Armbruster @ 2019-11-14  9:20 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth

Max Reitz <mreitz@redhat.com> writes:

> This function will be useful for code generation once we allow default
> values, so move it to the other "C helper functions".  In the process,
> rewrite it so it supports all nonprintable and non-ASCII characters.
>
> Signed-off-by: Max Reitz <mreitz@redhat.com>

Please have a close look at commit 56a8caff92 "qapi: Restrict strings to
printable ASCII".  Do we still need the rewrite?

If yes: the commit message title promises code motion, but the patch is
anything but.  Adjust the title, please.



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values
  2019-11-14  9:15   ` Markus Armbruster
@ 2019-11-14  9:50     ` Max Reitz
  2019-11-14 12:01       ` Markus Armbruster
  0 siblings, 1 reply; 41+ messages in thread
From: Max Reitz @ 2019-11-14  9:50 UTC (permalink / raw)
  To: Markus Armbruster; +Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth


[-- Attachment #1.1: Type: text/plain, Size: 9284 bytes --]

On 14.11.19 10:15, Markus Armbruster wrote:
> Max Reitz <mreitz@redhat.com> writes:
> 
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  tests/qapi-schema/bad-type-int.json      |  1 -
>>  tests/qapi-schema/enum-int-member.json   |  1 -
>>  scripts/qapi/common.py                   | 25 ++++++++++++++++++++----
>>  scripts/qapi/introspect.py               |  2 ++
>>  tests/qapi-schema/bad-type-int.err       |  2 +-
>>  tests/qapi-schema/enum-int-member.err    |  2 +-
>>  tests/qapi-schema/leading-comma-list.err |  2 +-
>>  7 files changed, 26 insertions(+), 9 deletions(-)
>>
>> diff --git a/tests/qapi-schema/bad-type-int.json b/tests/qapi-schema/bad-type-int.json
>> index 56fc6f8126..81355eb196 100644
>> --- a/tests/qapi-schema/bad-type-int.json
>> +++ b/tests/qapi-schema/bad-type-int.json
>> @@ -1,3 +1,2 @@
>>  # we reject an expression with a metatype that is not a string
>> -# FIXME: once the parser understands integer inputs, improve the error message
>>  { 'struct': 1, 'data': { } }
>> diff --git a/tests/qapi-schema/enum-int-member.json b/tests/qapi-schema/enum-int-member.json
>> index 6c9c32e149..6958440c6d 100644
>> --- a/tests/qapi-schema/enum-int-member.json
>> +++ b/tests/qapi-schema/enum-int-member.json
>> @@ -1,3 +1,2 @@
>>  # we reject any enum member that is not a string
>> -# FIXME: once the parser understands integer inputs, improve the error message
>>  { 'enum': 'MyEnum', 'data': [ 1 ] }
>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>> index d61bfdc526..3396ea4a09 100644
>> --- a/scripts/qapi/common.py
>> +++ b/scripts/qapi/common.py
>> @@ -498,6 +498,8 @@ class QAPISchemaParser(object):
>>              raise QAPISemError(info, "Unknown pragma '%s'" % name)
>>  
>>      def accept(self, skip_comment=True):
>> +        num_match = re.compile(r'([-+]?inf|nan|[-+0-9.][0-9a-f.ex]*)')
>> +
>>          while True:
>>              self.tok = self.src[self.cursor]
>>              self.pos = self.cursor
> 
> This is yet another extension over plain JSON.  RFC 8259:
> 
>       number = [ minus ] int [ frac ] [ exp ]
>       decimal-point = %x2E       ; .
>       digit1-9 = %x31-39         ; 1-9
>       e = %x65 / %x45            ; e E
>       exp = e [ minus / plus ] 1*DIGIT
>       frac = decimal-point 1*DIGIT
>       int = zero / ( digit1-9 *DIGIT )
>       minus = %x2D               ; -
>       plus = %x2B                ; +
>       zero = %x30                ; 0
> 
> Extensions are acceptable when we have an actual use for it, and we
> document them properly.

Well, it isn’t really an extension, because this isn’t a JSON parser but
just something that accepts anything that looks like a number and then
lets Python try a conversion on it.

> Isn't the parenthesis in your regular expression redundant?

You’re right, but on second thought, maybe I should surround it by \<
and \>.

> What use do you have in mind for 'inf' and 'nan'?

I could imagine inf being a useful default value, actually.  nan,
probably not so much.

> Why accept leading '+' as in '+123'?
> 
> Why accept empty integral part as in '.123'?
> 
> Why accept '.xe.'?  Kidding you, that must be a bug in your regexp.

Well, kind of.

I wanted to accept anything that looks in any way like a number and then
let Python try to convert it.  That’s also the reason why the case comes
last.

For that reason, I decided to keep the regex as simple as possible,
because the attempted conversions would reject anything that isn’t (to
Python) a valid number later.

It was my impression that the QAPI schema isn’t really JSON anyway and
that our QAPI schema parser isn’t a JSON parser.  Under that assumption
it simply seemed useful to me to accept anything that could potentially
be a number to Python and convert it.

Now, honestly, I still don’t see the point of having a strict JSON
“parser” here, but if you insist.  Seems possible to do in a regex.

Though I do think it makes sense to support hex integers as an extension.

> Please decide what number syntax you'd like to accept, then specify it
> in docs/devel/qapi-code-gen.txt, so we can first discuss the
> specification, and then check the regexp implements it.
> 
> docs/devel/qapi-code-gen.txt update goes here:
> 
>     === Schema syntax ===
> 
>     Syntax is loosely based on JSON (http://www.ietf.org/rfc/rfc8259.txt).
>     Differences:
> 
>     * Comments: start with a hash character (#) that is not part of a
>       string, and extend to the end of the line.
> 
>     * Strings are enclosed in 'single quotes', not "double quotes".
> 
>     * Strings are restricted to printable ASCII, and escape sequences to
>       just '\\'.
> 
> --> * Numbers and null are not supported.

OK.

> Hrmm, commit 9d55380b5a "qapi: Remove null from schema language" left
> two instances in error messages behind.  I'll fix them.
> 
>> @@ -584,7 +586,22 @@ class QAPISchemaParser(object):
>>                      return
>>                  self.line += 1
>>                  self.line_pos = self.cursor
>> -            elif not self.tok.isspace():
>> +            elif self.tok.isspace():
>> +                pass
>> +            elif num_match.match(self.src[self.pos:]):
>> +                match = num_match.match(self.src[self.pos:]).group(0)
> 
> Sadly, the walrus operator is Python 3.8.
> 
>> +                try:
>> +                    self.val = int(match, 0)
>> +                except ValueError:
>> +                    try:
>> +                        self.val = float(match)
>> +                    except ValueError:
>> +                        raise QAPIParseError(self,
>> +                                '"%s" is not a valid integer or float' % match)
>> +
>> +                self.cursor += len(match) - 1
>> +                return
>> +            else:
>>                  raise QAPIParseError(self, 'Stray "%s"' % self.tok)
> 
> Any particular reason for putting the number case last?

Because the match is so broad.

>>  
>>      def get_members(self):
>> @@ -617,9 +634,9 @@ class QAPISchemaParser(object):
>>          if self.tok == ']':
>>              self.accept()
>>              return expr
>> -        if self.tok not in "{['tfn":
>> +        if self.tok not in "{['tfn-+0123456789.i":
> 
> This is getting a bit ugly.  Let's not worry about it now.
> 
>>              raise QAPIParseError(self, 'Expected "{", "[", "]", string, '
>> -                                 'boolean or "null"')
>> +                                 'boolean, number or "null"')
>>          while True:
>>              expr.append(self.get_expr(True))
>>              if self.tok == ']':
>> @@ -638,7 +655,7 @@ class QAPISchemaParser(object):
>>          elif self.tok == '[':
>>              self.accept()
>>              expr = self.get_values()
>> -        elif self.tok in "'tfn":
>> +        elif self.tok in "'tfn-+0123456789.i":
>>              expr = self.val
>>              self.accept()
>>          else:
>> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
>> index f62cf0a2e1..6a61dd831f 100644
>> --- a/scripts/qapi/introspect.py
>> +++ b/scripts/qapi/introspect.py
>> @@ -57,6 +57,8 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
>>          ret += indent(level) + '}))'
>>      elif isinstance(obj, bool):
>>          ret += 'QLIT_QBOOL(%s)' % ('true' if obj else 'false')
>> +    elif isinstance(obj, int) and obj >= -(2 ** 63) and obj < 2 ** 63:
>> +        ret += 'QLIT_QNUM(%i)' % obj
> 
> Please explain the range check.

Will do.

>>      else:
>>          assert False                # not implemented
>>      if level > 0:
>> diff --git a/tests/qapi-schema/bad-type-int.err b/tests/qapi-schema/bad-type-int.err
>> index da89895404..e22fb4f655 100644
>> --- a/tests/qapi-schema/bad-type-int.err
>> +++ b/tests/qapi-schema/bad-type-int.err
>> @@ -1 +1 @@
>> -tests/qapi-schema/bad-type-int.json:3:13: Stray "1"
>> +tests/qapi-schema/bad-type-int.json:2: 'struct' key must have a string value
> 
> Test needs a rename, assuming it's not redundant now.

I’m not adding a test here, it’s just the value has changed in
4d42815587063d.

Thanks for reviewing!

Max

>> diff --git a/tests/qapi-schema/enum-int-member.err b/tests/qapi-schema/enum-int-member.err
>> index 071c5213d8..112175f79d 100644
>> --- a/tests/qapi-schema/enum-int-member.err
>> +++ b/tests/qapi-schema/enum-int-member.err
>> @@ -1 +1 @@
>> -tests/qapi-schema/enum-int-member.json:3:31: Stray "1"
>> +tests/qapi-schema/enum-int-member.json:2: Member of enum 'MyEnum' requires a string name
> 
> This one's name is still good.
> 
>> diff --git a/tests/qapi-schema/leading-comma-list.err b/tests/qapi-schema/leading-comma-list.err
>> index f5c870bb9c..fa9c80aa57 100644
>> --- a/tests/qapi-schema/leading-comma-list.err
>> +++ b/tests/qapi-schema/leading-comma-list.err
>> @@ -1 +1 @@
>> -tests/qapi-schema/leading-comma-list.json:2:13: Expected "{", "[", "]", string, boolean or "null"
>> +tests/qapi-schema/leading-comma-list.json:2:13: Expected "{", "[", "]", string, boolean, number or "null"



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 02/14] qapi: Move to_c_string() to common.py
  2019-11-14  9:20   ` Markus Armbruster
@ 2019-11-14  9:58     ` Max Reitz
  0 siblings, 0 replies; 41+ messages in thread
From: Max Reitz @ 2019-11-14  9:58 UTC (permalink / raw)
  To: Markus Armbruster; +Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth


[-- Attachment #1.1: Type: text/plain, Size: 1045 bytes --]

On 14.11.19 10:20, Markus Armbruster wrote:
> Max Reitz <mreitz@redhat.com> writes:
> 
>> This function will be useful for code generation once we allow default
>> values, so move it to the other "C helper functions".  In the process,
>> rewrite it so it supports all nonprintable and non-ASCII characters.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
> 
> Please have a close look at commit 56a8caff92 "qapi: Restrict strings to
> printable ASCII".  Do we still need the rewrite?

If that’s all that has changed, I think we will still need at least some
bits, like the " or \ escaping.

Also, actually, it looks like 56a8caff92 didn’t change the fact that
control characters are verbatim parts of the string, i.e. \u000a will
still be a literal 0xa byte, and as such must be escaped anew in the C
string.

So without having tried, I think this is still very much necessary.

> If yes: the commit message title promises code motion, but the patch is
> anything but.  Adjust the title, please.

OK.

Max



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values
  2019-11-14  9:50     ` Max Reitz
@ 2019-11-14 12:01       ` Markus Armbruster
  0 siblings, 0 replies; 41+ messages in thread
From: Markus Armbruster @ 2019-11-14 12:01 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth

Max Reitz <mreitz@redhat.com> writes:

> On 14.11.19 10:15, Markus Armbruster wrote:
>> Max Reitz <mreitz@redhat.com> writes:
>> 
>>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>>> ---
>>>  tests/qapi-schema/bad-type-int.json      |  1 -
>>>  tests/qapi-schema/enum-int-member.json   |  1 -
>>>  scripts/qapi/common.py                   | 25 ++++++++++++++++++++----
>>>  scripts/qapi/introspect.py               |  2 ++
>>>  tests/qapi-schema/bad-type-int.err       |  2 +-
>>>  tests/qapi-schema/enum-int-member.err    |  2 +-
>>>  tests/qapi-schema/leading-comma-list.err |  2 +-
>>>  7 files changed, 26 insertions(+), 9 deletions(-)
>>>
>>> diff --git a/tests/qapi-schema/bad-type-int.json b/tests/qapi-schema/bad-type-int.json
>>> index 56fc6f8126..81355eb196 100644
>>> --- a/tests/qapi-schema/bad-type-int.json
>>> +++ b/tests/qapi-schema/bad-type-int.json
>>> @@ -1,3 +1,2 @@
>>>  # we reject an expression with a metatype that is not a string
>>> -# FIXME: once the parser understands integer inputs, improve the error message
>>>  { 'struct': 1, 'data': { } }
>>> diff --git a/tests/qapi-schema/enum-int-member.json b/tests/qapi-schema/enum-int-member.json
>>> index 6c9c32e149..6958440c6d 100644
>>> --- a/tests/qapi-schema/enum-int-member.json
>>> +++ b/tests/qapi-schema/enum-int-member.json
>>> @@ -1,3 +1,2 @@
>>>  # we reject any enum member that is not a string
>>> -# FIXME: once the parser understands integer inputs, improve the error message
>>>  { 'enum': 'MyEnum', 'data': [ 1 ] }
>>> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
>>> index d61bfdc526..3396ea4a09 100644
>>> --- a/scripts/qapi/common.py
>>> +++ b/scripts/qapi/common.py
>>> @@ -498,6 +498,8 @@ class QAPISchemaParser(object):
>>>              raise QAPISemError(info, "Unknown pragma '%s'" % name)
>>>  
>>>      def accept(self, skip_comment=True):
>>> +        num_match = re.compile(r'([-+]?inf|nan|[-+0-9.][0-9a-f.ex]*)')
>>> +
>>>          while True:
>>>              self.tok = self.src[self.cursor]
>>>              self.pos = self.cursor
>> 
>> This is yet another extension over plain JSON.  RFC 8259:
>> 
>>       number = [ minus ] int [ frac ] [ exp ]
>>       decimal-point = %x2E       ; .
>>       digit1-9 = %x31-39         ; 1-9
>>       e = %x65 / %x45            ; e E
>>       exp = e [ minus / plus ] 1*DIGIT
>>       frac = decimal-point 1*DIGIT
>>       int = zero / ( digit1-9 *DIGIT )
>>       minus = %x2D               ; -
>>       plus = %x2B                ; +
>>       zero = %x30                ; 0
>> 
>> Extensions are acceptable when we have an actual use for it, and we
>> document them properly.
>
> Well, it isn’t really an extension, because this isn’t a JSON parser but
> just something that accepts anything that looks like a number and then
> lets Python try a conversion on it.

I'm totally cool with deviating from JSON, all I really care about is
proper schema language documentation.

If we stick to JSON form number syntax, this is easy: replace "Numbers
and null are not supported" by "null is not supported", done.
Implementation should also be easy enough: convert RFC 8259's EBNF to a
regexp, feed the matched string to Python's number parser, done.

Fancier syntax we'd need to document ourselves.  I'm willing to deal
with that if we have a sufficiently compelling use for them.

>> Isn't the parenthesis in your regular expression redundant?
>
> You’re right, but on second thought, maybe I should surround it by \<
> and \>.
>
>> What use do you have in mind for 'inf' and 'nan'?
>
> I could imagine inf being a useful default value, actually.  nan,
> probably not so much.
>
>> Why accept leading '+' as in '+123'?
>> 
>> Why accept empty integral part as in '.123'?
>> 
>> Why accept '.xe.'?  Kidding you, that must be a bug in your regexp.
>
> Well, kind of.
>
> I wanted to accept anything that looks in any way like a number and then
> let Python try to convert it.  That’s also the reason why the case comes
> last.
>
> For that reason, I decided to keep the regex as simple as possible,
> because the attempted conversions would reject anything that isn’t (to
> Python) a valid number later.

Ah, now I see.

> It was my impression that the QAPI schema isn’t really JSON anyway and
> that our QAPI schema parser isn’t a JSON parser.  Under that assumption
> it simply seemed useful to me to accept anything that could potentially
> be a number to Python and convert it.
>
> Now, honestly, I still don’t see the point of having a strict JSON
> “parser” here, but if you insist.  Seems possible to do in a regex.
>
> Though I do think it makes sense to support hex integers as an extension.

Let's start simple & stupid.  Extensions can be added on top.
Hexadecimal integers may well be compelling enough to justify an
extension.

>> Please decide what number syntax you'd like to accept, then specify it
>> in docs/devel/qapi-code-gen.txt, so we can first discuss the
>> specification, and then check the regexp implements it.
>> 
>> docs/devel/qapi-code-gen.txt update goes here:
>> 
>>     === Schema syntax ===
>> 
>>     Syntax is loosely based on JSON (http://www.ietf.org/rfc/rfc8259.txt).
>>     Differences:
>> 
>>     * Comments: start with a hash character (#) that is not part of a
>>       string, and extend to the end of the line.
>> 
>>     * Strings are enclosed in 'single quotes', not "double quotes".
>> 
>>     * Strings are restricted to printable ASCII, and escape sequences to
>>       just '\\'.
>> 
>> --> * Numbers and null are not supported.
>
> OK.
>
>> Hrmm, commit 9d55380b5a "qapi: Remove null from schema language" left
>> two instances in error messages behind.  I'll fix them.
>> 
>>> @@ -584,7 +586,22 @@ class QAPISchemaParser(object):
>>>                      return
>>>                  self.line += 1
>>>                  self.line_pos = self.cursor
>>> -            elif not self.tok.isspace():
>>> +            elif self.tok.isspace():
>>> +                pass
>>> +            elif num_match.match(self.src[self.pos:]):
>>> +                match = num_match.match(self.src[self.pos:]).group(0)
>> 
>> Sadly, the walrus operator is Python 3.8.
>> 
>>> +                try:
>>> +                    self.val = int(match, 0)
>>> +                except ValueError:
>>> +                    try:
>>> +                        self.val = float(match)
>>> +                    except ValueError:
>>> +                        raise QAPIParseError(self,
>>> +                                '"%s" is not a valid integer or float' % match)
>>> +
>>> +                self.cursor += len(match) - 1
>>> +                return
>>> +            else:
>>>                  raise QAPIParseError(self, 'Stray "%s"' % self.tok)
>> 
>> Any particular reason for putting the number case last?
>
> Because the match is so broad.
>
>>>  
>>>      def get_members(self):
>>> @@ -617,9 +634,9 @@ class QAPISchemaParser(object):
>>>          if self.tok == ']':
>>>              self.accept()
>>>              return expr
>>> -        if self.tok not in "{['tfn":
>>> +        if self.tok not in "{['tfn-+0123456789.i":
>> 
>> This is getting a bit ugly.  Let's not worry about it now.
>> 
>>>              raise QAPIParseError(self, 'Expected "{", "[", "]", string, '
>>> -                                 'boolean or "null"')
>>> +                                 'boolean, number or "null"')
>>>          while True:
>>>              expr.append(self.get_expr(True))
>>>              if self.tok == ']':
>>> @@ -638,7 +655,7 @@ class QAPISchemaParser(object):
>>>          elif self.tok == '[':
>>>              self.accept()
>>>              expr = self.get_values()
>>> -        elif self.tok in "'tfn":
>>> +        elif self.tok in "'tfn-+0123456789.i":
>>>              expr = self.val
>>>              self.accept()
>>>          else:
>>> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
>>> index f62cf0a2e1..6a61dd831f 100644
>>> --- a/scripts/qapi/introspect.py
>>> +++ b/scripts/qapi/introspect.py
>>> @@ -57,6 +57,8 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
>>>          ret += indent(level) + '}))'
>>>      elif isinstance(obj, bool):
>>>          ret += 'QLIT_QBOOL(%s)' % ('true' if obj else 'false')
>>> +    elif isinstance(obj, int) and obj >= -(2 ** 63) and obj < 2 ** 63:
>>> +        ret += 'QLIT_QNUM(%i)' % obj
>> 
>> Please explain the range check.
>
> Will do.
>
>>>      else:
>>>          assert False                # not implemented
>>>      if level > 0:
>>> diff --git a/tests/qapi-schema/bad-type-int.err b/tests/qapi-schema/bad-type-int.err
>>> index da89895404..e22fb4f655 100644
>>> --- a/tests/qapi-schema/bad-type-int.err
>>> +++ b/tests/qapi-schema/bad-type-int.err
>>> @@ -1 +1 @@
>>> -tests/qapi-schema/bad-type-int.json:3:13: Stray "1"
>>> +tests/qapi-schema/bad-type-int.json:2: 'struct' key must have a string value
>> 
>> Test needs a rename, assuming it's not redundant now.
>
> I’m not adding a test here, it’s just the value has changed in
> 4d42815587063d.

The test name 'bad-type-int' is now bad, because the int in it is no
longer bad (pardon my lame puns).

> Thanks for reviewing!

Better late than never, I guess...  You're welcome!

>
> Max
>
>>> diff --git a/tests/qapi-schema/enum-int-member.err b/tests/qapi-schema/enum-int-member.err
>>> index 071c5213d8..112175f79d 100644
>>> --- a/tests/qapi-schema/enum-int-member.err
>>> +++ b/tests/qapi-schema/enum-int-member.err
>>> @@ -1 +1 @@
>>> -tests/qapi-schema/enum-int-member.json:3:31: Stray "1"
>>> +tests/qapi-schema/enum-int-member.json:2: Member of enum 'MyEnum' requires a string name
>> 
>> This one's name is still good.
>> 
>>> diff --git a/tests/qapi-schema/leading-comma-list.err b/tests/qapi-schema/leading-comma-list.err
>>> index f5c870bb9c..fa9c80aa57 100644
>>> --- a/tests/qapi-schema/leading-comma-list.err
>>> +++ b/tests/qapi-schema/leading-comma-list.err
>>> @@ -1 +1 @@
>>> -tests/qapi-schema/leading-comma-list.json:2:13: Expected "{", "[", "]", string, boolean or "null"
>>> +tests/qapi-schema/leading-comma-list.json:2:13: Expected "{", "[", "]", string, boolean, number or "null"



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members Max Reitz
@ 2019-11-14 15:53   ` Markus Armbruster
  2019-11-21 15:07   ` Markus Armbruster
  1 sibling, 0 replies; 41+ messages in thread
From: Markus Armbruster @ 2019-11-14 15:53 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth

Less than thorough review, because I expect the necessary rebase will
require a bit of rewriting here and there.

Max Reitz <mreitz@redhat.com> writes:

> With this change, it is possible to give default values for struct
> members, as follows:
>
>   What you had to do so far:
>
>     # @member: Some description, defaults to 42.
>     { 'struct': 'Foo',
>       'data': { '*member': 'int' } }
>
>   What you can do now:
>
>     { 'struct': 'Foo',
>       'data': { '*member': { 'type': 'int', 'default': 42 } }
>
> On the C side, this change would remove Foo.has_member, because
> Foo.member is always valid now.  The input visitor deals with setting
> it.  (Naturally, this means that such defaults are useful only for input
> parameters.)
>
> At least three things are left unimplemented:
>
> First, support for alternate data types.  This is because supporting
> them would mean having to allocate the object in the input visitor, and
> then potentially going through multiple levels of nested types.  In any
> case, it would have been difficult and I do not think there is need for
> such support at this point.

I don't mind restricting the 'default' feature to uses we actually have,
at least initially.

I'm afraid I don't fully understand the difficulties you describe, but I
guess that's okay.

> Second, support for null.  The most important reason for this is that
> introspection already uses "'default': null" for "no default, but this
> field is optional".  The second reason is that without support for
> alternate data types, there is not really a point in supporting null.

Also, commit 9d55380b5a "qapi: Remove null from schema language" :)

> Third, full support for default lists.  This has a similar reason to the
> lack of support for alternate data types: Allocating a default list is
> not trivial -- unless the list is empty, which is exactly what we have
> support for.

Your commit message says "for struct members".  What about union
members?  Cases:

* Flat union 'base' members: 'base' is a a struct, possibly implicit.
  Do defaults work in implicit bases, like BlockdevOption's?

* Flat union branch members: these are always struct types, so there's
  nothing for me to ask.  I think.

* Simple union branch members: these are each wrapped in an implicit
  struct type.  Do defaults work?  I'd be totally fine with "nope, not
  implemented, not going to implement it" here.

> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  qapi/introspect.json       |   9 +-
>  scripts/qapi/commands.py   |   2 +-
>  scripts/qapi/common.py     | 167 +++++++++++++++++++++++++++++++++++--
>  scripts/qapi/doc.py        |  20 ++++-
>  scripts/qapi/introspect.py |   2 +-
>  scripts/qapi/types.py      |   2 +-
>  scripts/qapi/visit.py      |  38 ++++++++-
>  7 files changed, 217 insertions(+), 23 deletions(-)

Missing: docs/devel/qapi-code-gen.txt update.

>
> diff --git a/qapi/introspect.json b/qapi/introspect.json
> index 1843c1cb17..db703135f9 100644
> --- a/qapi/introspect.json
> +++ b/qapi/introspect.json
> @@ -198,11 +198,10 @@
>  #
>  # @default: default when used as command parameter.
>  #           If absent, the parameter is mandatory.
> -#           If present, the value must be null.  The parameter is
> -#           optional, and behavior when it's missing is not specified
> -#           here.
> -#           Future extension: if present and non-null, the parameter
> -#           is optional, and defaults to this value.
> +#           If present and null, the parameter is optional, and
> +#           behavior when it's missing is not specified here.
> +#           If present and non-null, the parameter is optional, and
> +#           defaults to this value.
>  #
>  # Since: 2.5
>  ##
> diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
> index b929e07be4..6c407cd4ba 100644
> --- a/scripts/qapi/commands.py
> +++ b/scripts/qapi/commands.py
> @@ -35,7 +35,7 @@ def gen_call(name, arg_type, boxed, ret_type):
>      elif arg_type:
>          assert not arg_type.variants
>          for memb in arg_type.members:
> -            if memb.optional:
> +            if memb.optional and memb.default is None:
>                  argstr += 'arg.has_%s, ' % c_name(memb.name)
>              argstr += 'arg.%s, ' % c_name(memb.name)
>  
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index c6754a5856..8c57d0c67a 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -14,6 +14,7 @@
>  from __future__ import print_function
>  from contextlib import contextmanager
>  import errno
> +import math
>  import os
>  import re
>  import string
> @@ -800,6 +801,136 @@ def check_if(expr, info):
>          check_if_str(ifcond, info)
>  
>  
> +def check_value_str(info, value):
> +    return 'g_strdup(%s)' % to_c_string(value) if type(value) is str else False

What's wrong with isinstance(value, str)?

I'm a happy user of ternaries myself, but this one results in a rather
long line.  Easier to read, I think:

       if isinstance(value, str):
           return 'g_strdup(%s)' % to_c_string(value)
       return False

> +
> +def check_value_number(info, value):
> +    if type(value) is not float:
> +        return False
> +    if math.isinf(value):
> +        return 'INFINITY' if value > 0 else '-INFINITY'
> +    elif math.isnan(value):
> +        return 'NAN'
> +    else:
> +        return '%.16e' % value

Why not str(value)?

> +
> +def check_value_bool(info, value):
> +    if type(value) is not bool:
> +        return False
> +    return 'true' if value else 'false'
> +
> +def is_int_type(value):
> +    if type(value) is int:
> +        return True
> +    # 'long' does not exist in Python 3
> +    try:
> +        if type(value) is long:
> +            return True
> +    except NameError:
> +        pass
> +
> +    return False

Python 2 busy-work...

> +
> +def gen_check_value_int(bits):
> +    def check_value_int(info, value):
> +        if not is_int_type(value) or \
> +           value < -(2 ** (bits - 1)) or value >= 2 ** (bits - 1):
> +            return False
> +        if bits > 32:
> +            return '%ill' % value
> +        else:
> +            return '%i' % value

Why not str(value) regardless of @bits?

> +
> +    return check_value_int
> +
> +def gen_check_value_uint(bits):
> +    def check_value_uint(info, value):
> +        if not is_int_type(value) or value < 0 or value >= 2 ** bits:
> +            return False
> +        if bits > 32:
> +            return '%uull' % value
> +        elif bits > 16:
> +            return '%uu' % value
> +        else:
> +            return '%u' % value

Likewise.

> +
> +    return check_value_uint

Your check_value_FOO(info, value) have a peculiar contract: 

    If @value is a valid FOO, convert it to str and return that.  Else
    return False.

    @info is unused.

I wouldn't guess that from the name.  What about this: rename to
str_if_FOO(), return @value converted to str if it's a valid FOO, else
None.  Ditch @info unless there's a reason to keep it.

> +
> +# Check whether the given value fits the given QAPI type.
> +# If so, return a C representation of the value (pointers point to
> +# newly allocated objects).
> +# Otherwise, raise an exception.

The parenthesis gave me pause (I figure my body's busy digesting lunch).
The only pointer-valued type is 'str', isn't it?

Type-checking is in schema.py now.  Stuff like @enum_types is gone.  See
commit fa110c6a9e "qapi: Move context-sensitive checking to the proper
place".

> +def check_value(info, qapi_type, value):
> +    builtin_type_checks = {
> +        'str':      check_value_str,
> +        'int':      gen_check_value_int(64),
> +        'number':   check_value_number,
> +        'bool':     check_value_bool,
> +        'int8':     gen_check_value_int(8),
> +        'int16':    gen_check_value_int(16),
> +        'int32':    gen_check_value_int(32),
> +        'int64':    gen_check_value_int(64),
> +        'uint8':    gen_check_value_uint(8),
> +        'uint16':   gen_check_value_uint(16),
> +        'uint32':   gen_check_value_uint(32),
> +        'uint64':   gen_check_value_uint(64),
> +        'size':     gen_check_value_uint(64),
> +    }
> +
> +    # Cannot support null because that would require a value of "None"
> +    # (which is reserved for no default)

This is another instance of the class of problems that led to commit
9d55380b5a "qapi: Remove null from schema language".

> +    unsupported_builtin_types = ['null', 'any', 'QType']

You give a clue for 'null'.  Should you give one for 'any' and 'QType',
too?

> +
> +    if type(qapi_type) is list:
> +        has_list = True
> +        qapi_type = qapi_type[0]
> +    elif qapi_type.endswith('List'):
> +        has_list = True
> +        qapi_type = qapi_type[:-4]
> +    else:
> +        has_list = False
> +
> +    if has_list:
> +        if value == []:
> +            return 'NULL'
> +        else:
> +            raise QAPISemError(info,
> +                "Support for non-empty lists as default values has not been " \
> +                "implemented yet: '{}'".format(value))
> +
> +    if qapi_type in builtin_type_checks:
> +        c_val = builtin_type_checks[qapi_type](info, value)
> +        if not c_val:
> +            raise QAPISemError(info,
> +                "Value '{}' does not match type {}".format(value, qapi_type))
> +        return c_val
> +
> +    if qapi_type in unsupported_builtin_types:
> +        raise QAPISemError(info,
> +                           "Cannot specify values for type %s" % qapi_type)
> +
> +    if qapi_type in enum_types:
> +        if not check_value_str(info, value):
> +            raise QAPISemError(info,
> +                "Enum values must be strings, but '{}' is no string" \
> +                        .format(value))
> +
> +        enum_values = enum_types[qapi_type]['data']
> +        for ev in enum_values:
> +            if ev['name'] == value:
> +                return c_enum_const(qapi_type, value,
> +                                    enum_types[qapi_type].get('prefix'))
> +
> +        raise QAPISemError(info,
> +            "Value '{}' does not occur in enum {}".format(value, qapi_type))
> +
> +    # TODO: Support alternates
> +
> +    raise QAPISemError(info,
> +        "Cannot specify values for type %s (not built-in or an enum)" %
> +        qapi_type)
> +
> +

check_value() furses checking and converting.  Its callers seem to need
either the checking or the converting.  I'm not sure fusing the two is a
good idea.

>  def check_type(info, source, value, allow_array=False,
>                 allow_dict=False, allow_optional=False,
>                 allow_metas=[]):
> @@ -842,15 +973,22 @@ def check_type(info, source, value, allow_array=False,
>          if c_name(key, False) == 'u' or c_name(key, False).startswith('has_'):
>              raise QAPISemError(info, "Member of %s uses reserved name '%s'"
>                                 % (source, key))
> -        # Todo: allow dictionaries to represent default values of
> -        # an optional argument.
> +
>          check_known_keys(info, "member '%s' of %s" % (key, source),
> -                         arg, ['type'], ['if'])
> +                         arg, ['type'], ['if', 'default'])
>          check_type(info, "Member '%s' of %s" % (key, source),
>                     arg['type'], allow_array=True,
>                     allow_metas=['built-in', 'union', 'alternate', 'struct',
>                                  'enum'])
>  
> +        if 'default' in arg:
> +            if key[0] != '*':
> +                raise QAPISemError(info,
> +                    "'%s' is not optional, so it cannot have a default value" %
> +                    key)
> +
> +            check_value(info, arg['type'], arg['default'])
> +
>  
>  def check_command(expr, info):
>      name = expr['command']
> @@ -1601,13 +1739,14 @@ class QAPISchemaFeature(QAPISchemaMember):
>  
>  
>  class QAPISchemaObjectTypeMember(QAPISchemaMember):
> -    def __init__(self, name, typ, optional, ifcond=None):
> +    def __init__(self, name, typ, optional, ifcond=None, default=None):
>          QAPISchemaMember.__init__(self, name, ifcond)
>          assert isinstance(typ, str)
>          assert isinstance(optional, bool)
>          self._type_name = typ
>          self.type = None
>          self.optional = optional
> +        self.default = default
>  
>      def check(self, schema):
>          assert self.owner
> @@ -1917,7 +2056,7 @@ class QAPISchema(object):
>              name, info, doc, ifcond,
>              self._make_enum_members(data), prefix))
>  
> -    def _make_member(self, name, typ, ifcond, info):
> +    def _make_member(self, name, typ, ifcond, default, info):
>          optional = False
>          if name.startswith('*'):
>              name = name[1:]
> @@ -1925,10 +2064,11 @@ class QAPISchema(object):
>          if isinstance(typ, list):
>              assert len(typ) == 1
>              typ = self._make_array_type(typ[0], info)
> -        return QAPISchemaObjectTypeMember(name, typ, optional, ifcond)
> +        return QAPISchemaObjectTypeMember(name, typ, optional, ifcond, default)
>  
>      def _make_members(self, data, info):
> -        return [self._make_member(key, value['type'], value.get('if'), info)
> +        return [self._make_member(key, value['type'], value.get('if'),
> +                                  value.get('default'), info)
>                  for (key, value) in data.items()]
>  
>      def _def_struct_type(self, expr, info, doc):
> @@ -1951,7 +2091,7 @@ class QAPISchema(object):
>              typ = self._make_array_type(typ[0], info)
>          typ = self._make_implicit_object_type(
>              typ, info, None, self.lookup_type(typ),
> -            'wrapper', [self._make_member('data', typ, None, info)])
> +            'wrapper', [self._make_member('data', typ, None, None, info)])
>          return QAPISchemaObjectTypeVariant(case, typ, ifcond)
>  
>      def _def_union_type(self, expr, info, doc):
> @@ -2234,6 +2374,15 @@ def to_c_string(string):
>      return result
>  
>  
> +# Translates a value for the given QAPI type to its C representation.
> +# The caller must have called check_value() during parsing to be sure
> +# that the given value fits the type.
> +def c_value(qapi_type, value):
> +    pseudo_info = {'file': '(generator bug)', 'line': 0, 'parent': None}
> +    # The caller guarantees this does not raise an exception
> +    return check_value(pseudo_info, qapi_type, value)
> +
> +
>  def guardstart(name):
>      return mcgen('''
>  #ifndef %(name)s
> @@ -2356,7 +2505,7 @@ def build_params(arg_type, boxed, extra=None):
>          for memb in arg_type.members:
>              ret += sep
>              sep = ', '
> -            if memb.optional:
> +            if memb.optional and memb.default is None:
>                  ret += 'bool has_%s, ' % c_name(memb.name)
>              ret += '%s %s' % (memb.type.c_param_type(),
>                                c_name(memb.name))
> diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
> index 5fc0fc7e06..78a9052738 100755
> --- a/scripts/qapi/doc.py
> +++ b/scripts/qapi/doc.py
> @@ -139,13 +139,29 @@ def texi_enum_value(value, desc, suffix):
>          value.name, desc, texi_if(value.ifcond, prefix='@*'))
>  
>  
> +def doc_value(value):
> +    if value is True:
> +        return 'true'
> +    elif value is False:
> +        return 'false'
> +    elif value is None:
> +        return 'null'
> +    else:
> +        return '{}'.format(value)
> +
>  def texi_member(member, desc, suffix):
>      """Format a table of members item for an object type member"""
>      typ = member.type.doc_type()
>      membertype = ': ' + typ if typ else ''
> +
> +    optional_info = ''
> +    if member.default is not None:
> +        optional_info = ' (optional, default: %s)' % doc_value(member.default)
> +    elif member.optional:
> +        optional_info = ' (optional)'
> +
>      return '@item @code{%s%s}%s%s\n%s%s' % (
> -        member.name, membertype,
> -        ' (optional)' if member.optional else '',
> +        member.name, membertype, optional_info,
>          suffix, desc, texi_if(member.ifcond, prefix='@*'))
>  
>  
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index 572e0b8331..7d73020a42 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -159,7 +159,7 @@ const QLitObject %(c_name)s = %(c_string)s;
>      def _gen_member(self, member):
>          ret = {'name': member.name, 'type': self._use_type(member.type)}
>          if member.optional:
> -            ret['default'] = None
> +            ret['default'] = member.default
>          if member.ifcond:
>              ret = (ret, {'if': member.ifcond})
>          return ret
> diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
> index 3edd9374aa..46a6d33379 100644
> --- a/scripts/qapi/types.py
> +++ b/scripts/qapi/types.py
> @@ -44,7 +44,7 @@ def gen_struct_members(members):
>      ret = ''
>      for memb in members:
>          ret += gen_if(memb.ifcond)
> -        if memb.optional:
> +        if memb.optional and memb.default is None:
>              ret += mcgen('''
>      bool has_%(c_name)s;
>  ''',
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index 484ebb66ad..0960e25a25 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -40,10 +40,14 @@ def gen_visit_object_members(name, base, members, variants):
>  void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
>  {
>      Error *err = NULL;
> -
>  ''',
>                  c_name=c_name(name))
>  
> +    if len([m for m in members if m.default is not None]) > 0:
> +        ret += mcgen('''
> +    bool has_optional;
> +''')
> +
>      if base:
>          ret += mcgen('''
>      visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, &err);
> @@ -53,13 +57,28 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
>  ''',
>                       c_type=base.c_name())
>  
> +    ret += mcgen('''
> +
> +''')
> +
>      for memb in members:
>          ret += gen_if(memb.ifcond)
>          if memb.optional:
> +            if memb.default is not None:
> +                optional_target = 'has_optional'
> +                # Visitors other than the input visitor do not have to implement
> +                # .optional().  Therefore, we have to initialize has_optional.

Suggest "Only input visitors must implement .optional()."


> +                # Initialize it to true, because the field's value is always
> +                # present when using any visitor but the input visitor.
> +                ret += mcgen('''
> +    has_optional = true;
> +''')
> +            else:
> +                optional_target = 'obj->has_' + c_name(memb.name)
>              ret += mcgen('''
> -    if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
> +    if (visit_optional(v, "%(name)s", &%(opt_target)s)) {
>  ''',
> -                         name=memb.name, c_name=c_name(memb.name))
> +                         name=memb.name, opt_target=optional_target)
>              push_indent()
>          ret += mcgen('''
>      visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, &err);

I've stared at this dumbly for too long.  It can't actually be that
hard.  I'm afraid I've run out of steam for today.  I'll continue when
my steam pressure is back to operational.


> @@ -69,7 +88,16 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
>  ''',
>                       c_type=memb.type.c_name(), name=memb.name,
>                       c_name=c_name(memb.name))
> -        if memb.optional:
> +        if memb.default is not None:
> +            pop_indent()
> +            ret += mcgen('''
> +    } else {
> +        obj->%(c_name)s = %(c_value)s;
> +    }
> +''',
> +                         c_name=c_name(memb.name),
> +                         c_value=c_value(memb._type_name, memb.default))
> +        elif memb.optional:
>              pop_indent()
>              ret += mcgen('''
>      }
> @@ -287,6 +315,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
>          self._add_system_module(None, ' * Built-in QAPI visitors')
>          self._genc.preamble_add(mcgen('''
>  #include "qemu/osdep.h"
> +#include <math.h>
>  #include "qapi/error.h"
>  #include "qapi/qapi-builtin-visit.h"
>  '''))
> @@ -302,6 +331,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
>          visit = self._module_basename('qapi-visit', name)
>          self._genc.preamble_add(mcgen('''
>  #include "qemu/osdep.h"
> +#include <math.h>
>  #include "qapi/error.h"
>  #include "qapi/qmp/qerror.h"
>  #include "%(visit)s.h"



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members Max Reitz
  2019-11-14 15:53   ` Markus Armbruster
@ 2019-11-21 15:07   ` Markus Armbruster
  2019-11-21 15:24     ` Eric Blake
  1 sibling, 1 reply; 41+ messages in thread
From: Markus Armbruster @ 2019-11-21 15:07 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth

Max Reitz <mreitz@redhat.com> writes:

> With this change, it is possible to give default values for struct
> members, as follows:
>
>   What you had to do so far:
>
>     # @member: Some description, defaults to 42.
>     { 'struct': 'Foo',
>       'data': { '*member': 'int' } }
>
>   What you can do now:
>
>     { 'struct': 'Foo',
>       'data': { '*member': { 'type': 'int', 'default': 42 } }

The '*' is redundant in this form.

Can anyone think of reasons for keeping it anyway?  Against?

> On the C side, this change would remove Foo.has_member, because
> Foo.member is always valid now.  The input visitor deals with setting
> it.  (Naturally, this means that such defaults are useful only for input
> parameters.)
>
> At least three things are left unimplemented:
>
> First, support for alternate data types.  This is because supporting
> them would mean having to allocate the object in the input visitor, and
> then potentially going through multiple levels of nested types.  In any
> case, it would have been difficult and I do not think there is need for
> such support at this point.
>
> Second, support for null.  The most important reason for this is that
> introspection already uses "'default': null" for "no default, but this
> field is optional".  The second reason is that without support for
> alternate data types, there is not really a point in supporting null.
>
> Third, full support for default lists.  This has a similar reason to the
> lack of support for alternate data types: Allocating a default list is
> not trivial -- unless the list is empty, which is exactly what we have
> support for.
>
> Signed-off-by: Max Reitz <mreitz@redhat.com>



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 04/14] qapi: Allow optional discriminators
  2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 04/14] qapi: Allow optional discriminators Max Reitz
@ 2019-11-21 15:13   ` Markus Armbruster
  0 siblings, 0 replies; 41+ messages in thread
From: Markus Armbruster @ 2019-11-21 15:13 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth

Max Reitz <mreitz@redhat.com> writes:

> Optional discriminators are fine, as long as there is a default value.
>
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  scripts/qapi/common.py | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 8c57d0c67a..203623795b 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -1052,11 +1052,21 @@ def check_union(expr, info):
>          base_members = find_base_members(base)
>          assert base_members is not None
>  
> -        # The value of member 'discriminator' must name a non-optional
> -        # member of the base struct.
> +        # The value of member 'discriminator' must name a member of
> +        # the base struct.  (Optional members are allowed, but the
> +        # discriminator name must not start with '*', so keep
> +        # allow_optional=False.)
>          check_name(info, "Discriminator of flat union '%s'" % name,
>                     discriminator)
> +
>          discriminator_value = base_members.get(discriminator)
> +        if not discriminator_value:
> +            discriminator_value = base_members.get('*' + discriminator)
> +            if discriminator_value and 'default' not in discriminator_value:
> +                raise QAPISemError(info,
> +                    "Optional discriminator '%s' has no default value" %
> +                    discriminator)
> +
>          if not discriminator_value:
>              raise QAPISemError(info,
>                                 "Discriminator '%s' is not a member of base "

Needs test coverage and doc update.

Oh, looks like later patches provide.  Please consider squashing.  Doc
updates and tests often make code changes easier to understand.



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [PATCH v4 00/14] block: Try to create well-typed json:{} filenames
  2019-11-14  8:54     ` Markus Armbruster
@ 2019-11-21 15:17       ` Markus Armbruster
  0 siblings, 0 replies; 41+ messages in thread
From: Markus Armbruster @ 2019-11-21 15:17 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, Michael Roth, qemu-block, qemu-devel

Markus Armbruster <armbru@redhat.com> writes:

> Max Reitz <mreitz@redhat.com> writes:
>
>> On 13.09.19 13:49, Max Reitz wrote:
>>> Another gentle ping.
>>
>> And another.
>
> Conflicts with the refactoring merged in commit 69717d0f890.  Please
> accept my apologies for the inconvenience caused by the excessive delay.
>
> I'll try to review anyway.

I reviewed the proposed changes to the QAPI schema language, and they
look reasonable enough to justify a rebase.

Thanks!



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-11-21 15:07   ` Markus Armbruster
@ 2019-11-21 15:24     ` Eric Blake
  2019-11-21 19:46       ` Markus Armbruster
  0 siblings, 1 reply; 41+ messages in thread
From: Eric Blake @ 2019-11-21 15:24 UTC (permalink / raw)
  To: Markus Armbruster, Max Reitz
  Cc: Kevin Wolf, qemu-devel, qemu-block, Michael Roth

On 11/21/19 9:07 AM, Markus Armbruster wrote:
> Max Reitz <mreitz@redhat.com> writes:
> 
>> With this change, it is possible to give default values for struct
>> members, as follows:
>>
>>    What you had to do so far:
>>
>>      # @member: Some description, defaults to 42.
>>      { 'struct': 'Foo',
>>        'data': { '*member': 'int' } }
>>
>>    What you can do now:
>>
>>      { 'struct': 'Foo',
>>        'data': { '*member': { 'type': 'int', 'default': 42 } }
> 
> The '*' is redundant in this form.
> 
> Can anyone think of reasons for keeping it anyway?  Against?

Is there ever a reason to allow an optional member but without a 
'default' value?  Or can we just blindly state that if 'default' is not 
present, that is the same as 'default':0/'default':null?

Or, applying your statement further,

'data': { '*a':'int', '*b':'str' }

is shorthand for:

'data': { 'a': { 'type':'int', 'default':0 },
           'b': { 'type':'str', 'default':null } }

So I could live with permitting '*' only in the shorthand form, and 
declaring that it is incompatible with longhand form because the 
existence of a 'default' key in longhand form is evidence that the 
member is therefore optional.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-11-21 15:24     ` Eric Blake
@ 2019-11-21 19:46       ` Markus Armbruster
  2019-11-21 19:56         ` Eric Blake
  0 siblings, 1 reply; 41+ messages in thread
From: Markus Armbruster @ 2019-11-21 19:46 UTC (permalink / raw)
  To: Eric Blake; +Cc: Kevin Wolf, Michael Roth, qemu-devel, qemu-block, Max Reitz

Eric Blake <eblake@redhat.com> writes:

> On 11/21/19 9:07 AM, Markus Armbruster wrote:
>> Max Reitz <mreitz@redhat.com> writes:
>>
>>> With this change, it is possible to give default values for struct
>>> members, as follows:
>>>
>>>    What you had to do so far:
>>>
>>>      # @member: Some description, defaults to 42.
>>>      { 'struct': 'Foo',
>>>        'data': { '*member': 'int' } }
>>>
>>>    What you can do now:
>>>
>>>      { 'struct': 'Foo',
>>>        'data': { '*member': { 'type': 'int', 'default': 42 } }
>>
>> The '*' is redundant in this form.
>>
>> Can anyone think of reasons for keeping it anyway?  Against?
>
> Is there ever a reason to allow an optional member but without a
> 'default' value?  Or can we just blindly state that if 'default' is
> not present, that is the same as 'default':0/'default':null?
>
> Or, applying your statement further,
>
> 'data': { '*a':'int', '*b':'str' }
>
> is shorthand for:
>
> 'data': { 'a': { 'type':'int', 'default':0 },
>           'b': { 'type':'str', 'default':null } }

You propose to default 'default' to a type-specific value.

I don't think that's a good idea.

Quoting myself on v3:

    In many programming languages, absent optional arguments / members
    default to a default value specified in the declaration.  Simple.

    In others, 'absent' is effectively an additional value.  The code
    consuming the argument / member can interpret 'absent' however it wants.
    It may eliminate the additional value by mapping it to a default value,
    but it can also interpret 'absent' unlike any value.  If there's more
    than one consumer, their interpretations need not be consistent.  The
    declaration provides no clue on semantics of 'absent'.

    QAPI is in the latter camp.  I trust you can already sense how much I
    like that.
    [...]
    If I could go back in time, I'd flip QAPI to "'absent' defaults to a
    default value".  Lacking a time machine, we're stuck with cases of
    "'absent' means something you can't express with a value" and "'absent'
    means different things in different contexts" that have been enshrined
    in external interfaces.  Is there anything we could do to improve
    matters for saner cases?

    I think there is: we could provide for an *optional* default value.  If
    the schema specifies it, that's what 'absent' means.  If it doesn't, all
    bets are off, just like they are now.

This patch implements this idea.

When an absent member behaves just like it was present with a certain
value DFLT, we want to be able to say in the schema 'default': DFLT.

But the schema language also needs to let us say "absent member behaves
unlike any value".

If we define 'default' to default to a value, then that value must have
this special meaning.

Where that value is also a valid value, the schema language cannot
express "absent member behaves like it was present with that value".

I think this makes 0 a poor default value for 'default': "behaves like
member was present with value 0" is fairly common, I think.

Defaulting 'default' to null regardless of member type could work.

null is a valid value of the 'null' type and of alternate types with a
member of type 'null'.  For optional members of such types, the schema
language then can't express ""absent member behaves like it was present
with value null".  I think the need to say that is much less common than
the needs to say "like value 0".

Checking...  *sigh*: there are a few optional members that can take null
values, e.g. MigrateSetParameters member @tls-creds.  I read its doc
comment twice, and I have to admit I can't make heads or tails of it.
Can't say for sure whether absent behaves like null, or some other
value, or unlike any value.

QAPI/QMP introspection already specifies null to have exactly this
special meaning.

> So I could live with permitting '*' only in the shorthand form, and
> declaring that it is incompatible with longhand form because the
> existence of a 'default' key in longhand form is evidence that the
> member is therefore optional.

Noted.

More opinions?



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-11-21 19:46       ` Markus Armbruster
@ 2019-11-21 19:56         ` Eric Blake
  2019-11-22  7:29           ` Markus Armbruster
  0 siblings, 1 reply; 41+ messages in thread
From: Eric Blake @ 2019-11-21 19:56 UTC (permalink / raw)
  To: Markus Armbruster
  Cc: Kevin Wolf, Michael Roth, qemu-devel, qemu-block, Max Reitz

On 11/21/19 1:46 PM, Markus Armbruster wrote:

>>> The '*' is redundant in this form.
>>>
>>> Can anyone think of reasons for keeping it anyway?  Against?
>>
>> Is there ever a reason to allow an optional member but without a
>> 'default' value?  Or can we just blindly state that if 'default' is
>> not present, that is the same as 'default':0/'default':null?
>>
>> Or, applying your statement further,
>>
>> 'data': { '*a':'int', '*b':'str' }
>>
>> is shorthand for:
>>
>> 'data': { 'a': { 'type':'int', 'default':0 },
>>            'b': { 'type':'str', 'default':null } }
> 
> You propose to default 'default' to a type-specific value.
> 
> I don't think that's a good idea.

...


> When an absent member behaves just like it was present with a certain
> value DFLT, we want to be able to say in the schema 'default': DFLT.
> 
> But the schema language also needs to let us say "absent member behaves
> unlike any value".
> 
> If we define 'default' to default to a value, then that value must have
> this special meaning.
> 
> Where that value is also a valid value, the schema language cannot
> express "absent member behaves like it was present with that value".
> 
> I think this makes 0 a poor default value for 'default': "behaves like
> member was present with value 0" is fairly common, I think.
> 
> Defaulting 'default' to null regardless of member type could work.
> 
> null is a valid value of the 'null' type and of alternate types with a
> member of type 'null'.  For optional members of such types, the schema
> language then can't express ""absent member behaves like it was present
> with value null".  I think the need to say that is much less common than
> the needs to say "like value 0".
> 
> Checking...  *sigh*: there are a few optional members that can take null
> values, e.g. MigrateSetParameters member @tls-creds.  I read its doc
> comment twice, and I have to admit I can't make heads or tails of it.
> Can't say for sure whether absent behaves like null, or some other
> value, or unlike any value.
> 
> QAPI/QMP introspection already specifies null to have exactly this
> special meaning.

Maybe that means we need '*name':'t' to expand into something longer, maybe
  'name':{'type':'t', 'optional':true}
which in  turn would be synonymous with your idea of ALL types accepting 
a default of null:
  'name':{'type':'t', 'optional':true, 'default':null}

At any rate, your counterpoint is taken - whatever we pick, we'll want 
to make sure that introspection can expose semantics, and whether we can 
make '*' redundant with some other form of longhand in the qapi language 
is in part determined by whether we also reflect that through 
introspection.  If that means that keeping '*' in the longhand form of 
optional members (whether or not those members have a default value), 
then so be it.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-11-21 19:56         ` Eric Blake
@ 2019-11-22  7:29           ` Markus Armbruster
  2019-11-22 10:25             ` Kevin Wolf
  0 siblings, 1 reply; 41+ messages in thread
From: Markus Armbruster @ 2019-11-22  7:29 UTC (permalink / raw)
  To: Eric Blake; +Cc: Kevin Wolf, Max Reitz, Michael Roth, qemu-block, qemu-devel

Eric Blake <eblake@redhat.com> writes:

> On 11/21/19 1:46 PM, Markus Armbruster wrote:
>
>>>> The '*' is redundant in this form.
>>>>
>>>> Can anyone think of reasons for keeping it anyway?  Against?
>>>
>>> Is there ever a reason to allow an optional member but without a
>>> 'default' value?  Or can we just blindly state that if 'default' is
>>> not present, that is the same as 'default':0/'default':null?
>>>
>>> Or, applying your statement further,
>>>
>>> 'data': { '*a':'int', '*b':'str' }
>>>
>>> is shorthand for:
>>>
>>> 'data': { 'a': { 'type':'int', 'default':0 },
>>>            'b': { 'type':'str', 'default':null } }
>>
>> You propose to default 'default' to a type-specific value.
>>
>> I don't think that's a good idea.
>
> ...
>
>
>> When an absent member behaves just like it was present with a certain
>> value DFLT, we want to be able to say in the schema 'default': DFLT.
>>
>> But the schema language also needs to let us say "absent member behaves
>> unlike any value".
>>
>> If we define 'default' to default to a value, then that value must have
>> this special meaning.
>>
>> Where that value is also a valid value, the schema language cannot
>> express "absent member behaves like it was present with that value".
>>
>> I think this makes 0 a poor default value for 'default': "behaves like
>> member was present with value 0" is fairly common, I think.
>>
>> Defaulting 'default' to null regardless of member type could work.
>>
>> null is a valid value of the 'null' type and of alternate types with a
>> member of type 'null'.  For optional members of such types, the schema
>> language then can't express ""absent member behaves like it was present
>> with value null".  I think the need to say that is much less common than
>> the needs to say "like value 0".
>>
>> Checking...  *sigh*: there are a few optional members that can take null
>> values, e.g. MigrateSetParameters member @tls-creds.  I read its doc
>> comment twice, and I have to admit I can't make heads or tails of it.
>> Can't say for sure whether absent behaves like null, or some other
>> value, or unlike any value.
>>
>> QAPI/QMP introspection already specifies null to have exactly this
>> special meaning.
>
> Maybe that means we need '*name':'t' to expand into something longer, maybe
>  'name':{'type':'t', 'optional':true}
> which in  turn would be synonymous with your idea of ALL types
> accepting a default of null:
>  'name':{'type':'t', 'optional':true, 'default':null}

Yes, this is something we can consider.

Currently, we normalize away the '*' prefix when we go from the abstract
parse tree (which we call "expressions") to the internal representation:

    def _make_member(self, name, typ, ifcond, info):
        optional = False
        if name.startswith('*'):
            name = name[1:]
            optional = True
        if isinstance(typ, list):
            assert len(typ) == 1
            typ = self._make_array_type(typ[0], info)
        return QAPISchemaObjectTypeMember(name, info, typ, optional, ifcond)

This keeps the normalization internal.

Other normalizations we perform on the abstract parse tree.  For
instance, here's the one that normalizes shorthand member "'KEY': 'ARG'"
to longhand "'KEY': { 'type': 'ARG' }":

    def normalize_members(members):
        if isinstance(members, OrderedDict):
            for key, arg in members.items():
                if isinstance(arg, dict):
                    continue
                members[key] = {'type': arg}

For these normalizations, both the shorthand and the longhand form are
part of the schema language.  We could do the same for '*'.

> At any rate, your counterpoint is taken - whatever we pick, we'll want
> to make sure that introspection can expose semantics, and whether we
> can make '*' redundant with some other form of longhand in the qapi
> language is in part determined by whether we also reflect that through
> introspection.

Introspection has the true member name, without the '*' prefix.

We'll also want to avoid unnecessary compromises on QAPI schema
expressiveness.  If we use null to mean "schema does not specify
behavior when member is absent", we can't use it to mean "absent member
behaves like the value null".  A bit of a blemish, but I think it's a
tolerable one.

>                 If that means that keeping '*' in the longhand form of
> optional members (whether or not those members have a default value),
> then so be it.

I believe both

    '*KEY': { 'type': ARG': 'default': null }

and

    'KEY': { 'type': ARG': 'default': null }

are viable longhand forms for '*KEY': 'ARG'.

I prefer the latter, but I'm open to arguments.



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-11-22  7:29           ` Markus Armbruster
@ 2019-11-22 10:25             ` Kevin Wolf
  2019-11-22 14:40               ` Markus Armbruster
  0 siblings, 1 reply; 41+ messages in thread
From: Kevin Wolf @ 2019-11-22 10:25 UTC (permalink / raw)
  To: Markus Armbruster; +Cc: Max Reitz, Michael Roth, qemu-block, qemu-devel

Am 22.11.2019 um 08:29 hat Markus Armbruster geschrieben:
> > At any rate, your counterpoint is taken - whatever we pick, we'll want
> > to make sure that introspection can expose semantics, and whether we
> > can make '*' redundant with some other form of longhand in the qapi
> > language is in part determined by whether we also reflect that through
> > introspection.
> 
> Introspection has the true member name, without the '*' prefix.
> 
> We'll also want to avoid unnecessary compromises on QAPI schema
> expressiveness.  If we use null to mean "schema does not specify
> behavior when member is absent", we can't use it to mean "absent member
> behaves like the value null".  A bit of a blemish, but I think it's a
> tolerable one.

If you want an example for an option that defaults to null, take the
backing option of BlockdevOptionsGenericCOWFormat.

What is the reason for even considering limiting the expressiveness? Do
you think that an additional 'optional' bool, at least for those options
that don't have a default, would be so bad in the longhand form? Or
keeping '*' even in the longhand form, as suggested below.

> >                 If that means that keeping '*' in the longhand form of
> > optional members (whether or not those members have a default value),
> > then so be it.
> 
> I believe both
> 
>     '*KEY': { 'type': ARG': 'default': null }
> 
> and
> 
>     'KEY': { 'type': ARG': 'default': null }
> 
> are viable longhand forms for '*KEY': 'ARG'.
> 
> I prefer the latter, but I'm open to arguments.

If you go for the former, then you certainly want to use absent
'default' to indicate no default, and allow a QNull default with
'default': null.

The only reason to abuse 'default': null for no default is that you
can't distinguish optional and non-optional if you use 'KEY' for both
instead of 'KEY' for mandatory and '*KEY' for optional.

So while I understand and to some degree share your dislike for the '*'
prefix, I think I cast my pragmatic vote for:

mandatory:                   'KEY':  { 'type': 'ARG' }
optional without a default:  '*KEY': { 'type': 'ARG' }
optional with QNull default: '*KEY': { 'type': 'ARG', 'default': null }

Kevin



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-11-22 10:25             ` Kevin Wolf
@ 2019-11-22 14:40               ` Markus Armbruster
  2019-11-22 16:12                 ` Kevin Wolf
  0 siblings, 1 reply; 41+ messages in thread
From: Markus Armbruster @ 2019-11-22 14:40 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: qemu-devel, Michael Roth, qemu-block, Max Reitz

Kevin Wolf <kwolf@redhat.com> writes:

> Am 22.11.2019 um 08:29 hat Markus Armbruster geschrieben:
>> > At any rate, your counterpoint is taken - whatever we pick, we'll want
>> > to make sure that introspection can expose semantics, and whether we
>> > can make '*' redundant with some other form of longhand in the qapi
>> > language is in part determined by whether we also reflect that through
>> > introspection.
>> 
>> Introspection has the true member name, without the '*' prefix.
>> 
>> We'll also want to avoid unnecessary compromises on QAPI schema
>> expressiveness.  If we use null to mean "schema does not specify
>> behavior when member is absent", we can't use it to mean "absent member
>> behaves like the value null".  A bit of a blemish, but I think it's a
>> tolerable one.
>
> If you want an example for an option that defaults to null, take the
> backing option of BlockdevOptionsGenericCOWFormat.
>
> What is the reason for even considering limiting the expressiveness? Do
> you think that an additional 'optional' bool, at least for those options
> that don't have a default, would be so bad in the longhand form? Or
> keeping '*' even in the longhand form, as suggested below.

Well, one reason is this:

    ##
    # @SchemaInfoObjectMember:
    #
    # An object member.
    #
    # @name: the member's name, as defined in the QAPI schema.
    #
    # @type: the name of the member's type.
    #
    # @default: default when used as command parameter.
    #           If absent, the parameter is mandatory.
    #           If present, the value must be null.  The parameter is
    #           optional, and behavior when it's missing is not specified
    #           here.
    #           Future extension: if present and non-null, the parameter
    #           is optional, and defaults to this value.
    #
    # Since: 2.5
    ##

If we want to be able to express the difference between "behavior when
absent is not specified here" and "absent behaves like value null", then
we need to somehow add that bit of information here.

Could use a feature.  Features are not yet implemented for members, but
we need them anyway.

>> >                 If that means that keeping '*' in the longhand form of
>> > optional members (whether or not those members have a default value),
>> > then so be it.
>> 
>> I believe both
>> 
>>     '*KEY': { 'type': ARG': 'default': null }
>> 
>> and
>> 
>>     'KEY': { 'type': ARG': 'default': null }
>> 
>> are viable longhand forms for '*KEY': 'ARG'.
>> 
>> I prefer the latter, but I'm open to arguments.
>
> If you go for the former, then you certainly want to use absent
> 'default' to indicate no default, and allow a QNull default with
> 'default': null.
>
> The only reason to abuse 'default': null for no default is that you
> can't distinguish optional and non-optional if you use 'KEY' for both
> instead of 'KEY' for mandatory and '*KEY' for optional.
>
> So while I understand and to some degree share your dislike for the '*'
> prefix, I think I cast my pragmatic vote for:
>
> mandatory:                   'KEY':  { 'type': 'ARG' }
> optional without a default:  '*KEY': { 'type': 'ARG' }
> optional with QNull default: '*KEY': { 'type': 'ARG', 'default': null }

The last one could also be     'KEY': { 'type': 'ARG', 'default': null }
without loss of expressiveness.

Differently ugly.

Here's yet another idea.  For the "absent is not specified here" case,
use

    'KEY': { 'type': 'ARG', optional: true }
    '*KEY': 'ARG'

For the "absent defaults to DEFVAL" case, use

    'KEY': { 'type': 'ARG', optional: true, 'default': DEFVAL }
    'KEY': { 'type': 'ARG', 'default': DEFVAL }



^ permalink raw reply	[flat|nested] 41+ messages in thread

* Re: [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members
  2019-11-22 14:40               ` Markus Armbruster
@ 2019-11-22 16:12                 ` Kevin Wolf
  0 siblings, 0 replies; 41+ messages in thread
From: Kevin Wolf @ 2019-11-22 16:12 UTC (permalink / raw)
  To: Markus Armbruster; +Cc: qemu-devel, Michael Roth, qemu-block, Max Reitz

Am 22.11.2019 um 15:40 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
> 
> > Am 22.11.2019 um 08:29 hat Markus Armbruster geschrieben:
> >> > At any rate, your counterpoint is taken - whatever we pick, we'll want
> >> > to make sure that introspection can expose semantics, and whether we
> >> > can make '*' redundant with some other form of longhand in the qapi
> >> > language is in part determined by whether we also reflect that through
> >> > introspection.
> >> 
> >> Introspection has the true member name, without the '*' prefix.
> >> 
> >> We'll also want to avoid unnecessary compromises on QAPI schema
> >> expressiveness.  If we use null to mean "schema does not specify
> >> behavior when member is absent", we can't use it to mean "absent member
> >> behaves like the value null".  A bit of a blemish, but I think it's a
> >> tolerable one.
> >
> > If you want an example for an option that defaults to null, take the
> > backing option of BlockdevOptionsGenericCOWFormat.
> >
> > What is the reason for even considering limiting the expressiveness? Do
> > you think that an additional 'optional' bool, at least for those options
> > that don't have a default, would be so bad in the longhand form? Or
> > keeping '*' even in the longhand form, as suggested below.
> 
> Well, one reason is this:
> 
>     ##
>     # @SchemaInfoObjectMember:
>     #
>     # An object member.
>     #
>     # @name: the member's name, as defined in the QAPI schema.
>     #
>     # @type: the name of the member's type.
>     #
>     # @default: default when used as command parameter.
>     #           If absent, the parameter is mandatory.
>     #           If present, the value must be null.  The parameter is
>     #           optional, and behavior when it's missing is not specified
>     #           here.
>     #           Future extension: if present and non-null, the parameter
>     #           is optional, and defaults to this value.
>     #
>     # Since: 2.5
>     ##
> 
> If we want to be able to express the difference between "behavior when
> absent is not specified here" and "absent behaves like value null", then
> we need to somehow add that bit of information here.
> 
> Could use a feature.  Features are not yet implemented for members, but
> we need them anyway.

That definition wasn't a great idea, I'm afraid. :-(

But "default is QNull" is still acceptable behaviour for "not
specified" if the client doesn't need to know what the default is.

> >> >                 If that means that keeping '*' in the longhand form of
> >> > optional members (whether or not those members have a default value),
> >> > then so be it.
> >> 
> >> I believe both
> >> 
> >>     '*KEY': { 'type': ARG': 'default': null }
> >> 
> >> and
> >> 
> >>     'KEY': { 'type': ARG': 'default': null }
> >> 
> >> are viable longhand forms for '*KEY': 'ARG'.
> >> 
> >> I prefer the latter, but I'm open to arguments.
> >
> > If you go for the former, then you certainly want to use absent
> > 'default' to indicate no default, and allow a QNull default with
> > 'default': null.
> >
> > The only reason to abuse 'default': null for no default is that you
> > can't distinguish optional and non-optional if you use 'KEY' for both
> > instead of 'KEY' for mandatory and '*KEY' for optional.
> >
> > So while I understand and to some degree share your dislike for the '*'
> > prefix, I think I cast my pragmatic vote for:
> >
> > mandatory:                   'KEY':  { 'type': 'ARG' }
> > optional without a default:  '*KEY': { 'type': 'ARG' }
> > optional with QNull default: '*KEY': { 'type': 'ARG', 'default': null }
> 
> The last one could also be     'KEY': { 'type': 'ARG', 'default': null }
> without loss of expressiveness.
> 
> Differently ugly.

Not loss of expressiveness, but loss of consistency.

> Here's yet another idea.  For the "absent is not specified here" case,
> use
> 
>     'KEY': { 'type': 'ARG', optional: true }
>     '*KEY': 'ARG'
> 
> For the "absent defaults to DEFVAL" case, use
> 
>     'KEY': { 'type': 'ARG', optional: true, 'default': DEFVAL }
>     'KEY': { 'type': 'ARG', 'default': DEFVAL }

I assume this means: 'optional' defaults to true if 'default' is
present, and to false if 'default' is absent. (It's an example of a
default that can't be expressed in the schema.)

Works for me.

Kevin



^ permalink raw reply	[flat|nested] 41+ messages in thread

end of thread, other threads:[~2019-11-22 16:14 UTC | newest]

Thread overview: 41+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-24 17:39 [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 01/14] qapi: Parse numeric values Max Reitz
2019-11-14  9:15   ` Markus Armbruster
2019-11-14  9:50     ` Max Reitz
2019-11-14 12:01       ` Markus Armbruster
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 02/14] qapi: Move to_c_string() to common.py Max Reitz
2019-11-14  9:20   ` Markus Armbruster
2019-11-14  9:58     ` Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 03/14] qapi: Introduce default values for struct members Max Reitz
2019-11-14 15:53   ` Markus Armbruster
2019-11-21 15:07   ` Markus Armbruster
2019-11-21 15:24     ` Eric Blake
2019-11-21 19:46       ` Markus Armbruster
2019-11-21 19:56         ` Eric Blake
2019-11-22  7:29           ` Markus Armbruster
2019-11-22 10:25             ` Kevin Wolf
2019-11-22 14:40               ` Markus Armbruster
2019-11-22 16:12                 ` Kevin Wolf
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 04/14] qapi: Allow optional discriminators Max Reitz
2019-11-21 15:13   ` Markus Armbruster
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 05/14] qapi: Document default values for struct members Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 06/14] test-qapi: Print struct members' default values Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 07/14] tests: Test QAPI default values for struct members Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 08/14] tests: Add QAPI optional discriminator tests Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 09/14] qapi: Formalize qcow2 encryption probing Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 10/14] qapi: Formalize qcow " Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 11/14] block: Try to create well typed json:{} filenames Max Reitz
2019-06-24 20:06   ` Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 12/14] iotests: Test internal option typing Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 13/14] iotests: qcow2's encrypt.format is now optional Max Reitz
2019-06-24 17:39 ` [Qemu-devel] [PATCH v4 14/14] block: Make use of QAPI defaults Max Reitz
2019-06-24 18:35 ` [Qemu-devel] [PATCH v4 00/14] block: Try to create well-typed json:{} filenames no-reply
2019-06-24 19:04   ` Max Reitz
2019-06-24 19:06     ` Max Reitz
2019-06-24 19:00 ` no-reply
2019-06-24 20:06   ` Max Reitz
2019-08-19 16:49 ` Max Reitz
2019-09-13 11:49 ` Max Reitz
2019-11-06 13:01   ` Max Reitz
2019-11-14  8:54     ` Markus Armbruster
2019-11-21 15:17       ` Markus Armbruster

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).