* [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames @ 2018-05-09 16:55 Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions Max Reitz ` (12 more replies) 0 siblings, 13 replies; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth The “Try” in the subject of this series means more or less the same as the “Try” in the subject line of http://lists.nongnu.org/archive/html/qemu-block/2018-05/msg00061.html . The difference is that it’s not so important here, because the worst outcome is that in very narrow edge cases[1] (e.g. you trying to use password-secret for rbd) we fail to generate a properly typed json:{} filename -- but that is no regression over what we currently have. In the vast majority of cases[1], this series will result in proper typing. Or, well, I should add that I’m not quite sure about all the edge cases[1], solely because the current bdrv_refresh_filename() code is a mess, especially when it comes to gathering the blockdev options. But I do know that after my series to address that (“block: Fix some filename generation issues”), it only comes down to a very small number of edge cases (the only hard edge case is rbd’s =keyvalue-pairs, but that is kaput anyway).[1] But since this series does not have any hard requirement on that other one, I decided to send it independently. [1] You might have noticed a number of [1]s above. This is because the above text is as it was in a version of this series with only five patches in it. Then I noticed a not-so-edge case where converting to QAPI broke down: qcow(2) encryption. Since qcow and qcow2 now support both the old AES and luks encryption, and every encryption type may offer its own runtime options, you now have a flat union there (under “encrypt”) which is discriminated based on encrypt.format. Now the qcow2 driver can of course detect the format from the image header, so you don’t really have to specify it. In fact, when your options don’t go through QAPI, you actually don’t have to. So this is exactly the not-so-edge case I was talking about. QAPI cannot parse encrypt before it knows encrypt.format. So it’s much too late when the qcow2 driver detects encrypt.format in its .bdrv_open() plementation. I’m more than happy to take alternative suggestions, but this is an issue we really cannot ignore (unlike =keyvalue-pairs), so the two solutions I came up with are: (1) Say auto-detection was a bad idea and make encrypt.format mandatory, even without QAPI. This is always a possibility, but it does seem kind of cumbersome to users, so I wanted to explore the other option first, not knowing how deep into the rabbit hole that would lead. (2) Allow flat union discriminators to be optional. Since both AES and luks encryption in practice only allow a key-secret option, we can add a new runtime encryption pseudo-type, which is “from-image” that only allows such a key-secret option. Now, when the user does not specify encrypt.format, the format is assumed to be “from-image” which means that you can specify a key-secret and qcow/qcow2 will figure out what encryption to use. This allows to retain compatibility with the current interface, but it also means this series exploded, and maybe the idea is universally hated. One issue I can see with this approach is how it would fare with new encryption types added; specifically, what if we add some encryption type that does not have a key-secret but something else? Do we add that something else to from-image and make everything optional? Or do we require users to explicitly specify the encryption type for such a new type when it is not compatible with from-image? In any case, one thing we have to watch out for from now on is that drivers must not allow options to be optional that are mandatory in the QAPI schema. More generally, we have to make sure that the driver’s interface on -drive is compatible to QAPI. But I guess everybody knows that, so it’s good to be a bit more specific and say that this doesn’t just mean that options must match. Mandatoriness must match also. So, OK, obviously I chose solution (2) for now. What this means is that in this series: - The QAPI code generator is modified to allow optional discriminators for flat unions. In such a case, a default-variant must be supplied to choose when the discriminator value is not present on the wire. - Accordingly, documentation, tests, and introspection are adjusted. - This is used to make qcow’s and qcow2’s encrypt.format parameter optional. It now defaults to “from-image” which is a new pseudo-format that allows a key-secret to be given, and otherwise leaves it to the format driver to determine the encryption format. - qdict_stringify_for_keyval() is added, as already proposed/hinted at in the rbd QAPI discussion. This includes movement of many block-specific QDict function declarations into an own header, as suggested by Markus. - json:{} filenames are attempted to be typed correctly when they are generated, by running bs->full_open_options through a healthy mix of qdict_flatten(), qdict_stringify_for_keyval(), qdict_crumple(), the keyval input visitor for BlockdevOptions, and the output visitor. This may not always work but I hope it usually will. Fingers crossed. (At least it won’t make things worse.) - Tests, tests, tests. (Yes, I know that “In this series tests, tests, tests.” is not a sentence.) Max Reitz (13): qapi: Add default-variant for flat unions docs/qapi: Document optional discriminators tests: Add QAPI optional discriminator tests qapi: Formalize qcow2 encryption probing qapi: Formalize qcow encryption probing block: Add block-specific QDict header qdict: Add qdict_stringify_for_keyval() tests: Add qdict_stringify_for_keyval() test qdict: Make qdict_flatten() shallow-clone-friendly tests: Add QDict clone-flatten test block: Try to create well typed json:{} filenames iotests: Test internal option typing iotests: qcow2's encrypt.format is now optional docs/devel/qapi-code-gen.txt | 21 +++++- tests/Makefile.include | 3 + qapi/block-core.json | 72 ++++++++++++++++-- qapi/introspect.json | 8 ++ ...ion-optional-discriminator-invalid-default.json | 12 +++ ...at-union-optional-discriminator-no-default.json | 11 +++ .../flat-union-optional-discriminator.json | 4 +- .../flat-union-superfluous-default-variant.json | 11 +++ include/block/qdict.h | 37 +++++++++ include/qapi/qmp/qdict.h | 17 ----- block.c | 70 ++++++++++++++++- block/gluster.c | 1 + block/iscsi.c | 1 + block/nbd.c | 1 + block/nfs.c | 1 + block/parallels.c | 1 + block/qcow.c | 1 + block/qcow2.c | 1 + block/qed.c | 1 + block/quorum.c | 1 + block/rbd.c | 1 + block/sheepdog.c | 1 + block/snapshot.c | 1 + block/ssh.c | 1 + block/vhdx.c | 1 + block/vpc.c | 1 + block/vvfat.c | 1 + block/vxhs.c | 1 + blockdev.c | 1 + qobject/qdict.c | 81 ++++++++++++++++++-- tests/check-qdict.c | 88 ++++++++++++++++++++++ tests/check-qobject.c | 1 + tests/test-replication.c | 1 + util/qemu-config.c | 1 + scripts/qapi/common.py | 57 +++++++++++--- scripts/qapi/doc.py | 8 +- scripts/qapi/introspect.py | 10 ++- scripts/qapi/visit.py | 33 +++++++- ...nion-optional-discriminator-invalid-default.err | 1 + ...ion-optional-discriminator-invalid-default.exit | 1 + ...nion-optional-discriminator-invalid-default.out | 0 ...lat-union-optional-discriminator-no-default.err | 1 + ...at-union-optional-discriminator-no-default.exit | 1 + ...lat-union-optional-discriminator-no-default.out | 0 .../flat-union-optional-discriminator.err | 1 - .../flat-union-optional-discriminator.exit | 2 +- .../flat-union-optional-discriminator.out | 15 ++++ .../flat-union-superfluous-default-variant.err | 1 + .../flat-union-superfluous-default-variant.exit | 1 + .../flat-union-superfluous-default-variant.out | 0 tests/qapi-schema/test-qapi.py | 2 + tests/qemu-iotests/059.out | 2 +- tests/qemu-iotests/087 | 2 - tests/qemu-iotests/089 | 25 ++++++ tests/qemu-iotests/089.out | 9 +++ tests/qemu-iotests/099.out | 4 +- tests/qemu-iotests/110.out | 2 +- tests/qemu-iotests/198.out | 4 +- tests/qemu-iotests/207.out | 10 +-- 59 files changed, 580 insertions(+), 68 deletions(-) create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.json create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.json create mode 100644 include/block/qdict.h create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.err create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.exit create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.out -- 2.14.3 ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-10 13:12 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 02/13] docs/qapi: Document optional discriminators Max Reitz ` (11 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth This patch allows specifying a discriminator that is an optional member of the base struct. In such a case, a default value must be provided that is used when no value is given. Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/introspect.json | 8 ++++++ scripts/qapi/common.py | 57 ++++++++++++++++++++++++++++++++++-------- scripts/qapi/doc.py | 8 ++++-- scripts/qapi/introspect.py | 10 +++++--- scripts/qapi/visit.py | 33 ++++++++++++++++++++++-- tests/qapi-schema/test-qapi.py | 2 ++ 6 files changed, 101 insertions(+), 17 deletions(-) diff --git a/qapi/introspect.json b/qapi/introspect.json index c7f67b7d78..2d7b1e3745 100644 --- a/qapi/introspect.json +++ b/qapi/introspect.json @@ -168,6 +168,13 @@ # @tag: the name of the member serving as type tag. # An element of @members with this name must exist. # +# @default-variant: if the @tag element of @members is optional, this +# is the default value for choosing a variant. Its +# value must be a valid value for @tag. +# Present exactly when @tag is present and the +# associated element of @members is optional. +# (Since: 2.13) +# # @variants: variant members, i.e. additional members that # depend on the type tag's value. Present exactly when # @tag is present. The variants are in no particular order, @@ -181,6 +188,7 @@ { 'struct': 'SchemaInfoObject', 'data': { 'members': [ 'SchemaInfoObjectMember' ], '*tag': 'str', + '*default-variant': 'str', '*variants': [ 'SchemaInfoObjectVariant' ] } } ## diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py index a032cec375..fbf0244f73 100644 --- a/scripts/qapi/common.py +++ b/scripts/qapi/common.py @@ -721,6 +721,7 @@ def check_union(expr, info): name = expr['union'] base = expr.get('base') discriminator = expr.get('discriminator') + default_variant = expr.get('default-variant') members = expr['data'] # Two types of unions, determined by discriminator. @@ -745,16 +746,37 @@ def check_union(expr, info): base_members = find_base_members(base) assert base_members is not None - # The value of member 'discriminator' must name a non-optional - # member of the base struct. + # The value of member 'discriminator' must name a member of + # the base struct. check_name(info, "Discriminator of flat union '%s'" % name, discriminator) - discriminator_type = base_members.get(discriminator) - if not discriminator_type: - raise QAPISemError(info, - "Discriminator '%s' is not a member of base " - "struct '%s'" - % (discriminator, base)) + if default_variant is None: + discriminator_type = base_members.get(discriminator) + if not discriminator_type: + if base_members.get('*' + discriminator) is None: + raise QAPISemError(info, + "Discriminator '%s' is not a member of " + "base struct '%s'" + % (discriminator, base)) + else: + raise QAPISemError(info, + "Default variant must be specified for " + "optional discriminator '%s'" + % discriminator) + else: + discriminator_type = base_members.get('*' + discriminator) + if not discriminator_type: + if base_members.get(discriminator) is None: + raise QAPISemError(info, + "Discriminator '%s' is not a member of " + "base struct '%s'" + % (discriminator, base)) + else: + raise QAPISemError(info, + "Must not specify a default variant for " + "non-optional discriminator '%s'" + % discriminator) + enum_define = enum_types.get(discriminator_type) allow_metas = ['struct'] # Do not allow string discriminator @@ -763,6 +785,15 @@ def check_union(expr, info): "Discriminator '%s' must be of enumeration " "type" % discriminator) + if default_variant is not None: + # Must be a value of the enumeration + if default_variant not in enum_define['data']: + raise QAPISemError(info, + "Default variant '%s' of flat union '%s' is " + "not part of '%s'" + % (default_variant, name, + discriminator_type)) + # Check every branch; don't allow an empty union if len(members) == 0: raise QAPISemError(info, "Union '%s' cannot have empty 'data'" % name) @@ -909,7 +940,7 @@ def check_exprs(exprs): elif 'union' in expr: meta = 'union' check_keys(expr_elem, 'union', ['data'], - ['base', 'discriminator']) + ['base', 'discriminator', 'default-variant']) union_types[expr[meta]] = expr elif 'alternate' in expr: meta = 'alternate' @@ -1335,12 +1366,14 @@ class QAPISchemaObjectTypeMember(QAPISchemaMember): class QAPISchemaObjectTypeVariants(object): - def __init__(self, tag_name, tag_member, variants): + def __init__(self, tag_name, tag_member, default_tag_value, variants): # Flat unions pass tag_name but not tag_member. # Simple unions and alternates pass tag_member but not tag_name. # After check(), tag_member is always set, and tag_name remains # a reliable witness of being used by a flat union. assert bool(tag_member) != bool(tag_name) + # default_tag_value is only passed for flat unions. + assert bool(tag_name) or not bool(default_tag_value) assert (isinstance(tag_name, str) or isinstance(tag_member, QAPISchemaObjectTypeMember)) assert len(variants) > 0 @@ -1348,6 +1381,7 @@ class QAPISchemaObjectTypeVariants(object): assert isinstance(v, QAPISchemaObjectTypeVariant) self._tag_name = tag_name self.tag_member = tag_member + self.default_tag_value = default_tag_value self.variants = variants def set_owner(self, name): @@ -1637,6 +1671,7 @@ class QAPISchema(object): data = expr['data'] base = expr.get('base') tag_name = expr.get('discriminator') + default_tag_value = expr.get('default-variant') tag_member = None if isinstance(base, dict): base = (self._make_implicit_object_type( @@ -1656,6 +1691,7 @@ class QAPISchema(object): QAPISchemaObjectType(name, info, doc, base, members, QAPISchemaObjectTypeVariants(tag_name, tag_member, + default_tag_value, variants))) def _def_alternate_type(self, expr, info, doc): @@ -1668,6 +1704,7 @@ class QAPISchema(object): QAPISchemaAlternateType(name, info, doc, QAPISchemaObjectTypeVariants(None, tag_member, + None, variants))) def _def_command(self, expr, info, doc): diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py index 9b312b2c51..91204dc4c6 100644 --- a/scripts/qapi/doc.py +++ b/scripts/qapi/doc.py @@ -160,8 +160,12 @@ def texi_members(doc, what, base, variants, member_func): items += '@item The members of @code{%s}\n' % base.doc_type() if variants: for v in variants.variants: - when = ' when @code{%s} is @t{"%s"}' % ( - variants.tag_member.name, v.name) + if v.name == variants.default_tag_value: + when = ' when @code{%s} is @t{"%s"} or not given' % ( + variants.tag_member.name, v.name) + else: + when = ' when @code{%s} is @t{"%s"}' % ( + variants.tag_member.name, v.name) if v.type.is_implicit(): assert not v.type.base and not v.type.variants for m in v.type.local_members: diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index f9e67e8227..2d1d4e320a 100644 --- a/scripts/qapi/introspect.py +++ b/scripts/qapi/introspect.py @@ -142,9 +142,12 @@ const QLitObject %(c_name)s = %(c_string)s; ret['default'] = None return ret - def _gen_variants(self, tag_name, variants): - return {'tag': tag_name, - 'variants': [self._gen_variant(v) for v in variants]} + def _gen_variants(self, tag_name, default_variant, variants): + ret = {'tag': tag_name, + 'variants': [self._gen_variant(v) for v in variants]} + if default_variant: + ret['default-variant'] = default_variant + return ret def _gen_variant(self, variant): return {'case': variant.name, 'type': self._use_type(variant.type)} @@ -163,6 +166,7 @@ const QLitObject %(c_name)s = %(c_string)s; obj = {'members': [self._gen_member(m) for m in members]} if variants: obj.update(self._gen_variants(variants.tag_member.name, + variants.default_tag_value, variants.variants)) self._gen_qlit(name, 'object', obj) diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py index 5d72d8936c..ecffc46bd3 100644 --- a/scripts/qapi/visit.py +++ b/scripts/qapi/visit.py @@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, variants): void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp) { Error *err = NULL; - ''', c_name=c_name(name)) + if variants: + ret += mcgen(''' + %(c_type)s %(c_name)s; +''', + c_type=variants.tag_member.type.c_name(), + c_name=c_name(variants.tag_member.name)) + + ret += mcgen(''' + +''') + if base: ret += mcgen(''' visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, &err); @@ -75,8 +85,27 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp) ''') if variants: + if variants.default_tag_value is None: + ret += mcgen(''' + %(c_name)s = obj->%(c_name)s; +''', + c_name=c_name(variants.tag_member.name)) + else: + ret += mcgen(''' + if (obj->has_%(c_name)s) { + %(c_name)s = obj->%(c_name)s; + } else { + %(c_name)s = %(enum_const)s; + } +''', + c_name=c_name(variants.tag_member.name), + enum_const=c_enum_const( + variants.tag_member.type.name, + variants.default_tag_value, + variants.tag_member.type.prefix)) + ret += mcgen(''' - switch (obj->%(c_name)s) { + switch (%(c_name)s) { ''', c_name=c_name(variants.tag_member.name)) diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py index c1a144ba29..f2a072b92e 100644 --- a/tests/qapi-schema/test-qapi.py +++ b/tests/qapi-schema/test-qapi.py @@ -56,6 +56,8 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor): def _print_variants(variants): if variants: print(' tag %s' % variants.tag_member.name) + if variants.default_tag_value: + print(' default variant: %s' % variants.default_tag_value) for v in variants.variants: print(' case %s: %s' % (v.name, v.type.name)) -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions 2018-05-09 16:55 ` [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions Max Reitz @ 2018-05-10 13:12 ` Eric Blake 2018-05-10 13:18 ` Eric Blake 2018-05-11 17:38 ` Max Reitz 0 siblings, 2 replies; 42+ messages in thread From: Eric Blake @ 2018-05-10 13:12 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > This patch allows specifying a discriminator that is an optional member > of the base struct. In such a case, a default value must be provided > that is used when no value is given. > > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > qapi/introspect.json | 8 ++++++ > scripts/qapi/common.py | 57 ++++++++++++++++++++++++++++++++++-------- > scripts/qapi/doc.py | 8 ++++-- > scripts/qapi/introspect.py | 10 +++++--- > scripts/qapi/visit.py | 33 ++++++++++++++++++++++-- > tests/qapi-schema/test-qapi.py | 2 ++ > 6 files changed, 101 insertions(+), 17 deletions(-) I've been threatening that we might need this for some time, so I'm glad to see it being implemented. We'll see if the tests in 2 and 3 cover the code added here. > > diff --git a/qapi/introspect.json b/qapi/introspect.json > index c7f67b7d78..2d7b1e3745 100644 > --- a/qapi/introspect.json > +++ b/qapi/introspect.json > @@ -168,6 +168,13 @@ > # @tag: the name of the member serving as type tag. > # An element of @members with this name must exist. > # > +# @default-variant: if the @tag element of @members is optional, this > +# is the default value for choosing a variant. Its > +# value must be a valid value for @tag. Perhaps s/must/will/ as this struct is used for output (and therefore we always meet the condition, rather than the user having to do something correctly). Nice that you remembered introspection. > +# Present exactly when @tag is present and the > +# associated element of @members is optional. > +# (Since: 2.13) > +# > # @variants: variant members, i.e. additional members that > # depend on the type tag's value. Present exactly when > # @tag is present. The variants are in no particular order, > @@ -181,6 +188,7 @@ > { 'struct': 'SchemaInfoObject', > 'data': { 'members': [ 'SchemaInfoObjectMember' ], > '*tag': 'str', > + '*default-variant': 'str', > '*variants': [ 'SchemaInfoObjectVariant' ] } } > > ## > diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py > index a032cec375..fbf0244f73 100644 > --- a/scripts/qapi/common.py > +++ b/scripts/qapi/common.py > @@ -721,6 +721,7 @@ def check_union(expr, info): > name = expr['union'] > base = expr.get('base') > discriminator = expr.get('discriminator') > + default_variant = expr.get('default-variant') > members = expr['data'] > > # Two types of unions, determined by discriminator. > @@ -745,16 +746,37 @@ def check_union(expr, info): > base_members = find_base_members(base) > assert base_members is not None > > - # The value of member 'discriminator' must name a non-optional > - # member of the base struct. > + # The value of member 'discriminator' must name a member of > + # the base struct. > check_name(info, "Discriminator of flat union '%s'" % name, > discriminator) > - discriminator_type = base_members.get(discriminator) > - if not discriminator_type: > - raise QAPISemError(info, > - "Discriminator '%s' is not a member of base " > - "struct '%s'" > - % (discriminator, base)) > + if default_variant is None: > + discriminator_type = base_members.get(discriminator) > + if not discriminator_type: > + if base_members.get('*' + discriminator) is None: Unrelated to your patch, but this reminds me - we already had a question on list about whether we should allow ANY member of the base struct, rather than the members directly declared in the struct. (The use case was that if you have a grandparent struct, then an intermediate base struct, the code as written does not permit the discriminator to come from the grandparent struct, which is awkward - but if I also recall correctly, the question came up in relation to the query-cpus-fast command, where we ended up not refactoring things after all due to deprecation of query-cpus). > + raise QAPISemError(info, > + "Discriminator '%s' is not a member of " > + "base struct '%s'" > + % (discriminator, base)) > + else: > + raise QAPISemError(info, > + "Default variant must be specified for " > + "optional discriminator '%s'" > + % discriminator) > + else: > + discriminator_type = base_members.get('*' + discriminator) > + if not discriminator_type: > + if base_members.get(discriminator) is None: > + raise QAPISemError(info, > + "Discriminator '%s' is not a member of " > + "base struct '%s'" > + % (discriminator, base)) > + else: > + raise QAPISemError(info, > + "Must not specify a default variant for " > + "non-optional discriminator '%s'" > + % discriminator) > + Looks like you've got good error messages for all the cases of enforcing the rule that default-variant and optional member must be used together. > enum_define = enum_types.get(discriminator_type) > allow_metas = ['struct'] > # Do not allow string discriminator > @@ -763,6 +785,15 @@ def check_union(expr, info): > "Discriminator '%s' must be of enumeration " > "type" % discriminator) > > + if default_variant is not None: > + # Must be a value of the enumeration > + if default_variant not in enum_define['data']: > + raise QAPISemError(info, > + "Default variant '%s' of flat union '%s' is " > + "not part of '%s'" > + % (default_variant, name, > + discriminator_type)) > + > # Check every branch; don't allow an empty union > if len(members) == 0: > raise QAPISemError(info, "Union '%s' cannot have empty 'data'" % name) > @@ -909,7 +940,7 @@ def check_exprs(exprs): > elif 'union' in expr: > meta = 'union' > check_keys(expr_elem, 'union', ['data'], > - ['base', 'discriminator']) > + ['base', 'discriminator', 'default-variant']) > union_types[expr[meta]] = expr > elif 'alternate' in expr: > meta = 'alternate' > @@ -1335,12 +1366,14 @@ class QAPISchemaObjectTypeMember(QAPISchemaMember): > > > class QAPISchemaObjectTypeVariants(object): > - def __init__(self, tag_name, tag_member, variants): > + def __init__(self, tag_name, tag_member, default_tag_value, variants): > # Flat unions pass tag_name but not tag_member. > # Simple unions and alternates pass tag_member but not tag_name. > # After check(), tag_member is always set, and tag_name remains > # a reliable witness of being used by a flat union. > assert bool(tag_member) != bool(tag_name) > + # default_tag_value is only passed for flat unions. > + assert bool(tag_name) or not bool(default_tag_value) Looks correct. > assert (isinstance(tag_name, str) or > isinstance(tag_member, QAPISchemaObjectTypeMember)) > assert len(variants) > 0 > @@ -1348,6 +1381,7 @@ class QAPISchemaObjectTypeVariants(object): > assert isinstance(v, QAPISchemaObjectTypeVariant) > self._tag_name = tag_name > self.tag_member = tag_member > + self.default_tag_value = default_tag_value > self.variants = variants > > def set_owner(self, name): > @@ -1637,6 +1671,7 @@ class QAPISchema(object): > data = expr['data'] > base = expr.get('base') > tag_name = expr.get('discriminator') > + default_tag_value = expr.get('default-variant') > tag_member = None > if isinstance(base, dict): > base = (self._make_implicit_object_type( > @@ -1656,6 +1691,7 @@ class QAPISchema(object): > QAPISchemaObjectType(name, info, doc, base, members, > QAPISchemaObjectTypeVariants(tag_name, > tag_member, > + default_tag_value, > variants))) > > def _def_alternate_type(self, expr, info, doc): > @@ -1668,6 +1704,7 @@ class QAPISchema(object): > QAPISchemaAlternateType(name, info, doc, > QAPISchemaObjectTypeVariants(None, > tag_member, > + None, > variants))) > > def _def_command(self, expr, info, doc): > diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py > index 9b312b2c51..91204dc4c6 100644 > --- a/scripts/qapi/doc.py > +++ b/scripts/qapi/doc.py > @@ -160,8 +160,12 @@ def texi_members(doc, what, base, variants, member_func): > items += '@item The members of @code{%s}\n' % base.doc_type() > if variants: > for v in variants.variants: > - when = ' when @code{%s} is @t{"%s"}' % ( > - variants.tag_member.name, v.name) > + if v.name == variants.default_tag_value: > + when = ' when @code{%s} is @t{"%s"} or not given' % ( > + variants.tag_member.name, v.name) Nice, you got the generated docs as well as introspection. > + else: > + when = ' when @code{%s} is @t{"%s"}' % ( > + variants.tag_member.name, v.name) > if v.type.is_implicit(): > assert not v.type.base and not v.type.variants > for m in v.type.local_members: > diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py > index f9e67e8227..2d1d4e320a 100644 > --- a/scripts/qapi/introspect.py > +++ b/scripts/qapi/introspect.py > @@ -142,9 +142,12 @@ const QLitObject %(c_name)s = %(c_string)s; > ret['default'] = None > return ret > > - def _gen_variants(self, tag_name, variants): > - return {'tag': tag_name, > - 'variants': [self._gen_variant(v) for v in variants]} > + def _gen_variants(self, tag_name, default_variant, variants): > + ret = {'tag': tag_name, > + 'variants': [self._gen_variant(v) for v in variants]} > + if default_variant: > + ret['default-variant'] = default_variant > + return ret > > def _gen_variant(self, variant): > return {'case': variant.name, 'type': self._use_type(variant.type)} > @@ -163,6 +166,7 @@ const QLitObject %(c_name)s = %(c_string)s; > obj = {'members': [self._gen_member(m) for m in members]} > if variants: > obj.update(self._gen_variants(variants.tag_member.name, > + variants.default_tag_value, > variants.variants)) > self._gen_qlit(name, 'object', obj) > > diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py > index 5d72d8936c..ecffc46bd3 100644 > --- a/scripts/qapi/visit.py > +++ b/scripts/qapi/visit.py > @@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, variants): > void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp) > { > Error *err = NULL; > - > ''', > c_name=c_name(name)) > > + if variants: > + ret += mcgen(''' > + %(c_type)s %(c_name)s; > +''', > + c_type=variants.tag_member.type.c_name(), > + c_name=c_name(variants.tag_member.name)) > + > + ret += mcgen(''' > + > +''') > + Creating a temp variable makes it easier to handle the default... > if base: > ret += mcgen(''' > visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, &err); > @@ -75,8 +85,27 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp) > ''') > > if variants: > + if variants.default_tag_value is None: > + ret += mcgen(''' > + %(c_name)s = obj->%(c_name)s; > +''', > + c_name=c_name(variants.tag_member.name)) > + else: > + ret += mcgen(''' > + if (obj->has_%(c_name)s) { > + %(c_name)s = obj->%(c_name)s; > + } else { > + %(c_name)s = %(enum_const)s; > + } > +''', > + c_name=c_name(variants.tag_member.name), > + enum_const=c_enum_const( > + variants.tag_member.type.name, > + variants.default_tag_value, > + variants.tag_member.type.prefix)) > + > ret += mcgen(''' > - switch (obj->%(c_name)s) { > + switch (%(c_name)s) { ...compared to the old code that just inlined the one thing that used to be assigned to what is now the temporary variable. It might be possible to inline things so that the generated code reads either: switch (obj->discriminator) { switch (!obj->has_discriminator ? DEFAULT : obj->discriminator) { but I don't think it's worth the effort; and the temporary variable is fine even though it makes the generated file bigger. > ''', > c_name=c_name(variants.tag_member.name)) > > diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py > index c1a144ba29..f2a072b92e 100644 > --- a/tests/qapi-schema/test-qapi.py > +++ b/tests/qapi-schema/test-qapi.py > @@ -56,6 +56,8 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor): > def _print_variants(variants): > if variants: > print(' tag %s' % variants.tag_member.name) > + if variants.default_tag_value: > + print(' default variant: %s' % variants.default_tag_value) > for v in variants.variants: > print(' case %s: %s' % (v.name, v.type.name)) > > Looks good! Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions 2018-05-10 13:12 ` Eric Blake @ 2018-05-10 13:18 ` Eric Blake 2018-05-11 17:59 ` Max Reitz 2018-05-11 17:38 ` Max Reitz 1 sibling, 1 reply; 42+ messages in thread From: Eric Blake @ 2018-05-10 13:18 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/10/2018 08:12 AM, Eric Blake wrote: Oh, I just had a thought: >> +++ b/scripts/qapi/visit.py >> @@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, >> if variants: >> + if variants.default_tag_value is None: >> + ret += mcgen(''' >> + %(c_name)s = obj->%(c_name)s; >> +''', >> + c_name=c_name(variants.tag_member.name)) >> + else: >> + ret += mcgen(''' >> + if (obj->has_%(c_name)s) { >> + %(c_name)s = obj->%(c_name)s; >> + } else { >> + %(c_name)s = %(enum_const)s; In this branch of code, is it worth also generating: %has_(c_name)s = true; That way, the rest of the C code does not have to check has_discriminator, because the process of assigning the default will ensure that has_discriminator is always true later on. It does have the effect that output would never omit the discriminator - but that might be a good thing: if we ever have an output union that used to have a mandatory discriminator and want to now make it optional, we don't want to break older clients that expected the discriminator to be present. It does obscure whether input relied on the default, but I don't think that hurts. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions 2018-05-10 13:18 ` Eric Blake @ 2018-05-11 17:59 ` Max Reitz 2018-05-11 18:13 ` Eric Blake 0 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-11 17:59 UTC (permalink / raw) To: Eric Blake, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth [-- Attachment #1: Type: text/plain, Size: 2176 bytes --] On 2018-05-10 15:18, Eric Blake wrote: > On 05/10/2018 08:12 AM, Eric Blake wrote: > > Oh, I just had a thought: > >>> +++ b/scripts/qapi/visit.py >>> @@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, > >>> if variants: >>> + if variants.default_tag_value is None: >>> + ret += mcgen(''' >>> + %(c_name)s = obj->%(c_name)s; >>> +''', >>> + c_name=c_name(variants.tag_member.name)) >>> + else: >>> + ret += mcgen(''' >>> + if (obj->has_%(c_name)s) { >>> + %(c_name)s = obj->%(c_name)s; >>> + } else { >>> + %(c_name)s = %(enum_const)s; > > In this branch of code, is it worth also generating: > > %has_(c_name)s = true; You mean obj->has_%(c_name)s, and then also set obj->%(c_name)s? > That way, the rest of the C code does not have to check > has_discriminator, because the process of assigning the default will > ensure that has_discriminator is always true later on. It does have the > effect that output would never omit the discriminator - but that might > be a good thing: if we ever have an output union that used to have a > mandatory discriminator and want to now make it optional, we don't want > to break older clients that expected the discriminator to be present. It > does obscure whether input relied on the default, but I don't think that > hurts. Also, it would make code a bit simpler because it can cover the !has_X case under X == default_X. But OTOH, you could make that case for any optional value -- except where "is missing" has special value. And that's the tricky part: I think it's hard to say that a missing discriminator never has special meaning. We only need the default-variant so we know which variant of the union to pick; but we don't know that the fact that the discriminator value was missing does not have special meaning. Of course, we can simply foreclose that by setting it here. I don't have money in this game, so I suppose it's yours and Markus's call. :-) Max [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions 2018-05-11 17:59 ` Max Reitz @ 2018-05-11 18:13 ` Eric Blake 0 siblings, 0 replies; 42+ messages in thread From: Eric Blake @ 2018-05-11 18:13 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/11/2018 12:59 PM, Max Reitz wrote: > On 2018-05-10 15:18, Eric Blake wrote: >> On 05/10/2018 08:12 AM, Eric Blake wrote: >> >> Oh, I just had a thought: >> >>>> +++ b/scripts/qapi/visit.py >>>> @@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, >> >>>> if variants: >>>> + if variants.default_tag_value is None: >>>> + ret += mcgen(''' >>>> + %(c_name)s = obj->%(c_name)s; >>>> +''', >>>> + c_name=c_name(variants.tag_member.name)) >>>> + else: >>>> + ret += mcgen(''' >>>> + if (obj->has_%(c_name)s) { >>>> + %(c_name)s = obj->%(c_name)s; >>>> + } else { >>>> + %(c_name)s = %(enum_const)s; >> >> In this branch of code, is it worth also generating: >> >> %has_(c_name)s = true; > > You mean obj->has_%(c_name)s, and then also set obj->%(c_name)s? Umm, yeah ;) In fact, I think it is as simple as: if variants: if variants.default_tag_value: ret += mcgen(''' if (!obj->has_%(c_name)s) { obj->has_%(c_name)s = true; obj->%(c_name)s = %(enum_const)s; } ''') ret += mcgen(''' switch (obj->%(c_name)s) { ... and you are back to not needing a temporary variable. > >> That way, the rest of the C code does not have to check >> has_discriminator, because the process of assigning the default will >> ensure that has_discriminator is always true later on. It does have the >> effect that output would never omit the discriminator - but that might >> be a good thing: if we ever have an output union that used to have a >> mandatory discriminator and want to now make it optional, we don't want >> to break older clients that expected the discriminator to be present. It >> does obscure whether input relied on the default, but I don't think that >> hurts. > > Also, it would make code a bit simpler because it can cover the !has_X > case under X == default_X. > > But OTOH, you could make that case for any optional value -- except > where "is missing" has special value. My preference - special-casing "is missing" is prone to abuse. I don't want to support that if we can at all avoid it. The sane semantics is that the default is populated as soon as we detect that something is missing, and whether the user relies on the default by leaving the discriminator absent, or explicitly supplies the discriminator set to the default, the behavior should always be the same. > > And that's the tricky part: I think it's hard to say that a missing > discriminator never has special meaning. It won't, if we decide right now that we don't want to let it :) > We only need the > default-variant so we know which variant of the union to pick; but we > don't know that the fact that the discriminator value was missing does > not have special meaning. > > Of course, we can simply foreclose that by setting it here. And that's the way I'm leaning. > > I don't have money in this game, so I suppose it's yours and Markus's > call. :-) Markus, what's your preference? -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions 2018-05-10 13:12 ` Eric Blake 2018-05-10 13:18 ` Eric Blake @ 2018-05-11 17:38 ` Max Reitz 1 sibling, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-11 17:38 UTC (permalink / raw) To: Eric Blake, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth [-- Attachment #1: Type: text/plain, Size: 6368 bytes --] On 2018-05-10 15:12, Eric Blake wrote: > On 05/09/2018 11:55 AM, Max Reitz wrote: >> This patch allows specifying a discriminator that is an optional member >> of the base struct. In such a case, a default value must be provided >> that is used when no value is given. >> >> Signed-off-by: Max Reitz <mreitz@redhat.com> >> --- >> qapi/introspect.json | 8 ++++++ >> scripts/qapi/common.py | 57 >> ++++++++++++++++++++++++++++++++++-------- >> scripts/qapi/doc.py | 8 ++++-- >> scripts/qapi/introspect.py | 10 +++++--- >> scripts/qapi/visit.py | 33 ++++++++++++++++++++++-- >> tests/qapi-schema/test-qapi.py | 2 ++ >> 6 files changed, 101 insertions(+), 17 deletions(-) > > I've been threatening that we might need this for some time, so I'm glad > to see it being implemented. We'll see if the tests in 2 and 3 cover > the code added here. > >> >> diff --git a/qapi/introspect.json b/qapi/introspect.json >> index c7f67b7d78..2d7b1e3745 100644 >> --- a/qapi/introspect.json >> +++ b/qapi/introspect.json >> @@ -168,6 +168,13 @@ >> # @tag: the name of the member serving as type tag. >> # An element of @members with this name must exist. >> # >> +# @default-variant: if the @tag element of @members is optional, this >> +# is the default value for choosing a variant. Its >> +# value must be a valid value for @tag. > > Perhaps s/must/will/ as this struct is used for output (and therefore we > always meet the condition, rather than the user having to do something > correctly). I mostly copied from the other descriptions which seemed to prefer "must", but I'm happy with either. > Nice that you remembered introspection. I didn't, because I had no idea how introspection works exactly before this series. But one of the tests broke, thus telling me I might have forgotten something. :-) >> +# Present exactly when @tag is present and the >> +# associated element of @members is optional. >> +# (Since: 2.13) >> +# >> # @variants: variant members, i.e. additional members that >> # depend on the type tag's value. Present exactly when >> # @tag is present. The variants are in no particular order, [...] >> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py >> index 5d72d8936c..ecffc46bd3 100644 >> --- a/scripts/qapi/visit.py >> +++ b/scripts/qapi/visit.py >> @@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, >> variants): >> void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, >> Error **errp) >> { >> Error *err = NULL; >> - >> ''', >> c_name=c_name(name)) >> + if variants: >> + ret += mcgen(''' >> + %(c_type)s %(c_name)s; >> +''', >> + c_type=variants.tag_member.type.c_name(), >> + c_name=c_name(variants.tag_member.name)) >> + >> + ret += mcgen(''' >> + >> +''') >> + > > Creating a temp variable makes it easier to handle the default... > >> if base: >> ret += mcgen(''' >> visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, &err); >> @@ -75,8 +85,27 @@ void visit_type_%(c_name)s_members(Visitor *v, >> %(c_name)s *obj, Error **errp) >> ''') >> if variants: >> + if variants.default_tag_value is None: >> + ret += mcgen(''' >> + %(c_name)s = obj->%(c_name)s; >> +''', >> + c_name=c_name(variants.tag_member.name)) >> + else: >> + ret += mcgen(''' >> + if (obj->has_%(c_name)s) { >> + %(c_name)s = obj->%(c_name)s; >> + } else { >> + %(c_name)s = %(enum_const)s; >> + } >> +''', >> + c_name=c_name(variants.tag_member.name), >> + enum_const=c_enum_const( >> + variants.tag_member.type.name, >> + variants.default_tag_value, >> + variants.tag_member.type.prefix)) >> + >> ret += mcgen(''' >> - switch (obj->%(c_name)s) { >> + switch (%(c_name)s) { > > ...compared to the old code that just inlined the one thing that used to > be assigned to what is now the temporary variable. > > It might be possible to inline things so that the generated code reads > either: > > switch (obj->discriminator) { > switch (!obj->has_discriminator ? DEFAULT : obj->discriminator) { > > but I don't think it's worth the effort; and the temporary variable is > fine even though it makes the generated file bigger. I don't really mind either way, but depending on the default value and the discriminator name, using ?: may lead to a rather long line. >> ''', >> c_name=c_name(variants.tag_member.name)) >> diff --git a/tests/qapi-schema/test-qapi.py >> b/tests/qapi-schema/test-qapi.py >> index c1a144ba29..f2a072b92e 100644 >> --- a/tests/qapi-schema/test-qapi.py >> +++ b/tests/qapi-schema/test-qapi.py >> @@ -56,6 +56,8 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor): >> def _print_variants(variants): >> if variants: >> print(' tag %s' % variants.tag_member.name) >> + if variants.default_tag_value: >> + print(' default variant: %s' % >> variants.default_tag_value) >> for v in variants.variants: >> print(' case %s: %s' % (v.name, v.type.name)) >> > > Looks good! > Reviewed-by: Eric Blake <eblake@redhat.com> Phew. :-) Thanks! Max [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 02/13] docs/qapi: Document optional discriminators 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-10 13:14 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests Max Reitz ` (10 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth Signed-off-by: Max Reitz <mreitz@redhat.com> --- docs/devel/qapi-code-gen.txt | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt index b9b6eabd08..35b9f0d99d 100644 --- a/docs/devel/qapi-code-gen.txt +++ b/docs/devel/qapi-code-gen.txt @@ -472,8 +472,8 @@ All branches of the union must be complex types, and the top-level members of the union dictionary on the wire will be combination of members from both the base type and the appropriate branch type (when merging two dictionaries, there must be no keys in common). The -'discriminator' member must be the name of a non-optional enum-typed -member of the base struct. +'discriminator' member must be the name of an enum-typed member of the +base struct. The following example enhances the above simple union example by adding an optional common member 'read-only', renaming the @@ -502,6 +502,23 @@ the enum). In the resulting generated C data types, a flat union is represented as a struct with the base members included directly, and then a union of structures for each branch of the struct. +If the discriminator points to an optional member of the base struct, +its default value must be specified as a 'default-variant'. In the +following example, the above BlockDriver struct is changed so it +defaults to the 'file' driver if that field is omitted on the wire: + + { 'union': 'BlockdevOptions', + 'base': { '*driver': 'BlockdevDriver', '*read-only': 'bool' }, + 'discriminator': 'driver', + 'default-variant': 'file', + 'data': { 'file': 'BlockdevOptionsFile', + 'qcow2': 'BlockdevOptionsQcow2' } } + +Now the 'file' JSON object can be abbreviated to: + + { "read-only": "true", + "filename": "/some/place/my-image" } + A simple union can always be re-written as a flat union where the base class has a single member named 'type', and where each branch of the union has a struct with a single member named 'data'. That is, -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 02/13] docs/qapi: Document optional discriminators 2018-05-09 16:55 ` [Qemu-devel] [PATCH 02/13] docs/qapi: Document optional discriminators Max Reitz @ 2018-05-10 13:14 ` Eric Blake 0 siblings, 0 replies; 42+ messages in thread From: Eric Blake @ 2018-05-10 13:14 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > docs/devel/qapi-code-gen.txt | 21 +++++++++++++++++++-- > 1 file changed, 19 insertions(+), 2 deletions(-) > I might have squashed this in with patch 1, but I can live with it separate. > @@ -502,6 +502,23 @@ the enum). In the resulting generated C data types, a flat union is > represented as a struct with the base members included directly, and > then a union of structures for each branch of the struct. > > +If the discriminator points to an optional member of the base struct, > +its default value must be specified as a 'default-variant'. In the > +following example, the above BlockDriver struct is changed so it > +defaults to the 'file' driver if that field is omitted on the wire: > + > + { 'union': 'BlockdevOptions', > + 'base': { '*driver': 'BlockdevDriver', '*read-only': 'bool' }, > + 'discriminator': 'driver', > + 'default-variant': 'file', > + 'data': { 'file': 'BlockdevOptionsFile', > + 'qcow2': 'BlockdevOptionsQcow2' } } > + > +Now the 'file' JSON object can be abbreviated to: > + > + { "read-only": "true", > + "filename": "/some/place/my-image" } > + Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 02/13] docs/qapi: Document optional discriminators Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-10 14:08 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing Max Reitz ` (9 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth There already is an optional discriminator test, although it also noted the discriminator name itself as optional. Changing that, and adding a default-variant field, makes that test pass instead of failing. Besides this one, a number of tests adding various error cases are added. Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/Makefile.include | 3 +++ ...flat-union-optional-discriminator-invalid-default.json | 12 ++++++++++++ .../flat-union-optional-discriminator-no-default.json | 11 +++++++++++ tests/qapi-schema/flat-union-optional-discriminator.json | 4 ++-- .../flat-union-superfluous-default-variant.json | 11 +++++++++++ .../flat-union-optional-discriminator-invalid-default.err | 1 + ...flat-union-optional-discriminator-invalid-default.exit | 1 + .../flat-union-optional-discriminator-invalid-default.out | 0 .../flat-union-optional-discriminator-no-default.err | 1 + .../flat-union-optional-discriminator-no-default.exit | 1 + .../flat-union-optional-discriminator-no-default.out | 0 tests/qapi-schema/flat-union-optional-discriminator.err | 1 - tests/qapi-schema/flat-union-optional-discriminator.exit | 2 +- tests/qapi-schema/flat-union-optional-discriminator.out | 15 +++++++++++++++ .../flat-union-superfluous-default-variant.err | 1 + .../flat-union-superfluous-default-variant.exit | 1 + .../flat-union-superfluous-default-variant.out | 0 17 files changed, 61 insertions(+), 4 deletions(-) create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.json create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.json create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.err create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.exit create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.out diff --git a/tests/Makefile.include b/tests/Makefile.include index 3b9a5e31a2..d01df12b09 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -500,7 +500,10 @@ qapi-schema += flat-union-invalid-branch-key.json qapi-schema += flat-union-invalid-discriminator.json qapi-schema += flat-union-no-base.json qapi-schema += flat-union-optional-discriminator.json +qapi-schema += flat-union-optional-discriminator-no-default.json +qapi-schema += flat-union-optional-discriminator-invalid-default.json qapi-schema += flat-union-string-discriminator.json +qapi-schema += flat-union-superfluous-default-variant.json qapi-schema += funny-char.json qapi-schema += ident-with-escape.json qapi-schema += include-before-err.json diff --git a/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json b/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json new file mode 100644 index 0000000000..015a47ba52 --- /dev/null +++ b/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json @@ -0,0 +1,12 @@ +# default-variant must refer to an actual value of the discriminator's +# enum +{ 'enum': 'Enum', 'data': [ 'one', 'two' ] } +{ 'struct': 'Base', + 'data': { '*switch': 'Enum' } } +{ 'struct': 'Branch', 'data': { 'name': 'str' } } +{ 'union': 'MyUnion', + 'base': 'Base', + 'discriminator': 'switch', + 'default-variant': 'three', + 'data': { 'one': 'Branch', + 'two': 'Branch' } } diff --git a/tests/qapi-schema/flat-union-optional-discriminator-no-default.json b/tests/qapi-schema/flat-union-optional-discriminator-no-default.json new file mode 100644 index 0000000000..0d612f5a62 --- /dev/null +++ b/tests/qapi-schema/flat-union-optional-discriminator-no-default.json @@ -0,0 +1,11 @@ +# Using an optional discriminator requires specifying a default +# variant +{ 'enum': 'Enum', 'data': [ 'one', 'two' ] } +{ 'struct': 'Base', + 'data': { '*switch': 'Enum' } } +{ 'struct': 'Branch', 'data': { 'name': 'str' } } +{ 'union': 'MyUnion', + 'base': 'Base', + 'discriminator': 'switch', + 'data': { 'one': 'Branch', + 'two': 'Branch' } } diff --git a/tests/qapi-schema/flat-union-optional-discriminator.json b/tests/qapi-schema/flat-union-optional-discriminator.json index 08a8f7ef8b..68e87083a1 100644 --- a/tests/qapi-schema/flat-union-optional-discriminator.json +++ b/tests/qapi-schema/flat-union-optional-discriminator.json @@ -1,10 +1,10 @@ -# we require the discriminator to be non-optional { 'enum': 'Enum', 'data': [ 'one', 'two' ] } { 'struct': 'Base', 'data': { '*switch': 'Enum' } } { 'struct': 'Branch', 'data': { 'name': 'str' } } { 'union': 'MyUnion', 'base': 'Base', - 'discriminator': '*switch', + 'discriminator': 'switch', + 'default-variant': 'one', 'data': { 'one': 'Branch', 'two': 'Branch' } } diff --git a/tests/qapi-schema/flat-union-superfluous-default-variant.json b/tests/qapi-schema/flat-union-superfluous-default-variant.json new file mode 100644 index 0000000000..8558165868 --- /dev/null +++ b/tests/qapi-schema/flat-union-superfluous-default-variant.json @@ -0,0 +1,11 @@ +# default-variant only makes sense with an optional discriminator +{ 'enum': 'Enum', 'data': [ 'one', 'two' ] } +{ 'struct': 'Base', + 'data': { 'switch': 'Enum' } } +{ 'struct': 'Branch', 'data': { 'name': 'str' } } +{ 'union': 'MyUnion', + 'base': 'Base', + 'discriminator': 'switch', + 'default-variant': 'one', + 'data': { 'one': 'Branch', + 'two': 'Branch' } } diff --git a/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err b/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err new file mode 100644 index 0000000000..b6bd3423d6 --- /dev/null +++ b/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err @@ -0,0 +1 @@ +tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json:7: Default variant 'three' of flat union 'MyUnion' is not part of 'Enum' diff --git a/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit b/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit new file mode 100644 index 0000000000..d00491fd7e --- /dev/null +++ b/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit @@ -0,0 +1 @@ +1 diff --git a/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out b/tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/qapi-schema/flat-union-optional-discriminator-no-default.err b/tests/qapi-schema/flat-union-optional-discriminator-no-default.err new file mode 100644 index 0000000000..1342efd9e8 --- /dev/null +++ b/tests/qapi-schema/flat-union-optional-discriminator-no-default.err @@ -0,0 +1 @@ +tests/qapi-schema/flat-union-optional-discriminator-no-default.json:7: Default variant must be specified for optional discriminator 'switch' diff --git a/tests/qapi-schema/flat-union-optional-discriminator-no-default.exit b/tests/qapi-schema/flat-union-optional-discriminator-no-default.exit new file mode 100644 index 0000000000..d00491fd7e --- /dev/null +++ b/tests/qapi-schema/flat-union-optional-discriminator-no-default.exit @@ -0,0 +1 @@ +1 diff --git a/tests/qapi-schema/flat-union-optional-discriminator-no-default.out b/tests/qapi-schema/flat-union-optional-discriminator-no-default.out new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/qapi-schema/flat-union-optional-discriminator.err b/tests/qapi-schema/flat-union-optional-discriminator.err index aaabedb3bd..e69de29bb2 100644 --- a/tests/qapi-schema/flat-union-optional-discriminator.err +++ b/tests/qapi-schema/flat-union-optional-discriminator.err @@ -1 +0,0 @@ -tests/qapi-schema/flat-union-optional-discriminator.json:6: Discriminator of flat union 'MyUnion' does not allow optional name '*switch' diff --git a/tests/qapi-schema/flat-union-optional-discriminator.exit b/tests/qapi-schema/flat-union-optional-discriminator.exit index d00491fd7e..573541ac97 100644 --- a/tests/qapi-schema/flat-union-optional-discriminator.exit +++ b/tests/qapi-schema/flat-union-optional-discriminator.exit @@ -1 +1 @@ -1 +0 diff --git a/tests/qapi-schema/flat-union-optional-discriminator.out b/tests/qapi-schema/flat-union-optional-discriminator.out index e69de29bb2..d3a72f63c2 100644 --- a/tests/qapi-schema/flat-union-optional-discriminator.out +++ b/tests/qapi-schema/flat-union-optional-discriminator.out @@ -0,0 +1,15 @@ +object q_empty +enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool'] + prefix QTYPE +module flat-union-optional-discriminator.json +enum Enum ['one', 'two'] +object Base + member switch: Enum optional=True +object Branch + member name: str optional=False +object MyUnion + base Base + tag switch + default variant: one + case one: Branch + case two: Branch diff --git a/tests/qapi-schema/flat-union-superfluous-default-variant.err b/tests/qapi-schema/flat-union-superfluous-default-variant.err new file mode 100644 index 0000000000..5230bbf645 --- /dev/null +++ b/tests/qapi-schema/flat-union-superfluous-default-variant.err @@ -0,0 +1 @@ +tests/qapi-schema/flat-union-superfluous-default-variant.json:6: Must not specify a default variant for non-optional discriminator 'switch' diff --git a/tests/qapi-schema/flat-union-superfluous-default-variant.exit b/tests/qapi-schema/flat-union-superfluous-default-variant.exit new file mode 100644 index 0000000000..d00491fd7e --- /dev/null +++ b/tests/qapi-schema/flat-union-superfluous-default-variant.exit @@ -0,0 +1 @@ +1 diff --git a/tests/qapi-schema/flat-union-superfluous-default-variant.out b/tests/qapi-schema/flat-union-superfluous-default-variant.out new file mode 100644 index 0000000000..e69de29bb2 -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests 2018-05-09 16:55 ` [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests Max Reitz @ 2018-05-10 14:08 ` Eric Blake 2018-05-11 18:06 ` Max Reitz 0 siblings, 1 reply; 42+ messages in thread From: Eric Blake @ 2018-05-10 14:08 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > There already is an optional discriminator test, although it also noted > the discriminator name itself as optional. Changing that, and adding a > default-variant field, makes that test pass instead of failing. I'm not sure I agree with that one. I think that you should instead add a positive test of a default variant into qapi-schema-test.json, especially since that is the one positive test where we also ensure that the C compiler is happy with the generated code (other positive tests prove that the generator produced something without error, but not that what it produced could be compiled). > 17 files changed, 61 insertions(+), 4 deletions(-) > create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json > create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.json > create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.json > create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err > create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit > create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out > create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err > create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit > create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out > create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.err > create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.exit > create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.out Patch 1 converted 2 error messages into 6, where 2 of them look identical: >> - # The value of member 'discriminator' must name a non-optional >> - # member of the base struct. >> + # The value of member 'discriminator' must name a member of >> + # the base struct. >> check_name(info, "Discriminator of flat union '%s'" % name, >> discriminator) [0] check_name() checks that 'discriminator':'*name' is rejected - this check is unchanged >> - discriminator_type = base_members.get(discriminator) >> - if not discriminator_type: >> - raise QAPISemError(info, >> - "Discriminator '%s' is not a member of base " >> - "struct '%s'" >> - % (discriminator, base)) The old code ensured that 'discriminator':'name' finds 'name' as a mandatory field in the base type; which changed into: >> + if default_variant is None: >> + discriminator_type = base_members.get(discriminator) >> + if not discriminator_type: >> + if base_members.get('*' + discriminator) is None: >> + raise QAPISemError(info, >> + "Discriminator '%s' is not a member of " >> + "base struct '%s'" >> + % (discriminator, base)) [1] the discriminator type must be a member field (we didn't find either a mandatory or an optional field) >> + else: >> + raise QAPISemError(info, >> + "Default variant must be specified for " >> + "optional discriminator '%s'" >> + % discriminator) [2] without default_variant, the member field must be mandatory >> + else: >> + discriminator_type = base_members.get('*' + discriminator) >> + if not discriminator_type: >> + if base_members.get(discriminator) is None: >> + raise QAPISemError(info, >> + "Discriminator '%s' is not a member of " >> + "base struct '%s'" >> + % (discriminator, base)) [3] the discriminator type must be a member field (we didn't find either a mandatory or an optional field), identical message to [1] >> + else: >> + raise QAPISemError(info, >> + "Must not specify a default variant for " >> + "non-optional discriminator '%s'" >> + % discriminator) [4] with default_variant, the member field must be optional >> + >> enum_define = enum_types.get(discriminator_type) >> allow_metas = ['struct'] >> # Do not allow string discriminator >> @@ -763,6 +785,15 @@ def check_union(expr, info): >> "Discriminator '%s' must be of enumeration " >> "type" % discriminator) >> >> + if default_variant is not None: >> + # Must be a value of the enumeration >> + if default_variant not in enum_define['data']: >> + raise QAPISemError(info, >> + "Default variant '%s' of flat union '%s' is " >> + "not part of '%s'" >> + % (default_variant, name, >> + discriminator_type)) >> + [5] the default discriminator value must belong to the discriminator's type Of those 6 messages, you now have coverage of: > > +++ b/tests/Makefile.include > @@ -500,7 +500,10 @@ qapi-schema += flat-union-invalid-branch-key.json > qapi-schema += flat-union-invalid-discriminator.json [1] no change to that test > qapi-schema += flat-union-no-base.json > qapi-schema += flat-union-optional-discriminator.json the old test for [0] - you turned it into success. See more below... > +qapi-schema += flat-union-optional-discriminator-no-default.json new test for [2] > +qapi-schema += flat-union-optional-discriminator-invalid-default.json new test for [5] > qapi-schema += flat-union-string-discriminator.json > +qapi-schema += flat-union-superfluous-default-variant.json new test for [4] I don't know if it is worth trying to explicitly test for [3], given that it is identical to [1], but the loss of the test for [0] bothers me. > +++ b/tests/qapi-schema/flat-union-optional-discriminator.err > @@ -1 +0,0 @@ > -tests/qapi-schema/flat-union-optional-discriminator.json:6: Discriminator of flat union 'MyUnion' does not allow optional name '*switch' This used to be covered by [0] - but you rewrote the test by changing 'discriminator':'*switch' to 'discriminator':'switch'. I'd argue that you WANT to keep 'discriminator':'*switch' in this test, to keep the testing of error [0] (even when the discriminator type is optional, we spell it without leading '*' in the 'union' description), and then only qapi-schema-test.json gets a positive test, which also gives us the additional coverage that the C compiler likes our generated code. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests 2018-05-10 14:08 ` Eric Blake @ 2018-05-11 18:06 ` Max Reitz 0 siblings, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-11 18:06 UTC (permalink / raw) To: Eric Blake, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth [-- Attachment #1: Type: text/plain, Size: 8813 bytes --] On 2018-05-10 16:08, Eric Blake wrote: > On 05/09/2018 11:55 AM, Max Reitz wrote: >> There already is an optional discriminator test, although it also noted >> the discriminator name itself as optional. Changing that, and adding a >> default-variant field, makes that test pass instead of failing. > > I'm not sure I agree with that one. I think that you should instead add > a positive test of a default variant into qapi-schema-test.json, > especially since that is the one positive test where we also ensure that > the C compiler is happy with the generated code (other positive tests > prove that the generator produced something without error, but not that > what it produced could be compiled). Oh, sure. >> 17 files changed, 61 insertions(+), 4 deletions(-) >> create mode 100644 >> tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json >> create mode 100644 >> tests/qapi-schema/flat-union-optional-discriminator-no-default.json >> create mode 100644 >> tests/qapi-schema/flat-union-superfluous-default-variant.json >> create mode 100644 >> tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err >> create mode 100644 >> tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit >> create mode 100644 >> tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out >> create mode 100644 >> tests/qapi-schema/flat-union-optional-discriminator-no-default.err >> create mode 100644 >> tests/qapi-schema/flat-union-optional-discriminator-no-default.exit >> create mode 100644 >> tests/qapi-schema/flat-union-optional-discriminator-no-default.out >> create mode 100644 >> tests/qapi-schema/flat-union-superfluous-default-variant.err >> create mode 100644 >> tests/qapi-schema/flat-union-superfluous-default-variant.exit >> create mode 100644 >> tests/qapi-schema/flat-union-superfluous-default-variant.out > > Patch 1 converted 2 error messages into 6, where 2 of them look identical: > >>> - # The value of member 'discriminator' must name a non-optional >>> - # member of the base struct. >>> + # The value of member 'discriminator' must name a member of >>> + # the base struct. >>> check_name(info, "Discriminator of flat union '%s'" % name, >>> discriminator) > > [0] check_name() checks that 'discriminator':'*name' is rejected - this > check is unchanged > >>> - discriminator_type = base_members.get(discriminator) >>> - if not discriminator_type: >>> - raise QAPISemError(info, >>> - "Discriminator '%s' is not a member >>> of base " >>> - "struct '%s'" >>> - % (discriminator, base)) > > The old code ensured that 'discriminator':'name' finds 'name' as a > mandatory field in the base type; which changed into: > >>> + if default_variant is None: >>> + discriminator_type = base_members.get(discriminator) >>> + if not discriminator_type: >>> + if base_members.get('*' + discriminator) is None: >>> + raise QAPISemError(info, >>> + "Discriminator '%s' is not a >>> member of " >>> + "base struct '%s'" >>> + % (discriminator, base)) > > [1] the discriminator type must be a member field (we didn't find either > a mandatory or an optional field) > >>> + else: >>> + raise QAPISemError(info, >>> + "Default variant must be >>> specified for " >>> + "optional discriminator '%s'" >>> + % discriminator) > > [2] without default_variant, the member field must be mandatory > >>> + else: >>> + discriminator_type = base_members.get('*' + discriminator) >>> + if not discriminator_type: >>> + if base_members.get(discriminator) is None: >>> + raise QAPISemError(info, >>> + "Discriminator '%s' is not a >>> member of " >>> + "base struct '%s'" >>> + % (discriminator, base)) > > [3] the discriminator type must be a member field (we didn't find either > a mandatory or an optional field), identical message to [1] > >>> + else: >>> + raise QAPISemError(info, >>> + "Must not specify a default >>> variant for " >>> + "non-optional discriminator >>> '%s'" >>> + % discriminator) > > [4] with default_variant, the member field must be optional > >>> + >>> enum_define = enum_types.get(discriminator_type) >>> allow_metas = ['struct'] >>> # Do not allow string discriminator >>> @@ -763,6 +785,15 @@ def check_union(expr, info): >>> "Discriminator '%s' must be of >>> enumeration " >>> "type" % discriminator) >>> >>> + if default_variant is not None: >>> + # Must be a value of the enumeration >>> + if default_variant not in enum_define['data']: >>> + raise QAPISemError(info, >>> + "Default variant '%s' of flat >>> union '%s' is " >>> + "not part of '%s'" >>> + % (default_variant, name, >>> + discriminator_type)) >>> + > > [5] the default discriminator value must belong to the discriminator's type > > Of those 6 messages, you now have coverage of: > >> >> +++ b/tests/Makefile.include >> @@ -500,7 +500,10 @@ qapi-schema += flat-union-invalid-branch-key.json >> qapi-schema += flat-union-invalid-discriminator.json > > [1] no change to that test > >> qapi-schema += flat-union-no-base.json >> qapi-schema += flat-union-optional-discriminator.json > > the old test for [0] - you turned it into success. See more below... > >> +qapi-schema += flat-union-optional-discriminator-no-default.json > > new test for [2] > >> +qapi-schema += flat-union-optional-discriminator-invalid-default.json > > new test for [5] > >> qapi-schema += flat-union-string-discriminator.json >> +qapi-schema += flat-union-superfluous-default-variant.json > > new test for [4] > > I don't know if it is worth trying to explicitly test for [3], given > that it is identical to [1], but the loss of the test for [0] bothers me. > >> +++ b/tests/qapi-schema/flat-union-optional-discriminator.err >> @@ -1 +0,0 @@ >> -tests/qapi-schema/flat-union-optional-discriminator.json:6: >> Discriminator of flat union 'MyUnion' does not allow optional name >> '*switch' > > This used to be covered by [0] - but you rewrote the test by changing > 'discriminator':'*switch' to 'discriminator':'switch'. > > I'd argue that you WANT to keep 'discriminator':'*switch' in this test, > to keep the testing of error [0] (even when the discriminator type is > optional, we spell it without leading '*' in the 'union' description), > and then only qapi-schema-test.json gets a positive test, which also > gives us the additional coverage that the C compiler likes our generated > code. OK, but I'd like to rename the test, then. Max [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (2 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-10 7:58 ` Daniel P. Berrangé 2018-05-09 16:55 ` [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow " Max Reitz ` (8 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth Currently, you can give no encryption format for a qcow2 file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it. Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 44 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 71c9ab8538..092a1aba2d 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -43,6 +43,19 @@ { 'struct': 'ImageInfoSpecificQCow2EncryptionBase', 'data': { 'format': 'BlockdevQcow2EncryptionFormat'}} +## +# @ImageInfoSpecificQCow2EncryptionNoInfo: +# +# Only used for the qcow2 encryption format "from-image" in which the +# actual encryption format is determined from the image header. +# Therefore, this encryption format will never be reported in +# ImageInfoSpecificQCow2Encryption. +# +# Since: 2.13 +## +{ 'struct': 'ImageInfoSpecificQCow2EncryptionNoInfo', + 'data': { } } + ## # @ImageInfoSpecificQCow2Encryption: # @@ -52,7 +65,8 @@ 'base': 'ImageInfoSpecificQCow2EncryptionBase', 'discriminator': 'format', 'data': { 'aes': 'QCryptoBlockInfoQCow', - 'luks': 'QCryptoBlockInfoLUKS' } } + 'luks': 'QCryptoBlockInfoLUKS', + 'from-image': 'ImageInfoSpecificQCow2EncryptionNoInfo' } } ## # @ImageInfoSpecificQCow2: @@ -2739,10 +2753,30 @@ # @BlockdevQcow2EncryptionFormat: # @aes: AES-CBC with plain64 initialization venctors # +# @from-image: Determine the encryption format from the image +# header. This only allows the use of the +# key-secret option. (Since: 2.13) +# # Since: 2.10 ## { 'enum': 'BlockdevQcow2EncryptionFormat', - 'data': [ 'aes', 'luks' ] } + 'data': [ 'aes', 'luks', 'from-image' ] } + +## +# @BlockdevQcow2EncryptionSecret: +# +# Allows specifying a key-secret without specifying the exact +# encryption format, which is determined automatically from the image +# header. +# +# @key-secret: The ID of a QCryptoSecret object providing the +# decryption key. Mandatory except when probing +# image for metadata only. +# +# Since: 2.13 +## +{ 'struct': 'BlockdevQcow2EncryptionSecret', + 'data': { '*key-secret': 'str' } } ## # @BlockdevQcow2Encryption: @@ -2750,10 +2784,12 @@ # Since: 2.10 ## { 'union': 'BlockdevQcow2Encryption', - 'base': { 'format': 'BlockdevQcow2EncryptionFormat' }, + 'base': { '*format': 'BlockdevQcow2EncryptionFormat' }, 'discriminator': 'format', + 'default-variant': 'from-image', 'data': { 'aes': 'QCryptoBlockOptionsQCow', - 'luks': 'QCryptoBlockOptionsLUKS'} } + 'luks': 'QCryptoBlockOptionsLUKS', + 'from-image': 'BlockdevQcow2EncryptionSecret' } } ## # @BlockdevOptionsQcow2: -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing 2018-05-09 16:55 ` [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing Max Reitz @ 2018-05-10 7:58 ` Daniel P. Berrangé 2018-05-10 14:22 ` Eric Blake 2018-05-11 17:32 ` Max Reitz 0 siblings, 2 replies; 42+ messages in thread From: Daniel P. Berrangé @ 2018-05-10 7:58 UTC (permalink / raw) To: Max Reitz Cc: qemu-block, Kevin Wolf, qemu-devel, Markus Armbruster, Michael Roth On Wed, May 09, 2018 at 06:55:21PM +0200, Max Reitz wrote: > Currently, you can give no encryption format for a qcow2 file while > still passing a key-secret. That does not conform to the schema, so > this patch changes the schema to allow it. > > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > qapi/block-core.json | 44 ++++++++++++++++++++++++++++++++++++++++---- > 1 file changed, 40 insertions(+), 4 deletions(-) > > diff --git a/qapi/block-core.json b/qapi/block-core.json > index 71c9ab8538..092a1aba2d 100644 > --- a/qapi/block-core.json > +++ b/qapi/block-core.json > @@ -43,6 +43,19 @@ > { 'struct': 'ImageInfoSpecificQCow2EncryptionBase', > 'data': { 'format': 'BlockdevQcow2EncryptionFormat'}} > > +## > +# @ImageInfoSpecificQCow2EncryptionNoInfo: > +# > +# Only used for the qcow2 encryption format "from-image" in which the > +# actual encryption format is determined from the image header. > +# Therefore, this encryption format will never be reported in > +# ImageInfoSpecificQCow2Encryption. > +# > +# Since: 2.13 > +## > +{ 'struct': 'ImageInfoSpecificQCow2EncryptionNoInfo', > + 'data': { } } > + > ## > # @ImageInfoSpecificQCow2Encryption: > # > @@ -52,7 +65,8 @@ > 'base': 'ImageInfoSpecificQCow2EncryptionBase', > 'discriminator': 'format', > 'data': { 'aes': 'QCryptoBlockInfoQCow', > - 'luks': 'QCryptoBlockInfoLUKS' } } > + 'luks': 'QCryptoBlockInfoLUKS', > + 'from-image': 'ImageInfoSpecificQCow2EncryptionNoInfo' } } > > ## > # @ImageInfoSpecificQCow2: > @@ -2739,10 +2753,30 @@ > # @BlockdevQcow2EncryptionFormat: > # @aes: AES-CBC with plain64 initialization venctors > # > +# @from-image: Determine the encryption format from the image > +# header. This only allows the use of the > +# key-secret option. (Since: 2.13) > +# > # Since: 2.10 > ## > { 'enum': 'BlockdevQcow2EncryptionFormat', > - 'data': [ 'aes', 'luks' ] } > + 'data': [ 'aes', 'luks', 'from-image' ] } > + > +## > +# @BlockdevQcow2EncryptionSecret: > +# > +# Allows specifying a key-secret without specifying the exact > +# encryption format, which is determined automatically from the image > +# header. > +# > +# @key-secret: The ID of a QCryptoSecret object providing the > +# decryption key. Mandatory except when probing > +# image for metadata only. > +# > +# Since: 2.13 > +## > +{ 'struct': 'BlockdevQcow2EncryptionSecret', > + 'data': { '*key-secret': 'str' } } > > ## > # @BlockdevQcow2Encryption: > @@ -2750,10 +2784,12 @@ > # Since: 2.10 > ## > { 'union': 'BlockdevQcow2Encryption', > - 'base': { 'format': 'BlockdevQcow2EncryptionFormat' }, > + 'base': { '*format': 'BlockdevQcow2EncryptionFormat' }, > 'discriminator': 'format', > + 'default-variant': 'from-image', > 'data': { 'aes': 'QCryptoBlockOptionsQCow', > - 'luks': 'QCryptoBlockOptionsLUKS'} } > + 'luks': 'QCryptoBlockOptionsLUKS', > + 'from-image': 'BlockdevQcow2EncryptionSecret' } } Bike-shedding on name, how about "auto" or "probe" ? IIUC, this schema addition means the QAPI parser now allows encrypt.format=from-image,encrypt.key-secret=sec0,...other opts... but the code will not accept "from-image" as a valid string. eg qcow2_update_options_prepare() will do case QCOW_CRYPT_AES: if (encryptfmt && !g_str_equal(encryptfmt, "aes")) { error_setg(errp, "Header reported 'aes' encryption format but " "options specify '%s'", encryptfmt); ret = -EINVAL; goto fail; } ...snip.... case QCOW_CRYPT_LUKS: if (encryptfmt && !g_str_equal(encryptfmt, "luks")) { error_setg(errp, "Header reported 'luks' encryption format but " "options specify '%s'", encryptfmt); ret = -EINVAL; goto fail; } Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing 2018-05-10 7:58 ` Daniel P. Berrangé @ 2018-05-10 14:22 ` Eric Blake 2018-05-11 17:32 ` Max Reitz 1 sibling, 0 replies; 42+ messages in thread From: Eric Blake @ 2018-05-10 14:22 UTC (permalink / raw) To: Daniel P. Berrangé, Max Reitz Cc: Kevin Wolf, Michael Roth, qemu-devel, qemu-block, Markus Armbruster On 05/10/2018 02:58 AM, Daniel P. Berrangé wrote: > On Wed, May 09, 2018 at 06:55:21PM +0200, Max Reitz wrote: >> Currently, you can give no encryption format for a qcow2 file while >> still passing a key-secret. That does not conform to the schema, so >> this patch changes the schema to allow it. >> >> Signed-off-by: Max Reitz <mreitz@redhat.com> >> --- >> qapi/block-core.json | 44 ++++++++++++++++++++++++++++++++++++++++---- >> 1 file changed, 40 insertions(+), 4 deletions(-) >> { 'union': 'BlockdevQcow2Encryption', >> - 'base': { 'format': 'BlockdevQcow2EncryptionFormat' }, >> + 'base': { '*format': 'BlockdevQcow2EncryptionFormat' }, >> 'discriminator': 'format', >> + 'default-variant': 'from-image', >> 'data': { 'aes': 'QCryptoBlockOptionsQCow', >> - 'luks': 'QCryptoBlockOptionsLUKS'} } >> + 'luks': 'QCryptoBlockOptionsLUKS', >> + 'from-image': 'BlockdevQcow2EncryptionSecret' } } > > Bike-shedding on name, how about "auto" or "probe" ? Either of those sounds nicer to me; 'auto' might be better in the context of creation (that way, we can state that creating a NEW image with x-blockdev-create maps 'auto' to 'luks'; while connecting to an EXISTING image maps 'auto' to either 'aes' or 'luks' as appropriate). > > IIUC, this schema addition means the QAPI parser now allows > > encrypt.format=from-image,encrypt.key-secret=sec0,...other opts... Yes. You could, perhaps, add a special case on the command line parsing code to reject an explicit use of format=from-image, but the QMP should not reject an explicit discriminator. Hmm, it plays in with my comment on 1/13 - should the QMP parser automatically set has_discriminator to true when it supplies the default? If it does, you lose the ability to see whether the user supplied an explicit encrypt.format=from-image (or the equivalent when using QMP instead of the command line), if you wanted to enforce that the user MUST omit format when relying on the from-image variant. I don't see a problem in allowing the user to explicitly specify the name of the default branch, but I _do_ think the patch is incomplete for not handling the new QCOW_CRYPT_FROM_IMAGE case and converting it as soon as possible back into one of the other two preferred enum values. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing 2018-05-10 7:58 ` Daniel P. Berrangé 2018-05-10 14:22 ` Eric Blake @ 2018-05-11 17:32 ` Max Reitz 1 sibling, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-11 17:32 UTC (permalink / raw) To: Daniel P. Berrangé Cc: qemu-block, Kevin Wolf, qemu-devel, Markus Armbruster, Michael Roth [-- Attachment #1: Type: text/plain, Size: 4554 bytes --] On 2018-05-10 09:58, Daniel P. Berrangé wrote: > On Wed, May 09, 2018 at 06:55:21PM +0200, Max Reitz wrote: >> Currently, you can give no encryption format for a qcow2 file while >> still passing a key-secret. That does not conform to the schema, so >> this patch changes the schema to allow it. >> >> Signed-off-by: Max Reitz <mreitz@redhat.com> >> --- >> qapi/block-core.json | 44 ++++++++++++++++++++++++++++++++++++++++---- >> 1 file changed, 40 insertions(+), 4 deletions(-) >> >> diff --git a/qapi/block-core.json b/qapi/block-core.json >> index 71c9ab8538..092a1aba2d 100644 >> --- a/qapi/block-core.json >> +++ b/qapi/block-core.json >> @@ -43,6 +43,19 @@ >> { 'struct': 'ImageInfoSpecificQCow2EncryptionBase', >> 'data': { 'format': 'BlockdevQcow2EncryptionFormat'}} >> >> +## >> +# @ImageInfoSpecificQCow2EncryptionNoInfo: >> +# >> +# Only used for the qcow2 encryption format "from-image" in which the >> +# actual encryption format is determined from the image header. >> +# Therefore, this encryption format will never be reported in >> +# ImageInfoSpecificQCow2Encryption. >> +# >> +# Since: 2.13 >> +## >> +{ 'struct': 'ImageInfoSpecificQCow2EncryptionNoInfo', >> + 'data': { } } >> + >> ## >> # @ImageInfoSpecificQCow2Encryption: >> # >> @@ -52,7 +65,8 @@ >> 'base': 'ImageInfoSpecificQCow2EncryptionBase', >> 'discriminator': 'format', >> 'data': { 'aes': 'QCryptoBlockInfoQCow', >> - 'luks': 'QCryptoBlockInfoLUKS' } } >> + 'luks': 'QCryptoBlockInfoLUKS', >> + 'from-image': 'ImageInfoSpecificQCow2EncryptionNoInfo' } } >> >> ## >> # @ImageInfoSpecificQCow2: >> @@ -2739,10 +2753,30 @@ >> # @BlockdevQcow2EncryptionFormat: >> # @aes: AES-CBC with plain64 initialization venctors >> # >> +# @from-image: Determine the encryption format from the image >> +# header. This only allows the use of the >> +# key-secret option. (Since: 2.13) >> +# >> # Since: 2.10 >> ## >> { 'enum': 'BlockdevQcow2EncryptionFormat', >> - 'data': [ 'aes', 'luks' ] } >> + 'data': [ 'aes', 'luks', 'from-image' ] } >> + >> +## >> +# @BlockdevQcow2EncryptionSecret: >> +# >> +# Allows specifying a key-secret without specifying the exact >> +# encryption format, which is determined automatically from the image >> +# header. >> +# >> +# @key-secret: The ID of a QCryptoSecret object providing the >> +# decryption key. Mandatory except when probing >> +# image for metadata only. >> +# >> +# Since: 2.13 >> +## >> +{ 'struct': 'BlockdevQcow2EncryptionSecret', >> + 'data': { '*key-secret': 'str' } } >> >> ## >> # @BlockdevQcow2Encryption: >> @@ -2750,10 +2784,12 @@ >> # Since: 2.10 >> ## >> { 'union': 'BlockdevQcow2Encryption', >> - 'base': { 'format': 'BlockdevQcow2EncryptionFormat' }, >> + 'base': { '*format': 'BlockdevQcow2EncryptionFormat' }, >> 'discriminator': 'format', >> + 'default-variant': 'from-image', >> 'data': { 'aes': 'QCryptoBlockOptionsQCow', >> - 'luks': 'QCryptoBlockOptionsLUKS'} } >> + 'luks': 'QCryptoBlockOptionsLUKS', >> + 'from-image': 'BlockdevQcow2EncryptionSecret' } } > > Bike-shedding on name, how about "auto" or "probe" ? Sure. I like "probe" a bit better than "auto", although "auto" is what we usually have... But I think "probe" is still a bit better. > > IIUC, this schema addition means the QAPI parser now allows > > encrypt.format=from-image,encrypt.key-secret=sec0,...other opts... > > but the code will not accept "from-image" as a valid string. Ah, right, I forgot that. Will fix. Thanks for reviewing! Max > eg qcow2_update_options_prepare() will do > > case QCOW_CRYPT_AES: > if (encryptfmt && !g_str_equal(encryptfmt, "aes")) { > error_setg(errp, > "Header reported 'aes' encryption format but " > "options specify '%s'", encryptfmt); > ret = -EINVAL; > goto fail; > } > > ...snip.... > > case QCOW_CRYPT_LUKS: > if (encryptfmt && !g_str_equal(encryptfmt, "luks")) { > error_setg(errp, > "Header reported 'luks' encryption format but " > "options specify '%s'", encryptfmt); > ret = -EINVAL; > goto fail; > } > > > Regards, > Daniel > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow encryption probing 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (3 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-10 14:24 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header Max Reitz ` (7 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth Currently, you can give no encryption format for a qcow file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it. Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 092a1aba2d..d98af0a71d 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -2712,15 +2712,35 @@ 'data': { 'flags': 'Qcow2OverlapCheckFlags', 'mode': 'Qcow2OverlapCheckMode' } } +## +# @BlockdevQcowEncryptionSecret: +# +# Allows specifying a key-secret without specifying the exact +# encryption format, which is determined automatically from the image +# header. +# +# @key-secret: The ID of a QCryptoSecret object providing the +# decryption key. Mandatory except when probing +# image for metadata only. +# +# Since: 2.13 +## +{ 'struct': 'BlockdevQcowEncryptionSecret', + 'data': { '*key-secret': 'str' } } + ## # @BlockdevQcowEncryptionFormat: # # @aes: AES-CBC with plain64 initialization vectors # +# @from-image: Determine the encryption format from the image +# header. This only allows the use of the +# key-secret option. (Since: 2.13) +# # Since: 2.10 ## { 'enum': 'BlockdevQcowEncryptionFormat', - 'data': [ 'aes' ] } + 'data': [ 'aes', 'from-image' ] } ## # @BlockdevQcowEncryption: @@ -2728,9 +2748,11 @@ # Since: 2.10 ## { 'union': 'BlockdevQcowEncryption', - 'base': { 'format': 'BlockdevQcowEncryptionFormat' }, + 'base': { '*format': 'BlockdevQcowEncryptionFormat' }, 'discriminator': 'format', - 'data': { 'aes': 'QCryptoBlockOptionsQCow' } } + 'default-variant': 'from-image', + 'data': { 'aes': 'QCryptoBlockOptionsQCow', + 'from-image': 'BlockdevQcowEncryptionSecret' } } ## # @BlockdevOptionsQcow: -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow encryption probing 2018-05-09 16:55 ` [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow " Max Reitz @ 2018-05-10 14:24 ` Eric Blake 2018-05-10 14:32 ` Daniel P. Berrangé 2018-05-11 18:07 ` Max Reitz 0 siblings, 2 replies; 42+ messages in thread From: Eric Blake @ 2018-05-10 14:24 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > Currently, you can give no encryption format for a qcow file while still > passing a key-secret. That does not conform to the schema, so this > patch changes the schema to allow it. > > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > ## > # @BlockdevQcowEncryptionFormat: > # > # @aes: AES-CBC with plain64 initialization vectors > # > +# @from-image: Determine the encryption format from the image > +# header. This only allows the use of the > +# key-secret option. (Since: 2.13) > +# > # Since: 2.10 > ## > { 'enum': 'BlockdevQcowEncryptionFormat', > - 'data': [ 'aes' ] } > + 'data': [ 'aes', 'from-image' ] } Overkill. Why not just: > > ## > # @BlockdevQcowEncryption: > @@ -2728,9 +2748,11 @@ > # Since: 2.10 > ## > { 'union': 'BlockdevQcowEncryption', > - 'base': { 'format': 'BlockdevQcowEncryptionFormat' }, > + 'base': { '*format': 'BlockdevQcowEncryptionFormat' }, > 'discriminator': 'format', > - 'data': { 'aes': 'QCryptoBlockOptionsQCow' } } > + 'default-variant': 'from-image', 'default-variant': 'aes' > + 'data': { 'aes': 'QCryptoBlockOptionsQCow', and call it good, because there are no other options to pick from, so 'from-image' would always resolve to 'aes' anyway. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow encryption probing 2018-05-10 14:24 ` Eric Blake @ 2018-05-10 14:32 ` Daniel P. Berrangé 2018-05-11 18:07 ` Max Reitz 1 sibling, 0 replies; 42+ messages in thread From: Daniel P. Berrangé @ 2018-05-10 14:32 UTC (permalink / raw) To: Eric Blake Cc: Max Reitz, qemu-block, Kevin Wolf, Michael Roth, qemu-devel, Markus Armbruster On Thu, May 10, 2018 at 09:24:24AM -0500, Eric Blake wrote: > On 05/09/2018 11:55 AM, Max Reitz wrote: > > Currently, you can give no encryption format for a qcow file while still > > passing a key-secret. That does not conform to the schema, so this > > patch changes the schema to allow it. > > > > Signed-off-by: Max Reitz <mreitz@redhat.com> > > --- > > > ## > > # @BlockdevQcowEncryptionFormat: > > # > > # @aes: AES-CBC with plain64 initialization vectors > > # > > +# @from-image: Determine the encryption format from the image > > +# header. This only allows the use of the > > +# key-secret option. (Since: 2.13) > > +# > > # Since: 2.10 > > ## > > { 'enum': 'BlockdevQcowEncryptionFormat', > > - 'data': [ 'aes' ] } > > + 'data': [ 'aes', 'from-image' ] } > > Overkill. Why not just: > > > ## > > # @BlockdevQcowEncryption: > > @@ -2728,9 +2748,11 @@ > > # Since: 2.10 > > ## > > { 'union': 'BlockdevQcowEncryption', > > - 'base': { 'format': 'BlockdevQcowEncryptionFormat' }, > > + 'base': { '*format': 'BlockdevQcowEncryptionFormat' }, > > 'discriminator': 'format', > > - 'data': { 'aes': 'QCryptoBlockOptionsQCow' } } > > + 'default-variant': 'from-image', > > 'default-variant': 'aes' > > > + 'data': { 'aes': 'QCryptoBlockOptionsQCow', > > and call it good, because there are no other options to pick from, so > 'from-image' would always resolve to 'aes' anyway. Sounds reasonable because qcowv1 is a dead format we don't intend to add more features to. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow encryption probing 2018-05-10 14:24 ` Eric Blake 2018-05-10 14:32 ` Daniel P. Berrangé @ 2018-05-11 18:07 ` Max Reitz 1 sibling, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-11 18:07 UTC (permalink / raw) To: Eric Blake, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth [-- Attachment #1: Type: text/plain, Size: 1558 bytes --] On 2018-05-10 16:24, Eric Blake wrote: > On 05/09/2018 11:55 AM, Max Reitz wrote: >> Currently, you can give no encryption format for a qcow file while still >> passing a key-secret. That does not conform to the schema, so this >> patch changes the schema to allow it. >> >> Signed-off-by: Max Reitz <mreitz@redhat.com> >> --- > >> ## >> # @BlockdevQcowEncryptionFormat: >> # >> # @aes: AES-CBC with plain64 initialization vectors >> # >> +# @from-image: Determine the encryption format from the image >> +# header. This only allows the use of the >> +# key-secret option. (Since: 2.13) >> +# >> # Since: 2.10 >> ## >> { 'enum': 'BlockdevQcowEncryptionFormat', >> - 'data': [ 'aes' ] } >> + 'data': [ 'aes', 'from-image' ] } > > Overkill. Why not just: > >> ## >> # @BlockdevQcowEncryption: >> @@ -2728,9 +2748,11 @@ >> # Since: 2.10 >> ## >> { 'union': 'BlockdevQcowEncryption', >> - 'base': { 'format': 'BlockdevQcowEncryptionFormat' }, >> + 'base': { '*format': 'BlockdevQcowEncryptionFormat' }, >> 'discriminator': 'format', >> - 'data': { 'aes': 'QCryptoBlockOptionsQCow' } } >> + 'default-variant': 'from-image', > > 'default-variant': 'aes' > >> + 'data': { 'aes': 'QCryptoBlockOptionsQCow', > > and call it good, because there are no other options to pick from, so > 'from-image' would always resolve to 'aes' anyway. Hmmmm. Yes. :-) Max [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (4 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow " Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-10 14:54 ` Eric Blake 2018-06-06 13:17 ` Markus Armbruster 2018-05-09 16:55 ` [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval() Max Reitz ` (6 subsequent siblings) 12 siblings, 2 replies; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth There are numerous QDict functions that have been introduced for and are used only by the block layer. Move their declarations into an own header file to reflect that. While qdict_extract_subqdict() is in fact used outside of the block layer (in util/qemu-config.c), it is still a function related very closely to how the block layer works with nested QDicts, namely by sometimes flattening them. Therefore, its declaration is put into this header as well and util/qemu-config.c includes it with a comment stating exactly which function it needs. Suggested-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/qdict.h | 35 +++++++++++++++++++++++++++++++++++ include/qapi/qmp/qdict.h | 17 ----------------- block.c | 1 + block/gluster.c | 1 + block/iscsi.c | 1 + block/nbd.c | 1 + block/nfs.c | 1 + block/parallels.c | 1 + block/qcow.c | 1 + block/qcow2.c | 1 + block/qed.c | 1 + block/quorum.c | 1 + block/rbd.c | 1 + block/sheepdog.c | 1 + block/snapshot.c | 1 + block/ssh.c | 1 + block/vhdx.c | 1 + block/vpc.c | 1 + block/vvfat.c | 1 + block/vxhs.c | 1 + blockdev.c | 1 + qobject/qdict.c | 1 + tests/check-qdict.c | 1 + tests/check-qobject.c | 1 + tests/test-replication.c | 1 + util/qemu-config.c | 1 + 26 files changed, 59 insertions(+), 17 deletions(-) create mode 100644 include/block/qdict.h diff --git a/include/block/qdict.h b/include/block/qdict.h new file mode 100644 index 0000000000..825e096a72 --- /dev/null +++ b/include/block/qdict.h @@ -0,0 +1,35 @@ +/* + * Special QDict functions used by the block layer + * + * Copyright (c) 2018 Red Hat, Inc. + * + * This work is licensed under the terms of the GNU LGPL, version 2.1 or later. + * See the COPYING.LIB file in the top-level directory. + */ + +#ifndef BLOCK_QDICT_H +#define BLOCK_QDICT_H + +#include "qapi/qmp/qdict.h" +#include "qapi/qmp/qobject.h" +#include "qapi/error.h" + + +void qdict_copy_default(QDict *dst, QDict *src, const char *key); +void qdict_set_default_str(QDict *dst, const char *key, const char *val); + +void qdict_join(QDict *dest, QDict *src, bool overwrite); + +void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start); +void qdict_array_split(QDict *src, QList **dst); +int qdict_array_entries(QDict *src, const char *subqdict); +QObject *qdict_crumple(const QDict *src, Error **errp); +void qdict_flatten(QDict *qdict); + +typedef struct QDictRenames { + const char *from; + const char *to; +} QDictRenames; +bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp); + +#endif diff --git a/include/qapi/qmp/qdict.h b/include/qapi/qmp/qdict.h index 921a28d2d3..7f3ec10a10 100644 --- a/include/qapi/qmp/qdict.h +++ b/include/qapi/qmp/qdict.h @@ -67,23 +67,6 @@ int64_t qdict_get_try_int(const QDict *qdict, const char *key, bool qdict_get_try_bool(const QDict *qdict, const char *key, bool def_value); const char *qdict_get_try_str(const QDict *qdict, const char *key); -void qdict_copy_default(QDict *dst, QDict *src, const char *key); -void qdict_set_default_str(QDict *dst, const char *key, const char *val); - QDict *qdict_clone_shallow(const QDict *src); -void qdict_flatten(QDict *qdict); - -void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start); -void qdict_array_split(QDict *src, QList **dst); -int qdict_array_entries(QDict *src, const char *subqdict); -QObject *qdict_crumple(const QDict *src, Error **errp); - -void qdict_join(QDict *dest, QDict *src, bool overwrite); - -typedef struct QDictRenames { - const char *from; - const char *to; -} QDictRenames; -bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp); #endif /* QDICT_H */ diff --git a/block.c b/block.c index 676e57f562..3c3e8fd11d 100644 --- a/block.c +++ b/block.c @@ -27,6 +27,7 @@ #include "block/block_int.h" #include "block/blockjob.h" #include "block/nbd.h" +#include "block/qdict.h" #include "qemu/error-report.h" #include "module_block.h" #include "qemu/module.h" diff --git a/block/gluster.c b/block/gluster.c index 9900b6420c..b5fe7f3e87 100644 --- a/block/gluster.c +++ b/block/gluster.c @@ -11,6 +11,7 @@ #include "qemu/osdep.h" #include <glusterfs/api/glfs.h> #include "block/block_int.h" +#include "block/qdict.h" #include "qapi/error.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qerror.h" diff --git a/block/iscsi.c b/block/iscsi.c index 3fd7203916..d70e8aea84 100644 --- a/block/iscsi.c +++ b/block/iscsi.c @@ -33,6 +33,7 @@ #include "qemu/bitops.h" #include "qemu/bitmap.h" #include "block/block_int.h" +#include "block/qdict.h" #include "scsi/constants.h" #include "qemu/iov.h" #include "qemu/option.h" diff --git a/block/nbd.c b/block/nbd.c index 3e1693cc55..f499830410 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -28,6 +28,7 @@ #include "qemu/osdep.h" #include "block/nbd-client.h" +#include "block/qdict.h" #include "qapi/error.h" #include "qemu/uri.h" #include "block/block_int.h" diff --git a/block/nfs.c b/block/nfs.c index 66fddf12d4..5159ef0aa7 100644 --- a/block/nfs.c +++ b/block/nfs.c @@ -29,6 +29,7 @@ #include "qemu/error-report.h" #include "qapi/error.h" #include "block/block_int.h" +#include "block/qdict.h" #include "trace.h" #include "qemu/iov.h" #include "qemu/option.h" diff --git a/block/parallels.c b/block/parallels.c index 6e9c37f44e..c1d9643498 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -31,6 +31,7 @@ #include "qemu/osdep.h" #include "qapi/error.h" #include "block/block_int.h" +#include "block/qdict.h" #include "sysemu/block-backend.h" #include "qemu/module.h" #include "qemu/option.h" diff --git a/block/qcow.c b/block/qcow.c index 3ba2ca25ea..e34eedc903 100644 --- a/block/qcow.c +++ b/block/qcow.c @@ -26,6 +26,7 @@ #include "qapi/error.h" #include "qemu/error-report.h" #include "block/block_int.h" +#include "block/qdict.h" #include "sysemu/block-backend.h" #include "qemu/module.h" #include "qemu/option.h" diff --git a/block/qcow2.c b/block/qcow2.c index 6d532470a8..416ab54298 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -24,6 +24,7 @@ #include "qemu/osdep.h" #include "block/block_int.h" +#include "block/qdict.h" #include "sysemu/block-backend.h" #include "qemu/module.h" #include <zlib.h> diff --git a/block/qed.c b/block/qed.c index 65cfe92393..324a953cbc 100644 --- a/block/qed.c +++ b/block/qed.c @@ -13,6 +13,7 @@ */ #include "qemu/osdep.h" +#include "block/qdict.h" #include "qapi/error.h" #include "qemu/timer.h" #include "qemu/bswap.h" diff --git a/block/quorum.c b/block/quorum.c index e448d7e384..0d90a02583 100644 --- a/block/quorum.c +++ b/block/quorum.c @@ -17,6 +17,7 @@ #include "qemu/cutils.h" #include "qemu/option.h" #include "block/block_int.h" +#include "block/qdict.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h" #include "qapi/qmp/qdict.h" diff --git a/block/rbd.c b/block/rbd.c index a16431e267..ad5fd30b43 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -18,6 +18,7 @@ #include "qemu/error-report.h" #include "qemu/option.h" #include "block/block_int.h" +#include "block/qdict.h" #include "crypto/secret.h" #include "qemu/cutils.h" #include "qapi/qmp/qstring.h" diff --git a/block/sheepdog.c b/block/sheepdog.c index 4237132419..bf7ab896d2 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -24,6 +24,7 @@ #include "qemu/option.h" #include "qemu/sockets.h" #include "block/block_int.h" +#include "block/qdict.h" #include "sysemu/block-backend.h" #include "qemu/bitops.h" #include "qemu/cutils.h" diff --git a/block/snapshot.c b/block/snapshot.c index 2953d96c06..f9903bc94e 100644 --- a/block/snapshot.c +++ b/block/snapshot.c @@ -25,6 +25,7 @@ #include "qemu/osdep.h" #include "block/snapshot.h" #include "block/block_int.h" +#include "block/qdict.h" #include "qapi/error.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qerror.h" diff --git a/block/ssh.c b/block/ssh.c index 4c4fa3ccfc..eec37dd27c 100644 --- a/block/ssh.c +++ b/block/ssh.c @@ -28,6 +28,7 @@ #include <libssh2_sftp.h> #include "block/block_int.h" +#include "block/qdict.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "qemu/option.h" diff --git a/block/vhdx.c b/block/vhdx.c index 0b1e21c750..b409acfeb2 100644 --- a/block/vhdx.c +++ b/block/vhdx.c @@ -18,6 +18,7 @@ #include "qemu/osdep.h" #include "qapi/error.h" #include "block/block_int.h" +#include "block/qdict.h" #include "sysemu/block-backend.h" #include "qemu/module.h" #include "qemu/option.h" diff --git a/block/vpc.c b/block/vpc.c index 0ebfcd3cc8..41c8c980f1 100644 --- a/block/vpc.c +++ b/block/vpc.c @@ -26,6 +26,7 @@ #include "qemu/osdep.h" #include "qapi/error.h" #include "block/block_int.h" +#include "block/qdict.h" #include "sysemu/block-backend.h" #include "qemu/module.h" #include "qemu/option.h" diff --git a/block/vvfat.c b/block/vvfat.c index 662dca0114..4595f335b8 100644 --- a/block/vvfat.c +++ b/block/vvfat.c @@ -27,6 +27,7 @@ #include <dirent.h> #include "qapi/error.h" #include "block/block_int.h" +#include "block/qdict.h" #include "qemu/module.h" #include "qemu/option.h" #include "qemu/bswap.h" diff --git a/block/vxhs.c b/block/vxhs.c index 339e23218d..0cb0a007e9 100644 --- a/block/vxhs.c +++ b/block/vxhs.c @@ -12,6 +12,7 @@ #include <qnio/qnio_api.h> #include <sys/param.h> #include "block/block_int.h" +#include "block/qdict.h" #include "qapi/qmp/qerror.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qstring.h" diff --git a/blockdev.c b/blockdev.c index 3808b1fc00..19c04d9276 100644 --- a/blockdev.c +++ b/blockdev.c @@ -35,6 +35,7 @@ #include "sysemu/blockdev.h" #include "hw/block/block.h" #include "block/blockjob.h" +#include "block/qdict.h" #include "block/throttle-groups.h" #include "monitor/monitor.h" #include "qemu/error-report.h" diff --git a/qobject/qdict.c b/qobject/qdict.c index 22800eeceb..0554c64553 100644 --- a/qobject/qdict.c +++ b/qobject/qdict.c @@ -11,6 +11,7 @@ */ #include "qemu/osdep.h" +#include "block/qdict.h" #include "qapi/qmp/qnum.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qbool.h" diff --git a/tests/check-qdict.c b/tests/check-qdict.c index eba5d3528e..93e2112b6d 100644 --- a/tests/check-qdict.c +++ b/tests/check-qdict.c @@ -11,6 +11,7 @@ */ #include "qemu/osdep.h" +#include "block/qdict.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qlist.h" #include "qapi/qmp/qnum.h" diff --git a/tests/check-qobject.c b/tests/check-qobject.c index 5cb08fcb63..16ccbde82c 100644 --- a/tests/check-qobject.c +++ b/tests/check-qobject.c @@ -8,6 +8,7 @@ */ #include "qemu/osdep.h" +#include "block/qdict.h" #include "qapi/qmp/qbool.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qlist.h" diff --git a/tests/test-replication.c b/tests/test-replication.c index 68c0d04f2a..c8165ae954 100644 --- a/tests/test-replication.c +++ b/tests/test-replication.c @@ -15,6 +15,7 @@ #include "qemu/option.h" #include "replication.h" #include "block/block_int.h" +#include "block/qdict.h" #include "sysemu/block-backend.h" #define IMG_SIZE (64 * 1024 * 1024) diff --git a/util/qemu-config.c b/util/qemu-config.c index 14d84022dc..9d2e278e29 100644 --- a/util/qemu-config.c +++ b/util/qemu-config.c @@ -1,4 +1,5 @@ #include "qemu/osdep.h" +#include "block/qdict.h" /* for qdict_extract_subqdict() */ #include "qapi/error.h" #include "qapi/qapi-commands-misc.h" #include "qapi/qmp/qdict.h" -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header 2018-05-09 16:55 ` [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header Max Reitz @ 2018-05-10 14:54 ` Eric Blake 2018-05-11 18:11 ` Max Reitz 2018-06-06 13:17 ` Markus Armbruster 1 sibling, 1 reply; 42+ messages in thread From: Eric Blake @ 2018-05-10 14:54 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > There are numerous QDict functions that have been introduced for and are > used only by the block layer. Move their declarations into an own s/an own/their own/ > header file to reflect that. > > While qdict_extract_subqdict() is in fact used outside of the block > layer (in util/qemu-config.c), it is still a function related very > closely to how the block layer works with nested QDicts, namely by > sometimes flattening them. Therefore, its declaration is put into this > header as well and util/qemu-config.c includes it with a comment stating > exactly which function it needs. > > Suggested-by: Markus Armbruster <armbru@redhat.com> > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > +++ b/include/block/qdict.h > @@ -0,0 +1,35 @@ > +/* > + * Special QDict functions used by the block layer > + * > + * Copyright (c) 2018 Red Hat, Inc. You are extracting this from qdict.h which has: * Copyright (C) 2009 Red Hat Inc. Is it worth listing 2009-2018, instead of just this year? > + > +void qdict_copy_default(QDict *dst, QDict *src, const char *key); > +void qdict_set_default_str(QDict *dst, const char *key, const char *val); These two might be useful outside of the block layer; we can move them back in a later patch if so. But for now, I agree with your choice of moving them. > + > +void qdict_join(QDict *dest, QDict *src, bool overwrite); This is borderline whether it would be useful outside of the block layer. > + > +void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start); > +void qdict_array_split(QDict *src, QList **dst); > +int qdict_array_entries(QDict *src, const char *subqdict); > +QObject *qdict_crumple(const QDict *src, Error **errp); > +void qdict_flatten(QDict *qdict); And these are definitely hacks that the block layer relies on, where hopefully someday long term we can rewrite the block layer to use QAPI types directly instead of constant reconversion between QemuOpts and QDict and QAPI types. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header 2018-05-10 14:54 ` Eric Blake @ 2018-05-11 18:11 ` Max Reitz 2018-06-06 13:10 ` Markus Armbruster 0 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-11 18:11 UTC (permalink / raw) To: Eric Blake, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth [-- Attachment #1: Type: text/plain, Size: 2793 bytes --] On 2018-05-10 16:54, Eric Blake wrote: > On 05/09/2018 11:55 AM, Max Reitz wrote: >> There are numerous QDict functions that have been introduced for and are >> used only by the block layer. Move their declarations into an own > > s/an own/their own/ > >> header file to reflect that. >> >> While qdict_extract_subqdict() is in fact used outside of the block >> layer (in util/qemu-config.c), it is still a function related very >> closely to how the block layer works with nested QDicts, namely by >> sometimes flattening them. Therefore, its declaration is put into this >> header as well and util/qemu-config.c includes it with a comment stating >> exactly which function it needs. >> >> Suggested-by: Markus Armbruster <armbru@redhat.com> >> Signed-off-by: Max Reitz <mreitz@redhat.com> >> --- > >> +++ b/include/block/qdict.h >> @@ -0,0 +1,35 @@ >> +/* >> + * Special QDict functions used by the block layer >> + * >> + * Copyright (c) 2018 Red Hat, Inc. > > You are extracting this from qdict.h which has: > * Copyright (C) 2009 Red Hat Inc. > > Is it worth listing 2009-2018, instead of just this year? I don't know, is it? I don't know whether it makes a real difference. >> + >> +void qdict_copy_default(QDict *dst, QDict *src, const char *key); >> +void qdict_set_default_str(QDict *dst, const char *key, const char >> *val); > > These two might be useful outside of the block layer; we can move them > back in a later patch if so. But for now, I agree with your choice of > moving them. > >> + >> +void qdict_join(QDict *dest, QDict *src, bool overwrite); > > This is borderline whether it would be useful outside of the block layer. I decided I wanted to move the *_default* functions, and if I did that, I would have to move this one as well. I decided I can't be biased just because I wrote this one. :-) But in general I'd say if anyone wants to use any of these functions outside of the block layer, they are welcome to move them back to the main header, provided they have a good reason to do so. I suppose a good reason for using qdict_join() may indeed turn up sooner or later. Max >> + >> +void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start); >> +void qdict_array_split(QDict *src, QList **dst); >> +int qdict_array_entries(QDict *src, const char *subqdict); >> +QObject *qdict_crumple(const QDict *src, Error **errp); >> +void qdict_flatten(QDict *qdict); > > And these are definitely hacks that the block layer relies on, where > hopefully someday long term we can rewrite the block layer to use QAPI > types directly instead of constant reconversion between QemuOpts and > QDict and QAPI types. > > Reviewed-by: Eric Blake <eblake@redhat.com> > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header 2018-05-11 18:11 ` Max Reitz @ 2018-06-06 13:10 ` Markus Armbruster 0 siblings, 0 replies; 42+ messages in thread From: Markus Armbruster @ 2018-06-06 13:10 UTC (permalink / raw) To: Max Reitz; +Cc: Eric Blake, qemu-block, Kevin Wolf, Michael Roth, qemu-devel Max Reitz <mreitz@redhat.com> writes: > On 2018-05-10 16:54, Eric Blake wrote: >> On 05/09/2018 11:55 AM, Max Reitz wrote: >>> There are numerous QDict functions that have been introduced for and are >>> used only by the block layer. Move their declarations into an own >> >> s/an own/their own/ >> >>> header file to reflect that. >>> >>> While qdict_extract_subqdict() is in fact used outside of the block >>> layer (in util/qemu-config.c), it is still a function related very >>> closely to how the block layer works with nested QDicts, namely by >>> sometimes flattening them. Therefore, its declaration is put into this >>> header as well and util/qemu-config.c includes it with a comment stating >>> exactly which function it needs. >>> >>> Suggested-by: Markus Armbruster <armbru@redhat.com> >>> Signed-off-by: Max Reitz <mreitz@redhat.com> >>> --- >> >>> +++ b/include/block/qdict.h >>> @@ -0,0 +1,35 @@ >>> +/* >>> + * Special QDict functions used by the block layer >>> + * >>> + * Copyright (c) 2018 Red Hat, Inc. >> >> You are extracting this from qdict.h which has: >> * Copyright (C) 2009 Red Hat Inc. >> >> Is it worth listing 2009-2018, instead of just this year? > > I don't know, is it? I don't know whether it makes a real difference. Where's your taste for pedantry? Here's mine: please make it 2013-2018, because the oldest code moved is from 2013. >>> + * >>> + * This work is licensed under the terms of the GNU LGPL, version 2.1 or later. >>> + * See the COPYING.LIB file in the top-level directory. >>> + */ >>> + >>> +#ifndef BLOCK_QDICT_H >>> +#define BLOCK_QDICT_H >>> + >>> +#include "qapi/qmp/qdict.h" >>> +#include "qapi/qmp/qobject.h" >>> +#include "qapi/error.h" Only the first #include is necessary, due to qemu/typedefs.h. Please drop the other two. >>> + >>> + >>> +void qdict_copy_default(QDict *dst, QDict *src, const char *key); >>> +void qdict_set_default_str(QDict *dst, const char *key, const char >>> *val); >> >> These two might be useful outside of the block layer; we can move them >> back in a later patch if so. But for now, I agree with your choice of >> moving them. >> >>> + >>> +void qdict_join(QDict *dest, QDict *src, bool overwrite); >> >> This is borderline whether it would be useful outside of the block layer. > > I decided I wanted to move the *_default* functions, and if I did that, > I would have to move this one as well. I decided I can't be biased just > because I wrote this one. :-) > > But in general I'd say if anyone wants to use any of these functions > outside of the block layer, they are welcome to move them back to the > main header, provided they have a good reason to do so. I suppose a > good reason for using qdict_join() may indeed turn up sooner or later. I like it this way. With the trivial changes I asked for: Reviewed-by: Markus Armbruster <armbru@redhat.com> ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header 2018-05-09 16:55 ` [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header Max Reitz 2018-05-10 14:54 ` Eric Blake @ 2018-06-06 13:17 ` Markus Armbruster 2018-06-06 14:19 ` Markus Armbruster 1 sibling, 1 reply; 42+ messages in thread From: Markus Armbruster @ 2018-06-06 13:17 UTC (permalink / raw) To: Max Reitz; +Cc: qemu-block, Kevin Wolf, qemu-devel, Michael Roth MAINTAINERS files include/block/qdict.h under "Block layer core". Let's split qobject/qdict.c as well, so it's there, too. Obvious patch appended. Feel free to squash it into yours without giving me credit for it. PS: I tried to move block-qdict.c to block/, but totally failed at persuading Make to link it into all programs that need it. Oh well. >From 4614533375729c84c0599e1fed44420b29a180a6 Mon Sep 17 00:00:00 2001 From: Markus Armbruster <armbru@redhat.com> Date: Wed, 6 Jun 2018 15:11:54 +0200 Subject: [PATCH] qobject: Move block-specific qdict code to block-qdict.c Signed-off-by: Markus Armbruster <armbru@redhat.com> --- MAINTAINERS | 1 + qobject/Makefile.objs | 1 + qobject/block-qdict.c | 637 ++++++++++++++++++++++++++++++++++++++++++ qobject/qdict.c | 629 ----------------------------------------- 4 files changed, 639 insertions(+), 629 deletions(-) create mode 100644 qobject/block-qdict.c diff --git a/MAINTAINERS b/MAINTAINERS index 41cd3736a9..9f9835e052 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1365,6 +1365,7 @@ F: qemu-img* F: qemu-io* F: tests/qemu-iotests/ F: util/qemu-progress.c +F: qobject/block-qdict.c T: git git://repo.or.cz/qemu/kevin.git block Block I/O path diff --git a/qobject/Makefile.objs b/qobject/Makefile.objs index 002d25873a..7b12c9cacf 100644 --- a/qobject/Makefile.objs +++ b/qobject/Makefile.objs @@ -1,2 +1,3 @@ util-obj-y = qnull.o qnum.o qstring.o qdict.o qlist.o qbool.o qlit.o util-obj-y += qjson.o qobject.o json-lexer.o json-streamer.o json-parser.o +util-obj-y += block-qdict.o diff --git a/qobject/block-qdict.c b/qobject/block-qdict.c new file mode 100644 index 0000000000..01c453a8c0 --- /dev/null +++ b/qobject/block-qdict.c @@ -0,0 +1,637 @@ +/* + * Special QDict functions used by the block layer + * + * Copyright (c) 2013-2018 Red Hat, Inc. + * + * This work is licensed under the terms of the GNU LGPL, version 2.1 or later. + * See the COPYING.LIB file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "block/qdict.h" +#include "qapi/qmp/qlist.h" +#include "qemu/cutils.h" +#include "qapi/error.h" + +/** + * qdict_copy_default(): If no entry mapped by 'key' exists in 'dst' yet, the + * value of 'key' in 'src' is copied there (and the refcount increased + * accordingly). + */ +void qdict_copy_default(QDict *dst, QDict *src, const char *key) +{ + QObject *val; + + if (qdict_haskey(dst, key)) { + return; + } + + val = qdict_get(src, key); + if (val) { + qdict_put_obj(dst, key, qobject_ref(val)); + } +} + +/** + * qdict_set_default_str(): If no entry mapped by 'key' exists in 'dst' yet, a + * new QString initialised by 'val' is put there. + */ +void qdict_set_default_str(QDict *dst, const char *key, const char *val) +{ + if (qdict_haskey(dst, key)) { + return; + } + + qdict_put_str(dst, key, val); +} + +static void qdict_flatten_qdict(QDict *qdict, QDict *target, + const char *prefix); + +static void qdict_flatten_qlist(QList *qlist, QDict *target, const char *prefix) +{ + QObject *value; + const QListEntry *entry; + char *new_key; + int i; + + /* This function is never called with prefix == NULL, i.e., it is always + * called from within qdict_flatten_q(list|dict)(). Therefore, it does not + * need to remove list entries during the iteration (the whole list will be + * deleted eventually anyway from qdict_flatten_qdict()). */ + assert(prefix); + + entry = qlist_first(qlist); + + for (i = 0; entry; entry = qlist_next(entry), i++) { + value = qlist_entry_obj(entry); + new_key = g_strdup_printf("%s.%i", prefix, i); + + if (qobject_type(value) == QTYPE_QDICT) { + qdict_flatten_qdict(qobject_to(QDict, value), target, new_key); + } else if (qobject_type(value) == QTYPE_QLIST) { + qdict_flatten_qlist(qobject_to(QList, value), target, new_key); + } else { + /* All other types are moved to the target unchanged. */ + qdict_put_obj(target, new_key, qobject_ref(value)); + } + + g_free(new_key); + } +} + +static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) +{ + QObject *value; + const QDictEntry *entry, *next; + char *new_key; + bool delete; + + entry = qdict_first(qdict); + + while (entry != NULL) { + + next = qdict_next(qdict, entry); + value = qdict_entry_value(entry); + new_key = NULL; + delete = false; + + if (prefix) { + new_key = g_strdup_printf("%s.%s", prefix, entry->key); + } + + if (qobject_type(value) == QTYPE_QDICT) { + /* Entries of QDicts are processed recursively, the QDict object + * itself disappears. */ + qdict_flatten_qdict(qobject_to(QDict, value), target, + new_key ? new_key : entry->key); + delete = true; + } else if (qobject_type(value) == QTYPE_QLIST) { + qdict_flatten_qlist(qobject_to(QList, value), target, + new_key ? new_key : entry->key); + delete = true; + } else if (prefix) { + /* All other objects are moved to the target unchanged. */ + qdict_put_obj(target, new_key, qobject_ref(value)); + delete = true; + } + + g_free(new_key); + + if (delete) { + qdict_del(qdict, entry->key); + + /* Restart loop after modifying the iterated QDict */ + entry = qdict_first(qdict); + continue; + } + + entry = next; + } +} + +/** + * qdict_flatten(): For each nested QDict with key x, all fields with key y + * are moved to this QDict and their key is renamed to "x.y". For each nested + * QList with key x, the field at index y is moved to this QDict with the key + * "x.y" (i.e., the reverse of what qdict_array_split() does). + * This operation is applied recursively for nested QDicts and QLists. + */ +void qdict_flatten(QDict *qdict) +{ + qdict_flatten_qdict(qdict, qdict, NULL); +} + +/* extract all the src QDict entries starting by start into dst */ +void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start) + +{ + const QDictEntry *entry, *next; + const char *p; + + *dst = qdict_new(); + entry = qdict_first(src); + + while (entry != NULL) { + next = qdict_next(src, entry); + if (strstart(entry->key, start, &p)) { + qdict_put_obj(*dst, p, qobject_ref(entry->value)); + qdict_del(src, entry->key); + } + entry = next; + } +} + +static int qdict_count_prefixed_entries(const QDict *src, const char *start) +{ + const QDictEntry *entry; + int count = 0; + + for (entry = qdict_first(src); entry; entry = qdict_next(src, entry)) { + if (strstart(entry->key, start, NULL)) { + if (count == INT_MAX) { + return -ERANGE; + } + count++; + } + } + + return count; +} + +/** + * qdict_array_split(): This function moves array-like elements of a QDict into + * a new QList. Every entry in the original QDict with a key "%u" or one + * prefixed "%u.", where %u designates an unsigned integer starting at 0 and + * incrementally counting up, will be moved to a new QDict at index %u in the + * output QList with the key prefix removed, if that prefix is "%u.". If the + * whole key is just "%u", the whole QObject will be moved unchanged without + * creating a new QDict. The function terminates when there is no entry in the + * QDict with a prefix directly (incrementally) following the last one; it also + * returns if there are both entries with "%u" and "%u." for the same index %u. + * Example: {"0.a": 42, "0.b": 23, "1.x": 0, "4.y": 1, "o.o": 7, "2": 66} + * (or {"1.x": 0, "4.y": 1, "0.a": 42, "o.o": 7, "0.b": 23, "2": 66}) + * => [{"a": 42, "b": 23}, {"x": 0}, 66] + * and {"4.y": 1, "o.o": 7} (remainder of the old QDict) + */ +void qdict_array_split(QDict *src, QList **dst) +{ + unsigned i; + + *dst = qlist_new(); + + for (i = 0; i < UINT_MAX; i++) { + QObject *subqobj; + bool is_subqdict; + QDict *subqdict; + char indexstr[32], prefix[32]; + size_t snprintf_ret; + + snprintf_ret = snprintf(indexstr, 32, "%u", i); + assert(snprintf_ret < 32); + + subqobj = qdict_get(src, indexstr); + + snprintf_ret = snprintf(prefix, 32, "%u.", i); + assert(snprintf_ret < 32); + + /* Overflow is the same as positive non-zero results */ + is_subqdict = qdict_count_prefixed_entries(src, prefix); + + // There may be either a single subordinate object (named "%u") or + // multiple objects (each with a key prefixed "%u."), but not both. + if (!subqobj == !is_subqdict) { + break; + } + + if (is_subqdict) { + qdict_extract_subqdict(src, &subqdict, prefix); + assert(qdict_size(subqdict) > 0); + } else { + qobject_ref(subqobj); + qdict_del(src, indexstr); + } + + qlist_append_obj(*dst, subqobj ?: QOBJECT(subqdict)); + } +} + +/** + * qdict_split_flat_key: + * @key: the key string to split + * @prefix: non-NULL pointer to hold extracted prefix + * @suffix: non-NULL pointer to remaining suffix + * + * Given a flattened key such as 'foo.0.bar', split it into two parts + * at the first '.' separator. Allows double dot ('..') to escape the + * normal separator. + * + * e.g. + * 'foo.0.bar' -> prefix='foo' and suffix='0.bar' + * 'foo..0.bar' -> prefix='foo.0' and suffix='bar' + * + * The '..' sequence will be unescaped in the returned 'prefix' + * string. The 'suffix' string will be left in escaped format, so it + * can be fed back into the qdict_split_flat_key() key as the input + * later. + * + * The caller is responsible for freeing the string returned in @prefix + * using g_free(). + */ +static void qdict_split_flat_key(const char *key, char **prefix, + const char **suffix) +{ + const char *separator; + size_t i, j; + + /* Find first '.' separator, but if there is a pair '..' + * that acts as an escape, so skip over '..' */ + separator = NULL; + do { + if (separator) { + separator += 2; + } else { + separator = key; + } + separator = strchr(separator, '.'); + } while (separator && separator[1] == '.'); + + if (separator) { + *prefix = g_strndup(key, separator - key); + *suffix = separator + 1; + } else { + *prefix = g_strdup(key); + *suffix = NULL; + } + + /* Unescape the '..' sequence into '.' */ + for (i = 0, j = 0; (*prefix)[i] != '\0'; i++, j++) { + if ((*prefix)[i] == '.') { + assert((*prefix)[i + 1] == '.'); + i++; + } + (*prefix)[j] = (*prefix)[i]; + } + (*prefix)[j] = '\0'; +} + +/** + * qdict_is_list: + * @maybe_list: dict to check if keys represent list elements. + * + * Determine whether all keys in @maybe_list are valid list elements. + * If @maybe_list is non-zero in length and all the keys look like + * valid list indexes, this will return 1. If @maybe_list is zero + * length or all keys are non-numeric then it will return 0 to indicate + * it is a normal qdict. If there is a mix of numeric and non-numeric + * keys, or the list indexes are non-contiguous, an error is reported. + * + * Returns: 1 if a valid list, 0 if a dict, -1 on error + */ +static int qdict_is_list(QDict *maybe_list, Error **errp) +{ + const QDictEntry *ent; + ssize_t len = 0; + ssize_t max = -1; + int is_list = -1; + int64_t val; + + for (ent = qdict_first(maybe_list); ent != NULL; + ent = qdict_next(maybe_list, ent)) { + + if (qemu_strtoi64(ent->key, NULL, 10, &val) == 0) { + if (is_list == -1) { + is_list = 1; + } else if (!is_list) { + error_setg(errp, + "Cannot mix list and non-list keys"); + return -1; + } + len++; + if (val > max) { + max = val; + } + } else { + if (is_list == -1) { + is_list = 0; + } else if (is_list) { + error_setg(errp, + "Cannot mix list and non-list keys"); + return -1; + } + } + } + + if (is_list == -1) { + assert(!qdict_size(maybe_list)); + is_list = 0; + } + + /* NB this isn't a perfect check - e.g. it won't catch + * a list containing '1', '+1', '01', '3', but that + * does not matter - we've still proved that the + * input is a list. It is up the caller to do a + * stricter check if desired */ + if (len != (max + 1)) { + error_setg(errp, "List indices are not contiguous, " + "saw %zd elements but %zd largest index", + len, max); + return -1; + } + + return is_list; +} + +/** + * qdict_crumple: + * @src: the original flat dictionary (only scalar values) to crumple + * + * Takes a flat dictionary whose keys use '.' separator to indicate + * nesting, and values are scalars, and crumples it into a nested + * structure. + * + * To include a literal '.' in a key name, it must be escaped as '..' + * + * For example, an input of: + * + * { 'foo.0.bar': 'one', 'foo.0.wizz': '1', + * 'foo.1.bar': 'two', 'foo.1.wizz': '2' } + * + * will result in an output of: + * + * { + * 'foo': [ + * { 'bar': 'one', 'wizz': '1' }, + * { 'bar': 'two', 'wizz': '2' } + * ], + * } + * + * The following scenarios in the input dict will result in an + * error being returned: + * + * - Any values in @src are non-scalar types + * - If keys in @src imply that a particular level is both a + * list and a dict. e.g., "foo.0.bar" and "foo.eek.bar". + * - If keys in @src imply that a particular level is a list, + * but the indices are non-contiguous. e.g. "foo.0.bar" and + * "foo.2.bar" without any "foo.1.bar" present. + * - If keys in @src represent list indexes, but are not in + * the "%zu" format. e.g. "foo.+0.bar" + * + * Returns: either a QDict or QList for the nested data structure, or NULL + * on error + */ +QObject *qdict_crumple(const QDict *src, Error **errp) +{ + const QDictEntry *ent; + QDict *two_level, *multi_level = NULL; + QObject *dst = NULL, *child; + size_t i; + char *prefix = NULL; + const char *suffix = NULL; + int is_list; + + two_level = qdict_new(); + + /* Step 1: split our totally flat dict into a two level dict */ + for (ent = qdict_first(src); ent != NULL; ent = qdict_next(src, ent)) { + if (qobject_type(ent->value) == QTYPE_QDICT || + qobject_type(ent->value) == QTYPE_QLIST) { + error_setg(errp, "Value %s is not a scalar", + ent->key); + goto error; + } + + qdict_split_flat_key(ent->key, &prefix, &suffix); + + child = qdict_get(two_level, prefix); + if (suffix) { + QDict *child_dict = qobject_to(QDict, child); + if (!child_dict) { + if (child) { + error_setg(errp, "Key %s prefix is already set as a scalar", + prefix); + goto error; + } + + child_dict = qdict_new(); + qdict_put_obj(two_level, prefix, QOBJECT(child_dict)); + } + + qdict_put_obj(child_dict, suffix, qobject_ref(ent->value)); + } else { + if (child) { + error_setg(errp, "Key %s prefix is already set as a dict", + prefix); + goto error; + } + qdict_put_obj(two_level, prefix, qobject_ref(ent->value)); + } + + g_free(prefix); + prefix = NULL; + } + + /* Step 2: optionally process the two level dict recursively + * into a multi-level dict */ + multi_level = qdict_new(); + for (ent = qdict_first(two_level); ent != NULL; + ent = qdict_next(two_level, ent)) { + QDict *dict = qobject_to(QDict, ent->value); + if (dict) { + child = qdict_crumple(dict, errp); + if (!child) { + goto error; + } + + qdict_put_obj(multi_level, ent->key, child); + } else { + qdict_put_obj(multi_level, ent->key, qobject_ref(ent->value)); + } + } + qobject_unref(two_level); + two_level = NULL; + + /* Step 3: detect if we need to turn our dict into list */ + is_list = qdict_is_list(multi_level, errp); + if (is_list < 0) { + goto error; + } + + if (is_list) { + dst = QOBJECT(qlist_new()); + + for (i = 0; i < qdict_size(multi_level); i++) { + char *key = g_strdup_printf("%zu", i); + + child = qdict_get(multi_level, key); + g_free(key); + + if (!child) { + error_setg(errp, "Missing list index %zu", i); + goto error; + } + + qlist_append_obj(qobject_to(QList, dst), qobject_ref(child)); + } + qobject_unref(multi_level); + multi_level = NULL; + } else { + dst = QOBJECT(multi_level); + } + + return dst; + + error: + g_free(prefix); + qobject_unref(multi_level); + qobject_unref(two_level); + qobject_unref(dst); + return NULL; +} + +/** + * qdict_array_entries(): Returns the number of direct array entries if the + * sub-QDict of src specified by the prefix in subqdict (or src itself for + * prefix == "") is valid as an array, i.e. the length of the created list if + * the sub-QDict would become empty after calling qdict_array_split() on it. If + * the array is not valid, -EINVAL is returned. + */ +int qdict_array_entries(QDict *src, const char *subqdict) +{ + const QDictEntry *entry; + unsigned i; + unsigned entries = 0; + size_t subqdict_len = strlen(subqdict); + + assert(!subqdict_len || subqdict[subqdict_len - 1] == '.'); + + /* qdict_array_split() loops until UINT_MAX, but as we want to return + * negative errors, we only have a signed return value here. Any additional + * entries will lead to -EINVAL. */ + for (i = 0; i < INT_MAX; i++) { + QObject *subqobj; + int subqdict_entries; + char *prefix = g_strdup_printf("%s%u.", subqdict, i); + + subqdict_entries = qdict_count_prefixed_entries(src, prefix); + + /* Remove ending "." */ + prefix[strlen(prefix) - 1] = 0; + subqobj = qdict_get(src, prefix); + + g_free(prefix); + + if (subqdict_entries < 0) { + return subqdict_entries; + } + + /* There may be either a single subordinate object (named "%u") or + * multiple objects (each with a key prefixed "%u."), but not both. */ + if (subqobj && subqdict_entries) { + return -EINVAL; + } else if (!subqobj && !subqdict_entries) { + break; + } + + entries += subqdict_entries ? subqdict_entries : 1; + } + + /* Consider everything handled that isn't part of the given sub-QDict */ + for (entry = qdict_first(src); entry; entry = qdict_next(src, entry)) { + if (!strstart(qdict_entry_key(entry), subqdict, NULL)) { + entries++; + } + } + + /* Anything left in the sub-QDict that wasn't handled? */ + if (qdict_size(src) != entries) { + return -EINVAL; + } + + return i; +} + +/** + * qdict_join(): Absorb the src QDict into the dest QDict, that is, move all + * elements from src to dest. + * + * If an element from src has a key already present in dest, it will not be + * moved unless overwrite is true. + * + * If overwrite is true, the conflicting values in dest will be discarded and + * replaced by the corresponding values from src. + * + * Therefore, with overwrite being true, the src QDict will always be empty when + * this function returns. If overwrite is false, the src QDict will be empty + * iff there were no conflicts. + */ +void qdict_join(QDict *dest, QDict *src, bool overwrite) +{ + const QDictEntry *entry, *next; + + entry = qdict_first(src); + while (entry) { + next = qdict_next(src, entry); + + if (overwrite || !qdict_haskey(dest, entry->key)) { + qdict_put_obj(dest, entry->key, qobject_ref(entry->value)); + qdict_del(src, entry->key); + } + + entry = next; + } +} + +/** + * qdict_rename_keys(): Rename keys in qdict according to the replacements + * specified in the array renames. The array must be terminated by an entry + * with from = NULL. + * + * The renames are performed individually in the order of the array, so entries + * may be renamed multiple times and may or may not conflict depending on the + * order of the renames array. + * + * Returns true for success, false in error cases. + */ +bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) +{ + QObject *qobj; + + while (renames->from) { + if (qdict_haskey(qdict, renames->from)) { + if (qdict_haskey(qdict, renames->to)) { + error_setg(errp, "'%s' and its alias '%s' can't be used at the " + "same time", renames->to, renames->from); + return false; + } + + qobj = qdict_get(qdict, renames->from); + qdict_put_obj(qdict, renames->to, qobject_ref(qobj)); + qdict_del(qdict, renames->from); + } + + renames++; + } + return true; +} diff --git a/qobject/qdict.c b/qobject/qdict.c index 0554c64553..3d8c2f7bbc 100644 --- a/qobject/qdict.c +++ b/qobject/qdict.c @@ -11,17 +11,11 @@ */ #include "qemu/osdep.h" -#include "block/qdict.h" #include "qapi/qmp/qnum.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qbool.h" -#include "qapi/qmp/qlist.h" #include "qapi/qmp/qnull.h" #include "qapi/qmp/qstring.h" -#include "qapi/error.h" -#include "qemu/queue.h" -#include "qemu-common.h" -#include "qemu/cutils.h" /** * qdict_new(): Create a new QDict @@ -464,626 +458,3 @@ void qdict_destroy_obj(QObject *obj) g_free(qdict); } - -/** - * qdict_copy_default(): If no entry mapped by 'key' exists in 'dst' yet, the - * value of 'key' in 'src' is copied there (and the refcount increased - * accordingly). - */ -void qdict_copy_default(QDict *dst, QDict *src, const char *key) -{ - QObject *val; - - if (qdict_haskey(dst, key)) { - return; - } - - val = qdict_get(src, key); - if (val) { - qdict_put_obj(dst, key, qobject_ref(val)); - } -} - -/** - * qdict_set_default_str(): If no entry mapped by 'key' exists in 'dst' yet, a - * new QString initialised by 'val' is put there. - */ -void qdict_set_default_str(QDict *dst, const char *key, const char *val) -{ - if (qdict_haskey(dst, key)) { - return; - } - - qdict_put_str(dst, key, val); -} - -static void qdict_flatten_qdict(QDict *qdict, QDict *target, - const char *prefix); - -static void qdict_flatten_qlist(QList *qlist, QDict *target, const char *prefix) -{ - QObject *value; - const QListEntry *entry; - char *new_key; - int i; - - /* This function is never called with prefix == NULL, i.e., it is always - * called from within qdict_flatten_q(list|dict)(). Therefore, it does not - * need to remove list entries during the iteration (the whole list will be - * deleted eventually anyway from qdict_flatten_qdict()). */ - assert(prefix); - - entry = qlist_first(qlist); - - for (i = 0; entry; entry = qlist_next(entry), i++) { - value = qlist_entry_obj(entry); - new_key = g_strdup_printf("%s.%i", prefix, i); - - if (qobject_type(value) == QTYPE_QDICT) { - qdict_flatten_qdict(qobject_to(QDict, value), target, new_key); - } else if (qobject_type(value) == QTYPE_QLIST) { - qdict_flatten_qlist(qobject_to(QList, value), target, new_key); - } else { - /* All other types are moved to the target unchanged. */ - qdict_put_obj(target, new_key, qobject_ref(value)); - } - - g_free(new_key); - } -} - -static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) -{ - QObject *value; - const QDictEntry *entry, *next; - char *new_key; - bool delete; - - entry = qdict_first(qdict); - - while (entry != NULL) { - - next = qdict_next(qdict, entry); - value = qdict_entry_value(entry); - new_key = NULL; - delete = false; - - if (prefix) { - new_key = g_strdup_printf("%s.%s", prefix, entry->key); - } - - if (qobject_type(value) == QTYPE_QDICT) { - /* Entries of QDicts are processed recursively, the QDict object - * itself disappears. */ - qdict_flatten_qdict(qobject_to(QDict, value), target, - new_key ? new_key : entry->key); - delete = true; - } else if (qobject_type(value) == QTYPE_QLIST) { - qdict_flatten_qlist(qobject_to(QList, value), target, - new_key ? new_key : entry->key); - delete = true; - } else if (prefix) { - /* All other objects are moved to the target unchanged. */ - qdict_put_obj(target, new_key, qobject_ref(value)); - delete = true; - } - - g_free(new_key); - - if (delete) { - qdict_del(qdict, entry->key); - - /* Restart loop after modifying the iterated QDict */ - entry = qdict_first(qdict); - continue; - } - - entry = next; - } -} - -/** - * qdict_flatten(): For each nested QDict with key x, all fields with key y - * are moved to this QDict and their key is renamed to "x.y". For each nested - * QList with key x, the field at index y is moved to this QDict with the key - * "x.y" (i.e., the reverse of what qdict_array_split() does). - * This operation is applied recursively for nested QDicts and QLists. - */ -void qdict_flatten(QDict *qdict) -{ - qdict_flatten_qdict(qdict, qdict, NULL); -} - -/* extract all the src QDict entries starting by start into dst */ -void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start) - -{ - const QDictEntry *entry, *next; - const char *p; - - *dst = qdict_new(); - entry = qdict_first(src); - - while (entry != NULL) { - next = qdict_next(src, entry); - if (strstart(entry->key, start, &p)) { - qdict_put_obj(*dst, p, qobject_ref(entry->value)); - qdict_del(src, entry->key); - } - entry = next; - } -} - -static int qdict_count_prefixed_entries(const QDict *src, const char *start) -{ - const QDictEntry *entry; - int count = 0; - - for (entry = qdict_first(src); entry; entry = qdict_next(src, entry)) { - if (strstart(entry->key, start, NULL)) { - if (count == INT_MAX) { - return -ERANGE; - } - count++; - } - } - - return count; -} - -/** - * qdict_array_split(): This function moves array-like elements of a QDict into - * a new QList. Every entry in the original QDict with a key "%u" or one - * prefixed "%u.", where %u designates an unsigned integer starting at 0 and - * incrementally counting up, will be moved to a new QDict at index %u in the - * output QList with the key prefix removed, if that prefix is "%u.". If the - * whole key is just "%u", the whole QObject will be moved unchanged without - * creating a new QDict. The function terminates when there is no entry in the - * QDict with a prefix directly (incrementally) following the last one; it also - * returns if there are both entries with "%u" and "%u." for the same index %u. - * Example: {"0.a": 42, "0.b": 23, "1.x": 0, "4.y": 1, "o.o": 7, "2": 66} - * (or {"1.x": 0, "4.y": 1, "0.a": 42, "o.o": 7, "0.b": 23, "2": 66}) - * => [{"a": 42, "b": 23}, {"x": 0}, 66] - * and {"4.y": 1, "o.o": 7} (remainder of the old QDict) - */ -void qdict_array_split(QDict *src, QList **dst) -{ - unsigned i; - - *dst = qlist_new(); - - for (i = 0; i < UINT_MAX; i++) { - QObject *subqobj; - bool is_subqdict; - QDict *subqdict; - char indexstr[32], prefix[32]; - size_t snprintf_ret; - - snprintf_ret = snprintf(indexstr, 32, "%u", i); - assert(snprintf_ret < 32); - - subqobj = qdict_get(src, indexstr); - - snprintf_ret = snprintf(prefix, 32, "%u.", i); - assert(snprintf_ret < 32); - - /* Overflow is the same as positive non-zero results */ - is_subqdict = qdict_count_prefixed_entries(src, prefix); - - // There may be either a single subordinate object (named "%u") or - // multiple objects (each with a key prefixed "%u."), but not both. - if (!subqobj == !is_subqdict) { - break; - } - - if (is_subqdict) { - qdict_extract_subqdict(src, &subqdict, prefix); - assert(qdict_size(subqdict) > 0); - } else { - qobject_ref(subqobj); - qdict_del(src, indexstr); - } - - qlist_append_obj(*dst, subqobj ?: QOBJECT(subqdict)); - } -} - -/** - * qdict_split_flat_key: - * @key: the key string to split - * @prefix: non-NULL pointer to hold extracted prefix - * @suffix: non-NULL pointer to remaining suffix - * - * Given a flattened key such as 'foo.0.bar', split it into two parts - * at the first '.' separator. Allows double dot ('..') to escape the - * normal separator. - * - * e.g. - * 'foo.0.bar' -> prefix='foo' and suffix='0.bar' - * 'foo..0.bar' -> prefix='foo.0' and suffix='bar' - * - * The '..' sequence will be unescaped in the returned 'prefix' - * string. The 'suffix' string will be left in escaped format, so it - * can be fed back into the qdict_split_flat_key() key as the input - * later. - * - * The caller is responsible for freeing the string returned in @prefix - * using g_free(). - */ -static void qdict_split_flat_key(const char *key, char **prefix, - const char **suffix) -{ - const char *separator; - size_t i, j; - - /* Find first '.' separator, but if there is a pair '..' - * that acts as an escape, so skip over '..' */ - separator = NULL; - do { - if (separator) { - separator += 2; - } else { - separator = key; - } - separator = strchr(separator, '.'); - } while (separator && separator[1] == '.'); - - if (separator) { - *prefix = g_strndup(key, separator - key); - *suffix = separator + 1; - } else { - *prefix = g_strdup(key); - *suffix = NULL; - } - - /* Unescape the '..' sequence into '.' */ - for (i = 0, j = 0; (*prefix)[i] != '\0'; i++, j++) { - if ((*prefix)[i] == '.') { - assert((*prefix)[i + 1] == '.'); - i++; - } - (*prefix)[j] = (*prefix)[i]; - } - (*prefix)[j] = '\0'; -} - -/** - * qdict_is_list: - * @maybe_list: dict to check if keys represent list elements. - * - * Determine whether all keys in @maybe_list are valid list elements. - * If @maybe_list is non-zero in length and all the keys look like - * valid list indexes, this will return 1. If @maybe_list is zero - * length or all keys are non-numeric then it will return 0 to indicate - * it is a normal qdict. If there is a mix of numeric and non-numeric - * keys, or the list indexes are non-contiguous, an error is reported. - * - * Returns: 1 if a valid list, 0 if a dict, -1 on error - */ -static int qdict_is_list(QDict *maybe_list, Error **errp) -{ - const QDictEntry *ent; - ssize_t len = 0; - ssize_t max = -1; - int is_list = -1; - int64_t val; - - for (ent = qdict_first(maybe_list); ent != NULL; - ent = qdict_next(maybe_list, ent)) { - - if (qemu_strtoi64(ent->key, NULL, 10, &val) == 0) { - if (is_list == -1) { - is_list = 1; - } else if (!is_list) { - error_setg(errp, - "Cannot mix list and non-list keys"); - return -1; - } - len++; - if (val > max) { - max = val; - } - } else { - if (is_list == -1) { - is_list = 0; - } else if (is_list) { - error_setg(errp, - "Cannot mix list and non-list keys"); - return -1; - } - } - } - - if (is_list == -1) { - assert(!qdict_size(maybe_list)); - is_list = 0; - } - - /* NB this isn't a perfect check - e.g. it won't catch - * a list containing '1', '+1', '01', '3', but that - * does not matter - we've still proved that the - * input is a list. It is up the caller to do a - * stricter check if desired */ - if (len != (max + 1)) { - error_setg(errp, "List indices are not contiguous, " - "saw %zd elements but %zd largest index", - len, max); - return -1; - } - - return is_list; -} - -/** - * qdict_crumple: - * @src: the original flat dictionary (only scalar values) to crumple - * - * Takes a flat dictionary whose keys use '.' separator to indicate - * nesting, and values are scalars, and crumples it into a nested - * structure. - * - * To include a literal '.' in a key name, it must be escaped as '..' - * - * For example, an input of: - * - * { 'foo.0.bar': 'one', 'foo.0.wizz': '1', - * 'foo.1.bar': 'two', 'foo.1.wizz': '2' } - * - * will result in an output of: - * - * { - * 'foo': [ - * { 'bar': 'one', 'wizz': '1' }, - * { 'bar': 'two', 'wizz': '2' } - * ], - * } - * - * The following scenarios in the input dict will result in an - * error being returned: - * - * - Any values in @src are non-scalar types - * - If keys in @src imply that a particular level is both a - * list and a dict. e.g., "foo.0.bar" and "foo.eek.bar". - * - If keys in @src imply that a particular level is a list, - * but the indices are non-contiguous. e.g. "foo.0.bar" and - * "foo.2.bar" without any "foo.1.bar" present. - * - If keys in @src represent list indexes, but are not in - * the "%zu" format. e.g. "foo.+0.bar" - * - * Returns: either a QDict or QList for the nested data structure, or NULL - * on error - */ -QObject *qdict_crumple(const QDict *src, Error **errp) -{ - const QDictEntry *ent; - QDict *two_level, *multi_level = NULL; - QObject *dst = NULL, *child; - size_t i; - char *prefix = NULL; - const char *suffix = NULL; - int is_list; - - two_level = qdict_new(); - - /* Step 1: split our totally flat dict into a two level dict */ - for (ent = qdict_first(src); ent != NULL; ent = qdict_next(src, ent)) { - if (qobject_type(ent->value) == QTYPE_QDICT || - qobject_type(ent->value) == QTYPE_QLIST) { - error_setg(errp, "Value %s is not a scalar", - ent->key); - goto error; - } - - qdict_split_flat_key(ent->key, &prefix, &suffix); - - child = qdict_get(two_level, prefix); - if (suffix) { - QDict *child_dict = qobject_to(QDict, child); - if (!child_dict) { - if (child) { - error_setg(errp, "Key %s prefix is already set as a scalar", - prefix); - goto error; - } - - child_dict = qdict_new(); - qdict_put_obj(two_level, prefix, QOBJECT(child_dict)); - } - - qdict_put_obj(child_dict, suffix, qobject_ref(ent->value)); - } else { - if (child) { - error_setg(errp, "Key %s prefix is already set as a dict", - prefix); - goto error; - } - qdict_put_obj(two_level, prefix, qobject_ref(ent->value)); - } - - g_free(prefix); - prefix = NULL; - } - - /* Step 2: optionally process the two level dict recursively - * into a multi-level dict */ - multi_level = qdict_new(); - for (ent = qdict_first(two_level); ent != NULL; - ent = qdict_next(two_level, ent)) { - QDict *dict = qobject_to(QDict, ent->value); - if (dict) { - child = qdict_crumple(dict, errp); - if (!child) { - goto error; - } - - qdict_put_obj(multi_level, ent->key, child); - } else { - qdict_put_obj(multi_level, ent->key, qobject_ref(ent->value)); - } - } - qobject_unref(two_level); - two_level = NULL; - - /* Step 3: detect if we need to turn our dict into list */ - is_list = qdict_is_list(multi_level, errp); - if (is_list < 0) { - goto error; - } - - if (is_list) { - dst = QOBJECT(qlist_new()); - - for (i = 0; i < qdict_size(multi_level); i++) { - char *key = g_strdup_printf("%zu", i); - - child = qdict_get(multi_level, key); - g_free(key); - - if (!child) { - error_setg(errp, "Missing list index %zu", i); - goto error; - } - - qlist_append_obj(qobject_to(QList, dst), qobject_ref(child)); - } - qobject_unref(multi_level); - multi_level = NULL; - } else { - dst = QOBJECT(multi_level); - } - - return dst; - - error: - g_free(prefix); - qobject_unref(multi_level); - qobject_unref(two_level); - qobject_unref(dst); - return NULL; -} - -/** - * qdict_array_entries(): Returns the number of direct array entries if the - * sub-QDict of src specified by the prefix in subqdict (or src itself for - * prefix == "") is valid as an array, i.e. the length of the created list if - * the sub-QDict would become empty after calling qdict_array_split() on it. If - * the array is not valid, -EINVAL is returned. - */ -int qdict_array_entries(QDict *src, const char *subqdict) -{ - const QDictEntry *entry; - unsigned i; - unsigned entries = 0; - size_t subqdict_len = strlen(subqdict); - - assert(!subqdict_len || subqdict[subqdict_len - 1] == '.'); - - /* qdict_array_split() loops until UINT_MAX, but as we want to return - * negative errors, we only have a signed return value here. Any additional - * entries will lead to -EINVAL. */ - for (i = 0; i < INT_MAX; i++) { - QObject *subqobj; - int subqdict_entries; - char *prefix = g_strdup_printf("%s%u.", subqdict, i); - - subqdict_entries = qdict_count_prefixed_entries(src, prefix); - - /* Remove ending "." */ - prefix[strlen(prefix) - 1] = 0; - subqobj = qdict_get(src, prefix); - - g_free(prefix); - - if (subqdict_entries < 0) { - return subqdict_entries; - } - - /* There may be either a single subordinate object (named "%u") or - * multiple objects (each with a key prefixed "%u."), but not both. */ - if (subqobj && subqdict_entries) { - return -EINVAL; - } else if (!subqobj && !subqdict_entries) { - break; - } - - entries += subqdict_entries ? subqdict_entries : 1; - } - - /* Consider everything handled that isn't part of the given sub-QDict */ - for (entry = qdict_first(src); entry; entry = qdict_next(src, entry)) { - if (!strstart(qdict_entry_key(entry), subqdict, NULL)) { - entries++; - } - } - - /* Anything left in the sub-QDict that wasn't handled? */ - if (qdict_size(src) != entries) { - return -EINVAL; - } - - return i; -} - -/** - * qdict_join(): Absorb the src QDict into the dest QDict, that is, move all - * elements from src to dest. - * - * If an element from src has a key already present in dest, it will not be - * moved unless overwrite is true. - * - * If overwrite is true, the conflicting values in dest will be discarded and - * replaced by the corresponding values from src. - * - * Therefore, with overwrite being true, the src QDict will always be empty when - * this function returns. If overwrite is false, the src QDict will be empty - * iff there were no conflicts. - */ -void qdict_join(QDict *dest, QDict *src, bool overwrite) -{ - const QDictEntry *entry, *next; - - entry = qdict_first(src); - while (entry) { - next = qdict_next(src, entry); - - if (overwrite || !qdict_haskey(dest, entry->key)) { - qdict_put_obj(dest, entry->key, qobject_ref(entry->value)); - qdict_del(src, entry->key); - } - - entry = next; - } -} - -/** - * qdict_rename_keys(): Rename keys in qdict according to the replacements - * specified in the array renames. The array must be terminated by an entry - * with from = NULL. - * - * The renames are performed individually in the order of the array, so entries - * may be renamed multiple times and may or may not conflict depending on the - * order of the renames array. - * - * Returns true for success, false in error cases. - */ -bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) -{ - QObject *qobj; - - while (renames->from) { - if (qdict_haskey(qdict, renames->from)) { - if (qdict_haskey(qdict, renames->to)) { - error_setg(errp, "'%s' and its alias '%s' can't be used at the " - "same time", renames->to, renames->from); - return false; - } - - qobj = qdict_get(qdict, renames->from); - qdict_put_obj(qdict, renames->to, qobject_ref(qobj)); - qdict_del(qdict, renames->from); - } - - renames++; - } - return true; -} -- 2.17.1 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header 2018-06-06 13:17 ` Markus Armbruster @ 2018-06-06 14:19 ` Markus Armbruster 2018-06-06 14:35 ` Markus Armbruster 0 siblings, 1 reply; 42+ messages in thread From: Markus Armbruster @ 2018-06-06 14:19 UTC (permalink / raw) Cc: qemu-block, Kevin Wolf, qemu-devel, Michael Roth Markus Armbruster <armbru@redhat.com> writes: > MAINTAINERS files include/block/qdict.h under "Block layer core". Let's > split qobject/qdict.c as well, so it's there, too. Obvious patch > appended. Feel free to squash it into yours without giving me credit > for it. Hmm, we should split tests/check-qdict.c for the same reason. Updated patch appended. > PS: I tried to move block-qdict.c to block/, but totally failed at > persuading Make to link it into all programs that need it. Oh well. >From bdb7f6581a3066db853bca6c81cc2c42f6cec8e9 Mon Sep 17 00:00:00 2001 From: Markus Armbruster <armbru@redhat.com> Date: Wed, 6 Jun 2018 15:11:54 +0200 Subject: [PATCH] qobject: Move block-specific qdict code to block-qdict.c Signed-off-by: Markus Armbruster <armbru@redhat.com> --- MAINTAINERS | 1 + qobject/Makefile.objs | 1 + qobject/block-qdict.c | 637 ++++++++++++++++++++++++++++++++++++ qobject/qdict.c | 629 ------------------------------------ tests/Makefile.include | 4 + tests/check-block-qdict.c | 657 ++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 642 ------------------------------------- 7 files changed, 1300 insertions(+), 1271 deletions(-) create mode 100644 qobject/block-qdict.c create mode 100644 tests/check-block-qdict.c diff --git a/MAINTAINERS b/MAINTAINERS index 41cd3736a9..9f9835e052 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1365,6 +1365,7 @@ F: qemu-img* F: qemu-io* F: tests/qemu-iotests/ F: util/qemu-progress.c +F: qobject/block-qdict.c T: git git://repo.or.cz/qemu/kevin.git block Block I/O path diff --git a/qobject/Makefile.objs b/qobject/Makefile.objs index 002d25873a..7b12c9cacf 100644 --- a/qobject/Makefile.objs +++ b/qobject/Makefile.objs @@ -1,2 +1,3 @@ util-obj-y = qnull.o qnum.o qstring.o qdict.o qlist.o qbool.o qlit.o util-obj-y += qjson.o qobject.o json-lexer.o json-streamer.o json-parser.o +util-obj-y += block-qdict.o diff --git a/qobject/block-qdict.c b/qobject/block-qdict.c new file mode 100644 index 0000000000..01c453a8c0 --- /dev/null +++ b/qobject/block-qdict.c @@ -0,0 +1,637 @@ +/* + * Special QDict functions used by the block layer + * + * Copyright (c) 2013-2018 Red Hat, Inc. + * + * This work is licensed under the terms of the GNU LGPL, version 2.1 or later. + * See the COPYING.LIB file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "block/qdict.h" +#include "qapi/qmp/qlist.h" +#include "qemu/cutils.h" +#include "qapi/error.h" + +/** + * qdict_copy_default(): If no entry mapped by 'key' exists in 'dst' yet, the + * value of 'key' in 'src' is copied there (and the refcount increased + * accordingly). + */ +void qdict_copy_default(QDict *dst, QDict *src, const char *key) +{ + QObject *val; + + if (qdict_haskey(dst, key)) { + return; + } + + val = qdict_get(src, key); + if (val) { + qdict_put_obj(dst, key, qobject_ref(val)); + } +} + +/** + * qdict_set_default_str(): If no entry mapped by 'key' exists in 'dst' yet, a + * new QString initialised by 'val' is put there. + */ +void qdict_set_default_str(QDict *dst, const char *key, const char *val) +{ + if (qdict_haskey(dst, key)) { + return; + } + + qdict_put_str(dst, key, val); +} + +static void qdict_flatten_qdict(QDict *qdict, QDict *target, + const char *prefix); + +static void qdict_flatten_qlist(QList *qlist, QDict *target, const char *prefix) +{ + QObject *value; + const QListEntry *entry; + char *new_key; + int i; + + /* This function is never called with prefix == NULL, i.e., it is always + * called from within qdict_flatten_q(list|dict)(). Therefore, it does not + * need to remove list entries during the iteration (the whole list will be + * deleted eventually anyway from qdict_flatten_qdict()). */ + assert(prefix); + + entry = qlist_first(qlist); + + for (i = 0; entry; entry = qlist_next(entry), i++) { + value = qlist_entry_obj(entry); + new_key = g_strdup_printf("%s.%i", prefix, i); + + if (qobject_type(value) == QTYPE_QDICT) { + qdict_flatten_qdict(qobject_to(QDict, value), target, new_key); + } else if (qobject_type(value) == QTYPE_QLIST) { + qdict_flatten_qlist(qobject_to(QList, value), target, new_key); + } else { + /* All other types are moved to the target unchanged. */ + qdict_put_obj(target, new_key, qobject_ref(value)); + } + + g_free(new_key); + } +} + +static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) +{ + QObject *value; + const QDictEntry *entry, *next; + char *new_key; + bool delete; + + entry = qdict_first(qdict); + + while (entry != NULL) { + + next = qdict_next(qdict, entry); + value = qdict_entry_value(entry); + new_key = NULL; + delete = false; + + if (prefix) { + new_key = g_strdup_printf("%s.%s", prefix, entry->key); + } + + if (qobject_type(value) == QTYPE_QDICT) { + /* Entries of QDicts are processed recursively, the QDict object + * itself disappears. */ + qdict_flatten_qdict(qobject_to(QDict, value), target, + new_key ? new_key : entry->key); + delete = true; + } else if (qobject_type(value) == QTYPE_QLIST) { + qdict_flatten_qlist(qobject_to(QList, value), target, + new_key ? new_key : entry->key); + delete = true; + } else if (prefix) { + /* All other objects are moved to the target unchanged. */ + qdict_put_obj(target, new_key, qobject_ref(value)); + delete = true; + } + + g_free(new_key); + + if (delete) { + qdict_del(qdict, entry->key); + + /* Restart loop after modifying the iterated QDict */ + entry = qdict_first(qdict); + continue; + } + + entry = next; + } +} + +/** + * qdict_flatten(): For each nested QDict with key x, all fields with key y + * are moved to this QDict and their key is renamed to "x.y". For each nested + * QList with key x, the field at index y is moved to this QDict with the key + * "x.y" (i.e., the reverse of what qdict_array_split() does). + * This operation is applied recursively for nested QDicts and QLists. + */ +void qdict_flatten(QDict *qdict) +{ + qdict_flatten_qdict(qdict, qdict, NULL); +} + +/* extract all the src QDict entries starting by start into dst */ +void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start) + +{ + const QDictEntry *entry, *next; + const char *p; + + *dst = qdict_new(); + entry = qdict_first(src); + + while (entry != NULL) { + next = qdict_next(src, entry); + if (strstart(entry->key, start, &p)) { + qdict_put_obj(*dst, p, qobject_ref(entry->value)); + qdict_del(src, entry->key); + } + entry = next; + } +} + +static int qdict_count_prefixed_entries(const QDict *src, const char *start) +{ + const QDictEntry *entry; + int count = 0; + + for (entry = qdict_first(src); entry; entry = qdict_next(src, entry)) { + if (strstart(entry->key, start, NULL)) { + if (count == INT_MAX) { + return -ERANGE; + } + count++; + } + } + + return count; +} + +/** + * qdict_array_split(): This function moves array-like elements of a QDict into + * a new QList. Every entry in the original QDict with a key "%u" or one + * prefixed "%u.", where %u designates an unsigned integer starting at 0 and + * incrementally counting up, will be moved to a new QDict at index %u in the + * output QList with the key prefix removed, if that prefix is "%u.". If the + * whole key is just "%u", the whole QObject will be moved unchanged without + * creating a new QDict. The function terminates when there is no entry in the + * QDict with a prefix directly (incrementally) following the last one; it also + * returns if there are both entries with "%u" and "%u." for the same index %u. + * Example: {"0.a": 42, "0.b": 23, "1.x": 0, "4.y": 1, "o.o": 7, "2": 66} + * (or {"1.x": 0, "4.y": 1, "0.a": 42, "o.o": 7, "0.b": 23, "2": 66}) + * => [{"a": 42, "b": 23}, {"x": 0}, 66] + * and {"4.y": 1, "o.o": 7} (remainder of the old QDict) + */ +void qdict_array_split(QDict *src, QList **dst) +{ + unsigned i; + + *dst = qlist_new(); + + for (i = 0; i < UINT_MAX; i++) { + QObject *subqobj; + bool is_subqdict; + QDict *subqdict; + char indexstr[32], prefix[32]; + size_t snprintf_ret; + + snprintf_ret = snprintf(indexstr, 32, "%u", i); + assert(snprintf_ret < 32); + + subqobj = qdict_get(src, indexstr); + + snprintf_ret = snprintf(prefix, 32, "%u.", i); + assert(snprintf_ret < 32); + + /* Overflow is the same as positive non-zero results */ + is_subqdict = qdict_count_prefixed_entries(src, prefix); + + // There may be either a single subordinate object (named "%u") or + // multiple objects (each with a key prefixed "%u."), but not both. + if (!subqobj == !is_subqdict) { + break; + } + + if (is_subqdict) { + qdict_extract_subqdict(src, &subqdict, prefix); + assert(qdict_size(subqdict) > 0); + } else { + qobject_ref(subqobj); + qdict_del(src, indexstr); + } + + qlist_append_obj(*dst, subqobj ?: QOBJECT(subqdict)); + } +} + +/** + * qdict_split_flat_key: + * @key: the key string to split + * @prefix: non-NULL pointer to hold extracted prefix + * @suffix: non-NULL pointer to remaining suffix + * + * Given a flattened key such as 'foo.0.bar', split it into two parts + * at the first '.' separator. Allows double dot ('..') to escape the + * normal separator. + * + * e.g. + * 'foo.0.bar' -> prefix='foo' and suffix='0.bar' + * 'foo..0.bar' -> prefix='foo.0' and suffix='bar' + * + * The '..' sequence will be unescaped in the returned 'prefix' + * string. The 'suffix' string will be left in escaped format, so it + * can be fed back into the qdict_split_flat_key() key as the input + * later. + * + * The caller is responsible for freeing the string returned in @prefix + * using g_free(). + */ +static void qdict_split_flat_key(const char *key, char **prefix, + const char **suffix) +{ + const char *separator; + size_t i, j; + + /* Find first '.' separator, but if there is a pair '..' + * that acts as an escape, so skip over '..' */ + separator = NULL; + do { + if (separator) { + separator += 2; + } else { + separator = key; + } + separator = strchr(separator, '.'); + } while (separator && separator[1] == '.'); + + if (separator) { + *prefix = g_strndup(key, separator - key); + *suffix = separator + 1; + } else { + *prefix = g_strdup(key); + *suffix = NULL; + } + + /* Unescape the '..' sequence into '.' */ + for (i = 0, j = 0; (*prefix)[i] != '\0'; i++, j++) { + if ((*prefix)[i] == '.') { + assert((*prefix)[i + 1] == '.'); + i++; + } + (*prefix)[j] = (*prefix)[i]; + } + (*prefix)[j] = '\0'; +} + +/** + * qdict_is_list: + * @maybe_list: dict to check if keys represent list elements. + * + * Determine whether all keys in @maybe_list are valid list elements. + * If @maybe_list is non-zero in length and all the keys look like + * valid list indexes, this will return 1. If @maybe_list is zero + * length or all keys are non-numeric then it will return 0 to indicate + * it is a normal qdict. If there is a mix of numeric and non-numeric + * keys, or the list indexes are non-contiguous, an error is reported. + * + * Returns: 1 if a valid list, 0 if a dict, -1 on error + */ +static int qdict_is_list(QDict *maybe_list, Error **errp) +{ + const QDictEntry *ent; + ssize_t len = 0; + ssize_t max = -1; + int is_list = -1; + int64_t val; + + for (ent = qdict_first(maybe_list); ent != NULL; + ent = qdict_next(maybe_list, ent)) { + + if (qemu_strtoi64(ent->key, NULL, 10, &val) == 0) { + if (is_list == -1) { + is_list = 1; + } else if (!is_list) { + error_setg(errp, + "Cannot mix list and non-list keys"); + return -1; + } + len++; + if (val > max) { + max = val; + } + } else { + if (is_list == -1) { + is_list = 0; + } else if (is_list) { + error_setg(errp, + "Cannot mix list and non-list keys"); + return -1; + } + } + } + + if (is_list == -1) { + assert(!qdict_size(maybe_list)); + is_list = 0; + } + + /* NB this isn't a perfect check - e.g. it won't catch + * a list containing '1', '+1', '01', '3', but that + * does not matter - we've still proved that the + * input is a list. It is up the caller to do a + * stricter check if desired */ + if (len != (max + 1)) { + error_setg(errp, "List indices are not contiguous, " + "saw %zd elements but %zd largest index", + len, max); + return -1; + } + + return is_list; +} + +/** + * qdict_crumple: + * @src: the original flat dictionary (only scalar values) to crumple + * + * Takes a flat dictionary whose keys use '.' separator to indicate + * nesting, and values are scalars, and crumples it into a nested + * structure. + * + * To include a literal '.' in a key name, it must be escaped as '..' + * + * For example, an input of: + * + * { 'foo.0.bar': 'one', 'foo.0.wizz': '1', + * 'foo.1.bar': 'two', 'foo.1.wizz': '2' } + * + * will result in an output of: + * + * { + * 'foo': [ + * { 'bar': 'one', 'wizz': '1' }, + * { 'bar': 'two', 'wizz': '2' } + * ], + * } + * + * The following scenarios in the input dict will result in an + * error being returned: + * + * - Any values in @src are non-scalar types + * - If keys in @src imply that a particular level is both a + * list and a dict. e.g., "foo.0.bar" and "foo.eek.bar". + * - If keys in @src imply that a particular level is a list, + * but the indices are non-contiguous. e.g. "foo.0.bar" and + * "foo.2.bar" without any "foo.1.bar" present. + * - If keys in @src represent list indexes, but are not in + * the "%zu" format. e.g. "foo.+0.bar" + * + * Returns: either a QDict or QList for the nested data structure, or NULL + * on error + */ +QObject *qdict_crumple(const QDict *src, Error **errp) +{ + const QDictEntry *ent; + QDict *two_level, *multi_level = NULL; + QObject *dst = NULL, *child; + size_t i; + char *prefix = NULL; + const char *suffix = NULL; + int is_list; + + two_level = qdict_new(); + + /* Step 1: split our totally flat dict into a two level dict */ + for (ent = qdict_first(src); ent != NULL; ent = qdict_next(src, ent)) { + if (qobject_type(ent->value) == QTYPE_QDICT || + qobject_type(ent->value) == QTYPE_QLIST) { + error_setg(errp, "Value %s is not a scalar", + ent->key); + goto error; + } + + qdict_split_flat_key(ent->key, &prefix, &suffix); + + child = qdict_get(two_level, prefix); + if (suffix) { + QDict *child_dict = qobject_to(QDict, child); + if (!child_dict) { + if (child) { + error_setg(errp, "Key %s prefix is already set as a scalar", + prefix); + goto error; + } + + child_dict = qdict_new(); + qdict_put_obj(two_level, prefix, QOBJECT(child_dict)); + } + + qdict_put_obj(child_dict, suffix, qobject_ref(ent->value)); + } else { + if (child) { + error_setg(errp, "Key %s prefix is already set as a dict", + prefix); + goto error; + } + qdict_put_obj(two_level, prefix, qobject_ref(ent->value)); + } + + g_free(prefix); + prefix = NULL; + } + + /* Step 2: optionally process the two level dict recursively + * into a multi-level dict */ + multi_level = qdict_new(); + for (ent = qdict_first(two_level); ent != NULL; + ent = qdict_next(two_level, ent)) { + QDict *dict = qobject_to(QDict, ent->value); + if (dict) { + child = qdict_crumple(dict, errp); + if (!child) { + goto error; + } + + qdict_put_obj(multi_level, ent->key, child); + } else { + qdict_put_obj(multi_level, ent->key, qobject_ref(ent->value)); + } + } + qobject_unref(two_level); + two_level = NULL; + + /* Step 3: detect if we need to turn our dict into list */ + is_list = qdict_is_list(multi_level, errp); + if (is_list < 0) { + goto error; + } + + if (is_list) { + dst = QOBJECT(qlist_new()); + + for (i = 0; i < qdict_size(multi_level); i++) { + char *key = g_strdup_printf("%zu", i); + + child = qdict_get(multi_level, key); + g_free(key); + + if (!child) { + error_setg(errp, "Missing list index %zu", i); + goto error; + } + + qlist_append_obj(qobject_to(QList, dst), qobject_ref(child)); + } + qobject_unref(multi_level); + multi_level = NULL; + } else { + dst = QOBJECT(multi_level); + } + + return dst; + + error: + g_free(prefix); + qobject_unref(multi_level); + qobject_unref(two_level); + qobject_unref(dst); + return NULL; +} + +/** + * qdict_array_entries(): Returns the number of direct array entries if the + * sub-QDict of src specified by the prefix in subqdict (or src itself for + * prefix == "") is valid as an array, i.e. the length of the created list if + * the sub-QDict would become empty after calling qdict_array_split() on it. If + * the array is not valid, -EINVAL is returned. + */ +int qdict_array_entries(QDict *src, const char *subqdict) +{ + const QDictEntry *entry; + unsigned i; + unsigned entries = 0; + size_t subqdict_len = strlen(subqdict); + + assert(!subqdict_len || subqdict[subqdict_len - 1] == '.'); + + /* qdict_array_split() loops until UINT_MAX, but as we want to return + * negative errors, we only have a signed return value here. Any additional + * entries will lead to -EINVAL. */ + for (i = 0; i < INT_MAX; i++) { + QObject *subqobj; + int subqdict_entries; + char *prefix = g_strdup_printf("%s%u.", subqdict, i); + + subqdict_entries = qdict_count_prefixed_entries(src, prefix); + + /* Remove ending "." */ + prefix[strlen(prefix) - 1] = 0; + subqobj = qdict_get(src, prefix); + + g_free(prefix); + + if (subqdict_entries < 0) { + return subqdict_entries; + } + + /* There may be either a single subordinate object (named "%u") or + * multiple objects (each with a key prefixed "%u."), but not both. */ + if (subqobj && subqdict_entries) { + return -EINVAL; + } else if (!subqobj && !subqdict_entries) { + break; + } + + entries += subqdict_entries ? subqdict_entries : 1; + } + + /* Consider everything handled that isn't part of the given sub-QDict */ + for (entry = qdict_first(src); entry; entry = qdict_next(src, entry)) { + if (!strstart(qdict_entry_key(entry), subqdict, NULL)) { + entries++; + } + } + + /* Anything left in the sub-QDict that wasn't handled? */ + if (qdict_size(src) != entries) { + return -EINVAL; + } + + return i; +} + +/** + * qdict_join(): Absorb the src QDict into the dest QDict, that is, move all + * elements from src to dest. + * + * If an element from src has a key already present in dest, it will not be + * moved unless overwrite is true. + * + * If overwrite is true, the conflicting values in dest will be discarded and + * replaced by the corresponding values from src. + * + * Therefore, with overwrite being true, the src QDict will always be empty when + * this function returns. If overwrite is false, the src QDict will be empty + * iff there were no conflicts. + */ +void qdict_join(QDict *dest, QDict *src, bool overwrite) +{ + const QDictEntry *entry, *next; + + entry = qdict_first(src); + while (entry) { + next = qdict_next(src, entry); + + if (overwrite || !qdict_haskey(dest, entry->key)) { + qdict_put_obj(dest, entry->key, qobject_ref(entry->value)); + qdict_del(src, entry->key); + } + + entry = next; + } +} + +/** + * qdict_rename_keys(): Rename keys in qdict according to the replacements + * specified in the array renames. The array must be terminated by an entry + * with from = NULL. + * + * The renames are performed individually in the order of the array, so entries + * may be renamed multiple times and may or may not conflict depending on the + * order of the renames array. + * + * Returns true for success, false in error cases. + */ +bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) +{ + QObject *qobj; + + while (renames->from) { + if (qdict_haskey(qdict, renames->from)) { + if (qdict_haskey(qdict, renames->to)) { + error_setg(errp, "'%s' and its alias '%s' can't be used at the " + "same time", renames->to, renames->from); + return false; + } + + qobj = qdict_get(qdict, renames->from); + qdict_put_obj(qdict, renames->to, qobject_ref(qobj)); + qdict_del(qdict, renames->from); + } + + renames++; + } + return true; +} diff --git a/qobject/qdict.c b/qobject/qdict.c index 0554c64553..3d8c2f7bbc 100644 --- a/qobject/qdict.c +++ b/qobject/qdict.c @@ -11,17 +11,11 @@ */ #include "qemu/osdep.h" -#include "block/qdict.h" #include "qapi/qmp/qnum.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qbool.h" -#include "qapi/qmp/qlist.h" #include "qapi/qmp/qnull.h" #include "qapi/qmp/qstring.h" -#include "qapi/error.h" -#include "qemu/queue.h" -#include "qemu-common.h" -#include "qemu/cutils.h" /** * qdict_new(): Create a new QDict @@ -464,626 +458,3 @@ void qdict_destroy_obj(QObject *obj) g_free(qdict); } - -/** - * qdict_copy_default(): If no entry mapped by 'key' exists in 'dst' yet, the - * value of 'key' in 'src' is copied there (and the refcount increased - * accordingly). - */ -void qdict_copy_default(QDict *dst, QDict *src, const char *key) -{ - QObject *val; - - if (qdict_haskey(dst, key)) { - return; - } - - val = qdict_get(src, key); - if (val) { - qdict_put_obj(dst, key, qobject_ref(val)); - } -} - -/** - * qdict_set_default_str(): If no entry mapped by 'key' exists in 'dst' yet, a - * new QString initialised by 'val' is put there. - */ -void qdict_set_default_str(QDict *dst, const char *key, const char *val) -{ - if (qdict_haskey(dst, key)) { - return; - } - - qdict_put_str(dst, key, val); -} - -static void qdict_flatten_qdict(QDict *qdict, QDict *target, - const char *prefix); - -static void qdict_flatten_qlist(QList *qlist, QDict *target, const char *prefix) -{ - QObject *value; - const QListEntry *entry; - char *new_key; - int i; - - /* This function is never called with prefix == NULL, i.e., it is always - * called from within qdict_flatten_q(list|dict)(). Therefore, it does not - * need to remove list entries during the iteration (the whole list will be - * deleted eventually anyway from qdict_flatten_qdict()). */ - assert(prefix); - - entry = qlist_first(qlist); - - for (i = 0; entry; entry = qlist_next(entry), i++) { - value = qlist_entry_obj(entry); - new_key = g_strdup_printf("%s.%i", prefix, i); - - if (qobject_type(value) == QTYPE_QDICT) { - qdict_flatten_qdict(qobject_to(QDict, value), target, new_key); - } else if (qobject_type(value) == QTYPE_QLIST) { - qdict_flatten_qlist(qobject_to(QList, value), target, new_key); - } else { - /* All other types are moved to the target unchanged. */ - qdict_put_obj(target, new_key, qobject_ref(value)); - } - - g_free(new_key); - } -} - -static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) -{ - QObject *value; - const QDictEntry *entry, *next; - char *new_key; - bool delete; - - entry = qdict_first(qdict); - - while (entry != NULL) { - - next = qdict_next(qdict, entry); - value = qdict_entry_value(entry); - new_key = NULL; - delete = false; - - if (prefix) { - new_key = g_strdup_printf("%s.%s", prefix, entry->key); - } - - if (qobject_type(value) == QTYPE_QDICT) { - /* Entries of QDicts are processed recursively, the QDict object - * itself disappears. */ - qdict_flatten_qdict(qobject_to(QDict, value), target, - new_key ? new_key : entry->key); - delete = true; - } else if (qobject_type(value) == QTYPE_QLIST) { - qdict_flatten_qlist(qobject_to(QList, value), target, - new_key ? new_key : entry->key); - delete = true; - } else if (prefix) { - /* All other objects are moved to the target unchanged. */ - qdict_put_obj(target, new_key, qobject_ref(value)); - delete = true; - } - - g_free(new_key); - - if (delete) { - qdict_del(qdict, entry->key); - - /* Restart loop after modifying the iterated QDict */ - entry = qdict_first(qdict); - continue; - } - - entry = next; - } -} - -/** - * qdict_flatten(): For each nested QDict with key x, all fields with key y - * are moved to this QDict and their key is renamed to "x.y". For each nested - * QList with key x, the field at index y is moved to this QDict with the key - * "x.y" (i.e., the reverse of what qdict_array_split() does). - * This operation is applied recursively for nested QDicts and QLists. - */ -void qdict_flatten(QDict *qdict) -{ - qdict_flatten_qdict(qdict, qdict, NULL); -} - -/* extract all the src QDict entries starting by start into dst */ -void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start) - -{ - const QDictEntry *entry, *next; - const char *p; - - *dst = qdict_new(); - entry = qdict_first(src); - - while (entry != NULL) { - next = qdict_next(src, entry); - if (strstart(entry->key, start, &p)) { - qdict_put_obj(*dst, p, qobject_ref(entry->value)); - qdict_del(src, entry->key); - } - entry = next; - } -} - -static int qdict_count_prefixed_entries(const QDict *src, const char *start) -{ - const QDictEntry *entry; - int count = 0; - - for (entry = qdict_first(src); entry; entry = qdict_next(src, entry)) { - if (strstart(entry->key, start, NULL)) { - if (count == INT_MAX) { - return -ERANGE; - } - count++; - } - } - - return count; -} - -/** - * qdict_array_split(): This function moves array-like elements of a QDict into - * a new QList. Every entry in the original QDict with a key "%u" or one - * prefixed "%u.", where %u designates an unsigned integer starting at 0 and - * incrementally counting up, will be moved to a new QDict at index %u in the - * output QList with the key prefix removed, if that prefix is "%u.". If the - * whole key is just "%u", the whole QObject will be moved unchanged without - * creating a new QDict. The function terminates when there is no entry in the - * QDict with a prefix directly (incrementally) following the last one; it also - * returns if there are both entries with "%u" and "%u." for the same index %u. - * Example: {"0.a": 42, "0.b": 23, "1.x": 0, "4.y": 1, "o.o": 7, "2": 66} - * (or {"1.x": 0, "4.y": 1, "0.a": 42, "o.o": 7, "0.b": 23, "2": 66}) - * => [{"a": 42, "b": 23}, {"x": 0}, 66] - * and {"4.y": 1, "o.o": 7} (remainder of the old QDict) - */ -void qdict_array_split(QDict *src, QList **dst) -{ - unsigned i; - - *dst = qlist_new(); - - for (i = 0; i < UINT_MAX; i++) { - QObject *subqobj; - bool is_subqdict; - QDict *subqdict; - char indexstr[32], prefix[32]; - size_t snprintf_ret; - - snprintf_ret = snprintf(indexstr, 32, "%u", i); - assert(snprintf_ret < 32); - - subqobj = qdict_get(src, indexstr); - - snprintf_ret = snprintf(prefix, 32, "%u.", i); - assert(snprintf_ret < 32); - - /* Overflow is the same as positive non-zero results */ - is_subqdict = qdict_count_prefixed_entries(src, prefix); - - // There may be either a single subordinate object (named "%u") or - // multiple objects (each with a key prefixed "%u."), but not both. - if (!subqobj == !is_subqdict) { - break; - } - - if (is_subqdict) { - qdict_extract_subqdict(src, &subqdict, prefix); - assert(qdict_size(subqdict) > 0); - } else { - qobject_ref(subqobj); - qdict_del(src, indexstr); - } - - qlist_append_obj(*dst, subqobj ?: QOBJECT(subqdict)); - } -} - -/** - * qdict_split_flat_key: - * @key: the key string to split - * @prefix: non-NULL pointer to hold extracted prefix - * @suffix: non-NULL pointer to remaining suffix - * - * Given a flattened key such as 'foo.0.bar', split it into two parts - * at the first '.' separator. Allows double dot ('..') to escape the - * normal separator. - * - * e.g. - * 'foo.0.bar' -> prefix='foo' and suffix='0.bar' - * 'foo..0.bar' -> prefix='foo.0' and suffix='bar' - * - * The '..' sequence will be unescaped in the returned 'prefix' - * string. The 'suffix' string will be left in escaped format, so it - * can be fed back into the qdict_split_flat_key() key as the input - * later. - * - * The caller is responsible for freeing the string returned in @prefix - * using g_free(). - */ -static void qdict_split_flat_key(const char *key, char **prefix, - const char **suffix) -{ - const char *separator; - size_t i, j; - - /* Find first '.' separator, but if there is a pair '..' - * that acts as an escape, so skip over '..' */ - separator = NULL; - do { - if (separator) { - separator += 2; - } else { - separator = key; - } - separator = strchr(separator, '.'); - } while (separator && separator[1] == '.'); - - if (separator) { - *prefix = g_strndup(key, separator - key); - *suffix = separator + 1; - } else { - *prefix = g_strdup(key); - *suffix = NULL; - } - - /* Unescape the '..' sequence into '.' */ - for (i = 0, j = 0; (*prefix)[i] != '\0'; i++, j++) { - if ((*prefix)[i] == '.') { - assert((*prefix)[i + 1] == '.'); - i++; - } - (*prefix)[j] = (*prefix)[i]; - } - (*prefix)[j] = '\0'; -} - -/** - * qdict_is_list: - * @maybe_list: dict to check if keys represent list elements. - * - * Determine whether all keys in @maybe_list are valid list elements. - * If @maybe_list is non-zero in length and all the keys look like - * valid list indexes, this will return 1. If @maybe_list is zero - * length or all keys are non-numeric then it will return 0 to indicate - * it is a normal qdict. If there is a mix of numeric and non-numeric - * keys, or the list indexes are non-contiguous, an error is reported. - * - * Returns: 1 if a valid list, 0 if a dict, -1 on error - */ -static int qdict_is_list(QDict *maybe_list, Error **errp) -{ - const QDictEntry *ent; - ssize_t len = 0; - ssize_t max = -1; - int is_list = -1; - int64_t val; - - for (ent = qdict_first(maybe_list); ent != NULL; - ent = qdict_next(maybe_list, ent)) { - - if (qemu_strtoi64(ent->key, NULL, 10, &val) == 0) { - if (is_list == -1) { - is_list = 1; - } else if (!is_list) { - error_setg(errp, - "Cannot mix list and non-list keys"); - return -1; - } - len++; - if (val > max) { - max = val; - } - } else { - if (is_list == -1) { - is_list = 0; - } else if (is_list) { - error_setg(errp, - "Cannot mix list and non-list keys"); - return -1; - } - } - } - - if (is_list == -1) { - assert(!qdict_size(maybe_list)); - is_list = 0; - } - - /* NB this isn't a perfect check - e.g. it won't catch - * a list containing '1', '+1', '01', '3', but that - * does not matter - we've still proved that the - * input is a list. It is up the caller to do a - * stricter check if desired */ - if (len != (max + 1)) { - error_setg(errp, "List indices are not contiguous, " - "saw %zd elements but %zd largest index", - len, max); - return -1; - } - - return is_list; -} - -/** - * qdict_crumple: - * @src: the original flat dictionary (only scalar values) to crumple - * - * Takes a flat dictionary whose keys use '.' separator to indicate - * nesting, and values are scalars, and crumples it into a nested - * structure. - * - * To include a literal '.' in a key name, it must be escaped as '..' - * - * For example, an input of: - * - * { 'foo.0.bar': 'one', 'foo.0.wizz': '1', - * 'foo.1.bar': 'two', 'foo.1.wizz': '2' } - * - * will result in an output of: - * - * { - * 'foo': [ - * { 'bar': 'one', 'wizz': '1' }, - * { 'bar': 'two', 'wizz': '2' } - * ], - * } - * - * The following scenarios in the input dict will result in an - * error being returned: - * - * - Any values in @src are non-scalar types - * - If keys in @src imply that a particular level is both a - * list and a dict. e.g., "foo.0.bar" and "foo.eek.bar". - * - If keys in @src imply that a particular level is a list, - * but the indices are non-contiguous. e.g. "foo.0.bar" and - * "foo.2.bar" without any "foo.1.bar" present. - * - If keys in @src represent list indexes, but are not in - * the "%zu" format. e.g. "foo.+0.bar" - * - * Returns: either a QDict or QList for the nested data structure, or NULL - * on error - */ -QObject *qdict_crumple(const QDict *src, Error **errp) -{ - const QDictEntry *ent; - QDict *two_level, *multi_level = NULL; - QObject *dst = NULL, *child; - size_t i; - char *prefix = NULL; - const char *suffix = NULL; - int is_list; - - two_level = qdict_new(); - - /* Step 1: split our totally flat dict into a two level dict */ - for (ent = qdict_first(src); ent != NULL; ent = qdict_next(src, ent)) { - if (qobject_type(ent->value) == QTYPE_QDICT || - qobject_type(ent->value) == QTYPE_QLIST) { - error_setg(errp, "Value %s is not a scalar", - ent->key); - goto error; - } - - qdict_split_flat_key(ent->key, &prefix, &suffix); - - child = qdict_get(two_level, prefix); - if (suffix) { - QDict *child_dict = qobject_to(QDict, child); - if (!child_dict) { - if (child) { - error_setg(errp, "Key %s prefix is already set as a scalar", - prefix); - goto error; - } - - child_dict = qdict_new(); - qdict_put_obj(two_level, prefix, QOBJECT(child_dict)); - } - - qdict_put_obj(child_dict, suffix, qobject_ref(ent->value)); - } else { - if (child) { - error_setg(errp, "Key %s prefix is already set as a dict", - prefix); - goto error; - } - qdict_put_obj(two_level, prefix, qobject_ref(ent->value)); - } - - g_free(prefix); - prefix = NULL; - } - - /* Step 2: optionally process the two level dict recursively - * into a multi-level dict */ - multi_level = qdict_new(); - for (ent = qdict_first(two_level); ent != NULL; - ent = qdict_next(two_level, ent)) { - QDict *dict = qobject_to(QDict, ent->value); - if (dict) { - child = qdict_crumple(dict, errp); - if (!child) { - goto error; - } - - qdict_put_obj(multi_level, ent->key, child); - } else { - qdict_put_obj(multi_level, ent->key, qobject_ref(ent->value)); - } - } - qobject_unref(two_level); - two_level = NULL; - - /* Step 3: detect if we need to turn our dict into list */ - is_list = qdict_is_list(multi_level, errp); - if (is_list < 0) { - goto error; - } - - if (is_list) { - dst = QOBJECT(qlist_new()); - - for (i = 0; i < qdict_size(multi_level); i++) { - char *key = g_strdup_printf("%zu", i); - - child = qdict_get(multi_level, key); - g_free(key); - - if (!child) { - error_setg(errp, "Missing list index %zu", i); - goto error; - } - - qlist_append_obj(qobject_to(QList, dst), qobject_ref(child)); - } - qobject_unref(multi_level); - multi_level = NULL; - } else { - dst = QOBJECT(multi_level); - } - - return dst; - - error: - g_free(prefix); - qobject_unref(multi_level); - qobject_unref(two_level); - qobject_unref(dst); - return NULL; -} - -/** - * qdict_array_entries(): Returns the number of direct array entries if the - * sub-QDict of src specified by the prefix in subqdict (or src itself for - * prefix == "") is valid as an array, i.e. the length of the created list if - * the sub-QDict would become empty after calling qdict_array_split() on it. If - * the array is not valid, -EINVAL is returned. - */ -int qdict_array_entries(QDict *src, const char *subqdict) -{ - const QDictEntry *entry; - unsigned i; - unsigned entries = 0; - size_t subqdict_len = strlen(subqdict); - - assert(!subqdict_len || subqdict[subqdict_len - 1] == '.'); - - /* qdict_array_split() loops until UINT_MAX, but as we want to return - * negative errors, we only have a signed return value here. Any additional - * entries will lead to -EINVAL. */ - for (i = 0; i < INT_MAX; i++) { - QObject *subqobj; - int subqdict_entries; - char *prefix = g_strdup_printf("%s%u.", subqdict, i); - - subqdict_entries = qdict_count_prefixed_entries(src, prefix); - - /* Remove ending "." */ - prefix[strlen(prefix) - 1] = 0; - subqobj = qdict_get(src, prefix); - - g_free(prefix); - - if (subqdict_entries < 0) { - return subqdict_entries; - } - - /* There may be either a single subordinate object (named "%u") or - * multiple objects (each with a key prefixed "%u."), but not both. */ - if (subqobj && subqdict_entries) { - return -EINVAL; - } else if (!subqobj && !subqdict_entries) { - break; - } - - entries += subqdict_entries ? subqdict_entries : 1; - } - - /* Consider everything handled that isn't part of the given sub-QDict */ - for (entry = qdict_first(src); entry; entry = qdict_next(src, entry)) { - if (!strstart(qdict_entry_key(entry), subqdict, NULL)) { - entries++; - } - } - - /* Anything left in the sub-QDict that wasn't handled? */ - if (qdict_size(src) != entries) { - return -EINVAL; - } - - return i; -} - -/** - * qdict_join(): Absorb the src QDict into the dest QDict, that is, move all - * elements from src to dest. - * - * If an element from src has a key already present in dest, it will not be - * moved unless overwrite is true. - * - * If overwrite is true, the conflicting values in dest will be discarded and - * replaced by the corresponding values from src. - * - * Therefore, with overwrite being true, the src QDict will always be empty when - * this function returns. If overwrite is false, the src QDict will be empty - * iff there were no conflicts. - */ -void qdict_join(QDict *dest, QDict *src, bool overwrite) -{ - const QDictEntry *entry, *next; - - entry = qdict_first(src); - while (entry) { - next = qdict_next(src, entry); - - if (overwrite || !qdict_haskey(dest, entry->key)) { - qdict_put_obj(dest, entry->key, qobject_ref(entry->value)); - qdict_del(src, entry->key); - } - - entry = next; - } -} - -/** - * qdict_rename_keys(): Rename keys in qdict according to the replacements - * specified in the array renames. The array must be terminated by an entry - * with from = NULL. - * - * The renames are performed individually in the order of the array, so entries - * may be renamed multiple times and may or may not conflict depending on the - * order of the renames array. - * - * Returns true for success, false in error cases. - */ -bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) -{ - QObject *qobj; - - while (renames->from) { - if (qdict_haskey(qdict, renames->from)) { - if (qdict_haskey(qdict, renames->to)) { - error_setg(errp, "'%s' and its alias '%s' can't be used at the " - "same time", renames->to, renames->from); - return false; - } - - qobj = qdict_get(qdict, renames->from); - qdict_put_obj(qdict, renames->to, qobject_ref(qobj)); - qdict_del(qdict, renames->from); - } - - renames++; - } - return true; -} diff --git a/tests/Makefile.include b/tests/Makefile.include index 86f90c0cb0..fd0274b5dc 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -40,6 +40,8 @@ SYSEMU_TARGET_LIST := $(subst -softmmu.mak,,$(notdir \ check-unit-y = tests/check-qdict$(EXESUF) gcov-files-check-qdict-y = qobject/qdict.c +check-unit-y = tests/check-block-qdict$(EXESUF) +gcov-files-check-block-qdict-y = qobject/block-qdict.c check-unit-y += tests/test-char$(EXESUF) gcov-files-check-qdict-y = chardev/char.c check-unit-y += tests/check-qnum$(EXESUF) @@ -582,6 +584,7 @@ GENERATED_FILES += tests/test-qapi-types.h tests/test-qapi-visit.h \ test-obj-y = tests/check-qnum.o tests/check-qstring.o tests/check-qdict.o \ tests/check-qlist.o tests/check-qnull.o tests/check-qobject.o \ tests/check-qjson.o tests/check-qlit.o \ + tests/check-block-qtest.o \ tests/test-coroutine.o tests/test-string-output-visitor.o \ tests/test-string-input-visitor.o tests/test-qobject-output-visitor.o \ tests/test-clone-visitor.o \ @@ -612,6 +615,7 @@ test-block-obj-y = $(block-obj-y) $(test-io-obj-y) tests/iothread.o tests/check-qnum$(EXESUF): tests/check-qnum.o $(test-util-obj-y) tests/check-qstring$(EXESUF): tests/check-qstring.o $(test-util-obj-y) tests/check-qdict$(EXESUF): tests/check-qdict.o $(test-util-obj-y) +tests/check-block-qdict$(EXESUF): tests/check-block-qdict.o $(test-util-obj-y) tests/check-qlist$(EXESUF): tests/check-qlist.o $(test-util-obj-y) tests/check-qnull$(EXESUF): tests/check-qnull.o $(test-util-obj-y) tests/check-qobject$(EXESUF): tests/check-qobject.o $(test-util-obj-y) diff --git a/tests/check-block-qdict.c b/tests/check-block-qdict.c new file mode 100644 index 0000000000..f9845fd1f0 --- /dev/null +++ b/tests/check-block-qdict.c @@ -0,0 +1,657 @@ +/* + * Unit-tests for Block layer QDict extras + * + * Copyright (c) 2013-2018 Red Hat, Inc. + * + * This work is licensed under the terms of the GNU LGPL, version 2.1 or later. + * See the COPYING.LIB file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "block/qdict.h" +#include "qapi/qmp/qlist.h" +#include "qapi/qmp/qnum.h" +#include "qapi/error.h" + +static void qdict_defaults_test(void) +{ + QDict *dict, *copy; + + dict = qdict_new(); + copy = qdict_new(); + + qdict_set_default_str(dict, "foo", "abc"); + qdict_set_default_str(dict, "foo", "def"); + g_assert_cmpstr(qdict_get_str(dict, "foo"), ==, "abc"); + qdict_set_default_str(dict, "bar", "ghi"); + + qdict_copy_default(copy, dict, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "foo"), ==, "abc"); + qdict_set_default_str(copy, "bar", "xyz"); + qdict_copy_default(copy, dict, "bar"); + g_assert_cmpstr(qdict_get_str(copy, "bar"), ==, "xyz"); + + qobject_unref(copy); + qobject_unref(dict); +} + +static void qdict_flatten_test(void) +{ + QList *list1 = qlist_new(); + QList *list2 = qlist_new(); + QDict *dict1 = qdict_new(); + QDict *dict2 = qdict_new(); + QDict *dict3 = qdict_new(); + + /* + * Test the flattening of + * + * { + * "e": [ + * 42, + * [ + * 23, + * 66, + * { + * "a": 0, + * "b": 1 + * } + * ] + * ], + * "f": { + * "c": 2, + * "d": 3, + * }, + * "g": 4 + * } + * + * to + * + * { + * "e.0": 42, + * "e.1.0": 23, + * "e.1.1": 66, + * "e.1.2.a": 0, + * "e.1.2.b": 1, + * "f.c": 2, + * "f.d": 3, + * "g": 4 + * } + */ + + qdict_put_int(dict1, "a", 0); + qdict_put_int(dict1, "b", 1); + + qlist_append_int(list1, 23); + qlist_append_int(list1, 66); + qlist_append(list1, dict1); + qlist_append_int(list2, 42); + qlist_append(list2, list1); + + qdict_put_int(dict2, "c", 2); + qdict_put_int(dict2, "d", 3); + qdict_put(dict3, "e", list2); + qdict_put(dict3, "f", dict2); + qdict_put_int(dict3, "g", 4); + + qdict_flatten(dict3); + + g_assert(qdict_get_int(dict3, "e.0") == 42); + g_assert(qdict_get_int(dict3, "e.1.0") == 23); + g_assert(qdict_get_int(dict3, "e.1.1") == 66); + g_assert(qdict_get_int(dict3, "e.1.2.a") == 0); + g_assert(qdict_get_int(dict3, "e.1.2.b") == 1); + g_assert(qdict_get_int(dict3, "f.c") == 2); + g_assert(qdict_get_int(dict3, "f.d") == 3); + g_assert(qdict_get_int(dict3, "g") == 4); + + g_assert(qdict_size(dict3) == 8); + + qobject_unref(dict3); +} + +static void qdict_array_split_test(void) +{ + QDict *test_dict = qdict_new(); + QDict *dict1, *dict2; + QNum *int1; + QList *test_list; + + /* + * Test the split of + * + * { + * "1.x": 0, + * "4.y": 1, + * "0.a": 42, + * "o.o": 7, + * "0.b": 23, + * "2": 66 + * } + * + * to + * + * [ + * { + * "a": 42, + * "b": 23 + * }, + * { + * "x": 0 + * }, + * 66 + * ] + * + * and + * + * { + * "4.y": 1, + * "o.o": 7 + * } + * + * (remaining in the old QDict) + * + * This example is given in the comment of qdict_array_split(). + */ + + qdict_put_int(test_dict, "1.x", 0); + qdict_put_int(test_dict, "4.y", 1); + qdict_put_int(test_dict, "0.a", 42); + qdict_put_int(test_dict, "o.o", 7); + qdict_put_int(test_dict, "0.b", 23); + qdict_put_int(test_dict, "2", 66); + + qdict_array_split(test_dict, &test_list); + + dict1 = qobject_to(QDict, qlist_pop(test_list)); + dict2 = qobject_to(QDict, qlist_pop(test_list)); + int1 = qobject_to(QNum, qlist_pop(test_list)); + + g_assert(dict1); + g_assert(dict2); + g_assert(int1); + g_assert(qlist_empty(test_list)); + + qobject_unref(test_list); + + g_assert(qdict_get_int(dict1, "a") == 42); + g_assert(qdict_get_int(dict1, "b") == 23); + + g_assert(qdict_size(dict1) == 2); + + qobject_unref(dict1); + + g_assert(qdict_get_int(dict2, "x") == 0); + + g_assert(qdict_size(dict2) == 1); + + qobject_unref(dict2); + + g_assert_cmpint(qnum_get_int(int1), ==, 66); + + qobject_unref(int1); + + g_assert(qdict_get_int(test_dict, "4.y") == 1); + g_assert(qdict_get_int(test_dict, "o.o") == 7); + + g_assert(qdict_size(test_dict) == 2); + + qobject_unref(test_dict); + + /* + * Test the split of + * + * { + * "0": 42, + * "1": 23, + * "1.x": 84 + * } + * + * to + * + * [ + * 42 + * ] + * + * and + * + * { + * "1": 23, + * "1.x": 84 + * } + * + * That is, test whether splitting stops if there is both an entry with key + * of "%u" and other entries with keys prefixed "%u." for the same index. + */ + + test_dict = qdict_new(); + + qdict_put_int(test_dict, "0", 42); + qdict_put_int(test_dict, "1", 23); + qdict_put_int(test_dict, "1.x", 84); + + qdict_array_split(test_dict, &test_list); + + int1 = qobject_to(QNum, qlist_pop(test_list)); + + g_assert(int1); + g_assert(qlist_empty(test_list)); + + qobject_unref(test_list); + + g_assert_cmpint(qnum_get_int(int1), ==, 42); + + qobject_unref(int1); + + g_assert(qdict_get_int(test_dict, "1") == 23); + g_assert(qdict_get_int(test_dict, "1.x") == 84); + + g_assert(qdict_size(test_dict) == 2); + + qobject_unref(test_dict); +} + +static void qdict_array_entries_test(void) +{ + QDict *dict = qdict_new(); + + g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, 0); + + qdict_put_int(dict, "bar", 0); + qdict_put_int(dict, "baz.0", 0); + g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, 0); + + qdict_put_int(dict, "foo.1", 0); + g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, -EINVAL); + qdict_put_int(dict, "foo.0", 0); + g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, 2); + qdict_put_int(dict, "foo.bar", 0); + g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, -EINVAL); + qdict_del(dict, "foo.bar"); + + qdict_put_int(dict, "foo.2.a", 0); + qdict_put_int(dict, "foo.2.b", 0); + qdict_put_int(dict, "foo.2.c", 0); + g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, 3); + g_assert_cmpint(qdict_array_entries(dict, ""), ==, -EINVAL); + + qobject_unref(dict); + + dict = qdict_new(); + qdict_put_int(dict, "1", 0); + g_assert_cmpint(qdict_array_entries(dict, ""), ==, -EINVAL); + qdict_put_int(dict, "0", 0); + g_assert_cmpint(qdict_array_entries(dict, ""), ==, 2); + qdict_put_int(dict, "bar", 0); + g_assert_cmpint(qdict_array_entries(dict, ""), ==, -EINVAL); + qdict_del(dict, "bar"); + + qdict_put_int(dict, "2.a", 0); + qdict_put_int(dict, "2.b", 0); + qdict_put_int(dict, "2.c", 0); + g_assert_cmpint(qdict_array_entries(dict, ""), ==, 3); + + qobject_unref(dict); +} + +static void qdict_join_test(void) +{ + QDict *dict1, *dict2; + bool overwrite = false; + int i; + + dict1 = qdict_new(); + dict2 = qdict_new(); + + /* Test everything once without overwrite and once with */ + do + { + /* Test empty dicts */ + qdict_join(dict1, dict2, overwrite); + + g_assert(qdict_size(dict1) == 0); + g_assert(qdict_size(dict2) == 0); + + /* First iteration: Test movement */ + /* Second iteration: Test empty source and non-empty destination */ + qdict_put_int(dict2, "foo", 42); + + for (i = 0; i < 2; i++) { + qdict_join(dict1, dict2, overwrite); + + g_assert(qdict_size(dict1) == 1); + g_assert(qdict_size(dict2) == 0); + + g_assert(qdict_get_int(dict1, "foo") == 42); + } + + /* Test non-empty source and destination without conflict */ + qdict_put_int(dict2, "bar", 23); + + qdict_join(dict1, dict2, overwrite); + + g_assert(qdict_size(dict1) == 2); + g_assert(qdict_size(dict2) == 0); + + g_assert(qdict_get_int(dict1, "foo") == 42); + g_assert(qdict_get_int(dict1, "bar") == 23); + + /* Test conflict */ + qdict_put_int(dict2, "foo", 84); + + qdict_join(dict1, dict2, overwrite); + + g_assert(qdict_size(dict1) == 2); + g_assert(qdict_size(dict2) == !overwrite); + + g_assert(qdict_get_int(dict1, "foo") == (overwrite ? 84 : 42)); + g_assert(qdict_get_int(dict1, "bar") == 23); + + if (!overwrite) { + g_assert(qdict_get_int(dict2, "foo") == 84); + } + + /* Check the references */ + g_assert(qdict_get(dict1, "foo")->base.refcnt == 1); + g_assert(qdict_get(dict1, "bar")->base.refcnt == 1); + + if (!overwrite) { + g_assert(qdict_get(dict2, "foo")->base.refcnt == 1); + } + + /* Clean up */ + qdict_del(dict1, "foo"); + qdict_del(dict1, "bar"); + + if (!overwrite) { + qdict_del(dict2, "foo"); + } + } + while (overwrite ^= true); + + qobject_unref(dict1); + qobject_unref(dict2); +} + +static void qdict_crumple_test_recursive(void) +{ + QDict *src, *dst, *rule, *vnc, *acl, *listen; + QList *rules; + + src = qdict_new(); + qdict_put_str(src, "vnc.listen.addr", "127.0.0.1"); + qdict_put_str(src, "vnc.listen.port", "5901"); + qdict_put_str(src, "vnc.acl.rules.0.match", "fred"); + qdict_put_str(src, "vnc.acl.rules.0.policy", "allow"); + qdict_put_str(src, "vnc.acl.rules.1.match", "bob"); + qdict_put_str(src, "vnc.acl.rules.1.policy", "deny"); + qdict_put_str(src, "vnc.acl.default", "deny"); + qdict_put_str(src, "vnc.acl..name", "acl0"); + qdict_put_str(src, "vnc.acl.rule..name", "acl0"); + + dst = qobject_to(QDict, qdict_crumple(src, &error_abort)); + g_assert(dst); + g_assert_cmpint(qdict_size(dst), ==, 1); + + vnc = qdict_get_qdict(dst, "vnc"); + g_assert(vnc); + g_assert_cmpint(qdict_size(vnc), ==, 3); + + listen = qdict_get_qdict(vnc, "listen"); + g_assert(listen); + g_assert_cmpint(qdict_size(listen), ==, 2); + g_assert_cmpstr("127.0.0.1", ==, qdict_get_str(listen, "addr")); + g_assert_cmpstr("5901", ==, qdict_get_str(listen, "port")); + + acl = qdict_get_qdict(vnc, "acl"); + g_assert(acl); + g_assert_cmpint(qdict_size(acl), ==, 3); + + rules = qdict_get_qlist(acl, "rules"); + g_assert(rules); + g_assert_cmpint(qlist_size(rules), ==, 2); + + rule = qobject_to(QDict, qlist_pop(rules)); + g_assert(rule); + g_assert_cmpint(qdict_size(rule), ==, 2); + g_assert_cmpstr("fred", ==, qdict_get_str(rule, "match")); + g_assert_cmpstr("allow", ==, qdict_get_str(rule, "policy")); + qobject_unref(rule); + + rule = qobject_to(QDict, qlist_pop(rules)); + g_assert(rule); + g_assert_cmpint(qdict_size(rule), ==, 2); + g_assert_cmpstr("bob", ==, qdict_get_str(rule, "match")); + g_assert_cmpstr("deny", ==, qdict_get_str(rule, "policy")); + qobject_unref(rule); + + /* With recursive crumpling, we should see all names unescaped */ + g_assert_cmpstr("acl0", ==, qdict_get_str(vnc, "acl.name")); + g_assert_cmpstr("acl0", ==, qdict_get_str(acl, "rule.name")); + + qobject_unref(src); + qobject_unref(dst); +} + +static void qdict_crumple_test_empty(void) +{ + QDict *src, *dst; + + src = qdict_new(); + + dst = qobject_to(QDict, qdict_crumple(src, &error_abort)); + + g_assert_cmpint(qdict_size(dst), ==, 0); + + qobject_unref(src); + qobject_unref(dst); +} + +static int qdict_count_entries(QDict *dict) +{ + const QDictEntry *e; + int count = 0; + + for (e = qdict_first(dict); e; e = qdict_next(dict, e)) { + count++; + } + + return count; +} + +static void qdict_rename_keys_test(void) +{ + QDict *dict = qdict_new(); + QDict *copy; + QDictRenames *renames; + Error *local_err = NULL; + + qdict_put_str(dict, "abc", "foo"); + qdict_put_str(dict, "abcdef", "bar"); + qdict_put_int(dict, "number", 42); + qdict_put_bool(dict, "flag", true); + qdict_put_null(dict, "nothing"); + + /* Empty rename list */ + renames = (QDictRenames[]) { + { NULL, "this can be anything" } + }; + copy = qdict_clone_shallow(dict); + qdict_rename_keys(copy, renames, &error_abort); + + g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "abcdef"), ==, "bar"); + g_assert_cmpint(qdict_get_int(copy, "number"), ==, 42); + g_assert_cmpint(qdict_get_bool(copy, "flag"), ==, true); + g_assert(qobject_type(qdict_get(copy, "nothing")) == QTYPE_QNULL); + g_assert_cmpint(qdict_count_entries(copy), ==, 5); + + qobject_unref(copy); + + /* Simple rename of all entries */ + renames = (QDictRenames[]) { + { "abc", "str1" }, + { "abcdef", "str2" }, + { "number", "int" }, + { "flag", "bool" }, + { "nothing", "null" }, + { NULL , NULL } + }; + copy = qdict_clone_shallow(dict); + qdict_rename_keys(copy, renames, &error_abort); + + g_assert(!qdict_haskey(copy, "abc")); + g_assert(!qdict_haskey(copy, "abcdef")); + g_assert(!qdict_haskey(copy, "number")); + g_assert(!qdict_haskey(copy, "flag")); + g_assert(!qdict_haskey(copy, "nothing")); + + g_assert_cmpstr(qdict_get_str(copy, "str1"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "str2"), ==, "bar"); + g_assert_cmpint(qdict_get_int(copy, "int"), ==, 42); + g_assert_cmpint(qdict_get_bool(copy, "bool"), ==, true); + g_assert(qobject_type(qdict_get(copy, "null")) == QTYPE_QNULL); + g_assert_cmpint(qdict_count_entries(copy), ==, 5); + + qobject_unref(copy); + + /* Renames are processed top to bottom */ + renames = (QDictRenames[]) { + { "abc", "tmp" }, + { "abcdef", "abc" }, + { "number", "abcdef" }, + { "flag", "number" }, + { "nothing", "flag" }, + { "tmp", "nothing" }, + { NULL , NULL } + }; + copy = qdict_clone_shallow(dict); + qdict_rename_keys(copy, renames, &error_abort); + + g_assert_cmpstr(qdict_get_str(copy, "nothing"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "bar"); + g_assert_cmpint(qdict_get_int(copy, "abcdef"), ==, 42); + g_assert_cmpint(qdict_get_bool(copy, "number"), ==, true); + g_assert(qobject_type(qdict_get(copy, "flag")) == QTYPE_QNULL); + g_assert(!qdict_haskey(copy, "tmp")); + g_assert_cmpint(qdict_count_entries(copy), ==, 5); + + qobject_unref(copy); + + /* Conflicting rename */ + renames = (QDictRenames[]) { + { "abcdef", "abc" }, + { NULL , NULL } + }; + copy = qdict_clone_shallow(dict); + qdict_rename_keys(copy, renames, &local_err); + + g_assert(local_err != NULL); + error_free(local_err); + local_err = NULL; + + g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "abcdef"), ==, "bar"); + g_assert_cmpint(qdict_get_int(copy, "number"), ==, 42); + g_assert_cmpint(qdict_get_bool(copy, "flag"), ==, true); + g_assert(qobject_type(qdict_get(copy, "nothing")) == QTYPE_QNULL); + g_assert_cmpint(qdict_count_entries(copy), ==, 5); + + qobject_unref(copy); + + /* Renames in an empty dict */ + renames = (QDictRenames[]) { + { "abcdef", "abc" }, + { NULL , NULL } + }; + + qobject_unref(dict); + dict = qdict_new(); + + qdict_rename_keys(dict, renames, &error_abort); + g_assert(qdict_first(dict) == NULL); + + qobject_unref(dict); +} + +static void qdict_crumple_test_bad_inputs(void) +{ + QDict *src; + Error *error = NULL; + + src = qdict_new(); + /* rule.0 can't be both a string and a dict */ + qdict_put_str(src, "rule.0", "fred"); + qdict_put_str(src, "rule.0.policy", "allow"); + + g_assert(qdict_crumple(src, &error) == NULL); + g_assert(error != NULL); + error_free(error); + error = NULL; + qobject_unref(src); + + src = qdict_new(); + /* rule can't be both a list and a dict */ + qdict_put_str(src, "rule.0", "fred"); + qdict_put_str(src, "rule.a", "allow"); + + g_assert(qdict_crumple(src, &error) == NULL); + g_assert(error != NULL); + error_free(error); + error = NULL; + qobject_unref(src); + + src = qdict_new(); + /* The input should be flat, ie no dicts or lists */ + qdict_put(src, "rule.a", qdict_new()); + qdict_put_str(src, "rule.b", "allow"); + + g_assert(qdict_crumple(src, &error) == NULL); + g_assert(error != NULL); + error_free(error); + error = NULL; + qobject_unref(src); + + src = qdict_new(); + /* List indexes must not have gaps */ + qdict_put_str(src, "rule.0", "deny"); + qdict_put_str(src, "rule.3", "allow"); + + g_assert(qdict_crumple(src, &error) == NULL); + g_assert(error != NULL); + error_free(error); + error = NULL; + qobject_unref(src); + + src = qdict_new(); + /* List indexes must be in %zu format */ + qdict_put_str(src, "rule.0", "deny"); + qdict_put_str(src, "rule.+1", "allow"); + + g_assert(qdict_crumple(src, &error) == NULL); + g_assert(error != NULL); + error_free(error); + error = NULL; + qobject_unref(src); +} + +int main(int argc, char **argv) +{ + g_test_init(&argc, &argv, NULL); + + g_test_add_func("/public/defaults", qdict_defaults_test); + g_test_add_func("/public/flatten", qdict_flatten_test); + g_test_add_func("/public/array_split", qdict_array_split_test); + g_test_add_func("/public/array_entries", qdict_array_entries_test); + g_test_add_func("/public/join", qdict_join_test); + g_test_add_func("/public/crumple/recursive", + qdict_crumple_test_recursive); + g_test_add_func("/public/crumple/empty", + qdict_crumple_test_empty); + g_test_add_func("/public/crumple/bad_inputs", + qdict_crumple_test_bad_inputs); + + g_test_add_func("/public/rename_keys", qdict_rename_keys_test); + + return g_test_run(); +} diff --git a/tests/check-qdict.c b/tests/check-qdict.c index 93e2112b6d..86e9fe7dc4 100644 --- a/tests/check-qdict.c +++ b/tests/check-qdict.c @@ -11,13 +11,7 @@ */ #include "qemu/osdep.h" -#include "block/qdict.h" #include "qapi/qmp/qdict.h" -#include "qapi/qmp/qlist.h" -#include "qapi/qmp/qnum.h" -#include "qapi/qmp/qstring.h" -#include "qapi/error.h" -#include "qemu-common.h" /* * Public Interface test-cases @@ -157,28 +151,6 @@ static void qdict_get_try_str_test(void) qobject_unref(tests_dict); } -static void qdict_defaults_test(void) -{ - QDict *dict, *copy; - - dict = qdict_new(); - copy = qdict_new(); - - qdict_set_default_str(dict, "foo", "abc"); - qdict_set_default_str(dict, "foo", "def"); - g_assert_cmpstr(qdict_get_str(dict, "foo"), ==, "abc"); - qdict_set_default_str(dict, "bar", "ghi"); - - qdict_copy_default(copy, dict, "foo"); - g_assert_cmpstr(qdict_get_str(copy, "foo"), ==, "abc"); - qdict_set_default_str(copy, "bar", "xyz"); - qdict_copy_default(copy, dict, "bar"); - g_assert_cmpstr(qdict_get_str(copy, "bar"), ==, "xyz"); - - qobject_unref(copy); - qobject_unref(dict); -} - static void qdict_haskey_not_test(void) { QDict *tests_dict = qdict_new(); @@ -254,606 +226,6 @@ static void qdict_iterapi_test(void) qobject_unref(tests_dict); } -static void qdict_flatten_test(void) -{ - QList *list1 = qlist_new(); - QList *list2 = qlist_new(); - QDict *dict1 = qdict_new(); - QDict *dict2 = qdict_new(); - QDict *dict3 = qdict_new(); - - /* - * Test the flattening of - * - * { - * "e": [ - * 42, - * [ - * 23, - * 66, - * { - * "a": 0, - * "b": 1 - * } - * ] - * ], - * "f": { - * "c": 2, - * "d": 3, - * }, - * "g": 4 - * } - * - * to - * - * { - * "e.0": 42, - * "e.1.0": 23, - * "e.1.1": 66, - * "e.1.2.a": 0, - * "e.1.2.b": 1, - * "f.c": 2, - * "f.d": 3, - * "g": 4 - * } - */ - - qdict_put_int(dict1, "a", 0); - qdict_put_int(dict1, "b", 1); - - qlist_append_int(list1, 23); - qlist_append_int(list1, 66); - qlist_append(list1, dict1); - qlist_append_int(list2, 42); - qlist_append(list2, list1); - - qdict_put_int(dict2, "c", 2); - qdict_put_int(dict2, "d", 3); - qdict_put(dict3, "e", list2); - qdict_put(dict3, "f", dict2); - qdict_put_int(dict3, "g", 4); - - qdict_flatten(dict3); - - g_assert(qdict_get_int(dict3, "e.0") == 42); - g_assert(qdict_get_int(dict3, "e.1.0") == 23); - g_assert(qdict_get_int(dict3, "e.1.1") == 66); - g_assert(qdict_get_int(dict3, "e.1.2.a") == 0); - g_assert(qdict_get_int(dict3, "e.1.2.b") == 1); - g_assert(qdict_get_int(dict3, "f.c") == 2); - g_assert(qdict_get_int(dict3, "f.d") == 3); - g_assert(qdict_get_int(dict3, "g") == 4); - - g_assert(qdict_size(dict3) == 8); - - qobject_unref(dict3); -} - -static void qdict_array_split_test(void) -{ - QDict *test_dict = qdict_new(); - QDict *dict1, *dict2; - QNum *int1; - QList *test_list; - - /* - * Test the split of - * - * { - * "1.x": 0, - * "4.y": 1, - * "0.a": 42, - * "o.o": 7, - * "0.b": 23, - * "2": 66 - * } - * - * to - * - * [ - * { - * "a": 42, - * "b": 23 - * }, - * { - * "x": 0 - * }, - * 66 - * ] - * - * and - * - * { - * "4.y": 1, - * "o.o": 7 - * } - * - * (remaining in the old QDict) - * - * This example is given in the comment of qdict_array_split(). - */ - - qdict_put_int(test_dict, "1.x", 0); - qdict_put_int(test_dict, "4.y", 1); - qdict_put_int(test_dict, "0.a", 42); - qdict_put_int(test_dict, "o.o", 7); - qdict_put_int(test_dict, "0.b", 23); - qdict_put_int(test_dict, "2", 66); - - qdict_array_split(test_dict, &test_list); - - dict1 = qobject_to(QDict, qlist_pop(test_list)); - dict2 = qobject_to(QDict, qlist_pop(test_list)); - int1 = qobject_to(QNum, qlist_pop(test_list)); - - g_assert(dict1); - g_assert(dict2); - g_assert(int1); - g_assert(qlist_empty(test_list)); - - qobject_unref(test_list); - - g_assert(qdict_get_int(dict1, "a") == 42); - g_assert(qdict_get_int(dict1, "b") == 23); - - g_assert(qdict_size(dict1) == 2); - - qobject_unref(dict1); - - g_assert(qdict_get_int(dict2, "x") == 0); - - g_assert(qdict_size(dict2) == 1); - - qobject_unref(dict2); - - g_assert_cmpint(qnum_get_int(int1), ==, 66); - - qobject_unref(int1); - - g_assert(qdict_get_int(test_dict, "4.y") == 1); - g_assert(qdict_get_int(test_dict, "o.o") == 7); - - g_assert(qdict_size(test_dict) == 2); - - qobject_unref(test_dict); - - /* - * Test the split of - * - * { - * "0": 42, - * "1": 23, - * "1.x": 84 - * } - * - * to - * - * [ - * 42 - * ] - * - * and - * - * { - * "1": 23, - * "1.x": 84 - * } - * - * That is, test whether splitting stops if there is both an entry with key - * of "%u" and other entries with keys prefixed "%u." for the same index. - */ - - test_dict = qdict_new(); - - qdict_put_int(test_dict, "0", 42); - qdict_put_int(test_dict, "1", 23); - qdict_put_int(test_dict, "1.x", 84); - - qdict_array_split(test_dict, &test_list); - - int1 = qobject_to(QNum, qlist_pop(test_list)); - - g_assert(int1); - g_assert(qlist_empty(test_list)); - - qobject_unref(test_list); - - g_assert_cmpint(qnum_get_int(int1), ==, 42); - - qobject_unref(int1); - - g_assert(qdict_get_int(test_dict, "1") == 23); - g_assert(qdict_get_int(test_dict, "1.x") == 84); - - g_assert(qdict_size(test_dict) == 2); - - qobject_unref(test_dict); -} - -static void qdict_array_entries_test(void) -{ - QDict *dict = qdict_new(); - - g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, 0); - - qdict_put_int(dict, "bar", 0); - qdict_put_int(dict, "baz.0", 0); - g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, 0); - - qdict_put_int(dict, "foo.1", 0); - g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, -EINVAL); - qdict_put_int(dict, "foo.0", 0); - g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, 2); - qdict_put_int(dict, "foo.bar", 0); - g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, -EINVAL); - qdict_del(dict, "foo.bar"); - - qdict_put_int(dict, "foo.2.a", 0); - qdict_put_int(dict, "foo.2.b", 0); - qdict_put_int(dict, "foo.2.c", 0); - g_assert_cmpint(qdict_array_entries(dict, "foo."), ==, 3); - g_assert_cmpint(qdict_array_entries(dict, ""), ==, -EINVAL); - - qobject_unref(dict); - - dict = qdict_new(); - qdict_put_int(dict, "1", 0); - g_assert_cmpint(qdict_array_entries(dict, ""), ==, -EINVAL); - qdict_put_int(dict, "0", 0); - g_assert_cmpint(qdict_array_entries(dict, ""), ==, 2); - qdict_put_int(dict, "bar", 0); - g_assert_cmpint(qdict_array_entries(dict, ""), ==, -EINVAL); - qdict_del(dict, "bar"); - - qdict_put_int(dict, "2.a", 0); - qdict_put_int(dict, "2.b", 0); - qdict_put_int(dict, "2.c", 0); - g_assert_cmpint(qdict_array_entries(dict, ""), ==, 3); - - qobject_unref(dict); -} - -static void qdict_join_test(void) -{ - QDict *dict1, *dict2; - bool overwrite = false; - int i; - - dict1 = qdict_new(); - dict2 = qdict_new(); - - /* Test everything once without overwrite and once with */ - do - { - /* Test empty dicts */ - qdict_join(dict1, dict2, overwrite); - - g_assert(qdict_size(dict1) == 0); - g_assert(qdict_size(dict2) == 0); - - /* First iteration: Test movement */ - /* Second iteration: Test empty source and non-empty destination */ - qdict_put_int(dict2, "foo", 42); - - for (i = 0; i < 2; i++) { - qdict_join(dict1, dict2, overwrite); - - g_assert(qdict_size(dict1) == 1); - g_assert(qdict_size(dict2) == 0); - - g_assert(qdict_get_int(dict1, "foo") == 42); - } - - /* Test non-empty source and destination without conflict */ - qdict_put_int(dict2, "bar", 23); - - qdict_join(dict1, dict2, overwrite); - - g_assert(qdict_size(dict1) == 2); - g_assert(qdict_size(dict2) == 0); - - g_assert(qdict_get_int(dict1, "foo") == 42); - g_assert(qdict_get_int(dict1, "bar") == 23); - - /* Test conflict */ - qdict_put_int(dict2, "foo", 84); - - qdict_join(dict1, dict2, overwrite); - - g_assert(qdict_size(dict1) == 2); - g_assert(qdict_size(dict2) == !overwrite); - - g_assert(qdict_get_int(dict1, "foo") == (overwrite ? 84 : 42)); - g_assert(qdict_get_int(dict1, "bar") == 23); - - if (!overwrite) { - g_assert(qdict_get_int(dict2, "foo") == 84); - } - - /* Check the references */ - g_assert(qdict_get(dict1, "foo")->base.refcnt == 1); - g_assert(qdict_get(dict1, "bar")->base.refcnt == 1); - - if (!overwrite) { - g_assert(qdict_get(dict2, "foo")->base.refcnt == 1); - } - - /* Clean up */ - qdict_del(dict1, "foo"); - qdict_del(dict1, "bar"); - - if (!overwrite) { - qdict_del(dict2, "foo"); - } - } - while (overwrite ^= true); - - qobject_unref(dict1); - qobject_unref(dict2); -} - -static void qdict_crumple_test_recursive(void) -{ - QDict *src, *dst, *rule, *vnc, *acl, *listen; - QList *rules; - - src = qdict_new(); - qdict_put_str(src, "vnc.listen.addr", "127.0.0.1"); - qdict_put_str(src, "vnc.listen.port", "5901"); - qdict_put_str(src, "vnc.acl.rules.0.match", "fred"); - qdict_put_str(src, "vnc.acl.rules.0.policy", "allow"); - qdict_put_str(src, "vnc.acl.rules.1.match", "bob"); - qdict_put_str(src, "vnc.acl.rules.1.policy", "deny"); - qdict_put_str(src, "vnc.acl.default", "deny"); - qdict_put_str(src, "vnc.acl..name", "acl0"); - qdict_put_str(src, "vnc.acl.rule..name", "acl0"); - - dst = qobject_to(QDict, qdict_crumple(src, &error_abort)); - g_assert(dst); - g_assert_cmpint(qdict_size(dst), ==, 1); - - vnc = qdict_get_qdict(dst, "vnc"); - g_assert(vnc); - g_assert_cmpint(qdict_size(vnc), ==, 3); - - listen = qdict_get_qdict(vnc, "listen"); - g_assert(listen); - g_assert_cmpint(qdict_size(listen), ==, 2); - g_assert_cmpstr("127.0.0.1", ==, qdict_get_str(listen, "addr")); - g_assert_cmpstr("5901", ==, qdict_get_str(listen, "port")); - - acl = qdict_get_qdict(vnc, "acl"); - g_assert(acl); - g_assert_cmpint(qdict_size(acl), ==, 3); - - rules = qdict_get_qlist(acl, "rules"); - g_assert(rules); - g_assert_cmpint(qlist_size(rules), ==, 2); - - rule = qobject_to(QDict, qlist_pop(rules)); - g_assert(rule); - g_assert_cmpint(qdict_size(rule), ==, 2); - g_assert_cmpstr("fred", ==, qdict_get_str(rule, "match")); - g_assert_cmpstr("allow", ==, qdict_get_str(rule, "policy")); - qobject_unref(rule); - - rule = qobject_to(QDict, qlist_pop(rules)); - g_assert(rule); - g_assert_cmpint(qdict_size(rule), ==, 2); - g_assert_cmpstr("bob", ==, qdict_get_str(rule, "match")); - g_assert_cmpstr("deny", ==, qdict_get_str(rule, "policy")); - qobject_unref(rule); - - /* With recursive crumpling, we should see all names unescaped */ - g_assert_cmpstr("acl0", ==, qdict_get_str(vnc, "acl.name")); - g_assert_cmpstr("acl0", ==, qdict_get_str(acl, "rule.name")); - - qobject_unref(src); - qobject_unref(dst); -} - -static void qdict_crumple_test_empty(void) -{ - QDict *src, *dst; - - src = qdict_new(); - - dst = qobject_to(QDict, qdict_crumple(src, &error_abort)); - - g_assert_cmpint(qdict_size(dst), ==, 0); - - qobject_unref(src); - qobject_unref(dst); -} - -static int qdict_count_entries(QDict *dict) -{ - const QDictEntry *e; - int count = 0; - - for (e = qdict_first(dict); e; e = qdict_next(dict, e)) { - count++; - } - - return count; -} - -static void qdict_rename_keys_test(void) -{ - QDict *dict = qdict_new(); - QDict *copy; - QDictRenames *renames; - Error *local_err = NULL; - - qdict_put_str(dict, "abc", "foo"); - qdict_put_str(dict, "abcdef", "bar"); - qdict_put_int(dict, "number", 42); - qdict_put_bool(dict, "flag", true); - qdict_put_null(dict, "nothing"); - - /* Empty rename list */ - renames = (QDictRenames[]) { - { NULL, "this can be anything" } - }; - copy = qdict_clone_shallow(dict); - qdict_rename_keys(copy, renames, &error_abort); - - g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "foo"); - g_assert_cmpstr(qdict_get_str(copy, "abcdef"), ==, "bar"); - g_assert_cmpint(qdict_get_int(copy, "number"), ==, 42); - g_assert_cmpint(qdict_get_bool(copy, "flag"), ==, true); - g_assert(qobject_type(qdict_get(copy, "nothing")) == QTYPE_QNULL); - g_assert_cmpint(qdict_count_entries(copy), ==, 5); - - qobject_unref(copy); - - /* Simple rename of all entries */ - renames = (QDictRenames[]) { - { "abc", "str1" }, - { "abcdef", "str2" }, - { "number", "int" }, - { "flag", "bool" }, - { "nothing", "null" }, - { NULL , NULL } - }; - copy = qdict_clone_shallow(dict); - qdict_rename_keys(copy, renames, &error_abort); - - g_assert(!qdict_haskey(copy, "abc")); - g_assert(!qdict_haskey(copy, "abcdef")); - g_assert(!qdict_haskey(copy, "number")); - g_assert(!qdict_haskey(copy, "flag")); - g_assert(!qdict_haskey(copy, "nothing")); - - g_assert_cmpstr(qdict_get_str(copy, "str1"), ==, "foo"); - g_assert_cmpstr(qdict_get_str(copy, "str2"), ==, "bar"); - g_assert_cmpint(qdict_get_int(copy, "int"), ==, 42); - g_assert_cmpint(qdict_get_bool(copy, "bool"), ==, true); - g_assert(qobject_type(qdict_get(copy, "null")) == QTYPE_QNULL); - g_assert_cmpint(qdict_count_entries(copy), ==, 5); - - qobject_unref(copy); - - /* Renames are processed top to bottom */ - renames = (QDictRenames[]) { - { "abc", "tmp" }, - { "abcdef", "abc" }, - { "number", "abcdef" }, - { "flag", "number" }, - { "nothing", "flag" }, - { "tmp", "nothing" }, - { NULL , NULL } - }; - copy = qdict_clone_shallow(dict); - qdict_rename_keys(copy, renames, &error_abort); - - g_assert_cmpstr(qdict_get_str(copy, "nothing"), ==, "foo"); - g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "bar"); - g_assert_cmpint(qdict_get_int(copy, "abcdef"), ==, 42); - g_assert_cmpint(qdict_get_bool(copy, "number"), ==, true); - g_assert(qobject_type(qdict_get(copy, "flag")) == QTYPE_QNULL); - g_assert(!qdict_haskey(copy, "tmp")); - g_assert_cmpint(qdict_count_entries(copy), ==, 5); - - qobject_unref(copy); - - /* Conflicting rename */ - renames = (QDictRenames[]) { - { "abcdef", "abc" }, - { NULL , NULL } - }; - copy = qdict_clone_shallow(dict); - qdict_rename_keys(copy, renames, &local_err); - - g_assert(local_err != NULL); - error_free(local_err); - local_err = NULL; - - g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "foo"); - g_assert_cmpstr(qdict_get_str(copy, "abcdef"), ==, "bar"); - g_assert_cmpint(qdict_get_int(copy, "number"), ==, 42); - g_assert_cmpint(qdict_get_bool(copy, "flag"), ==, true); - g_assert(qobject_type(qdict_get(copy, "nothing")) == QTYPE_QNULL); - g_assert_cmpint(qdict_count_entries(copy), ==, 5); - - qobject_unref(copy); - - /* Renames in an empty dict */ - renames = (QDictRenames[]) { - { "abcdef", "abc" }, - { NULL , NULL } - }; - - qobject_unref(dict); - dict = qdict_new(); - - qdict_rename_keys(dict, renames, &error_abort); - g_assert(qdict_first(dict) == NULL); - - qobject_unref(dict); -} - -static void qdict_crumple_test_bad_inputs(void) -{ - QDict *src; - Error *error = NULL; - - src = qdict_new(); - /* rule.0 can't be both a string and a dict */ - qdict_put_str(src, "rule.0", "fred"); - qdict_put_str(src, "rule.0.policy", "allow"); - - g_assert(qdict_crumple(src, &error) == NULL); - g_assert(error != NULL); - error_free(error); - error = NULL; - qobject_unref(src); - - src = qdict_new(); - /* rule can't be both a list and a dict */ - qdict_put_str(src, "rule.0", "fred"); - qdict_put_str(src, "rule.a", "allow"); - - g_assert(qdict_crumple(src, &error) == NULL); - g_assert(error != NULL); - error_free(error); - error = NULL; - qobject_unref(src); - - src = qdict_new(); - /* The input should be flat, ie no dicts or lists */ - qdict_put(src, "rule.a", qdict_new()); - qdict_put_str(src, "rule.b", "allow"); - - g_assert(qdict_crumple(src, &error) == NULL); - g_assert(error != NULL); - error_free(error); - error = NULL; - qobject_unref(src); - - src = qdict_new(); - /* List indexes must not have gaps */ - qdict_put_str(src, "rule.0", "deny"); - qdict_put_str(src, "rule.3", "allow"); - - g_assert(qdict_crumple(src, &error) == NULL); - g_assert(error != NULL); - error_free(error); - error = NULL; - qobject_unref(src); - - src = qdict_new(); - /* List indexes must be in %zu format */ - qdict_put_str(src, "rule.0", "deny"); - qdict_put_str(src, "rule.+1", "allow"); - - g_assert(qdict_crumple(src, &error) == NULL); - g_assert(error != NULL); - error_free(error); - error = NULL; - qobject_unref(src); -} - /* * Errors test-cases */ @@ -987,29 +359,15 @@ int main(int argc, char **argv) g_test_add_func("/public/get_try_int", qdict_get_try_int_test); g_test_add_func("/public/get_str", qdict_get_str_test); g_test_add_func("/public/get_try_str", qdict_get_try_str_test); - g_test_add_func("/public/defaults", qdict_defaults_test); g_test_add_func("/public/haskey_not", qdict_haskey_not_test); g_test_add_func("/public/haskey", qdict_haskey_test); g_test_add_func("/public/del", qdict_del_test); g_test_add_func("/public/to_qdict", qobject_to_qdict_test); g_test_add_func("/public/iterapi", qdict_iterapi_test); - g_test_add_func("/public/flatten", qdict_flatten_test); - g_test_add_func("/public/array_split", qdict_array_split_test); - g_test_add_func("/public/array_entries", qdict_array_entries_test); - g_test_add_func("/public/join", qdict_join_test); g_test_add_func("/errors/put_exists", qdict_put_exists_test); g_test_add_func("/errors/get_not_exists", qdict_get_not_exists_test); - g_test_add_func("/public/crumple/recursive", - qdict_crumple_test_recursive); - g_test_add_func("/public/crumple/empty", - qdict_crumple_test_empty); - g_test_add_func("/public/crumple/bad_inputs", - qdict_crumple_test_bad_inputs); - - g_test_add_func("/public/rename_keys", qdict_rename_keys_test); - /* The Big one */ if (g_test_slow()) { g_test_add_func("/stress/test", qdict_stress_test); -- 2.17.1 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header 2018-06-06 14:19 ` Markus Armbruster @ 2018-06-06 14:35 ` Markus Armbruster 0 siblings, 0 replies; 42+ messages in thread From: Markus Armbruster @ 2018-06-06 14:35 UTC (permalink / raw) To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Michael Roth Markus Armbruster <armbru@redhat.com> writes: > Markus Armbruster <armbru@redhat.com> writes: > >> MAINTAINERS files include/block/qdict.h under "Block layer core". Let's >> split qobject/qdict.c as well, so it's there, too. Obvious patch >> appended. Feel free to squash it into yours without giving me credit >> for it. > > Hmm, we should split tests/check-qdict.c for the same reason. Updated > patch appended. > >> PS: I tried to move block-qdict.c to block/, but totally failed at >> persuading Make to link it into all programs that need it. Oh well. > > > From bdb7f6581a3066db853bca6c81cc2c42f6cec8e9 Mon Sep 17 00:00:00 2001 > From: Markus Armbruster <armbru@redhat.com> > Date: Wed, 6 Jun 2018 15:11:54 +0200 > Subject: [PATCH] qobject: Move block-specific qdict code to block-qdict.c > > Signed-off-by: Markus Armbruster <armbru@redhat.com> > --- > MAINTAINERS | 1 + > qobject/Makefile.objs | 1 + > qobject/block-qdict.c | 637 ++++++++++++++++++++++++++++++++++++ > qobject/qdict.c | 629 ------------------------------------ > tests/Makefile.include | 4 + > tests/check-block-qdict.c | 657 ++++++++++++++++++++++++++++++++++++++ > tests/check-qdict.c | 642 ------------------------------------- > 7 files changed, 1300 insertions(+), 1271 deletions(-) > create mode 100644 qobject/block-qdict.c > create mode 100644 tests/check-block-qdict.c > > diff --git a/MAINTAINERS b/MAINTAINERS > index 41cd3736a9..9f9835e052 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -1365,6 +1365,7 @@ F: qemu-img* > F: qemu-io* > F: tests/qemu-iotests/ > F: util/qemu-progress.c > +F: qobject/block-qdict.c > T: git git://repo.or.cz/qemu/kevin.git block > > Block I/O path Forgot to update MAINTAINERS for tests/check-block-qdict.c. Fixup: diff --git a/MAINTAINERS b/MAINTAINERS index 9f9835e052..21f4bc1806 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1366,6 +1366,7 @@ F: qemu-io* F: tests/qemu-iotests/ F: util/qemu-progress.c F: qobject/block-qdict.c +F: test/check-block-qdict.c T: git git://repo.or.cz/qemu/kevin.git block Block I/O path [...] ^ permalink raw reply related [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval() 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (5 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-11 18:39 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test Max Reitz ` (5 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth The purpose of this function is to prepare a QDict for consumption by the keyval visitor, which only accepts strings and QNull. Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/qdict.h | 2 ++ qobject/qdict.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/include/block/qdict.h b/include/block/qdict.h index 825e096a72..05a24677b5 100644 --- a/include/block/qdict.h +++ b/include/block/qdict.h @@ -32,4 +32,6 @@ typedef struct QDictRenames { } QDictRenames; bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp); +void qdict_stringify_for_keyval(QDict *qdict); + #endif diff --git a/qobject/qdict.c b/qobject/qdict.c index 0554c64553..83678c4951 100644 --- a/qobject/qdict.c +++ b/qobject/qdict.c @@ -1087,3 +1087,60 @@ bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) } return true; } + +/** + * Convert all values in a QDict so it can be consumed by the keyval + * input visitor. QNull values are left as-is, all other values are + * converted to strings. + * + * @qdict must be flattened, i.e. it may not contain any nested QDicts + * or QLists. + */ +void qdict_stringify_for_keyval(QDict *qdict) +{ + const QDictEntry *e; + + for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) { + QString *new_value = NULL; + + switch (qobject_type(e->value)) { + case QTYPE_QNULL: + /* leave as-is */ + break; + + case QTYPE_QNUM: { + char *str = qnum_to_string(qobject_to(QNum, e->value)); + new_value = qstring_from_str(str); + g_free(str); + break; + } + + case QTYPE_QSTRING: + /* leave as-is */ + break; + + case QTYPE_QDICT: + case QTYPE_QLIST: + /* @qdict must be flattened */ + abort(); + + case QTYPE_QBOOL: + if (qbool_get_bool(qobject_to(QBool, e->value))) { + new_value = qstring_from_str("on"); + } else { + new_value = qstring_from_str("off"); + } + break; + + case QTYPE_NONE: + case QTYPE__MAX: + abort(); + } + + if (new_value) { + QDictEntry *mut_e = (QDictEntry *)e; + qobject_unref(mut_e->value); + mut_e->value = QOBJECT(new_value); + } + } +} -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval() 2018-05-09 16:55 ` [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval() Max Reitz @ 2018-05-11 18:39 ` Eric Blake 2018-05-11 21:42 ` Max Reitz 0 siblings, 1 reply; 42+ messages in thread From: Eric Blake @ 2018-05-11 18:39 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > The purpose of this function is to prepare a QDict for consumption by > the keyval visitor, which only accepts strings and QNull. > > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > include/block/qdict.h | 2 ++ > qobject/qdict.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 59 insertions(+) > > +/** > + * Convert all values in a QDict so it can be consumed by the keyval > + * input visitor. QNull values are left as-is, all other values are > + * converted to strings. > + * > + * @qdict must be flattened, i.e. it may not contain any nested QDicts > + * or QLists. Maybe s/flattened/flattened already/ or would it be worth letting this function PERFORM the flattening step automatically? > + */ > +void qdict_stringify_for_keyval(QDict *qdict) > +{ > + const QDictEntry *e; > + > + for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) { > + QString *new_value = NULL; > + > + switch (qobject_type(e->value)) { > + case QTYPE_QNULL: > + /* leave as-is */ > + break; > + > + case QTYPE_QNUM: { > + char *str = qnum_to_string(qobject_to(QNum, e->value)); > + new_value = qstring_from_str(str); > + g_free(str); > + break; > + } > + > + case QTYPE_QSTRING: > + /* leave as-is */ > + break; > + > + case QTYPE_QDICT: > + case QTYPE_QLIST: > + /* @qdict must be flattened */ > + abort(); Matches your documentation about requiring it to be already flattened. > + > + case QTYPE_QBOOL: > + if (qbool_get_bool(qobject_to(QBool, e->value))) { > + new_value = qstring_from_str("on"); > + } else { > + new_value = qstring_from_str("off"); > + } > + break; > + > + case QTYPE_NONE: > + case QTYPE__MAX: > + abort(); > + } > + > + if (new_value) { > + QDictEntry *mut_e = (QDictEntry *)e; A bit of a shame that you have to cast away const. It took me a couple reads to figure out this meant 'mutable_e'. But in the end, the code looks correct. > + qobject_unref(mut_e->value); > + mut_e->value = QOBJECT(new_value); If we're okay requiring the caller to pre-flatten before calling this, instead of offering to do it here, Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval() 2018-05-11 18:39 ` Eric Blake @ 2018-05-11 21:42 ` Max Reitz 0 siblings, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-11 21:42 UTC (permalink / raw) To: Eric Blake, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth [-- Attachment #1: Type: text/plain, Size: 3339 bytes --] On 2018-05-11 20:39, Eric Blake wrote: > On 05/09/2018 11:55 AM, Max Reitz wrote: >> The purpose of this function is to prepare a QDict for consumption by >> the keyval visitor, which only accepts strings and QNull. >> >> Signed-off-by: Max Reitz <mreitz@redhat.com> >> --- >> include/block/qdict.h | 2 ++ >> qobject/qdict.c | 57 >> +++++++++++++++++++++++++++++++++++++++++++++++++++ >> 2 files changed, 59 insertions(+) >> > >> +/** >> + * Convert all values in a QDict so it can be consumed by the keyval >> + * input visitor. QNull values are left as-is, all other values are >> + * converted to strings. >> + * >> + * @qdict must be flattened, i.e. it may not contain any nested QDicts >> + * or QLists. > > Maybe s/flattened/flattened already/ > > or would it be worth letting this function PERFORM the flattening step > automatically? Possibly, but I don't think it would be any more efficient, so I'd rather leave it up to the caller to do it explicitly than to do it here and maybe surprise someone. (Indeed I personally prefer to be surprised by an abort() than by some unintended data change.) Max >> + */ >> +void qdict_stringify_for_keyval(QDict *qdict) >> +{ >> + const QDictEntry *e; >> + >> + for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) { >> + QString *new_value = NULL; >> + >> + switch (qobject_type(e->value)) { >> + case QTYPE_QNULL: >> + /* leave as-is */ >> + break; >> + >> + case QTYPE_QNUM: { >> + char *str = qnum_to_string(qobject_to(QNum, e->value)); >> + new_value = qstring_from_str(str); >> + g_free(str); >> + break; >> + } >> + >> + case QTYPE_QSTRING: >> + /* leave as-is */ >> + break; >> + >> + case QTYPE_QDICT: >> + case QTYPE_QLIST: >> + /* @qdict must be flattened */ >> + abort(); > > Matches your documentation about requiring it to be already flattened. > >> + >> + case QTYPE_QBOOL: >> + if (qbool_get_bool(qobject_to(QBool, e->value))) { >> + new_value = qstring_from_str("on"); >> + } else { >> + new_value = qstring_from_str("off"); >> + } >> + break; >> + >> + case QTYPE_NONE: >> + case QTYPE__MAX: >> + abort(); >> + } >> + >> + if (new_value) { >> + QDictEntry *mut_e = (QDictEntry *)e; > > A bit of a shame that you have to cast away const. It took me a couple > reads to figure out this meant 'mutable_e'. But in the end, the code > looks correct. > >> + qobject_unref(mut_e->value); >> + mut_e->value = QOBJECT(new_value); > > If we're okay requiring the caller to pre-flatten before calling this, > instead of offering to do it here, > Reviewed-by: Eric Blake <eblake@redhat.com> > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (6 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval() Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-10 16:02 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 09/13] qdict: Make qdict_flatten() shallow-clone-friendly Max Reitz ` (4 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/check-qdict.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/tests/check-qdict.c b/tests/check-qdict.c index 93e2112b6d..ef5d17f815 100644 --- a/tests/check-qdict.c +++ b/tests/check-qdict.c @@ -973,6 +973,57 @@ static void qdict_stress_test(void) qobject_unref(qdict); } +static void qdict_stringify_for_keyval_test(void) +{ + QDict *dict = qdict_new(); + + /* + * Test stringification of: + * + * { + * "a": "null", + * "b": 42, + * "c": -23, + * "d": false, + * "e": null, + * "f": "", + * "g": 0.5, + * "h": 0xffffffffffffffff, + * "i": true, + * "j": 0 + * } + * + * Note that null is not transformed into a string and remains a + * QNull. + */ + + qdict_put_str(dict, "a", "null"); + qdict_put_int(dict, "b", 42); + qdict_put_int(dict, "c", -23); + qdict_put_bool(dict, "d", false); + qdict_put_null(dict, "e"); + qdict_put_str(dict, "f", ""); + qdict_put(dict, "g", qnum_from_double(0.5)); + qdict_put(dict, "h", qnum_from_uint(0xffffffffffffffffull)); + qdict_put_bool(dict, "i", true); + qdict_put_int(dict, "j", 0); + + qdict_stringify_for_keyval(dict); + + g_assert(!strcmp(qdict_get_str(dict, "a"), "null")); + g_assert(!strcmp(qdict_get_str(dict, "b"), "42")); + g_assert(!strcmp(qdict_get_str(dict, "c"), "-23")); + g_assert(!strcmp(qdict_get_str(dict, "d"), "off")); + g_assert(qobject_type(qdict_get(dict, "e")) == QTYPE_QNULL); + g_assert(!strcmp(qdict_get_str(dict, "f"), "")); + g_assert(!strcmp(qdict_get_str(dict, "g"), "0.5")); + g_assert(!strcmp(qdict_get_str(dict, "h"), "18446744073709551615")); + g_assert(!strcmp(qdict_get_str(dict, "i"), "on")); + g_assert(!strcmp(qdict_get_str(dict, "j"), "0")); + + qobject_unref(dict); +} + int main(int argc, char **argv) { g_test_init(&argc, &argv, NULL); @@ -1010,6 +1061,9 @@ int main(int argc, char **argv) g_test_add_func("/public/rename_keys", qdict_rename_keys_test); + g_test_add_func("/public/stringify_for_keyval", + qdict_stringify_for_keyval_test); + /* The Big one */ if (g_test_slow()) { g_test_add_func("/stress/test", qdict_stress_test); -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test 2018-05-09 16:55 ` [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test Max Reitz @ 2018-05-10 16:02 ` Eric Blake 2018-05-11 18:13 ` Max Reitz 0 siblings, 1 reply; 42+ messages in thread From: Eric Blake @ 2018-05-10 16:02 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > tests/check-qdict.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 54 insertions(+) > > +static void qdict_stringify_for_keyval_test(void) > +{ > + QDict *dict = qdict_new(); > + > + /* > + * Test stringification of: > + * > + * { > + * "a": "null", > + * "b": 42, > + * "c": -23, > + * "d": false, > + * "e": null, > + * "f": "", > + * "g": 0.5, > + * "h": 0xffffffffffffffff, > + * "i": true, > + * "j": 0 Is it worth testing fun things like '-0.0'? > + g_assert(!strcmp(qdict_get_str(dict, "a"), "null")); > + g_assert(!strcmp(qdict_get_str(dict, "b"), "42")); > + g_assert(!strcmp(qdict_get_str(dict, "c"), "-23")); > + g_assert(!strcmp(qdict_get_str(dict, "d"), "off")); > + g_assert(qobject_type(qdict_get(dict, "e")) == QTYPE_QNULL); Is it worth shortening this line to: g_assert(qobject_to(QNull, qdict_get(dict, "e"))); Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test 2018-05-10 16:02 ` Eric Blake @ 2018-05-11 18:13 ` Max Reitz 2018-05-11 18:33 ` Eric Blake 0 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-11 18:13 UTC (permalink / raw) To: Eric Blake, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth [-- Attachment #1: Type: text/plain, Size: 1539 bytes --] On 2018-05-10 18:02, Eric Blake wrote: > On 05/09/2018 11:55 AM, Max Reitz wrote: >> Signed-off-by: Max Reitz <mreitz@redhat.com> >> --- >> tests/check-qdict.c | 54 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 54 insertions(+) >> > >> +static void qdict_stringify_for_keyval_test(void) >> +{ >> + QDict *dict = qdict_new(); >> + >> + /* >> + * Test stringification of: >> + * >> + * { >> + * "a": "null", >> + * "b": 42, >> + * "c": -23, >> + * "d": false, >> + * "e": null, >> + * "f": "", >> + * "g": 0.5, >> + * "h": 0xffffffffffffffff, >> + * "i": true, >> + * "j": 0 > > Is it worth testing fun things like '-0.0'? Sure, why not. Maybe even infinity, although I'm not quite sure the input visitor can handle it... >> + g_assert(!strcmp(qdict_get_str(dict, "a"), "null")); >> + g_assert(!strcmp(qdict_get_str(dict, "b"), "42")); >> + g_assert(!strcmp(qdict_get_str(dict, "c"), "-23")); >> + g_assert(!strcmp(qdict_get_str(dict, "d"), "off")); >> + g_assert(qobject_type(qdict_get(dict, "e")) == QTYPE_QNULL); > > Is it worth shortening this line to: > g_assert(qobject_to(QNull, qdict_get(dict, "e"))); I think explicitly checking the type is a bit more expressive. Max > Reviewed-by: Eric Blake <eblake@redhat.com> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test 2018-05-11 18:13 ` Max Reitz @ 2018-05-11 18:33 ` Eric Blake 0 siblings, 0 replies; 42+ messages in thread From: Eric Blake @ 2018-05-11 18:33 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/11/2018 01:13 PM, Max Reitz wrote: >>> + * "h": 0xffffffffffffffff, >>> + * "i": true, >>> + * "j": 0 >> >> Is it worth testing fun things like '-0.0'? > > Sure, why not. Maybe even infinity, although I'm not quite sure the > input visitor can handle it... JSON can't handle Inf or NaN, even if the input visitor can. So probably best to not worry about those. >>> + g_assert(!strcmp(qdict_get_str(dict, "a"), "null")); >>> + g_assert(!strcmp(qdict_get_str(dict, "b"), "42")); >>> + g_assert(!strcmp(qdict_get_str(dict, "c"), "-23")); >>> + g_assert(!strcmp(qdict_get_str(dict, "d"), "off")); >>> + g_assert(qobject_type(qdict_get(dict, "e")) == QTYPE_QNULL); >> >> Is it worth shortening this line to: >> g_assert(qobject_to(QNull, qdict_get(dict, "e"))); > > I think explicitly checking the type is a bit more expressive. Okay (qobject_to() checks the type under the hood, and returns non-NULL only when it was the right type - but I see your point about that being a bit more magic) -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 09/13] qdict: Make qdict_flatten() shallow-clone-friendly 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (7 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-11 18:44 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test Max Reitz ` (3 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth In its current form, qdict_flatten() removes all entries from nested QDicts that are moved to the root QDict. It is completely sufficient to remove all old entries from the root QDict, however. If the nested dicts have a refcount of 1, this will automatically delete them, too. And if they have a greater refcount, we probably do not want to modify them in the first place. The latter observation means that it was currently (in general) impossible to qdict_flatten() a shallowly cloned dict because that would empty nested QDicts in the original dict as well. This patch changes this, so you can now use qdict_flatten(qdict_shallow_clone(dict)) to get a flattened copy without disturbing the original. Signed-off-by: Max Reitz <mreitz@redhat.com> --- qobject/qdict.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/qobject/qdict.c b/qobject/qdict.c index 83678c4951..63dcb43e73 100644 --- a/qobject/qdict.c +++ b/qobject/qdict.c @@ -537,7 +537,7 @@ static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) QObject *value; const QDictEntry *entry, *next; char *new_key; - bool delete; + bool copied_value; entry = qdict_first(qdict); @@ -546,7 +546,7 @@ static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) next = qdict_next(qdict, entry); value = qdict_entry_value(entry); new_key = NULL; - delete = false; + copied_value = false; if (prefix) { new_key = g_strdup_printf("%s.%s", prefix, entry->key); @@ -557,23 +557,32 @@ static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) * itself disappears. */ qdict_flatten_qdict(qobject_to(QDict, value), target, new_key ? new_key : entry->key); - delete = true; + copied_value = true; } else if (qobject_type(value) == QTYPE_QLIST) { qdict_flatten_qlist(qobject_to(QList, value), target, new_key ? new_key : entry->key); - delete = true; + copied_value = true; } else if (prefix) { /* All other objects are moved to the target unchanged. */ qdict_put_obj(target, new_key, qobject_ref(value)); - delete = true; + copied_value = true; } g_free(new_key); - if (delete) { + if (copied_value && qdict == target) { + /* If we have copied a value, and if we are on the root + * level, we need to remove the old entry. Otherwise, we + * do not, because by removing these entries on the root + * level, the reference counts of nested dicts and listed + * will be reduced automatically. In fact, we probably do + * not want to modify nested dicts and lists with + * refcounts greater than 1 anyway. */ qdict_del(qdict, entry->key); - /* Restart loop after modifying the iterated QDict */ + /* Restart loop after modifying the iterated QDict. We + * only need to do this if qdict == target, because + * otherwise copying the value did not affect qdict. */ entry = qdict_first(qdict); continue; } -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 09/13] qdict: Make qdict_flatten() shallow-clone-friendly 2018-05-09 16:55 ` [Qemu-devel] [PATCH 09/13] qdict: Make qdict_flatten() shallow-clone-friendly Max Reitz @ 2018-05-11 18:44 ` Eric Blake 0 siblings, 0 replies; 42+ messages in thread From: Eric Blake @ 2018-05-11 18:44 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > In its current form, qdict_flatten() removes all entries from nested > QDicts that are moved to the root QDict. It is completely sufficient to > remove all old entries from the root QDict, however. If the nested > dicts have a refcount of 1, this will automatically delete them, too. > And if they have a greater refcount, we probably do not want to modify > them in the first place. > > The latter observation means that it was currently (in general) > impossible to qdict_flatten() a shallowly cloned dict because that would > empty nested QDicts in the original dict as well. This patch changes > this, so you can now use qdict_flatten(qdict_shallow_clone(dict)) to get > a flattened copy without disturbing the original. > > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > qobject/qdict.c | 23 ++++++++++++++++------- > 1 file changed, 16 insertions(+), 7 deletions(-) > > > g_free(new_key); > > - if (delete) { > + if (copied_value && qdict == target) { > + /* If we have copied a value, and if we are on the root > + * level, we need to remove the old entry. Otherwise, we > + * do not, because by removing these entries on the root > + * level, the reference counts of nested dicts and listed s/listed/lists/ > + * will be reduced automatically. In fact, we probably do > + * not want to modify nested dicts and lists with > + * refcounts greater than 1 anyway. */ > qdict_del(qdict, entry->key); > > - /* Restart loop after modifying the iterated QDict */ > + /* Restart loop after modifying the iterated QDict. We > + * only need to do this if qdict == target, because > + * otherwise copying the value did not affect qdict. */ > entry = qdict_first(qdict); > continue; > } > With the typo fix, Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (8 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 09/13] qdict: Make qdict_flatten() shallow-clone-friendly Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-11 18:46 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 11/13] block: Try to create well typed json:{} filenames Max Reitz ` (2 subsequent siblings) 12 siblings, 1 reply; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth This new test verifies that qdict_flatten() does not modify a shallow clone of the given QDict. Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/check-qdict.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/tests/check-qdict.c b/tests/check-qdict.c index ef5d17f815..7960b4a385 100644 --- a/tests/check-qdict.c +++ b/tests/check-qdict.c @@ -329,6 +329,38 @@ static void qdict_flatten_test(void) qobject_unref(dict3); } +static void qdict_clone_flatten_test(void) +{ + QDict *dict1 = qdict_new(); + QDict *dict2 = qdict_new(); + QDict *cloned_dict1; + + /* + * Test that we can clone and flatten + * { "a": { "b": 42 } } + * without modifying the clone. + */ + + qdict_put_int(dict2, "b", 42); + qdict_put(dict1, "a", dict2); + + cloned_dict1 = qdict_clone_shallow(dict1); + + qdict_flatten(dict1); + + g_assert(qdict_size(dict1) == 1); + g_assert(qdict_get_int(dict1, "a.b") == 42); + + g_assert(qdict_size(cloned_dict1) == 1); + g_assert(qdict_get_qdict(cloned_dict1, "a") == dict2); + + g_assert(qdict_size(dict2) == 1); + g_assert(qdict_get_int(dict2, "b") == 42); + + qobject_unref(dict1); + qobject_unref(cloned_dict1); +} + static void qdict_array_split_test(void) { QDict *test_dict = qdict_new(); @@ -1045,6 +1077,7 @@ int main(int argc, char **argv) g_test_add_func("/public/to_qdict", qobject_to_qdict_test); g_test_add_func("/public/iterapi", qdict_iterapi_test); g_test_add_func("/public/flatten", qdict_flatten_test); + g_test_add_func("/public/clone-flatten", qdict_clone_flatten_test); g_test_add_func("/public/array_split", qdict_array_split_test); g_test_add_func("/public/array_entries", qdict_array_entries_test); g_test_add_func("/public/join", qdict_join_test); -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test 2018-05-09 16:55 ` [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test Max Reitz @ 2018-05-11 18:46 ` Eric Blake 2018-05-11 21:41 ` Max Reitz 0 siblings, 1 reply; 42+ messages in thread From: Eric Blake @ 2018-05-11 18:46 UTC (permalink / raw) To: Max Reitz, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth On 05/09/2018 11:55 AM, Max Reitz wrote: > This new test verifies that qdict_flatten() does not modify a shallow > clone of the given QDict. > > Signed-off-by: Max Reitz <mreitz@redhat.com> > --- > tests/check-qdict.c | 33 +++++++++++++++++++++++++++++++++ > 1 file changed, 33 insertions(+) > I'm not sure I even want to know how long it took you to debug the crash that you obviously hit before adding the fix in 9/13 plus this test ;) Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test 2018-05-11 18:46 ` Eric Blake @ 2018-05-11 21:41 ` Max Reitz 0 siblings, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-11 21:41 UTC (permalink / raw) To: Eric Blake, qemu-block Cc: qemu-devel, Markus Armbruster, Kevin Wolf, Michael Roth [-- Attachment #1: Type: text/plain, Size: 1217 bytes --] On 2018-05-11 20:46, Eric Blake wrote: > On 05/09/2018 11:55 AM, Max Reitz wrote: >> This new test verifies that qdict_flatten() does not modify a shallow >> clone of the given QDict. >> >> Signed-off-by: Max Reitz <mreitz@redhat.com> >> --- >> tests/check-qdict.c | 33 +++++++++++++++++++++++++++++++++ >> 1 file changed, 33 insertions(+) >> > > I'm not sure I even want to know how long it took you to debug the crash > that you obviously hit before adding the fix in 9/13 plus this test ;) Thank you very much, I made myself forget about that trauma already. In short, I wondered why the whole thing worked for null-co directly: driver=null-co,size=512 => {"driver": "null-co", "size": 512} But not for null-co through raw: driver=raw,file.driver=null-co,file.size=512 => {"file": {}} (Or something like that, I don't remember exactly.) With some debugging sprinkled into block.c, I could see that the correct options were there on the null-co level... But for some reason they disappeared one level above. Then I recalled that dict cloning is just a shallow cloning and looked for the culprit... Of course, in reality, much more cursing was involved. Max [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 11/13] block: Try to create well typed json:{} filenames 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (9 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 12/13] iotests: Test internal option typing Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 13/13] iotests: qcow2's encrypt.format is now optional Max Reitz 12 siblings, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth By applying a health mix of qdict_flatten(), qdict_crumple(), qdict_stringify_for_keyval(), the keyval input visitor, and the QObject output visitor (not necessarily in that order), we can at least try to bring bs->full_open_options into accordance with the QAPI schema. This may not always work (there are some options left that have not been QAPI-fied yet), but in practice it usually will. In any case, sometimes emitting wrongly typed json:{} filenames is better than doing it effectively half the time. This affects some iotests because json:{} filenames are now usually crumpled. Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1534396 Signed-off-by: Max Reitz <mreitz@redhat.com> --- block.c | 69 +++++++++++++++++++++++++++++++++++++++++++++- tests/qemu-iotests/059.out | 2 +- tests/qemu-iotests/099.out | 4 +-- tests/qemu-iotests/110.out | 2 +- tests/qemu-iotests/198.out | 4 +-- tests/qemu-iotests/207.out | 10 +++---- 6 files changed, 79 insertions(+), 12 deletions(-) diff --git a/block.c b/block.c index 3c3e8fd11d..9e4a6c0d30 100644 --- a/block.c +++ b/block.c @@ -36,6 +36,7 @@ #include "qapi/qmp/qjson.h" #include "qapi/qmp/qnull.h" #include "qapi/qmp/qstring.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/qobject-output-visitor.h" #include "qapi/qapi-visit-block-core.h" #include "sysemu/block-backend.h" @@ -5143,6 +5144,59 @@ static bool append_open_options(QDict *d, BlockDriverState *bs) return found_any; } +/** + * Take a blockdev @options QDict and convert its values to the + * correct type. + * + * Fail if @options does not match the QAPI schema of BlockdevOptions. + * + * In case of failure, return NULL and set @errp. + * + * In case of success, return a correctly typed new QDict. + */ +static QDict *bdrv_type_blockdev_opts(const QDict *options, Error **errp) +{ + Visitor *v; + BlockdevOptions *blockdev_options; + QObject *typed_opts, *crumpled_opts; + QDict *string_options; + Error *local_err = NULL; + + string_options = qdict_clone_shallow(options); + + qdict_flatten(string_options); + qdict_stringify_for_keyval(string_options); + crumpled_opts = qdict_crumple(string_options, errp); + qobject_unref(string_options); + if (!crumpled_opts) { + error_prepend(errp, "Failed to crumple options: "); + return NULL; + } + + v = qobject_input_visitor_new_keyval(crumpled_opts); + visit_type_BlockdevOptions(v, NULL, &blockdev_options, &local_err); + visit_free(v); + if (local_err) { + error_propagate(errp, local_err); + error_prepend(errp, "Not a valid BlockdevOptions object: "); + return NULL; + } + + v = qobject_output_visitor_new(&typed_opts); + visit_type_BlockdevOptions(v, NULL, &blockdev_options, &local_err); + if (!local_err) { + visit_complete(v, &typed_opts); + } + visit_free(v); + qapi_free_BlockdevOptions(blockdev_options); + if (local_err) { + error_propagate(errp, local_err); + return NULL; + } + + return qobject_to(QDict, typed_opts); +} + /* Updates the following BDS fields: * - exact_filename: A filename which may be used for opening a block device * which (mostly) equals the given BDS (even without any @@ -5244,10 +5298,23 @@ void bdrv_refresh_filename(BlockDriverState *bs) if (bs->exact_filename[0]) { pstrcpy(bs->filename, sizeof(bs->filename), bs->exact_filename); } else if (bs->full_open_options) { - QString *json = qobject_to_json(QOBJECT(bs->full_open_options)); + QString *json; + QDict *typed_opts, *json_opts; + + typed_opts = bdrv_type_blockdev_opts(bs->full_open_options, NULL); + + /* We cannot be certain that bs->full_open_options matches + * BlockdevOptions, so bdrv_type_blockdev_opts() may fail. + * That is not fatal, we can just emit bs->full_open_options + * directly -- qemu will accept that, even if it does not + * match the schema. */ + json_opts = typed_opts ?: bs->full_open_options; + + json = qobject_to_json(QOBJECT(json_opts)); snprintf(bs->filename, sizeof(bs->filename), "json:%s", qstring_get_str(json)); qobject_unref(json); + qobject_unref(typed_opts); } } diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out index f6dce7947c..0238b9e9a8 100644 --- a/tests/qemu-iotests/059.out +++ b/tests/qemu-iotests/059.out @@ -2050,7 +2050,7 @@ wrote 512/512 bytes at offset 10240 === Testing monolithicFlat with internally generated JSON file name === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat -can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}' +can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"inject-error": [{"event": "read_aio"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}' === Testing version 3 === image: TEST_DIR/iotest-version3.IMGFMT diff --git a/tests/qemu-iotests/099.out b/tests/qemu-iotests/099.out index 8cce627529..0a9c434148 100644 --- a/tests/qemu-iotests/099.out +++ b/tests/qemu-iotests/099.out @@ -12,11 +12,11 @@ blkverify:TEST_DIR/t.IMGFMT.compare:TEST_DIR/t.IMGFMT === Testing JSON filename for blkdebug === -json:{"driver": "IMGFMT", "file": {"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "l1_update"}} +json:{"driver": "IMGFMT", "file": {"inject-error": [{"event": "l1_update"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}} === Testing indirectly enforced JSON filename === -json:{"driver": "raw", "file": {"test": {"driver": "IMGFMT", "file": {"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "l1_update"}}, "driver": "blkverify", "raw": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.compare"}}} +json:{"driver": "raw", "file": {"test": {"driver": "IMGFMT", "file": {"inject-error": [{"event": "l1_update"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}, "driver": "blkverify", "raw": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.compare"}}} === Testing plain filename for blkdebug === diff --git a/tests/qemu-iotests/110.out b/tests/qemu-iotests/110.out index b3584ff87f..fa19315dfb 100644 --- a/tests/qemu-iotests/110.out +++ b/tests/qemu-iotests/110.out @@ -11,7 +11,7 @@ backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base) === Non-reconstructable filename === -image: json:{"driver": "IMGFMT", "file": {"set-state.0.event": "read_aio", "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "set-state.0.new_state": 42}} +image: json:{"driver": "IMGFMT", "file": {"set-state": [{"new_state": 42, "event": "read_aio"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}} file format: IMGFMT virtual size: 64M (67108864 bytes) backing file: t.IMGFMT.base (cannot determine actual path) diff --git a/tests/qemu-iotests/198.out b/tests/qemu-iotests/198.out index adb805cce9..2a02077d0b 100644 --- a/tests/qemu-iotests/198.out +++ b/tests/qemu-iotests/198.out @@ -32,7 +32,7 @@ read 16777216/16777216 bytes at offset 0 16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) == checking image base == -image: json:{"encrypt.key-secret": "sec0", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.base"}} +image: json:{"driver": "IMGFMT", "encrypt": {"key-secret": "sec0"}, "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.base"}} file format: IMGFMT virtual size: 16M (16777216 bytes) Format specific information: @@ -74,7 +74,7 @@ Format specific information: master key iters: 1024 == checking image layer == -image: json:{"encrypt.key-secret": "sec1", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}} +image: json:{"driver": "IMGFMT", "encrypt": {"key-secret": "sec1"}, "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}} file format: IMGFMT virtual size: 16M (16777216 bytes) backing file: TEST_DIR/t.IMGFMT.base diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out index 417deee970..effb74bf02 100644 --- a/tests/qemu-iotests/207.out +++ b/tests/qemu-iotests/207.out @@ -9,7 +9,7 @@ QMP_VERSION {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} -image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}} file format: IMGFMT virtual size: 4.0M (4194304 bytes) @@ -26,7 +26,7 @@ QMP_VERSION {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} -image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}} file format: IMGFMT virtual size: 8.0M (8388608 bytes) Testing: @@ -36,7 +36,7 @@ QMP_VERSION {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} -image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}} file format: IMGFMT virtual size: 4.0M (4194304 bytes) Testing: @@ -47,7 +47,7 @@ QMP_VERSION {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} -image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}} file format: IMGFMT virtual size: 8.0M (8388608 bytes) Testing: @@ -58,7 +58,7 @@ QMP_VERSION {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} -image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}} file format: IMGFMT virtual size: 4.0M (4194304 bytes) -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 12/13] iotests: Test internal option typing 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (10 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 11/13] block: Try to create well typed json:{} filenames Max Reitz @ 2018-05-09 16:55 ` Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 13/13] iotests: qcow2's encrypt.format is now optional Max Reitz 12 siblings, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth It would be nice if qemu used the correct types for blockdev options internally, even if the user specified string values (either through -drive or by being not so nice and using json:{} with string values). This patch adds a test verifying that fact. Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/089 | 25 +++++++++++++++++++++++++ tests/qemu-iotests/089.out | 9 +++++++++ 2 files changed, 34 insertions(+) diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089 index aa1ba4a98e..0682d08f39 100755 --- a/tests/qemu-iotests/089 +++ b/tests/qemu-iotests/089 @@ -36,6 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 # get standard environment, filters and checks . ./common.rc . ./common.filter +. ./common.qemu _supported_fmt qcow2 _supported_proto file @@ -145,6 +146,30 @@ $QEMU_IO -c "open -o driver=qcow2 json:{\"file.filename\":\"$TEST_IMG\"}" \ $QEMU_IO -c "open -o driver=qcow2 json:{\"driver\":\"raw\",\"file.filename\":\"$TEST_IMG\"}" \ -c "info" 2>&1 | _filter_img_info +echo +echo "=== Testing option typing ===" +echo + +# json:{} accepts both strings and correctly typed values (even mixed, +# although we probably do not want to support that...), but when +# creating a json:{} filename, it should be correctly typed. +# Therefore, both of these should make the "size" value an integer. + +TEST_IMG="json:{'driver': 'null-co', 'size': 42 }" _img_info | grep '^image' +TEST_IMG="json:{'driver': 'null-co', 'size': '42'}" _img_info | grep '^image' + +echo + +# This should even work when some driver abuses bs->options to store +# non-QAPI options (and the given -drive options are not complete) +# (Watch for whether file.align appears as an int or a string) +qemu_comm_method=monitor _launch_qemu \ + -drive if=none,id=drv0,node-name=node0,format=raw,file=blkdebug::null-co://,file.align=512 + +_send_qemu_cmd $QEMU_HANDLE 'info block' 'json:' + +_cleanup_qemu + # success, all done echo "*** done" diff --git a/tests/qemu-iotests/089.out b/tests/qemu-iotests/089.out index 89e3e4340a..35ffbabe77 100644 --- a/tests/qemu-iotests/089.out +++ b/tests/qemu-iotests/089.out @@ -49,4 +49,13 @@ vm state offset: 512 MiB format name: IMGFMT cluster size: 64 KiB vm state offset: 512 MiB + +=== Testing option typing === + +image: json:{"driver": "null-co", "size": 42} +image: json:{"driver": "null-co", "size": 42} + +QEMU X.Y.Z monitor - type 'help' for more information +(qemu) info block +drv0 (node0): json:{"driver": "raw", "file": {"image": {"driver": "null-co"}, "driver": "blkdebug", "align": 512}} (raw) *** done -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
* [Qemu-devel] [PATCH 13/13] iotests: qcow2's encrypt.format is now optional 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz ` (11 preceding siblings ...) 2018-05-09 16:55 ` [Qemu-devel] [PATCH 12/13] iotests: Test internal option typing Max Reitz @ 2018-05-09 16:55 ` Max Reitz 12 siblings, 0 replies; 42+ messages in thread From: Max Reitz @ 2018-05-09 16:55 UTC (permalink / raw) To: qemu-block Cc: qemu-devel, Max Reitz, Markus Armbruster, Kevin Wolf, Eric Blake, Michael Roth Remove the encrypt.format option from the two blockdev-add test cases for encrypted qcow2 images in 087 to explicitly test that this parameter is now optional. Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/087 | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087 index 2561a14456..1eba457985 100755 --- a/tests/qemu-iotests/087 +++ b/tests/qemu-iotests/087 @@ -150,7 +150,6 @@ run_qemu <<EOF "filename": "$TEST_IMG" }, "encrypt": { - "format": "aes", "key-secret": "sec0" } } @@ -183,7 +182,6 @@ run_qemu <<EOF "filename": "$TEST_IMG" }, "encrypt": { - "format": "luks", "key-secret": "sec0" } } -- 2.14.3 ^ permalink raw reply related [flat|nested] 42+ messages in thread
end of thread, other threads:[~2018-06-06 14:44 UTC | newest] Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-05-09 16:55 [Qemu-devel] [PATCH 00/13] block: Try to create well typed json:{} filenames Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions Max Reitz 2018-05-10 13:12 ` Eric Blake 2018-05-10 13:18 ` Eric Blake 2018-05-11 17:59 ` Max Reitz 2018-05-11 18:13 ` Eric Blake 2018-05-11 17:38 ` Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 02/13] docs/qapi: Document optional discriminators Max Reitz 2018-05-10 13:14 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests Max Reitz 2018-05-10 14:08 ` Eric Blake 2018-05-11 18:06 ` Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing Max Reitz 2018-05-10 7:58 ` Daniel P. Berrangé 2018-05-10 14:22 ` Eric Blake 2018-05-11 17:32 ` Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow " Max Reitz 2018-05-10 14:24 ` Eric Blake 2018-05-10 14:32 ` Daniel P. Berrangé 2018-05-11 18:07 ` Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header Max Reitz 2018-05-10 14:54 ` Eric Blake 2018-05-11 18:11 ` Max Reitz 2018-06-06 13:10 ` Markus Armbruster 2018-06-06 13:17 ` Markus Armbruster 2018-06-06 14:19 ` Markus Armbruster 2018-06-06 14:35 ` Markus Armbruster 2018-05-09 16:55 ` [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval() Max Reitz 2018-05-11 18:39 ` Eric Blake 2018-05-11 21:42 ` Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test Max Reitz 2018-05-10 16:02 ` Eric Blake 2018-05-11 18:13 ` Max Reitz 2018-05-11 18:33 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 09/13] qdict: Make qdict_flatten() shallow-clone-friendly Max Reitz 2018-05-11 18:44 ` Eric Blake 2018-05-09 16:55 ` [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test Max Reitz 2018-05-11 18:46 ` Eric Blake 2018-05-11 21:41 ` Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 11/13] block: Try to create well typed json:{} filenames Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 12/13] iotests: Test internal option typing Max Reitz 2018-05-09 16:55 ` [Qemu-devel] [PATCH 13/13] iotests: qcow2's encrypt.format is now optional Max Reitz
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.