* [PATCH v3 0/6] qapi: Add support for aliases
@ 2021-08-12 16:11 Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 1/6] qapi: Add interfaces for alias support to Visitor Kevin Wolf
` (7 more replies)
0 siblings, 8 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-08-12 16:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, armbru
This series introduces alias definitions for QAPI object types (structs
and unions).
This allows using the same QAPI type and visitor even when the syntax
has some variations between different external interfaces such as QMP
and the command line.
It also provides a new tool for evolving the schema while maintaining
backwards compatibility (possibly during a deprecation period).
The first user is intended to be a QAPIfied -chardev command line
option, for which I'll send a separate series. A git tag is available
that contains both this series and the chardev changes that make use of
it:
https://repo.or.cz/qemu/kevin.git qapi-alias-chardev-v3
v3:
- Mention the new functions in the big comment in visitor.h. However,
since the comment is about users of the visitor rather than the
generated code, it seems like to wrong place to go into details.
- Updated commit message for patch 3 ('Simplify full_name_nth() ...')
- Patch 4 ('qapi: Apply aliases in qobject-input-visitor'):
- Multiple matching wildcard aliases are considered conflicting now
- Improved comments for several functions
- Renamed bool *implicit_object into *is_alias_prefix, which
describes better what it is rather than what it is used for
- Simplified alias_present() into input_present()
- Fixed potential use of wrong StackObject in error message
- Patch 5 ('qapi: Add support for aliases'):
- Made QAPISchemaAlias a QAPISchemaMember
- Check validity of alias source paths (must exist in at least one
variant, no optional objects in the path of a wildcard alias, no
alias loops)
- Many new tests cases, both positive and negative, including unit tests
of the generated visit functions
- Coding style changes
- Rebased documentation (.txt -> .rst conversion in master)
v2:
- Renamed 'alias' to 'name' in all data structures describing aliases
- Tons of new or changed comments and other documentation
- Be more explicit that empty 'source' is invalid and assert it
- Fixed full_name_so() for lists (added a parameter to tell the function
whether the name of a list member or the list itself is meant)
- Changed some QAPI generator error messages
- Assert the type of parameters in QAPISchemaAlias.__init__()
Kevin Wolf (6):
qapi: Add interfaces for alias support to Visitor
qapi: Remember alias definitions in qobject-input-visitor
qapi: Simplify full_name_nth() in qobject-input-visitor
qapi: Apply aliases in qobject-input-visitor
qapi: Add support for aliases
tests/qapi-schema: Test cases for aliases
docs/devel/qapi-code-gen.rst | 104 ++++-
docs/sphinx/qapidoc.py | 2 +-
include/qapi/visitor-impl.h | 12 +
include/qapi/visitor.h | 59 ++-
qapi/qapi-visit-core.c | 22 +
qapi/qobject-input-visitor.c | 417 ++++++++++++++++--
tests/unit/test-qobject-input-visitor.c | 218 +++++++++
scripts/qapi/expr.py | 47 +-
scripts/qapi/schema.py | 116 ++++-
scripts/qapi/types.py | 4 +-
scripts/qapi/visit.py | 34 +-
tests/qapi-schema/test-qapi.py | 7 +-
tests/qapi-schema/alias-bad-type.err | 2 +
tests/qapi-schema/alias-bad-type.json | 3 +
tests/qapi-schema/alias-bad-type.out | 0
tests/qapi-schema/alias-missing-source.err | 2 +
tests/qapi-schema/alias-missing-source.json | 3 +
tests/qapi-schema/alias-missing-source.out | 0
tests/qapi-schema/alias-name-bad-type.err | 2 +
tests/qapi-schema/alias-name-bad-type.json | 3 +
tests/qapi-schema/alias-name-bad-type.out | 0
tests/qapi-schema/alias-name-conflict.err | 2 +
tests/qapi-schema/alias-name-conflict.json | 4 +
tests/qapi-schema/alias-name-conflict.out | 0
tests/qapi-schema/alias-recursive.err | 2 +
tests/qapi-schema/alias-recursive.json | 4 +
tests/qapi-schema/alias-recursive.out | 0
tests/qapi-schema/alias-source-bad-type.err | 2 +
tests/qapi-schema/alias-source-bad-type.json | 3 +
tests/qapi-schema/alias-source-bad-type.out | 0
.../alias-source-elem-bad-type.err | 2 +
.../alias-source-elem-bad-type.json | 3 +
.../alias-source-elem-bad-type.out | 0
tests/qapi-schema/alias-source-empty.err | 2 +
tests/qapi-schema/alias-source-empty.json | 3 +
tests/qapi-schema/alias-source-empty.out | 0
.../alias-source-inexistent-variants.err | 2 +
.../alias-source-inexistent-variants.json | 12 +
.../alias-source-inexistent-variants.out | 0
tests/qapi-schema/alias-source-inexistent.err | 2 +
.../qapi-schema/alias-source-inexistent.json | 3 +
tests/qapi-schema/alias-source-inexistent.out | 0
.../alias-source-non-object-path.err | 2 +
.../alias-source-non-object-path.json | 3 +
.../alias-source-non-object-path.out | 0
.../alias-source-non-object-wildcard.err | 2 +
.../alias-source-non-object-wildcard.json | 3 +
.../alias-source-non-object-wildcard.out | 0
...lias-source-optional-wildcard-indirect.err | 2 +
...ias-source-optional-wildcard-indirect.json | 6 +
...lias-source-optional-wildcard-indirect.out | 0
.../alias-source-optional-wildcard.err | 2 +
.../alias-source-optional-wildcard.json | 5 +
.../alias-source-optional-wildcard.out | 0
tests/qapi-schema/alias-unknown-key.err | 3 +
tests/qapi-schema/alias-unknown-key.json | 3 +
tests/qapi-schema/alias-unknown-key.out | 0
tests/qapi-schema/aliases-bad-type.err | 2 +
tests/qapi-schema/aliases-bad-type.json | 3 +
tests/qapi-schema/aliases-bad-type.out | 0
tests/qapi-schema/double-type.err | 2 +-
tests/qapi-schema/meson.build | 16 +
tests/qapi-schema/qapi-schema-test.json | 26 ++
tests/qapi-schema/qapi-schema-test.out | 31 ++
tests/qapi-schema/unknown-expr-key.err | 2 +-
65 files changed, 1163 insertions(+), 53 deletions(-)
create mode 100644 tests/qapi-schema/alias-bad-type.err
create mode 100644 tests/qapi-schema/alias-bad-type.json
create mode 100644 tests/qapi-schema/alias-bad-type.out
create mode 100644 tests/qapi-schema/alias-missing-source.err
create mode 100644 tests/qapi-schema/alias-missing-source.json
create mode 100644 tests/qapi-schema/alias-missing-source.out
create mode 100644 tests/qapi-schema/alias-name-bad-type.err
create mode 100644 tests/qapi-schema/alias-name-bad-type.json
create mode 100644 tests/qapi-schema/alias-name-bad-type.out
create mode 100644 tests/qapi-schema/alias-name-conflict.err
create mode 100644 tests/qapi-schema/alias-name-conflict.json
create mode 100644 tests/qapi-schema/alias-name-conflict.out
create mode 100644 tests/qapi-schema/alias-recursive.err
create mode 100644 tests/qapi-schema/alias-recursive.json
create mode 100644 tests/qapi-schema/alias-recursive.out
create mode 100644 tests/qapi-schema/alias-source-bad-type.err
create mode 100644 tests/qapi-schema/alias-source-bad-type.json
create mode 100644 tests/qapi-schema/alias-source-bad-type.out
create mode 100644 tests/qapi-schema/alias-source-elem-bad-type.err
create mode 100644 tests/qapi-schema/alias-source-elem-bad-type.json
create mode 100644 tests/qapi-schema/alias-source-elem-bad-type.out
create mode 100644 tests/qapi-schema/alias-source-empty.err
create mode 100644 tests/qapi-schema/alias-source-empty.json
create mode 100644 tests/qapi-schema/alias-source-empty.out
create mode 100644 tests/qapi-schema/alias-source-inexistent-variants.err
create mode 100644 tests/qapi-schema/alias-source-inexistent-variants.json
create mode 100644 tests/qapi-schema/alias-source-inexistent-variants.out
create mode 100644 tests/qapi-schema/alias-source-inexistent.err
create mode 100644 tests/qapi-schema/alias-source-inexistent.json
create mode 100644 tests/qapi-schema/alias-source-inexistent.out
create mode 100644 tests/qapi-schema/alias-source-non-object-path.err
create mode 100644 tests/qapi-schema/alias-source-non-object-path.json
create mode 100644 tests/qapi-schema/alias-source-non-object-path.out
create mode 100644 tests/qapi-schema/alias-source-non-object-wildcard.err
create mode 100644 tests/qapi-schema/alias-source-non-object-wildcard.json
create mode 100644 tests/qapi-schema/alias-source-non-object-wildcard.out
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard-indirect.err
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard-indirect.json
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard-indirect.out
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard.err
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard.json
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard.out
create mode 100644 tests/qapi-schema/alias-unknown-key.err
create mode 100644 tests/qapi-schema/alias-unknown-key.json
create mode 100644 tests/qapi-schema/alias-unknown-key.out
create mode 100644 tests/qapi-schema/aliases-bad-type.err
create mode 100644 tests/qapi-schema/aliases-bad-type.json
create mode 100644 tests/qapi-schema/aliases-bad-type.out
--
2.31.1
^ permalink raw reply [flat|nested] 43+ messages in thread
* [PATCH v3 1/6] qapi: Add interfaces for alias support to Visitor
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
@ 2021-08-12 16:11 ` Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 2/6] qapi: Remember alias definitions in qobject-input-visitor Kevin Wolf
` (6 subsequent siblings)
7 siblings, 0 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-08-12 16:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, armbru
This adds functions to the Visitor interface that can be used to define
aliases and alias scopes.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
include/qapi/visitor-impl.h | 12 ++++++++
include/qapi/visitor.h | 59 ++++++++++++++++++++++++++++++++++---
qapi/qapi-visit-core.c | 22 ++++++++++++++
3 files changed, 89 insertions(+), 4 deletions(-)
diff --git a/include/qapi/visitor-impl.h b/include/qapi/visitor-impl.h
index 3b950f6e3d..704c5ad2d9 100644
--- a/include/qapi/visitor-impl.h
+++ b/include/qapi/visitor-impl.h
@@ -119,6 +119,18 @@ struct Visitor
/* Optional */
bool (*deprecated)(Visitor *v, const char *name);
+ /*
+ * Optional; intended for input visitors. If not given, aliases are
+ * ignored.
+ */
+ void (*define_alias)(Visitor *v, const char *name, const char **source);
+
+ /* Must be set if define_alias is set */
+ void (*start_alias_scope)(Visitor *v);
+
+ /* Must be set if define_alias is set */
+ void (*end_alias_scope)(Visitor *v);
+
/* Must be set */
VisitorType type;
diff --git a/include/qapi/visitor.h b/include/qapi/visitor.h
index b3c9ef7a81..3bf0f4dad2 100644
--- a/include/qapi/visitor.h
+++ b/include/qapi/visitor.h
@@ -220,10 +220,17 @@
* </example>
*
* This file provides helpers for use by the generated
- * visit_type_FOO(): visit_optional() for the 'has_member' field
- * associated with optional 'member' in the C struct,
- * visit_next_list() for advancing through a FooList linked list, and
- * visit_is_input() for cleaning up on failure.
+ * visit_type_FOO():
+ *
+ * - visit_optional() for the 'has_member' field associated with
+ * optional 'member' in the C struct,
+ * - visit_next_list() for advancing through a FooList linked list
+ * - visit_is_input() for cleaning up on failure
+ * - visit_define_alias() for defining alternative names for object
+ * members in input visitors
+ * - visit_start/end_alias_scope() to limit the scope of aliases
+ * within a single input object (e.g. aliases defined in the base
+ * struct should not provide values for the parent struct)
*/
/*** Useful types ***/
@@ -477,6 +484,50 @@ bool visit_deprecated_accept(Visitor *v, const char *name, Error **errp);
*/
bool visit_deprecated(Visitor *v, const char *name);
+/*
+ * Defines a new alias rule.
+ *
+ * If @name is non-NULL, the member called @name in the external
+ * representation of the currently visited object is defined as an
+ * alias for the member described by @source. It is not allowed to
+ * call this function when the currently visited type is not an
+ * object.
+ *
+ * If @name is NULL, all members of the object described by @source
+ * are considered to have alias members with the same key in the
+ * currently visited object.
+ *
+ * @source is a NULL-terminated non-empty array of names that describe
+ * the path to a member, starting from the currently visited object.
+ * All elements in @source except the last one should describe
+ * objects. If an intermediate element refers to a member with a
+ * non-object type, the alias won't work (this case can legitimately
+ * happen in unions where an alias only makes sense for one branch,
+ * but not for another).
+ *
+ * The alias stays valid until the current alias scope ends.
+ * visit_start/end_struct() implicitly start/end an alias scope.
+ * Additionally, visit_start/end_alias_scope() can be used to explicitly
+ * create a nested alias scope.
+ */
+void visit_define_alias(Visitor *v, const char *name, const char **source);
+
+/*
+ * Begins an explicit alias scope.
+ *
+ * Alias definitions after here will only stay valid until the
+ * corresponding visit_end_alias_scope() is called.
+ */
+void visit_start_alias_scope(Visitor *v);
+
+/*
+ * Ends an explicit alias scope.
+ *
+ * Alias definitions between the correspoding visit_start_alias_scope()
+ * call and here go out of scope and won't apply in later code any more.
+ */
+void visit_end_alias_scope(Visitor *v);
+
/*
* Visit an enum value.
*
diff --git a/qapi/qapi-visit-core.c b/qapi/qapi-visit-core.c
index a641adec51..79df6901ae 100644
--- a/qapi/qapi-visit-core.c
+++ b/qapi/qapi-visit-core.c
@@ -153,6 +153,28 @@ bool visit_deprecated(Visitor *v, const char *name)
return true;
}
+void visit_define_alias(Visitor *v, const char *name, const char **source)
+{
+ assert(source[0] != NULL);
+ if (v->define_alias) {
+ v->define_alias(v, name, source);
+ }
+}
+
+void visit_start_alias_scope(Visitor *v)
+{
+ if (v->start_alias_scope) {
+ v->start_alias_scope(v);
+ }
+}
+
+void visit_end_alias_scope(Visitor *v)
+{
+ if (v->end_alias_scope) {
+ v->end_alias_scope(v);
+ }
+}
+
bool visit_is_input(Visitor *v)
{
return v->type == VISITOR_INPUT;
--
2.31.1
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 2/6] qapi: Remember alias definitions in qobject-input-visitor
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 1/6] qapi: Add interfaces for alias support to Visitor Kevin Wolf
@ 2021-08-12 16:11 ` Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 3/6] qapi: Simplify full_name_nth() " Kevin Wolf
` (5 subsequent siblings)
7 siblings, 0 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-08-12 16:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, armbru
This makes qobject-input-visitor remember the currently valid aliases in
each StackObject. It doesn't actually allow using the aliases yet.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
qapi/qobject-input-visitor.c | 147 +++++++++++++++++++++++++++++++++++
1 file changed, 147 insertions(+)
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index 04b790412e..d0061d33d6 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -30,6 +30,50 @@
#include "qemu/cutils.h"
#include "qemu/option.h"
+/*
+ * Describes an alias that is relevant for the current StackObject,
+ * either because it aliases a member of the currently visited object
+ * or because it aliases a member of a nested object.
+ *
+ * When processing a nested object, all InputVisitorAlias objects that
+ * are relevant for the nested object are propagated, i.e. copied with
+ * the name of the nested object removed from @source.
+ */
+typedef struct InputVisitorAlias {
+ /* StackObject in which the alias was defined */
+ struct StackObject *alias_so;
+
+ /*
+ * Alias name as defined for @alias_so.
+ * NULL means that this is a wildcard alias, i.e. all members of
+ * @src get an alias in @alias_so with the same name.
+ */
+ const char *name;
+
+ /*
+ * NULL-terminated array representing a path to the source member
+ * that the alias refers to.
+ *
+ * Must contain at least one non-NULL element if @alias is not NULL.
+ *
+ * If it contains no non-NULL element, @alias_so must be different
+ * from the StackObject which contains this InputVisitorAlias in
+ * its aliases list. In this case, all elements in the currently
+ * visited object have an alias with the same name in @alias_so.
+ */
+ const char **src;
+
+ /*
+ * The alias remains valid as long as the StackObject which
+ * contains this InputVisitorAlias in its aliases list has
+ * StackObject.alias_scope_nesting >= InputVisitorAlias.scope_nesting
+ * or until the whole StackObject is removed.
+ */
+ int scope_nesting;
+
+ QSLIST_ENTRY(InputVisitorAlias) next;
+} InputVisitorAlias;
+
typedef struct StackObject {
const char *name; /* Name of @obj in its parent, if any */
QObject *obj; /* QDict or QList being visited */
@@ -39,6 +83,9 @@ typedef struct StackObject {
const QListEntry *entry; /* If @obj is QList: unvisited tail */
unsigned index; /* If @obj is QList: list index of @entry */
+ QSLIST_HEAD(, InputVisitorAlias) aliases;
+ int alias_scope_nesting; /* Number of open alias scopes */
+
QSLIST_ENTRY(StackObject) node; /* parent */
} StackObject;
@@ -205,6 +252,45 @@ static const char *qobject_input_get_keyval(QObjectInputVisitor *qiv,
return qstring_get_str(qstr);
}
+/*
+ * Propagate aliases from the parent StackObject @src to its direct
+ * child StackObject @dst, which is representing the child struct @name.
+ *
+ * Every alias whose source path begins with @dst->name and which still
+ * applies in @dst (i.e. it is either a wildcard alias or has at least
+ * one more source path element) is propagated to @dst with the first
+ * element (i.e. @dst->name) removed from the source path.
+ */
+static void propagate_aliases(StackObject *dst, StackObject *src)
+{
+ InputVisitorAlias *a;
+ InputVisitorAlias *propagated_alias;
+
+ QSLIST_FOREACH(a, &src->aliases, next) {
+ if (!a->src[0] || strcmp(a->src[0], dst->name)) {
+ continue;
+ }
+
+ /*
+ * If this is not a wildcard alias, but a->src[1] is NULL,
+ * then it referred to a->name in src and doesn't apply inside
+ * dst any more.
+ */
+ if (a->name && !a->src[1]) {
+ continue;
+ }
+
+ propagated_alias = g_new(InputVisitorAlias, 1);
+ *propagated_alias = (InputVisitorAlias) {
+ .name = a->name,
+ .alias_so = a->alias_so,
+ .src = &a->src[1],
+ };
+
+ QSLIST_INSERT_HEAD(&dst->aliases, propagated_alias, next);
+ }
+}
+
static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,
const char *name,
QObject *obj, void *qapi)
@@ -228,6 +314,9 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,
g_hash_table_insert(h, (void *)qdict_entry_key(entry), NULL);
}
tos->h = h;
+ if (!QSLIST_EMPTY(&qiv->stack)) {
+ propagate_aliases(tos, QSLIST_FIRST(&qiv->stack));
+ }
} else {
assert(qlist);
tos->entry = qlist_first(qlist);
@@ -259,10 +348,17 @@ static bool qobject_input_check_struct(Visitor *v, Error **errp)
static void qobject_input_stack_object_free(StackObject *tos)
{
+ InputVisitorAlias *a;
+
if (tos->h) {
g_hash_table_unref(tos->h);
}
+ while ((a = QSLIST_FIRST(&tos->aliases))) {
+ QSLIST_REMOVE_HEAD(&tos->aliases, next);
+ g_free(a);
+ }
+
g_free(tos);
}
@@ -276,6 +372,54 @@ static void qobject_input_pop(Visitor *v, void **obj)
qobject_input_stack_object_free(tos);
}
+static void qobject_input_start_alias_scope(Visitor *v)
+{
+ QObjectInputVisitor *qiv = to_qiv(v);
+ StackObject *tos = QSLIST_FIRST(&qiv->stack);
+
+ tos->alias_scope_nesting++;
+}
+
+static void qobject_input_end_alias_scope(Visitor *v)
+{
+ QObjectInputVisitor *qiv = to_qiv(v);
+ StackObject *tos = QSLIST_FIRST(&qiv->stack);
+ InputVisitorAlias *a, *next;
+
+ assert(tos->alias_scope_nesting > 0);
+ tos->alias_scope_nesting--;
+
+ QSLIST_FOREACH_SAFE(a, &tos->aliases, next, next) {
+ if (a->scope_nesting > tos->alias_scope_nesting) {
+ QSLIST_REMOVE(&tos->aliases, a, InputVisitorAlias, next);
+ g_free(a);
+ }
+ }
+}
+
+static void qobject_input_define_alias(Visitor *v, const char *name,
+ const char **source)
+{
+ QObjectInputVisitor *qiv = to_qiv(v);
+ StackObject *tos = QSLIST_FIRST(&qiv->stack);
+ InputVisitorAlias *alias = g_new(InputVisitorAlias, 1);
+
+ /*
+ * The source path can become empty during alias propagation for
+ * wildcard aliases, but not when defining an alias (it would map
+ * all names onto themselves, which doesn't make sense).
+ */
+ assert(source[0]);
+
+ *alias = (InputVisitorAlias) {
+ .name = name,
+ .alias_so = tos,
+ .src = source,
+ };
+
+ QSLIST_INSERT_HEAD(&tos->aliases, alias, next);
+}
+
static bool qobject_input_start_struct(Visitor *v, const char *name, void **obj,
size_t size, Error **errp)
{
@@ -717,6 +861,9 @@ static QObjectInputVisitor *qobject_input_visitor_base_new(QObject *obj)
v->visitor.start_alternate = qobject_input_start_alternate;
v->visitor.optional = qobject_input_optional;
v->visitor.deprecated_accept = qobject_input_deprecated_accept;
+ v->visitor.define_alias = qobject_input_define_alias;
+ v->visitor.start_alias_scope = qobject_input_start_alias_scope;
+ v->visitor.end_alias_scope = qobject_input_end_alias_scope;
v->visitor.free = qobject_input_free;
v->root = qobject_ref(obj);
--
2.31.1
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 3/6] qapi: Simplify full_name_nth() in qobject-input-visitor
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 1/6] qapi: Add interfaces for alias support to Visitor Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 2/6] qapi: Remember alias definitions in qobject-input-visitor Kevin Wolf
@ 2021-08-12 16:11 ` Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 4/6] qapi: Apply aliases " Kevin Wolf
` (4 subsequent siblings)
7 siblings, 0 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-08-12 16:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, armbru
Instead of counting how many elements from the top of the stack we need
to ignore until we find the thing we're interested in, we can just
directly pass the StackObject pointer because all callers already know
it.
We only need a different way now to tell if we want to know the name of
something contained in the given StackObject or of the StackObject
itself. Make this explicit with a new boolean parameter.
This makes the function easier to use in cases where we have the
StackObject, but don't know how many steps down the stack it is. The
following patches will introduce such a caller.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
qapi/qobject-input-visitor.c | 43 ++++++++++++++++++++----------------
1 file changed, 24 insertions(+), 19 deletions(-)
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index d0061d33d6..16a75442ff 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -110,20 +110,20 @@ static QObjectInputVisitor *to_qiv(Visitor *v)
}
/*
- * Find the full name of something @qiv is currently visiting.
- * @qiv is visiting something named @name in the stack of containers
- * @qiv->stack.
- * If @n is zero, return its full name.
- * If @n is positive, return the full name of the @n-th container
- * counting from the top. The stack of containers must have at least
- * @n elements.
- * The returned string is valid until the next full_name_nth(@v) or
- * destruction of @v.
+ * Find the full name of a member in @so which @qiv is currently
+ * visiting. If the currently visited thing is an object, @name is
+ * the (local) name of the member to describe. If it is a list, @name
+ * is ignored and the current index (so->index) is included.
+ *
+ * If @skip_member is true, find the full name of @so itself instead.
+ * @name must be NULL then.
+ *
+ * The returned string is valid until the next full_name_so(@qiv) or
+ * destruction of @qiv.
*/
-static const char *full_name_nth(QObjectInputVisitor *qiv, const char *name,
- int n)
+static const char *full_name_so(QObjectInputVisitor *qiv, const char *name,
+ bool skip_member, StackObject *so)
{
- StackObject *so;
char buf[32];
if (qiv->errname) {
@@ -132,10 +132,14 @@ static const char *full_name_nth(QObjectInputVisitor *qiv, const char *name,
qiv->errname = g_string_new("");
}
- QSLIST_FOREACH(so , &qiv->stack, node) {
- if (n) {
- n--;
- } else if (qobject_type(so->obj) == QTYPE_QDICT) {
+ if (skip_member && so) {
+ assert(name == NULL);
+ name = so->name;
+ so = QSLIST_NEXT(so, node);
+ }
+
+ for (; so; so = QSLIST_NEXT(so, node)) {
+ if (qobject_type(so->obj) == QTYPE_QDICT) {
g_string_prepend(qiv->errname, name ?: "<anonymous>");
g_string_prepend_c(qiv->errname, '.');
} else {
@@ -146,7 +150,6 @@ static const char *full_name_nth(QObjectInputVisitor *qiv, const char *name,
}
name = so->name;
}
- assert(!n);
if (name) {
g_string_prepend(qiv->errname, name);
@@ -161,7 +164,9 @@ static const char *full_name_nth(QObjectInputVisitor *qiv, const char *name,
static const char *full_name(QObjectInputVisitor *qiv, const char *name)
{
- return full_name_nth(qiv, name, 0);
+ StackObject *tos = QSLIST_FIRST(&qiv->stack);
+
+ return full_name_so(qiv, name, false, tos);
}
static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
@@ -507,7 +512,7 @@ static bool qobject_input_check_list(Visitor *v, Error **errp)
if (tos->entry) {
error_setg(errp, "Only %u list elements expected in %s",
- tos->index + 1, full_name_nth(qiv, NULL, 1));
+ tos->index + 1, full_name_so(qiv, NULL, true, tos));
return false;
}
return true;
--
2.31.1
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 4/6] qapi: Apply aliases in qobject-input-visitor
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
` (2 preceding siblings ...)
2021-08-12 16:11 ` [PATCH v3 3/6] qapi: Simplify full_name_nth() " Kevin Wolf
@ 2021-08-12 16:11 ` Kevin Wolf
2021-09-06 15:16 ` Markus Armbruster
2021-08-12 16:11 ` [PATCH v3 5/6] qapi: Add support for aliases Kevin Wolf
` (3 subsequent siblings)
7 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-08-12 16:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, armbru
When looking for an object in a struct in the external representation,
check not only the currently visited struct, but also whether an alias
in the current StackObject matches and try to fetch the value from the
alias then. Providing two values for the same object through different
aliases is an error.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
qapi/qobject-input-visitor.c | 227 +++++++++++++++++++++++++++++++++--
1 file changed, 218 insertions(+), 9 deletions(-)
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index 16a75442ff..6193df28a5 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -97,6 +97,8 @@ struct QObjectInputVisitor {
QObject *root;
bool keyval; /* Assume @root made with keyval_parse() */
+ QDict *empty_qdict; /* Used for implicit objects */
+
/* Stack of objects being visited (all entries will be either
* QDict or QList). */
QSLIST_HEAD(, StackObject) stack;
@@ -169,9 +171,190 @@ static const char *full_name(QObjectInputVisitor *qiv, const char *name)
return full_name_so(qiv, name, false, tos);
}
+static bool find_object_member(QObjectInputVisitor *qiv,
+ StackObject **so, const char **name,
+ bool *is_alias_prefix, Error **errp);
+
+/*
+ * Check whether the member @name in @so, or an alias for it, is
+ * present in the input and can be used to obtain the value.
+ */
+static bool input_present(QObjectInputVisitor *qiv, StackObject *so,
+ const char *name)
+{
+ /*
+ * Check whether the alias member is present in the input
+ * (possibly recursively because aliases are transitive).
+ * The QAPI generator makes sure that alises cannot form loops, so
+ * the recursion guaranteed to terminate.
+ */
+ if (!find_object_member(qiv, &so, &name, NULL, NULL)) {
+ return false;
+ }
+
+ /*
+ * Every source can be used only once. If a value in the input
+ * would end up being used twice through aliases, we'll fail the
+ * second access.
+ */
+ if (!g_hash_table_contains(so->h, name)) {
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Check whether the member @name in the object visited by @so can be
+ * specified in the input by using the alias described by @a (which
+ * must be an alias contained in so->aliases).
+ *
+ * If @name is only a prefix of the alias source, but doesn't match
+ * immediately, false is returned and *is_alias_prefix is set to true
+ * if it is non-NULL. In all other cases, *is_alias_prefix is left
+ * unchanged.
+ */
+static bool alias_source_matches(QObjectInputVisitor *qiv,
+ StackObject *so, InputVisitorAlias *a,
+ const char *name, bool *is_alias_prefix)
+{
+ if (a->src[0] == NULL) {
+ assert(a->name == NULL);
+ return true;
+ }
+
+ if (!strcmp(a->src[0], name)) {
+ if (a->name && a->src[1] == NULL) {
+ /*
+ * We're matching an exact member, the source for this alias is
+ * immediately in @so.
+ */
+ return true;
+ } else if (is_alias_prefix) {
+ /*
+ * We're only looking at a prefix of the source path for the alias.
+ * If the input contains no object of the requested name, we will
+ * implicitly create an empty one so that the alias can still be
+ * used.
+ *
+ * We want to create the implicit object only if the alias is
+ * actually used, but we can't tell here for wildcard aliases (only
+ * a later visitor call will determine this). This means that
+ * wildcard aliases must never have optional keys in their source
+ * path. The QAPI generator checks this condition.
+ */
+ if (!a->name || input_present(qiv, a->alias_so, a->name)) {
+ *is_alias_prefix = true;
+ }
+ }
+ }
+
+ return false;
+}
+
+/*
+ * Find the place in the input where the value for the object member
+ * @name in @so is specified, considering applicable aliases.
+ *
+ * If a value could be found, true is returned and @so and @name are
+ * updated to identify the key name and StackObject where the value
+ * can be found in the input. (This is either unchanged or the
+ * alias_so/name of an alias.) The value of @is_alias_prefix on
+ * return is undefined in this case.
+ *
+ * If no value could be found in the input, false is returned and @so
+ * and @name are set to NULL. This is not an error and @errp remains
+ * unchanged. If @is_alias_prefix is non-NULL, it is set to true if
+ * the given name is a prefix of the source path of an alias for which
+ * a value may be present in the input. It is set to false otherwise.
+ *
+ * If an error occurs (e.g. two values are specified for the member
+ * through different names), false is returned and @errp is set. The
+ * value of @is_alias_prefix on return is undefined in this case.
+ */
+static bool find_object_member(QObjectInputVisitor *qiv,
+ StackObject **so, const char **name,
+ bool *is_alias_prefix, Error **errp)
+{
+ QDict *qdict = qobject_to(QDict, (*so)->obj);
+ const char *found_name = NULL;
+ StackObject *found_so = NULL;
+ bool found_is_wildcard = false;
+ InputVisitorAlias *a;
+
+ if (is_alias_prefix) {
+ *is_alias_prefix = false;
+ }
+
+ /* Directly present in the container */
+ if (qdict_haskey(qdict, *name)) {
+ found_name = *name;
+ found_so = *so;
+ }
+
+ /*
+ * Find aliases whose source path matches @name in this StackObject. We can
+ * then get the value with the key a->name from a->alias_so.
+ */
+ QSLIST_FOREACH(a, &(*so)->aliases, next) {
+ if (a->name == NULL && found_name && !found_is_wildcard) {
+ /*
+ * Skip wildcard aliases if we already have a match. This is
+ * not a conflict that should result in an error.
+ *
+ * However, multiple wildcard aliases matching is an error
+ * and will be caught below.
+ */
+ continue;
+ }
+
+ if (!alias_source_matches(qiv, *so, a, *name, is_alias_prefix)) {
+ continue;
+ }
+
+ /*
+ * For single-member aliases, an alias name is specified in the
+ * alias definition. For wildcard aliases, the alias has the same
+ * name as the member in the source object, i.e. *name.
+ */
+ if (!input_present(qiv, a->alias_so, a->name ?: *name)) {
+ continue;
+ }
+
+ /*
+ * A non-wildcard alias simply overrides a wildcard alias, but
+ * two matching non-wildcard aliases or two matching wildcard
+ * aliases conflict with each other.
+ */
+ if (found_name && (!found_is_wildcard || a->name == NULL)) {
+ error_setg(errp, "Value for parameter %s was already given "
+ "through an alias",
+ full_name_so(qiv, *name, false, *so));
+ return false;
+ } else {
+ found_name = a->name ?: *name;
+ found_so = a->alias_so;
+ found_is_wildcard = !a->name;
+ }
+ }
+
+ /*
+ * Chained aliases: *found_so/found_name might be the source of
+ * another alias.
+ */
+ if (found_name && (found_so != *so || found_name != *name)) {
+ find_object_member(qiv, &found_so, &found_name, NULL, errp);
+ }
+
+ *so = found_so;
+ *name = found_name;
+
+ return found_name != NULL;
+}
+
static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
const char *name,
- bool consume)
+ bool consume, Error **errp)
{
StackObject *tos;
QObject *qobj;
@@ -189,10 +372,31 @@ static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
assert(qobj);
if (qobject_type(qobj) == QTYPE_QDICT) {
- assert(name);
- ret = qdict_get(qobject_to(QDict, qobj), name);
- if (tos->h && consume && ret) {
- bool removed = g_hash_table_remove(tos->h, name);
+ StackObject *so = tos;
+ const char *key = name;
+ bool is_alias_prefix;
+
+ assert(key);
+ if (!find_object_member(qiv, &so, &key, &is_alias_prefix, errp)) {
+ if (is_alias_prefix) {
+ /*
+ * The member is not present in the input, but
+ * something inside of it might still be given through
+ * an alias. Pretend there was an empty object in the
+ * input.
+ */
+ if (!qiv->empty_qdict) {
+ qiv->empty_qdict = qdict_new();
+ }
+ return QOBJECT(qiv->empty_qdict);
+ } else {
+ return NULL;
+ }
+ }
+ ret = qdict_get(qobject_to(QDict, so->obj), key);
+ assert(ret != NULL);
+ if (so->h && consume) {
+ bool removed = g_hash_table_remove(so->h, key);
assert(removed);
}
} else {
@@ -218,9 +422,10 @@ static QObject *qobject_input_get_object(QObjectInputVisitor *qiv,
const char *name,
bool consume, Error **errp)
{
- QObject *obj = qobject_input_try_get_object(qiv, name, consume);
+ ERRP_GUARD();
+ QObject *obj = qobject_input_try_get_object(qiv, name, consume, errp);
- if (!obj) {
+ if (!obj && !*errp) {
error_setg(errp, QERR_MISSING_PARAMETER, full_name(qiv, name));
}
return obj;
@@ -803,13 +1008,16 @@ static bool qobject_input_type_size_keyval(Visitor *v, const char *name,
static void qobject_input_optional(Visitor *v, const char *name, bool *present)
{
QObjectInputVisitor *qiv = to_qiv(v);
- QObject *qobj = qobject_input_try_get_object(qiv, name, false);
+ Error *local_err = NULL;
+ QObject *qobj = qobject_input_try_get_object(qiv, name, false, &local_err);
- if (!qobj) {
+ /* If there was an error, let the caller try and run into the error */
+ if (!qobj && !local_err) {
*present = false;
return;
}
+ error_free(local_err);
*present = true;
}
@@ -842,6 +1050,7 @@ static void qobject_input_free(Visitor *v)
qobject_input_stack_object_free(tos);
}
+ qobject_unref(qiv->empty_qdict);
qobject_unref(qiv->root);
if (qiv->errname) {
g_string_free(qiv->errname, TRUE);
--
2.31.1
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 5/6] qapi: Add support for aliases
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
` (3 preceding siblings ...)
2021-08-12 16:11 ` [PATCH v3 4/6] qapi: Apply aliases " Kevin Wolf
@ 2021-08-12 16:11 ` Kevin Wolf
2021-09-06 15:24 ` Markus Armbruster
2021-09-16 7:49 ` Markus Armbruster
2021-08-12 16:11 ` [PATCH v3 6/6] tests/qapi-schema: Test cases " Kevin Wolf
` (2 subsequent siblings)
7 siblings, 2 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-08-12 16:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, armbru
Introduce alias definitions for object types (structs and unions). This
allows using the same QAPI type and visitor for many syntax variations
that exist in the external representation, like between QMP and the
command line. It also provides a new tool for evolving the schema while
maintaining backwards compatibility during a deprecation period.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
docs/devel/qapi-code-gen.rst | 104 +++++++++++++++++++++-
docs/sphinx/qapidoc.py | 2 +-
scripts/qapi/expr.py | 47 +++++++++-
scripts/qapi/schema.py | 116 +++++++++++++++++++++++--
scripts/qapi/types.py | 4 +-
scripts/qapi/visit.py | 34 +++++++-
tests/qapi-schema/test-qapi.py | 7 +-
tests/qapi-schema/double-type.err | 2 +-
tests/qapi-schema/unknown-expr-key.err | 2 +-
9 files changed, 297 insertions(+), 21 deletions(-)
diff --git a/docs/devel/qapi-code-gen.rst b/docs/devel/qapi-code-gen.rst
index 26c62b0e7b..c0883507a8 100644
--- a/docs/devel/qapi-code-gen.rst
+++ b/docs/devel/qapi-code-gen.rst
@@ -262,7 +262,8 @@ Syntax::
'data': MEMBERS,
'*base': STRING,
'*if': COND,
- '*features': FEATURES }
+ '*features': FEATURES,
+ '*aliases': ALIASES }
MEMBERS = { MEMBER, ... }
MEMBER = STRING : TYPE-REF
| STRING : { 'type': TYPE-REF,
@@ -312,6 +313,9 @@ the schema`_ below for more on this.
The optional 'features' member specifies features. See Features_
below for more on this.
+The optional 'aliases' member specifies aliases. See Aliases_ below
+for more on this.
+
Union types
-----------
@@ -321,13 +325,15 @@ Syntax::
UNION = { 'union': STRING,
'data': BRANCHES,
'*if': COND,
- '*features': FEATURES }
+ '*features': FEATURES,
+ '*aliases': ALIASES }
| { 'union': STRING,
'data': BRANCHES,
'base': ( MEMBERS | STRING ),
'discriminator': STRING,
'*if': COND,
- '*features': FEATURES }
+ '*features': FEATURES,
+ '*aliases': ALIASES }
BRANCHES = { BRANCH, ... }
BRANCH = STRING : TYPE-REF
| STRING : { 'type': TYPE-REF, '*if': COND }
@@ -437,6 +443,9 @@ the schema`_ below for more on this.
The optional 'features' member specifies features. See Features_
below for more on this.
+The optional 'aliases' member specifies aliases. See Aliases_ below
+for more on this.
+
Alternate types
---------------
@@ -888,6 +897,95 @@ shows a conditional entity only when the condition is satisfied in
this particular build.
+Aliases
+-------
+
+Object types, including structs and unions, can contain alias
+definitions.
+
+Aliases define alternative member names that may be used in wire input
+to provide a value for a member in the same object or in a nested
+object.
+
+Syntax::
+
+ ALIASES = [ ALIAS, ... ]
+ ALIAS = { '*name': STRING,
+ 'source': [ STRING, ... ] }
+
+If ``name`` is present, then the single member referred to by ``source``
+is made accessible with the name given by ``name`` in the type where the
+alias definition is specified.
+
+If ``name`` is not present, then this is a wildcard alias and all
+members in the object referred to by ``source`` are made accessible in
+the type where the alias definition is specified with the same name as
+they have in ``source``.
+
+``source`` is a non-empty list of member names representing the path to
+an object member. The first name is resolved in the same object. Each
+subsequent member is resolved in the object named by the preceding
+member.
+
+Do not use optional objects in the path of a wildcard alias unless there
+is no semantic difference between an empty object and an absent object.
+Absent objects are implicitly turned into empty ones if an alias could
+apply and provide a value in the nested object, which is always the case
+for wildcard aliases.
+
+Example: Alternative name for a member in the same object ::
+
+ { 'struct': 'File',
+ 'data': { 'path': 'str' },
+ 'aliases': [ { 'name': 'filename', 'source': ['path'] } ] }
+
+The member ``path`` may instead be given through its alias ``filename``
+in input.
+
+Example: Alias for a member in a nested object ::
+
+ { 'struct': 'A',
+ 'data': { 'zahl': 'int' } }
+ { 'struct': 'B',
+ 'data': { 'drei': 'A' } }
+ { 'struct': 'C',
+ 'data': { 'zwei': 'B' } }
+ { 'struct': 'D',
+ 'data': { 'eins': 'C' },
+ 'aliases': [ { 'name': 'number',
+ 'source': ['eins', 'zwei', 'drei', 'zahl' ] },
+ { 'name': 'the_B',
+ 'source': ['eins','zwei'] } ] }
+
+With this definition, each of the following inputs for ``D`` mean the
+same::
+
+ { 'eins': { 'zwei': { 'drei': { 'zahl': 42 } } } }
+
+ { 'the_B': { 'drei': { 'zahl': 42 } } }
+
+ { 'number': 42 }
+
+Example: Flattening a simple union with a wildcard alias that maps all
+members of ``data`` to the top level ::
+
+ { 'union': 'SocketAddress',
+ 'data': {
+ 'inet': 'InetSocketAddress',
+ 'unix': 'UnixSocketAddress' },
+ 'aliases': [ { 'source': ['data'] } ] }
+
+Aliases are transitive: ``source`` may refer to another alias name. In
+this case, the alias is effectively an alternative name for the source
+of the other alias.
+
+In order to accommodate unions where variants differ in structure, it
+is allowed to use a path that doesn't necessarily match an existing
+member in every variant; in this case, the alias remains unused. The
+QAPI generator checks that there is at least one branch for which an
+alias could match.
+
+
Documentation comments
----------------------
diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 87c67ab23f..68340b8529 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -313,7 +313,7 @@ def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+ self._nodes_for_if_section(ifcond))
def visit_object_type(self, name, info, ifcond, features,
- base, members, variants):
+ base, members, variants, aliases):
doc = self._cur_doc
if base and base.is_implicit():
base = None
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index cf98923fa6..054fef8d8e 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -430,6 +430,45 @@ def check_features(features: Optional[object],
check_if(feat, info, source)
+def check_aliases(aliases: Optional[object],
+ info: QAPISourceInfo) -> None:
+ """
+ Normalize and validate the ``aliases`` member.
+
+ :param aliases: The aliases member value to validate.
+ :param info: QAPI schema source file information.
+
+ :raise QAPISemError: When ``aliases`` fails validation.
+ :return: None, ``aliases`` is normalized in-place as needed.
+ """
+
+ if aliases is None:
+ return
+ if not isinstance(aliases, list):
+ raise QAPISemError(info, "'aliases' must be an array")
+ for a in aliases:
+ if not isinstance(a, dict):
+ raise QAPISemError(info, "'aliases' members must be objects")
+ check_keys(a, info, "'aliases' member", ['source'], ['name'])
+
+ if 'name' in a:
+ source = "alias member 'name'"
+ check_name_is_str(a['name'], info, source)
+ check_name_str(a['name'], info, source)
+
+ if not isinstance(a['source'], list):
+ raise QAPISemError(info,
+ "alias member 'source' must be an array")
+ if not a['source']:
+ raise QAPISemError(info,
+ "alias member 'source' must not be empty")
+
+ source = "member of alias member 'source'"
+ for s in a['source']:
+ check_name_is_str(s, info, source)
+ check_name_str(s, info, source)
+
+
def check_enum(expr: _JSONObject, info: QAPISourceInfo) -> None:
"""
Normalize and validate this expression as an ``enum`` definition.
@@ -483,6 +522,7 @@ def check_struct(expr: _JSONObject, info: QAPISourceInfo) -> None:
check_type(members, info, "'data'", allow_dict=name)
check_type(expr.get('base'), info, "'base'")
+ check_aliases(expr.get('aliases'), info)
def check_union(expr: _JSONObject, info: QAPISourceInfo) -> None:
@@ -509,6 +549,8 @@ def check_union(expr: _JSONObject, info: QAPISourceInfo) -> None:
raise QAPISemError(info, "'discriminator' requires 'base'")
check_name_is_str(discriminator, info, "'discriminator'")
+ check_aliases(expr.get('aliases'), info)
+
if not isinstance(members, dict):
raise QAPISemError(info, "'data' must be an object")
@@ -653,7 +695,7 @@ def check_exprs(exprs: List[_JSONObject]) -> List[_JSONObject]:
elif meta == 'union':
check_keys(expr, info, meta,
['union', 'data'],
- ['base', 'discriminator', 'if', 'features'])
+ ['base', 'discriminator', 'if', 'features', 'aliases'])
normalize_members(expr.get('base'))
normalize_members(expr['data'])
check_union(expr, info)
@@ -664,7 +706,8 @@ def check_exprs(exprs: List[_JSONObject]) -> List[_JSONObject]:
check_alternate(expr, info)
elif meta == 'struct':
check_keys(expr, info, meta,
- ['struct', 'data'], ['base', 'if', 'features'])
+ ['struct', 'data'],
+ ['base', 'if', 'features', 'aliases'])
normalize_members(expr['data'])
check_struct(expr, info)
elif meta == 'command':
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index d1d27ff7ee..fc75635f4e 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -118,7 +118,7 @@ def visit_array_type(self, name, info, ifcond, element_type):
pass
def visit_object_type(self, name, info, ifcond, features,
- base, members, variants):
+ base, members, variants, aliases):
pass
def visit_object_type_flat(self, name, info, ifcond, features,
@@ -364,7 +364,7 @@ def describe(self):
class QAPISchemaObjectType(QAPISchemaType):
def __init__(self, name, info, doc, ifcond, features,
- base, local_members, variants):
+ base, local_members, variants, aliases=None):
# struct has local_members, optional base, and no variants
# flat union has base, variants, and no local_members
# simple union has local_members, variants, and no base
@@ -382,6 +382,9 @@ def __init__(self, name, info, doc, ifcond, features,
self.local_members = local_members
self.variants = variants
self.members = None
+ self.aliases = aliases or []
+ for a in self.aliases:
+ a.set_defined_in(name)
def check(self, schema):
# This calls another type T's .check() exactly when the C
@@ -413,12 +416,16 @@ def check(self, schema):
for m in self.local_members:
m.check(schema)
m.check_clash(self.info, seen)
- members = seen.values()
+ members = list(seen.values())
if self.variants:
self.variants.check(schema, seen)
self.variants.check_clash(self.info, seen)
+ for a in self.aliases:
+ a.check_clash(self.info, seen)
+ self.check_path(a, list(a.source), members)
+
self.members = members # mark completed
# Check that the members of this type do not cause duplicate JSON members,
@@ -430,6 +437,68 @@ def check_clash(self, info, seen):
for m in self.members:
m.check_clash(info, seen)
+ # Deletes elements from path, so pass a copy if you still need them
+ def check_path(self, alias, path, members=None, local_aliases_seen=()):
+ assert isinstance(path, list)
+
+ if not path:
+ return
+ first = path.pop(0)
+
+ for a in self.aliases:
+ if a.name == first:
+ if a in local_aliases_seen:
+ raise QAPISemError(
+ self.info,
+ "%s resolving to '%s' makes '%s' an alias for itself"
+ % (a.describe(self.info), a.source[0], a.source[0]))
+
+ path = a.source + path
+ return self.check_path(alias, path, members,
+ (*local_aliases_seen, a))
+
+ if members is None:
+ assert self.members is not None
+ members = self.members
+ else:
+ assert isinstance(members, list)
+
+ for m in members:
+ if m.name == first:
+ # Wildcard aliases can only accept object types in the whole
+ # path; for single-member aliases, the last element can be
+ # any type
+ need_obj = (alias.name is None) or path
+ if need_obj and not isinstance(m.type, QAPISchemaObjectType):
+ raise QAPISemError(
+ self.info,
+ "%s has non-object '%s' in its source path"
+ % (alias.describe(self.info), m.name))
+ if alias.name is None and m.optional:
+ raise QAPISemError(
+ self.info,
+ "%s has optional object %s in its source path"
+ % (alias.describe(self.info), m.describe(self.info)))
+ if path:
+ m.type.check_path(alias, path)
+ return
+
+ # It is sufficient that the path is valid in at least one variant
+ if self.variants:
+ for v in self.variants.variants:
+ try:
+ return v.type.check_path(alias, [first, *path])
+ except QAPISemError:
+ pass
+ raise QAPISemError(
+ self.info,
+ "%s has a source path that does not exist in any variant of %s"
+ % (alias.describe(self.info), self.describe()))
+
+ raise QAPISemError(
+ self.info,
+ "%s has inexistent source" % alias.describe(self.info))
+
def connect_doc(self, doc=None):
super().connect_doc(doc)
doc = doc or self.doc
@@ -474,7 +543,7 @@ def visit(self, visitor):
super().visit(visitor)
visitor.visit_object_type(
self.name, self.info, self.ifcond, self.features,
- self.base, self.local_members, self.variants)
+ self.base, self.local_members, self.variants, self.aliases)
visitor.visit_object_type_flat(
self.name, self.info, self.ifcond, self.features,
self.members, self.variants)
@@ -639,7 +708,7 @@ def check_clash(self, info, seen):
class QAPISchemaMember:
- """ Represents object members, enum members and features """
+ """ Represents object members, enum members, features and aliases """
role = 'member'
def __init__(self, name, info, ifcond=None):
@@ -705,6 +774,30 @@ class QAPISchemaFeature(QAPISchemaMember):
role = 'feature'
+class QAPISchemaAlias(QAPISchemaMember):
+ role = 'alias'
+
+ def __init__(self, name, info, source):
+ assert name is None or isinstance(name, str)
+ assert source
+ for member in source:
+ assert isinstance(member, str)
+
+ super().__init__(name or '*', info)
+ self.name = name
+ self.source = source
+
+ def check_clash(self, info, seen):
+ if self.name:
+ super().check_clash(info, seen)
+
+ def describe(self, info):
+ if self.name:
+ return super().describe(info)
+ else:
+ return "wildcard alias"
+
+
class QAPISchemaObjectTypeMember(QAPISchemaMember):
def __init__(self, name, info, typ, optional, ifcond=None, features=None):
super().__init__(name, info, ifcond)
@@ -971,6 +1064,12 @@ def _make_features(self, features, info):
return [QAPISchemaFeature(f['name'], info, f.get('if'))
for f in features]
+ def _make_aliases(self, aliases, info):
+ if aliases is None:
+ return []
+ return [QAPISchemaAlias(a.get('name'), info, a['source'])
+ for a in aliases]
+
def _make_enum_members(self, values, info):
return [QAPISchemaEnumMember(v['name'], info, v.get('if'))
for v in values]
@@ -1045,11 +1144,12 @@ def _def_struct_type(self, expr, info, doc):
base = expr.get('base')
data = expr['data']
ifcond = expr.get('if')
+ aliases = self._make_aliases(expr.get('aliases'), info)
features = self._make_features(expr.get('features'), info)
self._def_entity(QAPISchemaObjectType(
name, info, doc, ifcond, features, base,
self._make_members(data, info),
- None))
+ None, aliases))
def _make_variant(self, case, typ, ifcond, info):
return QAPISchemaVariant(case, info, typ, ifcond)
@@ -1068,6 +1168,7 @@ def _def_union_type(self, expr, info, doc):
data = expr['data']
base = expr.get('base')
ifcond = expr.get('if')
+ aliases = self._make_aliases(expr.get('aliases'), info)
features = self._make_features(expr.get('features'), info)
tag_name = expr.get('discriminator')
tag_member = None
@@ -1092,7 +1193,8 @@ def _def_union_type(self, expr, info, doc):
QAPISchemaObjectType(name, info, doc, ifcond, features,
base, members,
QAPISchemaVariants(
- tag_name, info, tag_member, variants)))
+ tag_name, info, tag_member, variants),
+ aliases))
def _def_alternate_type(self, expr, info, doc):
name = expr['alternate']
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 20d572a23a..3bc451baa9 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -25,6 +25,7 @@
from .gen import QAPISchemaModularCVisitor, ifcontext
from .schema import (
QAPISchema,
+ QAPISchemaAlias,
QAPISchemaEnumMember,
QAPISchemaFeature,
QAPISchemaObjectType,
@@ -332,7 +333,8 @@ def visit_object_type(self,
features: List[QAPISchemaFeature],
base: Optional[QAPISchemaObjectType],
members: List[QAPISchemaObjectTypeMember],
- variants: Optional[QAPISchemaVariants]) -> None:
+ variants: Optional[QAPISchemaVariants],
+ aliases: List[QAPISchemaAlias]) -> None:
# Nothing to do for the special empty builtin
if name == 'q_empty':
return
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 9e96f3c566..0aa0764755 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -26,6 +26,7 @@
from .gen import QAPISchemaModularCVisitor, ifcontext
from .schema import (
QAPISchema,
+ QAPISchemaAlias,
QAPISchemaEnumMember,
QAPISchemaEnumType,
QAPISchemaFeature,
@@ -60,7 +61,8 @@ def gen_visit_members_decl(name: str) -> str:
def gen_visit_object_members(name: str,
base: Optional[QAPISchemaObjectType],
members: List[QAPISchemaObjectTypeMember],
- variants: Optional[QAPISchemaVariants]) -> str:
+ variants: Optional[QAPISchemaVariants],
+ aliases: List[QAPISchemaAlias]) -> str:
ret = mcgen('''
bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
@@ -68,6 +70,24 @@ def gen_visit_object_members(name: str,
''',
c_name=c_name(name))
+ if aliases:
+ ret += mcgen('''
+ visit_start_alias_scope(v);
+''')
+
+ for a in aliases:
+ if a.name:
+ name = '"%s"' % a.name
+ else:
+ name = "NULL"
+
+ source = ", ".join('"%s"' % x for x in a.source)
+
+ ret += mcgen('''
+ visit_define_alias(v, %(name)s, (const char * []) { %(source)s, NULL });
+''',
+ name=name, source=source)
+
if base:
ret += mcgen('''
if (!visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, errp)) {
@@ -148,6 +168,11 @@ def gen_visit_object_members(name: str,
}
''')
+ if aliases:
+ ret += mcgen('''
+ visit_end_alias_scope(v);
+''')
+
ret += mcgen('''
return true;
}
@@ -376,14 +401,15 @@ def visit_object_type(self,
features: List[QAPISchemaFeature],
base: Optional[QAPISchemaObjectType],
members: List[QAPISchemaObjectTypeMember],
- variants: Optional[QAPISchemaVariants]) -> None:
+ variants: Optional[QAPISchemaVariants],
+ aliases: List[QAPISchemaAlias]) -> None:
# Nothing to do for the special empty builtin
if name == 'q_empty':
return
with ifcontext(ifcond, self._genh, self._genc):
self._genh.add(gen_visit_members_decl(name))
- self._genc.add(gen_visit_object_members(name, base,
- members, variants))
+ self._genc.add(gen_visit_object_members(
+ name, base, members, variants, aliases))
# TODO Worth changing the visitor signature, so we could
# directly use rather than repeat type.is_implicit()?
if not name.startswith('q_'):
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index f1c4deb9a5..376630901b 100755
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -47,7 +47,7 @@ def visit_array_type(self, name, info, ifcond, element_type):
self._print_if(ifcond)
def visit_object_type(self, name, info, ifcond, features,
- base, members, variants):
+ base, members, variants, aliases):
print('object %s' % name)
if base:
print(' base %s' % base.name)
@@ -56,6 +56,11 @@ def visit_object_type(self, name, info, ifcond, features,
% (m.name, m.type.name, m.optional))
self._print_if(m.ifcond, 8)
self._print_features(m.features, indent=8)
+ for a in aliases:
+ if a.name:
+ print(' alias %s -> %s' % (a.name, '.'.join(a.source)))
+ else:
+ print(' alias * -> %s.*' % '.'.join(a.source))
self._print_variants(variants)
self._print_if(ifcond)
self._print_features(features)
diff --git a/tests/qapi-schema/double-type.err b/tests/qapi-schema/double-type.err
index 576e716197..c382e61d88 100644
--- a/tests/qapi-schema/double-type.err
+++ b/tests/qapi-schema/double-type.err
@@ -1,3 +1,3 @@
double-type.json: In struct 'Bar':
double-type.json:2: struct has unknown key 'command'
-Valid keys are 'base', 'data', 'features', 'if', 'struct'.
+Valid keys are 'aliases', 'base', 'data', 'features', 'if', 'struct'.
diff --git a/tests/qapi-schema/unknown-expr-key.err b/tests/qapi-schema/unknown-expr-key.err
index f2538e3ce7..354916968f 100644
--- a/tests/qapi-schema/unknown-expr-key.err
+++ b/tests/qapi-schema/unknown-expr-key.err
@@ -1,3 +1,3 @@
unknown-expr-key.json: In struct 'Bar':
unknown-expr-key.json:2: struct has unknown keys 'bogus', 'phony'
-Valid keys are 'base', 'data', 'features', 'if', 'struct'.
+Valid keys are 'aliases', 'base', 'data', 'features', 'if', 'struct'.
--
2.31.1
^ permalink raw reply related [flat|nested] 43+ messages in thread
* [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
` (4 preceding siblings ...)
2021-08-12 16:11 ` [PATCH v3 5/6] qapi: Add support for aliases Kevin Wolf
@ 2021-08-12 16:11 ` Kevin Wolf
2021-09-06 15:28 ` Markus Armbruster
2021-08-24 9:36 ` [PATCH v3 0/6] qapi: Add support " Markus Armbruster
2021-09-06 15:32 ` Markus Armbruster
7 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-08-12 16:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, armbru
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
tests/unit/test-qobject-input-visitor.c | 218 ++++++++++++++++++
tests/qapi-schema/alias-bad-type.err | 2 +
tests/qapi-schema/alias-bad-type.json | 3 +
tests/qapi-schema/alias-bad-type.out | 0
tests/qapi-schema/alias-missing-source.err | 2 +
tests/qapi-schema/alias-missing-source.json | 3 +
tests/qapi-schema/alias-missing-source.out | 0
tests/qapi-schema/alias-name-bad-type.err | 2 +
tests/qapi-schema/alias-name-bad-type.json | 3 +
tests/qapi-schema/alias-name-bad-type.out | 0
tests/qapi-schema/alias-name-conflict.err | 2 +
tests/qapi-schema/alias-name-conflict.json | 4 +
tests/qapi-schema/alias-name-conflict.out | 0
tests/qapi-schema/alias-recursive.err | 2 +
tests/qapi-schema/alias-recursive.json | 4 +
tests/qapi-schema/alias-recursive.out | 0
tests/qapi-schema/alias-source-bad-type.err | 2 +
tests/qapi-schema/alias-source-bad-type.json | 3 +
tests/qapi-schema/alias-source-bad-type.out | 0
.../alias-source-elem-bad-type.err | 2 +
.../alias-source-elem-bad-type.json | 3 +
.../alias-source-elem-bad-type.out | 0
tests/qapi-schema/alias-source-empty.err | 2 +
tests/qapi-schema/alias-source-empty.json | 3 +
tests/qapi-schema/alias-source-empty.out | 0
.../alias-source-inexistent-variants.err | 2 +
.../alias-source-inexistent-variants.json | 12 +
.../alias-source-inexistent-variants.out | 0
tests/qapi-schema/alias-source-inexistent.err | 2 +
.../qapi-schema/alias-source-inexistent.json | 3 +
tests/qapi-schema/alias-source-inexistent.out | 0
.../alias-source-non-object-path.err | 2 +
.../alias-source-non-object-path.json | 3 +
.../alias-source-non-object-path.out | 0
.../alias-source-non-object-wildcard.err | 2 +
.../alias-source-non-object-wildcard.json | 3 +
.../alias-source-non-object-wildcard.out | 0
...lias-source-optional-wildcard-indirect.err | 2 +
...ias-source-optional-wildcard-indirect.json | 6 +
...lias-source-optional-wildcard-indirect.out | 0
.../alias-source-optional-wildcard.err | 2 +
.../alias-source-optional-wildcard.json | 5 +
.../alias-source-optional-wildcard.out | 0
tests/qapi-schema/alias-unknown-key.err | 3 +
tests/qapi-schema/alias-unknown-key.json | 3 +
tests/qapi-schema/alias-unknown-key.out | 0
tests/qapi-schema/aliases-bad-type.err | 2 +
tests/qapi-schema/aliases-bad-type.json | 3 +
tests/qapi-schema/aliases-bad-type.out | 0
tests/qapi-schema/meson.build | 16 ++
tests/qapi-schema/qapi-schema-test.json | 26 +++
tests/qapi-schema/qapi-schema-test.out | 31 +++
52 files changed, 388 insertions(+)
create mode 100644 tests/qapi-schema/alias-bad-type.err
create mode 100644 tests/qapi-schema/alias-bad-type.json
create mode 100644 tests/qapi-schema/alias-bad-type.out
create mode 100644 tests/qapi-schema/alias-missing-source.err
create mode 100644 tests/qapi-schema/alias-missing-source.json
create mode 100644 tests/qapi-schema/alias-missing-source.out
create mode 100644 tests/qapi-schema/alias-name-bad-type.err
create mode 100644 tests/qapi-schema/alias-name-bad-type.json
create mode 100644 tests/qapi-schema/alias-name-bad-type.out
create mode 100644 tests/qapi-schema/alias-name-conflict.err
create mode 100644 tests/qapi-schema/alias-name-conflict.json
create mode 100644 tests/qapi-schema/alias-name-conflict.out
create mode 100644 tests/qapi-schema/alias-recursive.err
create mode 100644 tests/qapi-schema/alias-recursive.json
create mode 100644 tests/qapi-schema/alias-recursive.out
create mode 100644 tests/qapi-schema/alias-source-bad-type.err
create mode 100644 tests/qapi-schema/alias-source-bad-type.json
create mode 100644 tests/qapi-schema/alias-source-bad-type.out
create mode 100644 tests/qapi-schema/alias-source-elem-bad-type.err
create mode 100644 tests/qapi-schema/alias-source-elem-bad-type.json
create mode 100644 tests/qapi-schema/alias-source-elem-bad-type.out
create mode 100644 tests/qapi-schema/alias-source-empty.err
create mode 100644 tests/qapi-schema/alias-source-empty.json
create mode 100644 tests/qapi-schema/alias-source-empty.out
create mode 100644 tests/qapi-schema/alias-source-inexistent-variants.err
create mode 100644 tests/qapi-schema/alias-source-inexistent-variants.json
create mode 100644 tests/qapi-schema/alias-source-inexistent-variants.out
create mode 100644 tests/qapi-schema/alias-source-inexistent.err
create mode 100644 tests/qapi-schema/alias-source-inexistent.json
create mode 100644 tests/qapi-schema/alias-source-inexistent.out
create mode 100644 tests/qapi-schema/alias-source-non-object-path.err
create mode 100644 tests/qapi-schema/alias-source-non-object-path.json
create mode 100644 tests/qapi-schema/alias-source-non-object-path.out
create mode 100644 tests/qapi-schema/alias-source-non-object-wildcard.err
create mode 100644 tests/qapi-schema/alias-source-non-object-wildcard.json
create mode 100644 tests/qapi-schema/alias-source-non-object-wildcard.out
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard-indirect.err
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard-indirect.json
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard-indirect.out
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard.err
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard.json
create mode 100644 tests/qapi-schema/alias-source-optional-wildcard.out
create mode 100644 tests/qapi-schema/alias-unknown-key.err
create mode 100644 tests/qapi-schema/alias-unknown-key.json
create mode 100644 tests/qapi-schema/alias-unknown-key.out
create mode 100644 tests/qapi-schema/aliases-bad-type.err
create mode 100644 tests/qapi-schema/aliases-bad-type.json
create mode 100644 tests/qapi-schema/aliases-bad-type.out
diff --git a/tests/unit/test-qobject-input-visitor.c b/tests/unit/test-qobject-input-visitor.c
index e41b91a2a6..f2891b6f5d 100644
--- a/tests/unit/test-qobject-input-visitor.c
+++ b/tests/unit/test-qobject-input-visitor.c
@@ -952,6 +952,214 @@ static void test_visitor_in_list_union_number(TestInputVisitorData *data,
g_string_free(gstr_list, true);
}
+static void test_visitor_in_alias_struct_local(TestInputVisitorData *data,
+ const void *unused)
+{
+ AliasStruct1 *tmp = NULL;
+ Error *err = NULL;
+ Visitor *v;
+
+ /* Can still specify the real member name with alias support */
+ v = visitor_input_test_init(data, "{ 'foo': 42 }");
+ visit_type_AliasStruct1(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->foo, ==, 42);
+ qapi_free_AliasStruct1(tmp);
+
+ /* The alias is a working alternative */
+ v = visitor_input_test_init(data, "{ 'bar': 42 }");
+ visit_type_AliasStruct1(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->foo, ==, 42);
+ qapi_free_AliasStruct1(tmp);
+
+ /* But you can't use both at the same time */
+ v = visitor_input_test_init(data, "{ 'foo': 5, 'bar': 42 }");
+ visit_type_AliasStruct1(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+}
+
+static void test_visitor_in_alias_struct_nested(TestInputVisitorData *data,
+ const void *unused)
+{
+ AliasStruct2 *tmp = NULL;
+ Error *err = NULL;
+ Visitor *v;
+
+ /* Can still specify the real member names with alias support */
+ v = visitor_input_test_init(data, "{ 'nested': { 'foo': 42 } }");
+ visit_type_AliasStruct2(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->nested->foo, ==, 42);
+ qapi_free_AliasStruct2(tmp);
+
+ /* The inner alias is a working alternative */
+ v = visitor_input_test_init(data, "{ 'nested': { 'bar': 42 } }");
+ visit_type_AliasStruct2(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->nested->foo, ==, 42);
+ qapi_free_AliasStruct2(tmp);
+
+ /* So is the outer alias */
+ v = visitor_input_test_init(data, "{ 'bar': 42 }");
+ visit_type_AliasStruct2(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->nested->foo, ==, 42);
+ qapi_free_AliasStruct2(tmp);
+
+ /* You can't use more than one option at the same time */
+ v = visitor_input_test_init(data, "{ 'bar': 5, 'nested': { 'foo': 42 } }");
+ visit_type_AliasStruct2(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'bar': 5, 'nested': { 'bar': 42 } }");
+ visit_type_AliasStruct2(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'nested': { 'foo': 42, 'bar': 42 } }");
+ visit_type_AliasStruct2(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'bar': 5, "
+ " 'nested': { 'foo': 42, 'bar': 42 } }");
+ visit_type_AliasStruct2(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+}
+
+static void test_visitor_in_alias_wildcard(TestInputVisitorData *data,
+ const void *unused)
+{
+ AliasStruct3 *tmp = NULL;
+ Error *err = NULL;
+ Visitor *v;
+
+ /* Can still specify the real member names with alias support */
+ v = visitor_input_test_init(data, "{ 'nested': { 'foo': 42 } }");
+ visit_type_AliasStruct3(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->nested->foo, ==, 42);
+ qapi_free_AliasStruct3(tmp);
+
+ /* The wildcard alias makes it work on the top level */
+ v = visitor_input_test_init(data, "{ 'foo': 42 }");
+ visit_type_AliasStruct3(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->nested->foo, ==, 42);
+ qapi_free_AliasStruct3(tmp);
+
+ /* It makes the inner alias available, too */
+ v = visitor_input_test_init(data, "{ 'bar': 42 }");
+ visit_type_AliasStruct3(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->nested->foo, ==, 42);
+ qapi_free_AliasStruct3(tmp);
+
+ /* You can't use more than one option at the same time */
+ v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'foo': 42 } }");
+ visit_type_AliasStruct3(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'bar': 42, 'nested': { 'foo': 42 } }");
+ visit_type_AliasStruct3(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'bar': 42 } }");
+ visit_type_AliasStruct3(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'bar': 42, 'nested': { 'bar': 42 } }");
+ visit_type_AliasStruct3(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'foo': 42, 'bar': 42 }");
+ visit_type_AliasStruct3(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+}
+
+static void test_visitor_in_alias_flat_union(TestInputVisitorData *data,
+ const void *unused)
+{
+ AliasFlatUnion *tmp = NULL;
+ Error *err = NULL;
+ Visitor *v;
+
+ /* Can still specify the real member name with alias support */
+ v = visitor_input_test_init(data, "{ 'tag': 'drei' }");
+ visit_type_AliasFlatUnion(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->tag, ==, FEATURE_ENUM1_DREI);
+ qapi_free_AliasFlatUnion(tmp);
+
+ /* Use alias for a base member (the discriminator even) */
+ v = visitor_input_test_init(data, "{ 'variant': 'zwei' }");
+ visit_type_AliasFlatUnion(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->tag, ==, FEATURE_ENUM1_ZWEI);
+ qapi_free_AliasFlatUnion(tmp);
+
+ /* Use alias for a variant member */
+ v = visitor_input_test_init(data, "{ 'tag': 'eins', 'bar': 42 }");
+ visit_type_AliasFlatUnion(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->tag, ==, FEATURE_ENUM1_EINS);
+ g_assert_cmpint(tmp->u.eins.foo, ==, 42);
+ qapi_free_AliasFlatUnion(tmp);
+
+ /* Both together */
+ v = visitor_input_test_init(data, "{ 'variant': 'eins', 'bar': 42 }");
+ visit_type_AliasFlatUnion(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->tag, ==, FEATURE_ENUM1_EINS);
+ g_assert_cmpint(tmp->u.eins.foo, ==, 42);
+ qapi_free_AliasFlatUnion(tmp);
+
+ /* You can't use more than one option at the same time for each alias */
+ v = visitor_input_test_init(data, "{ 'variant': 'zwei', 'tag': 'drei' }");
+ visit_type_AliasFlatUnion(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'tag': 'eins', 'foo': 6, 'bar': 9 }");
+ visit_type_AliasFlatUnion(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+}
+
+static void test_visitor_in_alias_simple_union(TestInputVisitorData *data,
+ const void *unused)
+{
+ AliasSimpleUnion *tmp = NULL;
+ Error *err = NULL;
+ Visitor *v;
+
+ /* Can still specify the real member name with alias support */
+ v = visitor_input_test_init(data, "{ 'type': 'eins', "
+ " 'data': { 'foo': 42 } }");
+ visit_type_AliasSimpleUnion(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->type, ==, ALIAS_SIMPLE_UNION_KIND_EINS);
+ g_assert_cmpint(tmp->u.eins.data->foo, ==, 42);
+ qapi_free_AliasSimpleUnion(tmp);
+
+ /* 'type' can be aliased */
+ v = visitor_input_test_init(data, "{ 'tag': 'eins', "
+ " 'data': { 'foo': 42 } }");
+ visit_type_AliasSimpleUnion(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->type, ==, ALIAS_SIMPLE_UNION_KIND_EINS);
+ g_assert_cmpint(tmp->u.eins.data->foo, ==, 42);
+ qapi_free_AliasSimpleUnion(tmp);
+
+ /* The wildcard alias makes it work on the top level */
+ v = visitor_input_test_init(data, "{ 'type': 'eins', 'foo': 42 }");
+ visit_type_AliasSimpleUnion(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->type, ==, ALIAS_SIMPLE_UNION_KIND_EINS);
+ g_assert_cmpint(tmp->u.eins.data->foo, ==, 42);
+ qapi_free_AliasSimpleUnion(tmp);
+
+ /* It makes the inner alias available, too */
+ v = visitor_input_test_init(data, "{ 'type': 'eins', 'bar': 42 }");
+ visit_type_AliasSimpleUnion(v, NULL, &tmp, &error_abort);
+ g_assert_cmpint(tmp->type, ==, ALIAS_SIMPLE_UNION_KIND_EINS);
+ g_assert_cmpint(tmp->u.eins.data->foo, ==, 42);
+ qapi_free_AliasSimpleUnion(tmp);
+
+ /* You can't use more than one option at the same time for each alias */
+ v = visitor_input_test_init(data, "{ 'type': 'eins', 'tag': 'eins' }");
+ visit_type_AliasSimpleUnion(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+
+ v = visitor_input_test_init(data, "{ 'type': 'eins', "
+ " 'bar': 123, "
+ " 'data': { 'foo': 312 } }");
+ visit_type_AliasSimpleUnion(v, NULL, &tmp, &err);
+ error_free_or_abort(&err);
+}
+
static void input_visitor_test_add(const char *testpath,
const void *user_data,
void (*test_func)(TestInputVisitorData *data,
@@ -1350,6 +1558,16 @@ int main(int argc, char **argv)
NULL, test_visitor_in_list_union_string);
input_visitor_test_add("/visitor/input/list_union/number",
NULL, test_visitor_in_list_union_number);
+ input_visitor_test_add("/visitor/input/alias/struct-local",
+ NULL, test_visitor_in_alias_struct_local);
+ input_visitor_test_add("/visitor/input/alias/struct-nested",
+ NULL, test_visitor_in_alias_struct_nested);
+ input_visitor_test_add("/visitor/input/alias/wildcard",
+ NULL, test_visitor_in_alias_wildcard);
+ input_visitor_test_add("/visitor/input/alias/flat-union",
+ NULL, test_visitor_in_alias_flat_union);
+ input_visitor_test_add("/visitor/input/alias/simple-union",
+ NULL, test_visitor_in_alias_simple_union);
input_visitor_test_add("/visitor/input/fail/struct",
NULL, test_visitor_in_fail_struct);
input_visitor_test_add("/visitor/input/fail/struct-nested",
diff --git a/tests/qapi-schema/alias-bad-type.err b/tests/qapi-schema/alias-bad-type.err
new file mode 100644
index 0000000000..820e18ed9c
--- /dev/null
+++ b/tests/qapi-schema/alias-bad-type.err
@@ -0,0 +1,2 @@
+alias-bad-type.json: In struct 'AliasStruct0':
+alias-bad-type.json:1: 'aliases' members must be objects
diff --git a/tests/qapi-schema/alias-bad-type.json b/tests/qapi-schema/alias-bad-type.json
new file mode 100644
index 0000000000..0aa5d206fe
--- /dev/null
+++ b/tests/qapi-schema/alias-bad-type.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ 'must be an object' ] }
diff --git a/tests/qapi-schema/alias-bad-type.out b/tests/qapi-schema/alias-bad-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-missing-source.err b/tests/qapi-schema/alias-missing-source.err
new file mode 100644
index 0000000000..8b7d601fbf
--- /dev/null
+++ b/tests/qapi-schema/alias-missing-source.err
@@ -0,0 +1,2 @@
+alias-missing-source.json: In struct 'AliasStruct0':
+alias-missing-source.json:1: 'aliases' member misses key 'source'
diff --git a/tests/qapi-schema/alias-missing-source.json b/tests/qapi-schema/alias-missing-source.json
new file mode 100644
index 0000000000..b6c91a9488
--- /dev/null
+++ b/tests/qapi-schema/alias-missing-source.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar' } ] }
diff --git a/tests/qapi-schema/alias-missing-source.out b/tests/qapi-schema/alias-missing-source.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-name-bad-type.err b/tests/qapi-schema/alias-name-bad-type.err
new file mode 100644
index 0000000000..489f45ff9b
--- /dev/null
+++ b/tests/qapi-schema/alias-name-bad-type.err
@@ -0,0 +1,2 @@
+alias-name-bad-type.json: In struct 'AliasStruct0':
+alias-name-bad-type.json:1: alias member 'name' requires a string name
diff --git a/tests/qapi-schema/alias-name-bad-type.json b/tests/qapi-schema/alias-name-bad-type.json
new file mode 100644
index 0000000000..17442d5939
--- /dev/null
+++ b/tests/qapi-schema/alias-name-bad-type.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': ['bar'], 'source': ['foo'] } ] }
diff --git a/tests/qapi-schema/alias-name-bad-type.out b/tests/qapi-schema/alias-name-bad-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-name-conflict.err b/tests/qapi-schema/alias-name-conflict.err
new file mode 100644
index 0000000000..d5825a0285
--- /dev/null
+++ b/tests/qapi-schema/alias-name-conflict.err
@@ -0,0 +1,2 @@
+alias-name-conflict.json: In struct 'AliasStruct0':
+alias-name-conflict.json:1: alias 'bar' collides with member 'bar'
diff --git a/tests/qapi-schema/alias-name-conflict.json b/tests/qapi-schema/alias-name-conflict.json
new file mode 100644
index 0000000000..bdb5bd4eab
--- /dev/null
+++ b/tests/qapi-schema/alias-name-conflict.json
@@ -0,0 +1,4 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int',
+ 'bar': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': ['foo'] } ] }
diff --git a/tests/qapi-schema/alias-name-conflict.out b/tests/qapi-schema/alias-name-conflict.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-recursive.err b/tests/qapi-schema/alias-recursive.err
new file mode 100644
index 0000000000..127ce019a8
--- /dev/null
+++ b/tests/qapi-schema/alias-recursive.err
@@ -0,0 +1,2 @@
+alias-recursive.json: In struct 'AliasStruct0':
+alias-recursive.json:1: alias 'baz' resolving to 'bar' makes 'bar' an alias for itself
diff --git a/tests/qapi-schema/alias-recursive.json b/tests/qapi-schema/alias-recursive.json
new file mode 100644
index 0000000000..e25b935324
--- /dev/null
+++ b/tests/qapi-schema/alias-recursive.json
@@ -0,0 +1,4 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': ['baz'] },
+ { 'name': 'baz', 'source': ['bar'] } ] }
diff --git a/tests/qapi-schema/alias-recursive.out b/tests/qapi-schema/alias-recursive.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-bad-type.err b/tests/qapi-schema/alias-source-bad-type.err
new file mode 100644
index 0000000000..b1779cbb8e
--- /dev/null
+++ b/tests/qapi-schema/alias-source-bad-type.err
@@ -0,0 +1,2 @@
+alias-source-bad-type.json: In struct 'AliasStruct0':
+alias-source-bad-type.json:1: alias member 'source' must be an array
diff --git a/tests/qapi-schema/alias-source-bad-type.json b/tests/qapi-schema/alias-source-bad-type.json
new file mode 100644
index 0000000000..d6a7430ee3
--- /dev/null
+++ b/tests/qapi-schema/alias-source-bad-type.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': 'foo' } ] }
diff --git a/tests/qapi-schema/alias-source-bad-type.out b/tests/qapi-schema/alias-source-bad-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-elem-bad-type.err b/tests/qapi-schema/alias-source-elem-bad-type.err
new file mode 100644
index 0000000000..f73fbece77
--- /dev/null
+++ b/tests/qapi-schema/alias-source-elem-bad-type.err
@@ -0,0 +1,2 @@
+alias-source-elem-bad-type.json: In struct 'AliasStruct0':
+alias-source-elem-bad-type.json:1: member of alias member 'source' requires a string name
diff --git a/tests/qapi-schema/alias-source-elem-bad-type.json b/tests/qapi-schema/alias-source-elem-bad-type.json
new file mode 100644
index 0000000000..1d08f56492
--- /dev/null
+++ b/tests/qapi-schema/alias-source-elem-bad-type.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': ['foo', true] } ] }
diff --git a/tests/qapi-schema/alias-source-elem-bad-type.out b/tests/qapi-schema/alias-source-elem-bad-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-empty.err b/tests/qapi-schema/alias-source-empty.err
new file mode 100644
index 0000000000..2848e762cb
--- /dev/null
+++ b/tests/qapi-schema/alias-source-empty.err
@@ -0,0 +1,2 @@
+alias-source-empty.json: In struct 'AliasStruct0':
+alias-source-empty.json:1: alias member 'source' must not be empty
diff --git a/tests/qapi-schema/alias-source-empty.json b/tests/qapi-schema/alias-source-empty.json
new file mode 100644
index 0000000000..74b529de4a
--- /dev/null
+++ b/tests/qapi-schema/alias-source-empty.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': [] } ] }
diff --git a/tests/qapi-schema/alias-source-empty.out b/tests/qapi-schema/alias-source-empty.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-inexistent-variants.err b/tests/qapi-schema/alias-source-inexistent-variants.err
new file mode 100644
index 0000000000..a5d4a4c334
--- /dev/null
+++ b/tests/qapi-schema/alias-source-inexistent-variants.err
@@ -0,0 +1,2 @@
+alias-source-inexistent-variants.json: In union 'AliasStruct0':
+alias-source-inexistent-variants.json:7: alias 'test' has a source path that does not exist in any variant of union type 'AliasStruct0'
diff --git a/tests/qapi-schema/alias-source-inexistent-variants.json b/tests/qapi-schema/alias-source-inexistent-variants.json
new file mode 100644
index 0000000000..6328095b86
--- /dev/null
+++ b/tests/qapi-schema/alias-source-inexistent-variants.json
@@ -0,0 +1,12 @@
+{ 'enum': 'Variants',
+ 'data': [ 'a', 'b' ] }
+{ 'struct': 'Variant0',
+ 'data': { 'foo': 'int' } }
+{ 'struct': 'Variant1',
+ 'data': { 'bar': 'int' } }
+{ 'union': 'AliasStruct0',
+ 'base': { 'type': 'Variants' },
+ 'discriminator': 'type',
+ 'data': { 'a': 'Variant0',
+ 'b': 'Variant1' },
+ 'aliases': [ { 'name': 'test', 'source': ['baz'] } ] }
diff --git a/tests/qapi-schema/alias-source-inexistent-variants.out b/tests/qapi-schema/alias-source-inexistent-variants.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-inexistent.err b/tests/qapi-schema/alias-source-inexistent.err
new file mode 100644
index 0000000000..2d65d3f588
--- /dev/null
+++ b/tests/qapi-schema/alias-source-inexistent.err
@@ -0,0 +1,2 @@
+alias-source-inexistent.json: In struct 'AliasStruct0':
+alias-source-inexistent.json:1: alias 'bar' has inexistent source
diff --git a/tests/qapi-schema/alias-source-inexistent.json b/tests/qapi-schema/alias-source-inexistent.json
new file mode 100644
index 0000000000..5106d3609f
--- /dev/null
+++ b/tests/qapi-schema/alias-source-inexistent.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': ['baz'] } ] }
diff --git a/tests/qapi-schema/alias-source-inexistent.out b/tests/qapi-schema/alias-source-inexistent.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-non-object-path.err b/tests/qapi-schema/alias-source-non-object-path.err
new file mode 100644
index 0000000000..b3c748350f
--- /dev/null
+++ b/tests/qapi-schema/alias-source-non-object-path.err
@@ -0,0 +1,2 @@
+alias-source-non-object-path.json: In struct 'AliasStruct0':
+alias-source-non-object-path.json:1: alias 'bar' has non-object 'foo' in its source path
diff --git a/tests/qapi-schema/alias-source-non-object-path.json b/tests/qapi-schema/alias-source-non-object-path.json
new file mode 100644
index 0000000000..808a3e6281
--- /dev/null
+++ b/tests/qapi-schema/alias-source-non-object-path.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': ['foo', 'baz'] } ] }
diff --git a/tests/qapi-schema/alias-source-non-object-path.out b/tests/qapi-schema/alias-source-non-object-path.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-non-object-wildcard.err b/tests/qapi-schema/alias-source-non-object-wildcard.err
new file mode 100644
index 0000000000..4adc0d2281
--- /dev/null
+++ b/tests/qapi-schema/alias-source-non-object-wildcard.err
@@ -0,0 +1,2 @@
+alias-source-non-object-wildcard.json: In struct 'AliasStruct0':
+alias-source-non-object-wildcard.json:1: wildcard alias has non-object 'foo' in its source path
diff --git a/tests/qapi-schema/alias-source-non-object-wildcard.json b/tests/qapi-schema/alias-source-non-object-wildcard.json
new file mode 100644
index 0000000000..59ce1081ef
--- /dev/null
+++ b/tests/qapi-schema/alias-source-non-object-wildcard.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'source': ['foo'] } ] }
diff --git a/tests/qapi-schema/alias-source-non-object-wildcard.out b/tests/qapi-schema/alias-source-non-object-wildcard.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-optional-wildcard-indirect.err b/tests/qapi-schema/alias-source-optional-wildcard-indirect.err
new file mode 100644
index 0000000000..b58b8ff00f
--- /dev/null
+++ b/tests/qapi-schema/alias-source-optional-wildcard-indirect.err
@@ -0,0 +1,2 @@
+alias-source-optional-wildcard-indirect.json: In struct 'AliasStruct0':
+alias-source-optional-wildcard-indirect.json:3: wildcard alias has optional object member 'nested' in its source path
diff --git a/tests/qapi-schema/alias-source-optional-wildcard-indirect.json b/tests/qapi-schema/alias-source-optional-wildcard-indirect.json
new file mode 100644
index 0000000000..fcf04969dc
--- /dev/null
+++ b/tests/qapi-schema/alias-source-optional-wildcard-indirect.json
@@ -0,0 +1,6 @@
+{ 'struct': 'Nested',
+ 'data': { 'foo': 'int' } }
+{ 'struct': 'AliasStruct0',
+ 'data': { '*nested': 'Nested' },
+ 'aliases': [ { 'name': 'nested-alias', 'source': ['nested'] },
+ { 'source': ['nested-alias'] } ] }
diff --git a/tests/qapi-schema/alias-source-optional-wildcard-indirect.out b/tests/qapi-schema/alias-source-optional-wildcard-indirect.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-source-optional-wildcard.err b/tests/qapi-schema/alias-source-optional-wildcard.err
new file mode 100644
index 0000000000..e39200bd3d
--- /dev/null
+++ b/tests/qapi-schema/alias-source-optional-wildcard.err
@@ -0,0 +1,2 @@
+alias-source-optional-wildcard.json: In struct 'AliasStruct0':
+alias-source-optional-wildcard.json:3: wildcard alias has optional object member 'nested' in its source path
diff --git a/tests/qapi-schema/alias-source-optional-wildcard.json b/tests/qapi-schema/alias-source-optional-wildcard.json
new file mode 100644
index 0000000000..1a315f2ae0
--- /dev/null
+++ b/tests/qapi-schema/alias-source-optional-wildcard.json
@@ -0,0 +1,5 @@
+{ 'struct': 'Nested',
+ 'data': { 'foo': 'int' } }
+{ 'struct': 'AliasStruct0',
+ 'data': { '*nested': 'Nested' },
+ 'aliases': [ { 'source': ['nested'] } ] }
diff --git a/tests/qapi-schema/alias-source-optional-wildcard.out b/tests/qapi-schema/alias-source-optional-wildcard.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/alias-unknown-key.err b/tests/qapi-schema/alias-unknown-key.err
new file mode 100644
index 0000000000..c7b8cb9498
--- /dev/null
+++ b/tests/qapi-schema/alias-unknown-key.err
@@ -0,0 +1,3 @@
+alias-unknown-key.json: In struct 'AliasStruct0':
+alias-unknown-key.json:1: 'aliases' member has unknown key 'known'
+Valid keys are 'name', 'source'.
diff --git a/tests/qapi-schema/alias-unknown-key.json b/tests/qapi-schema/alias-unknown-key.json
new file mode 100644
index 0000000000..cdb8fc3d07
--- /dev/null
+++ b/tests/qapi-schema/alias-unknown-key.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': ['foo'], 'known': false } ] }
diff --git a/tests/qapi-schema/alias-unknown-key.out b/tests/qapi-schema/alias-unknown-key.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/aliases-bad-type.err b/tests/qapi-schema/aliases-bad-type.err
new file mode 100644
index 0000000000..7ffe789ec0
--- /dev/null
+++ b/tests/qapi-schema/aliases-bad-type.err
@@ -0,0 +1,2 @@
+aliases-bad-type.json: In struct 'AliasStruct0':
+aliases-bad-type.json:1: 'aliases' must be an array
diff --git a/tests/qapi-schema/aliases-bad-type.json b/tests/qapi-schema/aliases-bad-type.json
new file mode 100644
index 0000000000..4bbf6d6b20
--- /dev/null
+++ b/tests/qapi-schema/aliases-bad-type.json
@@ -0,0 +1,3 @@
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': 'this must be an array' }
diff --git a/tests/qapi-schema/aliases-bad-type.out b/tests/qapi-schema/aliases-bad-type.out
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qapi-schema/meson.build b/tests/qapi-schema/meson.build
index b8de58116a..f937de1c35 100644
--- a/tests/qapi-schema/meson.build
+++ b/tests/qapi-schema/meson.build
@@ -3,6 +3,22 @@ test_env.set('PYTHONPATH', meson.source_root() / 'scripts')
test_env.set('PYTHONIOENCODING', 'utf-8')
schemas = [
+ 'alias-bad-type.json',
+ 'aliases-bad-type.json',
+ 'alias-missing-source.json',
+ 'alias-name-bad-type.json',
+ 'alias-name-conflict.json',
+ 'alias-recursive.json',
+ 'alias-source-bad-type.json',
+ 'alias-source-elem-bad-type.json',
+ 'alias-source-empty.json',
+ 'alias-source-inexistent.json',
+ 'alias-source-inexistent-variants.json',
+ 'alias-source-non-object-path.json',
+ 'alias-source-non-object-wildcard.json',
+ 'alias-source-optional-wildcard.json',
+ 'alias-source-optional-wildcard-indirect.json',
+ 'alias-unknown-key.json',
'alternate-any.json',
'alternate-array.json',
'alternate-base.json',
diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index 84b9d41f15..c5e81a883c 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -336,3 +336,29 @@
{ 'event': 'TEST_EVENT_FEATURES1',
'features': [ 'deprecated' ] }
+
+# test 'aliases'
+
+{ 'struct': 'AliasStruct0',
+ 'data': { 'foo': 'int' },
+ 'aliases': [] }
+{ 'struct': 'AliasStruct1',
+ 'data': { 'foo': 'int' },
+ 'aliases': [ { 'name': 'bar', 'source': ['foo'] } ] }
+{ 'struct': 'AliasStruct2',
+ 'data': { 'nested': 'AliasStruct1' },
+ 'aliases': [ { 'name': 'bar', 'source': ['nested', 'foo'] } ] }
+{ 'struct': 'AliasStruct3',
+ 'data': { 'nested': 'AliasStruct1' },
+ 'aliases': [ { 'source': ['nested'] } ] }
+
+{ 'union': 'AliasFlatUnion',
+ 'base': { 'tag': 'FeatureEnum1' },
+ 'discriminator': 'tag',
+ 'data': { 'eins': 'FeatureStruct1' },
+ 'aliases': [ { 'name': 'variant', 'source': ['tag'] },
+ { 'name': 'bar', 'source': ['foo'] } ] }
+{ 'union': 'AliasSimpleUnion',
+ 'data': { 'eins': 'AliasStruct1' },
+ 'aliases': [ { 'source': ['data'] },
+ { 'name': 'tag', 'source': ['type'] } ] }
diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
index e0b8a5f0b6..f6b8a98b7c 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -445,6 +445,37 @@ event TEST_EVENT_FEATURES0 FeatureStruct1
event TEST_EVENT_FEATURES1 None
boxed=False
feature deprecated
+object AliasStruct0
+ member foo: int optional=False
+object AliasStruct1
+ member foo: int optional=False
+ alias bar -> foo
+object AliasStruct2
+ member nested: AliasStruct1 optional=False
+ alias bar -> nested.foo
+object AliasStruct3
+ member nested: AliasStruct1 optional=False
+ alias * -> nested.*
+object q_obj_AliasFlatUnion-base
+ member tag: FeatureEnum1 optional=False
+object AliasFlatUnion
+ base q_obj_AliasFlatUnion-base
+ alias variant -> tag
+ alias bar -> foo
+ tag tag
+ case eins: FeatureStruct1
+ case zwei: q_empty
+ case drei: q_empty
+object q_obj_AliasStruct1-wrapper
+ member data: AliasStruct1 optional=False
+enum AliasSimpleUnionKind
+ member eins
+object AliasSimpleUnion
+ member type: AliasSimpleUnionKind optional=False
+ alias * -> data.*
+ alias tag -> type
+ tag type
+ case eins: q_obj_AliasStruct1-wrapper
module include/sub-module.json
include sub-sub-module.json
object SecondArrayRef
--
2.31.1
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH v3 0/6] qapi: Add support for aliases
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
` (5 preceding siblings ...)
2021-08-12 16:11 ` [PATCH v3 6/6] tests/qapi-schema: Test cases " Kevin Wolf
@ 2021-08-24 9:36 ` Markus Armbruster
2021-09-06 15:32 ` Markus Armbruster
7 siblings, 0 replies; 43+ messages in thread
From: Markus Armbruster @ 2021-08-24 9:36 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Conflicts with Marc-André's "[PATCH v7 00/10] qapi: untie 'if'
conditions from C preprocessor", which I queued for 6.2. The conflicts
look harmless to me.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 4/6] qapi: Apply aliases in qobject-input-visitor
2021-08-12 16:11 ` [PATCH v3 4/6] qapi: Apply aliases " Kevin Wolf
@ 2021-09-06 15:16 ` Markus Armbruster
2021-09-08 13:01 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-09-06 15:16 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> When looking for an object in a struct in the external representation,
> check not only the currently visited struct, but also whether an alias
> in the current StackObject matches and try to fetch the value from the
> alias then. Providing two values for the same object through different
> aliases is an error.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
> qapi/qobject-input-visitor.c | 227 +++++++++++++++++++++++++++++++++--
> 1 file changed, 218 insertions(+), 9 deletions(-)
>
> diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
> index 16a75442ff..6193df28a5 100644
> --- a/qapi/qobject-input-visitor.c
> +++ b/qapi/qobject-input-visitor.c
> @@ -97,6 +97,8 @@ struct QObjectInputVisitor {
> QObject *root;
> bool keyval; /* Assume @root made with keyval_parse() */
>
> + QDict *empty_qdict; /* Used for implicit objects */
Would
/* For visiting objects where all members are from aliases */
be clearer?
> +
> /* Stack of objects being visited (all entries will be either
> * QDict or QList). */
> QSLIST_HEAD(, StackObject) stack;
> @@ -169,9 +171,190 @@ static const char *full_name(QObjectInputVisitor *qiv, const char *name)
> return full_name_so(qiv, name, false, tos);
> }
>
> +static bool find_object_member(QObjectInputVisitor *qiv,
> + StackObject **so, const char **name,
> + bool *is_alias_prefix, Error **errp);
According to the function's contract below, three cases:
* Input present: update *so, *name, return true.
* Input absent: zap *so, *name, set *is_alias_prefix, return false.
* Error: set *errp, leave *is_alias_prefix undefined, return false.
> +
> +/*
> + * Check whether the member @name in @so, or an alias for it, is
> + * present in the input and can be used to obtain the value.
> + */
> +static bool input_present(QObjectInputVisitor *qiv, StackObject *so,
> + const char *name)
> +{
> + /*
> + * Check whether the alias member is present in the input
> + * (possibly recursively because aliases are transitive).
> + * The QAPI generator makes sure that alises cannot form loops, so
> + * the recursion guaranteed to terminate.
> + */
> + if (!find_object_member(qiv, &so, &name, NULL, NULL)) {
* Input absent: zap @so and @name.
* Error: don't zap.
Since @so and @name aren't used anymore, the difference doesn't matter.
Okay.
> + return false;
> + }
> +
> + /*
> + * Every source can be used only once. If a value in the input
> + * would end up being used twice through aliases, we'll fail the
> + * second access.
> + */
> + if (!g_hash_table_contains(so->h, name)) {
> + return false;
> + }
> +
> + return true;
> +}
> +
> +/*
> + * Check whether the member @name in the object visited by @so can be
> + * specified in the input by using the alias described by @a (which
> + * must be an alias contained in so->aliases).
> + *
> + * If @name is only a prefix of the alias source, but doesn't match
> + * immediately, false is returned and *is_alias_prefix is set to true
> + * if it is non-NULL. In all other cases, *is_alias_prefix is left
> + * unchanged.
> + */
> +static bool alias_source_matches(QObjectInputVisitor *qiv,
> + StackObject *so, InputVisitorAlias *a,
> + const char *name, bool *is_alias_prefix)
> +{
> + if (a->src[0] == NULL) {
> + assert(a->name == NULL);
> + return true;
> + }
> +
> + if (!strcmp(a->src[0], name)) {
> + if (a->name && a->src[1] == NULL) {
> + /*
> + * We're matching an exact member, the source for this alias is
> + * immediately in @so.
> + */
> + return true;
> + } else if (is_alias_prefix) {
> + /*
> + * We're only looking at a prefix of the source path for the alias.
> + * If the input contains no object of the requested name, we will
> + * implicitly create an empty one so that the alias can still be
> + * used.
> + *
> + * We want to create the implicit object only if the alias is
> + * actually used, but we can't tell here for wildcard aliases (only
> + * a later visitor call will determine this). This means that
> + * wildcard aliases must never have optional keys in their source
> + * path. The QAPI generator checks this condition.
> + */
Double-checking: this actually ensures that we only ever create the
implicit object when it will not remain empty. Correct?
> + if (!a->name || input_present(qiv, a->alias_so, a->name)) {
> + *is_alias_prefix = true;
> + }
> + }
> + }
> +
> + return false;
> +}
> +
> +/*
> + * Find the place in the input where the value for the object member
> + * @name in @so is specified, considering applicable aliases.
> + *
> + * If a value could be found, true is returned and @so and @name are
> + * updated to identify the key name and StackObject where the value
> + * can be found in the input. (This is either unchanged or the
> + * alias_so/name of an alias.) The value of @is_alias_prefix on
> + * return is undefined in this case.
> + *
> + * If no value could be found in the input, false is returned and @so
> + * and @name are set to NULL. This is not an error and @errp remains
> + * unchanged. If @is_alias_prefix is non-NULL, it is set to true if
> + * the given name is a prefix of the source path of an alias for which
> + * a value may be present in the input. It is set to false otherwise.
> + *
> + * If an error occurs (e.g. two values are specified for the member
> + * through different names), false is returned and @errp is set. The
> + * value of @is_alias_prefix on return is undefined in this case.
> + */
> +static bool find_object_member(QObjectInputVisitor *qiv,
> + StackObject **so, const char **name,
> + bool *is_alias_prefix, Error **errp)
> +{
> + QDict *qdict = qobject_to(QDict, (*so)->obj);
> + const char *found_name = NULL;
> + StackObject *found_so = NULL;
> + bool found_is_wildcard = false;
> + InputVisitorAlias *a;
> +
> + if (is_alias_prefix) {
> + *is_alias_prefix = false;
> + }
> +
> + /* Directly present in the container */
> + if (qdict_haskey(qdict, *name)) {
> + found_name = *name;
> + found_so = *so;
> + }
> +
> + /*
> + * Find aliases whose source path matches @name in this StackObject. We can
> + * then get the value with the key a->name from a->alias_so.
> + */
> + QSLIST_FOREACH(a, &(*so)->aliases, next) {
> + if (a->name == NULL && found_name && !found_is_wildcard) {
> + /*
> + * Skip wildcard aliases if we already have a match. This is
> + * not a conflict that should result in an error.
> + *
> + * However, multiple wildcard aliases matching is an error
> + * and will be caught below.
> + */
> + continue;
> + }
> +
> + if (!alias_source_matches(qiv, *so, a, *name, is_alias_prefix)) {
> + continue;
> + }
According to the contract of alias_source_matches() above, three cases:
* No match: try next alias
* Partial match: set *is_alias_prefix = true if non-null
* Full match: leave it alone
> +
> + /*
> + * For single-member aliases, an alias name is specified in the
> + * alias definition. For wildcard aliases, the alias has the same
> + * name as the member in the source object, i.e. *name.
> + */
> + if (!input_present(qiv, a->alias_so, a->name ?: *name)) {
> + continue;
What if alias_source_matches() already set *is_alias_prefix = true?
I figure this can't happen, because it guards the assignment with the
exact same call of input_present(). In other words, we can get here
only for "full match". Correct?
Such repeated calls of helpers can be a sign of awkward interfaces.
Let's not worry about that now.
> + }
> +
> + /*
> + * A non-wildcard alias simply overrides a wildcard alias, but
> + * two matching non-wildcard aliases or two matching wildcard
> + * aliases conflict with each other.
> + */
> + if (found_name && (!found_is_wildcard || a->name == NULL)) {
> + error_setg(errp, "Value for parameter %s was already given "
> + "through an alias",
> + full_name_so(qiv, *name, false, *so));
> + return false;
> + } else {
> + found_name = a->name ?: *name;
> + found_so = a->alias_so;
> + found_is_wildcard = !a->name;
> + }
> + }
> +
> + /*
> + * Chained aliases: *found_so/found_name might be the source of
> + * another alias.
> + */
> + if (found_name && (found_so != *so || found_name != *name)) {
> + find_object_member(qiv, &found_so, &found_name, NULL, errp);
* Input present: update @found_so, @found_name.
* Input absent: zap @found_name, @found_name.
* Error: set *errp.
Can @found_name be non-null? If yes, we can set *errp and return
true, which would be bad.
> + }
> +
> + *so = found_so;
> + *name = found_name;
> +
> + return found_name != NULL;
> +}
> +
> static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
> const char *name,
> - bool consume)
> + bool consume, Error **errp)
Before the patch, two cases:
* Input present: consume input if @consume, return the input
* Input absent: return null
The patch adds
* Other error: set *errp, return null
Slightly awkward to use, as we shall see below at [1] and [2].
Observation, not demand.
> {
> StackObject *tos;
> QObject *qobj;
> @@ -189,10 +372,31 @@ static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
> assert(qobj);
>
> if (qobject_type(qobj) == QTYPE_QDICT) {
> - assert(name);
> - ret = qdict_get(qobject_to(QDict, qobj), name);
> - if (tos->h && consume && ret) {
> - bool removed = g_hash_table_remove(tos->h, name);
> + StackObject *so = tos;
> + const char *key = name;
> + bool is_alias_prefix;
> +
> + assert(key);
> + if (!find_object_member(qiv, &so, &key, &is_alias_prefix, errp)) {
* Input absent: zap @so, @key, set @is_alias_prefix.
* Error: set *errp, leave @is_alias_prefix undefined.
> + if (is_alias_prefix) {
Use of undefined @is_alias_prefix in case "Error". Bug in code or in
contract?
> + /*
> + * The member is not present in the input, but
> + * something inside of it might still be given through
> + * an alias. Pretend there was an empty object in the
> + * input.
> + */
We pretend there was an object to make the calling visitor enter the
object and visit its members. Visiting a member first looks for input
in the (empty) object, then follows aliases to look for it elsewhere.
Is "might" still correct? The comment in alias_source_matches() makes
me hope it's actually "will".
> + if (!qiv->empty_qdict) {
> + qiv->empty_qdict = qdict_new();
> + }
> + return QOBJECT(qiv->empty_qdict);
> + } else {
> + return NULL;
> + }
> + }
> + ret = qdict_get(qobject_to(QDict, so->obj), key);
> + assert(ret != NULL);
> + if (so->h && consume) {
> + bool removed = g_hash_table_remove(so->h, key);
> assert(removed);
> }
> } else {
> @@ -218,9 +422,10 @@ static QObject *qobject_input_get_object(QObjectInputVisitor *qiv,
> const char *name,
> bool consume, Error **errp)
> {
> - QObject *obj = qobject_input_try_get_object(qiv, name, consume);
> + ERRP_GUARD();
> + QObject *obj = qobject_input_try_get_object(qiv, name, consume, errp);
>
> - if (!obj) {
> + if (!obj && !*errp) {
> error_setg(errp, QERR_MISSING_PARAMETER, full_name(qiv, name));
> }
[1] Squash case "Input absent" into case "Error".
> return obj;
> @@ -803,13 +1008,16 @@ static bool qobject_input_type_size_keyval(Visitor *v, const char *name,
> static void qobject_input_optional(Visitor *v, const char *name, bool *present)
> {
> QObjectInputVisitor *qiv = to_qiv(v);
> - QObject *qobj = qobject_input_try_get_object(qiv, name, false);
> + Error *local_err = NULL;
> + QObject *qobj = qobject_input_try_get_object(qiv, name, false, &local_err);
>
> - if (!qobj) {
> + /* If there was an error, let the caller try and run into the error */
> + if (!qobj && !local_err) {
[2] Case "Input absent", and ...
> *present = false;
> return;
> }
... cases "Input present" and "Error".
>
> + error_free(local_err);
> *present = true;
> }
>
> @@ -842,6 +1050,7 @@ static void qobject_input_free(Visitor *v)
> qobject_input_stack_object_free(tos);
> }
>
> + qobject_unref(qiv->empty_qdict);
> qobject_unref(qiv->root);
> if (qiv->errname) {
> g_string_free(qiv->errname, TRUE);
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 5/6] qapi: Add support for aliases
2021-08-12 16:11 ` [PATCH v3 5/6] qapi: Add support for aliases Kevin Wolf
@ 2021-09-06 15:24 ` Markus Armbruster
2021-09-09 16:39 ` Kevin Wolf
2021-09-16 7:49 ` Markus Armbruster
1 sibling, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-09-06 15:24 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Introduce alias definitions for object types (structs and unions). This
> allows using the same QAPI type and visitor for many syntax variations
> that exist in the external representation, like between QMP and the
> command line. It also provides a new tool for evolving the schema while
> maintaining backwards compatibility during a deprecation period.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
> docs/devel/qapi-code-gen.rst | 104 +++++++++++++++++++++-
> docs/sphinx/qapidoc.py | 2 +-
> scripts/qapi/expr.py | 47 +++++++++-
> scripts/qapi/schema.py | 116 +++++++++++++++++++++++--
> scripts/qapi/types.py | 4 +-
> scripts/qapi/visit.py | 34 +++++++-
> tests/qapi-schema/test-qapi.py | 7 +-
> tests/qapi-schema/double-type.err | 2 +-
> tests/qapi-schema/unknown-expr-key.err | 2 +-
> 9 files changed, 297 insertions(+), 21 deletions(-)
>
> diff --git a/docs/devel/qapi-code-gen.rst b/docs/devel/qapi-code-gen.rst
> index 26c62b0e7b..c0883507a8 100644
> --- a/docs/devel/qapi-code-gen.rst
> +++ b/docs/devel/qapi-code-gen.rst
> @@ -262,7 +262,8 @@ Syntax::
> 'data': MEMBERS,
> '*base': STRING,
> '*if': COND,
> - '*features': FEATURES }
> + '*features': FEATURES,
> + '*aliases': ALIASES }
> MEMBERS = { MEMBER, ... }
> MEMBER = STRING : TYPE-REF
> | STRING : { 'type': TYPE-REF,
> @@ -312,6 +313,9 @@ the schema`_ below for more on this.
> The optional 'features' member specifies features. See Features_
> below for more on this.
>
> +The optional 'aliases' member specifies aliases. See Aliases_ below
> +for more on this.
> +
>
> Union types
> -----------
> @@ -321,13 +325,15 @@ Syntax::
> UNION = { 'union': STRING,
> 'data': BRANCHES,
> '*if': COND,
> - '*features': FEATURES }
> + '*features': FEATURES,
> + '*aliases': ALIASES }
> | { 'union': STRING,
> 'data': BRANCHES,
> 'base': ( MEMBERS | STRING ),
> 'discriminator': STRING,
> '*if': COND,
> - '*features': FEATURES }
> + '*features': FEATURES,
> + '*aliases': ALIASES }
> BRANCHES = { BRANCH, ... }
> BRANCH = STRING : TYPE-REF
> | STRING : { 'type': TYPE-REF, '*if': COND }
> @@ -437,6 +443,9 @@ the schema`_ below for more on this.
> The optional 'features' member specifies features. See Features_
> below for more on this.
>
> +The optional 'aliases' member specifies aliases. See Aliases_ below
> +for more on this.
> +
>
> Alternate types
> ---------------
> @@ -888,6 +897,95 @@ shows a conditional entity only when the condition is satisfied in
> this particular build.
>
>
> +Aliases
> +-------
> +
> +Object types, including structs and unions, can contain alias
> +definitions.
> +
> +Aliases define alternative member names that may be used in wire input
> +to provide a value for a member in the same object or in a nested
> +object.
Explaining intended use would be nice. From your cover letter:
This allows using the same QAPI type and visitor even when the syntax
has some variations between different external interfaces such as QMP
and the command line.
It also provides a new tool for evolving the schema while maintaining
backwards compatibility (possibly during a deprecation period).
For the second use, we need to be able to tack feature 'deprecated' to
exactly one of the two.
We can already tack it to the "real" member. The real member's
'deprecated' must not apply to its aliases.
We can't tack it to the alias, yet. More on that in review of PATCH 6.
> +
> +Syntax::
> +
> + ALIASES = [ ALIAS, ... ]
> + ALIAS = { '*name': STRING,
> + 'source': [ STRING, ... ] }
> +
> +If ``name`` is present, then the single member referred to by ``source``
> +is made accessible with the name given by ``name`` in the type where the
> +alias definition is specified.
> +
> +If ``name`` is not present, then this is a wildcard alias and all
> +members in the object referred to by ``source`` are made accessible in
> +the type where the alias definition is specified with the same name as
> +they have in ``source``.
> +
> +``source`` is a non-empty list of member names representing the path to
> +an object member. The first name is resolved in the same object. Each
> +subsequent member is resolved in the object named by the preceding
> +member.
> +
> +Do not use optional objects in the path of a wildcard alias unless there
> +is no semantic difference between an empty object and an absent object.
> +Absent objects are implicitly turned into empty ones if an alias could
> +apply and provide a value in the nested object, which is always the case
> +for wildcard aliases.
> +
> +Example: Alternative name for a member in the same object ::
> +
> + { 'struct': 'File',
> + 'data': { 'path': 'str' },
> + 'aliases': [ { 'name': 'filename', 'source': ['path'] } ] }
> +
> +The member ``path`` may instead be given through its alias ``filename``
> +in input.
> +
> +Example: Alias for a member in a nested object ::
> +
> + { 'struct': 'A',
> + 'data': { 'zahl': 'int' } }
> + { 'struct': 'B',
> + 'data': { 'drei': 'A' } }
> + { 'struct': 'C',
> + 'data': { 'zwei': 'B' } }
> + { 'struct': 'D',
> + 'data': { 'eins': 'C' },
> + 'aliases': [ { 'name': 'number',
> + 'source': ['eins', 'zwei', 'drei', 'zahl' ] },
> + { 'name': 'the_B',
> + 'source': ['eins','zwei'] } ] }
> +
> +With this definition, each of the following inputs for ``D`` mean the
> +same::
> +
> + { 'eins': { 'zwei': { 'drei': { 'zahl': 42 } } } }
> +
> + { 'the_B': { 'drei': { 'zahl': 42 } } }
> +
> + { 'number': 42 }
> +
> +Example: Flattening a simple union with a wildcard alias that maps all
> +members of ``data`` to the top level ::
> +
> + { 'union': 'SocketAddress',
> + 'data': {
> + 'inet': 'InetSocketAddress',
> + 'unix': 'UnixSocketAddress' },
> + 'aliases': [ { 'source': ['data'] } ] }
> +
> +Aliases are transitive: ``source`` may refer to another alias name. In
> +this case, the alias is effectively an alternative name for the source
> +of the other alias.
> +
> +In order to accommodate unions where variants differ in structure, it
> +is allowed to use a path that doesn't necessarily match an existing
> +member in every variant; in this case, the alias remains unused. The
> +QAPI generator checks that there is at least one branch for which an
> +alias could match.
> +
> +
> Documentation comments
> ----------------------
>
> diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> index 87c67ab23f..68340b8529 100644
> --- a/docs/sphinx/qapidoc.py
> +++ b/docs/sphinx/qapidoc.py
> @@ -313,7 +313,7 @@ def visit_enum_type(self, name, info, ifcond, features, members, prefix):
> + self._nodes_for_if_section(ifcond))
>
> def visit_object_type(self, name, info, ifcond, features,
> - base, members, variants):
> + base, members, variants, aliases):
> doc = self._cur_doc
> if base and base.is_implicit():
> base = None
> diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
> index cf98923fa6..054fef8d8e 100644
> --- a/scripts/qapi/expr.py
> +++ b/scripts/qapi/expr.py
> @@ -430,6 +430,45 @@ def check_features(features: Optional[object],
> check_if(feat, info, source)
>
>
> +def check_aliases(aliases: Optional[object],
> + info: QAPISourceInfo) -> None:
> + """
> + Normalize and validate the ``aliases`` member.
> +
> + :param aliases: The aliases member value to validate.
> + :param info: QAPI schema source file information.
> +
> + :raise QAPISemError: When ``aliases`` fails validation.
> + :return: None, ``aliases`` is normalized in-place as needed.
> + """
> +
> + if aliases is None:
> + return
> + if not isinstance(aliases, list):
> + raise QAPISemError(info, "'aliases' must be an array")
Covered by PATCH 6's aliases-bad-type. Good.
> + for a in aliases:
> + if not isinstance(a, dict):
> + raise QAPISemError(info, "'aliases' members must be objects")
Convered by alias-bad-type.
Doesn't identify the offending member. Same for all errors reported in
this loop. Users should have no trouble identifying this one
themselves. Less obvious ones might be confusing.
Class QAPISchemaAlias identifies like 'alias ' + a['name'] and 'wildcard
alias', as several test results show, e.g. alias-name-conflict.err and
alias-source-non-object-wildcard.err. Could be improved on top.
> + check_keys(a, info, "'aliases' member", ['source'], ['name'])
Covered by alias-missing-source and alias-unknown-key.
> +
> + if 'name' in a:
> + source = "alias member 'name'"
> + check_name_is_str(a['name'], info, source)
Covered by alias-name-bad-type.
I understand the desire to reuse an existing check, but the resulting
error message feels awkward: "alias member 'name' requires a string
name". We can improve suboptimal error messages on top.
> + check_name_str(a['name'], info, source)
Not covered. Tolerable.
However, aliases are like members, and we therefore need
check_name_lower() here. Necessary to reject
'aliases': [{'name': 'Bar', 'source': ['foo']}]
Can fix in my tree.
Note: if we simply replace check_name_str by check_name_lower here, we
don't support pragma member-name-exceptions. We can address that when
we need it. Worth a comment, though.
> +
> + if not isinstance(a['source'], list):
> + raise QAPISemError(info,
> + "alias member 'source' must be an array")
Covered by alias-source-bad-type.
pycodestyle-3 gripes:
scripts/qapi/expr.py:461:17: E128 continuation line under-indented for visual indent
Obvious fix:
raise QAPISemError(
info, "alias member 'source' must be an array")
> + if not a['source']:
> + raise QAPISemError(info,
> + "alias member 'source' must not be empty")
Covered by alias-source-empty.
pycodestyle-3:
scripts/qapi/expr.py:464:17: E128 continuation line under-indented for visual indent
> +
> + source = "member of alias member 'source'"
> + for s in a['source']:
> + check_name_is_str(s, info, source)
Covered by alias-source-elem-bad-type.
> + check_name_str(s, info, source)
Not covered. Tolerable, but should we check at all? We also check that
the alias can resolve, don't we? If it resolves, then the elements of
@source match names that are checked elsewhere.
In short, we enforce naming conventions for definitions, not uses, and
this is a use.
If I'm wrong and we need to check, then check_name_lower(). Adding
support for pragma member-name-exceptions could be hairy.
> +
> +
> def check_enum(expr: _JSONObject, info: QAPISourceInfo) -> None:
> """
> Normalize and validate this expression as an ``enum`` definition.
> @@ -483,6 +522,7 @@ def check_struct(expr: _JSONObject, info: QAPISourceInfo) -> None:
>
> check_type(members, info, "'data'", allow_dict=name)
> check_type(expr.get('base'), info, "'base'")
> + check_aliases(expr.get('aliases'), info)
>
>
> def check_union(expr: _JSONObject, info: QAPISourceInfo) -> None:
> @@ -509,6 +549,8 @@ def check_union(expr: _JSONObject, info: QAPISourceInfo) -> None:
> raise QAPISemError(info, "'discriminator' requires 'base'")
> check_name_is_str(discriminator, info, "'discriminator'")
>
> + check_aliases(expr.get('aliases'), info)
> +
> if not isinstance(members, dict):
> raise QAPISemError(info, "'data' must be an object")
>
> @@ -653,7 +695,7 @@ def check_exprs(exprs: List[_JSONObject]) -> List[_JSONObject]:
> elif meta == 'union':
> check_keys(expr, info, meta,
> ['union', 'data'],
> - ['base', 'discriminator', 'if', 'features'])
> + ['base', 'discriminator', 'if', 'features', 'aliases'])
I'll break this line if you don't mind.
> normalize_members(expr.get('base'))
> normalize_members(expr['data'])
> check_union(expr, info)
> @@ -664,7 +706,8 @@ def check_exprs(exprs: List[_JSONObject]) -> List[_JSONObject]:
> check_alternate(expr, info)
> elif meta == 'struct':
> check_keys(expr, info, meta,
> - ['struct', 'data'], ['base', 'if', 'features'])
> + ['struct', 'data'],
> + ['base', 'if', 'features', 'aliases'])
> normalize_members(expr['data'])
> check_struct(expr, info)
> elif meta == 'command':
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index d1d27ff7ee..fc75635f4e 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -118,7 +118,7 @@ def visit_array_type(self, name, info, ifcond, element_type):
> pass
>
> def visit_object_type(self, name, info, ifcond, features,
> - base, members, variants):
> + base, members, variants, aliases):
> pass
>
> def visit_object_type_flat(self, name, info, ifcond, features,
> @@ -364,7 +364,7 @@ def describe(self):
>
> class QAPISchemaObjectType(QAPISchemaType):
> def __init__(self, name, info, doc, ifcond, features,
> - base, local_members, variants):
> + base, local_members, variants, aliases=None):
> # struct has local_members, optional base, and no variants
> # flat union has base, variants, and no local_members
> # simple union has local_members, variants, and no base
> @@ -382,6 +382,9 @@ def __init__(self, name, info, doc, ifcond, features,
> self.local_members = local_members
> self.variants = variants
> self.members = None
> + self.aliases = aliases or []
> + for a in self.aliases:
> + a.set_defined_in(name)
>
> def check(self, schema):
> # This calls another type T's .check() exactly when the C
> @@ -413,12 +416,16 @@ def check(self, schema):
> for m in self.local_members:
> m.check(schema)
> m.check_clash(self.info, seen)
> - members = seen.values()
> + members = list(seen.values())
Uh, why do you need this? If I back it out, .check_path()'s assert
isinstance(members, list) fails. If I take that out as well, "make
check" passes. So does asserting isinstance(members, ValuesView).
For what it's worth: when this code was written, we still used Python2,
where .values() returns a list. The switch to Python3 silently made
@members and self.members (assigned below) track changes of @seen. It
only ever changes in QAPISchemaMember.check_clash().
Hmmm, does your patch add such changes after this point?
>
> if self.variants:
> self.variants.check(schema, seen)
> self.variants.check_clash(self.info, seen)
>
> + for a in self.aliases:
> + a.check_clash(self.info, seen)
Covered by alias-name-conflict.
Is such a change to @seen hiding behind a.check_clash()?
> + self.check_path(a, list(a.source), members)
> +
> self.members = members # mark completed
>
> # Check that the members of this type do not cause duplicate JSON members,
> @@ -430,6 +437,68 @@ def check_clash(self, info, seen):
> for m in self.members:
> m.check_clash(info, seen)
>
> + # Deletes elements from path, so pass a copy if you still need them
Suggest "from @path", to make it immediately obvious what you're talking
about.
The side effect is a bit ugly. We can tidy it up later if we care.
> + def check_path(self, alias, path, members=None, local_aliases_seen=()):
Hmm, @local_aliases_seen is a tuple, whereas the @seen elsewhere are
dict mapping name to QAPISchemaEntity, so we can .describe().
Observation, not demand.
> + assert isinstance(path, list)
> +
> + if not path:
> + return
> + first = path.pop(0)
> +
> + for a in self.aliases:
> + if a.name == first:
> + if a in local_aliases_seen:
> + raise QAPISemError(
> + self.info,
> + "%s resolving to '%s' makes '%s' an alias for itself"
> + % (a.describe(self.info), a.source[0], a.source[0]))
Covered by alias-recursive.
Suggest to call the test case alias-loop.
The error message shows just one arc of the loop:
alias-recursive.json: In struct 'AliasStruct0':
alias-recursive.json:1: alias 'baz' resolving to 'bar' makes 'bar' an alias for itself
Showing the complete loop would be nice. Not a demand. Might require
turning @local_aliases_seen into a dict.
> +
> + path = a.source + path
> + return self.check_path(alias, path, members,
> + (*local_aliases_seen, a))
> +
> + if members is None:
> + assert self.members is not None
> + members = self.members
> + else:
> + assert isinstance(members, list)
> +
> + for m in members:
> + if m.name == first:
> + # Wildcard aliases can only accept object types in the whole
> + # path; for single-member aliases, the last element can be
> + # any type
> + need_obj = (alias.name is None) or path
> + if need_obj and not isinstance(m.type, QAPISchemaObjectType):
> + raise QAPISemError(
> + self.info,
> + "%s has non-object '%s' in its source path"
> + % (alias.describe(self.info), m.name))
Covered by alias-source-non-object-path and
alias-source-non-object-wildcard.
> + if alias.name is None and m.optional:
> + raise QAPISemError(
> + self.info,
> + "%s has optional object %s in its source path"
> + % (alias.describe(self.info), m.describe(self.info)))
Covered by alias-source-optional-wildcard-indirect and
alias-source-optional-wildcard.
> + if path:
> + m.type.check_path(alias, path)
> + return
> +
> + # It is sufficient that the path is valid in at least one variant
> + if self.variants:
> + for v in self.variants.variants:
> + try:
> + return v.type.check_path(alias, [first, *path])
> + except QAPISemError:
> + pass
Code smell: abuse of exception for perfectly non-exceptional control
flow. I'm willing to tolerate this for now.
> + raise QAPISemError(
> + self.info,
> + "%s has a source path that does not exist in any variant of %s"
> + % (alias.describe(self.info), self.describe()))
Covered by alias-source-inexistent-variants.
> +
> + raise QAPISemError(
> + self.info,
> + "%s has inexistent source" % alias.describe(self.info))
Covered by alias-source-inexistent.
pycodestyle-3 points out:
scripts/qapi/schema.py:441:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
Obvious fix: replace
return FOO.check_path(..)
by
FOO.check_path(..)
return
Now let's see what this function does. It detects the following errors:
1. Alias loop
2. Alias "dotting through" a non-object
3. Wildcard alias "dotting through" an optional object
4. Alias must resolve to something (common or variant member, possibly
nested)
Lovely! But how does it work?
The first loop takes care of 1. Looks like we try to resolve the alias,
then recurse, keeping track of things meanwhile so we can detect loops.
Isn't this more complicated than it needs to be?
Aliases can only resolve to the same or deeper nesting levels.
An alias that resolves to a deeper level cannot be part of a loop
(because we can't resolve to back to the alias's level).
So this should do:
local_aliases_seen = {}
for all aliases that resolve to the same level:
if local_aliases_seen[alias.name]:
# loop! we can retrace it using @local_aliases_seen if we
# care
raise ...
local_aliases_seen[alias.name] = alias
Or am I missing something?
Moving on.
To do 2. and 3., we try to resolve the alias, one element of source
after the other.
The first element can match either a common member, or a member of any
number of variants.
We first look for a common member, by searching @members for a match.
If there is one, we check it, then recurse to check the remaining
elements of source, and are done.
Else, we try variant members. We reduce the problem to "common member"
by recursing into the variant types. Neat! If this doesn't find any,
the alias doesn't resolve, taking care of 4.
Works. Searching @members is kind of brutish, though. We do have a map
from member name to QAPISchemaObjectTypeMember: @seen. To use it, we'd
have to fuse .check_path() into .check_clash(). Let's not worry about
that right now.
> +
> def connect_doc(self, doc=None):
> super().connect_doc(doc)
> doc = doc or self.doc
> @@ -474,7 +543,7 @@ def visit(self, visitor):
> super().visit(visitor)
> visitor.visit_object_type(
> self.name, self.info, self.ifcond, self.features,
> - self.base, self.local_members, self.variants)
> + self.base, self.local_members, self.variants, self.aliases)
> visitor.visit_object_type_flat(
> self.name, self.info, self.ifcond, self.features,
> self.members, self.variants)
> @@ -639,7 +708,7 @@ def check_clash(self, info, seen):
>
>
> class QAPISchemaMember:
> - """ Represents object members, enum members and features """
> + """ Represents object members, enum members, features and aliases """
> role = 'member'
>
> def __init__(self, name, info, ifcond=None):
> @@ -705,6 +774,30 @@ class QAPISchemaFeature(QAPISchemaMember):
> role = 'feature'
>
>
> +class QAPISchemaAlias(QAPISchemaMember):
> + role = 'alias'
I like this :)
> +
> + def __init__(self, name, info, source):
> + assert name is None or isinstance(name, str)
> + assert source
> + for member in source:
> + assert isinstance(member, str)
> +
> + super().__init__(name or '*', info)
> + self.name = name
> + self.source = source
> +
> + def check_clash(self, info, seen):
> + if self.name:
> + super().check_clash(info, seen)
> +
> + def describe(self, info):
> + if self.name:
> + return super().describe(info)
> + else:
> + return "wildcard alias"
pycodestyle-3 gripes:
scripts/qapi/schema.py:795:8: R1705: Unnecessary "else" after "return" (no-else-return)
> +
> +
> class QAPISchemaObjectTypeMember(QAPISchemaMember):
> def __init__(self, name, info, typ, optional, ifcond=None, features=None):
> super().__init__(name, info, ifcond)
> @@ -971,6 +1064,12 @@ def _make_features(self, features, info):
> return [QAPISchemaFeature(f['name'], info, f.get('if'))
> for f in features]
>
> + def _make_aliases(self, aliases, info):
> + if aliases is None:
> + return []
> + return [QAPISchemaAlias(a.get('name'), info, a['source'])
> + for a in aliases]
> +
> def _make_enum_members(self, values, info):
> return [QAPISchemaEnumMember(v['name'], info, v.get('if'))
> for v in values]
> @@ -1045,11 +1144,12 @@ def _def_struct_type(self, expr, info, doc):
> base = expr.get('base')
> data = expr['data']
> ifcond = expr.get('if')
> + aliases = self._make_aliases(expr.get('aliases'), info)
> features = self._make_features(expr.get('features'), info)
> self._def_entity(QAPISchemaObjectType(
> name, info, doc, ifcond, features, base,
> self._make_members(data, info),
> - None))
> + None, aliases))
>
> def _make_variant(self, case, typ, ifcond, info):
> return QAPISchemaVariant(case, info, typ, ifcond)
> @@ -1068,6 +1168,7 @@ def _def_union_type(self, expr, info, doc):
> data = expr['data']
> base = expr.get('base')
> ifcond = expr.get('if')
> + aliases = self._make_aliases(expr.get('aliases'), info)
> features = self._make_features(expr.get('features'), info)
> tag_name = expr.get('discriminator')
> tag_member = None
> @@ -1092,7 +1193,8 @@ def _def_union_type(self, expr, info, doc):
> QAPISchemaObjectType(name, info, doc, ifcond, features,
> base, members,
> QAPISchemaVariants(
> - tag_name, info, tag_member, variants)))
> + tag_name, info, tag_member, variants),
> + aliases))
>
> def _def_alternate_type(self, expr, info, doc):
> name = expr['alternate']
> diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
> index 20d572a23a..3bc451baa9 100644
> --- a/scripts/qapi/types.py
> +++ b/scripts/qapi/types.py
> @@ -25,6 +25,7 @@
> from .gen import QAPISchemaModularCVisitor, ifcontext
> from .schema import (
> QAPISchema,
> + QAPISchemaAlias,
> QAPISchemaEnumMember,
> QAPISchemaFeature,
> QAPISchemaObjectType,
> @@ -332,7 +333,8 @@ def visit_object_type(self,
> features: List[QAPISchemaFeature],
> base: Optional[QAPISchemaObjectType],
> members: List[QAPISchemaObjectTypeMember],
> - variants: Optional[QAPISchemaVariants]) -> None:
> + variants: Optional[QAPISchemaVariants],
> + aliases: List[QAPISchemaAlias]) -> None:
> # Nothing to do for the special empty builtin
> if name == 'q_empty':
> return
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index 9e96f3c566..0aa0764755 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -26,6 +26,7 @@
> from .gen import QAPISchemaModularCVisitor, ifcontext
> from .schema import (
> QAPISchema,
> + QAPISchemaAlias,
> QAPISchemaEnumMember,
> QAPISchemaEnumType,
> QAPISchemaFeature,
> @@ -60,7 +61,8 @@ def gen_visit_members_decl(name: str) -> str:
> def gen_visit_object_members(name: str,
> base: Optional[QAPISchemaObjectType],
> members: List[QAPISchemaObjectTypeMember],
> - variants: Optional[QAPISchemaVariants]) -> str:
> + variants: Optional[QAPISchemaVariants],
> + aliases: List[QAPISchemaAlias]) -> str:
> ret = mcgen('''
>
> bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
> @@ -68,6 +70,24 @@ def gen_visit_object_members(name: str,
> ''',
> c_name=c_name(name))
>
> + if aliases:
> + ret += mcgen('''
> + visit_start_alias_scope(v);
> +''')
> +
> + for a in aliases:
> + if a.name:
> + name = '"%s"' % a.name
> + else:
> + name = "NULL"
> +
> + source = ", ".join('"%s"' % x for x in a.source)
@x is a poor choice for a loop control variable. @elt?
> +
> + ret += mcgen('''
> + visit_define_alias(v, %(name)s, (const char * []) { %(source)s, NULL });
> +''',
> + name=name, source=source)
> +
> if base:
> ret += mcgen('''
> if (!visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, errp)) {
> @@ -148,6 +168,11 @@ def gen_visit_object_members(name: str,
> }
> ''')
>
> + if aliases:
> + ret += mcgen('''
> + visit_end_alias_scope(v);
> +''')
> +
> ret += mcgen('''
> return true;
> }
> @@ -376,14 +401,15 @@ def visit_object_type(self,
> features: List[QAPISchemaFeature],
> base: Optional[QAPISchemaObjectType],
> members: List[QAPISchemaObjectTypeMember],
> - variants: Optional[QAPISchemaVariants]) -> None:
> + variants: Optional[QAPISchemaVariants],
> + aliases: List[QAPISchemaAlias]) -> None:
> # Nothing to do for the special empty builtin
> if name == 'q_empty':
> return
> with ifcontext(ifcond, self._genh, self._genc):
> self._genh.add(gen_visit_members_decl(name))
> - self._genc.add(gen_visit_object_members(name, base,
> - members, variants))
> + self._genc.add(gen_visit_object_members(
> + name, base, members, variants, aliases))
> # TODO Worth changing the visitor signature, so we could
> # directly use rather than repeat type.is_implicit()?
> if not name.startswith('q_'):
> diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
> index f1c4deb9a5..376630901b 100755
> --- a/tests/qapi-schema/test-qapi.py
> +++ b/tests/qapi-schema/test-qapi.py
> @@ -47,7 +47,7 @@ def visit_array_type(self, name, info, ifcond, element_type):
> self._print_if(ifcond)
>
> def visit_object_type(self, name, info, ifcond, features,
> - base, members, variants):
> + base, members, variants, aliases):
> print('object %s' % name)
> if base:
> print(' base %s' % base.name)
> @@ -56,6 +56,11 @@ def visit_object_type(self, name, info, ifcond, features,
> % (m.name, m.type.name, m.optional))
> self._print_if(m.ifcond, 8)
> self._print_features(m.features, indent=8)
> + for a in aliases:
> + if a.name:
> + print(' alias %s -> %s' % (a.name, '.'.join(a.source)))
> + else:
> + print(' alias * -> %s.*' % '.'.join(a.source))
> self._print_variants(variants)
> self._print_if(ifcond)
> self._print_features(features)
> diff --git a/tests/qapi-schema/double-type.err b/tests/qapi-schema/double-type.err
> index 576e716197..c382e61d88 100644
> --- a/tests/qapi-schema/double-type.err
> +++ b/tests/qapi-schema/double-type.err
> @@ -1,3 +1,3 @@
> double-type.json: In struct 'Bar':
> double-type.json:2: struct has unknown key 'command'
> -Valid keys are 'base', 'data', 'features', 'if', 'struct'.
> +Valid keys are 'aliases', 'base', 'data', 'features', 'if', 'struct'.
> diff --git a/tests/qapi-schema/unknown-expr-key.err b/tests/qapi-schema/unknown-expr-key.err
> index f2538e3ce7..354916968f 100644
> --- a/tests/qapi-schema/unknown-expr-key.err
> +++ b/tests/qapi-schema/unknown-expr-key.err
> @@ -1,3 +1,3 @@
> unknown-expr-key.json: In struct 'Bar':
> unknown-expr-key.json:2: struct has unknown keys 'bogus', 'phony'
> -Valid keys are 'base', 'data', 'features', 'if', 'struct'.
> +Valid keys are 'aliases', 'base', 'data', 'features', 'if', 'struct'.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-08-12 16:11 ` [PATCH v3 6/6] tests/qapi-schema: Test cases " Kevin Wolf
@ 2021-09-06 15:28 ` Markus Armbruster
2021-09-10 15:04 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-09-06 15:28 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel, armbru
Kevin Wolf <kwolf@redhat.com> writes:
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
[...]
> diff --git a/tests/unit/test-qobject-input-visitor.c b/tests/unit/test-qobject-input-visitor.c
> index e41b91a2a6..f2891b6f5d 100644
> --- a/tests/unit/test-qobject-input-visitor.c
> +++ b/tests/unit/test-qobject-input-visitor.c
> @@ -952,6 +952,214 @@ static void test_visitor_in_list_union_number(TestInputVisitorData *data,
> g_string_free(gstr_list, true);
> }
>
> +static void test_visitor_in_alias_struct_local(TestInputVisitorData *data,
> + const void *unused)
> +{
> + AliasStruct1 *tmp = NULL;
> + Error *err = NULL;
> + Visitor *v;
> +
Context: the schema makes 'bar' an alias for 'foo'.
> + /* Can still specify the real member name with alias support */
> + v = visitor_input_test_init(data, "{ 'foo': 42 }");
> + visit_type_AliasStruct1(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->foo, ==, 42);
> + qapi_free_AliasStruct1(tmp);
> +
> + /* The alias is a working alternative */
> + v = visitor_input_test_init(data, "{ 'bar': 42 }");
> + visit_type_AliasStruct1(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->foo, ==, 42);
> + qapi_free_AliasStruct1(tmp);
> +
> + /* But you can't use both at the same time */
> + v = visitor_input_test_init(data, "{ 'foo': 5, 'bar': 42 }");
> + visit_type_AliasStruct1(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
I double-checked this reports "Value for parameter foo was already given
through an alias", as it should.
Pointing to what exactly is giving values to foo already would be nice.
In this case, 'foo' is obvious, but 'bar' is not. This is not a demand.
> +}
> +
> +static void test_visitor_in_alias_struct_nested(TestInputVisitorData *data,
> + const void *unused)
> +{
> + AliasStruct2 *tmp = NULL;
> + Error *err = NULL;
> + Visitor *v;
> +
Context: the schema makes 'bar' and 'nested.bar' aliases for
'nested.foo'.
> + /* Can still specify the real member names with alias support */
> + v = visitor_input_test_init(data, "{ 'nested': { 'foo': 42 } }");
> + visit_type_AliasStruct2(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->nested->foo, ==, 42);
> + qapi_free_AliasStruct2(tmp);
> +
> + /* The inner alias is a working alternative */
> + v = visitor_input_test_init(data, "{ 'nested': { 'bar': 42 } }");
> + visit_type_AliasStruct2(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->nested->foo, ==, 42);
> + qapi_free_AliasStruct2(tmp);
> +
> + /* So is the outer alias */
> + v = visitor_input_test_init(data, "{ 'bar': 42 }");
> + visit_type_AliasStruct2(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->nested->foo, ==, 42);
> + qapi_free_AliasStruct2(tmp);
> +
> + /* You can't use more than one option at the same time */
> + v = visitor_input_test_init(data, "{ 'bar': 5, 'nested': { 'foo': 42 } }");
> + visit_type_AliasStruct2(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Value for parameter nested.foo was already given through an alias".
Good.
> +
> + v = visitor_input_test_init(data, "{ 'bar': 5, 'nested': { 'bar': 42 } }");
> + visit_type_AliasStruct2(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
Likewise.
> +
> + v = visitor_input_test_init(data, "{ 'nested': { 'foo': 42, 'bar': 42 } }");
> + visit_type_AliasStruct2(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
Likewise.
> +
> + v = visitor_input_test_init(data, "{ 'bar': 5, "
> + " 'nested': { 'foo': 42, 'bar': 42 } }");
> + visit_type_AliasStruct2(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
Likewise.
In the second of these four cases, none of the things giving values to
nested.foo is obvious. Still not a demand.
> +}
> +
> +static void test_visitor_in_alias_wildcard(TestInputVisitorData *data,
> + const void *unused)
> +{
> + AliasStruct3 *tmp = NULL;
> + Error *err = NULL;
> + Visitor *v;
> +
Context: the schema makes 'foo', 'bar', and 'nested.bar' aliases for
'nested.foo', using a wildcard alias for the former two.
> + /* Can still specify the real member names with alias support */
> + v = visitor_input_test_init(data, "{ 'nested': { 'foo': 42 } }");
> + visit_type_AliasStruct3(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->nested->foo, ==, 42);
> + qapi_free_AliasStruct3(tmp);
> +
> + /* The wildcard alias makes it work on the top level */
> + v = visitor_input_test_init(data, "{ 'foo': 42 }");
> + visit_type_AliasStruct3(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->nested->foo, ==, 42);
> + qapi_free_AliasStruct3(tmp);
> +
> + /* It makes the inner alias available, too */
> + v = visitor_input_test_init(data, "{ 'bar': 42 }");
> + visit_type_AliasStruct3(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->nested->foo, ==, 42);
> + qapi_free_AliasStruct3(tmp);
> +
> + /* You can't use more than one option at the same time */
> + v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'foo': 42 } }");
> + visit_type_AliasStruct3(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Parameter 'foo' is unexpected". Say what? It *is* expected, it just
clashes with 'nested.foo'.
I figure this is what happens:
* visit_type_AliasStruct3()
- visit_start_struct()
- visit_type_AliasStruct3_members()
• visit_type_AliasStruct1() for member @nested.
This consumes consumes input nested.foo.
- visit_check_struct()
Error: input foo has not been consumed.
Any ideas on how to report this error more clearly?
> +
> + v = visitor_input_test_init(data, "{ 'bar': 42, 'nested': { 'foo': 42 } }");
> + visit_type_AliasStruct3(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Value for parameter nested.foo was already given through an alias".
Good (but I have no idea how we avoid the bad error reporting in this
case).
> +
> + v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'bar': 42 } }");
> + visit_type_AliasStruct3(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Parameter 'foo' is unexpected"
> +
> + v = visitor_input_test_init(data, "{ 'bar': 42, 'nested': { 'bar': 42 } }");
> + visit_type_AliasStruct3(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Parameter 'bar' is unexpected"
> +
> + v = visitor_input_test_init(data, "{ 'foo': 42, 'bar': 42 }");
> + visit_type_AliasStruct3(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Parameter 'foo' is unexpected"
> +}
> +
> +static void test_visitor_in_alias_flat_union(TestInputVisitorData *data,
> + const void *unused)
> +{
> + AliasFlatUnion *tmp = NULL;
> + Error *err = NULL;
> + Visitor *v;
> +
Context: the schema makes 'variant' an alias for 'tag', and 'bar' an
alias for 'foo'.
> + /* Can still specify the real member name with alias support */
> + v = visitor_input_test_init(data, "{ 'tag': 'drei' }");
> + visit_type_AliasFlatUnion(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->tag, ==, FEATURE_ENUM1_DREI);
> + qapi_free_AliasFlatUnion(tmp);
> +
> + /* Use alias for a base member (the discriminator even) */
> + v = visitor_input_test_init(data, "{ 'variant': 'zwei' }");
> + visit_type_AliasFlatUnion(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->tag, ==, FEATURE_ENUM1_ZWEI);
> + qapi_free_AliasFlatUnion(tmp);
> +
> + /* Use alias for a variant member */
> + v = visitor_input_test_init(data, "{ 'tag': 'eins', 'bar': 42 }");
> + visit_type_AliasFlatUnion(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->tag, ==, FEATURE_ENUM1_EINS);
> + g_assert_cmpint(tmp->u.eins.foo, ==, 42);
> + qapi_free_AliasFlatUnion(tmp);
> +
> + /* Both together */
> + v = visitor_input_test_init(data, "{ 'variant': 'eins', 'bar': 42 }");
> + visit_type_AliasFlatUnion(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->tag, ==, FEATURE_ENUM1_EINS);
> + g_assert_cmpint(tmp->u.eins.foo, ==, 42);
> + qapi_free_AliasFlatUnion(tmp);
> +
> + /* You can't use more than one option at the same time for each alias */
> + v = visitor_input_test_init(data, "{ 'variant': 'zwei', 'tag': 'drei' }");
> + visit_type_AliasFlatUnion(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Value for parameter tag was already given through an alias". Good.
> +
> + v = visitor_input_test_init(data, "{ 'tag': 'eins', 'foo': 6, 'bar': 9 }");
> + visit_type_AliasFlatUnion(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Value for parameter foo was already given through an alias". Good,
except I'm getting a feeling "already" may be confusing. It's "already"
only in the sense that we already got the value via alias, which is an
implementation detail. It may or may not be given already in the
input. Here it's not: 'bar' follows 'foo'.
What about "is also given through an alias"?
> +}
> +
> +static void test_visitor_in_alias_simple_union(TestInputVisitorData *data,
> + const void *unused)
> +{
> + AliasSimpleUnion *tmp = NULL;
> + Error *err = NULL;
> + Visitor *v;
> +
Context: the schema makes 'foo' and 'bar' aliases for 'data.foo' and
'data.bar' (using wildcard alias), 'tag' an alias for 'type', and
'data.bar' an alias for 'data.foo'.
> + /* Can still specify the real member name with alias support */
> + v = visitor_input_test_init(data, "{ 'type': 'eins', "
> + " 'data': { 'foo': 42 } }");
> + visit_type_AliasSimpleUnion(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->type, ==, ALIAS_SIMPLE_UNION_KIND_EINS);
> + g_assert_cmpint(tmp->u.eins.data->foo, ==, 42);
> + qapi_free_AliasSimpleUnion(tmp);
> +
> + /* 'type' can be aliased */
> + v = visitor_input_test_init(data, "{ 'tag': 'eins', "
> + " 'data': { 'foo': 42 } }");
> + visit_type_AliasSimpleUnion(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->type, ==, ALIAS_SIMPLE_UNION_KIND_EINS);
> + g_assert_cmpint(tmp->u.eins.data->foo, ==, 42);
> + qapi_free_AliasSimpleUnion(tmp);
> +
> + /* The wildcard alias makes it work on the top level */
> + v = visitor_input_test_init(data, "{ 'type': 'eins', 'foo': 42 }");
> + visit_type_AliasSimpleUnion(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->type, ==, ALIAS_SIMPLE_UNION_KIND_EINS);
> + g_assert_cmpint(tmp->u.eins.data->foo, ==, 42);
> + qapi_free_AliasSimpleUnion(tmp);
> +
> + /* It makes the inner alias available, too */
> + v = visitor_input_test_init(data, "{ 'type': 'eins', 'bar': 42 }");
> + visit_type_AliasSimpleUnion(v, NULL, &tmp, &error_abort);
> + g_assert_cmpint(tmp->type, ==, ALIAS_SIMPLE_UNION_KIND_EINS);
> + g_assert_cmpint(tmp->u.eins.data->foo, ==, 42);
> + qapi_free_AliasSimpleUnion(tmp);
> +
> + /* You can't use more than one option at the same time for each alias */
> + v = visitor_input_test_init(data, "{ 'type': 'eins', 'tag': 'eins' }");
> + visit_type_AliasSimpleUnion(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Value for parameter type was already given through an alias". Good,
except "parameter type" is confusing. Make it "parameter 'type'".
> +
> + v = visitor_input_test_init(data, "{ 'type': 'eins', "
> + " 'bar': 123, "
> + " 'data': { 'foo': 312 } }");
> + visit_type_AliasSimpleUnion(v, NULL, &tmp, &err);
> + error_free_or_abort(&err);
"Value for parameter data.foo was already given through an alias".
Good.
> +}
> +
> static void input_visitor_test_add(const char *testpath,
> const void *user_data,
> void (*test_func)(TestInputVisitorData *data,
> @@ -1350,6 +1558,16 @@ int main(int argc, char **argv)
> NULL, test_visitor_in_list_union_string);
> input_visitor_test_add("/visitor/input/list_union/number",
> NULL, test_visitor_in_list_union_number);
> + input_visitor_test_add("/visitor/input/alias/struct-local",
> + NULL, test_visitor_in_alias_struct_local);
> + input_visitor_test_add("/visitor/input/alias/struct-nested",
> + NULL, test_visitor_in_alias_struct_nested);
> + input_visitor_test_add("/visitor/input/alias/wildcard",
> + NULL, test_visitor_in_alias_wildcard);
> + input_visitor_test_add("/visitor/input/alias/flat-union",
> + NULL, test_visitor_in_alias_flat_union);
> + input_visitor_test_add("/visitor/input/alias/simple-union",
> + NULL, test_visitor_in_alias_simple_union);
> input_visitor_test_add("/visitor/input/fail/struct",
> NULL, test_visitor_in_fail_struct);
> input_visitor_test_add("/visitor/input/fail/struct-nested",
[Negative tests snipped, I checked them in review of PATCH 5, they're
fine]
> diff --git a/tests/qapi-schema/meson.build b/tests/qapi-schema/meson.build
> index b8de58116a..f937de1c35 100644
> --- a/tests/qapi-schema/meson.build
> +++ b/tests/qapi-schema/meson.build
> @@ -3,6 +3,22 @@ test_env.set('PYTHONPATH', meson.source_root() / 'scripts')
> test_env.set('PYTHONIOENCODING', 'utf-8')
>
> schemas = [
> + 'alias-bad-type.json',
> + 'aliases-bad-type.json',
> + 'alias-missing-source.json',
> + 'alias-name-bad-type.json',
> + 'alias-name-conflict.json',
> + 'alias-recursive.json',
> + 'alias-source-bad-type.json',
> + 'alias-source-elem-bad-type.json',
> + 'alias-source-empty.json',
> + 'alias-source-inexistent.json',
> + 'alias-source-inexistent-variants.json',
> + 'alias-source-non-object-path.json',
> + 'alias-source-non-object-wildcard.json',
> + 'alias-source-optional-wildcard.json',
> + 'alias-source-optional-wildcard-indirect.json',
> + 'alias-unknown-key.json',
> 'alternate-any.json',
> 'alternate-array.json',
> 'alternate-base.json',
> diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
> index 84b9d41f15..c5e81a883c 100644
> --- a/tests/qapi-schema/qapi-schema-test.json
> +++ b/tests/qapi-schema/qapi-schema-test.json
> @@ -336,3 +336,29 @@
>
> { 'event': 'TEST_EVENT_FEATURES1',
> 'features': [ 'deprecated' ] }
> +
> +# test 'aliases'
> +
> +{ 'struct': 'AliasStruct0',
> + 'data': { 'foo': 'int' },
> + 'aliases': [] }
> +{ 'struct': 'AliasStruct1',
> + 'data': { 'foo': 'int' },
> + 'aliases': [ { 'name': 'bar', 'source': ['foo'] } ] }
> +{ 'struct': 'AliasStruct2',
> + 'data': { 'nested': 'AliasStruct1' },
> + 'aliases': [ { 'name': 'bar', 'source': ['nested', 'foo'] } ] }
> +{ 'struct': 'AliasStruct3',
> + 'data': { 'nested': 'AliasStruct1' },
> + 'aliases': [ { 'source': ['nested'] } ] }
> +
> +{ 'union': 'AliasFlatUnion',
> + 'base': { 'tag': 'FeatureEnum1' },
> + 'discriminator': 'tag',
> + 'data': { 'eins': 'FeatureStruct1' },
> + 'aliases': [ { 'name': 'variant', 'source': ['tag'] },
> + { 'name': 'bar', 'source': ['foo'] } ] }
> +{ 'union': 'AliasSimpleUnion',
> + 'data': { 'eins': 'AliasStruct1' },
> + 'aliases': [ { 'source': ['data'] },
> + { 'name': 'tag', 'source': ['type'] } ] }
> diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
> index e0b8a5f0b6..f6b8a98b7c 100644
> --- a/tests/qapi-schema/qapi-schema-test.out
> +++ b/tests/qapi-schema/qapi-schema-test.out
> @@ -445,6 +445,37 @@ event TEST_EVENT_FEATURES0 FeatureStruct1
> event TEST_EVENT_FEATURES1 None
> boxed=False
> feature deprecated
> +object AliasStruct0
> + member foo: int optional=False
> +object AliasStruct1
> + member foo: int optional=False
> + alias bar -> foo
> +object AliasStruct2
> + member nested: AliasStruct1 optional=False
> + alias bar -> nested.foo
> +object AliasStruct3
> + member nested: AliasStruct1 optional=False
> + alias * -> nested.*
> +object q_obj_AliasFlatUnion-base
> + member tag: FeatureEnum1 optional=False
> +object AliasFlatUnion
> + base q_obj_AliasFlatUnion-base
> + alias variant -> tag
> + alias bar -> foo
> + tag tag
> + case eins: FeatureStruct1
> + case zwei: q_empty
> + case drei: q_empty
> +object q_obj_AliasStruct1-wrapper
> + member data: AliasStruct1 optional=False
> +enum AliasSimpleUnionKind
> + member eins
> +object AliasSimpleUnion
> + member type: AliasSimpleUnionKind optional=False
> + alias * -> data.*
> + alias tag -> type
> + tag type
> + case eins: q_obj_AliasStruct1-wrapper
> module include/sub-module.json
> include sub-sub-module.json
> object SecondArrayRef
Positive tests look good to me, except they neglect to use any of the
types using the alias features in QMP. I think we need something like
the appended incremental patch.
Oh, with that, backing out the hunk
- members = seen.values()
+ members = list(seen.values())
as described in review of PATCH 5 actually fails "make check"!
The generated test-qapi-introspect.c doesn't show aliases. Here's
AliasStruct1:
/* "63" = AliasStruct1 */
QLIT_QDICT(((QLitDictEntry[]) {
{ "members", QLIT_QLIST(((QLitObject[]) {
QLIT_QDICT(((QLitDictEntry[]) {
{ "name", QLIT_QSTR("foo"), },
{ "type", QLIT_QSTR("int"), },
{}
})),
{}
})), },
{ "meta-type", QLIT_QSTR("object"), },
{ "name", QLIT_QSTR("63"), },
{}
})),
Not a peep about member 'bar'.
We need to address this for use case "compatible schema evolution", so
that management applications can detect presence of the new interface.
Actual use of aliases for this purpose requires coordination with
libvirt developers, of course.
How could introspection show aliases? We can't simply add an entry for
"bar" to "members", because that would show two mandatory members "foo"
and "bar", which is wrong.
If we add "aliases" next to "members", aliases remain invisible for
older management applications. I don't have better ideas.
Let's have a closer look at "compatible schema evolution". We want to
move / rename a member, and use aliases to support both the new and the
old name for compatibility. We want to be able to deprecate the old
name.
Example 1: move 'foo' to 'bar'
Two ways:
1. Replace member 'foo' by 'bar', then add alias 'foo'
Old management applications can't see the alias. To them, it
looks like 'foo' vanished without replacement, which is a
compatibility break. May well cause trouble.
2. Add alias 'bar'
Old management applications can't see the alias. If we deprecate
'foo', they see that. Unlikely to cause trouble, I think. If we
remove 'foo', compatibility break, but that's intentional.
Always use way 2. Documentation should spell that out.
Example 2: move 'nested.foo' to 'bar'
Due to the way aliases work, we need to make 'bar' the alias, like
'aliases': [ { 'name': 'bar', 'source': ['nested', 'foo'] } ] }
This is way 2. again. Fine.
Example 1: move 'bar' to 'nested.foo'
Due to the way aliases work, we need to replace 'bar' by
'nested.foo', then add alias 'bar'.
Here, we can only use problematic way 1. Better ideas than
"document the limitation?"
diff --git a/tests/unit/test-qmp-cmds.c b/tests/unit/test-qmp-cmds.c
index 1b0b7d99df..907468b157 100644
--- a/tests/unit/test-qmp-cmds.c
+++ b/tests/unit/test-qmp-cmds.c
@@ -76,6 +76,16 @@ void qmp_test_command_cond_features3(Error **errp)
{
}
+void qmp_test_aliases0(bool has_as0, AliasStruct0 *as0,
+ bool has_as1, AliasStruct1 *as1,
+ bool has_as2, AliasStruct2 *as2,
+ bool has_as3, AliasStruct3 *as3,
+ bool has_afu, AliasFlatUnion *afu,
+ bool has_asu, AliasSimpleUnion *asu,
+ Error **errp)
+{
+}
+
UserDefTwo *qmp_user_def_cmd2(UserDefOne *ud1a,
bool has_udb1, UserDefOne *ud1b,
Error **errp)
diff --git a/tests/qapi-schema/qapi-schema-test.json b/tests/qapi-schema/qapi-schema-test.json
index c5e81a883c..4d3a5039b4 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -362,3 +362,11 @@
'data': { 'eins': 'AliasStruct1' },
'aliases': [ { 'source': ['data'] },
{ 'name': 'tag', 'source': ['type'] } ] }
+
+{ 'command': 'test-aliases0',
+ 'data': { '*as0': 'AliasStruct0',
+ '*as1': 'AliasStruct1',
+ '*as2': 'AliasStruct2',
+ '*as3': 'AliasStruct3',
+ '*afu': 'AliasFlatUnion',
+ '*asu': 'AliasSimpleUnion' } }
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH v3 0/6] qapi: Add support for aliases
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
` (6 preceding siblings ...)
2021-08-24 9:36 ` [PATCH v3 0/6] qapi: Add support " Markus Armbruster
@ 2021-09-06 15:32 ` Markus Armbruster
7 siblings, 0 replies; 43+ messages in thread
From: Markus Armbruster @ 2021-09-06 15:32 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> This series introduces alias definitions for QAPI object types (structs
> and unions).
>
> This allows using the same QAPI type and visitor even when the syntax
> has some variations between different external interfaces such as QMP
> and the command line.
>
> It also provides a new tool for evolving the schema while maintaining
> backwards compatibility (possibly during a deprecation period).
>
> The first user is intended to be a QAPIfied -chardev command line
> option, for which I'll send a separate series. A git tag is available
> that contains both this series and the chardev changes that make use of
> it:
>
> https://repo.or.cz/qemu/kevin.git qapi-alias-chardev-v3
Review complete. Let's discuss my findings, decide what we'd rather
improve on top, then see whether the remainder needs a respin.
> v3:
> - Mention the new functions in the big comment in visitor.h. However,
> since the comment is about users of the visitor rather than the
> generated code, it seems like to wrong place to go into details.
> - Updated commit message for patch 3 ('Simplify full_name_nth() ...')
> - Patch 4 ('qapi: Apply aliases in qobject-input-visitor'):
> - Multiple matching wildcard aliases are considered conflicting now
> - Improved comments for several functions
> - Renamed bool *implicit_object into *is_alias_prefix, which
> describes better what it is rather than what it is used for
> - Simplified alias_present() into input_present()
> - Fixed potential use of wrong StackObject in error message
> - Patch 5 ('qapi: Add support for aliases'):
> - Made QAPISchemaAlias a QAPISchemaMember
> - Check validity of alias source paths (must exist in at least one
> variant, no optional objects in the path of a wildcard alias, no
> alias loops)
I love this one, thanks!
> - Many new tests cases, both positive and negative, including unit tests
> of the generated visit functions
Tests look good now.
> - Coding style changes
> - Rebased documentation (.txt -> .rst conversion in master)
[...]
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 4/6] qapi: Apply aliases in qobject-input-visitor
2021-09-06 15:16 ` Markus Armbruster
@ 2021-09-08 13:01 ` Kevin Wolf
2021-09-14 6:58 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-09-08 13:01 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 06.09.2021 um 17:16 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
>
> > When looking for an object in a struct in the external representation,
> > check not only the currently visited struct, but also whether an alias
> > in the current StackObject matches and try to fetch the value from the
> > alias then. Providing two values for the same object through different
> > aliases is an error.
> >
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> > qapi/qobject-input-visitor.c | 227 +++++++++++++++++++++++++++++++++--
> > 1 file changed, 218 insertions(+), 9 deletions(-)
> >
> > diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
> > index 16a75442ff..6193df28a5 100644
> > --- a/qapi/qobject-input-visitor.c
> > +++ b/qapi/qobject-input-visitor.c
> > @@ -97,6 +97,8 @@ struct QObjectInputVisitor {
> > QObject *root;
> > bool keyval; /* Assume @root made with keyval_parse() */
> >
> > + QDict *empty_qdict; /* Used for implicit objects */
>
> Would
>
> /* For visiting objects where all members are from aliases */
>
> be clearer?
I know what I meant with "implicit objects", so not to me, but if you
think it's clearer this way, then it probably is.
> > +
> > /* Stack of objects being visited (all entries will be either
> > * QDict or QList). */
> > QSLIST_HEAD(, StackObject) stack;
> > @@ -169,9 +171,190 @@ static const char *full_name(QObjectInputVisitor *qiv, const char *name)
> > return full_name_so(qiv, name, false, tos);
> > }
> >
> > +static bool find_object_member(QObjectInputVisitor *qiv,
> > + StackObject **so, const char **name,
> > + bool *is_alias_prefix, Error **errp);
>
> According to the function's contract below, three cases:
>
> * Input present: update *so, *name, return true.
>
> * Input absent: zap *so, *name, set *is_alias_prefix, return false.
>
> * Error: set *errp, leave *is_alias_prefix undefined, return false.
>
> > +
> > +/*
> > + * Check whether the member @name in @so, or an alias for it, is
> > + * present in the input and can be used to obtain the value.
> > + */
> > +static bool input_present(QObjectInputVisitor *qiv, StackObject *so,
> > + const char *name)
> > +{
> > + /*
> > + * Check whether the alias member is present in the input
> > + * (possibly recursively because aliases are transitive).
> > + * The QAPI generator makes sure that alises cannot form loops, so
> > + * the recursion guaranteed to terminate.
> > + */
> > + if (!find_object_member(qiv, &so, &name, NULL, NULL)) {
>
> * Input absent: zap @so and @name.
>
> * Error: don't zap.
>
> Since @so and @name aren't used anymore, the difference doesn't matter.
> Okay.
>
> > + return false;
> > + }
> > +
> > + /*
> > + * Every source can be used only once. If a value in the input
> > + * would end up being used twice through aliases, we'll fail the
> > + * second access.
> > + */
> > + if (!g_hash_table_contains(so->h, name)) {
> > + return false;
> > + }
> > +
> > + return true;
> > +}
> > +
> > +/*
> > + * Check whether the member @name in the object visited by @so can be
> > + * specified in the input by using the alias described by @a (which
> > + * must be an alias contained in so->aliases).
> > + *
> > + * If @name is only a prefix of the alias source, but doesn't match
> > + * immediately, false is returned and *is_alias_prefix is set to true
> > + * if it is non-NULL. In all other cases, *is_alias_prefix is left
> > + * unchanged.
> > + */
> > +static bool alias_source_matches(QObjectInputVisitor *qiv,
> > + StackObject *so, InputVisitorAlias *a,
> > + const char *name, bool *is_alias_prefix)
> > +{
> > + if (a->src[0] == NULL) {
> > + assert(a->name == NULL);
> > + return true;
> > + }
> > +
> > + if (!strcmp(a->src[0], name)) {
> > + if (a->name && a->src[1] == NULL) {
> > + /*
> > + * We're matching an exact member, the source for this alias is
> > + * immediately in @so.
> > + */
> > + return true;
> > + } else if (is_alias_prefix) {
> > + /*
> > + * We're only looking at a prefix of the source path for the alias.
> > + * If the input contains no object of the requested name, we will
> > + * implicitly create an empty one so that the alias can still be
> > + * used.
> > + *
> > + * We want to create the implicit object only if the alias is
> > + * actually used, but we can't tell here for wildcard aliases (only
> > + * a later visitor call will determine this). This means that
> > + * wildcard aliases must never have optional keys in their source
> > + * path. The QAPI generator checks this condition.
> > + */
>
> Double-checking: this actually ensures that we only ever create the
> implicit object when it will not remain empty. Correct?
For wildcard aliases, we still can't know which keys will be visited
later. Checking that we don't have optional keys only avoids the
confusion between absent and present, but empty objects that you would
get from the implicit objects. So it means that creating an implicit
object is never wrong, either the nested object can be visited (which
means we needed the implicit object) or it errors out.
> > + if (!a->name || input_present(qiv, a->alias_so, a->name)) {
> > + *is_alias_prefix = true;
> > + }
> > + }
> > + }
> > +
> > + return false;
> > +}
> > +
> > +/*
> > + * Find the place in the input where the value for the object member
> > + * @name in @so is specified, considering applicable aliases.
> > + *
> > + * If a value could be found, true is returned and @so and @name are
> > + * updated to identify the key name and StackObject where the value
> > + * can be found in the input. (This is either unchanged or the
> > + * alias_so/name of an alias.) The value of @is_alias_prefix on
> > + * return is undefined in this case.
> > + *
> > + * If no value could be found in the input, false is returned and @so
> > + * and @name are set to NULL. This is not an error and @errp remains
> > + * unchanged. If @is_alias_prefix is non-NULL, it is set to true if
> > + * the given name is a prefix of the source path of an alias for which
> > + * a value may be present in the input. It is set to false otherwise.
> > + *
> > + * If an error occurs (e.g. two values are specified for the member
> > + * through different names), false is returned and @errp is set. The
> > + * value of @is_alias_prefix on return is undefined in this case.
> > + */
> > +static bool find_object_member(QObjectInputVisitor *qiv,
> > + StackObject **so, const char **name,
> > + bool *is_alias_prefix, Error **errp)
> > +{
> > + QDict *qdict = qobject_to(QDict, (*so)->obj);
> > + const char *found_name = NULL;
> > + StackObject *found_so = NULL;
> > + bool found_is_wildcard = false;
> > + InputVisitorAlias *a;
> > +
> > + if (is_alias_prefix) {
> > + *is_alias_prefix = false;
> > + }
> > +
> > + /* Directly present in the container */
> > + if (qdict_haskey(qdict, *name)) {
> > + found_name = *name;
> > + found_so = *so;
> > + }
> > +
> > + /*
> > + * Find aliases whose source path matches @name in this StackObject. We can
> > + * then get the value with the key a->name from a->alias_so.
> > + */
> > + QSLIST_FOREACH(a, &(*so)->aliases, next) {
> > + if (a->name == NULL && found_name && !found_is_wildcard) {
> > + /*
> > + * Skip wildcard aliases if we already have a match. This is
> > + * not a conflict that should result in an error.
> > + *
> > + * However, multiple wildcard aliases matching is an error
> > + * and will be caught below.
> > + */
> > + continue;
> > + }
> > +
> > + if (!alias_source_matches(qiv, *so, a, *name, is_alias_prefix)) {
> > + continue;
> > + }
>
> According to the contract of alias_source_matches() above, three cases:
>
> * No match: try next alias
>
> * Partial match: set *is_alias_prefix = true if non-null
>
> * Full match: leave it alone
>
> > +
> > + /*
> > + * For single-member aliases, an alias name is specified in the
> > + * alias definition. For wildcard aliases, the alias has the same
> > + * name as the member in the source object, i.e. *name.
> > + */
> > + if (!input_present(qiv, a->alias_so, a->name ?: *name)) {
> > + continue;
>
> What if alias_source_matches() already set *is_alias_prefix = true?
>
> I figure this can't happen, because it guards the assignment with the
> exact same call of input_present(). In other words, we can get here
> only for "full match". Correct?
Probably, but my reasoning is much simpler: If alias_source_matches()
sets *is_alias_prefix, it also returns false, so we would already have
taken a different path above.
> Such repeated calls of helpers can be a sign of awkward interfaces.
> Let's not worry about that now.
>
> > + }
> > +
> > + /*
> > + * A non-wildcard alias simply overrides a wildcard alias, but
> > + * two matching non-wildcard aliases or two matching wildcard
> > + * aliases conflict with each other.
> > + */
> > + if (found_name && (!found_is_wildcard || a->name == NULL)) {
> > + error_setg(errp, "Value for parameter %s was already given "
> > + "through an alias",
> > + full_name_so(qiv, *name, false, *so));
> > + return false;
> > + } else {
> > + found_name = a->name ?: *name;
> > + found_so = a->alias_so;
> > + found_is_wildcard = !a->name;
> > + }
> > + }
> > +
> > + /*
> > + * Chained aliases: *found_so/found_name might be the source of
> > + * another alias.
> > + */
> > + if (found_name && (found_so != *so || found_name != *name)) {
> > + find_object_member(qiv, &found_so, &found_name, NULL, errp);
>
> * Input present: update @found_so, @found_name.
>
> * Input absent: zap @found_name, @found_name.
>
> * Error: set *errp.
>
> Can @found_name be non-null? If yes, we can set *errp and return
> true, which would be bad.
Good catch, this looks like a bug.
Maybe the best solution is to guarantee *name == NULL on return for
error cases. Apart from updating the contract, the only relevant place
is the error_setg() above where we would add an explicit *name = NULL.
> > + }
> > +
> > + *so = found_so;
> > + *name = found_name;
> > +
> > + return found_name != NULL;
> > +}
> > +
> > static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
> > const char *name,
> > - bool consume)
> > + bool consume, Error **errp)
>
> Before the patch, two cases:
>
> * Input present: consume input if @consume, return the input
>
> * Input absent: return null
>
> The patch adds
>
> * Other error: set *errp, return null
>
> Slightly awkward to use, as we shall see below at [1] and [2].
> Observation, not demand.
>
> > {
> > StackObject *tos;
> > QObject *qobj;
> > @@ -189,10 +372,31 @@ static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
> > assert(qobj);
> >
> > if (qobject_type(qobj) == QTYPE_QDICT) {
> > - assert(name);
> > - ret = qdict_get(qobject_to(QDict, qobj), name);
> > - if (tos->h && consume && ret) {
> > - bool removed = g_hash_table_remove(tos->h, name);
> > + StackObject *so = tos;
> > + const char *key = name;
> > + bool is_alias_prefix;
> > +
> > + assert(key);
> > + if (!find_object_member(qiv, &so, &key, &is_alias_prefix, errp)) {
>
> * Input absent: zap @so, @key, set @is_alias_prefix.
>
> * Error: set *errp, leave @is_alias_prefix undefined.
>
> > + if (is_alias_prefix) {
>
> Use of undefined @is_alias_prefix in case "Error". Bug in code or in
> contract?
We should probably use ERRP_GUARD() and check for !*errp here.
> > + /*
> > + * The member is not present in the input, but
> > + * something inside of it might still be given through
> > + * an alias. Pretend there was an empty object in the
> > + * input.
> > + */
>
> We pretend there was an object to make the calling visitor enter the
> object and visit its members. Visiting a member first looks for input
> in the (empty) object, then follows aliases to look for it elsewhere.
>
> Is "might" still correct? The comment in alias_source_matches() makes
> me hope it's actually "will".
No, unfortunately it's still "might". This is not a problem of the new
code, but a limitation of the way how visitors work. We just can't know
which keys the caller will visit later and whether the given input will
match up with them.
>
> > + if (!qiv->empty_qdict) {
> > + qiv->empty_qdict = qdict_new();
> > + }
> > + return QOBJECT(qiv->empty_qdict);
> > + } else {
> > + return NULL;
> > + }
> > + }
> > + ret = qdict_get(qobject_to(QDict, so->obj), key);
> > + assert(ret != NULL);
> > + if (so->h && consume) {
> > + bool removed = g_hash_table_remove(so->h, key);
> > assert(removed);
> > }
> > } else {
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 5/6] qapi: Add support for aliases
2021-09-06 15:24 ` Markus Armbruster
@ 2021-09-09 16:39 ` Kevin Wolf
2021-09-14 8:42 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-09-09 16:39 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 06.09.2021 um 17:24 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
>
> > Introduce alias definitions for object types (structs and unions). This
> > allows using the same QAPI type and visitor for many syntax variations
> > that exist in the external representation, like between QMP and the
> > command line. It also provides a new tool for evolving the schema while
> > maintaining backwards compatibility during a deprecation period.
> >
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> > docs/devel/qapi-code-gen.rst | 104 +++++++++++++++++++++-
> > docs/sphinx/qapidoc.py | 2 +-
> > scripts/qapi/expr.py | 47 +++++++++-
> > scripts/qapi/schema.py | 116 +++++++++++++++++++++++--
> > scripts/qapi/types.py | 4 +-
> > scripts/qapi/visit.py | 34 +++++++-
> > tests/qapi-schema/test-qapi.py | 7 +-
> > tests/qapi-schema/double-type.err | 2 +-
> > tests/qapi-schema/unknown-expr-key.err | 2 +-
> > 9 files changed, 297 insertions(+), 21 deletions(-)
> >
> > diff --git a/docs/devel/qapi-code-gen.rst b/docs/devel/qapi-code-gen.rst
> > index 26c62b0e7b..c0883507a8 100644
> > --- a/docs/devel/qapi-code-gen.rst
> > +++ b/docs/devel/qapi-code-gen.rst
> > @@ -262,7 +262,8 @@ Syntax::
> > 'data': MEMBERS,
> > '*base': STRING,
> > '*if': COND,
> > - '*features': FEATURES }
> > + '*features': FEATURES,
> > + '*aliases': ALIASES }
> > MEMBERS = { MEMBER, ... }
> > MEMBER = STRING : TYPE-REF
> > | STRING : { 'type': TYPE-REF,
> > @@ -312,6 +313,9 @@ the schema`_ below for more on this.
> > The optional 'features' member specifies features. See Features_
> > below for more on this.
> >
> > +The optional 'aliases' member specifies aliases. See Aliases_ below
> > +for more on this.
> > +
> >
> > Union types
> > -----------
> > @@ -321,13 +325,15 @@ Syntax::
> > UNION = { 'union': STRING,
> > 'data': BRANCHES,
> > '*if': COND,
> > - '*features': FEATURES }
> > + '*features': FEATURES,
> > + '*aliases': ALIASES }
> > | { 'union': STRING,
> > 'data': BRANCHES,
> > 'base': ( MEMBERS | STRING ),
> > 'discriminator': STRING,
> > '*if': COND,
> > - '*features': FEATURES }
> > + '*features': FEATURES,
> > + '*aliases': ALIASES }
> > BRANCHES = { BRANCH, ... }
> > BRANCH = STRING : TYPE-REF
> > | STRING : { 'type': TYPE-REF, '*if': COND }
> > @@ -437,6 +443,9 @@ the schema`_ below for more on this.
> > The optional 'features' member specifies features. See Features_
> > below for more on this.
> >
> > +The optional 'aliases' member specifies aliases. See Aliases_ below
> > +for more on this.
> > +
> >
> > Alternate types
> > ---------------
> > @@ -888,6 +897,95 @@ shows a conditional entity only when the condition is satisfied in
> > this particular build.
> >
> >
> > +Aliases
> > +-------
> > +
> > +Object types, including structs and unions, can contain alias
> > +definitions.
> > +
> > +Aliases define alternative member names that may be used in wire input
> > +to provide a value for a member in the same object or in a nested
> > +object.
>
> Explaining intended use would be nice. From your cover letter:
>
> This allows using the same QAPI type and visitor even when the syntax
> has some variations between different external interfaces such as QMP
> and the command line.
I can add this exact paragraph here if you think this is useful. On the
other hand, it might make it look like this is the only valid use.
> It also provides a new tool for evolving the schema while maintaining
> backwards compatibility (possibly during a deprecation period).
>
> For the second use, we need to be able to tack feature 'deprecated' to
> exactly one of the two.
>
> We can already tack it to the "real" member. The real member's
> 'deprecated' must not apply to its aliases.
>
> We can't tack it to the alias, yet. More on that in review of PATCH 6.
Let's ignore this part for now. It's more an idea for a future
direction. The first use is what I actually need now for -chardev.
In an early version of the series, I tried to make aliases visible in
introspection, but as my use case doesn't need it and I soon found out
that it's not completely obvious how things should be exposed, I decided
to leave it out in this series.
> > +
> > +Syntax::
> > +
> > + ALIASES = [ ALIAS, ... ]
> > + ALIAS = { '*name': STRING,
> > + 'source': [ STRING, ... ] }
> > +
> > +If ``name`` is present, then the single member referred to by ``source``
> > +is made accessible with the name given by ``name`` in the type where the
> > +alias definition is specified.
> > +
> > +If ``name`` is not present, then this is a wildcard alias and all
> > +members in the object referred to by ``source`` are made accessible in
> > +the type where the alias definition is specified with the same name as
> > +they have in ``source``.
> > +
> > +``source`` is a non-empty list of member names representing the path to
> > +an object member. The first name is resolved in the same object. Each
> > +subsequent member is resolved in the object named by the preceding
> > +member.
> > +
> > +Do not use optional objects in the path of a wildcard alias unless there
> > +is no semantic difference between an empty object and an absent object.
> > +Absent objects are implicitly turned into empty ones if an alias could
> > +apply and provide a value in the nested object, which is always the case
> > +for wildcard aliases.
> > +
> > +Example: Alternative name for a member in the same object ::
> > +
> > + { 'struct': 'File',
> > + 'data': { 'path': 'str' },
> > + 'aliases': [ { 'name': 'filename', 'source': ['path'] } ] }
> > +
> > +The member ``path`` may instead be given through its alias ``filename``
> > +in input.
> > +
> > +Example: Alias for a member in a nested object ::
> > +
> > + { 'struct': 'A',
> > + 'data': { 'zahl': 'int' } }
> > + { 'struct': 'B',
> > + 'data': { 'drei': 'A' } }
> > + { 'struct': 'C',
> > + 'data': { 'zwei': 'B' } }
> > + { 'struct': 'D',
> > + 'data': { 'eins': 'C' },
> > + 'aliases': [ { 'name': 'number',
> > + 'source': ['eins', 'zwei', 'drei', 'zahl' ] },
> > + { 'name': 'the_B',
> > + 'source': ['eins','zwei'] } ] }
> > +
> > +With this definition, each of the following inputs for ``D`` mean the
> > +same::
> > +
> > + { 'eins': { 'zwei': { 'drei': { 'zahl': 42 } } } }
> > +
> > + { 'the_B': { 'drei': { 'zahl': 42 } } }
> > +
> > + { 'number': 42 }
> > +
> > +Example: Flattening a simple union with a wildcard alias that maps all
> > +members of ``data`` to the top level ::
> > +
> > + { 'union': 'SocketAddress',
> > + 'data': {
> > + 'inet': 'InetSocketAddress',
> > + 'unix': 'UnixSocketAddress' },
> > + 'aliases': [ { 'source': ['data'] } ] }
> > +
> > +Aliases are transitive: ``source`` may refer to another alias name. In
> > +this case, the alias is effectively an alternative name for the source
> > +of the other alias.
> > +
> > +In order to accommodate unions where variants differ in structure, it
> > +is allowed to use a path that doesn't necessarily match an existing
> > +member in every variant; in this case, the alias remains unused. The
> > +QAPI generator checks that there is at least one branch for which an
> > +alias could match.
> > +
> > +
> > Documentation comments
> > ----------------------
> >
> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> > index 87c67ab23f..68340b8529 100644
> > --- a/docs/sphinx/qapidoc.py
> > +++ b/docs/sphinx/qapidoc.py
> > @@ -313,7 +313,7 @@ def visit_enum_type(self, name, info, ifcond, features, members, prefix):
> > + self._nodes_for_if_section(ifcond))
> >
> > def visit_object_type(self, name, info, ifcond, features,
> > - base, members, variants):
> > + base, members, variants, aliases):
> > doc = self._cur_doc
> > if base and base.is_implicit():
> > base = None
> > diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
> > index cf98923fa6..054fef8d8e 100644
> > --- a/scripts/qapi/expr.py
> > +++ b/scripts/qapi/expr.py
> > @@ -430,6 +430,45 @@ def check_features(features: Optional[object],
> > check_if(feat, info, source)
> >
> >
> > +def check_aliases(aliases: Optional[object],
> > + info: QAPISourceInfo) -> None:
> > + """
> > + Normalize and validate the ``aliases`` member.
> > +
> > + :param aliases: The aliases member value to validate.
> > + :param info: QAPI schema source file information.
> > +
> > + :raise QAPISemError: When ``aliases`` fails validation.
> > + :return: None, ``aliases`` is normalized in-place as needed.
> > + """
> > +
> > + if aliases is None:
> > + return
> > + if not isinstance(aliases, list):
> > + raise QAPISemError(info, "'aliases' must be an array")
>
> Covered by PATCH 6's aliases-bad-type. Good.
>
> > + for a in aliases:
> > + if not isinstance(a, dict):
> > + raise QAPISemError(info, "'aliases' members must be objects")
>
> Convered by alias-bad-type.
>
> Doesn't identify the offending member. Same for all errors reported in
> this loop. Users should have no trouble identifying this one
> themselves. Less obvious ones might be confusing.
>
> Class QAPISchemaAlias identifies like 'alias ' + a['name'] and 'wildcard
> alias', as several test results show, e.g. alias-name-conflict.err and
> alias-source-non-object-wildcard.err. Could be improved on top.
We don't have a QAPISchemaAlias here, and more importantly, we don't
have a name because the object that should contain the name is actually
not an object.
So the best we could do is using an index like "'aliases' member 2 must
be an object". Not sure if this is much more useful, and as you say,
this one is obvious.
I think the two test results you mention identify the offending member
quite clearly? Maybe not if you have more than one wildcard alias. Do
you have another idea apart from using the list index?
> > @@ -413,12 +416,16 @@ def check(self, schema):
> > for m in self.local_members:
> > m.check(schema)
> > m.check_clash(self.info, seen)
> > - members = seen.values()
> > + members = list(seen.values())
>
> Uh, why do you need this? If I back it out, .check_path()'s assert
> isinstance(members, list) fails. If I take that out as well, "make
> check" passes. So does asserting isinstance(members, ValuesView).
>
> For what it's worth: when this code was written, we still used Python2,
> where .values() returns a list. The switch to Python3 silently made
> @members and self.members (assigned below) track changes of @seen. It
> only ever changes in QAPISchemaMember.check_clash().
>
> Hmmm, does your patch add such changes after this point?
>
> >
> > if self.variants:
> > self.variants.check(schema, seen)
> > self.variants.check_clash(self.info, seen)
> >
> > + for a in self.aliases:
> > + a.check_clash(self.info, seen)
>
> Covered by alias-name-conflict.
>
> Is such a change to @seen hiding behind a.check_clash()?
It don't remember the details of why I needed the list(), but
a.check_clash() is (a wrapper around) QAPISchemaMember.check_clash(), so
yes, it does change @seen. Specifically, it adds alias names to it.
> > + self.check_path(a, list(a.source), members)
> > +
> > self.members = members # mark completed
> >
> > # Check that the members of this type do not cause duplicate JSON members,
> > @@ -430,6 +437,68 @@ def check_clash(self, info, seen):
> > for m in self.members:
> > m.check_clash(info, seen)
> >
> > + # Deletes elements from path, so pass a copy if you still need them
>
> Suggest "from @path", to make it immediately obvious what you're talking
> about.
>
> The side effect is a bit ugly. We can tidy it up later if we care.
It's not hard to change.
> > + def check_path(self, alias, path, members=None, local_aliases_seen=()):
>
> Hmm, @local_aliases_seen is a tuple, whereas the @seen elsewhere are
> dict mapping name to QAPISchemaEntity, so we can .describe().
> Observation, not demand.
QAPISchemaMember, not QAPISchemaEntity, I think? Which is what
@local_aliases_seen contains, too.
> > + assert isinstance(path, list)
> > +
> > + if not path:
> > + return
> > + first = path.pop(0)
> > +
> > + for a in self.aliases:
> > + if a.name == first:
> > + if a in local_aliases_seen:
> > + raise QAPISemError(
> > + self.info,
> > + "%s resolving to '%s' makes '%s' an alias for itself"
> > + % (a.describe(self.info), a.source[0], a.source[0]))
>
> Covered by alias-recursive.
>
> Suggest to call the test case alias-loop.
>
> The error message shows just one arc of the loop:
>
> alias-recursive.json: In struct 'AliasStruct0':
> alias-recursive.json:1: alias 'baz' resolving to 'bar' makes 'bar' an alias for itself
>
> Showing the complete loop would be nice. Not a demand. Might require
> turning @local_aliases_seen into a dict.
>
> > +
> > + path = a.source + path
> > + return self.check_path(alias, path, members,
> > + (*local_aliases_seen, a))
> > +
> > + if members is None:
> > + assert self.members is not None
> > + members = self.members
> > + else:
> > + assert isinstance(members, list)
> > +
> > + for m in members:
> > + if m.name == first:
> > + # Wildcard aliases can only accept object types in the whole
> > + # path; for single-member aliases, the last element can be
> > + # any type
> > + need_obj = (alias.name is None) or path
> > + if need_obj and not isinstance(m.type, QAPISchemaObjectType):
> > + raise QAPISemError(
> > + self.info,
> > + "%s has non-object '%s' in its source path"
> > + % (alias.describe(self.info), m.name))
>
> Covered by alias-source-non-object-path and
> alias-source-non-object-wildcard.
>
> > + if alias.name is None and m.optional:
> > + raise QAPISemError(
> > + self.info,
> > + "%s has optional object %s in its source path"
> > + % (alias.describe(self.info), m.describe(self.info)))
>
> Covered by alias-source-optional-wildcard-indirect and
> alias-source-optional-wildcard.
>
> > + if path:
> > + m.type.check_path(alias, path)
> > + return
> > +
> > + # It is sufficient that the path is valid in at least one variant
> > + if self.variants:
> > + for v in self.variants.variants:
> > + try:
> > + return v.type.check_path(alias, [first, *path])
> > + except QAPISemError:
> > + pass
>
> Code smell: abuse of exception for perfectly non-exceptional control
> flow. I'm willing to tolerate this for now.
>
> > + raise QAPISemError(
> > + self.info,
> > + "%s has a source path that does not exist in any variant of %s"
> > + % (alias.describe(self.info), self.describe()))
>
> Covered by alias-source-inexistent-variants.
>
> > +
> > + raise QAPISemError(
> > + self.info,
> > + "%s has inexistent source" % alias.describe(self.info))
>
> Covered by alias-source-inexistent.
>
> pycodestyle-3 points out:
>
> scripts/qapi/schema.py:441:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
>
> Obvious fix: replace
>
> return FOO.check_path(..)
>
> by
>
> FOO.check_path(..)
> return
>
> Now let's see what this function does. It detects the following errors:
>
> 1. Alias loop
>
> 2. Alias "dotting through" a non-object
>
> 3. Wildcard alias "dotting through" an optional object
>
> 4. Alias must resolve to something (common or variant member, possibly
> nested)
>
> Lovely! But how does it work?
>
> The first loop takes care of 1. Looks like we try to resolve the alias,
> then recurse, keeping track of things meanwhile so we can detect loops.
> Isn't this more complicated than it needs to be?
>
> Aliases can only resolve to the same or deeper nesting levels.
>
> An alias that resolves to a deeper level cannot be part of a loop
> (because we can't resolve to back to the alias's level).
>
> So this should do:
>
> local_aliases_seen = {}
> for all aliases that resolve to the same level:
> if local_aliases_seen[alias.name]:
> # loop! we can retrace it using @local_aliases_seen if we
> # care
> raise ...
> local_aliases_seen[alias.name] = alias
>
> Or am I missing something?
You can change the recursion into something iterative, but the resulting
code would neither be shorter nor easier to understand (well, the latter
is subjective).
Essentially, instead of the recursive call, you would update @first and
then wrap the whole thing in a while loop. Either a 'while True' if
Python can break out of two loops (I thought it could, but doesn't look
like it?), or with some additional boolean variables.
> Moving on.
>
> To do 2. and 3., we try to resolve the alias, one element of source
> after the other.
>
> The first element can match either a common member, or a member of any
> number of variants.
>
> We first look for a common member, by searching @members for a match.
> If there is one, we check it, then recurse to check the remaining
> elements of source, and are done.
>
> Else, we try variant members. We reduce the problem to "common member"
> by recursing into the variant types. Neat! If this doesn't find any,
> the alias doesn't resolve, taking care of 4.
>
> Works. Searching @members is kind of brutish, though. We do have a map
> from member name to QAPISchemaObjectTypeMember: @seen. To use it, we'd
> have to fuse .check_path() into .check_clash(). Let's not worry about
> that right now.
>
> > +
> > def connect_doc(self, doc=None):
> > super().connect_doc(doc)
> > doc = doc or self.doc
> > @@ -474,7 +543,7 @@ def visit(self, visitor):
> > super().visit(visitor)
> > visitor.visit_object_type(
> > self.name, self.info, self.ifcond, self.features,
> > - self.base, self.local_members, self.variants)
> > + self.base, self.local_members, self.variants, self.aliases)
> > visitor.visit_object_type_flat(
> > self.name, self.info, self.ifcond, self.features,
> > self.members, self.variants)
> > @@ -639,7 +708,7 @@ def check_clash(self, info, seen):
> >
> >
> > class QAPISchemaMember:
> > - """ Represents object members, enum members and features """
> > + """ Represents object members, enum members, features and aliases """
> > role = 'member'
> >
> > def __init__(self, name, info, ifcond=None):
> > @@ -705,6 +774,30 @@ class QAPISchemaFeature(QAPISchemaMember):
> > role = 'feature'
> >
> >
> > +class QAPISchemaAlias(QAPISchemaMember):
> > + role = 'alias'
>
> I like this :)
>
> > +
> > + def __init__(self, name, info, source):
> > + assert name is None or isinstance(name, str)
> > + assert source
> > + for member in source:
> > + assert isinstance(member, str)
> > +
> > + super().__init__(name or '*', info)
> > + self.name = name
> > + self.source = source
> > +
> > + def check_clash(self, info, seen):
> > + if self.name:
> > + super().check_clash(info, seen)
> > +
> > + def describe(self, info):
> > + if self.name:
> > + return super().describe(info)
> > + else:
> > + return "wildcard alias"
>
> pycodestyle-3 gripes:
>
> scripts/qapi/schema.py:795:8: R1705: Unnecessary "else" after "return" (no-else-return)
I disagree with pycodestyle-3 on this one (and I know we disabled checks
like this for iotests), but I can change it.
> > +
> > +
> > class QAPISchemaObjectTypeMember(QAPISchemaMember):
> > def __init__(self, name, info, typ, optional, ifcond=None, features=None):
> > super().__init__(name, info, ifcond)
> > @@ -971,6 +1064,12 @@ def _make_features(self, features, info):
> > return [QAPISchemaFeature(f['name'], info, f.get('if'))
> > for f in features]
> >
> > + def _make_aliases(self, aliases, info):
> > + if aliases is None:
> > + return []
> > + return [QAPISchemaAlias(a.get('name'), info, a['source'])
> > + for a in aliases]
> > +
> > def _make_enum_members(self, values, info):
> > return [QAPISchemaEnumMember(v['name'], info, v.get('if'))
> > for v in values]
> > @@ -1045,11 +1144,12 @@ def _def_struct_type(self, expr, info, doc):
> > base = expr.get('base')
> > data = expr['data']
> > ifcond = expr.get('if')
> > + aliases = self._make_aliases(expr.get('aliases'), info)
> > features = self._make_features(expr.get('features'), info)
> > self._def_entity(QAPISchemaObjectType(
> > name, info, doc, ifcond, features, base,
> > self._make_members(data, info),
> > - None))
> > + None, aliases))
> >
> > def _make_variant(self, case, typ, ifcond, info):
> > return QAPISchemaVariant(case, info, typ, ifcond)
> > @@ -1068,6 +1168,7 @@ def _def_union_type(self, expr, info, doc):
> > data = expr['data']
> > base = expr.get('base')
> > ifcond = expr.get('if')
> > + aliases = self._make_aliases(expr.get('aliases'), info)
> > features = self._make_features(expr.get('features'), info)
> > tag_name = expr.get('discriminator')
> > tag_member = None
> > @@ -1092,7 +1193,8 @@ def _def_union_type(self, expr, info, doc):
> > QAPISchemaObjectType(name, info, doc, ifcond, features,
> > base, members,
> > QAPISchemaVariants(
> > - tag_name, info, tag_member, variants)))
> > + tag_name, info, tag_member, variants),
> > + aliases))
> >
> > def _def_alternate_type(self, expr, info, doc):
> > name = expr['alternate']
> > diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
> > index 20d572a23a..3bc451baa9 100644
> > --- a/scripts/qapi/types.py
> > +++ b/scripts/qapi/types.py
> > @@ -25,6 +25,7 @@
> > from .gen import QAPISchemaModularCVisitor, ifcontext
> > from .schema import (
> > QAPISchema,
> > + QAPISchemaAlias,
> > QAPISchemaEnumMember,
> > QAPISchemaFeature,
> > QAPISchemaObjectType,
> > @@ -332,7 +333,8 @@ def visit_object_type(self,
> > features: List[QAPISchemaFeature],
> > base: Optional[QAPISchemaObjectType],
> > members: List[QAPISchemaObjectTypeMember],
> > - variants: Optional[QAPISchemaVariants]) -> None:
> > + variants: Optional[QAPISchemaVariants],
> > + aliases: List[QAPISchemaAlias]) -> None:
> > # Nothing to do for the special empty builtin
> > if name == 'q_empty':
> > return
> > diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> > index 9e96f3c566..0aa0764755 100644
> > --- a/scripts/qapi/visit.py
> > +++ b/scripts/qapi/visit.py
> > @@ -26,6 +26,7 @@
> > from .gen import QAPISchemaModularCVisitor, ifcontext
> > from .schema import (
> > QAPISchema,
> > + QAPISchemaAlias,
> > QAPISchemaEnumMember,
> > QAPISchemaEnumType,
> > QAPISchemaFeature,
> > @@ -60,7 +61,8 @@ def gen_visit_members_decl(name: str) -> str:
> > def gen_visit_object_members(name: str,
> > base: Optional[QAPISchemaObjectType],
> > members: List[QAPISchemaObjectTypeMember],
> > - variants: Optional[QAPISchemaVariants]) -> str:
> > + variants: Optional[QAPISchemaVariants],
> > + aliases: List[QAPISchemaAlias]) -> str:
> > ret = mcgen('''
> >
> > bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
> > @@ -68,6 +70,24 @@ def gen_visit_object_members(name: str,
> > ''',
> > c_name=c_name(name))
> >
> > + if aliases:
> > + ret += mcgen('''
> > + visit_start_alias_scope(v);
> > +''')
> > +
> > + for a in aliases:
> > + if a.name:
> > + name = '"%s"' % a.name
> > + else:
> > + name = "NULL"
> > +
> > + source = ", ".join('"%s"' % x for x in a.source)
>
> @x is a poor choice for a loop control variable. @elt?
I couldn't figure out what @elt was supposed to mean at first, but I
mean, it's a really local name, so I don't care much if I understand
what the name is supposed to mean. I like 'x' because it doesn't even
pretend to mean anything specific.
> > +
> > + ret += mcgen('''
> > + visit_define_alias(v, %(name)s, (const char * []) { %(source)s, NULL });
> > +''',
> > + name=name, source=source)
> > +
> > if base:
> > ret += mcgen('''
> > if (!visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, errp)) {
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-09-06 15:28 ` Markus Armbruster
@ 2021-09-10 15:04 ` Kevin Wolf
2021-09-14 8:59 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-09-10 15:04 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 06.09.2021 um 17:28 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
> > + /* Can still specify the real member name with alias support */
> > + v = visitor_input_test_init(data, "{ 'foo': 42 }");
> > + visit_type_AliasStruct1(v, NULL, &tmp, &error_abort);
> > + g_assert_cmpint(tmp->foo, ==, 42);
> > + qapi_free_AliasStruct1(tmp);
> > +
> > + /* The alias is a working alternative */
> > + v = visitor_input_test_init(data, "{ 'bar': 42 }");
> > + visit_type_AliasStruct1(v, NULL, &tmp, &error_abort);
> > + g_assert_cmpint(tmp->foo, ==, 42);
> > + qapi_free_AliasStruct1(tmp);
> > +
> > + /* But you can't use both at the same time */
> > + v = visitor_input_test_init(data, "{ 'foo': 5, 'bar': 42 }");
> > + visit_type_AliasStruct1(v, NULL, &tmp, &err);
> > + error_free_or_abort(&err);
>
> I double-checked this reports "Value for parameter foo was already given
> through an alias", as it should.
>
> Pointing to what exactly is giving values to foo already would be nice.
> In this case, 'foo' is obvious, but 'bar' is not. This is not a demand.
We have the name, so we could print it, but it could be in a different
StackObject. I'm not sure if we have a good way to identify a parent
StackObject, and without it the message could be very confusing.
If you have a good idea what the message should look like, I can make an
attempt.
> > + /* You can't use more than one option at the same time */
> > + v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'foo': 42 } }");
> > + visit_type_AliasStruct3(v, NULL, &tmp, &err);
> > + error_free_or_abort(&err);
>
> "Parameter 'foo' is unexpected". Say what? It *is* expected, it just
> clashes with 'nested.foo'.
>
> I figure this is what happens:
>
> * visit_type_AliasStruct3()
>
> - visit_start_struct()
>
> - visit_type_AliasStruct3_members()
>
> • visit_type_AliasStruct1() for member @nested.
>
> This consumes consumes input nested.foo.
>
> - visit_check_struct()
>
> Error: input foo has not been consumed.
>
> Any ideas on how to report this error more clearly?
It's a result of the logic that wildcard aliases are silently ignored
when they aren't needed. The reason why I included this is that it would
allow you to have two members with the same name in the object
containing the alias and in the aliased object without conflicting as
long as both are given.
Never skipping wildcard aliases makes the code simpler and results in
the expected error message here. So I'll do that for v4.
Note that parsing something like '--chardev type=socket,addr.type=unix,
path=/tmp/sock,id=c' now depends on the order in the generated code. If
the top level 'type' weren't parsed and removed from the input first,
visiting 'addr.type' would now detect a conflict. For union types, we
know that 'type' is always parsed first, so it's not a problem, but in
the general case you need to be careful with the order.
> > +
> > + v = visitor_input_test_init(data, "{ 'tag': 'eins', 'foo': 6, 'bar': 9 }");
> > + visit_type_AliasFlatUnion(v, NULL, &tmp, &err);
> > + error_free_or_abort(&err);
>
> "Value for parameter foo was already given through an alias". Good,
> except I'm getting a feeling "already" may be confusing. It's "already"
> only in the sense that we already got the value via alias, which is an
> implementation detail. It may or may not be given already in the
> input. Here it's not: 'bar' follows 'foo'.
>
> What about "is also given through an alias"?
Sounds good.
> Positive tests look good to me, except they neglect to use any of the
> types using the alias features in QMP. I think we need something like
> the appended incremental patch.
I'm squashing it in.
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 4/6] qapi: Apply aliases in qobject-input-visitor
2021-09-08 13:01 ` Kevin Wolf
@ 2021-09-14 6:58 ` Markus Armbruster
2021-09-14 9:35 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-09-14 6:58 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 06.09.2021 um 17:16 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>>
>> > When looking for an object in a struct in the external representation,
>> > check not only the currently visited struct, but also whether an alias
>> > in the current StackObject matches and try to fetch the value from the
>> > alias then. Providing two values for the same object through different
>> > aliases is an error.
>> >
>> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>> > ---
>> > qapi/qobject-input-visitor.c | 227 +++++++++++++++++++++++++++++++++--
>> > 1 file changed, 218 insertions(+), 9 deletions(-)
>> >
>> > diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
>> > index 16a75442ff..6193df28a5 100644
>> > --- a/qapi/qobject-input-visitor.c
>> > +++ b/qapi/qobject-input-visitor.c
>> > @@ -97,6 +97,8 @@ struct QObjectInputVisitor {
>> > QObject *root;
>> > bool keyval; /* Assume @root made with keyval_parse() */
>> >
>> > + QDict *empty_qdict; /* Used for implicit objects */
>>
>> Would
>>
>> /* For visiting objects where all members are from aliases */
>>
>> be clearer?
>
> I know what I meant with "implicit objects", so not to me, but if you
> think it's clearer this way, then it probably is.
We're writing comments not so much for you and me now, but for someone
else trying to understand the code. That someone else may well be
forgetful me six months from now :)
>> > +
>> > /* Stack of objects being visited (all entries will be either
>> > * QDict or QList). */
>> > QSLIST_HEAD(, StackObject) stack;
>> > @@ -169,9 +171,190 @@ static const char *full_name(QObjectInputVisitor *qiv, const char *name)
>> > return full_name_so(qiv, name, false, tos);
>> > }
>> >
>> > +static bool find_object_member(QObjectInputVisitor *qiv,
>> > + StackObject **so, const char **name,
>> > + bool *is_alias_prefix, Error **errp);
>>
>> According to the function's contract below, three cases:
>>
>> * Input present: update *so, *name, return true.
>>
>> * Input absent: zap *so, *name, set *is_alias_prefix, return false.
>>
>> * Error: set *errp, leave *is_alias_prefix undefined, return false.
>>
>> > +
>> > +/*
>> > + * Check whether the member @name in @so, or an alias for it, is
>> > + * present in the input and can be used to obtain the value.
>> > + */
>> > +static bool input_present(QObjectInputVisitor *qiv, StackObject *so,
>> > + const char *name)
>> > +{
>> > + /*
>> > + * Check whether the alias member is present in the input
>> > + * (possibly recursively because aliases are transitive).
>> > + * The QAPI generator makes sure that alises cannot form loops, so
>> > + * the recursion guaranteed to terminate.
>> > + */
>> > + if (!find_object_member(qiv, &so, &name, NULL, NULL)) {
>>
>> * Input absent: zap @so and @name.
>>
>> * Error: don't zap.
>>
>> Since @so and @name aren't used anymore, the difference doesn't matter.
>> Okay.
>>
>> > + return false;
>> > + }
>> > +
>> > + /*
>> > + * Every source can be used only once. If a value in the input
>> > + * would end up being used twice through aliases, we'll fail the
>> > + * second access.
>> > + */
>> > + if (!g_hash_table_contains(so->h, name)) {
>> > + return false;
>> > + }
>> > +
>> > + return true;
>> > +}
>> > +
>> > +/*
>> > + * Check whether the member @name in the object visited by @so can be
>> > + * specified in the input by using the alias described by @a (which
>> > + * must be an alias contained in so->aliases).
>> > + *
>> > + * If @name is only a prefix of the alias source, but doesn't match
>> > + * immediately, false is returned and *is_alias_prefix is set to true
>> > + * if it is non-NULL. In all other cases, *is_alias_prefix is left
>> > + * unchanged.
>> > + */
>> > +static bool alias_source_matches(QObjectInputVisitor *qiv,
>> > + StackObject *so, InputVisitorAlias *a,
>> > + const char *name, bool *is_alias_prefix)
>> > +{
>> > + if (a->src[0] == NULL) {
>> > + assert(a->name == NULL);
>> > + return true;
>> > + }
>> > +
>> > + if (!strcmp(a->src[0], name)) {
>> > + if (a->name && a->src[1] == NULL) {
>> > + /*
>> > + * We're matching an exact member, the source for this alias is
>> > + * immediately in @so.
>> > + */
>> > + return true;
>> > + } else if (is_alias_prefix) {
>> > + /*
>> > + * We're only looking at a prefix of the source path for the alias.
>> > + * If the input contains no object of the requested name, we will
>> > + * implicitly create an empty one so that the alias can still be
>> > + * used.
>> > + *
>> > + * We want to create the implicit object only if the alias is
>> > + * actually used, but we can't tell here for wildcard aliases (only
>> > + * a later visitor call will determine this). This means that
>> > + * wildcard aliases must never have optional keys in their source
>> > + * path. The QAPI generator checks this condition.
>> > + */
>>
>> Double-checking: this actually ensures that we only ever create the
>> implicit object when it will not remain empty. Correct?
>
> For wildcard aliases, we still can't know which keys will be visited
> later. Checking that we don't have optional keys only avoids the
> confusion between absent and present, but empty objects that you would
> get from the implicit objects. So it means that creating an implicit
> object is never wrong, either the nested object can be visited (which
> means we needed the implicit object) or it errors out.
What I'm trying to understand is whether aliases may make up an empty
object, and if yes, under what conditions. Can you help me?
"Make up an empty object" = have an empty QDict in the result where the
JSON input doesn't have a {}.
>> > + if (!a->name || input_present(qiv, a->alias_so, a->name)) {
>> > + *is_alias_prefix = true;
>> > + }
>> > + }
>> > + }
>> > +
>> > + return false;
>> > +}
>> > +
>> > +/*
>> > + * Find the place in the input where the value for the object member
>> > + * @name in @so is specified, considering applicable aliases.
>> > + *
>> > + * If a value could be found, true is returned and @so and @name are
>> > + * updated to identify the key name and StackObject where the value
>> > + * can be found in the input. (This is either unchanged or the
>> > + * alias_so/name of an alias.) The value of @is_alias_prefix on
>> > + * return is undefined in this case.
>> > + *
>> > + * If no value could be found in the input, false is returned and @so
>> > + * and @name are set to NULL. This is not an error and @errp remains
>> > + * unchanged. If @is_alias_prefix is non-NULL, it is set to true if
>> > + * the given name is a prefix of the source path of an alias for which
>> > + * a value may be present in the input. It is set to false otherwise.
>> > + *
>> > + * If an error occurs (e.g. two values are specified for the member
>> > + * through different names), false is returned and @errp is set. The
>> > + * value of @is_alias_prefix on return is undefined in this case.
>> > + */
>> > +static bool find_object_member(QObjectInputVisitor *qiv,
>> > + StackObject **so, const char **name,
>> > + bool *is_alias_prefix, Error **errp)
>> > +{
>> > + QDict *qdict = qobject_to(QDict, (*so)->obj);
>> > + const char *found_name = NULL;
>> > + StackObject *found_so = NULL;
>> > + bool found_is_wildcard = false;
>> > + InputVisitorAlias *a;
>> > +
>> > + if (is_alias_prefix) {
>> > + *is_alias_prefix = false;
>> > + }
>> > +
>> > + /* Directly present in the container */
>> > + if (qdict_haskey(qdict, *name)) {
>> > + found_name = *name;
>> > + found_so = *so;
>> > + }
>> > +
>> > + /*
>> > + * Find aliases whose source path matches @name in this StackObject. We can
>> > + * then get the value with the key a->name from a->alias_so.
>> > + */
>> > + QSLIST_FOREACH(a, &(*so)->aliases, next) {
>> > + if (a->name == NULL && found_name && !found_is_wildcard) {
>> > + /*
>> > + * Skip wildcard aliases if we already have a match. This is
>> > + * not a conflict that should result in an error.
>> > + *
>> > + * However, multiple wildcard aliases matching is an error
>> > + * and will be caught below.
>> > + */
>> > + continue;
>> > + }
>> > +
>> > + if (!alias_source_matches(qiv, *so, a, *name, is_alias_prefix)) {
>> > + continue;
>> > + }
>>
>> According to the contract of alias_source_matches() above, three cases:
>>
>> * No match: try next alias
>>
>> * Partial match: set *is_alias_prefix = true if non-null
>>
>> * Full match: leave it alone
>>
>> > +
>> > + /*
>> > + * For single-member aliases, an alias name is specified in the
>> > + * alias definition. For wildcard aliases, the alias has the same
>> > + * name as the member in the source object, i.e. *name.
>> > + */
>> > + if (!input_present(qiv, a->alias_so, a->name ?: *name)) {
>> > + continue;
>>
>> What if alias_source_matches() already set *is_alias_prefix = true?
>>
>> I figure this can't happen, because it guards the assignment with the
>> exact same call of input_present(). In other words, we can get here
>> only for "full match". Correct?
>
> Probably, but my reasoning is much simpler: If alias_source_matches()
> sets *is_alias_prefix, it also returns false, so we would already have
> taken a different path above.
I see. The contract even says so: "false is returned and
*is_alias_prefix is set to true". It's actually the only way for
*is_alias_prefix to become true.
Output parameters that are set only sometimes require the caller to
initialize the receiving variable, or use it only under the same
condition it is set. The former is easy to forget, and the latter is
easy to get wrong.
"Set sometimes" can be useful, say when you have several calls where the
output should "accumulate". When you don't, I prefer to set always,
because it makes the function harder to misuse. Would you mind?
>> Such repeated calls of helpers can be a sign of awkward interfaces.
>> Let's not worry about that now.
>>
>> > + }
>> > +
>> > + /*
>> > + * A non-wildcard alias simply overrides a wildcard alias, but
>> > + * two matching non-wildcard aliases or two matching wildcard
>> > + * aliases conflict with each other.
>> > + */
>> > + if (found_name && (!found_is_wildcard || a->name == NULL)) {
>> > + error_setg(errp, "Value for parameter %s was already given "
>> > + "through an alias",
>> > + full_name_so(qiv, *name, false, *so));
>> > + return false;
>> > + } else {
>> > + found_name = a->name ?: *name;
>> > + found_so = a->alias_so;
>> > + found_is_wildcard = !a->name;
>> > + }
>> > + }
>> > +
>> > + /*
>> > + * Chained aliases: *found_so/found_name might be the source of
>> > + * another alias.
>> > + */
>> > + if (found_name && (found_so != *so || found_name != *name)) {
>> > + find_object_member(qiv, &found_so, &found_name, NULL, errp);
>>
>> * Input present: update @found_so, @found_name.
>>
>> * Input absent: zap @found_name, @found_name.
>>
>> * Error: set *errp.
>>
>> Can @found_name be non-null? If yes, we can set *errp and return
>> true, which would be bad.
>
> Good catch, this looks like a bug.
>
> Maybe the best solution is to guarantee *name == NULL on return for
> error cases. Apart from updating the contract, the only relevant place
> is the error_setg() above where we would add an explicit *name = NULL.
Yup, "set always" :)
>> > + }
>> > +
>> > + *so = found_so;
>> > + *name = found_name;
>> > +
>> > + return found_name != NULL;
>> > +}
>> > +
>> > static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
>> > const char *name,
>> > - bool consume)
>> > + bool consume, Error **errp)
>>
>> Before the patch, two cases:
>>
>> * Input present: consume input if @consume, return the input
>>
>> * Input absent: return null
>>
>> The patch adds
>>
>> * Other error: set *errp, return null
>>
>> Slightly awkward to use, as we shall see below at [1] and [2].
>> Observation, not demand.
>>
>> > {
>> > StackObject *tos;
>> > QObject *qobj;
>> > @@ -189,10 +372,31 @@ static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
>> > assert(qobj);
>> >
>> > if (qobject_type(qobj) == QTYPE_QDICT) {
>> > - assert(name);
>> > - ret = qdict_get(qobject_to(QDict, qobj), name);
>> > - if (tos->h && consume && ret) {
>> > - bool removed = g_hash_table_remove(tos->h, name);
>> > + StackObject *so = tos;
>> > + const char *key = name;
>> > + bool is_alias_prefix;
>> > +
>> > + assert(key);
>> > + if (!find_object_member(qiv, &so, &key, &is_alias_prefix, errp)) {
>>
>> * Input absent: zap @so, @key, set @is_alias_prefix.
>>
>> * Error: set *errp, leave @is_alias_prefix undefined.
>>
>> > + if (is_alias_prefix) {
>>
>> Use of undefined @is_alias_prefix in case "Error". Bug in code or in
>> contract?
>
> We should probably use ERRP_GUARD() and check for !*errp here.
A need to use ERRP_GUARD() often signals "awkward interface".
What about always setting is_alias_prefix? Then it's false on error.
>> > + /*
>> > + * The member is not present in the input, but
>> > + * something inside of it might still be given through
>> > + * an alias. Pretend there was an empty object in the
>> > + * input.
>> > + */
>>
>> We pretend there was an object to make the calling visitor enter the
>> object and visit its members. Visiting a member first looks for input
>> in the (empty) object, then follows aliases to look for it elsewhere.
>>
>> Is "might" still correct? The comment in alias_source_matches() makes
>> me hope it's actually "will".
>
> No, unfortunately it's still "might". This is not a problem of the new
> code, but a limitation of the way how visitors work. We just can't know
> which keys the caller will visit later and whether the given input will
> match up with them.
I see. Thanks!
>>
>> > + if (!qiv->empty_qdict) {
>> > + qiv->empty_qdict = qdict_new();
>> > + }
>> > + return QOBJECT(qiv->empty_qdict);
>> > + } else {
>> > + return NULL;
>> > + }
>> > + }
>> > + ret = qdict_get(qobject_to(QDict, so->obj), key);
>> > + assert(ret != NULL);
>> > + if (so->h && consume) {
>> > + bool removed = g_hash_table_remove(so->h, key);
>> > assert(removed);
>> > }
>> > } else {
>
> Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 5/6] qapi: Add support for aliases
2021-09-09 16:39 ` Kevin Wolf
@ 2021-09-14 8:42 ` Markus Armbruster
2021-09-14 11:00 ` Markus Armbruster
2021-09-14 14:24 ` Kevin Wolf
0 siblings, 2 replies; 43+ messages in thread
From: Markus Armbruster @ 2021-09-14 8:42 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 06.09.2021 um 17:24 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>>
>> > Introduce alias definitions for object types (structs and unions). This
>> > allows using the same QAPI type and visitor for many syntax variations
>> > that exist in the external representation, like between QMP and the
>> > command line. It also provides a new tool for evolving the schema while
>> > maintaining backwards compatibility during a deprecation period.
>> >
>> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>> > ---
>> > docs/devel/qapi-code-gen.rst | 104 +++++++++++++++++++++-
>> > docs/sphinx/qapidoc.py | 2 +-
>> > scripts/qapi/expr.py | 47 +++++++++-
>> > scripts/qapi/schema.py | 116 +++++++++++++++++++++++--
>> > scripts/qapi/types.py | 4 +-
>> > scripts/qapi/visit.py | 34 +++++++-
>> > tests/qapi-schema/test-qapi.py | 7 +-
>> > tests/qapi-schema/double-type.err | 2 +-
>> > tests/qapi-schema/unknown-expr-key.err | 2 +-
>> > 9 files changed, 297 insertions(+), 21 deletions(-)
>> >
>> > diff --git a/docs/devel/qapi-code-gen.rst b/docs/devel/qapi-code-gen.rst
>> > index 26c62b0e7b..c0883507a8 100644
>> > --- a/docs/devel/qapi-code-gen.rst
>> > +++ b/docs/devel/qapi-code-gen.rst
>> > @@ -262,7 +262,8 @@ Syntax::
>> > 'data': MEMBERS,
>> > '*base': STRING,
>> > '*if': COND,
>> > - '*features': FEATURES }
>> > + '*features': FEATURES,
>> > + '*aliases': ALIASES }
>> > MEMBERS = { MEMBER, ... }
>> > MEMBER = STRING : TYPE-REF
>> > | STRING : { 'type': TYPE-REF,
>> > @@ -312,6 +313,9 @@ the schema`_ below for more on this.
>> > The optional 'features' member specifies features. See Features_
>> > below for more on this.
>> >
>> > +The optional 'aliases' member specifies aliases. See Aliases_ below
>> > +for more on this.
>> > +
>> >
>> > Union types
>> > -----------
>> > @@ -321,13 +325,15 @@ Syntax::
>> > UNION = { 'union': STRING,
>> > 'data': BRANCHES,
>> > '*if': COND,
>> > - '*features': FEATURES }
>> > + '*features': FEATURES,
>> > + '*aliases': ALIASES }
>> > | { 'union': STRING,
>> > 'data': BRANCHES,
>> > 'base': ( MEMBERS | STRING ),
>> > 'discriminator': STRING,
>> > '*if': COND,
>> > - '*features': FEATURES }
>> > + '*features': FEATURES,
>> > + '*aliases': ALIASES }
>> > BRANCHES = { BRANCH, ... }
>> > BRANCH = STRING : TYPE-REF
>> > | STRING : { 'type': TYPE-REF, '*if': COND }
>> > @@ -437,6 +443,9 @@ the schema`_ below for more on this.
>> > The optional 'features' member specifies features. See Features_
>> > below for more on this.
>> >
>> > +The optional 'aliases' member specifies aliases. See Aliases_ below
>> > +for more on this.
>> > +
>> >
>> > Alternate types
>> > ---------------
>> > @@ -888,6 +897,95 @@ shows a conditional entity only when the condition is satisfied in
>> > this particular build.
>> >
>> >
>> > +Aliases
>> > +-------
>> > +
>> > +Object types, including structs and unions, can contain alias
>> > +definitions.
>> > +
>> > +Aliases define alternative member names that may be used in wire input
>> > +to provide a value for a member in the same object or in a nested
>> > +object.
>>
>> Explaining intended use would be nice. From your cover letter:
>>
>> This allows using the same QAPI type and visitor even when the syntax
>> has some variations between different external interfaces such as QMP
>> and the command line.
>
> I can add this exact paragraph here if you think this is useful. On the
> other hand, it might make it look like this is the only valid use.
Intended use is hard to describe without actual use in view :)
>> It also provides a new tool for evolving the schema while maintaining
>> backwards compatibility (possibly during a deprecation period).
>>
>> For the second use, we need to be able to tack feature 'deprecated' to
>> exactly one of the two.
>>
>> We can already tack it to the "real" member. The real member's
>> 'deprecated' must not apply to its aliases.
>>
>> We can't tack it to the alias, yet. More on that in review of PATCH 6.
>
> Let's ignore this part for now. It's more an idea for a future
> direction. The first use is what I actually need now for -chardev.
>
> In an early version of the series, I tried to make aliases visible in
> introspection, but as my use case doesn't need it and I soon found out
> that it's not completely obvious how things should be exposed, I decided
> to leave it out in this series.
Fair. One step at a time. We just have to be clear on limitations. I
think we should caution readers that aliases are not visible in
introspection, and therefore implementing QMP input as aliases is wrong.
What uses are left then? I figure it's just CLI, and only because it
lacks introspection. Thoughts?
>> > +
>> > +Syntax::
>> > +
>> > + ALIASES = [ ALIAS, ... ]
>> > + ALIAS = { '*name': STRING,
>> > + 'source': [ STRING, ... ] }
>> > +
>> > +If ``name`` is present, then the single member referred to by ``source``
>> > +is made accessible with the name given by ``name`` in the type where the
>> > +alias definition is specified.
>> > +
>> > +If ``name`` is not present, then this is a wildcard alias and all
>> > +members in the object referred to by ``source`` are made accessible in
>> > +the type where the alias definition is specified with the same name as
>> > +they have in ``source``.
>> > +
>> > +``source`` is a non-empty list of member names representing the path to
>> > +an object member. The first name is resolved in the same object. Each
>> > +subsequent member is resolved in the object named by the preceding
>> > +member.
>> > +
>> > +Do not use optional objects in the path of a wildcard alias unless there
>> > +is no semantic difference between an empty object and an absent object.
>> > +Absent objects are implicitly turned into empty ones if an alias could
>> > +apply and provide a value in the nested object, which is always the case
>> > +for wildcard aliases.
>> > +
>> > +Example: Alternative name for a member in the same object ::
>> > +
>> > + { 'struct': 'File',
>> > + 'data': { 'path': 'str' },
>> > + 'aliases': [ { 'name': 'filename', 'source': ['path'] } ] }
>> > +
>> > +The member ``path`` may instead be given through its alias ``filename``
>> > +in input.
>> > +
>> > +Example: Alias for a member in a nested object ::
>> > +
>> > + { 'struct': 'A',
>> > + 'data': { 'zahl': 'int' } }
>> > + { 'struct': 'B',
>> > + 'data': { 'drei': 'A' } }
>> > + { 'struct': 'C',
>> > + 'data': { 'zwei': 'B' } }
>> > + { 'struct': 'D',
>> > + 'data': { 'eins': 'C' },
>> > + 'aliases': [ { 'name': 'number',
>> > + 'source': ['eins', 'zwei', 'drei', 'zahl' ] },
>> > + { 'name': 'the_B',
>> > + 'source': ['eins','zwei'] } ] }
>> > +
>> > +With this definition, each of the following inputs for ``D`` mean the
>> > +same::
>> > +
>> > + { 'eins': { 'zwei': { 'drei': { 'zahl': 42 } } } }
>> > +
>> > + { 'the_B': { 'drei': { 'zahl': 42 } } }
>> > +
>> > + { 'number': 42 }
>> > +
>> > +Example: Flattening a simple union with a wildcard alias that maps all
>> > +members of ``data`` to the top level ::
>> > +
>> > + { 'union': 'SocketAddress',
>> > + 'data': {
>> > + 'inet': 'InetSocketAddress',
>> > + 'unix': 'UnixSocketAddress' },
>> > + 'aliases': [ { 'source': ['data'] } ] }
>> > +
>> > +Aliases are transitive: ``source`` may refer to another alias name. In
>> > +this case, the alias is effectively an alternative name for the source
>> > +of the other alias.
>> > +
>> > +In order to accommodate unions where variants differ in structure, it
>> > +is allowed to use a path that doesn't necessarily match an existing
>> > +member in every variant; in this case, the alias remains unused. The
>> > +QAPI generator checks that there is at least one branch for which an
>> > +alias could match.
>> > +
>> > +
>> > Documentation comments
>> > ----------------------
>> >
>> > diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
>> > index 87c67ab23f..68340b8529 100644
>> > --- a/docs/sphinx/qapidoc.py
>> > +++ b/docs/sphinx/qapidoc.py
>> > @@ -313,7 +313,7 @@ def visit_enum_type(self, name, info, ifcond, features, members, prefix):
>> > + self._nodes_for_if_section(ifcond))
>> >
>> > def visit_object_type(self, name, info, ifcond, features,
>> > - base, members, variants):
>> > + base, members, variants, aliases):
>> > doc = self._cur_doc
>> > if base and base.is_implicit():
>> > base = None
>> > diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
>> > index cf98923fa6..054fef8d8e 100644
>> > --- a/scripts/qapi/expr.py
>> > +++ b/scripts/qapi/expr.py
>> > @@ -430,6 +430,45 @@ def check_features(features: Optional[object],
>> > check_if(feat, info, source)
>> >
>> >
>> > +def check_aliases(aliases: Optional[object],
>> > + info: QAPISourceInfo) -> None:
>> > + """
>> > + Normalize and validate the ``aliases`` member.
>> > +
>> > + :param aliases: The aliases member value to validate.
>> > + :param info: QAPI schema source file information.
>> > +
>> > + :raise QAPISemError: When ``aliases`` fails validation.
>> > + :return: None, ``aliases`` is normalized in-place as needed.
>> > + """
>> > +
>> > + if aliases is None:
>> > + return
>> > + if not isinstance(aliases, list):
>> > + raise QAPISemError(info, "'aliases' must be an array")
>>
>> Covered by PATCH 6's aliases-bad-type. Good.
>>
>> > + for a in aliases:
>> > + if not isinstance(a, dict):
>> > + raise QAPISemError(info, "'aliases' members must be objects")
>>
>> Convered by alias-bad-type.
>>
>> Doesn't identify the offending member. Same for all errors reported in
>> this loop. Users should have no trouble identifying this one
>> themselves. Less obvious ones might be confusing.
>>
>> Class QAPISchemaAlias identifies like 'alias ' + a['name'] and 'wildcard
>> alias', as several test results show, e.g. alias-name-conflict.err and
>> alias-source-non-object-wildcard.err. Could be improved on top.
>
> We don't have a QAPISchemaAlias here, and more importantly, we don't
> have a name because the object that should contain the name is actually
> not an object.
You're right for this error, and as I said, this error is good enough as
is. I was wondering whether the loop's remaining errors would also be
good enough. There, we still don't have a QAPISchemaAlias. The point I
was trying to make is that the way QAPISchemaAlias describes itself
would work there as well: we survived the not isinstance(a, dict) check
visible above, so @a either has a member @name, or it is a wildcard.
QAPISchemaAlias.describe() describes the former as "alias 'NAME'", and
the latter as "wildcard alias".
> So the best we could do is using an index like "'aliases' member 2 must
> be an object". Not sure if this is much more useful, and as you say,
> this one is obvious.
>
> I think the two test results you mention identify the offending member
> quite clearly? Maybe not if you have more than one wildcard alias. Do
> you have another idea apart from using the list index?
I believe we don't use the list index for pinpointing error locations
elsewhere.
Let's examine the loop's remaining errors more closely:
check_keys(a, info, "'aliases' member", ['source'], ['name'])
Covered by alias-missing-source and alias-unknown-key.
alias-missing-source.json: In struct 'AliasStruct0':
alias-missing-source.json:1: 'aliases' member misses key 'source'
alias-unknown-key.json: In struct 'AliasStruct0':
alias-unknown-key.json:1: 'aliases' member has unknown key 'known'
Valid keys are 'name', 'source'.
Good enough, although describing the member like
QAPISchemaAlias.describe() does wouldn't hurt. Leave for later?
Same for most of the errors below; not noting it each time.
if 'name' in a:
source = "alias member 'name'"
check_name_is_str(a['name'], info, source)
Covered by alias-name-bad-type.
alias-name-bad-type.json: In struct 'AliasStruct0':
alias-name-bad-type.json:1: alias member 'name' requires a string name
check_name_str(a['name'], info, source)
Not covered. Manual test yields:
$ echo -n "{ 'struct': 'AliasStruct0', 'data': { 'foo': 'int' }, 'aliases': [ { 'name': '@bar', 'source': ['foo'] } ] }" | python3 scripts/qapi-gen.py /dev/stdin
scripts/qapi-gen.py: /dev/stdin: In struct 'AliasStruct0':
/dev/stdin:1: alias member 'name' has an invalid name
This one's crappy. I figure following my suggestion not to press
check_name_str() into service will get us to "good enough".
if not isinstance(a['source'], list):
raise QAPISemError(info,
"alias member 'source' must be an array")
Covered by alias-source-bad-type.
alias-source-bad-type.json: In struct 'AliasStruct0':
alias-source-bad-type.json:1: alias member 'source' must be an array
if not a['source']:
raise QAPISemError(info,
"alias member 'source' must not be empty")
Covered by alias-source-empty.
alias-source-empty.json: In struct 'AliasStruct0':
alias-source-empty.json:1: alias member 'source' must not be empty
source = "member of alias member 'source'"
for s in a['source']:
check_name_is_str(s, info, source)
Covered by alias-source-elem-bad-type.
alias-source-elem-bad-type.json: In struct 'AliasStruct0':
alias-source-elem-bad-type.json:1: member of alias member 'source' requires a string name
check_name_str(s, info, source)
Not covered. Also not necessary, I believe.
>> > @@ -413,12 +416,16 @@ def check(self, schema):
>> > for m in self.local_members:
>> > m.check(schema)
>> > m.check_clash(self.info, seen)
>> > - members = seen.values()
>> > + members = list(seen.values())
>>
>> Uh, why do you need this? If I back it out, .check_path()'s assert
>> isinstance(members, list) fails. If I take that out as well, "make
>> check" passes. So does asserting isinstance(members, ValuesView).
>>
>> For what it's worth: when this code was written, we still used Python2,
>> where .values() returns a list. The switch to Python3 silently made
>> @members and self.members (assigned below) track changes of @seen. It
>> only ever changes in QAPISchemaMember.check_clash().
>>
>> Hmmm, does your patch add such changes after this point?
>>
>> >
>> > if self.variants:
>> > self.variants.check(schema, seen)
>> > self.variants.check_clash(self.info, seen)
>> >
>> > + for a in self.aliases:
>> > + a.check_clash(self.info, seen)
>>
>> Covered by alias-name-conflict.
>>
>> Is such a change to @seen hiding behind a.check_clash()?
>
> It don't remember the details of why I needed the list(), but
> a.check_clash() is (a wrapper around) QAPISchemaMember.check_clash(), so
> yes, it does change @seen. Specifically, it adds alias names to it.
That's why you need it: seen.values() is a dictionary view object, but
you need something writable.
The silent change from list to view we got with Python 3 is kind of
iffy: we store the view in self.members (visible right below), which
keeps @seen alive.
Would you mind reverting this silent change in a separate one-liner
patch?
>> > + self.check_path(a, list(a.source), members)
>> > +
>> > self.members = members # mark completed
>> >
>> > # Check that the members of this type do not cause duplicate JSON members,
>> > @@ -430,6 +437,68 @@ def check_clash(self, info, seen):
>> > for m in self.members:
>> > m.check_clash(info, seen)
>> >
>> > + # Deletes elements from path, so pass a copy if you still need them
>>
>> Suggest "from @path", to make it immediately obvious what you're talking
>> about.
>>
>> The side effect is a bit ugly. We can tidy it up later if we care.
>
> It's not hard to change.
>
>> > + def check_path(self, alias, path, members=None, local_aliases_seen=()):
>>
>> Hmm, @local_aliases_seen is a tuple, whereas the @seen elsewhere are
>> dict mapping name to QAPISchemaEntity, so we can .describe().
>> Observation, not demand.
>
> QAPISchemaMember, not QAPISchemaEntity, I think? Which is what
> @local_aliases_seen contains, too.
You're right.
>> > + assert isinstance(path, list)
>> > +
>> > + if not path:
>> > + return
>> > + first = path.pop(0)
>> > +
>> > + for a in self.aliases:
>> > + if a.name == first:
>> > + if a in local_aliases_seen:
>> > + raise QAPISemError(
>> > + self.info,
>> > + "%s resolving to '%s' makes '%s' an alias for itself"
>> > + % (a.describe(self.info), a.source[0], a.source[0]))
>>
>> Covered by alias-recursive.
>>
>> Suggest to call the test case alias-loop.
>>
>> The error message shows just one arc of the loop:
>>
>> alias-recursive.json: In struct 'AliasStruct0':
>> alias-recursive.json:1: alias 'baz' resolving to 'bar' makes 'bar' an alias for itself
>>
>> Showing the complete loop would be nice. Not a demand. Might require
>> turning @local_aliases_seen into a dict.
>>
>> > +
>> > + path = a.source + path
>> > + return self.check_path(alias, path, members,
>> > + (*local_aliases_seen, a))
>> > +
>> > + if members is None:
>> > + assert self.members is not None
>> > + members = self.members
>> > + else:
>> > + assert isinstance(members, list)
>> > +
>> > + for m in members:
>> > + if m.name == first:
>> > + # Wildcard aliases can only accept object types in the whole
>> > + # path; for single-member aliases, the last element can be
>> > + # any type
>> > + need_obj = (alias.name is None) or path
>> > + if need_obj and not isinstance(m.type, QAPISchemaObjectType):
>> > + raise QAPISemError(
>> > + self.info,
>> > + "%s has non-object '%s' in its source path"
>> > + % (alias.describe(self.info), m.name))
>>
>> Covered by alias-source-non-object-path and
>> alias-source-non-object-wildcard.
>>
>> > + if alias.name is None and m.optional:
>> > + raise QAPISemError(
>> > + self.info,
>> > + "%s has optional object %s in its source path"
>> > + % (alias.describe(self.info), m.describe(self.info)))
>>
>> Covered by alias-source-optional-wildcard-indirect and
>> alias-source-optional-wildcard.
>>
>> > + if path:
>> > + m.type.check_path(alias, path)
>> > + return
>> > +
>> > + # It is sufficient that the path is valid in at least one variant
>> > + if self.variants:
>> > + for v in self.variants.variants:
>> > + try:
>> > + return v.type.check_path(alias, [first, *path])
>> > + except QAPISemError:
>> > + pass
>>
>> Code smell: abuse of exception for perfectly non-exceptional control
>> flow. I'm willing to tolerate this for now.
>>
>> > + raise QAPISemError(
>> > + self.info,
>> > + "%s has a source path that does not exist in any variant of %s"
>> > + % (alias.describe(self.info), self.describe()))
>>
>> Covered by alias-source-inexistent-variants.
>>
>> > +
>> > + raise QAPISemError(
>> > + self.info,
>> > + "%s has inexistent source" % alias.describe(self.info))
>>
>> Covered by alias-source-inexistent.
>>
>> pycodestyle-3 points out:
>>
>> scripts/qapi/schema.py:441:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
>>
>> Obvious fix: replace
>>
>> return FOO.check_path(..)
>>
>> by
>>
>> FOO.check_path(..)
>> return
>>
>> Now let's see what this function does. It detects the following errors:
>>
>> 1. Alias loop
>>
>> 2. Alias "dotting through" a non-object
>>
>> 3. Wildcard alias "dotting through" an optional object
>>
>> 4. Alias must resolve to something (common or variant member, possibly
>> nested)
>>
>> Lovely! But how does it work?
>>
>> The first loop takes care of 1. Looks like we try to resolve the alias,
>> then recurse, keeping track of things meanwhile so we can detect loops.
>> Isn't this more complicated than it needs to be?
>>
>> Aliases can only resolve to the same or deeper nesting levels.
>>
>> An alias that resolves to a deeper level cannot be part of a loop
>> (because we can't resolve to back to the alias's level).
>>
>> So this should do:
>>
>> local_aliases_seen = {}
>> for all aliases that resolve to the same level:
>> if local_aliases_seen[alias.name]:
>> # loop! we can retrace it using @local_aliases_seen if we
>> # care
>> raise ...
>> local_aliases_seen[alias.name] = alias
>>
>> Or am I missing something?
>
> You can change the recursion into something iterative, but the resulting
> code would neither be shorter nor easier to understand (well, the latter
> is subjective).
I wasn't talking about recursion vs. (equivalent) iteration.
When searching for alias loops, recursing into aliases that resolve
"deeper" is pointless, because they can't ever be part of a loop.
What's left is a simple linear chase of alias resolutions with loop
detection. Slightly complicated by not using a map from member name to
definition, but searching through the alias list instead. We build such
a map: @seen.
> Essentially, instead of the recursive call, you would update @first and
> then wrap the whole thing in a while loop. Either a 'while True' if
> Python can break out of two loops (I thought it could, but doesn't look
> like it?), or with some additional boolean variables.
If you don't bite, I can always simplify on top :)
>> Moving on.
>>
>> To do 2. and 3., we try to resolve the alias, one element of source
>> after the other.
>>
>> The first element can match either a common member, or a member of any
>> number of variants.
>>
>> We first look for a common member, by searching @members for a match.
>> If there is one, we check it, then recurse to check the remaining
>> elements of source, and are done.
>>
>> Else, we try variant members. We reduce the problem to "common member"
>> by recursing into the variant types. Neat! If this doesn't find any,
>> the alias doesn't resolve, taking care of 4.
>>
>> Works. Searching @members is kind of brutish, though. We do have a map
>> from member name to QAPISchemaObjectTypeMember: @seen. To use it, we'd
>> have to fuse .check_path() into .check_clash(). Let's not worry about
>> that right now.
>>
>> > +
>> > def connect_doc(self, doc=None):
>> > super().connect_doc(doc)
>> > doc = doc or self.doc
>> > @@ -474,7 +543,7 @@ def visit(self, visitor):
>> > super().visit(visitor)
>> > visitor.visit_object_type(
>> > self.name, self.info, self.ifcond, self.features,
>> > - self.base, self.local_members, self.variants)
>> > + self.base, self.local_members, self.variants, self.aliases)
>> > visitor.visit_object_type_flat(
>> > self.name, self.info, self.ifcond, self.features,
>> > self.members, self.variants)
>> > @@ -639,7 +708,7 @@ def check_clash(self, info, seen):
>> >
>> >
>> > class QAPISchemaMember:
>> > - """ Represents object members, enum members and features """
>> > + """ Represents object members, enum members, features and aliases """
>> > role = 'member'
>> >
>> > def __init__(self, name, info, ifcond=None):
>> > @@ -705,6 +774,30 @@ class QAPISchemaFeature(QAPISchemaMember):
>> > role = 'feature'
>> >
>> >
>> > +class QAPISchemaAlias(QAPISchemaMember):
>> > + role = 'alias'
>>
>> I like this :)
>>
>> > +
>> > + def __init__(self, name, info, source):
>> > + assert name is None or isinstance(name, str)
>> > + assert source
>> > + for member in source:
>> > + assert isinstance(member, str)
>> > +
>> > + super().__init__(name or '*', info)
>> > + self.name = name
>> > + self.source = source
>> > +
>> > + def check_clash(self, info, seen):
>> > + if self.name:
>> > + super().check_clash(info, seen)
>> > +
>> > + def describe(self, info):
>> > + if self.name:
>> > + return super().describe(info)
>> > + else:
>> > + return "wildcard alias"
>>
>> pycodestyle-3 gripes:
>>
>> scripts/qapi/schema.py:795:8: R1705: Unnecessary "else" after "return" (no-else-return)
>
> I disagree with pycodestyle-3 on this one (and I know we disabled checks
> like this for iotests), but I can change it.
Yes, please.
>> > +
>> > +
>> > class QAPISchemaObjectTypeMember(QAPISchemaMember):
>> > def __init__(self, name, info, typ, optional, ifcond=None, features=None):
>> > super().__init__(name, info, ifcond)
>> > @@ -971,6 +1064,12 @@ def _make_features(self, features, info):
>> > return [QAPISchemaFeature(f['name'], info, f.get('if'))
>> > for f in features]
>> >
>> > + def _make_aliases(self, aliases, info):
>> > + if aliases is None:
>> > + return []
>> > + return [QAPISchemaAlias(a.get('name'), info, a['source'])
>> > + for a in aliases]
>> > +
>> > def _make_enum_members(self, values, info):
>> > return [QAPISchemaEnumMember(v['name'], info, v.get('if'))
>> > for v in values]
>> > @@ -1045,11 +1144,12 @@ def _def_struct_type(self, expr, info, doc):
>> > base = expr.get('base')
>> > data = expr['data']
>> > ifcond = expr.get('if')
>> > + aliases = self._make_aliases(expr.get('aliases'), info)
>> > features = self._make_features(expr.get('features'), info)
>> > self._def_entity(QAPISchemaObjectType(
>> > name, info, doc, ifcond, features, base,
>> > self._make_members(data, info),
>> > - None))
>> > + None, aliases))
>> >
>> > def _make_variant(self, case, typ, ifcond, info):
>> > return QAPISchemaVariant(case, info, typ, ifcond)
>> > @@ -1068,6 +1168,7 @@ def _def_union_type(self, expr, info, doc):
>> > data = expr['data']
>> > base = expr.get('base')
>> > ifcond = expr.get('if')
>> > + aliases = self._make_aliases(expr.get('aliases'), info)
>> > features = self._make_features(expr.get('features'), info)
>> > tag_name = expr.get('discriminator')
>> > tag_member = None
>> > @@ -1092,7 +1193,8 @@ def _def_union_type(self, expr, info, doc):
>> > QAPISchemaObjectType(name, info, doc, ifcond, features,
>> > base, members,
>> > QAPISchemaVariants(
>> > - tag_name, info, tag_member, variants)))
>> > + tag_name, info, tag_member, variants),
>> > + aliases))
>> >
>> > def _def_alternate_type(self, expr, info, doc):
>> > name = expr['alternate']
>> > diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
>> > index 20d572a23a..3bc451baa9 100644
>> > --- a/scripts/qapi/types.py
>> > +++ b/scripts/qapi/types.py
>> > @@ -25,6 +25,7 @@
>> > from .gen import QAPISchemaModularCVisitor, ifcontext
>> > from .schema import (
>> > QAPISchema,
>> > + QAPISchemaAlias,
>> > QAPISchemaEnumMember,
>> > QAPISchemaFeature,
>> > QAPISchemaObjectType,
>> > @@ -332,7 +333,8 @@ def visit_object_type(self,
>> > features: List[QAPISchemaFeature],
>> > base: Optional[QAPISchemaObjectType],
>> > members: List[QAPISchemaObjectTypeMember],
>> > - variants: Optional[QAPISchemaVariants]) -> None:
>> > + variants: Optional[QAPISchemaVariants],
>> > + aliases: List[QAPISchemaAlias]) -> None:
>> > # Nothing to do for the special empty builtin
>> > if name == 'q_empty':
>> > return
>> > diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
>> > index 9e96f3c566..0aa0764755 100644
>> > --- a/scripts/qapi/visit.py
>> > +++ b/scripts/qapi/visit.py
>> > @@ -26,6 +26,7 @@
>> > from .gen import QAPISchemaModularCVisitor, ifcontext
>> > from .schema import (
>> > QAPISchema,
>> > + QAPISchemaAlias,
>> > QAPISchemaEnumMember,
>> > QAPISchemaEnumType,
>> > QAPISchemaFeature,
>> > @@ -60,7 +61,8 @@ def gen_visit_members_decl(name: str) -> str:
>> > def gen_visit_object_members(name: str,
>> > base: Optional[QAPISchemaObjectType],
>> > members: List[QAPISchemaObjectTypeMember],
>> > - variants: Optional[QAPISchemaVariants]) -> str:
>> > + variants: Optional[QAPISchemaVariants],
>> > + aliases: List[QAPISchemaAlias]) -> str:
>> > ret = mcgen('''
>> >
>> > bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
>> > @@ -68,6 +70,24 @@ def gen_visit_object_members(name: str,
>> > ''',
>> > c_name=c_name(name))
>> >
>> > + if aliases:
>> > + ret += mcgen('''
>> > + visit_start_alias_scope(v);
>> > +''')
>> > +
>> > + for a in aliases:
>> > + if a.name:
>> > + name = '"%s"' % a.name
>> > + else:
>> > + name = "NULL"
>> > +
>> > + source = ", ".join('"%s"' % x for x in a.source)
>>
>> @x is a poor choice for a loop control variable. @elt?
>
> I couldn't figure out what @elt was supposed to mean at first, but I
> mean, it's a really local name, so I don't care much if I understand
> what the name is supposed to mean. I like 'x' because it doesn't even
> pretend to mean anything specific.
>
>> > +
>> > + ret += mcgen('''
>> > + visit_define_alias(v, %(name)s, (const char * []) { %(source)s, NULL });
>> > +''',
>> > + name=name, source=source)
>> > +
>> > if base:
>> > ret += mcgen('''
>> > if (!visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, errp)) {
>
> Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-09-10 15:04 ` Kevin Wolf
@ 2021-09-14 8:59 ` Markus Armbruster
2021-09-14 10:05 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-09-14 8:59 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 06.09.2021 um 17:28 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>> > + /* Can still specify the real member name with alias support */
>> > + v = visitor_input_test_init(data, "{ 'foo': 42 }");
>> > + visit_type_AliasStruct1(v, NULL, &tmp, &error_abort);
>> > + g_assert_cmpint(tmp->foo, ==, 42);
>> > + qapi_free_AliasStruct1(tmp);
>> > +
>> > + /* The alias is a working alternative */
>> > + v = visitor_input_test_init(data, "{ 'bar': 42 }");
>> > + visit_type_AliasStruct1(v, NULL, &tmp, &error_abort);
>> > + g_assert_cmpint(tmp->foo, ==, 42);
>> > + qapi_free_AliasStruct1(tmp);
>> > +
>> > + /* But you can't use both at the same time */
>> > + v = visitor_input_test_init(data, "{ 'foo': 5, 'bar': 42 }");
>> > + visit_type_AliasStruct1(v, NULL, &tmp, &err);
>> > + error_free_or_abort(&err);
>>
>> I double-checked this reports "Value for parameter foo was already given
>> through an alias", as it should.
>>
>> Pointing to what exactly is giving values to foo already would be nice.
>> In this case, 'foo' is obvious, but 'bar' is not. This is not a demand.
>
> We have the name, so we could print it, but it could be in a different
> StackObject. I'm not sure if we have a good way to identify a parent
> StackObject, and without it the message could be very confusing.
All we have is full_name_so(), which can describe members / aliases in
any StackObject currently on qiv->stack, i.e. in the current object, the
object that contains it, and so forth up to the root object.
> If you have a good idea what the message should look like, I can make an
> attempt.
Of course, users don't want to know about alias chains, they want to
know what part of their input makes things explode. The answer to that
question could be like "'a.b.c' clashes with 'c'" (too terse, but you
get the idea).
Perhaps users then want to know *why* it clashes. To answer that
question, we'd have to present the complete alias chain. I'm not sure
this is worth the trouble.
>> > + /* You can't use more than one option at the same time */
>> > + v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'foo': 42 } }");
>> > + visit_type_AliasStruct3(v, NULL, &tmp, &err);
>> > + error_free_or_abort(&err);
>>
>> "Parameter 'foo' is unexpected". Say what? It *is* expected, it just
>> clashes with 'nested.foo'.
>>
>> I figure this is what happens:
>>
>> * visit_type_AliasStruct3()
>>
>> - visit_start_struct()
>>
>> - visit_type_AliasStruct3_members()
>>
>> • visit_type_AliasStruct1() for member @nested.
>>
>> This consumes consumes input nested.foo.
>>
>> - visit_check_struct()
>>
>> Error: input foo has not been consumed.
>>
>> Any ideas on how to report this error more clearly?
>
> It's a result of the logic that wildcard aliases are silently ignored
> when they aren't needed. The reason why I included this is that it would
> allow you to have two members with the same name in the object
> containing the alias and in the aliased object without conflicting as
> long as both are given.
*brain cramp*
Example?
> Never skipping wildcard aliases makes the code simpler and results in
> the expected error message here. So I'll do that for v4.
Trusting your judgement.
> Note that parsing something like '--chardev type=socket,addr.type=unix,
> path=/tmp/sock,id=c' now depends on the order in the generated code. If
> the top level 'type' weren't parsed and removed from the input first,
> visiting 'addr.type' would now detect a conflict. For union types, we
> know that 'type' is always parsed first, so it's not a problem, but in
> the general case you need to be careful with the order.
Uff! I think I'd like to understand this better. No need to delay v4
for that.
Can't yet say whether we need to spell out the limitation in commit
message and/or documentation.
>> > +
>> > + v = visitor_input_test_init(data, "{ 'tag': 'eins', 'foo': 6, 'bar': 9 }");
>> > + visit_type_AliasFlatUnion(v, NULL, &tmp, &err);
>> > + error_free_or_abort(&err);
>>
>> "Value for parameter foo was already given through an alias". Good,
>> except I'm getting a feeling "already" may be confusing. It's "already"
>> only in the sense that we already got the value via alias, which is an
>> implementation detail. It may or may not be given already in the
>> input. Here it's not: 'bar' follows 'foo'.
>>
>> What about "is also given through an alias"?
>
> Sounds good.
>
>> Positive tests look good to me, except they neglect to use any of the
>> types using the alias features in QMP. I think we need something like
>> the appended incremental patch.
>
> I'm squashing it in.
Thanks!
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 4/6] qapi: Apply aliases in qobject-input-visitor
2021-09-14 6:58 ` Markus Armbruster
@ 2021-09-14 9:35 ` Kevin Wolf
2021-09-14 14:24 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-09-14 9:35 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 14.09.2021 um 08:58 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
>
> > Am 06.09.2021 um 17:16 hat Markus Armbruster geschrieben:
> >> Kevin Wolf <kwolf@redhat.com> writes:
> >>
> >> > When looking for an object in a struct in the external representation,
> >> > check not only the currently visited struct, but also whether an alias
> >> > in the current StackObject matches and try to fetch the value from the
> >> > alias then. Providing two values for the same object through different
> >> > aliases is an error.
> >> >
> >> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> >> > +/*
> >> > + * Check whether the member @name in the object visited by @so can be
> >> > + * specified in the input by using the alias described by @a (which
> >> > + * must be an alias contained in so->aliases).
> >> > + *
> >> > + * If @name is only a prefix of the alias source, but doesn't match
> >> > + * immediately, false is returned and *is_alias_prefix is set to true
> >> > + * if it is non-NULL. In all other cases, *is_alias_prefix is left
> >> > + * unchanged.
> >> > + */
> >> > +static bool alias_source_matches(QObjectInputVisitor *qiv,
> >> > + StackObject *so, InputVisitorAlias *a,
> >> > + const char *name, bool *is_alias_prefix)
> >> > +{
> >> > + if (a->src[0] == NULL) {
> >> > + assert(a->name == NULL);
> >> > + return true;
> >> > + }
> >> > +
> >> > + if (!strcmp(a->src[0], name)) {
> >> > + if (a->name && a->src[1] == NULL) {
> >> > + /*
> >> > + * We're matching an exact member, the source for this alias is
> >> > + * immediately in @so.
> >> > + */
> >> > + return true;
> >> > + } else if (is_alias_prefix) {
> >> > + /*
> >> > + * We're only looking at a prefix of the source path for the alias.
> >> > + * If the input contains no object of the requested name, we will
> >> > + * implicitly create an empty one so that the alias can still be
> >> > + * used.
> >> > + *
> >> > + * We want to create the implicit object only if the alias is
> >> > + * actually used, but we can't tell here for wildcard aliases (only
> >> > + * a later visitor call will determine this). This means that
> >> > + * wildcard aliases must never have optional keys in their source
> >> > + * path. The QAPI generator checks this condition.
> >> > + */
> >>
> >> Double-checking: this actually ensures that we only ever create the
> >> implicit object when it will not remain empty. Correct?
> >
> > For wildcard aliases, we still can't know which keys will be visited
> > later. Checking that we don't have optional keys only avoids the
> > confusion between absent and present, but empty objects that you would
> > get from the implicit objects. So it means that creating an implicit
> > object is never wrong, either the nested object can be visited (which
> > means we needed the implicit object) or it errors out.
>
> What I'm trying to understand is whether aliases may make up an empty
> object, and if yes, under what conditions. Can you help me?
>
> "Make up an empty object" = have an empty QDict in the result where the
> JSON input doesn't have a {}.
Well, the result is a C type, not a QDict. We never build a single QDict
for the object including values resolved from aliases, we just fetch the
values from different QDicts if necessary.
But for what I think you're trying to get at: Yes, it can happen that we
start visiting a struct which was not present in the JSON, and for which
no members will match. This is if you have a wildcard alias for the
members of this object because we must assume that the alias might
provide the necessary values - but it might as well not have them.
There are two cases here:
1. The nested object contains non-optional members. This is an error
case. The error message will change from missing struct to missing
member, but this isn't too bad because the missing member does in
fact exist on the outer level, too, as an alias. So I think the error
message is still good.
2. The nested object only contains optional members. Then the alias
allows just not specifying the empty nested object, all of the zero
required members are taken from the outer object.
This would be a problem if the nested object were optional itself
because it would turn absent into present, but empty. So this is the
reason why we check in the generator that you don't have optional
members in the path of wildcard aliases.
> >> > +
> >> > + /*
> >> > + * For single-member aliases, an alias name is specified in the
> >> > + * alias definition. For wildcard aliases, the alias has the same
> >> > + * name as the member in the source object, i.e. *name.
> >> > + */
> >> > + if (!input_present(qiv, a->alias_so, a->name ?: *name)) {
> >> > + continue;
> >>
> >> What if alias_source_matches() already set *is_alias_prefix = true?
> >>
> >> I figure this can't happen, because it guards the assignment with the
> >> exact same call of input_present(). In other words, we can get here
> >> only for "full match". Correct?
> >
> > Probably, but my reasoning is much simpler: If alias_source_matches()
> > sets *is_alias_prefix, it also returns false, so we would already have
> > taken a different path above.
>
> I see. The contract even says so: "false is returned and
> *is_alias_prefix is set to true". It's actually the only way for
> *is_alias_prefix to become true.
>
> Output parameters that are set only sometimes require the caller to
> initialize the receiving variable, or use it only under the same
> condition it is set. The former is easy to forget, and the latter is
> easy to get wrong.
>
> "Set sometimes" can be useful, say when you have several calls where the
> output should "accumulate". When you don't, I prefer to set always,
> because it makes the function harder to misuse. Would you mind?
It does "accumulate" here. We want to return true if any of the aliases
make it true.
> >> > @@ -189,10 +372,31 @@ static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
> >> > assert(qobj);
> >> >
> >> > if (qobject_type(qobj) == QTYPE_QDICT) {
> >> > - assert(name);
> >> > - ret = qdict_get(qobject_to(QDict, qobj), name);
> >> > - if (tos->h && consume && ret) {
> >> > - bool removed = g_hash_table_remove(tos->h, name);
> >> > + StackObject *so = tos;
> >> > + const char *key = name;
> >> > + bool is_alias_prefix;
> >> > +
> >> > + assert(key);
> >> > + if (!find_object_member(qiv, &so, &key, &is_alias_prefix, errp)) {
> >>
> >> * Input absent: zap @so, @key, set @is_alias_prefix.
> >>
> >> * Error: set *errp, leave @is_alias_prefix undefined.
> >>
> >> > + if (is_alias_prefix) {
> >>
> >> Use of undefined @is_alias_prefix in case "Error". Bug in code or in
> >> contract?
> >
> > We should probably use ERRP_GUARD() and check for !*errp here.
>
> A need to use ERRP_GUARD() often signals "awkward interface".
>
> What about always setting is_alias_prefix? Then it's false on error.
Ok, I can define it as false for error cases if you prefer.
I'm not sure if I find it more readable than !*errp && ..., though. It
makes is_alias_prefix carry more information than its name suggests.
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-09-14 8:59 ` Markus Armbruster
@ 2021-09-14 10:05 ` Kevin Wolf
2021-09-14 13:29 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-09-14 10:05 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 14.09.2021 um 10:59 hat Markus Armbruster geschrieben:
> >> > + /* You can't use more than one option at the same time */
> >> > + v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'foo': 42 } }");
> >> > + visit_type_AliasStruct3(v, NULL, &tmp, &err);
> >> > + error_free_or_abort(&err);
> >>
> >> "Parameter 'foo' is unexpected". Say what? It *is* expected, it just
> >> clashes with 'nested.foo'.
> >>
> >> I figure this is what happens:
> >>
> >> * visit_type_AliasStruct3()
> >>
> >> - visit_start_struct()
> >>
> >> - visit_type_AliasStruct3_members()
> >>
> >> • visit_type_AliasStruct1() for member @nested.
> >>
> >> This consumes consumes input nested.foo.
> >>
> >> - visit_check_struct()
> >>
> >> Error: input foo has not been consumed.
> >>
> >> Any ideas on how to report this error more clearly?
> >
> > It's a result of the logic that wildcard aliases are silently ignored
> > when they aren't needed. The reason why I included this is that it would
> > allow you to have two members with the same name in the object
> > containing the alias and in the aliased object without conflicting as
> > long as both are given.
>
> *brain cramp*
>
> Example?
Let's use the real-world example I mentioned below:
{ 'union': 'ChardevBackend',
'data': { ...,
'socket': 'ChardevSocket',
... },
'aliases': [ { 'source': ['data'] } ] }
{ 'struct': 'ChardevSocket',
'data': { 'addr': 'SocketAddressLegacy',
... },
'base': 'ChardevCommon',
'aliases': [ { 'source': ['addr'] } ] }
{ 'union': 'SocketAddressLegacy',
'data': {
'inet': 'InetSocketAddress',
'unix': 'UnixSocketAddress',
'vsock': 'VsockSocketAddress',
'fd': 'String' },
'aliases': [
{ 'source': ['data'] },
{ 'name': 'fd', 'source': ['data', 'str'] }
] }
We have two simple unions there, and wildcard aliases all the way
through, so that you can have things like "hostname" on the top level.
However, two simple unions mean that "type" could refer to either
ChardevBackend.type or to SocketAddressLegacy.type.
Even though strictly speaking 'type=socket' is ambiguous, you don't want
to error out, but interpret it as a value for the outer one.
> > Never skipping wildcard aliases makes the code simpler and results in
> > the expected error message here. So I'll do that for v4.
>
> Trusting your judgement.
>
> > Note that parsing something like '--chardev type=socket,addr.type=unix,
> > path=/tmp/sock,id=c' now depends on the order in the generated code. If
> > the top level 'type' weren't parsed and removed from the input first,
> > visiting 'addr.type' would now detect a conflict. For union types, we
> > know that 'type' is always parsed first, so it's not a problem, but in
> > the general case you need to be careful with the order.
>
> Uff! I think I'd like to understand this better. No need to delay v4
> for that.
>
> Can't yet say whether we need to spell out the limitation in commit
> message and/or documentation.
The point where we could error out is when parsing SocketAddressLegacy,
because there can be two possible providers for "type".
The idea in the current code of this series was that we'll just ignore
wildcard aliases if we already have a value, because then the value must
be meant for somewhere else. So it doesn't error out and leaves the
value in the QDict for someone else to pick it up. If nobody picks it
up, it's an error "'foo' is unexpected".
If we change it and do error out when there are multiple possible values
through wildcard aliases, then the only thing that makes it work is that
ChardevBackend.type is parsed first and is therefore not a possible
value for SocketAddressLegacy.type any more.
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 5/6] qapi: Add support for aliases
2021-09-14 8:42 ` Markus Armbruster
@ 2021-09-14 11:00 ` Markus Armbruster
2021-09-14 14:24 ` Kevin Wolf
1 sibling, 0 replies; 43+ messages in thread
From: Markus Armbruster @ 2021-09-14 11:00 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Markus Armbruster <armbru@redhat.com> writes:
> Kevin Wolf <kwolf@redhat.com> writes:
>
>> Am 06.09.2021 um 17:24 hat Markus Armbruster geschrieben:
>>> Kevin Wolf <kwolf@redhat.com> writes:
>>>
>>> > Introduce alias definitions for object types (structs and unions). This
>>> > allows using the same QAPI type and visitor for many syntax variations
>>> > that exist in the external representation, like between QMP and the
>>> > command line. It also provides a new tool for evolving the schema while
>>> > maintaining backwards compatibility during a deprecation period.
>>> >
>>> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
[...]
>> It don't remember the details of why I needed the list(), but
>> a.check_clash() is (a wrapper around) QAPISchemaMember.check_clash(), so
>> yes, it does change @seen. Specifically, it adds alias names to it.
>
> That's why you need it: seen.values() is a dictionary view object, but
> you need something writable.
>
> The silent change from list to view we got with Python 3 is kind of
> iffy: we store the view in self.members (visible right below), which
> keeps @seen alive.
>
> Would you mind reverting this silent change in a separate one-liner
> patch?
Appending one for your convenience.
[...]
From 4eee60a6a02e425d25167761c47e858e240fe3f8 Mon Sep 17 00:00:00 2001
From: Markus Armbruster <armbru@redhat.com>
Date: Tue, 14 Sep 2021 12:25:09 +0200
Subject: [PATCH] qapi: Revert an accidental change from list to view object
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
A long time ago, commit 23a4b2c6f1 "qapi: Eliminate
QAPISchemaObjectType.check() variable members" replaced the manual
building of the list of members by seen.values(), where @seen is an
OrderedDict mapping names to members. The list is then stored in
self.members.
With Python 2, this is an innocent change: seen.values() returns "a
copy of the dictionary’s list of values".
With Python 3, it returns a dictionary view object instad. These
"provide a dynamic view on the dictionary’s entries, which means that
when the dictionary changes, the view reflects these changes."
Commit 23a4b2c6f1 predates the first mention of Python 3 in
scripts/qapi/ by years. If we had wanted a view object then, we'd
have used seen.viewvalues().
The accidental change of self.members from list to view object keeps
@seen alive longer. Not wanted, but harmless enough. I believe
that's all.
However, the change is in the next commit's way, which wants to mess
with self.members. Revert it.
All other uses of .values() in scripts/qapi/ are of the form
for ... in dict.values():
where the change to view object is just fine. Same for .keys() and
.items().
Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
scripts/qapi/schema.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index d1d27ff7ee..f313dbea27 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -413,7 +413,7 @@ def check(self, schema):
for m in self.local_members:
m.check(schema)
m.check_clash(self.info, seen)
- members = seen.values()
+ members = list(seen.values())
if self.variants:
self.variants.check(schema, seen)
--
2.31.1
^ permalink raw reply related [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-09-14 10:05 ` Kevin Wolf
@ 2021-09-14 13:29 ` Markus Armbruster
2021-09-15 9:24 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-09-14 13:29 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 14.09.2021 um 10:59 hat Markus Armbruster geschrieben:
>> >> > + /* You can't use more than one option at the same time */
>> >> > + v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'foo': 42 } }");
>> >> > + visit_type_AliasStruct3(v, NULL, &tmp, &err);
>> >> > + error_free_or_abort(&err);
>> >>
>> >> "Parameter 'foo' is unexpected". Say what? It *is* expected, it just
>> >> clashes with 'nested.foo'.
>> >>
>> >> I figure this is what happens:
>> >>
>> >> * visit_type_AliasStruct3()
>> >>
>> >> - visit_start_struct()
>> >>
>> >> - visit_type_AliasStruct3_members()
>> >>
>> >> • visit_type_AliasStruct1() for member @nested.
>> >>
>> >> This consumes consumes input nested.foo.
>> >>
>> >> - visit_check_struct()
>> >>
>> >> Error: input foo has not been consumed.
>> >>
>> >> Any ideas on how to report this error more clearly?
>> >
>> > It's a result of the logic that wildcard aliases are silently ignored
>> > when they aren't needed. The reason why I included this is that it would
>> > allow you to have two members with the same name in the object
>> > containing the alias and in the aliased object without conflicting as
>> > long as both are given.
>>
>> *brain cramp*
>>
>> Example?
>
> Let's use the real-world example I mentioned below:
>
> { 'union': 'ChardevBackend',
> 'data': { ...,
> 'socket': 'ChardevSocket',
> ... },
> 'aliases': [ { 'source': ['data'] } ] }
To pretend the simple union was flat, i.e. peel off its 'data', because
that nesting doesn't exist in the CLI you want to QAPIfy.
>
> { 'struct': 'ChardevSocket',
> 'data': { 'addr': 'SocketAddressLegacy',
> ... },
> 'base': 'ChardevCommon',
> 'aliases': [ { 'source': ['addr'] } ] }
To unbox struct SocketAddressLegacy, i.e. peel off its 'addr', for the
same reason.
>
> { 'union': 'SocketAddressLegacy',
> 'data': {
> 'inet': 'InetSocketAddress',
> 'unix': 'UnixSocketAddress',
> 'vsock': 'VsockSocketAddress',
> 'fd': 'String' },
> 'aliases': [
> { 'source': ['data'] },
To pretend the simple union was flat, i.e. peel off its 'data',
> { 'name': 'fd', 'source': ['data', 'str'] }
To unbox struct String, i.e. peel off its 'data'.
> ] }
Okay, I understand what you're trying to do. However:
> We have two simple unions there, and wildcard aliases all the way
> through, so that you can have things like "hostname" on the top level.
> However, two simple unions mean that "type" could refer to either
> ChardevBackend.type or to SocketAddressLegacy.type.
Yup. In ChardevBackend, we have both a (common) member @type, and a
chain of aliases that resolves @type to data.addr.data.type.
> Even though strictly speaking 'type=socket' is ambiguous, you don't want
> to error out, but interpret it as a value for the outer one.
I'm not sure.
When exactly are collisions an error? How exactly are non-erroneous
collisions resolved? qapi-code-gen.rst needs to answer this.
Back to the example. If 'type' resolves to ChardevBackend's member, how
should I specify SocketAddressLegacy's member? 'addr.type'?
Aside: existing -chardev infers SocketAddressLegacy's tag addr.type from
the presence of variant members, but that's a separate QAPIfication
problem.
I figure aliases let me refer to these guys at any level I like:
'data.addr.data.FOO', 'data.addr.FOO', 'addr.data.FOO', 'addr.FOO', or
just 'FOO'. Except for 'type', where just 'type' means something else.
Bizarre...
We actually require much less: for QMP chardev-add, we need
'data.addr.data.FOO' and nothing else, and for CLI -chardev, we need
'FOO' and nothing else (I think). The unneeded ones become accidental
parts of the external interface. If experience is any guide, we'll have
plenty of opportunity to regret such accidents :)
Can we somehow restrict external interfaces to what we actually need?
>> > Never skipping wildcard aliases makes the code simpler and results in
>> > the expected error message here. So I'll do that for v4.
>>
>> Trusting your judgement.
>>
>> > Note that parsing something like '--chardev type=socket,addr.type=unix,
>> > path=/tmp/sock,id=c' now depends on the order in the generated code. If
>> > the top level 'type' weren't parsed and removed from the input first,
>> > visiting 'addr.type' would now detect a conflict. For union types, we
>> > know that 'type' is always parsed first, so it's not a problem, but in
>> > the general case you need to be careful with the order.
>>
>> Uff! I think I'd like to understand this better. No need to delay v4
>> for that.
>>
>> Can't yet say whether we need to spell out the limitation in commit
>> message and/or documentation.
>
> The point where we could error out is when parsing SocketAddressLegacy,
> because there can be two possible providers for "type".
>
> The idea in the current code of this series was that we'll just ignore
> wildcard aliases if we already have a value, because then the value must
> be meant for somewhere else. So it doesn't error out and leaves the
> value in the QDict for someone else to pick it up. If nobody picks it
> up, it's an error "'foo' is unexpected".
>
> If we change it and do error out when there are multiple possible values
> through wildcard aliases, then the only thing that makes it work is that
> ChardevBackend.type is parsed first and is therefore not a possible
> value for SocketAddressLegacy.type any more.
You wrote you're picking the alternative with the simpler code for v4.
Fine with me, as long as it's reasonably easy to explain, in
qapi-code-gen.rst.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 4/6] qapi: Apply aliases in qobject-input-visitor
2021-09-14 9:35 ` Kevin Wolf
@ 2021-09-14 14:24 ` Markus Armbruster
0 siblings, 0 replies; 43+ messages in thread
From: Markus Armbruster @ 2021-09-14 14:24 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 14.09.2021 um 08:58 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>>
>> > Am 06.09.2021 um 17:16 hat Markus Armbruster geschrieben:
>> >> Kevin Wolf <kwolf@redhat.com> writes:
>> >>
>> >> > When looking for an object in a struct in the external representation,
>> >> > check not only the currently visited struct, but also whether an alias
>> >> > in the current StackObject matches and try to fetch the value from the
>> >> > alias then. Providing two values for the same object through different
>> >> > aliases is an error.
>> >> >
>> >> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>
>> >> > +/*
>> >> > + * Check whether the member @name in the object visited by @so can be
>> >> > + * specified in the input by using the alias described by @a (which
>> >> > + * must be an alias contained in so->aliases).
>> >> > + *
>> >> > + * If @name is only a prefix of the alias source, but doesn't match
>> >> > + * immediately, false is returned and *is_alias_prefix is set to true
>> >> > + * if it is non-NULL. In all other cases, *is_alias_prefix is left
>> >> > + * unchanged.
>> >> > + */
>> >> > +static bool alias_source_matches(QObjectInputVisitor *qiv,
>> >> > + StackObject *so, InputVisitorAlias *a,
>> >> > + const char *name, bool *is_alias_prefix)
>> >> > +{
>> >> > + if (a->src[0] == NULL) {
>> >> > + assert(a->name == NULL);
>> >> > + return true;
>> >> > + }
>> >> > +
>> >> > + if (!strcmp(a->src[0], name)) {
>> >> > + if (a->name && a->src[1] == NULL) {
>> >> > + /*
>> >> > + * We're matching an exact member, the source for this alias is
>> >> > + * immediately in @so.
>> >> > + */
>> >> > + return true;
>> >> > + } else if (is_alias_prefix) {
>> >> > + /*
>> >> > + * We're only looking at a prefix of the source path for the alias.
>> >> > + * If the input contains no object of the requested name, we will
>> >> > + * implicitly create an empty one so that the alias can still be
>> >> > + * used.
>> >> > + *
>> >> > + * We want to create the implicit object only if the alias is
>> >> > + * actually used, but we can't tell here for wildcard aliases (only
>> >> > + * a later visitor call will determine this). This means that
>> >> > + * wildcard aliases must never have optional keys in their source
>> >> > + * path. The QAPI generator checks this condition.
>> >> > + */
>> >>
>> >> Double-checking: this actually ensures that we only ever create the
>> >> implicit object when it will not remain empty. Correct?
>> >
>> > For wildcard aliases, we still can't know which keys will be visited
>> > later. Checking that we don't have optional keys only avoids the
>> > confusion between absent and present, but empty objects that you would
>> > get from the implicit objects. So it means that creating an implicit
>> > object is never wrong, either the nested object can be visited (which
>> > means we needed the implicit object) or it errors out.
>>
>> What I'm trying to understand is whether aliases may make up an empty
>> object, and if yes, under what conditions. Can you help me?
>>
>> "Make up an empty object" = have an empty QDict in the result where the
>> JSON input doesn't have a {}.
>
> Well, the result is a C type, not a QDict. We never build a single QDict
> for the object including values resolved from aliases, we just fetch the
> values from different QDicts if necessary.
I managed to confuse myself. Fortunately, it looks like I failed to
confuse you.
> But for what I think you're trying to get at: Yes, it can happen that we
> start visiting a struct which was not present in the JSON, and for which
> no members will match. This is if you have a wildcard alias for the
> members of this object because we must assume that the alias might
> provide the necessary values - but it might as well not have them.
>
> There are two cases here:
>
> 1. The nested object contains non-optional members. This is an error
> case. The error message will change from missing struct to missing
> member, but this isn't too bad because the missing member does in
> fact exist on the outer level, too, as an alias. So I think the error
> message is still good.
>
> 2. The nested object only contains optional members. Then the alias
> allows just not specifying the empty nested object, all of the zero
> required members are taken from the outer object.
>
> This would be a problem if the nested object were optional itself
> because it would turn absent into present, but empty. So this is the
> reason why we check in the generator that you don't have optional
> members in the path of wildcard aliases.
I'm too tired / stupid to grasp this in the abstract. I'll try again
tomorrow, with concrete examples.
>> >> > +
>> >> > + /*
>> >> > + * For single-member aliases, an alias name is specified in the
>> >> > + * alias definition. For wildcard aliases, the alias has the same
>> >> > + * name as the member in the source object, i.e. *name.
>> >> > + */
>> >> > + if (!input_present(qiv, a->alias_so, a->name ?: *name)) {
>> >> > + continue;
>> >>
>> >> What if alias_source_matches() already set *is_alias_prefix = true?
>> >>
>> >> I figure this can't happen, because it guards the assignment with the
>> >> exact same call of input_present(). In other words, we can get here
>> >> only for "full match". Correct?
>> >
>> > Probably, but my reasoning is much simpler: If alias_source_matches()
>> > sets *is_alias_prefix, it also returns false, so we would already have
>> > taken a different path above.
>>
>> I see. The contract even says so: "false is returned and
>> *is_alias_prefix is set to true". It's actually the only way for
>> *is_alias_prefix to become true.
>>
>> Output parameters that are set only sometimes require the caller to
>> initialize the receiving variable, or use it only under the same
>> condition it is set. The former is easy to forget, and the latter is
>> easy to get wrong.
>>
>> "Set sometimes" can be useful, say when you have several calls where the
>> output should "accumulate". When you don't, I prefer to set always,
>> because it makes the function harder to misuse. Would you mind?
>
> It does "accumulate" here. We want to return true if any of the aliases
> make it true.
You're right.
It doesn't accumulate with find_object_member() below. There, the code
always sets, but the contract doesn't actually promise it.
>> >> > @@ -189,10 +372,31 @@ static QObject *qobject_input_try_get_object(QObjectInputVisitor *qiv,
>> >> > assert(qobj);
>> >> >
>> >> > if (qobject_type(qobj) == QTYPE_QDICT) {
>> >> > - assert(name);
>> >> > - ret = qdict_get(qobject_to(QDict, qobj), name);
>> >> > - if (tos->h && consume && ret) {
>> >> > - bool removed = g_hash_table_remove(tos->h, name);
>> >> > + StackObject *so = tos;
>> >> > + const char *key = name;
>> >> > + bool is_alias_prefix;
>> >> > +
>> >> > + assert(key);
>> >> > + if (!find_object_member(qiv, &so, &key, &is_alias_prefix, errp)) {
>> >>
>> >> * Input absent: zap @so, @key, set @is_alias_prefix.
>> >>
>> >> * Error: set *errp, leave @is_alias_prefix undefined.
>> >>
>> >> > + if (is_alias_prefix) {
>> >>
>> >> Use of undefined @is_alias_prefix in case "Error". Bug in code or in
>> >> contract?
>> >
>> > We should probably use ERRP_GUARD() and check for !*errp here.
>>
>> A need to use ERRP_GUARD() often signals "awkward interface".
>>
>> What about always setting is_alias_prefix? Then it's false on error.
>
> Ok, I can define it as false for error cases if you prefer.
>
> I'm not sure if I find it more readable than !*errp && ..., though. It
> makes is_alias_prefix carry more information than its name suggests.
>
> Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 5/6] qapi: Add support for aliases
2021-09-14 8:42 ` Markus Armbruster
2021-09-14 11:00 ` Markus Armbruster
@ 2021-09-14 14:24 ` Kevin Wolf
1 sibling, 0 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-09-14 14:24 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 14.09.2021 um 10:42 hat Markus Armbruster geschrieben:
> >> It also provides a new tool for evolving the schema while maintaining
> >> backwards compatibility (possibly during a deprecation period).
> >>
> >> For the second use, we need to be able to tack feature 'deprecated' to
> >> exactly one of the two.
> >>
> >> We can already tack it to the "real" member. The real member's
> >> 'deprecated' must not apply to its aliases.
> >>
> >> We can't tack it to the alias, yet. More on that in review of PATCH 6.
> >
> > Let's ignore this part for now. It's more an idea for a future
> > direction. The first use is what I actually need now for -chardev.
> >
> > In an early version of the series, I tried to make aliases visible in
> > introspection, but as my use case doesn't need it and I soon found out
> > that it's not completely obvious how things should be exposed, I decided
> > to leave it out in this series.
>
> Fair. One step at a time. We just have to be clear on limitations. I
> think we should caution readers that aliases are not visible in
> introspection, and therefore implementing QMP input as aliases is wrong.
>
> What uses are left then? I figure it's just CLI, and only because it
> lacks introspection. Thoughts?
My current use case is CLI, yes. It might be the only one that fully
works after this series.
If we have to, we could use it in QMP with the usual hacks for detecting
features that are otherwise invisible in the schema (tie it to some
visible change, or add a feature flag), but preferably, introspection
support would be added before we use it in QMP.
> >> > + for a in aliases:
> >> > + if not isinstance(a, dict):
> >> > + raise QAPISemError(info, "'aliases' members must be objects")
> >>
> >> Convered by alias-bad-type.
> >>
> >> Doesn't identify the offending member. Same for all errors reported in
> >> this loop. Users should have no trouble identifying this one
> >> themselves. Less obvious ones might be confusing.
> >>
> >> Class QAPISchemaAlias identifies like 'alias ' + a['name'] and 'wildcard
> >> alias', as several test results show, e.g. alias-name-conflict.err and
> >> alias-source-non-object-wildcard.err. Could be improved on top.
> >
> > We don't have a QAPISchemaAlias here, and more importantly, we don't
> > have a name because the object that should contain the name is actually
> > not an object.
>
> You're right for this error, and as I said, this error is good enough as
> is. I was wondering whether the loop's remaining errors would also be
> good enough. There, we still don't have a QAPISchemaAlias. The point I
> was trying to make is that the way QAPISchemaAlias describes itself
> would work there as well: we survived the not isinstance(a, dict) check
> visible above, so @a either has a member @name, or it is a wildcard.
> QAPISchemaAlias.describe() describes the former as "alias 'NAME'", and
> the latter as "wildcard alias".
Ok, makes sense, I'll add the alias name to the later error messages. I
was just confused because you made the point under an error where this
doesn't apply.
> >> Now let's see what this function does. It detects the following errors:
> >>
> >> 1. Alias loop
> >>
> >> 2. Alias "dotting through" a non-object
> >>
> >> 3. Wildcard alias "dotting through" an optional object
> >>
> >> 4. Alias must resolve to something (common or variant member, possibly
> >> nested)
> >>
> >> Lovely! But how does it work?
> >>
> >> The first loop takes care of 1. Looks like we try to resolve the alias,
> >> then recurse, keeping track of things meanwhile so we can detect loops.
> >> Isn't this more complicated than it needs to be?
> >>
> >> Aliases can only resolve to the same or deeper nesting levels.
> >>
> >> An alias that resolves to a deeper level cannot be part of a loop
> >> (because we can't resolve to back to the alias's level).
> >>
> >> So this should do:
> >>
> >> local_aliases_seen = {}
> >> for all aliases that resolve to the same level:
> >> if local_aliases_seen[alias.name]:
> >> # loop! we can retrace it using @local_aliases_seen if we
> >> # care
> >> raise ...
> >> local_aliases_seen[alias.name] = alias
> >>
> >> Or am I missing something?
> >
> > You can change the recursion into something iterative, but the resulting
> > code would neither be shorter nor easier to understand (well, the latter
> > is subjective).
>
> I wasn't talking about recursion vs. (equivalent) iteration.
>
> When searching for alias loops, recursing into aliases that resolve
> "deeper" is pointless, because they can't ever be part of a loop.
>
> What's left is a simple linear chase of alias resolutions with loop
> detection. Slightly complicated by not using a map from member name to
> definition, but searching through the alias list instead. We build such
> a map: @seen.
The current recursive call of the function doesn't only detect loops,
but performs all of the four checks you identified above.
So you still need to resolve non-local aliases and check the whole path
for them even if they can't be part of a loop. There are three more
error cases that could apply for them.
For a moment I thought you're right and we don't need to resolve aliases
for these cases because other aliases will be checked separately. But
consider a case with these two aliases:
alias1 -> alias2/bar
alias2 -> foo
Here we need to make sure that the path foo/bar exists, which
check_path() for alias2 won't do anyway. We also must not check whether
the path alias2/bar exists without resolving aliases, because it would
incorrectly fail.
In the example, alias2 was a local alias, but the same is also true if
it were in a nested object. We still need to cjeck the full path with
resolved aliases.
> > Essentially, instead of the recursive call, you would update @first and
> > then wrap the whole thing in a while loop. Either a 'while True' if
> > Python can break out of two loops (I thought it could, but doesn't look
> > like it?), or with some additional boolean variables.
>
> If you don't bite, I can always simplify on top :)
I'll leave the code as it is for now. Feel free to simplify on top if
you find a way that is actually simple and correct.
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-09-14 13:29 ` Markus Armbruster
@ 2021-09-15 9:24 ` Kevin Wolf
2021-09-17 8:26 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-09-15 9:24 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 14.09.2021 um 15:29 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
>
> > Am 14.09.2021 um 10:59 hat Markus Armbruster geschrieben:
> >> >> > + /* You can't use more than one option at the same time */
> >> >> > + v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'foo': 42 } }");
> >> >> > + visit_type_AliasStruct3(v, NULL, &tmp, &err);
> >> >> > + error_free_or_abort(&err);
> >> >>
> >> >> "Parameter 'foo' is unexpected". Say what? It *is* expected, it just
> >> >> clashes with 'nested.foo'.
> >> >>
> >> >> I figure this is what happens:
> >> >>
> >> >> * visit_type_AliasStruct3()
> >> >>
> >> >> - visit_start_struct()
> >> >>
> >> >> - visit_type_AliasStruct3_members()
> >> >>
> >> >> • visit_type_AliasStruct1() for member @nested.
> >> >>
> >> >> This consumes consumes input nested.foo.
> >> >>
> >> >> - visit_check_struct()
> >> >>
> >> >> Error: input foo has not been consumed.
> >> >>
> >> >> Any ideas on how to report this error more clearly?
> >> >
> >> > It's a result of the logic that wildcard aliases are silently ignored
> >> > when they aren't needed. The reason why I included this is that it would
> >> > allow you to have two members with the same name in the object
> >> > containing the alias and in the aliased object without conflicting as
> >> > long as both are given.
> >>
> >> *brain cramp*
> >>
> >> Example?
> >
> > Let's use the real-world example I mentioned below:
> >
> > { 'union': 'ChardevBackend',
> > 'data': { ...,
> > 'socket': 'ChardevSocket',
> > ... },
> > 'aliases': [ { 'source': ['data'] } ] }
>
> To pretend the simple union was flat, i.e. peel off its 'data', because
> that nesting doesn't exist in the CLI you want to QAPIfy.
>
> >
> > { 'struct': 'ChardevSocket',
> > 'data': { 'addr': 'SocketAddressLegacy',
> > ... },
> > 'base': 'ChardevCommon',
> > 'aliases': [ { 'source': ['addr'] } ] }
>
> To unbox struct SocketAddressLegacy, i.e. peel off its 'addr', for the
> same reason.
>
> >
> > { 'union': 'SocketAddressLegacy',
> > 'data': {
> > 'inet': 'InetSocketAddress',
> > 'unix': 'UnixSocketAddress',
> > 'vsock': 'VsockSocketAddress',
> > 'fd': 'String' },
> > 'aliases': [
> > { 'source': ['data'] },
>
> To pretend the simple union was flat, i.e. peel off its 'data',
>
> > { 'name': 'fd', 'source': ['data', 'str'] }
>
> To unbox struct String, i.e. peel off its 'data'.
>
> > ] }
>
> Okay, I understand what you're trying to do. However:
>
> > We have two simple unions there, and wildcard aliases all the way
> > through, so that you can have things like "hostname" on the top level.
> > However, two simple unions mean that "type" could refer to either
> > ChardevBackend.type or to SocketAddressLegacy.type.
>
> Yup. In ChardevBackend, we have both a (common) member @type, and a
> chain of aliases that resolves @type to data.addr.data.type.
>
> > Even though strictly speaking 'type=socket' is ambiguous, you don't want
> > to error out, but interpret it as a value for the outer one.
>
> I'm not sure.
It's the only possible syntax for specifying ChardevBackend.type, so if
you don't allow this, everything breaks down.
> When exactly are collisions an error? How exactly are non-erroneous
> collisions resolved? qapi-code-gen.rst needs to answer this.
The strategy implemented in this version is: Collisions are generally an
error, except for wildcard aliases conflicting with a non-wildcard-alias
value. In this case the wildcard alias is ignored and the value is
assumed to belong elsewhere.
If it doesn't belong elsewhere in the end, it still sits in the QDict
when qobject_input_check_struct() runs, so you get an error.
> Back to the example. If 'type' resolves to ChardevBackend's member, how
> should I specify SocketAddressLegacy's member? 'addr.type'?
>
> Aside: existing -chardev infers SocketAddressLegacy's tag addr.type from
> the presence of variant members, but that's a separate QAPIfication
> problem.
So does the new -chardev in its compatibility code, but if you want to
specify it explicitly, addr.type is what you should use.
> I figure aliases let me refer to these guys at any level I like:
> 'data.addr.data.FOO', 'data.addr.FOO', 'addr.data.FOO', 'addr.FOO', or
> just 'FOO'. Except for 'type', where just 'type' means something else.
> Bizarre...
About as bizarre as shadowing variables in C. The more local one wins.
> We actually require much less: for QMP chardev-add, we need
> 'data.addr.data.FOO' and nothing else, and for CLI -chardev, we need
> 'FOO' and nothing else (I think). The unneeded ones become accidental
> parts of the external interface. If experience is any guide, we'll have
> plenty of opportunity to regret such accidents :)
>
> Can we somehow restrict external interfaces to what we actually need?
Not reasonably, I would say. Of course, you could try to cover all
paths with aliases in the top level, but the top level really shouldn't
have to know about the details of types deep inside some union variants.
The solution for reducing the allowed options here is to work on
introspection, mark 'data' deprecated everywhere and get rid of the
simple union nonsense.
> >> > Never skipping wildcard aliases makes the code simpler and results in
> >> > the expected error message here. So I'll do that for v4.
> >>
> >> Trusting your judgement.
> >>
> >> > Note that parsing something like '--chardev type=socket,addr.type=unix,
> >> > path=/tmp/sock,id=c' now depends on the order in the generated code. If
> >> > the top level 'type' weren't parsed and removed from the input first,
> >> > visiting 'addr.type' would now detect a conflict. For union types, we
> >> > know that 'type' is always parsed first, so it's not a problem, but in
> >> > the general case you need to be careful with the order.
> >>
> >> Uff! I think I'd like to understand this better. No need to delay v4
> >> for that.
> >>
> >> Can't yet say whether we need to spell out the limitation in commit
> >> message and/or documentation.
> >
> > The point where we could error out is when parsing SocketAddressLegacy,
> > because there can be two possible providers for "type".
> >
> > The idea in the current code of this series was that we'll just ignore
> > wildcard aliases if we already have a value, because then the value must
> > be meant for somewhere else. So it doesn't error out and leaves the
> > value in the QDict for someone else to pick it up. If nobody picks it
> > up, it's an error "'foo' is unexpected".
> >
> > If we change it and do error out when there are multiple possible values
> > through wildcard aliases, then the only thing that makes it work is that
> > ChardevBackend.type is parsed first and is therefore not a possible
> > value for SocketAddressLegacy.type any more.
>
> You wrote you're picking the alternative with the simpler code for v4.
> Fine with me, as long as it's reasonably easy to explain, in
> qapi-code-gen.rst.
I think I've come to the conclusion that it's not easy enough to
explain. As long as the parsing order should remain an implementation
detail that schema authors shouldn't rely on, it's not possible at all.
It's a pity because the code would have been simpler and it would
probably have worked for the cases we're interested in. But it doesn't
work in all hypothetical cases and we can't document the conditions for
that without making people rely on the parsing order.
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 5/6] qapi: Add support for aliases
2021-08-12 16:11 ` [PATCH v3 5/6] qapi: Add support for aliases Kevin Wolf
2021-09-06 15:24 ` Markus Armbruster
@ 2021-09-16 7:49 ` Markus Armbruster
1 sibling, 0 replies; 43+ messages in thread
From: Markus Armbruster @ 2021-09-16 7:49 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel, armbru
Kevin Wolf <kwolf@redhat.com> writes:
> Introduce alias definitions for object types (structs and unions). This
> allows using the same QAPI type and visitor for many syntax variations
> that exist in the external representation, like between QMP and the
> command line. It also provides a new tool for evolving the schema while
> maintaining backwards compatibility during a deprecation period.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
> docs/devel/qapi-code-gen.rst | 104 +++++++++++++++++++++-
> docs/sphinx/qapidoc.py | 2 +-
> scripts/qapi/expr.py | 47 +++++++++-
> scripts/qapi/schema.py | 116 +++++++++++++++++++++++--
> scripts/qapi/types.py | 4 +-
> scripts/qapi/visit.py | 34 +++++++-
> tests/qapi-schema/test-qapi.py | 7 +-
> tests/qapi-schema/double-type.err | 2 +-
> tests/qapi-schema/unknown-expr-key.err | 2 +-
> 9 files changed, 297 insertions(+), 21 deletions(-)
>
> diff --git a/docs/devel/qapi-code-gen.rst b/docs/devel/qapi-code-gen.rst
> index 26c62b0e7b..c0883507a8 100644
> --- a/docs/devel/qapi-code-gen.rst
> +++ b/docs/devel/qapi-code-gen.rst
> @@ -262,7 +262,8 @@ Syntax::
> 'data': MEMBERS,
> '*base': STRING,
> '*if': COND,
> - '*features': FEATURES }
> + '*features': FEATURES,
> + '*aliases': ALIASES }
> MEMBERS = { MEMBER, ... }
> MEMBER = STRING : TYPE-REF
> | STRING : { 'type': TYPE-REF,
> @@ -312,6 +313,9 @@ the schema`_ below for more on this.
> The optional 'features' member specifies features. See Features_
> below for more on this.
>
> +The optional 'aliases' member specifies aliases. See Aliases_ below
> +for more on this.
> +
>
> Union types
> -----------
> @@ -321,13 +325,15 @@ Syntax::
> UNION = { 'union': STRING,
> 'data': BRANCHES,
> '*if': COND,
> - '*features': FEATURES }
> + '*features': FEATURES,
> + '*aliases': ALIASES }
> | { 'union': STRING,
> 'data': BRANCHES,
> 'base': ( MEMBERS | STRING ),
> 'discriminator': STRING,
> '*if': COND,
> - '*features': FEATURES }
> + '*features': FEATURES,
> + '*aliases': ALIASES }
> BRANCHES = { BRANCH, ... }
> BRANCH = STRING : TYPE-REF
> | STRING : { 'type': TYPE-REF, '*if': COND }
> @@ -437,6 +443,9 @@ the schema`_ below for more on this.
> The optional 'features' member specifies features. See Features_
> below for more on this.
>
> +The optional 'aliases' member specifies aliases. See Aliases_ below
> +for more on this.
> +
>
> Alternate types
> ---------------
> @@ -888,6 +897,95 @@ shows a conditional entity only when the condition is satisfied in
> this particular build.
>
>
> +Aliases
> +-------
> +
> +Object types, including structs and unions, can contain alias
> +definitions.
> +
> +Aliases define alternative member names that may be used in wire input
> +to provide a value for a member in the same object or in a nested
> +object.
> +
> +Syntax::
> +
> + ALIASES = [ ALIAS, ... ]
> + ALIAS = { '*name': STRING,
> + 'source': [ STRING, ... ] }
> +
> +If ``name`` is present, then the single member referred to by ``source``
> +is made accessible with the name given by ``name`` in the type where the
> +alias definition is specified.
> +
> +If ``name`` is not present, then this is a wildcard alias and all
> +members in the object referred to by ``source`` are made accessible in
> +the type where the alias definition is specified with the same name as
> +they have in ``source``.
> +
> +``source`` is a non-empty list of member names representing the path to
> +an object member. The first name is resolved in the same object. Each
> +subsequent member is resolved in the object named by the preceding
> +member.
> +
> +Do not use optional objects in the path of a wildcard alias unless there
> +is no semantic difference between an empty object and an absent object.
> +Absent objects are implicitly turned into empty ones if an alias could
> +apply and provide a value in the nested object, which is always the case
> +for wildcard aliases.
Is this still correct?
> +
> +Example: Alternative name for a member in the same object ::
> +
> + { 'struct': 'File',
> + 'data': { 'path': 'str' },
> + 'aliases': [ { 'name': 'filename', 'source': ['path'] } ] }
> +
> +The member ``path`` may instead be given through its alias ``filename``
> +in input.
> +
> +Example: Alias for a member in a nested object ::
> +
> + { 'struct': 'A',
> + 'data': { 'zahl': 'int' } }
> + { 'struct': 'B',
> + 'data': { 'drei': 'A' } }
> + { 'struct': 'C',
> + 'data': { 'zwei': 'B' } }
> + { 'struct': 'D',
> + 'data': { 'eins': 'C' },
> + 'aliases': [ { 'name': 'number',
> + 'source': ['eins', 'zwei', 'drei', 'zahl' ] },
> + { 'name': 'the_B',
> + 'source': ['eins','zwei'] } ] }
> +
> +With this definition, each of the following inputs for ``D`` mean the
> +same::
> +
> + { 'eins': { 'zwei': { 'drei': { 'zahl': 42 } } } }
> +
> + { 'the_B': { 'drei': { 'zahl': 42 } } }
> +
> + { 'number': 42 }
> +
> +Example: Flattening a simple union with a wildcard alias that maps all
> +members of ``data`` to the top level ::
> +
> + { 'union': 'SocketAddress',
> + 'data': {
> + 'inet': 'InetSocketAddress',
> + 'unix': 'UnixSocketAddress' },
> + 'aliases': [ { 'source': ['data'] } ] }
> +
> +Aliases are transitive: ``source`` may refer to another alias name. In
> +this case, the alias is effectively an alternative name for the source
> +of the other alias.
> +
> +In order to accommodate unions where variants differ in structure, it
> +is allowed to use a path that doesn't necessarily match an existing
> +member in every variant; in this case, the alias remains unused. The
> +QAPI generator checks that there is at least one branch for which an
> +alias could match.
> +
> +
> Documentation comments
> ----------------------
>
> diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> index 87c67ab23f..68340b8529 100644
> --- a/docs/sphinx/qapidoc.py
> +++ b/docs/sphinx/qapidoc.py
> @@ -313,7 +313,7 @@ def visit_enum_type(self, name, info, ifcond, features, members, prefix):
> + self._nodes_for_if_section(ifcond))
>
> def visit_object_type(self, name, info, ifcond, features,
> - base, members, variants):
> + base, members, variants, aliases):
> doc = self._cur_doc
> if base and base.is_implicit():
> base = None
[...]
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-09-15 9:24 ` Kevin Wolf
@ 2021-09-17 8:26 ` Markus Armbruster
2021-09-17 15:03 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-09-17 8:26 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 14.09.2021 um 15:29 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>>
>> > Am 14.09.2021 um 10:59 hat Markus Armbruster geschrieben:
>> >> >> > + /* You can't use more than one option at the same time */
>> >> >> > + v = visitor_input_test_init(data, "{ 'foo': 42, 'nested': { 'foo': 42 } }");
>> >> >> > + visit_type_AliasStruct3(v, NULL, &tmp, &err);
>> >> >> > + error_free_or_abort(&err);
>> >> >>
>> >> >> "Parameter 'foo' is unexpected". Say what? It *is* expected, it just
>> >> >> clashes with 'nested.foo'.
>> >> >>
>> >> >> I figure this is what happens:
>> >> >>
>> >> >> * visit_type_AliasStruct3()
>> >> >>
>> >> >> - visit_start_struct()
>> >> >>
>> >> >> - visit_type_AliasStruct3_members()
>> >> >>
>> >> >> • visit_type_AliasStruct1() for member @nested.
>> >> >>
>> >> >> This consumes consumes input nested.foo.
>> >> >>
>> >> >> - visit_check_struct()
>> >> >>
>> >> >> Error: input foo has not been consumed.
>> >> >>
>> >> >> Any ideas on how to report this error more clearly?
>> >> >
>> >> > It's a result of the logic that wildcard aliases are silently ignored
>> >> > when they aren't needed. The reason why I included this is that it would
>> >> > allow you to have two members with the same name in the object
>> >> > containing the alias and in the aliased object without conflicting as
>> >> > long as both are given.
>> >>
>> >> *brain cramp*
>> >>
>> >> Example?
>> >
>> > Let's use the real-world example I mentioned below:
>> >
>> > { 'union': 'ChardevBackend',
>> > 'data': { ...,
>> > 'socket': 'ChardevSocket',
>> > ... },
>> > 'aliases': [ { 'source': ['data'] } ] }
>>
>> To pretend the simple union was flat, i.e. peel off its 'data', because
>> that nesting doesn't exist in the CLI you want to QAPIfy.
>>
>> >
>> > { 'struct': 'ChardevSocket',
>> > 'data': { 'addr': 'SocketAddressLegacy',
>> > ... },
>> > 'base': 'ChardevCommon',
>> > 'aliases': [ { 'source': ['addr'] } ] }
>>
>> To unbox struct SocketAddressLegacy, i.e. peel off its 'addr', for the
>> same reason.
>>
>> >
>> > { 'union': 'SocketAddressLegacy',
>> > 'data': {
>> > 'inet': 'InetSocketAddress',
>> > 'unix': 'UnixSocketAddress',
>> > 'vsock': 'VsockSocketAddress',
>> > 'fd': 'String' },
>> > 'aliases': [
>> > { 'source': ['data'] },
>>
>> To pretend the simple union was flat, i.e. peel off its 'data',
>>
>> > { 'name': 'fd', 'source': ['data', 'str'] }
>>
>> To unbox struct String, i.e. peel off its 'data'.
>>
>> > ] }
>>
>> Okay, I understand what you're trying to do. However:
>>
>> > We have two simple unions there, and wildcard aliases all the way
>> > through, so that you can have things like "hostname" on the top level.
>> > However, two simple unions mean that "type" could refer to either
>> > ChardevBackend.type or to SocketAddressLegacy.type.
>>
>> Yup. In ChardevBackend, we have both a (common) member @type, and a
>> chain of aliases that resolves @type to data.addr.data.type.
>>
>> > Even though strictly speaking 'type=socket' is ambiguous, you don't want
>> > to error out, but interpret it as a value for the outer one.
>>
>> I'm not sure.
>
> It's the only possible syntax for specifying ChardevBackend.type, so if
> you don't allow this, everything breaks down.
>
>> When exactly are collisions an error? How exactly are non-erroneous
>> collisions resolved? qapi-code-gen.rst needs to answer this.
>
> The strategy implemented in this version is: Collisions are generally an
> error, except for wildcard aliases conflicting with a non-wildcard-alias
> value. In this case the wildcard alias is ignored and the value is
> assumed to belong elsewhere.
Kinds of collisions:
member ordinary alias wildcard alias
member error[1] error[2] member wins[4]
ordinary alias error[3] ordinary wins[4]
wildcard alias ???[5]
[1] Test case duplicate-key demonstrates.
[2] Test case alias-name-conflict demonstrates.
[3] No test case, manual testing results in "alias 'a' collides with
alias 'a'".
[4] Please confirm I got this right.
[5] No test case, manual testing results in no error. What's the
intended behavior?
> If it doesn't belong elsewhere in the end, it still sits in the QDict
> when qobject_input_check_struct() runs, so you get an error.
>
>> Back to the example. If 'type' resolves to ChardevBackend's member, how
>> should I specify SocketAddressLegacy's member? 'addr.type'?
>>
>> Aside: existing -chardev infers SocketAddressLegacy's tag addr.type from
>> the presence of variant members, but that's a separate QAPIfication
>> problem.
>
> So does the new -chardev in its compatibility code, but if you want to
> specify it explicitly, addr.type is what you should use.
>
>> I figure aliases let me refer to these guys at any level I like:
>> 'data.addr.data.FOO', 'data.addr.FOO', 'addr.data.FOO', 'addr.FOO', or
>> just 'FOO'. Except for 'type', where just 'type' means something else.
>> Bizarre...
>
> About as bizarre as shadowing variables in C. The more local one wins.
The part where member 'type' shadows alias 'type' is intentional, and
you're right, it's hardly bizarre, just risks confusion (so does
shadowing in C).
The part where intermediate aliases contaminate the external interface
is an artifact of how we provide the "final" alias. The combination of
these unwanted artifacts with the shadowing feels bizarre to me. Eye of
the beholder, etc., etc.
>> We actually require much less: for QMP chardev-add, we need
>> 'data.addr.data.FOO' and nothing else, and for CLI -chardev, we need
>> 'FOO' and nothing else (I think). The unneeded ones become accidental
>> parts of the external interface. If experience is any guide, we'll have
>> plenty of opportunity to regret such accidents :)
>>
>> Can we somehow restrict external interfaces to what we actually need?
>
> Not reasonably, I would say. Of course, you could try to cover all
> paths with aliases in the top level, but the top level really shouldn't
> have to know about the details of types deep inside some union variants.
>
> The solution for reducing the allowed options here is to work on
> introspection, mark 'data' deprecated everywhere and get rid of the
> simple union nonsense.
Accidental extension of QMP to enable QAPIfication elsewhere would be a
mistake. Elsewhere right now: -chardev.
The knee-jerk short-term solution for QMP is to ignore aliases there
completely. Without introspection, they can't be used seriously anyway.
Of course, we eventually want to use them for evolving QMP, e.g. to
flatten simple unions. The knee-jerk solution sets up another obstacle.
The issue also exists in -chardev with a JSON argument. We can apply
the knee-jerk solution to any JSON-based interface, not just to QMP.
The issue also exists in -chardev with a dotted keys argument. There,
we definitely need the outermost alias (e.g. "host") for compatibility,
and we may want the member ("data.addr.data.host") for symmetry with
JSON. I can't see an argument for exposing the intermediate aliases as
dotted keys, though.
I find the argument "for symmetry with JSON" quite weak. But exposing
the member seems unlikely to create problems later on.
You argue that "the top level really shouldn't have to know about the
details of types deep inside some union variants." That's a valid
argument about the QAPI schema language's support for abstraction. But
the QAPI schema language is means, while the external interfaces are
ends. They come first. A nicer schema language is certainly desirable,
but the niceties shouldn't leak crap into the external interfaces.
Let me work through an example to ground this. Consider chardev-add /
-chardev. Structure of chardev-add's argument:
id: str
backend:
type: enum ChardevBackendKind
data: one of the following, selected by the value of @type:
for socket:
addr:
type: enum SocketAddressType
data: one of the following, selected by the value of @type:
for inet:
host: str
...
...
In contrast, -chardev's argument is flat. To QAPIfy it, we could use
aliases from the root into the object nest:
from type to backend.type
from host to backend.data.addr.data.host
...
We'd certainly design chardev-add's argument differently today, namely
with much less nesting. Say we want to evolve to this structure:
id: str
type: enum ChardevBackendKind
one of the following, selected by the value of @type:
for socket:
addr:
type: enum SocketAddressType
one of the following, selected by the value of @type:
for inet:
host: str
...
...
We obviously need to keep the old structure around for compatibility.
For that, we could use a *different* set of aliases:
from type to backend.type
from addr.host to backend.data.addr.data.host
...
What's the plan for supporting different uses wanting different aliases?
Throw all the aliases together and shake? Then one interface's
requirements will contaminate other interfaces with unwanted aliases.
Getting one interface right is hard enough, having to reason about *all*
QAPI-based interfaces would set us up for failure. And what if we run
into contradictory requirements?
Could we instead tag aliases somehow, pass a tag to the visitor, and
have it ignore aliases with a different tag?
Reminds me of the problem of generating multiple QMPs from a single
schema, e.g. one for qemu-system-FOO, and another one for
qemu-storage-daemon. Inchoate idea: use tags for that somehow. But I
digress.
I'm actually tempted to try how far we can get with just one level of
aliases, i.e. aliases that can only resolve to a member, not to another
alias. I'd expect the code to become quite a bit simpler.
One of the main reasons why this work has such a rough time in review is
that thinking through possible effects make my head explode. I sure
wish I had a stronger head. Weaker effects are commonly easier to get,
though.
I apologize for this wall of text. I've been thinking in writing again.
>> >> > Never skipping wildcard aliases makes the code simpler and results in
>> >> > the expected error message here. So I'll do that for v4.
>> >>
>> >> Trusting your judgement.
>> >>
>> >> > Note that parsing something like '--chardev type=socket,addr.type=unix,
>> >> > path=/tmp/sock,id=c' now depends on the order in the generated code. If
>> >> > the top level 'type' weren't parsed and removed from the input first,
>> >> > visiting 'addr.type' would now detect a conflict. For union types, we
>> >> > know that 'type' is always parsed first, so it's not a problem, but in
>> >> > the general case you need to be careful with the order.
>> >>
>> >> Uff! I think I'd like to understand this better. No need to delay v4
>> >> for that.
>> >>
>> >> Can't yet say whether we need to spell out the limitation in commit
>> >> message and/or documentation.
>> >
>> > The point where we could error out is when parsing SocketAddressLegacy,
>> > because there can be two possible providers for "type".
>> >
>> > The idea in the current code of this series was that we'll just ignore
>> > wildcard aliases if we already have a value, because then the value must
>> > be meant for somewhere else. So it doesn't error out and leaves the
>> > value in the QDict for someone else to pick it up. If nobody picks it
>> > up, it's an error "'foo' is unexpected".
>> >
>> > If we change it and do error out when there are multiple possible values
>> > through wildcard aliases, then the only thing that makes it work is that
>> > ChardevBackend.type is parsed first and is therefore not a possible
>> > value for SocketAddressLegacy.type any more.
>>
>> You wrote you're picking the alternative with the simpler code for v4.
>> Fine with me, as long as it's reasonably easy to explain, in
>> qapi-code-gen.rst.
>
> I think I've come to the conclusion that it's not easy enough to
> explain. As long as the parsing order should remain an implementation
> detail that schema authors shouldn't rely on, it's not possible at all.
>
> It's a pity because the code would have been simpler and it would
> probably have worked for the cases we're interested in. But it doesn't
> work in all hypothetical cases and we can't document the conditions for
> that without making people rely on the parsing order.
The order in which the visitors visit members is not really specified.
The examples in qapi-code-gen.rst show it's in source order, and the
order is quite unlikely to change. So, making it official is not out of
the question.
However, depending on member order for anything feels iffy. I'd like to
be able to shuffle them around without breaking anything.
Thoughts?
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-09-17 8:26 ` Markus Armbruster
@ 2021-09-17 15:03 ` Kevin Wolf
2021-10-02 13:33 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-09-17 15:03 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 17.09.2021 um 10:26 hat Markus Armbruster geschrieben:
> >> When exactly are collisions an error? How exactly are non-erroneous
> >> collisions resolved? qapi-code-gen.rst needs to answer this.
> >
> > The strategy implemented in this version is: Collisions are generally an
> > error, except for wildcard aliases conflicting with a non-wildcard-alias
> > value. In this case the wildcard alias is ignored and the value is
> > assumed to belong elsewhere.
>
> Kinds of collisions:
>
> member ordinary alias wildcard alias
> member error[1] error[2] member wins[4]
> ordinary alias error[3] ordinary wins[4]
> wildcard alias ???[5]
>
> [1] Test case duplicate-key demonstrates.
>
> [2] Test case alias-name-conflict demonstrates.
>
> [3] No test case, manual testing results in "alias 'a' collides with
> alias 'a'".
>
> [4] Please confirm I got this right.
>
> [5] No test case, manual testing results in no error. What's the
> intended behavior?
[5] is going to become "more local wins". In v3 it's "runtime error if
both are specified, except sometimes".
Both [4] and [5] are runtime errors if two values are specified and
there aren't two consumers for them.
> >> We actually require much less: for QMP chardev-add, we need
> >> 'data.addr.data.FOO' and nothing else, and for CLI -chardev, we need
> >> 'FOO' and nothing else (I think). The unneeded ones become accidental
> >> parts of the external interface. If experience is any guide, we'll have
> >> plenty of opportunity to regret such accidents :)
> >>
> >> Can we somehow restrict external interfaces to what we actually need?
> >
> > Not reasonably, I would say. Of course, you could try to cover all
> > paths with aliases in the top level, but the top level really shouldn't
> > have to know about the details of types deep inside some union variants.
> >
> > The solution for reducing the allowed options here is to work on
> > introspection, mark 'data' deprecated everywhere and get rid of the
> > simple union nonsense.
>
> Accidental extension of QMP to enable QAPIfication elsewhere would be a
> mistake. Elsewhere right now: -chardev.
>
> The knee-jerk short-term solution for QMP is to ignore aliases there
> completely. Without introspection, they can't be used seriously anyway.
I would say it's intentional enough. If we can flatten simple unions for
the CLI, why not accept them in QMP, too? (And management tools will
only be happier if they can use the same representation for QMP and
CLI.) I hope that we can get introspection done for 6.2, but even if we
can't, making the case already work shouldn't hurt anyone.
Now you could argue that some aliases to be introduced for -chardev have
no place in QMP because they have no practical use there. But isn't a
consistent QAPI structure on all external interfaces more valuable than
keeping the interface in QMP minimal, but inconsistent with the CLI?
The problem I would generally see with accidental extension of QMP is
that it may restrict future changes for no reason. But if we already
get the restriction because we must stay compatible with the CLI, too,
then this doesn't apply any more.
> Of course, we eventually want to use them for evolving QMP, e.g. to
> flatten simple unions. The knee-jerk solution sets up another obstacle.
>
> The issue also exists in -chardev with a JSON argument. We can apply
> the knee-jerk solution to any JSON-based interface, not just to QMP.
>
> The issue also exists in -chardev with a dotted keys argument. There,
> we definitely need the outermost alias (e.g. "host") for compatibility,
> and we may want the member ("data.addr.data.host") for symmetry with
> JSON. I can't see an argument for exposing the intermediate aliases as
> dotted keys, though.
>
> I find the argument "for symmetry with JSON" quite weak. But exposing
> the member seems unlikely to create problems later on.
Well, my simple argument is: It's hard to get rid of them, so why bother
with extra complexity to get rid of them?
But I think there is a better argument, and "symmetry with JSON"
actually covers support for the intermediate aliases, too:
The alias that flattens SocketAddressLegacy isn't an alias for the
command chardev-add, it's an alias for the type. If you have code that
formats JSON for SocketAddressLegacy, then you should be able to use it
everywhere where a SocketAddressLegacy is required.
So if your code to format JSON for SocketAddressLegacy uses the alias to
provide a flat representation, but the caller producing ChardevBackend
doesn't flatten the union yet, then that should be fine. And if you have
code for flat ChardevBackend, but your common SocketAddressLegacy code
still produces the nesting, then that should be fine, too.
Essentially partial use of aliases in JSON is a feature to allow libvirt
adopting changes incrementally. And just having a mapping of JSON to the
command line is why it should be there in dotted key syntax, too.
> You argue that "the top level really shouldn't have to know about the
> details of types deep inside some union variants." That's a valid
> argument about the QAPI schema language's support for abstraction. But
> the QAPI schema language is means, while the external interfaces are
> ends. They come first. A nicer schema language is certainly desirable,
> but the niceties shouldn't leak crap into the external interfaces.
>
> Let me work through an example to ground this. Consider chardev-add /
> -chardev. Structure of chardev-add's argument:
>
> id: str
> backend:
> type: enum ChardevBackendKind
> data: one of the following, selected by the value of @type:
> for socket:
> addr:
> type: enum SocketAddressType
> data: one of the following, selected by the value of @type:
> for inet:
> host: str
> ...
> ...
>
> In contrast, -chardev's argument is flat. To QAPIfy it, we could use
> aliases from the root into the object nest:
>
> from type to backend.type
> from host to backend.data.addr.data.host
> ...
>
> We'd certainly design chardev-add's argument differently today, namely
> with much less nesting. Say we want to evolve to this structure:
>
> id: str
> type: enum ChardevBackendKind
> one of the following, selected by the value of @type:
> for socket:
> addr:
> type: enum SocketAddressType
> one of the following, selected by the value of @type:
> for inet:
> host: str
> ...
> ...
>
> We obviously need to keep the old structure around for compatibility.
> For that, we could use a *different* set of aliases:
>
> from type to backend.type
> from addr.host to backend.data.addr.data.host
> ...
>
> What's the plan for supporting different uses wanting different aliases?
> Throw all the aliases together and shake? Then one interface's
> requirements will contaminate other interfaces with unwanted aliases.
> Getting one interface right is hard enough, having to reason about *all*
> QAPI-based interfaces would set us up for failure. And what if we run
> into contradictory requirements?
Are there legitimate reasons for exposing the same QAPI type in
different ways on different interfaces? This sounds like a bad idea to
me. If it's the same thing, it should look the same.
The biggest reason for QAPIfying things is to unify interfaces instead
of having different parsers everywhere. Intentionally accepting some
keys only in QMP and others only in the CLI seems to go against this
goal.
> Could we instead tag aliases somehow, pass a tag to the visitor, and
> have it ignore aliases with a different tag?
>
> Reminds me of the problem of generating multiple QMPs from a single
> schema, e.g. one for qemu-system-FOO, and another one for
> qemu-storage-daemon. Inchoate idea: use tags for that somehow. But I
> digress.
>
> I'm actually tempted to try how far we can get with just one level of
> aliases, i.e. aliases that can only resolve to a member, not to another
> alias. I'd expect the code to become quite a bit simpler.
The visitor code would become slightly simpler, but the schema would
become much less maintainable. If someone adds a new field to, say,
InetSocketAddress, review would have to catch this and request that a
new alias be added to ChardevOptions. I don't think this is a realistic
option.
> > I think I've come to the conclusion that it's not easy enough to
> > explain. As long as the parsing order should remain an implementation
> > detail that schema authors shouldn't rely on, it's not possible at all.
> >
> > It's a pity because the code would have been simpler and it would
> > probably have worked for the cases we're interested in. But it doesn't
> > work in all hypothetical cases and we can't document the conditions for
> > that without making people rely on the parsing order.
>
> The order in which the visitors visit members is not really specified.
> The examples in qapi-code-gen.rst show it's in source order, and the
> order is quite unlikely to change. So, making it official is not out of
> the question.
>
> However, depending on member order for anything feels iffy. I'd like to
> be able to shuffle them around without breaking anything.
>
> Thoughts?
No, I agree. As I said it's not simple enough to explain, so I'll just
have the deleted code back and add a bit to improve error reporting. Not
a big deal for me - though it might be one for your review...
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-09-17 15:03 ` Kevin Wolf
@ 2021-10-02 13:33 ` Markus Armbruster
2021-10-04 14:07 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-10-02 13:33 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 17.09.2021 um 10:26 hat Markus Armbruster geschrieben:
[...]
>> >> We actually require much less: for QMP chardev-add, we need
>> >> 'data.addr.data.FOO' and nothing else, and for CLI -chardev, we need
>> >> 'FOO' and nothing else (I think). The unneeded ones become accidental
>> >> parts of the external interface. If experience is any guide, we'll have
>> >> plenty of opportunity to regret such accidents :)
>> >>
>> >> Can we somehow restrict external interfaces to what we actually need?
>> >
>> > Not reasonably, I would say. Of course, you could try to cover all
>> > paths with aliases in the top level, but the top level really shouldn't
>> > have to know about the details of types deep inside some union variants.
>> >
>> > The solution for reducing the allowed options here is to work on
>> > introspection, mark 'data' deprecated everywhere and get rid of the
>> > simple union nonsense.
>>
>> Accidental extension of QMP to enable QAPIfication elsewhere would be a
>> mistake. Elsewhere right now: -chardev.
>>
>> The knee-jerk short-term solution for QMP is to ignore aliases there
>> completely. Without introspection, they can't be used seriously anyway.
>
> I would say it's intentional enough. If we can flatten simple unions for
> the CLI, why not accept them in QMP, too? (And management tools will
> only be happier if they can use the same representation for QMP and
> CLI.) I hope that we can get introspection done for 6.2, but even if we
> can't, making the case already work shouldn't hurt anyone.
>
> Now you could argue that some aliases to be introduced for -chardev have
> no place in QMP because they have no practical use there. But isn't a
> consistent QAPI structure on all external interfaces more valuable than
> keeping the interface in QMP minimal, but inconsistent with the CLI?
>
> The problem I would generally see with accidental extension of QMP is
> that it may restrict future changes for no reason. But if we already
> get the restriction because we must stay compatible with the CLI, too,
> then this doesn't apply any more.
I agree consistency matters. But what exactly should be consistent?
I believe we all agree that a CLI option's *JSON* argument should be
consistent with the corresponding QMP command. This is easy: since both
JSON arguments are fed to the same QAPI machinery, consistency is the
default, and inconsistency would take extra effort.
But what about the dotted keys argument?
One point of view is the difference between the JSON and the dotted keys
argument should be concrete syntax only. Fine print: there may be
arguments dotted keys can't express, but let's ignore that here.
Another point of view is that dotted keys arguments are to JSON
arguments what HMP is to QMP: a (hopefully) human-friendly layer on top,
where we have a certain degree of freedom to improve on the
machine-friendly interface for human use.
Let me try to explain why I believe the latter one makes more sense.
When QAPIfying existing CLI options, the new dotted keys option argument
must be backward compatible to the old option argument, and the QMP
command must also remain backward compatible.
If their abstract syntax is already perfectly consistent, we can simply
use qobject_input_visitor_new_str(). All we have to worry then is the
differences between dotted keys syntax and whatever it replaces (almost
always QemuOpts), which is of no concern to us right now.
What if their abstract syntax differs, as it does for -chardev and
QMP chardev-add?
One solution is to use the *union* of the two languages both for CLI and
QMP. Complicates both. I don't like this. We can try to mitigate by
deprecating unwanted parts of the abstract syntax. More on that below.
If we treat dotted keys as separate, human-friendly layer on top, we can
instead *translate* from one language to the other. This is actually
what -chardev, HMP chardev-add, and all the other HMP commands wrapping
around QMP do today, or close to it.
Different tack. I've been struggling with distilling my intuitions
about the proposed QAPI aliases feature into thought (one reason for me
taking so painfully long to reply). I believe my difficulties are in
part due to a disconnect from use cases: there's a lot aliases could
enable us to do, and I keep getting lost in possibilities. So let's try
to ground this by looking at QAPIfication of -chardev. This way, we
discuss how to solve a specific problem in its entirety instead of
trying to come to a conclusion on a building block that may be useful
(but isn't sufficient) to solve an unclear set of problems (unclear to
*me*).
The equivalent QMP command is chardev-add. The definition of its
argument is spread over several QAPI type definitions. It boils down
to:
id: str
backend: ChardevBackend
type: ChardevBackendKind tag
data: ChardevFile when file
*logfile: str
*logappend: bool
*in: str
out: str
*append: bool
...
data: ChardevSocket when socket
*logfile: str
*logappend: bool
addr: SocketAddressLegacy
type: SocketAddressType tag
data: InetSocketAddress when inet
host: str
port: str
*numeric: bool
...
...
...
This is old stuff; we'd use a lot less nesting today.
The argument of -chardev is flat, as output of -help shows:
...
-chardev socket,id=id[,host=host],port=port[,to=to][,ipv4=on|off][,ipv6=on|off][,nodelay=on|off][,reconnect=seconds]
...
-chardev file,id=id,path=path[,mux=on|off][,logfile=PATH][,logappend=on|off]
...
The historical reason for -chardev being flat is that QemuOpts supports
only flat out of the box. Nested requires gymnastics of the kind we
generally perform only under duress.
There's also a reason for *keeping* it flat: nesting is awful in dotted
keys syntax. With JSON, you simply throw another pair of braces on the
heap. With dotted keys, you get to repeat long key prefixes over and
over.
Let me stitch the two interfaces together:
-chardev file backend.type
,id=id, id
path=path backend.data.out
backend.data.in
backend.data.append
[,mux=on|off]
[,logfile=PATH] backend.data.logfile
[,logappend=on|off] backend.data.logappend
...
-chardev socket backend.type
,id=id id
backend.data.addr.type
[,host=host] backend.data.addr.data.host + default
,port=port backend.data.addr.data.port
backend.data.addr.data.numeric
[,to=to] backend.data.addr.data.to
[,ipv4=on|off] backend.data.addr.data.ipv4
[,ipv6=on|off] backend.data.addr.data.ipv6
backend.data.addr.data.keep-alive
backend.data.addr.data.mptcp
[,nodelay=on|off] backend.data.nodelay
[,reconnect=seconds] backend.data.reconnect
[,server=on|off] backend.data.server
[,wait=on|off] backend.data.wait
[,telnet=on|off] backend.data.telnet
backend.data.tn3270
[,websocket=on|off] backend.data.websocket
[,mux=on|off]
[,logfile=PATH] backend.data.logfile
[,logappend=on|off] backend.data.logappend
[,tls-creds=ID] backend.data.tls-creds
[,tls-authz=ID] backend.data.tls-authz
CLI parameter @mux has nothing in the right column. The code looks like
it's syntactic sugar for something I don't understand. Moving on.
A few things in the right column have nothing in the left column. I can
see three possible cases:
* It's only missing in help. Example: parameter append actually exists,
and corresponds to backend.data.append
* -chardev can't do. Example, maybe: backend.data.in.
* Magic. Example: the value of backend.data.addr.type is derived from
presence of path.
Both -chardev and QMP chardev-add use the same helpers (there are minor
differences that don't matter here). The former basically translates
its flat argument into the latter's arguments. HMP chardev-add is just
like -chardev.
The quickest way to QAPIfy existing -chardev is probably the one we
chose for -object: if the argument is JSON, use the new, QAPI-based
interface, else use the old interface.
Sugar is only available with the old QemuOpts argument.
Feature-parity with QMP is only available with the new JSON argument.
This kicks the "retire QemuOpts" can down the road. The hope is to
eventually weaken the compatibility promise for the non-JSON argument
(like we did for HMP), then switch it to dotted keys. Whether we ditch
the translation from flat to nested then is an open question.
A different problem would be adding a new -chardev somewhere. Not
having to drag all the compatibility baggage along would be nice. On
the other hand, dotted keys without flattening would result in an awful
human interface. Flattening by hand like the existing code does is
unappealing. We could use a bit of help from the QAPI generator.
Perhaps we hate QMP chardev-add's nesting enough to flatten it to
something like
id: str
type: ChardevBackendKind tag
# ChardevFile when file
*logfile: str
*logappend: bool
*in: str
out: str
*append: bool
...
# ChardevSocket when socket
*logfile: str
*logappend: bool
addr: SocketAddress
type: SocketAddressType tag
# InetSocketAddress when inet
host: str
port: str
*numeric: bool
To avoid a compatibility break, we'd actually have to keep the old
structure around, too. Deprecate, drop after grace period expires.
Now the question that matters for this series: how can QAPI aliases help
us with the things discussed above?
1. Aliases can help with flattening input wire formats. Missing parts:
introspection, feature deprecated for aliases.
Note:
* Input: aliases don't affect output.
* Wire format: aliases don't affect the generated C types.
* Aliases can only be "flatter" than the thing they alias. The members
remain nested until drop the old interface.
* Feature deprecated for aliases: straightforward, I think.
* Introspection: requires a schema extension, which means users of
introspection will need an update to actually see aliases.
Corollary: until all of the users that matter are updated, we can't
deprecate members, only aliases. Unfortunately, members is what we
actually *want* to deprecate here.
Why do we need a schema extension? We can't simply expose aliases as
if they were members, because that would look as if you had to specify
both (which is actually an error).
When we drop the old interface, the old, deprecated members go away,
and the aliases morph into members. Well-behaved users of
introspection should deal with the morphing. Still, there's a risk.
As much as I hate excessive nesting, I'm not sure flattening old
interfaces such as QMP chardev-add is worth the trouble. More on that
below, in reply to your arguments.
Flattening could reduce the difference to the current, non-QAPIfied
-chardev, though, which could help with QAPIfying it. How exactly
remains unclear, at least to me, especially for sugar and magic.
2. Aliases can help with adding flatter options to existing input wire
formats. This is 1. less the hope to get rid of the nested option some
day. Missing parts: introspection. Old introspection users simply
can't see the flat option, which is fine.
Is adding a flat option to old interfaces such as QMP chardev-add worth
the trouble? If you have to check whether the flat option is available
before you can use it, and else fall back to the nested option, always
using the nested option is simpler.
I figure the most likely user of these aliases would be a QAPIfied
-chardev. That use is legitimate. But if it remains the only one, all
the extra work to expose them in the external interface in a consumable
way (the "missing parts" above) is wasted.
3. Aliases might help with creating flatter variations of old input wire
formats. This is 2. plus a way to limit each interface to one option.
Missing parts: a way to limit, introspection.
Note:
* "A way to limit" is too vague to be useful. Needs design work.
Stupidest solution that could possibly work: hard-code two options,
with and without aliases.
* Introspection is needed only for machine-friendly interfaces.
Implementing it might be expensive.
If we use the flat variation just internally, say for -chardev's dotted
keys argument, then introspection is not needed. We'd use "with
aliases" just for translating -chardev's dotted keys argument.
We could also use the flat variation to create alternate, more modern
interfaces with much less code duplication in the schema, and much less
boring translation code between the differently nested duplicates.
Example: we have SocketAddress for use by "modern" interfaces, and
SocketAddressLegacy strictly for existing ones. We have hand-written
code to convert SocketAddressLegacy to SocketAddress. Not terrible for
a simple type like SocketAddress, but imagine doing the same for
flattening ChardevBackend.
>> Of course, we eventually want to use them for evolving QMP, e.g. to
>> flatten simple unions. The knee-jerk solution sets up another obstacle.
>>
>> The issue also exists in -chardev with a JSON argument. We can apply
>> the knee-jerk solution to any JSON-based interface, not just to QMP.
>>
>> The issue also exists in -chardev with a dotted keys argument. There,
>> we definitely need the outermost alias (e.g. "host") for compatibility,
>> and we may want the member ("data.addr.data.host") for symmetry with
>> JSON. I can't see an argument for exposing the intermediate aliases as
>> dotted keys, though.
>>
>> I find the argument "for symmetry with JSON" quite weak. But exposing
>> the member seems unlikely to create problems later on.
>
> Well, my simple argument is: It's hard to get rid of them, so why bother
> with extra complexity to get rid of them?
It's either that, or writing documentation explaining the now many ways
to do the same thing.
> But I think there is a better argument, and "symmetry with JSON"
> actually covers support for the intermediate aliases, too:
>
> The alias that flattens SocketAddressLegacy isn't an alias for the
> command chardev-add, it's an alias for the type. If you have code that
> formats JSON for SocketAddressLegacy, then you should be able to use it
> everywhere where a SocketAddressLegacy is required.
>
> So if your code to format JSON for SocketAddressLegacy uses the alias to
> provide a flat representation, but the caller producing ChardevBackend
> doesn't flatten the union yet, then that should be fine. And if you have
> code for flat ChardevBackend, but your common SocketAddressLegacy code
> still produces the nesting, then that should be fine, too.
>
> Essentially partial use of aliases in JSON is a feature to allow libvirt
> adopting changes incrementally.
This is about evolving existing JSON-based interfaces (QMP, basically)
compatibly.
To be able to use aliases for this, we need to supply missing parts
(detailed above), and they are anything but trivial. Users of
introspection need updates, too.
I hate excessive nesting, I really do. I'd love to see excessively
nested existing interfaces cleaned up. Sadly, the paths towards that
goal I can see feel too costly just for cleanliness.
> And just having a mapping of JSON to the
> command line is why it should be there in dotted key syntax, too.
We don't have a mapping from HMP to QMP, only the other way round.
I believe we should make CLI with dotted keys like HMP (not least
because it already is for many complex options). Then we don't have a
mapping from QMP to CLI with dotted keys. We do have a (trivial!)
mapping to CLI with JSON.
>> You argue that "the top level really shouldn't have to know about the
>> details of types deep inside some union variants." That's a valid
>> argument about the QAPI schema language's support for abstraction. But
>> the QAPI schema language is means, while the external interfaces are
>> ends. They come first. A nicer schema language is certainly desirable,
>> but the niceties shouldn't leak crap into the external interfaces.
>>
>> Let me work through an example to ground this. Consider chardev-add /
>> -chardev. Structure of chardev-add's argument:
>>
>> id: str
>> backend:
>> type: enum ChardevBackendKind
>> data: one of the following, selected by the value of @type:
>> for socket:
>> addr:
>> type: enum SocketAddressType
>> data: one of the following, selected by the value of @type:
>> for inet:
>> host: str
>> ...
>> ...
>>
>> In contrast, -chardev's argument is flat. To QAPIfy it, we could use
>> aliases from the root into the object nest:
>>
>> from type to backend.type
>> from host to backend.data.addr.data.host
>> ...
>>
>> We'd certainly design chardev-add's argument differently today, namely
>> with much less nesting. Say we want to evolve to this structure:
>>
>> id: str
>> type: enum ChardevBackendKind
>> one of the following, selected by the value of @type:
>> for socket:
>> addr:
>> type: enum SocketAddressType
>> one of the following, selected by the value of @type:
>> for inet:
>> host: str
>> ...
>> ...
>>
>> We obviously need to keep the old structure around for compatibility.
>> For that, we could use a *different* set of aliases:
>>
>> from type to backend.type
>> from addr.host to backend.data.addr.data.host
>> ...
>>
>> What's the plan for supporting different uses wanting different aliases?
>> Throw all the aliases together and shake? Then one interface's
>> requirements will contaminate other interfaces with unwanted aliases.
>> Getting one interface right is hard enough, having to reason about *all*
>> QAPI-based interfaces would set us up for failure. And what if we run
>> into contradictory requirements?
>
> Are there legitimate reasons for exposing the same QAPI type in
> different ways on different interfaces? This sounds like a bad idea to
> me. If it's the same thing, it should look the same.
>
> The biggest reason for QAPIfying things is to unify interfaces instead
> of having different parsers everywhere. Intentionally accepting some
> keys only in QMP and others only in the CLI seems to go against this
> goal.
Valid argument. CLI with JSON argument should match QMP. Even when
that means both are excessively nested.
CLI with dotted keys is a different matter, in my opinion.
>> Could we instead tag aliases somehow, pass a tag to the visitor, and
>> have it ignore aliases with a different tag?
>>
>> Reminds me of the problem of generating multiple QMPs from a single
>> schema, e.g. one for qemu-system-FOO, and another one for
>> qemu-storage-daemon. Inchoate idea: use tags for that somehow. But I
>> digress.
>>
>> I'm actually tempted to try how far we can get with just one level of
>> aliases, i.e. aliases that can only resolve to a member, not to another
>> alias. I'd expect the code to become quite a bit simpler.
>
> The visitor code would become slightly simpler, but the schema would
> become much less maintainable. If someone adds a new field to, say,
> InetSocketAddress, review would have to catch this and request that a
> new alias be added to ChardevOptions. I don't think this is a realistic
> option.
Shouldn't a wildcard alias in ChardevOptions take care of that?
[...]
I apologize for this wall of text. It's a desparate attempt to cut
through the complexity and my confusion, and make sense of the actual
problems we're trying to solve.
So, what problems exactly are we trying to solve?
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-02 13:33 ` Markus Armbruster
@ 2021-10-04 14:07 ` Kevin Wolf
2021-10-05 13:49 ` Markus Armbruster
2021-10-11 7:44 ` Markus Armbruster
0 siblings, 2 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-10-04 14:07 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 02.10.2021 um 15:33 hat Markus Armbruster geschrieben:
> I apologize for this wall of text. It's a desparate attempt to cut
> through the complexity and my confusion, and make sense of the actual
> problems we're trying to solve.
>
> So, what problems exactly are we trying to solve?
I'll start with replying to your final question because I think it's
more helpful to start with the big picture than with details.
So tools like libvirt want to have a single consistent interface to
configure things on startup and at runtime. We also intend to support
configuration files that should this time support all of the options and
not just a few chosen ones.
The hypothesis is that QAPIfying the command line is the correct
solution for both of these problems, i.e. all available command line
options must be present in the QAPI schema and will be processed by a
single parser shared with QMP to make sure they are consistent.
Adding QAPIfied versions of individual command line options are steps
towards this goal. As soon as they exist for every option, the final
conversion from an open coded getopt() loop (or in fact a hand crafted
parser in the case of vl.c) to something generated from the QAPI schema
should be reasonably easy.
You're right that adding a second JSON-based command line interface for
every option can already achieve the goal of providing a unified
external interface, at the cost of (interface and code) duplication. Is
this duplication desirable? Certainly no. Is it acceptable? You might
get different opinions on this one.
In my opinion, we should try to get rid of hand crafted parsers where
it's reasonably possible, and take advantage of the single unified
option structure that QAPI provides. -chardev specifically has a hand
crafted parser that essentially duplicates the automatically generated
QAPI visitor code, except for the nesting and some differences in option
names.
Aliases are one tool that can help bridge these differences in a generic
way with minimal effort in the individual case. They are not _necessary_
to solve the problem; we could instead just use manually written code to
manipulate input QDicts so that QAPI visitors accept them. Even with
aliases, there are a few things left in the chardev code that are
converted this way. Aliases just greatly reduce the amount of this code
and make the conversion declarative instead.
Now a key point in the previous paragraph is that aliases add a generic
way to do this. So even if they are immediately motivated by -chardev,
it might be worth looking at other cases they could enable if you think
that -chardev alone isn't sufficient justification to have this tool.
I guess this is the point where things become a bit less clear because
people are just hand waving with vague ideas for additional uses.
Do we need to invest more thought on these other cases? We probably do
if it makes a difference for the answer to the question whether we want
to add aliases to our toolbox. Does it?
> But what about the dotted keys argument?
>
> One point of view is the difference between the JSON and the dotted keys
> argument should be concrete syntax only. Fine print: there may be
> arguments dotted keys can't express, but let's ignore that here.
>
> Another point of view is that dotted keys arguments are to JSON
> arguments what HMP is to QMP: a (hopefully) human-friendly layer on top,
> where we have a certain degree of freedom to improve on the
> machine-friendly interface for human use.
This doesn't feel unreasonable because with HMP, there is precedence.
But this is not all what we have used dotted key syntax for so far. I'm
not against making a change there if we feel it makes sense, but we
should be clear that it is a change.
I think we all agree that -blockdev isn't human-friendly. Dotted key
syntax makes it humanly possible to use it on the command line, but
friendly is something else entirely. It is still a 1:1 mapping of QMP to
the command line, and this is how we advertised it, too. This has some
important advantages, like we don't have a separate parser for a
human-friendly syntax, but the generic keyval parser covers it.
HMP in contrast means separate code for every command, or translated to
the CLI for each command line option. HMP is not defined in QAPI, it's
a separate thing that just happens to call into QAPI-based QMP code in
the common case.
Is this what you have in mind for non-JSON command line options?
What I had in mind was using the schema to generate the necessary code,
using the generic keyval parser everywhere, and just providing a hook
where the QDict could be modified after the keyval parser and before the
visitor. Most command line options would not have any explicit code for
parsing input, only the declarative schema and the final option handler
getting a QAPI type (which could actually be the corresponding QMP
command handler in the common case).
I think these are very different ideas. If we want HMP-like, then all
the keyval based code paths we've built so far don't make much sense.
> Both -chardev and QMP chardev-add use the same helpers (there are minor
> differences that don't matter here). The former basically translates
> its flat argument into the latter's arguments. HMP chardev-add is just
> like -chardev.
>
> The quickest way to QAPIfy existing -chardev is probably the one we
> chose for -object: if the argument is JSON, use the new, QAPI-based
> interface, else use the old interface.
-chardev doesn't quite translate into chardev-add arguments. Converting
into the input of a QMP command normally means providing a QDict that
can be accepted by the QAPI-generated visitor code, and then using that
QAPI parser to create the C object. What -chardev does instead is using
an entirely separate parser to create the C object directly, without
going through any QAPI code.
After QAPIfication, both code paths go through the QAPI code and undergo
the same validations etc.
If we still have code paths that completely bypass QAPI, it's hard to
call the command line option fully QAPIfied. Of course, if you don't
care about duplication and inconsistencies between the interfaces,
though, and just want to achieve the initially stated goal of providing
at least one interface consistent between QMP and CLI (besides others)
and a config file, then it might be QAPIfied enough.
> Now the question that matters for this series: how can QAPI aliases
> help us with the things discussed above?
The translation needs to be implemented somehow. The obvious way is just
writing, reviewing and maintaining code that manually translates. (We
don't have this code yet for a fully QAPIfied -chardev. We only have
code that bypasses QAPI, i.e. translates to the wrong target format.)
Aliases help because they allow reducing the amount of code in favour of
data, the alias declarations in the schema.
> If we use the flat variation just internally, say for -chardev's dotted
> keys argument, then introspection is not needed. We'd use "with
> aliases" just for translating -chardev's dotted keys argument.
Note that we're doing more translations that just flattening with
aliases, some options have different names in QMP and the CLI.
But either way, yes, alias support could be optional so that you need to
explicitly enable it when creating a visitor, and then only the CLI code
could actually enable it.
Limits the possible future uses for the new tool in our toolbox, but is
sufficient for -chardev. We can always extend support for it later
if/when we actually want to make use of it outside of the CLI.
> Valid argument. CLI with JSON argument should match QMP. Even when
> that means both are excessively nested.
>
> CLI with dotted keys is a different matter, in my opinion.
I'm skipping quite a bit of text here, mainly because I think we need an
answer first if we really want to switch the keyval options to be
HMP-like in more than just the support status (see above).
Obviously, the other relevant question would then be if it also ends up
like HMP in that most interesting functionality isn't even available and
you're forced to use QMP/JSON when you're serious.
I guess I can go into more details once we have answered these
fundamental questions.
(We could and should have talked about them and about whether you want
to have aliases at all a year ago, before going into detailed code
review and making error messages perfect in code that we might throw
away anyway...)
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-04 14:07 ` Kevin Wolf
@ 2021-10-05 13:49 ` Markus Armbruster
2021-10-05 17:05 ` Kevin Wolf
2021-10-11 7:44 ` Markus Armbruster
1 sibling, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-10-05 13:49 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 02.10.2021 um 15:33 hat Markus Armbruster geschrieben:
>> I apologize for this wall of text. It's a desparate attempt to cut
>> through the complexity and my confusion, and make sense of the actual
>> problems we're trying to solve.
>>
>> So, what problems exactly are we trying to solve?
>
> I'll start with replying to your final question because I think it's
> more helpful to start with the big picture than with details.
>
> So tools like libvirt want to have a single consistent interface to
> configure things on startup and at runtime. We also intend to support
> configuration files that should this time support all of the options and
> not just a few chosen ones.
Yes.
> The hypothesis is that QAPIfying the command line is the correct
> solution for both of these problems, i.e. all available command line
> options must be present in the QAPI schema and will be processed by a
> single parser shared with QMP to make sure they are consistent.
Yes.
This leads us to JSON option arguments and configuration files.
Well-suited for management applications that already use QMP.
> Adding QAPIfied versions of individual command line options are steps
> towards this goal. As soon as they exist for every option, the final
> conversion from an open coded getopt() loop (or in fact a hand crafted
> parser in the case of vl.c) to something generated from the QAPI schema
> should be reasonably easy.
Yes.
> You're right that adding a second JSON-based command line interface for
> every option can already achieve the goal of providing a unified
> external interface, at the cost of (interface and code) duplication. Is
> this duplication desirable? Certainly no. Is it acceptable? You might
> get different opinions on this one.
We'd certainly prefer CLI options to match corresponding QMP commands
exactly.
Unfortunately, existing CLI options deviate from corresponding QMP
commands, and existing CLI options without corresponding QMP commands
may violate QMP design rules.
Note: these issues pertain to human-friendly option syntax. The
machine-friendly option syntax is still limited to just a few options,
and it does match QMP there.
> In my opinion, we should try to get rid of hand crafted parsers where
> it's reasonably possible, and take advantage of the single unified
> option structure that QAPI provides. -chardev specifically has a hand
> crafted parser that essentially duplicates the automatically generated
> QAPI visitor code, except for the nesting and some differences in option
> names.
We should definitely parse JSON option arguments with the QAPI
machinery, and nothing more.
Parsing human-friendly arguments with it is desirable, but the need for
backward compatibility can make it difficult. Even where compatibility
is of no concern, simply swapping concrete JSON syntax for dotted keys
may result in human interfaces that are less than friendly.
Are we in agreement that this is the problem at hand?
> Aliases are one tool that can help bridge these differences in a generic
> way with minimal effort in the individual case. They are not _necessary_
> to solve the problem; we could instead just use manually written code to
> manipulate input QDicts so that QAPI visitors accept them. Even with
> aliases, there are a few things left in the chardev code that are
> converted this way. Aliases just greatly reduce the amount of this code
> and make the conversion declarative instead.
Understood.
> Now a key point in the previous paragraph is that aliases add a generic
> way to do this. So even if they are immediately motivated by -chardev,
> it might be worth looking at other cases they could enable if you think
> that -chardev alone isn't sufficient justification to have this tool.
> I guess this is the point where things become a bit less clear because
> people are just hand waving with vague ideas for additional uses.
>
> Do we need to invest more thought on these other cases? We probably do
> if it makes a difference for the answer to the question whether we want
> to add aliases to our toolbox. Does it?
I hope we can make a case for aliases without looking beyond CLI
QAPIfication. That's a wide field already, with enough opportunity to
get lost in details.
If we later put aliases to other uses, we might have to adapt them some.
That's okay. Designing for one problem we have and understand has a
much better chance of success than trying to design for all problems we
might have.
There are many CLI options to be QAPIfied. -chardev is one of the more
thornier ones, which makes it a useful example.
>> But what about the dotted keys argument?
>>
>> One point of view is the difference between the JSON and the dotted keys
>> argument should be concrete syntax only. Fine print: there may be
>> arguments dotted keys can't express, but let's ignore that here.
>>
>> Another point of view is that dotted keys arguments are to JSON
>> arguments what HMP is to QMP: a (hopefully) human-friendly layer on top,
>> where we have a certain degree of freedom to improve on the
>> machine-friendly interface for human use.
>
> This doesn't feel unreasonable because with HMP, there is precedence.
> But this is not all what we have used dotted key syntax for so far. I'm
> not against making a change there if we feel it makes sense, but we
> should be clear that it is a change.
Our first uses of dotted keys were green fields. Sticking to QAPI/QMP's
abstract syntax was simplest.
> I think we all agree that -blockdev isn't human-friendly. Dotted key
> syntax makes it humanly possible to use it on the command line, but
> friendly is something else entirely. It is still a 1:1 mapping of QMP to
> the command line, and this is how we advertised it, too.
Yes.
> This has some
> important advantages, like we don't have a separate parser for a
> human-friendly syntax, but the generic keyval parser covers it.
There are two generic parts in play: the keyval parser, and the QAPI
input visitor.
Using identical abstract syntax both for JSON and dotted keys arguments
makes use of both parts simple: pass the argument to
qobject_input_visitor_new_str() to create a visitor, then visit the QAPI
type with it.
When abstract syntax differs, using the keyval parser is still simple,
but to use the QAPI input visitor, we need separate QAPI types. Massive
code duplication in the QAPI schema.
To avoid the duplication, we can instead translate the parse tree
produced for the dotted keys argument. Plenty of boring code.
In short: yes, using the same abstract syntax for both has advantages.
> HMP in contrast means separate code for every command, or translated to
> the CLI for each command line option. HMP is not defined in QAPI, it's
> a separate thing that just happens to call into QAPI-based QMP code in
> the common case.
>
> Is this what you have in mind for non-JSON command line options?
We should separate the idea "HMP wraps around QMP" from its
implementation as handwritten, boilerplate-heavy code.
All but the latest, QAPI-based CLI options work pretty much like HMP: a
bit of code generation (hxtool), a mix of common and ad hoc parsers,
boring handwritten code to translate from parse tree to internal C
interfaces (which are often QMP command handlers), and to stitch it all
together.
Reducing the amount of boring handwritten code is equally desirable for
HMP and CLI.
So is specifying the interface in QAPI instead of older, less powerful
schema languages like hxtool and QemuOpts.
There are at least three problems:
1. Specifying CLI in QAPI hasn't progressed beyond the proof-of-concept
stage.
2. Backward compatibility issues and doubts have defeated attempts to
replace gnarly stuff, in particular QemuOpts.
3. How to best bridge abstract syntax differences has been unclear.
Whether the differences are historical accidents or intentional for ease
of human use doesn't matter.
Aliases feel like a contribution towards solving 3.
I don't think 1. is particularly difficult. It's just a big chunk of
work that isn't really useful without solutions for 2. and 3.
To me, 2. feels intractable head on. Perhaps we better bypass the
problem by weakening the compatibility promise like we did for HMP.
> What I had in mind was using the schema to generate the necessary code,
> using the generic keyval parser everywhere, and just providing a hook
> where the QDict could be modified after the keyval parser and before the
> visitor. Most command line options would not have any explicit code for
> parsing input, only the declarative schema and the final option handler
> getting a QAPI type (which could actually be the corresponding QMP
> command handler in the common case).
>
> I think these are very different ideas. If we want HMP-like, then all
> the keyval based code paths we've built so far don't make much sense.
I'm not sure the differences are "very" :)
I think you start at "human-friendly and machine-friendly should use the
abstract syntax defined in the QAPI schema, except where human-friendly
has to differ, say for backward compatibility".
This approach considers differences a blemish. The "standard" abstract
syntax (the one defined in the QAPI schema) should be accepted in
addition to the "alternate" one whenever possible, so "modern" use can
avoid the blemishes, and blemishes can be removed once they have fallen
out of use.
"Alternate" should not be limited to human-friendly. Not limiting keeps
things consistent, which is good, because differences are bad.
Is that a fair description?
I start at "human-friendly syntax should be as friendly as we can make
it, except where we have to compromise, say for backward compatibility".
This approach embraces differences. Mind, not differences just for
differences sake, only where they help users.
Additionally accepting the "standard" abstract syntax is useful only
where it helps users.
"Alternate" should be limited to human-friendly.
Different approaches, without doubt. Yet both have to deal with
differences, and both could use similar techniques and machinery for
that. You're proposing to do the bulk of the work with aliases, and to
have a tree-transforming hook for the remainder. Makes sense to me.
However, in sufficiently gnarly cases, we might have to bypass all this
and keep using handwritten code until the backward compatibility promise
is history: see 2. above.
In addition each approach has its own, special needs.
Yours adds "alternate" syntax to QMP. This poses documentation and
introspection problems. The introspection solutions will then
necessitate management application updates.
Mine trades these problems for others, namely how to generate parsers
for two different syntaxes from one QAPI schema.
Do I make sense?
>> Both -chardev and QMP chardev-add use the same helpers (there are minor
>> differences that don't matter here). The former basically translates
>> its flat argument into the latter's arguments. HMP chardev-add is just
>> like -chardev.
>>
>> The quickest way to QAPIfy existing -chardev is probably the one we
>> chose for -object: if the argument is JSON, use the new, QAPI-based
>> interface, else use the old interface.
>
> -chardev doesn't quite translate into chardev-add arguments. Converting
> into the input of a QMP command normally means providing a QDict that
> can be accepted by the QAPI-generated visitor code, and then using that
> QAPI parser to create the C object. What -chardev does instead is using
> an entirely separate parser to create the C object directly, without
> going through any QAPI code.
Yes, and that's quite some extra code, with plenty of opportunity for
inconsistency.
> After QAPIfication, both code paths go through the QAPI code and undergo
> the same validations etc.
>
> If we still have code paths that completely bypass QAPI, it's hard to
> call the command line option fully QAPIfied. Of course, if you don't
> care about duplication and inconsistencies between the interfaces,
> though, and just want to achieve the initially stated goal of providing
> at least one interface consistent between QMP and CLI (besides others)
> and a config file, then it might be QAPIfied enough.
Reworking human-friendly -chardev to target QMP instead of a C interface
shared with QMP would be nice. Just because prior attempts to replace
gnarly stuff compatibly have failed doesn't mean this one must fail.
>> Now the question that matters for this series: how can QAPI aliases
>> help us with the things discussed above?
>
> The translation needs to be implemented somehow. The obvious way is just
> writing, reviewing and maintaining code that manually translates. (We
> don't have this code yet for a fully QAPIfied -chardev. We only have
> code that bypasses QAPI, i.e. translates to the wrong target format.)
>
> Aliases help because they allow reducing the amount of code in favour of
> data, the alias declarations in the schema.
Understood.
>> If we use the flat variation just internally, say for -chardev's dotted
>> keys argument, then introspection is not needed. We'd use "with
>> aliases" just for translating -chardev's dotted keys argument.
>
> Note that we're doing more translations that just flattening with
> aliases, some options have different names in QMP and the CLI.
>
> But either way, yes, alias support could be optional so that you need to
> explicitly enable it when creating a visitor, and then only the CLI code
> could actually enable it.
>
> Limits the possible future uses for the new tool in our toolbox, but is
> sufficient for -chardev. We can always extend support for it later
> if/when we actually want to make use of it outside of the CLI.
Yes.
>> Valid argument. CLI with JSON argument should match QMP. Even when
>> that means both are excessively nested.
>>
>> CLI with dotted keys is a different matter, in my opinion.
>
> I'm skipping quite a bit of text here, mainly because I think we need an
> answer first if we really want to switch the keyval options to be
> HMP-like in more than just the support status (see above).
>
> Obviously, the other relevant question would then be if it also ends up
> like HMP in that most interesting functionality isn't even available and
> you're forced to use QMP/JSON when you're serious.
I figure this is because we have let "QMP first" degenerate into "QMP
first, HMP never" sometimes, and we did so mostly because HMP is extra
work that was hard to justify.
What if you could get the 1:1 HMP command basically for free? It may
not be the most human-friendly command possible. But it should be a
start.
> I guess I can go into more details once we have answered these
> fundamental questions.
>
> (We could and should have talked about them and about whether you want
> to have aliases at all a year ago, before going into detailed code
> review and making error messages perfect in code that we might throw
> away anyway...)
I doubt you could possibly be more right than that!
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-05 13:49 ` Markus Armbruster
@ 2021-10-05 17:05 ` Kevin Wolf
2021-10-06 13:11 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-10-05 17:05 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 05.10.2021 um 15:49 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
>
> > Am 02.10.2021 um 15:33 hat Markus Armbruster geschrieben:
> >> I apologize for this wall of text. It's a desparate attempt to cut
> >> through the complexity and my confusion, and make sense of the actual
> >> problems we're trying to solve.
> >>
> >> So, what problems exactly are we trying to solve?
> >
> > I'll start with replying to your final question because I think it's
> > more helpful to start with the big picture than with details.
> >
> > So tools like libvirt want to have a single consistent interface to
> > configure things on startup and at runtime. We also intend to support
> > configuration files that should this time support all of the options and
> > not just a few chosen ones.
>
> Yes.
>
> > The hypothesis is that QAPIfying the command line is the correct
> > solution for both of these problems, i.e. all available command line
> > options must be present in the QAPI schema and will be processed by a
> > single parser shared with QMP to make sure they are consistent.
>
> Yes.
>
> This leads us to JSON option arguments and configuration files.
> Well-suited for management applications that already use QMP.
>
> > Adding QAPIfied versions of individual command line options are steps
> > towards this goal. As soon as they exist for every option, the final
> > conversion from an open coded getopt() loop (or in fact a hand crafted
> > parser in the case of vl.c) to something generated from the QAPI schema
> > should be reasonably easy.
>
> Yes.
>
> > You're right that adding a second JSON-based command line interface for
> > every option can already achieve the goal of providing a unified
> > external interface, at the cost of (interface and code) duplication. Is
> > this duplication desirable? Certainly no. Is it acceptable? You might
> > get different opinions on this one.
>
> We'd certainly prefer CLI options to match corresponding QMP commands
> exactly.
>
> Unfortunately, existing CLI options deviate from corresponding QMP
> commands, and existing CLI options without corresponding QMP commands
> may violate QMP design rules.
>
> Note: these issues pertain to human-friendly option syntax. The
> machine-friendly option syntax is still limited to just a few options,
> and it does match QMP there.
On the other hand, we only have a handful of existing options that are
very complex. Most of them aren't even structured and just take a single
scalar value. So I'm relatively sure that we know the big ones, and
we're working on them.
Another point here is that I would argue that there is a difference
between existing options and human-friendly options. Not all existing
options are human-friendly just because they aren't machine friendly
either, and there is probably potential for human-friendly syntax that
doesn't exist yet.
So I would treat support for existing options (i.e. compatibility) and
support for human-friendly options (i.e. usability) as two separate
problems.
> > In my opinion, we should try to get rid of hand crafted parsers where
> > it's reasonably possible, and take advantage of the single unified
> > option structure that QAPI provides. -chardev specifically has a hand
> > crafted parser that essentially duplicates the automatically generated
> > QAPI visitor code, except for the nesting and some differences in option
> > names.
>
> We should definitely parse JSON option arguments with the QAPI
> machinery, and nothing more.
Yes, no doubt. (And we don't even consistently do that yet, like
device-add going through QemuOpts after going through QAPI and throwing
away all type information and silently ignoring anything it doesn't know
to handle.)
> Parsing human-friendly arguments with it is desirable, but the need for
> backward compatibility can make it difficult. Even where compatibility
> is of no concern, simply swapping concrete JSON syntax for dotted keys
> may result in human interfaces that are less than friendly.
>
> Are we in agreement that this is the problem at hand?
As far as I am concerned, compatibility is the problem at hand,
usability isn't.
This doesn't mean that I'm opposed to having actually human friendly
options, quite the opposite. But getting machine friendly options is
already a big project. Making human interfaces friendlier would be
adding another project of similar size, and I don't feel like tackling
a second project at the same time when the first one is already hard.
> > Aliases are one tool that can help bridge these differences in a generic
> > way with minimal effort in the individual case. They are not _necessary_
> > to solve the problem; we could instead just use manually written code to
> > manipulate input QDicts so that QAPI visitors accept them. Even with
> > aliases, there are a few things left in the chardev code that are
> > converted this way. Aliases just greatly reduce the amount of this code
> > and make the conversion declarative instead.
>
> Understood.
>
> > Now a key point in the previous paragraph is that aliases add a generic
> > way to do this. So even if they are immediately motivated by -chardev,
> > it might be worth looking at other cases they could enable if you think
> > that -chardev alone isn't sufficient justification to have this tool.
> > I guess this is the point where things become a bit less clear because
> > people are just hand waving with vague ideas for additional uses.
> >
> > Do we need to invest more thought on these other cases? We probably do
> > if it makes a difference for the answer to the question whether we want
> > to add aliases to our toolbox. Does it?
>
> I hope we can make a case for aliases without looking beyond CLI
> QAPIfication. That's a wide field already, with enough opportunity to
> get lost in details.
>
> If we later put aliases to other uses, we might have to adapt them some.
> That's okay. Designing for one problem we have and understand has a
> much better chance of success than trying to design for all problems we
> might have.
>
> There are many CLI options to be QAPIfied. -chardev is one of the more
> thornier ones, which makes it a useful example.
Good, I agree.
> >> But what about the dotted keys argument?
> >>
> >> One point of view is the difference between the JSON and the dotted keys
> >> argument should be concrete syntax only. Fine print: there may be
> >> arguments dotted keys can't express, but let's ignore that here.
> >>
> >> Another point of view is that dotted keys arguments are to JSON
> >> arguments what HMP is to QMP: a (hopefully) human-friendly layer on top,
> >> where we have a certain degree of freedom to improve on the
> >> machine-friendly interface for human use.
> >
> > This doesn't feel unreasonable because with HMP, there is precedence.
> > But this is not all what we have used dotted key syntax for so far. I'm
> > not against making a change there if we feel it makes sense, but we
> > should be clear that it is a change.
>
> Our first uses of dotted keys were green fields. Sticking to QAPI/QMP's
> abstract syntax was simplest.
>
> > I think we all agree that -blockdev isn't human-friendly. Dotted key
> > syntax makes it humanly possible to use it on the command line, but
> > friendly is something else entirely. It is still a 1:1 mapping of QMP to
> > the command line, and this is how we advertised it, too.
>
> Yes.
>
> > This has some
> > important advantages, like we don't have a separate parser for a
> > human-friendly syntax, but the generic keyval parser covers it.
>
> There are two generic parts in play: the keyval parser, and the QAPI
> input visitor.
Fair point.
> Using identical abstract syntax both for JSON and dotted keys arguments
> makes use of both parts simple: pass the argument to
> qobject_input_visitor_new_str() to create a visitor, then visit the QAPI
> type with it.
>
> When abstract syntax differs, using the keyval parser is still simple,
> but to use the QAPI input visitor, we need separate QAPI types. Massive
> code duplication in the QAPI schema.
>
> To avoid the duplication, we can instead translate the parse tree
> produced for the dotted keys argument. Plenty of boring code.
>
> In short: yes, using the same abstract syntax for both has advantages.
Even with duplication in the schema, you still have to translate unless
you want to duplicate all of the logic, too. The difference is just that
instead of translating between QObjects, you would be translating
between C structs.
So yes, using different abstract syntax means translation, which in turn
can mean plenty of boring code (hopefully - might also be buggy rather
than boring) if we don't support ways to automate the conversion.
> > HMP in contrast means separate code for every command, or translated to
> > the CLI for each command line option. HMP is not defined in QAPI, it's
> > a separate thing that just happens to call into QAPI-based QMP code in
> > the common case.
> >
> > Is this what you have in mind for non-JSON command line options?
>
> We should separate the idea "HMP wraps around QMP" from its
> implementation as handwritten, boilerplate-heavy code.
>
> All but the latest, QAPI-based CLI options work pretty much like HMP: a
> bit of code generation (hxtool), a mix of common and ad hoc parsers,
> boring handwritten code to translate from parse tree to internal C
> interfaces (which are often QMP command handlers), and to stitch it all
> together.
>
> Reducing the amount of boring handwritten code is equally desirable for
> HMP and CLI.
>
> So is specifying the interface in QAPI instead of older, less powerful
> schema languages like hxtool and QemuOpts.
>
> There are at least three problems:
>
> 1. Specifying CLI in QAPI hasn't progressed beyond the proof-of-concept
> stage.
>
> 2. Backward compatibility issues and doubts have defeated attempts to
> replace gnarly stuff, in particular QemuOpts.
>
> 3. How to best bridge abstract syntax differences has been unclear.
> Whether the differences are historical accidents or intentional for ease
> of human use doesn't matter.
>
> Aliases feel like a contribution towards solving 3.
>
> I don't think 1. is particularly difficult. It's just a big chunk of
> work that isn't really useful without solutions for 2. and 3.
>
> To me, 2. feels intractable head on. Perhaps we better bypass the
> problem by weakening the compatibility promise like we did for HMP.
Can we define 2 in a bit more specific terms? I feel the biggest part
of 2 is actually 3.
You mention QemuOpts, but how commonly are the potentially problematic
features of it even used? Short booleans are deprecated and could be
dropped in 6.2. merge_lists may or may not have to be replicated.
I know building lists from repeated keys is said to be a thing, where
does it happen? I've worked on some of the big ones (blockdev, chardev,
object, device) and haven't come across it yet.
Can we have an intermediate state where CLI parsing involves both QAPI
generated options and manually written ones? So that we can actually put
the QAPIfication work already done to use instead of just having a
promise that it will make things possible eventually, in a few years?
> > What I had in mind was using the schema to generate the necessary code,
> > using the generic keyval parser everywhere, and just providing a hook
> > where the QDict could be modified after the keyval parser and before the
> > visitor. Most command line options would not have any explicit code for
> > parsing input, only the declarative schema and the final option handler
> > getting a QAPI type (which could actually be the corresponding QMP
> > command handler in the common case).
> >
> > I think these are very different ideas. If we want HMP-like, then all
> > the keyval based code paths we've built so far don't make much sense.
>
> I'm not sure the differences are "very" :)
>
> I think you start at "human-friendly and machine-friendly should use the
> abstract syntax defined in the QAPI schema, except where human-friendly
> has to differ, say for backward compatibility".
>
> This approach considers differences a blemish. The "standard" abstract
> syntax (the one defined in the QAPI schema) should be accepted in
> addition to the "alternate" one whenever possible, so "modern" use can
> avoid the blemishes, and blemishes can be removed once they have fallen
> out of use.
>
> "Alternate" should not be limited to human-friendly. Not limiting keeps
> things consistent, which is good, because differences are bad.
>
> Is that a fair description?
Hm, yes and no.
I do think that making the overlap between "standard" and "alternate"
abstract syntax as big as possible is a good thing because it means less
translation has to go on between them, and ultimately the interfaces are
more consistent with each other which actually improves the human
friendliness for people who get in touch with both syntaxes.
The -chardev conversion mostly deals with differences that I do consider
a blemish: There is no real reason why options need to have different
names on both interfaces, and there is also a lot of nesting in QMP for
which there is no real reason.
The reason why we need to accept both is compatibility, not usability.
There are also some differences that are justified by friendliness.
Having "addr" as a nested struct in QMP just makes sense, and on the
command line having it flattened makes more sense.
So accepting "path" for "device" in ChardevHostdev is probably a case
where switching both QMP and CLI to "modern" use and remove the
"blemish" eventually makes sense to me. The nesting for "addr" isn't.
We may even add more differences to make things human friendly. My
standard for this is whether the difference serves only for
compatibility (should go away eventually) or usability (should stay).
Now with this, it's still open whether the "standard" syntax should only
be supported in QMP or also in the CLI, and whether "alternate" syntax
should only be supported in the CLI or also in QMP.
Is usability actually improved by rejecting the "standard" syntax on the
command line, or by rejecting "alternate" syntax in QMP? Hardly so. It's
also not compatibility. So what is the justification for the difference?
Maintainability? I honestly don't see the improvement there either.
So I don't really see a reason for differences, but at the same time
it's also not a very important question to me. If you prefer restricting
things, so be it.
> I start at "human-friendly syntax should be as friendly as we can make
> it, except where we have to compromise, say for backward compatibility".
>
> This approach embraces differences. Mind, not differences just for
> differences sake, only where they help users.
>
> Additionally accepting the "standard" abstract syntax is useful only
> where it helps users.
>
> "Alternate" should be limited to human-friendly.
I think there is a small inconsistency in this reasoning: You say that
differences must help users, but then this is not the measuring stick
you use in the next paragraph. If it were, you would be asking whether
rejecting "standard" abstract syntax helps users, rather than whether
adding it does. (Even more so because rejecting it is more work! Not
much more work, but it adds a little bit of complexity.)
So it seems that in practice your approach is more like "different by
default, making it the same needs justification", whereas I am leaning
more towards "same by default, making it different needs justification".
Your idea of "human-friendly syntax should be as friendly as we can make
it" isn't in conflict with either approach. The thing that the idea
might actually conflict with is our time budget.
> Different approaches, without doubt. Yet both have to deal with
> differences, and both could use similar techniques and machinery for
> that. You're proposing to do the bulk of the work with aliases, and to
> have a tree-transforming hook for the remainder. Makes sense to me.
> However, in sufficiently gnarly cases, we might have to bypass all this
> and keep using handwritten code until the backward compatibility promise
> is history: see 2. above.
Possibly. I honestly can't tell how many of these cases we will have.
In all of -object, we had exactly one problematic option. This could
easily be handled in a tree-transforming hook.
> In addition each approach has its own, special needs.
>
> Yours adds "alternate" syntax to QMP. This poses documentation and
> introspection problems. The introspection solutions will then
> necessitate management application updates.
>
> Mine trades these problems for others, namely how to generate parsers
> for two different syntaxes from one QAPI schema.
>
> Do I make sense?
In the long run, won't we need documentation and introspection even for
things that are limited to the command line? Introspection is one of the
big features enabled by CLI QAPIfication.
But otherwise yes.
> >> Both -chardev and QMP chardev-add use the same helpers (there are minor
> >> differences that don't matter here). The former basically translates
> >> its flat argument into the latter's arguments. HMP chardev-add is just
> >> like -chardev.
> >>
> >> The quickest way to QAPIfy existing -chardev is probably the one we
> >> chose for -object: if the argument is JSON, use the new, QAPI-based
> >> interface, else use the old interface.
> >
> > -chardev doesn't quite translate into chardev-add arguments. Converting
> > into the input of a QMP command normally means providing a QDict that
> > can be accepted by the QAPI-generated visitor code, and then using that
> > QAPI parser to create the C object. What -chardev does instead is using
> > an entirely separate parser to create the C object directly, without
> > going through any QAPI code.
>
> Yes, and that's quite some extra code, with plenty of opportunity for
> inconsistency.
Needless to say, opportunity that we happily made use of.
> > After QAPIfication, both code paths go through the QAPI code and undergo
> > the same validations etc.
> >
> > If we still have code paths that completely bypass QAPI, it's hard to
> > call the command line option fully QAPIfied. Of course, if you don't
> > care about duplication and inconsistencies between the interfaces,
> > though, and just want to achieve the initially stated goal of providing
> > at least one interface consistent between QMP and CLI (besides others)
> > and a config file, then it might be QAPIfied enough.
>
> Reworking human-friendly -chardev to target QMP instead of a C interface
> shared with QMP would be nice. Just because prior attempts to replace
> gnarly stuff compatibly have failed doesn't mean this one must fail.
I mean, for -chardev specifically, you don't even have to take a guess.
The patches exist, a git tag with them is mentioned in the cover letter
of this series, and they have just been waiting for about a year for
their dependency (QAPI aliases) to be merged.
If we don't do aliases, I'll have to rework them to do everything in
code instead. It's doable, even if the chardev code wouldn't shrink as
nicely as with the current patches. I just need to know whether it has
to be done or not.
> >> Now the question that matters for this series: how can QAPI aliases
> >> help us with the things discussed above?
> >
> > The translation needs to be implemented somehow. The obvious way is just
> > writing, reviewing and maintaining code that manually translates. (We
> > don't have this code yet for a fully QAPIfied -chardev. We only have
> > code that bypasses QAPI, i.e. translates to the wrong target format.)
> >
> > Aliases help because they allow reducing the amount of code in favour of
> > data, the alias declarations in the schema.
>
> Understood.
>
> >> If we use the flat variation just internally, say for -chardev's dotted
> >> keys argument, then introspection is not needed. We'd use "with
> >> aliases" just for translating -chardev's dotted keys argument.
> >
> > Note that we're doing more translations that just flattening with
> > aliases, some options have different names in QMP and the CLI.
> >
> > But either way, yes, alias support could be optional so that you need to
> > explicitly enable it when creating a visitor, and then only the CLI code
> > could actually enable it.
> >
> > Limits the possible future uses for the new tool in our toolbox, but is
> > sufficient for -chardev. We can always extend support for it later
> > if/when we actually want to make use of it outside of the CLI.
>
> Yes.
>
> >> Valid argument. CLI with JSON argument should match QMP. Even when
> >> that means both are excessively nested.
> >>
> >> CLI with dotted keys is a different matter, in my opinion.
> >
> > I'm skipping quite a bit of text here, mainly because I think we need an
> > answer first if we really want to switch the keyval options to be
> > HMP-like in more than just the support status (see above).
> >
> > Obviously, the other relevant question would then be if it also ends up
> > like HMP in that most interesting functionality isn't even available and
> > you're forced to use QMP/JSON when you're serious.
>
> I figure this is because we have let "QMP first" degenerate into "QMP
> first, HMP never" sometimes, and we did so mostly because HMP is extra
> work that was hard to justify.
>
> What if you could get the 1:1 HMP command basically for free? It may
> not be the most human-friendly command possible. But it should be a
> start.
How would you do this?
The obvious way is mapping QMP 1:1 like we do for some of the command
line options. But you just argued that this is not what you would prefer
for the command line, so it's probably not what you have in mind for HMP
either?
> > I guess I can go into more details once we have answered these
> > fundamental questions.
> >
> > (We could and should have talked about them and about whether you want
> > to have aliases at all a year ago, before going into detailed code
> > review and making error messages perfect in code that we might throw
> > away anyway...)
>
> I doubt you could possibly be more right than that!
So what questions need to be answered before we can make a decision?
You talked a lot about how you like QMP to be different from the command
line. So is restricting aliases to the command line enough to make them
acceptable? Or do we need larger changes? Should I just throw them away
and write translation code for -chardev manually?
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-05 17:05 ` Kevin Wolf
@ 2021-10-06 13:11 ` Markus Armbruster
2021-10-06 16:36 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-10-06 13:11 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 05.10.2021 um 15:49 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>>
>> > Am 02.10.2021 um 15:33 hat Markus Armbruster geschrieben:
>> >> I apologize for this wall of text. It's a desparate attempt to cut
>> >> through the complexity and my confusion, and make sense of the actual
>> >> problems we're trying to solve.
>> >>
>> >> So, what problems exactly are we trying to solve?
>> >
>> > I'll start with replying to your final question because I think it's
>> > more helpful to start with the big picture than with details.
>> >
>> > So tools like libvirt want to have a single consistent interface to
>> > configure things on startup and at runtime. We also intend to support
>> > configuration files that should this time support all of the options and
>> > not just a few chosen ones.
>>
>> Yes.
>>
>> > The hypothesis is that QAPIfying the command line is the correct
>> > solution for both of these problems, i.e. all available command line
>> > options must be present in the QAPI schema and will be processed by a
>> > single parser shared with QMP to make sure they are consistent.
>>
>> Yes.
>>
>> This leads us to JSON option arguments and configuration files.
>> Well-suited for management applications that already use QMP.
>>
>> > Adding QAPIfied versions of individual command line options are steps
>> > towards this goal. As soon as they exist for every option, the final
>> > conversion from an open coded getopt() loop (or in fact a hand crafted
>> > parser in the case of vl.c) to something generated from the QAPI schema
>> > should be reasonably easy.
>>
>> Yes.
>>
>> > You're right that adding a second JSON-based command line interface for
>> > every option can already achieve the goal of providing a unified
>> > external interface, at the cost of (interface and code) duplication. Is
>> > this duplication desirable? Certainly no. Is it acceptable? You might
>> > get different opinions on this one.
>>
>> We'd certainly prefer CLI options to match corresponding QMP commands
>> exactly.
>>
>> Unfortunately, existing CLI options deviate from corresponding QMP
>> commands, and existing CLI options without corresponding QMP commands
>> may violate QMP design rules.
>>
>> Note: these issues pertain to human-friendly option syntax. The
>> machine-friendly option syntax is still limited to just a few options,
>> and it does match QMP there.
>
> On the other hand, we only have a handful of existing options that are
> very complex. Most of them aren't even structured and just take a single
> scalar value. So I'm relatively sure that we know the big ones, and
> we're working on them.
>
> Another point here is that I would argue that there is a difference
> between existing options and human-friendly options. Not all existing
> options are human-friendly just because they aren't machine friendly
> either, and there is probably potential for human-friendly syntax that
> doesn't exist yet.
>
> So I would treat support for existing options (i.e. compatibility) and
> support for human-friendly options (i.e. usability) as two separate
> problems.
That's fair.
Instead of "human-friendly option syntax", I could've written "our
traditional option syntax (which is better suited to humans than to
machines, although it still fails to be truly friendly to anyone)", but
that's a tad long, isn't it?
>> > In my opinion, we should try to get rid of hand crafted parsers where
>> > it's reasonably possible, and take advantage of the single unified
>> > option structure that QAPI provides. -chardev specifically has a hand
>> > crafted parser that essentially duplicates the automatically generated
>> > QAPI visitor code, except for the nesting and some differences in option
>> > names.
>>
>> We should definitely parse JSON option arguments with the QAPI
>> machinery, and nothing more.
>
> Yes, no doubt. (And we don't even consistently do that yet, like
> device-add going through QemuOpts after going through QAPI and throwing
> away all type information and silently ignoring anything it doesn't know
> to handle.)
At least device-add is the last remaining user of 'gen': false.
The 'any' type also leaves type checking to handwritten code, but at
least it doesn't throw away type information first. Used by qom-get and
qom-set.
>> Parsing human-friendly arguments with it is desirable, but the need for
>> backward compatibility can make it difficult. Even where compatibility
>> is of no concern, simply swapping concrete JSON syntax for dotted keys
>> may result in human interfaces that are less than friendly.
>>
>> Are we in agreement that this is the problem at hand?
>
> As far as I am concerned, compatibility is the problem at hand,
> usability isn't.
>
> This doesn't mean that I'm opposed to having actually human friendly
> options, quite the opposite. But getting machine friendly options is
> already a big project. Making human interfaces friendlier would be
> adding another project of similar size, and I don't feel like tackling
> a second project at the same time when the first one is already hard.
Fair enough.
>> > Aliases are one tool that can help bridge these differences in a generic
>> > way with minimal effort in the individual case. They are not _necessary_
>> > to solve the problem; we could instead just use manually written code to
>> > manipulate input QDicts so that QAPI visitors accept them. Even with
>> > aliases, there are a few things left in the chardev code that are
>> > converted this way. Aliases just greatly reduce the amount of this code
>> > and make the conversion declarative instead.
>>
>> Understood.
>>
>> > Now a key point in the previous paragraph is that aliases add a generic
>> > way to do this. So even if they are immediately motivated by -chardev,
>> > it might be worth looking at other cases they could enable if you think
>> > that -chardev alone isn't sufficient justification to have this tool.
>> > I guess this is the point where things become a bit less clear because
>> > people are just hand waving with vague ideas for additional uses.
>> >
>> > Do we need to invest more thought on these other cases? We probably do
>> > if it makes a difference for the answer to the question whether we want
>> > to add aliases to our toolbox. Does it?
>>
>> I hope we can make a case for aliases without looking beyond CLI
>> QAPIfication. That's a wide field already, with enough opportunity to
>> get lost in details.
>>
>> If we later put aliases to other uses, we might have to adapt them some.
>> That's okay. Designing for one problem we have and understand has a
>> much better chance of success than trying to design for all problems we
>> might have.
>>
>> There are many CLI options to be QAPIfied. -chardev is one of the more
>> thornier ones, which makes it a useful example.
>
> Good, I agree.
>
>> >> But what about the dotted keys argument?
>> >>
>> >> One point of view is the difference between the JSON and the dotted keys
>> >> argument should be concrete syntax only. Fine print: there may be
>> >> arguments dotted keys can't express, but let's ignore that here.
>> >>
>> >> Another point of view is that dotted keys arguments are to JSON
>> >> arguments what HMP is to QMP: a (hopefully) human-friendly layer on top,
>> >> where we have a certain degree of freedom to improve on the
>> >> machine-friendly interface for human use.
>> >
>> > This doesn't feel unreasonable because with HMP, there is precedence.
>> > But this is not all what we have used dotted key syntax for so far. I'm
>> > not against making a change there if we feel it makes sense, but we
>> > should be clear that it is a change.
>>
>> Our first uses of dotted keys were green fields. Sticking to QAPI/QMP's
>> abstract syntax was simplest.
>>
>> > I think we all agree that -blockdev isn't human-friendly. Dotted key
>> > syntax makes it humanly possible to use it on the command line, but
>> > friendly is something else entirely. It is still a 1:1 mapping of QMP to
>> > the command line, and this is how we advertised it, too.
>>
>> Yes.
>>
>> > This has some
>> > important advantages, like we don't have a separate parser for a
>> > human-friendly syntax, but the generic keyval parser covers it.
>>
>> There are two generic parts in play: the keyval parser, and the QAPI
>> input visitor.
>
> Fair point.
>
>> Using identical abstract syntax both for JSON and dotted keys arguments
>> makes use of both parts simple: pass the argument to
>> qobject_input_visitor_new_str() to create a visitor, then visit the QAPI
>> type with it.
>>
>> When abstract syntax differs, using the keyval parser is still simple,
>> but to use the QAPI input visitor, we need separate QAPI types. Massive
>> code duplication in the QAPI schema.
>>
>> To avoid the duplication, we can instead translate the parse tree
>> produced for the dotted keys argument. Plenty of boring code.
>>
>> In short: yes, using the same abstract syntax for both has advantages.
>
> Even with duplication in the schema, you still have to translate unless
> you want to duplicate all of the logic, too. The difference is just that
> instead of translating between QObjects, you would be translating
> between C structs.
True.
> So yes, using different abstract syntax means translation, which in turn
> can mean plenty of boring code (hopefully - might also be buggy rather
> than boring) if we don't support ways to automate the conversion.
We did it manually for SocketAddress / SocketAddressLegacy. Worked out
okay. We don't want to do it manually for chardev-add's argument type,
because the amount of translation / duplication is off-putting.
>> > HMP in contrast means separate code for every command, or translated to
>> > the CLI for each command line option. HMP is not defined in QAPI, it's
>> > a separate thing that just happens to call into QAPI-based QMP code in
>> > the common case.
>> >
>> > Is this what you have in mind for non-JSON command line options?
>>
>> We should separate the idea "HMP wraps around QMP" from its
>> implementation as handwritten, boilerplate-heavy code.
>>
>> All but the latest, QAPI-based CLI options work pretty much like HMP: a
>> bit of code generation (hxtool), a mix of common and ad hoc parsers,
>> boring handwritten code to translate from parse tree to internal C
>> interfaces (which are often QMP command handlers), and to stitch it all
>> together.
>>
>> Reducing the amount of boring handwritten code is equally desirable for
>> HMP and CLI.
>>
>> So is specifying the interface in QAPI instead of older, less powerful
>> schema languages like hxtool and QemuOpts.
>>
>> There are at least three problems:
>>
>> 1. Specifying CLI in QAPI hasn't progressed beyond the proof-of-concept
>> stage.
>>
>> 2. Backward compatibility issues and doubts have defeated attempts to
>> replace gnarly stuff, in particular QemuOpts.
>>
>> 3. How to best bridge abstract syntax differences has been unclear.
>> Whether the differences are historical accidents or intentional for ease
>> of human use doesn't matter.
>>
>> Aliases feel like a contribution towards solving 3.
>>
>> I don't think 1. is particularly difficult. It's just a big chunk of
>> work that isn't really useful without solutions for 2. and 3.
>>
>> To me, 2. feels intractable head on. Perhaps we better bypass the
>> problem by weakening the compatibility promise like we did for HMP.
>
> Can we define 2 in a bit more specific terms? I feel the biggest part
> of 2 is actually 3.
>
> You mention QemuOpts, but how commonly are the potentially problematic
> features of it even used? Short booleans are deprecated and could be
> dropped in 6.2. merge_lists may or may not have to be replicated.
> I know building lists from repeated keys is said to be a thing, where
> does it happen? I've worked on some of the big ones (blockdev, chardev,
> object, device) and haven't come across it yet.
Alright, I'll look for you :) Check out commit 2d6dcbf93fb, 396f935a9af,
54cb65d8588, f1672e6f2b6. These are fairly recent, which is kind of
depressing. There may be more.
When I wrote 2., I was specifically thinking of -object, where we
decided not to mess the human-friendly syntax in part because we didn't
fully understand what a conversion from QemuOpts to keyval could break.
> Can we have an intermediate state where CLI parsing involves both QAPI
> generated options and manually written ones? So that we can actually put
> the QAPIfication work already done to use instead of just having a
> promise that it will make things possible eventually, in a few years?
We kind of sort of have that already: a few option arguments are
QAPI-based.
Perhaps it's time to crack 1., but in a way that accommodates
handwritten options. Could be like "this option takes a string
argument, and the function to process the option is not to be generated,
only called (just like 'gen': false for commands). We may want to make
introspection describe the argument as "unknown", to distinguish it from
genuine string arguments.
>> > What I had in mind was using the schema to generate the necessary code,
>> > using the generic keyval parser everywhere, and just providing a hook
>> > where the QDict could be modified after the keyval parser and before the
>> > visitor. Most command line options would not have any explicit code for
>> > parsing input, only the declarative schema and the final option handler
>> > getting a QAPI type (which could actually be the corresponding QMP
>> > command handler in the common case).
>> >
>> > I think these are very different ideas. If we want HMP-like, then all
>> > the keyval based code paths we've built so far don't make much sense.
>>
>> I'm not sure the differences are "very" :)
>>
>> I think you start at "human-friendly and machine-friendly should use the
>> abstract syntax defined in the QAPI schema, except where human-friendly
>> has to differ, say for backward compatibility".
>>
>> This approach considers differences a blemish. The "standard" abstract
>> syntax (the one defined in the QAPI schema) should be accepted in
>> addition to the "alternate" one whenever possible, so "modern" use can
>> avoid the blemishes, and blemishes can be removed once they have fallen
>> out of use.
>>
>> "Alternate" should not be limited to human-friendly. Not limiting keeps
>> things consistent, which is good, because differences are bad.
>>
>> Is that a fair description?
>
> Hm, yes and no.
>
>
> I do think that making the overlap between "standard" and "alternate"
> abstract syntax as big as possible is a good thing because it means less
> translation has to go on between them, and ultimately the interfaces are
> more consistent with each other which actually improves the human
> friendliness for people who get in touch with both syntaxes.
>
> The -chardev conversion mostly deals with differences that I do consider
> a blemish: There is no real reason why options need to have different
> names on both interfaces, and there is also a lot of nesting in QMP for
> which there is no real reason.
>
> The reason why we need to accept both is compatibility, not usability.
Compatibility requires us to accept both, but each in its place. It
doesn't actually require us to accept both in either place.
> There are also some differences that are justified by friendliness.
> Having "addr" as a nested struct in QMP just makes sense, and on the
> command line having it flattened makes more sense.
>
> So accepting "path" for "device" in ChardevHostdev is probably a case
> where switching both QMP and CLI to "modern" use and remove the
> "blemish" eventually makes sense to me. The nesting for "addr" isn't.
>
> We may even add more differences to make things human friendly. My
> standard for this is whether the difference serves only for
> compatibility (should go away eventually) or usability (should stay).
In theory, cleaning up QMP is nice. In practice, we don't have a good
way to rename or move members. When we do, we have to make both old and
new optional, even when you actually have to specify exactly one. This
isn't the end of the world, but it is trading one kind of blemish for
another.
Improving on that would involve a non-trivial extension to
introspection. Pain right away, gain only when all management
applications that matter grok it.
I think the realistic assumption is that QMP blemishes stay.
Of course, we'd prefer not to copy existing QMP blemishes into new
interfaces. That's why we have SocketAddress for new code, and
SocketAddressLegacy for old code. Workable for simple utility types. I
don't have a good solution for "bigger" types.
> Now with this, it's still open whether the "standard" syntax should only
> be supported in QMP or also in the CLI, and whether "alternate" syntax
> should only be supported in the CLI or also in QMP.
>
> Is usability actually improved by rejecting the "standard" syntax on the
> command line, or by rejecting "alternate" syntax in QMP? Hardly so. It's
> also not compatibility. So what is the justification for the difference?
> Maintainability? I honestly don't see the improvement there either.
>
> So I don't really see a reason for differences, but at the same time
> it's also not a very important question to me. If you prefer restricting
> things, so be it.
I dislike adding "alternate" to machine-friendly QMP / CLI poses,
because:
* Adding "alternate" poses documentation and introspection problems.
The introspection solutions will then necessitate management
application updates. Anything we do becomes ABI immediately, so we
better get it right on the first try.
* Ignoring the documentation problems is a bad idea, because
undocumented interfaces are.
* If we ignore the introspection problems, then machines are well
advised to stick to introspectable "standard". Why add "alternate"
then? Complexity for nothing.
* Adding "alternate" to existing interfaces is pretty much pointless.
To use "alternate", machines first have to check whether this QEMU
provides it, and if not, fall back to "standard". Just use "standard"
and call it a day.
* For new interfaces, the differences between "standard" and "alternate"
are hopefully smaller than for old interfaces. Machines will likely
not care for improvements "alternate" may provide over "standard".
I haven't made up my mind on adding "standard" to human-friendly CLI.
Too many ways to do the same thing are bound to confuse. We could end
up with the standard way, the alternate way we provide for backward
compatibility, the alternate way we provide for usability, and any
combination thereof.
Each of the three ways has its uses, though. Combinations not so much.
Adding "standard" to human-friendly CLI only poses documentation
problems (there is no introspection). If we limit the backward
compatibility promise to machine-friendly, getting things wrong is no
big deal.
>> I start at "human-friendly syntax should be as friendly as we can make
>> it, except where we have to compromise, say for backward compatibility".
>>
>> This approach embraces differences. Mind, not differences just for
>> differences sake, only where they help users.
>>
>> Additionally accepting the "standard" abstract syntax is useful only
>> where it helps users.
>>
>> "Alternate" should be limited to human-friendly.
>
> I think there is a small inconsistency in this reasoning: You say that
> differences must help users, but then this is not the measuring stick
> you use in the next paragraph. If it were, you would be asking whether
> rejecting "standard" abstract syntax helps users, rather than whether
> adding it does. (Even more so because rejecting it is more work! Not
> much more work, but it adds a little bit of complexity.)
Additionally accepting "standard" complicates the interface and its
documentation. Whether this helps users more than it hurts them is not
obvious.
> So it seems that in practice your approach is more like "different by
> default, making it the same needs justification", whereas I am leaning
> more towards "same by default, making it different needs justification".
>
> Your idea of "human-friendly syntax should be as friendly as we can make
> it" isn't in conflict with either approach. The thing that the idea
> might actually conflict with is our time budget.
Reasonable people often have quite different ideas on "human-friendly".
>> Different approaches, without doubt. Yet both have to deal with
>> differences, and both could use similar techniques and machinery for
>> that. You're proposing to do the bulk of the work with aliases, and to
>> have a tree-transforming hook for the remainder. Makes sense to me.
>> However, in sufficiently gnarly cases, we might have to bypass all this
>> and keep using handwritten code until the backward compatibility promise
>> is history: see 2. above.
>
> Possibly. I honestly can't tell how many of these cases we will have.
> In all of -object, we had exactly one problematic option. This could
> easily be handled in a tree-transforming hook.
>
>> In addition each approach has its own, special needs.
>>
>> Yours adds "alternate" syntax to QMP. This poses documentation and
>> introspection problems. The introspection solutions will then
>> necessitate management application updates.
>>
>> Mine trades these problems for others, namely how to generate parsers
>> for two different syntaxes from one QAPI schema.
>>
>> Do I make sense?
>
> In the long run, won't we need documentation and introspection even for
> things that are limited to the command line? Introspection is one of the
> big features enabled by CLI QAPIfication.
CLI introspection is about the machine-friendly syntax, not the
human-friendly one, just like monitor introspection is about QMP, not
HMP.
> But otherwise yes.
>
>> >> Both -chardev and QMP chardev-add use the same helpers (there are minor
>> >> differences that don't matter here). The former basically translates
>> >> its flat argument into the latter's arguments. HMP chardev-add is just
>> >> like -chardev.
>> >>
>> >> The quickest way to QAPIfy existing -chardev is probably the one we
>> >> chose for -object: if the argument is JSON, use the new, QAPI-based
>> >> interface, else use the old interface.
>> >
>> > -chardev doesn't quite translate into chardev-add arguments. Converting
>> > into the input of a QMP command normally means providing a QDict that
>> > can be accepted by the QAPI-generated visitor code, and then using that
>> > QAPI parser to create the C object. What -chardev does instead is using
>> > an entirely separate parser to create the C object directly, without
>> > going through any QAPI code.
>>
>> Yes, and that's quite some extra code, with plenty of opportunity for
>> inconsistency.
>
> Needless to say, opportunity that we happily made use of.
Hah!
>> > After QAPIfication, both code paths go through the QAPI code and undergo
>> > the same validations etc.
>> >
>> > If we still have code paths that completely bypass QAPI, it's hard to
>> > call the command line option fully QAPIfied. Of course, if you don't
>> > care about duplication and inconsistencies between the interfaces,
>> > though, and just want to achieve the initially stated goal of providing
>> > at least one interface consistent between QMP and CLI (besides others)
>> > and a config file, then it might be QAPIfied enough.
>>
>> Reworking human-friendly -chardev to target QMP instead of a C interface
>> shared with QMP would be nice. Just because prior attempts to replace
>> gnarly stuff compatibly have failed doesn't mean this one must fail.
>
> I mean, for -chardev specifically, you don't even have to take a guess.
> The patches exist, a git tag with them is mentioned in the cover letter
> of this series, and they have just been waiting for about a year for
> their dependency (QAPI aliases) to be merged.
>
> If we don't do aliases, I'll have to rework them to do everything in
> code instead. It's doable, even if the chardev code wouldn't shrink as
> nicely as with the current patches. I just need to know whether it has
> to be done or not.
>
>> >> Now the question that matters for this series: how can QAPI aliases
>> >> help us with the things discussed above?
>> >
>> > The translation needs to be implemented somehow. The obvious way is just
>> > writing, reviewing and maintaining code that manually translates. (We
>> > don't have this code yet for a fully QAPIfied -chardev. We only have
>> > code that bypasses QAPI, i.e. translates to the wrong target format.)
>> >
>> > Aliases help because they allow reducing the amount of code in favour of
>> > data, the alias declarations in the schema.
>>
>> Understood.
>>
>> >> If we use the flat variation just internally, say for -chardev's dotted
>> >> keys argument, then introspection is not needed. We'd use "with
>> >> aliases" just for translating -chardev's dotted keys argument.
>> >
>> > Note that we're doing more translations that just flattening with
>> > aliases, some options have different names in QMP and the CLI.
>> >
>> > But either way, yes, alias support could be optional so that you need to
>> > explicitly enable it when creating a visitor, and then only the CLI code
>> > could actually enable it.
>> >
>> > Limits the possible future uses for the new tool in our toolbox, but is
>> > sufficient for -chardev. We can always extend support for it later
>> > if/when we actually want to make use of it outside of the CLI.
>>
>> Yes.
>>
>> >> Valid argument. CLI with JSON argument should match QMP. Even when
>> >> that means both are excessively nested.
>> >>
>> >> CLI with dotted keys is a different matter, in my opinion.
>> >
>> > I'm skipping quite a bit of text here, mainly because I think we need an
>> > answer first if we really want to switch the keyval options to be
>> > HMP-like in more than just the support status (see above).
>> >
>> > Obviously, the other relevant question would then be if it also ends up
>> > like HMP in that most interesting functionality isn't even available and
>> > you're forced to use QMP/JSON when you're serious.
>>
>> I figure this is because we have let "QMP first" degenerate into "QMP
>> first, HMP never" sometimes, and we did so mostly because HMP is extra
>> work that was hard to justify.
>>
>> What if you could get the 1:1 HMP command basically for free? It may
>> not be the most human-friendly command possible. But it should be a
>> start.
>
> How would you do this?
>
> The obvious way is mapping QMP 1:1 like we do for some of the command
> line options. But you just argued that this is not what you would prefer
> for the command line, so it's probably not what you have in mind for HMP
> either?
I think there are QMP commands where a 1:1 HMP command would be just
fine.
For others, it would be awkward, but that could still better than
nothing.
Infrastructure we build to QAPIfy the command line could probably be
used to get less awkward HMP commands without writing so much code.
Note I'm not talking about HMP commands somebody *wants* to write, say
to provide convenience magic that is inappropriate for QMP, or
specialized higher level commands where QMP only provides general
building blocks.
>> > I guess I can go into more details once we have answered these
>> > fundamental questions.
>> >
>> > (We could and should have talked about them and about whether you want
>> > to have aliases at all a year ago, before going into detailed code
>> > review and making error messages perfect in code that we might throw
>> > away anyway...)
>>
>> I doubt you could possibly be more right than that!
>
> So what questions need to be answered before we can make a decision?
>
> You talked a lot about how you like QMP to be different from the command
> line. So is restricting aliases to the command line enough to make them
> acceptable? Or do we need larger changes? Should I just throw them away
> and write translation code for -chardev manually?
Fair question, but I'm out of time for right now. Let's continue
tomorrow.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-06 13:11 ` Markus Armbruster
@ 2021-10-06 16:36 ` Kevin Wolf
2021-10-07 11:06 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-10-06 16:36 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 06.10.2021 um 15:11 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
>
> > Am 05.10.2021 um 15:49 hat Markus Armbruster geschrieben:
> >> Kevin Wolf <kwolf@redhat.com> writes:
> >>
> >> > Am 02.10.2021 um 15:33 hat Markus Armbruster geschrieben:
> >> >> I apologize for this wall of text. It's a desparate attempt to cut
> >> >> through the complexity and my confusion, and make sense of the actual
> >> >> problems we're trying to solve.
> >> >>
> >> >> So, what problems exactly are we trying to solve?
> >> >
> >> > I'll start with replying to your final question because I think it's
> >> > more helpful to start with the big picture than with details.
> >> >
> >> > So tools like libvirt want to have a single consistent interface to
> >> > configure things on startup and at runtime. We also intend to support
> >> > configuration files that should this time support all of the options and
> >> > not just a few chosen ones.
> >>
> >> Yes.
> >>
> >> > The hypothesis is that QAPIfying the command line is the correct
> >> > solution for both of these problems, i.e. all available command line
> >> > options must be present in the QAPI schema and will be processed by a
> >> > single parser shared with QMP to make sure they are consistent.
> >>
> >> Yes.
> >>
> >> This leads us to JSON option arguments and configuration files.
> >> Well-suited for management applications that already use QMP.
> >>
> >> > Adding QAPIfied versions of individual command line options are steps
> >> > towards this goal. As soon as they exist for every option, the final
> >> > conversion from an open coded getopt() loop (or in fact a hand crafted
> >> > parser in the case of vl.c) to something generated from the QAPI schema
> >> > should be reasonably easy.
> >>
> >> Yes.
> >>
> >> > You're right that adding a second JSON-based command line interface for
> >> > every option can already achieve the goal of providing a unified
> >> > external interface, at the cost of (interface and code) duplication. Is
> >> > this duplication desirable? Certainly no. Is it acceptable? You might
> >> > get different opinions on this one.
> >>
> >> We'd certainly prefer CLI options to match corresponding QMP commands
> >> exactly.
> >>
> >> Unfortunately, existing CLI options deviate from corresponding QMP
> >> commands, and existing CLI options without corresponding QMP commands
> >> may violate QMP design rules.
> >>
> >> Note: these issues pertain to human-friendly option syntax. The
> >> machine-friendly option syntax is still limited to just a few options,
> >> and it does match QMP there.
> >
> > On the other hand, we only have a handful of existing options that are
> > very complex. Most of them aren't even structured and just take a single
> > scalar value. So I'm relatively sure that we know the big ones, and
> > we're working on them.
> >
> > Another point here is that I would argue that there is a difference
> > between existing options and human-friendly options. Not all existing
> > options are human-friendly just because they aren't machine friendly
> > either, and there is probably potential for human-friendly syntax that
> > doesn't exist yet.
> >
> > So I would treat support for existing options (i.e. compatibility) and
> > support for human-friendly options (i.e. usability) as two separate
> > problems.
>
> That's fair.
>
> Instead of "human-friendly option syntax", I could've written "our
> traditional option syntax (which is better suited to humans than to
> machines, although it still fails to be truly friendly to anyone)", but
> that's a tad long, isn't it?
It is, but I had the impression that you used the same term for both
things without making a distinction.
> >> > In my opinion, we should try to get rid of hand crafted parsers where
> >> > it's reasonably possible, and take advantage of the single unified
> >> > option structure that QAPI provides. -chardev specifically has a hand
> >> > crafted parser that essentially duplicates the automatically generated
> >> > QAPI visitor code, except for the nesting and some differences in option
> >> > names.
> >>
> >> We should definitely parse JSON option arguments with the QAPI
> >> machinery, and nothing more.
> >
> > Yes, no doubt. (And we don't even consistently do that yet, like
> > device-add going through QemuOpts after going through QAPI and throwing
> > away all type information and silently ignoring anything it doesn't know
> > to handle.)
>
> At least device-add is the last remaining user of 'gen': false.
>
> The 'any' type also leaves type checking to handwritten code, but at
> least it doesn't throw away type information first. Used by qom-get and
> qom-set.
Even if device-add wasn't 'gen': false, in theory nothing would stop it
from going back from the QAPI C object through a output visitor and
QemuOpts and throwing the type information away again. Of course, with
explicit type checks in QAPI, we're less likely to mess up the other
side because otherwise things just wouldn't work.
But I actually know a subsystem where such conversions happen... *cough*
> >> > HMP in contrast means separate code for every command, or translated to
> >> > the CLI for each command line option. HMP is not defined in QAPI, it's
> >> > a separate thing that just happens to call into QAPI-based QMP code in
> >> > the common case.
> >> >
> >> > Is this what you have in mind for non-JSON command line options?
> >>
> >> We should separate the idea "HMP wraps around QMP" from its
> >> implementation as handwritten, boilerplate-heavy code.
> >>
> >> All but the latest, QAPI-based CLI options work pretty much like HMP: a
> >> bit of code generation (hxtool), a mix of common and ad hoc parsers,
> >> boring handwritten code to translate from parse tree to internal C
> >> interfaces (which are often QMP command handlers), and to stitch it all
> >> together.
> >>
> >> Reducing the amount of boring handwritten code is equally desirable for
> >> HMP and CLI.
> >>
> >> So is specifying the interface in QAPI instead of older, less powerful
> >> schema languages like hxtool and QemuOpts.
> >>
> >> There are at least three problems:
> >>
> >> 1. Specifying CLI in QAPI hasn't progressed beyond the proof-of-concept
> >> stage.
> >>
> >> 2. Backward compatibility issues and doubts have defeated attempts to
> >> replace gnarly stuff, in particular QemuOpts.
> >>
> >> 3. How to best bridge abstract syntax differences has been unclear.
> >> Whether the differences are historical accidents or intentional for ease
> >> of human use doesn't matter.
> >>
> >> Aliases feel like a contribution towards solving 3.
> >>
> >> I don't think 1. is particularly difficult. It's just a big chunk of
> >> work that isn't really useful without solutions for 2. and 3.
> >>
> >> To me, 2. feels intractable head on. Perhaps we better bypass the
> >> problem by weakening the compatibility promise like we did for HMP.
> >
> > Can we define 2 in a bit more specific terms? I feel the biggest part
> > of 2 is actually 3.
> >
> > You mention QemuOpts, but how commonly are the potentially problematic
> > features of it even used? Short booleans are deprecated and could be
> > dropped in 6.2. merge_lists may or may not have to be replicated.
> > I know building lists from repeated keys is said to be a thing, where
> > does it happen? I've worked on some of the big ones (blockdev, chardev,
> > object, device) and haven't come across it yet.
>
> Alright, I'll look for you :) Check out commit 2d6dcbf93fb, 396f935a9af,
> 54cb65d8588, f1672e6f2b6. These are fairly recent, which is kind of
> depressing. There may be more.
Should we patch QemuOpts to disallow this by default and require setting
QemuOptsList.legacy_enable_repeated_options just to avoid new instances
sneaking in?
> When I wrote 2., I was specifically thinking of -object, where we
> decided not to mess the human-friendly syntax in part because we didn't
> fully understand what a conversion from QemuOpts to keyval could break.
I seem to remember we were confident that only the various list hacks
would be problematic. They only property in -object that makes use of
them is memory-backend.host-nodes. We should deprecate the old syntax
for it and move on.
> > Can we have an intermediate state where CLI parsing involves both QAPI
> > generated options and manually written ones? So that we can actually put
> > the QAPIfication work already done to use instead of just having a
> > promise that it will make things possible eventually, in a few years?
>
> We kind of sort of have that already: a few option arguments are
> QAPI-based.
>
> Perhaps it's time to crack 1., but in a way that accommodates
> handwritten options. Could be like "this option takes a string
> argument, and the function to process the option is not to be generated,
> only called (just like 'gen': false for commands). We may want to make
> introspection describe the argument as "unknown", to distinguish it from
> genuine string arguments.
Maybe 'data': 'any' rather than 'gen': false if the keyval parser makes
sense for the option. Or as a first step, just call into a single C
function for any unknown option so that we don't have to split the big
switch block immediately and can instead reduce it incrementally. Then
'data': 'any' is an intermediate step we can take when a function is too
complicated to be fully QAPIfied in a single commit.
But yes, I guess cracking 1. is what I had in mind: Let QAPI control the
command line parsing and only call out into handwritten code for not yet
converted options. I think this would be a big step forward compared to
the current state of only calling into QAPI for selected options because
it makes it very obvious that new code should use QAPI, and it would
also give us more a visible list of things that still need to be
converted.
> >> > What I had in mind was using the schema to generate the necessary code,
> >> > using the generic keyval parser everywhere, and just providing a hook
> >> > where the QDict could be modified after the keyval parser and before the
> >> > visitor. Most command line options would not have any explicit code for
> >> > parsing input, only the declarative schema and the final option handler
> >> > getting a QAPI type (which could actually be the corresponding QMP
> >> > command handler in the common case).
> >> >
> >> > I think these are very different ideas. If we want HMP-like, then all
> >> > the keyval based code paths we've built so far don't make much sense.
> >>
> >> I'm not sure the differences are "very" :)
> >>
> >> I think you start at "human-friendly and machine-friendly should use the
> >> abstract syntax defined in the QAPI schema, except where human-friendly
> >> has to differ, say for backward compatibility".
> >>
> >> This approach considers differences a blemish. The "standard" abstract
> >> syntax (the one defined in the QAPI schema) should be accepted in
> >> addition to the "alternate" one whenever possible, so "modern" use can
> >> avoid the blemishes, and blemishes can be removed once they have fallen
> >> out of use.
> >>
> >> "Alternate" should not be limited to human-friendly. Not limiting keeps
> >> things consistent, which is good, because differences are bad.
> >>
> >> Is that a fair description?
> >
> > Hm, yes and no.
> >
> >
> > I do think that making the overlap between "standard" and "alternate"
> > abstract syntax as big as possible is a good thing because it means less
> > translation has to go on between them, and ultimately the interfaces are
> > more consistent with each other which actually improves the human
> > friendliness for people who get in touch with both syntaxes.
> >
> > The -chardev conversion mostly deals with differences that I do consider
> > a blemish: There is no real reason why options need to have different
> > names on both interfaces, and there is also a lot of nesting in QMP for
> > which there is no real reason.
> >
> > The reason why we need to accept both is compatibility, not usability.
>
> Compatibility requires us to accept both, but each in its place. It
> doesn't actually require us to accept both in either place.
Yes.
> > There are also some differences that are justified by friendliness.
> > Having "addr" as a nested struct in QMP just makes sense, and on the
> > command line having it flattened makes more sense.
> >
> > So accepting "path" for "device" in ChardevHostdev is probably a case
> > where switching both QMP and CLI to "modern" use and remove the
> > "blemish" eventually makes sense to me. The nesting for "addr" isn't.
> >
> > We may even add more differences to make things human friendly. My
> > standard for this is whether the difference serves only for
> > compatibility (should go away eventually) or usability (should stay).
>
> In theory, cleaning up QMP is nice. In practice, we don't have a good
> way to rename or move members. When we do, we have to make both old and
> new optional, even when you actually have to specify exactly one. This
> isn't the end of the world, but it is trading one kind of blemish for
> another.
>
> Improving on that would involve a non-trivial extension to
> introspection. Pain right away, gain only when all management
> applications that matter grok it.
>
> I think the realistic assumption is that QMP blemishes stay.
>
> Of course, we'd prefer not to copy existing QMP blemishes into new
> interfaces. That's why we have SocketAddress for new code, and
> SocketAddressLegacy for old code. Workable for simple utility types. I
> don't have a good solution for "bigger" types.
If this is a common problem and we want to solve it in a declarative
way, presumably a way to define mappings similar to aliases is not
unthinkable. Apart from the effort to implement it, there is no reason
why two types on the external interface couldn't share a single
representation in C, just with different mappings.
So far I think we only did this for SocketAddressLegacy, though, so it
might not be a very high priority.
> > Now with this, it's still open whether the "standard" syntax should only
> > be supported in QMP or also in the CLI, and whether "alternate" syntax
> > should only be supported in the CLI or also in QMP.
> >
> > Is usability actually improved by rejecting the "standard" syntax on the
> > command line, or by rejecting "alternate" syntax in QMP? Hardly so. It's
> > also not compatibility. So what is the justification for the difference?
> > Maintainability? I honestly don't see the improvement there either.
> >
> > So I don't really see a reason for differences, but at the same time
> > it's also not a very important question to me. If you prefer restricting
> > things, so be it.
>
> I dislike adding "alternate" to machine-friendly QMP / CLI poses,
> because:
>
> * Adding "alternate" poses documentation and introspection problems.
> The introspection solutions will then necessitate management
> application updates. Anything we do becomes ABI immediately, so we
> better get it right on the first try.
>
> * Ignoring the documentation problems is a bad idea, because
> undocumented interfaces are.
>
> * If we ignore the introspection problems, then machines are well
> advised to stick to introspectable "standard". Why add "alternate"
> then? Complexity for nothing.
>
> * Adding "alternate" to existing interfaces is pretty much pointless.
> To use "alternate", machines first have to check whether this QEMU
> provides it, and if not, fall back to "standard". Just use "standard"
> and call it a day.
>
> * For new interfaces, the differences between "standard" and "alternate"
> are hopefully smaller than for old interfaces. Machines will likely
> not care for improvements "alternate" may provide over "standard".
>
> I haven't made up my mind on adding "standard" to human-friendly CLI.
Removing "alternate" from QMP is easy, at least as long as we don't want
to have a different set of "alternate" still enabled in QMP.
Removing "standard" from the CLI is much harder. If we can avoid this,
let's please avoid it.
> Too many ways to do the same thing are bound to confuse. We could end
> up with the standard way, the alternate way we provide for backward
> compatibility, the alternate way we provide for usability, and any
> combination thereof.
>
> Each of the three ways has its uses, though. Combinations not so much.
>
> Adding "standard" to human-friendly CLI only poses documentation
> problems (there is no introspection). If we limit the backward
> compatibility promise to machine-friendly, getting things wrong is no
> big deal.
That's the plan, but it requires getting machine friendly as a first
step so that there is a stable alternative and we can actually lower the
backwards compatibility promise for the existing syntax.
> >> I start at "human-friendly syntax should be as friendly as we can make
> >> it, except where we have to compromise, say for backward compatibility".
> >>
> >> This approach embraces differences. Mind, not differences just for
> >> differences sake, only where they help users.
> >>
> >> Additionally accepting the "standard" abstract syntax is useful only
> >> where it helps users.
> >>
> >> "Alternate" should be limited to human-friendly.
> >
> > I think there is a small inconsistency in this reasoning: You say that
> > differences must help users, but then this is not the measuring stick
> > you use in the next paragraph. If it were, you would be asking whether
> > rejecting "standard" abstract syntax helps users, rather than whether
> > adding it does. (Even more so because rejecting it is more work! Not
> > much more work, but it adds a little bit of complexity.)
>
> Additionally accepting "standard" complicates the interface and its
> documentation. Whether this helps users more than it hurts them is not
> obvious.
Fair. When it's unclear for users, let's consider developers? :-)
I really don't want to invest a lot of time and effort (and code
complexity) just to get "standard" rejected.
> > So it seems that in practice your approach is more like "different by
> > default, making it the same needs justification", whereas I am leaning
> > more towards "same by default, making it different needs justification".
> >
> > Your idea of "human-friendly syntax should be as friendly as we can make
> > it" isn't in conflict with either approach. The thing that the idea
> > might actually conflict with is our time budget.
>
> Reasonable people often have quite different ideas on "human-friendly".
>
> >> Different approaches, without doubt. Yet both have to deal with
> >> differences, and both could use similar techniques and machinery for
> >> that. You're proposing to do the bulk of the work with aliases, and to
> >> have a tree-transforming hook for the remainder. Makes sense to me.
> >> However, in sufficiently gnarly cases, we might have to bypass all this
> >> and keep using handwritten code until the backward compatibility promise
> >> is history: see 2. above.
> >
> > Possibly. I honestly can't tell how many of these cases we will have.
> > In all of -object, we had exactly one problematic option. This could
> > easily be handled in a tree-transforming hook.
> >
> >> In addition each approach has its own, special needs.
> >>
> >> Yours adds "alternate" syntax to QMP. This poses documentation and
> >> introspection problems. The introspection solutions will then
> >> necessitate management application updates.
> >>
> >> Mine trades these problems for others, namely how to generate parsers
> >> for two different syntaxes from one QAPI schema.
> >>
> >> Do I make sense?
> >
> > In the long run, won't we need documentation and introspection even for
> > things that are limited to the command line? Introspection is one of the
> > big features enabled by CLI QAPIfication.
>
> CLI introspection is about the machine-friendly syntax, not the
> human-friendly one, just like monitor introspection is about QMP, not
> HMP.
Good point.
> >> >> Valid argument. CLI with JSON argument should match QMP. Even when
> >> >> that means both are excessively nested.
> >> >>
> >> >> CLI with dotted keys is a different matter, in my opinion.
> >> >
> >> > I'm skipping quite a bit of text here, mainly because I think we need an
> >> > answer first if we really want to switch the keyval options to be
> >> > HMP-like in more than just the support status (see above).
> >> >
> >> > Obviously, the other relevant question would then be if it also ends up
> >> > like HMP in that most interesting functionality isn't even available and
> >> > you're forced to use QMP/JSON when you're serious.
> >>
> >> I figure this is because we have let "QMP first" degenerate into "QMP
> >> first, HMP never" sometimes, and we did so mostly because HMP is extra
> >> work that was hard to justify.
> >>
> >> What if you could get the 1:1 HMP command basically for free? It may
> >> not be the most human-friendly command possible. But it should be a
> >> start.
> >
> > How would you do this?
> >
> > The obvious way is mapping QMP 1:1 like we do for some of the command
> > line options. But you just argued that this is not what you would prefer
> > for the command line, so it's probably not what you have in mind for HMP
> > either?
>
> I think there are QMP commands where a 1:1 HMP command would be just
> fine.
>
> For others, it would be awkward, but that could still better than
> nothing.
>
> Infrastructure we build to QAPIfy the command line could probably be
> used to get less awkward HMP commands without writing so much code.
>
> Note I'm not talking about HMP commands somebody *wants* to write, say
> to provide convenience magic that is inappropriate for QMP, or
> specialized higher level commands where QMP only provides general
> building blocks.
So basically, if a handwritten HMP handler doesn't exist for a given QMP
command, just generate one automatically? Sounds reasonable enough to
me.
> >> > I guess I can go into more details once we have answered these
> >> > fundamental questions.
> >> >
> >> > (We could and should have talked about them and about whether you want
> >> > to have aliases at all a year ago, before going into detailed code
> >> > review and making error messages perfect in code that we might throw
> >> > away anyway...)
> >>
> >> I doubt you could possibly be more right than that!
> >
> > So what questions need to be answered before we can make a decision?
> >
> > You talked a lot about how you like QMP to be different from the command
> > line. So is restricting aliases to the command line enough to make them
> > acceptable? Or do we need larger changes? Should I just throw them away
> > and write translation code for -chardev manually?
>
> Fair question, but I'm out of time for right now. Let's continue
> tomorrow.
:-)
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-06 16:36 ` Kevin Wolf
@ 2021-10-07 11:06 ` Markus Armbruster
2021-10-07 16:12 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-10-07 11:06 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 06.10.2021 um 15:11 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>>
>> > Am 05.10.2021 um 15:49 hat Markus Armbruster geschrieben:
>> >> Kevin Wolf <kwolf@redhat.com> writes:
>> >>
>> >> > Am 02.10.2021 um 15:33 hat Markus Armbruster geschrieben:
>> >> >> I apologize for this wall of text. It's a desparate attempt to cut
>> >> >> through the complexity and my confusion, and make sense of the actual
>> >> >> problems we're trying to solve.
>> >> >>
>> >> >> So, what problems exactly are we trying to solve?
>> >> >
>> >> > I'll start with replying to your final question because I think it's
>> >> > more helpful to start with the big picture than with details.
>> >> >
>> >> > So tools like libvirt want to have a single consistent interface to
>> >> > configure things on startup and at runtime. We also intend to support
>> >> > configuration files that should this time support all of the options and
>> >> > not just a few chosen ones.
>> >>
>> >> Yes.
>> >>
>> >> > The hypothesis is that QAPIfying the command line is the correct
>> >> > solution for both of these problems, i.e. all available command line
>> >> > options must be present in the QAPI schema and will be processed by a
>> >> > single parser shared with QMP to make sure they are consistent.
>> >>
>> >> Yes.
>> >>
>> >> This leads us to JSON option arguments and configuration files.
>> >> Well-suited for management applications that already use QMP.
>> >>
>> >> > Adding QAPIfied versions of individual command line options are steps
>> >> > towards this goal. As soon as they exist for every option, the final
>> >> > conversion from an open coded getopt() loop (or in fact a hand crafted
>> >> > parser in the case of vl.c) to something generated from the QAPI schema
>> >> > should be reasonably easy.
>> >>
>> >> Yes.
>> >>
>> >> > You're right that adding a second JSON-based command line interface for
>> >> > every option can already achieve the goal of providing a unified
>> >> > external interface, at the cost of (interface and code) duplication. Is
>> >> > this duplication desirable? Certainly no. Is it acceptable? You might
>> >> > get different opinions on this one.
>> >>
>> >> We'd certainly prefer CLI options to match corresponding QMP commands
>> >> exactly.
>> >>
>> >> Unfortunately, existing CLI options deviate from corresponding QMP
>> >> commands, and existing CLI options without corresponding QMP commands
>> >> may violate QMP design rules.
>> >>
>> >> Note: these issues pertain to human-friendly option syntax. The
>> >> machine-friendly option syntax is still limited to just a few options,
>> >> and it does match QMP there.
>> >
>> > On the other hand, we only have a handful of existing options that are
>> > very complex. Most of them aren't even structured and just take a single
>> > scalar value. So I'm relatively sure that we know the big ones, and
>> > we're working on them.
>> >
>> > Another point here is that I would argue that there is a difference
>> > between existing options and human-friendly options. Not all existing
>> > options are human-friendly just because they aren't machine friendly
>> > either, and there is probably potential for human-friendly syntax that
>> > doesn't exist yet.
>> >
>> > So I would treat support for existing options (i.e. compatibility) and
>> > support for human-friendly options (i.e. usability) as two separate
>> > problems.
>>
>> That's fair.
>>
>> Instead of "human-friendly option syntax", I could've written "our
>> traditional option syntax (which is better suited to humans than to
>> machines, although it still fails to be truly friendly to anyone)", but
>> that's a tad long, isn't it?
>
> It is, but I had the impression that you used the same term for both
> things without making a distinction.
Unclear writing is usually a symptom of unclear thinking :)
>> >> > In my opinion, we should try to get rid of hand crafted parsers where
>> >> > it's reasonably possible, and take advantage of the single unified
>> >> > option structure that QAPI provides. -chardev specifically has a hand
>> >> > crafted parser that essentially duplicates the automatically generated
>> >> > QAPI visitor code, except for the nesting and some differences in option
>> >> > names.
>> >>
>> >> We should definitely parse JSON option arguments with the QAPI
>> >> machinery, and nothing more.
>> >
>> > Yes, no doubt. (And we don't even consistently do that yet, like
>> > device-add going through QemuOpts after going through QAPI and throwing
>> > away all type information and silently ignoring anything it doesn't know
>> > to handle.)
>>
>> At least device-add is the last remaining user of 'gen': false.
>>
>> The 'any' type also leaves type checking to handwritten code, but at
>> least it doesn't throw away type information first. Used by qom-get and
>> qom-set.
>
> Even if device-add wasn't 'gen': false, in theory nothing would stop it
> from going back from the QAPI C object through a output visitor and
> QemuOpts and throwing the type information away again. Of course, with
> explicit type checks in QAPI, we're less likely to mess up the other
> side because otherwise things just wouldn't work.
>
> But I actually know a subsystem where such conversions happen... *cough*
Lalalala, nothing to see here, move on...
>> >> > HMP in contrast means separate code for every command, or translated to
>> >> > the CLI for each command line option. HMP is not defined in QAPI, it's
>> >> > a separate thing that just happens to call into QAPI-based QMP code in
>> >> > the common case.
>> >> >
>> >> > Is this what you have in mind for non-JSON command line options?
>> >>
>> >> We should separate the idea "HMP wraps around QMP" from its
>> >> implementation as handwritten, boilerplate-heavy code.
>> >>
>> >> All but the latest, QAPI-based CLI options work pretty much like HMP: a
>> >> bit of code generation (hxtool), a mix of common and ad hoc parsers,
>> >> boring handwritten code to translate from parse tree to internal C
>> >> interfaces (which are often QMP command handlers), and to stitch it all
>> >> together.
>> >>
>> >> Reducing the amount of boring handwritten code is equally desirable for
>> >> HMP and CLI.
>> >>
>> >> So is specifying the interface in QAPI instead of older, less powerful
>> >> schema languages like hxtool and QemuOpts.
>> >>
>> >> There are at least three problems:
>> >>
>> >> 1. Specifying CLI in QAPI hasn't progressed beyond the proof-of-concept
>> >> stage.
>> >>
>> >> 2. Backward compatibility issues and doubts have defeated attempts to
>> >> replace gnarly stuff, in particular QemuOpts.
>> >>
>> >> 3. How to best bridge abstract syntax differences has been unclear.
>> >> Whether the differences are historical accidents or intentional for ease
>> >> of human use doesn't matter.
>> >>
>> >> Aliases feel like a contribution towards solving 3.
>> >>
>> >> I don't think 1. is particularly difficult. It's just a big chunk of
>> >> work that isn't really useful without solutions for 2. and 3.
>> >>
>> >> To me, 2. feels intractable head on. Perhaps we better bypass the
>> >> problem by weakening the compatibility promise like we did for HMP.
>> >
>> > Can we define 2 in a bit more specific terms? I feel the biggest part
>> > of 2 is actually 3.
>> >
>> > You mention QemuOpts, but how commonly are the potentially problematic
>> > features of it even used? Short booleans are deprecated and could be
>> > dropped in 6.2. merge_lists may or may not have to be replicated.
>> > I know building lists from repeated keys is said to be a thing, where
>> > does it happen? I've worked on some of the big ones (blockdev, chardev,
>> > object, device) and haven't come across it yet.
>>
>> Alright, I'll look for you :) Check out commit 2d6dcbf93fb, 396f935a9af,
>> 54cb65d8588, f1672e6f2b6. These are fairly recent, which is kind of
>> depressing. There may be more.
>
> Should we patch QemuOpts to disallow this by default and require setting
> QemuOptsList.legacy_enable_repeated_options just to avoid new instances
> sneaking in?
I honestly don't know.
On the one hand, the use of repeated keys for lists wasn't designed into
QemuOpts, it was a "clever" (ab)use of a QemuOpts implementation detail.
On the other hand, -vnc vnc=localhost:1,vnc=unix:/tmp/vnc feels
friendlier to human me than -vnc vnc.0=localhost:1,vnc.1=unix:/tmp/vnc.
Repeated keys just don't fit into the QAPI parsing pipeline. "Object
member names are unique" is baked into data structures and code. Quite
reasonable when you start with JSON.
But this is a digression.
>> When I wrote 2., I was specifically thinking of -object, where we
>> decided not to mess the human-friendly syntax in part because we didn't
>> fully understand what a conversion from QemuOpts to keyval could break.
>
> I seem to remember we were confident that only the various list hacks
> would be problematic. They only property in -object that makes use of
> them is memory-backend.host-nodes. We should deprecate the old syntax
> for it and move on.
Yes, we should work on an orderly transition away from QemuOpts.
>> > Can we have an intermediate state where CLI parsing involves both QAPI
>> > generated options and manually written ones? So that we can actually put
>> > the QAPIfication work already done to use instead of just having a
>> > promise that it will make things possible eventually, in a few years?
>>
>> We kind of sort of have that already: a few option arguments are
>> QAPI-based.
>>
>> Perhaps it's time to crack 1., but in a way that accommodates
>> handwritten options. Could be like "this option takes a string
>> argument, and the function to process the option is not to be generated,
>> only called (just like 'gen': false for commands). We may want to make
>> introspection describe the argument as "unknown", to distinguish it from
>> genuine string arguments.
>
> Maybe 'data': 'any' rather than 'gen': false if the keyval parser makes
> sense for the option.
Yes, providing for "keyval without visitor" may be useful.
> Or as a first step, just call into a single C
> function for any unknown option so that we don't have to split the big
> switch block immediately and can instead reduce it incrementally. Then
> 'data': 'any' is an intermediate step we can take when a function is too
> complicated to be fully QAPIfied in a single commit.
Yes, we could have some options defined in the QAPI schema, the
remainder in .hx, then stitch the generated code together. Enables
moving commands from .hx to the QAPI schema one by one.
I'm not sure that's simpler than replacing .hx wholesale.
> But yes, I guess cracking 1. is what I had in mind: Let QAPI control the
> command line parsing and only call out into handwritten code for not yet
> converted options. I think this would be a big step forward compared to
> the current state of only calling into QAPI for selected options because
> it makes it very obvious that new code should use QAPI, and it would
> also give us more a visible list of things that still need to be
> converted.
No argument.
I have (dusty) RFC patches.
>> >> > What I had in mind was using the schema to generate the necessary code,
>> >> > using the generic keyval parser everywhere, and just providing a hook
>> >> > where the QDict could be modified after the keyval parser and before the
>> >> > visitor. Most command line options would not have any explicit code for
>> >> > parsing input, only the declarative schema and the final option handler
>> >> > getting a QAPI type (which could actually be the corresponding QMP
>> >> > command handler in the common case).
>> >> >
>> >> > I think these are very different ideas. If we want HMP-like, then all
>> >> > the keyval based code paths we've built so far don't make much sense.
>> >>
>> >> I'm not sure the differences are "very" :)
>> >>
>> >> I think you start at "human-friendly and machine-friendly should use the
>> >> abstract syntax defined in the QAPI schema, except where human-friendly
>> >> has to differ, say for backward compatibility".
>> >>
>> >> This approach considers differences a blemish. The "standard" abstract
>> >> syntax (the one defined in the QAPI schema) should be accepted in
>> >> addition to the "alternate" one whenever possible, so "modern" use can
>> >> avoid the blemishes, and blemishes can be removed once they have fallen
>> >> out of use.
>> >>
>> >> "Alternate" should not be limited to human-friendly. Not limiting keeps
>> >> things consistent, which is good, because differences are bad.
>> >>
>> >> Is that a fair description?
>> >
>> > Hm, yes and no.
>> >
>> >
>> > I do think that making the overlap between "standard" and "alternate"
>> > abstract syntax as big as possible is a good thing because it means less
>> > translation has to go on between them, and ultimately the interfaces are
>> > more consistent with each other which actually improves the human
>> > friendliness for people who get in touch with both syntaxes.
>> >
>> > The -chardev conversion mostly deals with differences that I do consider
>> > a blemish: There is no real reason why options need to have different
>> > names on both interfaces, and there is also a lot of nesting in QMP for
>> > which there is no real reason.
>> >
>> > The reason why we need to accept both is compatibility, not usability.
>>
>> Compatibility requires us to accept both, but each in its place. It
>> doesn't actually require us to accept both in either place.
>
> Yes.
>
>> > There are also some differences that are justified by friendliness.
>> > Having "addr" as a nested struct in QMP just makes sense, and on the
>> > command line having it flattened makes more sense.
>> >
>> > So accepting "path" for "device" in ChardevHostdev is probably a case
>> > where switching both QMP and CLI to "modern" use and remove the
>> > "blemish" eventually makes sense to me. The nesting for "addr" isn't.
>> >
>> > We may even add more differences to make things human friendly. My
>> > standard for this is whether the difference serves only for
>> > compatibility (should go away eventually) or usability (should stay).
>>
>> In theory, cleaning up QMP is nice. In practice, we don't have a good
>> way to rename or move members. When we do, we have to make both old and
>> new optional, even when you actually have to specify exactly one. This
>> isn't the end of the world, but it is trading one kind of blemish for
>> another.
>>
>> Improving on that would involve a non-trivial extension to
>> introspection. Pain right away, gain only when all management
>> applications that matter grok it.
>>
>> I think the realistic assumption is that QMP blemishes stay.
>>
>> Of course, we'd prefer not to copy existing QMP blemishes into new
>> interfaces. That's why we have SocketAddress for new code, and
>> SocketAddressLegacy for old code. Workable for simple utility types. I
>> don't have a good solution for "bigger" types.
>
> If this is a common problem and we want to solve it in a declarative
> way, presumably a way to define mappings similar to aliases is not
> unthinkable. Apart from the effort to implement it, there is no reason
> why two types on the external interface couldn't share a single
> representation in C, just with different mappings.
True.
If we use aliases to provide multiple wire formats for a common C type,
the common C type can only be the most nested one. I'd prefer the least
nested one. We'll live.
> So far I think we only did this for SocketAddressLegacy, though, so it
> might not be a very high priority.
Yes.
>> > Now with this, it's still open whether the "standard" syntax should only
>> > be supported in QMP or also in the CLI, and whether "alternate" syntax
>> > should only be supported in the CLI or also in QMP.
>> >
>> > Is usability actually improved by rejecting the "standard" syntax on the
>> > command line, or by rejecting "alternate" syntax in QMP? Hardly so. It's
>> > also not compatibility. So what is the justification for the difference?
>> > Maintainability? I honestly don't see the improvement there either.
>> >
>> > So I don't really see a reason for differences, but at the same time
>> > it's also not a very important question to me. If you prefer restricting
>> > things, so be it.
>>
>> I dislike adding "alternate" to machine-friendly QMP / CLI poses,
>> because:
>>
>> * Adding "alternate" poses documentation and introspection problems.
>> The introspection solutions will then necessitate management
>> application updates. Anything we do becomes ABI immediately, so we
>> better get it right on the first try.
>>
>> * Ignoring the documentation problems is a bad idea, because
>> undocumented interfaces are.
>>
>> * If we ignore the introspection problems, then machines are well
>> advised to stick to introspectable "standard". Why add "alternate"
>> then? Complexity for nothing.
>>
>> * Adding "alternate" to existing interfaces is pretty much pointless.
>> To use "alternate", machines first have to check whether this QEMU
>> provides it, and if not, fall back to "standard". Just use "standard"
>> and call it a day.
>>
>> * For new interfaces, the differences between "standard" and "alternate"
>> are hopefully smaller than for old interfaces. Machines will likely
>> not care for improvements "alternate" may provide over "standard".
>>
>> I haven't made up my mind on adding "standard" to human-friendly CLI.
>
> Removing "alternate" from QMP is easy, at least as long as we don't want
> to have a different set of "alternate" still enabled in QMP.
>
> Removing "standard" from the CLI is much harder. If we can avoid this,
> let's please avoid it.
Noted.
>> Too many ways to do the same thing are bound to confuse. We could end
>> up with the standard way, the alternate way we provide for backward
>> compatibility, the alternate way we provide for usability, and any
>> combination thereof.
>>
>> Each of the three ways has its uses, though. Combinations not so much.
>>
>> Adding "standard" to human-friendly CLI only poses documentation
>> problems (there is no introspection). If we limit the backward
>> compatibility promise to machine-friendly, getting things wrong is no
>> big deal.
>
> That's the plan, but it requires getting machine friendly as a first
> step so that there is a stable alternative and we can actually lower the
> backwards compatibility promise for the existing syntax.
True.
>> >> I start at "human-friendly syntax should be as friendly as we can make
>> >> it, except where we have to compromise, say for backward compatibility".
>> >>
>> >> This approach embraces differences. Mind, not differences just for
>> >> differences sake, only where they help users.
>> >>
>> >> Additionally accepting the "standard" abstract syntax is useful only
>> >> where it helps users.
>> >>
>> >> "Alternate" should be limited to human-friendly.
>> >
>> > I think there is a small inconsistency in this reasoning: You say that
>> > differences must help users, but then this is not the measuring stick
>> > you use in the next paragraph. If it were, you would be asking whether
>> > rejecting "standard" abstract syntax helps users, rather than whether
>> > adding it does. (Even more so because rejecting it is more work! Not
>> > much more work, but it adds a little bit of complexity.)
>>
>> Additionally accepting "standard" complicates the interface and its
>> documentation. Whether this helps users more than it hurts them is not
>> obvious.
>
> Fair. When it's unclear for users, let's consider developers? :-)
>
> I really don't want to invest a lot of time and effort (and code
> complexity) just to get "standard" rejected.
I acknowledge the existence of working code :)
If we had understood the problem back then as well as we do now, you
might have written different code.
Aliases are for *adding* "alternate" to "standard" by nature.
To get "alternate" *instead* of "standard", something else might be a
better fit. "T2 inherits from T1 with the following renames" comes to
mind.
>> > So it seems that in practice your approach is more like "different by
>> > default, making it the same needs justification", whereas I am leaning
>> > more towards "same by default, making it different needs justification".
>> >
>> > Your idea of "human-friendly syntax should be as friendly as we can make
>> > it" isn't in conflict with either approach. The thing that the idea
>> > might actually conflict with is our time budget.
>>
>> Reasonable people often have quite different ideas on "human-friendly".
>>
>> >> Different approaches, without doubt. Yet both have to deal with
>> >> differences, and both could use similar techniques and machinery for
>> >> that. You're proposing to do the bulk of the work with aliases, and to
>> >> have a tree-transforming hook for the remainder. Makes sense to me.
>> >> However, in sufficiently gnarly cases, we might have to bypass all this
>> >> and keep using handwritten code until the backward compatibility promise
>> >> is history: see 2. above.
>> >
>> > Possibly. I honestly can't tell how many of these cases we will have.
>> > In all of -object, we had exactly one problematic option. This could
>> > easily be handled in a tree-transforming hook.
>> >
>> >> In addition each approach has its own, special needs.
>> >>
>> >> Yours adds "alternate" syntax to QMP. This poses documentation and
>> >> introspection problems. The introspection solutions will then
>> >> necessitate management application updates.
>> >>
>> >> Mine trades these problems for others, namely how to generate parsers
>> >> for two different syntaxes from one QAPI schema.
>> >>
>> >> Do I make sense?
>> >
>> > In the long run, won't we need documentation and introspection even for
>> > things that are limited to the command line? Introspection is one of the
>> > big features enabled by CLI QAPIfication.
>>
>> CLI introspection is about the machine-friendly syntax, not the
>> human-friendly one, just like monitor introspection is about QMP, not
>> HMP.
>
> Good point.
>
>> >> >> Valid argument. CLI with JSON argument should match QMP. Even when
>> >> >> that means both are excessively nested.
>> >> >>
>> >> >> CLI with dotted keys is a different matter, in my opinion.
>> >> >
>> >> > I'm skipping quite a bit of text here, mainly because I think we need an
>> >> > answer first if we really want to switch the keyval options to be
>> >> > HMP-like in more than just the support status (see above).
>> >> >
>> >> > Obviously, the other relevant question would then be if it also ends up
>> >> > like HMP in that most interesting functionality isn't even available and
>> >> > you're forced to use QMP/JSON when you're serious.
>> >>
>> >> I figure this is because we have let "QMP first" degenerate into "QMP
>> >> first, HMP never" sometimes, and we did so mostly because HMP is extra
>> >> work that was hard to justify.
>> >>
>> >> What if you could get the 1:1 HMP command basically for free? It may
>> >> not be the most human-friendly command possible. But it should be a
>> >> start.
>> >
>> > How would you do this?
>> >
>> > The obvious way is mapping QMP 1:1 like we do for some of the command
>> > line options. But you just argued that this is not what you would prefer
>> > for the command line, so it's probably not what you have in mind for HMP
>> > either?
>>
>> I think there are QMP commands where a 1:1 HMP command would be just
>> fine.
>>
>> For others, it would be awkward, but that could still better than
>> nothing.
>>
>> Infrastructure we build to QAPIfy the command line could probably be
>> used to get less awkward HMP commands without writing so much code.
>>
>> Note I'm not talking about HMP commands somebody *wants* to write, say
>> to provide convenience magic that is inappropriate for QMP, or
>> specialized higher level commands where QMP only provides general
>> building blocks.
>
> So basically, if a handwritten HMP handler doesn't exist for a given QMP
> command, just generate one automatically? Sounds reasonable enough to
> me.
Fun project, but I can't make it mine.
>> >> > I guess I can go into more details once we have answered these
>> >> > fundamental questions.
>> >> >
>> >> > (We could and should have talked about them and about whether you want
>> >> > to have aliases at all a year ago, before going into detailed code
>> >> > review and making error messages perfect in code that we might throw
>> >> > away anyway...)
>> >>
>> >> I doubt you could possibly be more right than that!
>> >
>> > So what questions need to be answered before we can make a decision?
>> >
>> > You talked a lot about how you like QMP to be different from the command
>> > line. So is restricting aliases to the command line enough to make them
>> > acceptable? Or do we need larger changes? Should I just throw them away
>> > and write translation code for -chardev manually?
>>
>> Fair question, but I'm out of time for right now. Let's continue
>> tomorrow.
>
> :-)
What's the smallest set of aliases sufficient to make your -chardev
QAPIfication work?
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-07 11:06 ` Markus Armbruster
@ 2021-10-07 16:12 ` Kevin Wolf
2021-10-08 10:17 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-10-07 16:12 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 07.10.2021 um 13:06 hat Markus Armbruster geschrieben:
> >> >> > HMP in contrast means separate code for every command, or translated to
> >> >> > the CLI for each command line option. HMP is not defined in QAPI, it's
> >> >> > a separate thing that just happens to call into QAPI-based QMP code in
> >> >> > the common case.
> >> >> >
> >> >> > Is this what you have in mind for non-JSON command line options?
> >> >>
> >> >> We should separate the idea "HMP wraps around QMP" from its
> >> >> implementation as handwritten, boilerplate-heavy code.
> >> >>
> >> >> All but the latest, QAPI-based CLI options work pretty much like HMP: a
> >> >> bit of code generation (hxtool), a mix of common and ad hoc parsers,
> >> >> boring handwritten code to translate from parse tree to internal C
> >> >> interfaces (which are often QMP command handlers), and to stitch it all
> >> >> together.
> >> >>
> >> >> Reducing the amount of boring handwritten code is equally desirable for
> >> >> HMP and CLI.
> >> >>
> >> >> So is specifying the interface in QAPI instead of older, less powerful
> >> >> schema languages like hxtool and QemuOpts.
> >> >>
> >> >> There are at least three problems:
> >> >>
> >> >> 1. Specifying CLI in QAPI hasn't progressed beyond the proof-of-concept
> >> >> stage.
> >> >>
> >> >> 2. Backward compatibility issues and doubts have defeated attempts to
> >> >> replace gnarly stuff, in particular QemuOpts.
> >> >>
> >> >> 3. How to best bridge abstract syntax differences has been unclear.
> >> >> Whether the differences are historical accidents or intentional for ease
> >> >> of human use doesn't matter.
> >> >>
> >> >> Aliases feel like a contribution towards solving 3.
> >> >>
> >> >> I don't think 1. is particularly difficult. It's just a big chunk of
> >> >> work that isn't really useful without solutions for 2. and 3.
> >> >>
> >> >> To me, 2. feels intractable head on. Perhaps we better bypass the
> >> >> problem by weakening the compatibility promise like we did for HMP.
> >> >
> >> > Can we define 2 in a bit more specific terms? I feel the biggest part
> >> > of 2 is actually 3.
> >> >
> >> > You mention QemuOpts, but how commonly are the potentially problematic
> >> > features of it even used? Short booleans are deprecated and could be
> >> > dropped in 6.2. merge_lists may or may not have to be replicated.
> >> > I know building lists from repeated keys is said to be a thing, where
> >> > does it happen? I've worked on some of the big ones (blockdev, chardev,
> >> > object, device) and haven't come across it yet.
> >>
> >> Alright, I'll look for you :) Check out commit 2d6dcbf93fb, 396f935a9af,
> >> 54cb65d8588, f1672e6f2b6. These are fairly recent, which is kind of
> >> depressing. There may be more.
> >
> > Should we patch QemuOpts to disallow this by default and require setting
> > QemuOptsList.legacy_enable_repeated_options just to avoid new instances
> > sneaking in?
>
> I honestly don't know.
>
> On the one hand, the use of repeated keys for lists wasn't designed into
> QemuOpts, it was a "clever" (ab)use of a QemuOpts implementation detail.
>
> On the other hand, -vnc vnc=localhost:1,vnc=unix:/tmp/vnc feels
> friendlier to human me than -vnc vnc.0=localhost:1,vnc.1=unix:/tmp/vnc.
>
> Repeated keys just don't fit into the QAPI parsing pipeline. "Object
> member names are unique" is baked into data structures and code. Quite
> reasonable when you start with JSON.
>
> But this is a digression.
Making a list out of repeated keys should be easy in the keyval parser.
Turning a single or an absent key into a list in the right cirumstances
is harder because the keyval parser knows nothing about the schema.
Of course, we use the keyval parser in combination with the keyval
visitor, and when visiting lists, we could accept absent or scalar
values by wrapping a list around the input. (Doing the same for objects
would feel a bit crazier, but I don't think we need that anyway.)
So I think this is actually doable if we want.
But in general, the split between parser and schema keeps giving us
trouble, and visitors not knowing anything about future visitor calls
can make things harder, too. Probably too late to use a fundamentally
different way to parse things, though.
> >> > Can we have an intermediate state where CLI parsing involves both QAPI
> >> > generated options and manually written ones? So that we can actually put
> >> > the QAPIfication work already done to use instead of just having a
> >> > promise that it will make things possible eventually, in a few years?
> >>
> >> We kind of sort of have that already: a few option arguments are
> >> QAPI-based.
> >>
> >> Perhaps it's time to crack 1., but in a way that accommodates
> >> handwritten options. Could be like "this option takes a string
> >> argument, and the function to process the option is not to be generated,
> >> only called (just like 'gen': false for commands). We may want to make
> >> introspection describe the argument as "unknown", to distinguish it from
> >> genuine string arguments.
> >
> > Maybe 'data': 'any' rather than 'gen': false if the keyval parser makes
> > sense for the option.
>
> Yes, providing for "keyval without visitor" may be useful.
>
> > Or as a first step, just call into a single C
> > function for any unknown option so that we don't have to split the big
> > switch block immediately and can instead reduce it incrementally. Then
> > 'data': 'any' is an intermediate step we can take when a function is too
> > complicated to be fully QAPIfied in a single commit.
>
> Yes, we could have some options defined in the QAPI schema, the
> remainder in .hx, then stitch the generated code together. Enables
> moving commands from .hx to the QAPI schema one by one.
>
> I'm not sure that's simpler than replacing .hx wholesale.
I didn't even think of .hx any more. So the above refers only to the
parsing loop in qemu_init(). Moving things from .hx to the schema can
probably be done mostly independent from converting the implementation
when you start with 'gen': false.
Actually, the other thing .hx contains is help texts. I seem to remember
that you had another half-finished set of patches to display help from
QAPI documentation?
> > But yes, I guess cracking 1. is what I had in mind: Let QAPI control the
> > command line parsing and only call out into handwritten code for not yet
> > converted options. I think this would be a big step forward compared to
> > the current state of only calling into QAPI for selected options because
> > it makes it very obvious that new code should use QAPI, and it would
> > also give us more a visible list of things that still need to be
> > converted.
>
> No argument.
>
> I have (dusty) RFC patches.
Time to undust them?
> >> >> > What I had in mind was using the schema to generate the necessary code,
> >> >> > using the generic keyval parser everywhere, and just providing a hook
> >> >> > where the QDict could be modified after the keyval parser and before the
> >> >> > visitor. Most command line options would not have any explicit code for
> >> >> > parsing input, only the declarative schema and the final option handler
> >> >> > getting a QAPI type (which could actually be the corresponding QMP
> >> >> > command handler in the common case).
> >> >> >
> >> >> > I think these are very different ideas. If we want HMP-like, then all
> >> >> > the keyval based code paths we've built so far don't make much sense.
> >> >>
> >> >> I'm not sure the differences are "very" :)
> >> >>
> >> >> I think you start at "human-friendly and machine-friendly should use the
> >> >> abstract syntax defined in the QAPI schema, except where human-friendly
> >> >> has to differ, say for backward compatibility".
> >> >>
> >> >> This approach considers differences a blemish. The "standard" abstract
> >> >> syntax (the one defined in the QAPI schema) should be accepted in
> >> >> addition to the "alternate" one whenever possible, so "modern" use can
> >> >> avoid the blemishes, and blemishes can be removed once they have fallen
> >> >> out of use.
> >> >>
> >> >> "Alternate" should not be limited to human-friendly. Not limiting keeps
> >> >> things consistent, which is good, because differences are bad.
> >> >>
> >> >> Is that a fair description?
> >> >
> >> > Hm, yes and no.
> >> >
> >> >
> >> > I do think that making the overlap between "standard" and "alternate"
> >> > abstract syntax as big as possible is a good thing because it means less
> >> > translation has to go on between them, and ultimately the interfaces are
> >> > more consistent with each other which actually improves the human
> >> > friendliness for people who get in touch with both syntaxes.
> >> >
> >> > The -chardev conversion mostly deals with differences that I do consider
> >> > a blemish: There is no real reason why options need to have different
> >> > names on both interfaces, and there is also a lot of nesting in QMP for
> >> > which there is no real reason.
> >> >
> >> > The reason why we need to accept both is compatibility, not usability.
> >>
> >> Compatibility requires us to accept both, but each in its place. It
> >> doesn't actually require us to accept both in either place.
> >
> > Yes.
> >
> >> > There are also some differences that are justified by friendliness.
> >> > Having "addr" as a nested struct in QMP just makes sense, and on the
> >> > command line having it flattened makes more sense.
> >> >
> >> > So accepting "path" for "device" in ChardevHostdev is probably a case
> >> > where switching both QMP and CLI to "modern" use and remove the
> >> > "blemish" eventually makes sense to me. The nesting for "addr" isn't.
> >> >
> >> > We may even add more differences to make things human friendly. My
> >> > standard for this is whether the difference serves only for
> >> > compatibility (should go away eventually) or usability (should stay).
> >>
> >> In theory, cleaning up QMP is nice. In practice, we don't have a good
> >> way to rename or move members. When we do, we have to make both old and
> >> new optional, even when you actually have to specify exactly one. This
> >> isn't the end of the world, but it is trading one kind of blemish for
> >> another.
> >>
> >> Improving on that would involve a non-trivial extension to
> >> introspection. Pain right away, gain only when all management
> >> applications that matter grok it.
> >>
> >> I think the realistic assumption is that QMP blemishes stay.
> >>
> >> Of course, we'd prefer not to copy existing QMP blemishes into new
> >> interfaces. That's why we have SocketAddress for new code, and
> >> SocketAddressLegacy for old code. Workable for simple utility types. I
> >> don't have a good solution for "bigger" types.
> >
> > If this is a common problem and we want to solve it in a declarative
> > way, presumably a way to define mappings similar to aliases is not
> > unthinkable. Apart from the effort to implement it, there is no reason
> > why two types on the external interface couldn't share a single
> > representation in C, just with different mappings.
>
> True.
>
> If we use aliases to provide multiple wire formats for a common C type,
> the common C type can only be the most nested one. I'd prefer the least
> nested one. We'll live.
This direction would actually have been easier to implement, but it's
also much less powerful and couldn't have solved the problem I tried to
solve.
It could be used for flattening simple unions and for local aliases, but
pulling things like addr.* to the top level would be impossible unless
you directly embed the fields of SocketAddress into ChardevBackend.
> >> >> I start at "human-friendly syntax should be as friendly as we can make
> >> >> it, except where we have to compromise, say for backward compatibility".
> >> >>
> >> >> This approach embraces differences. Mind, not differences just for
> >> >> differences sake, only where they help users.
> >> >>
> >> >> Additionally accepting the "standard" abstract syntax is useful only
> >> >> where it helps users.
> >> >>
> >> >> "Alternate" should be limited to human-friendly.
> >> >
> >> > I think there is a small inconsistency in this reasoning: You say that
> >> > differences must help users, but then this is not the measuring stick
> >> > you use in the next paragraph. If it were, you would be asking whether
> >> > rejecting "standard" abstract syntax helps users, rather than whether
> >> > adding it does. (Even more so because rejecting it is more work! Not
> >> > much more work, but it adds a little bit of complexity.)
> >>
> >> Additionally accepting "standard" complicates the interface and its
> >> documentation. Whether this helps users more than it hurts them is not
> >> obvious.
> >
> > Fair. When it's unclear for users, let's consider developers? :-)
> >
> > I really don't want to invest a lot of time and effort (and code
> > complexity) just to get "standard" rejected.
>
> I acknowledge the existence of working code :)
>
> If we had understood the problem back then as well as we do now, you
> might have written different code.
s/the problem/the maintainer's preferences/ ;-)
The implicit requirement I wasn't aware of is "there should be only one
way to express the same thing on each external interface". Aliases go
against that by their very nature - they provide a second way.
> Aliases are for *adding* "alternate" to "standard" by nature.
>
> To get "alternate" *instead* of "standard", something else might be a
> better fit. "T2 inherits from T1 with the following renames" comes to
> mind.
Right.
> >> >> > I guess I can go into more details once we have answered these
> >> >> > fundamental questions.
> >> >> >
> >> >> > (We could and should have talked about them and about whether you want
> >> >> > to have aliases at all a year ago, before going into detailed code
> >> >> > review and making error messages perfect in code that we might throw
> >> >> > away anyway...)
> >> >>
> >> >> I doubt you could possibly be more right than that!
> >> >
> >> > So what questions need to be answered before we can make a decision?
> >> >
> >> > You talked a lot about how you like QMP to be different from the command
> >> > line. So is restricting aliases to the command line enough to make them
> >> > acceptable? Or do we need larger changes? Should I just throw them away
> >> > and write translation code for -chardev manually?
> >>
> >> Fair question, but I'm out of time for right now. Let's continue
> >> tomorrow.
> >
> > :-)
>
> What's the smallest set of aliases sufficient to make your -chardev
> QAPIfication work?
How do you define "make the QAPIfication work"?
The -chardev conversion adds the following aliases to the schema:
* SocketAddressLegacy: Flatten a simple union (wildcard alias +
non-local alias)
{ 'source': ['data'] },
{ 'name': 'fd', 'source': ['data', 'str'] }
* ChardevFile: Rename local member
{ 'name': 'path', 'source': ['out'] }
* ChardevHostdev: Rename local member
{ 'name': 'path', 'source': ['device'] }
* ChardevSocket: Flatten embedded struct (wildcard alias)
{ 'source': ['addr'] }
* ChardevUdp: Flatten two embedded structs with renaming (wildcard
alias + non-local alias)
{ 'source': ['remote'] },
{ 'name': 'localaddr', 'source': ['local', 'data', 'host'] },
{ 'name': 'localport', 'source': ['local', 'data', 'port'] }
* ChardevSpiceChannel: Rename local member
{ 'name': 'name', 'source': ['type'] }
* ChardevSpicePort: Rename local member
{ 'name': 'name', 'source': ['fqdn'] }
* ChardevBackend: Flatten a simple union (wildcard alias)
{ 'source': ['data'] }
* ChardevOptions: Flatten embedded struct (wildcard alias)
{ 'source': ['backend'] }
The deeper they are nested in the type hierarchy, especially when unions
with different variants come into play, the nastier they are to replace
with C code. The C code stays the simplest if all of the aliases are
there, and it gets uglier the more of them you leave out.
I don't know your idea of "sufficient", so I'll leave mapping that to a
scale of sufficiency to you.
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-07 16:12 ` Kevin Wolf
@ 2021-10-08 10:17 ` Markus Armbruster
2021-10-12 14:00 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-10-08 10:17 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 07.10.2021 um 13:06 hat Markus Armbruster geschrieben:
>> >> >> > HMP in contrast means separate code for every command, or translated to
>> >> >> > the CLI for each command line option. HMP is not defined in QAPI, it's
>> >> >> > a separate thing that just happens to call into QAPI-based QMP code in
>> >> >> > the common case.
>> >> >> >
>> >> >> > Is this what you have in mind for non-JSON command line options?
>> >> >>
>> >> >> We should separate the idea "HMP wraps around QMP" from its
>> >> >> implementation as handwritten, boilerplate-heavy code.
>> >> >>
>> >> >> All but the latest, QAPI-based CLI options work pretty much like HMP: a
>> >> >> bit of code generation (hxtool), a mix of common and ad hoc parsers,
>> >> >> boring handwritten code to translate from parse tree to internal C
>> >> >> interfaces (which are often QMP command handlers), and to stitch it all
>> >> >> together.
>> >> >>
>> >> >> Reducing the amount of boring handwritten code is equally desirable for
>> >> >> HMP and CLI.
>> >> >>
>> >> >> So is specifying the interface in QAPI instead of older, less powerful
>> >> >> schema languages like hxtool and QemuOpts.
>> >> >>
>> >> >> There are at least three problems:
>> >> >>
>> >> >> 1. Specifying CLI in QAPI hasn't progressed beyond the proof-of-concept
>> >> >> stage.
>> >> >>
>> >> >> 2. Backward compatibility issues and doubts have defeated attempts to
>> >> >> replace gnarly stuff, in particular QemuOpts.
>> >> >>
>> >> >> 3. How to best bridge abstract syntax differences has been unclear.
>> >> >> Whether the differences are historical accidents or intentional for ease
>> >> >> of human use doesn't matter.
>> >> >>
>> >> >> Aliases feel like a contribution towards solving 3.
>> >> >>
>> >> >> I don't think 1. is particularly difficult. It's just a big chunk of
>> >> >> work that isn't really useful without solutions for 2. and 3.
>> >> >>
>> >> >> To me, 2. feels intractable head on. Perhaps we better bypass the
>> >> >> problem by weakening the compatibility promise like we did for HMP.
>> >> >
>> >> > Can we define 2 in a bit more specific terms? I feel the biggest part
>> >> > of 2 is actually 3.
>> >> >
>> >> > You mention QemuOpts, but how commonly are the potentially problematic
>> >> > features of it even used? Short booleans are deprecated and could be
>> >> > dropped in 6.2. merge_lists may or may not have to be replicated.
>> >> > I know building lists from repeated keys is said to be a thing, where
>> >> > does it happen? I've worked on some of the big ones (blockdev, chardev,
>> >> > object, device) and haven't come across it yet.
>> >>
>> >> Alright, I'll look for you :) Check out commit 2d6dcbf93fb, 396f935a9af,
>> >> 54cb65d8588, f1672e6f2b6. These are fairly recent, which is kind of
>> >> depressing. There may be more.
>> >
>> > Should we patch QemuOpts to disallow this by default and require setting
>> > QemuOptsList.legacy_enable_repeated_options just to avoid new instances
>> > sneaking in?
>>
>> I honestly don't know.
>>
>> On the one hand, the use of repeated keys for lists wasn't designed into
>> QemuOpts, it was a "clever" (ab)use of a QemuOpts implementation detail.
>>
>> On the other hand, -vnc vnc=localhost:1,vnc=unix:/tmp/vnc feels
>> friendlier to human me than -vnc vnc.0=localhost:1,vnc.1=unix:/tmp/vnc.
>>
>> Repeated keys just don't fit into the QAPI parsing pipeline. "Object
>> member names are unique" is baked into data structures and code. Quite
>> reasonable when you start with JSON.
>>
>> But this is a digression.
>
> Making a list out of repeated keys should be easy in the keyval parser.
> Turning a single or an absent key into a list in the right cirumstances
> is harder because the keyval parser knows nothing about the schema.
This is our JSON parsing pipeline:
JSON qobject
parser visitor
text ---> QObject ---> C object
Our representation of JSON objects as QDict assumes unique keys.
The JSON parser treats duplicate keys as an error.
The dotted keys parsing pipeline came later, and has the same structure:
keyval qobject
parser visitor
text ---> QObject ---> C object
It differs in the details, though: scalars can only be QString, and the
visitor is tailored for that.
It's identical in the detail that matters here: can only do unique keys.
The keyval parser accepts duplicate keys. The last value wins.
The clean way to support "repeated keys are lists in list context, else
last one wins" is to change the abstract syntax tree to reflect that:
instead of QDict, we use something that represents all the (key, value)
pairs given. When the visitor visits a list, it uses all the pairs.
When it visits a scalar, it uses the last one.
The gap between the two abstract syntax trees widens. Isn't that bad;
they are different already, and using the same C type for both risks
confusing one for the other.
Falls apart for type 'any': a single key could be a scalar or a list
with one member, but the visitor can't decide. Likewise, an absent key
could be an absent optional list or an empty list.
Note that "repeated keys are lists in list context, else last one wins"
is internally inconsistent: you can overwrite a key's value, except when
you can't.
QemuOpts makes use of the overwrite feature when it combines multiple
sources. Doesn't apply to keyval. Perhaps I should have chosen to make
keyval incompatible there.
> Of course, we use the keyval parser in combination with the keyval
> visitor, and when visiting lists, we could accept absent or scalar
> values by wrapping a list around the input. (Doing the same for objects
> would feel a bit crazier, but I don't think we need that anyway.)
This is how the opts visitor works, and I don't like it.
> So I think this is actually doable if we want.
>
> But in general, the split between parser and schema keeps giving us
> trouble, and visitors not knowing anything about future visitor calls
> can make things harder, too. Probably too late to use a fundamentally
> different way to parse things, though.
The split between JSON parser and visitor makes sense, and simplifies
the code.
The same split is problematic for dotted keys, because there the syntax
conveys less structure. We accepted the resulting flaws (see "Design
flaw:" in util.keyval.c) to keep the code simple.
To truly do better, we'd have to replace the dotted keys parsing
pipeline wholesale, I believe: do everything in a new visitor, so the
visitor can guide the parsing. Still flawed for 'any', though.
>> >> > Can we have an intermediate state where CLI parsing involves both QAPI
>> >> > generated options and manually written ones? So that we can actually put
>> >> > the QAPIfication work already done to use instead of just having a
>> >> > promise that it will make things possible eventually, in a few years?
>> >>
>> >> We kind of sort of have that already: a few option arguments are
>> >> QAPI-based.
>> >>
>> >> Perhaps it's time to crack 1., but in a way that accommodates
>> >> handwritten options. Could be like "this option takes a string
>> >> argument, and the function to process the option is not to be generated,
>> >> only called (just like 'gen': false for commands). We may want to make
>> >> introspection describe the argument as "unknown", to distinguish it from
>> >> genuine string arguments.
>> >
>> > Maybe 'data': 'any' rather than 'gen': false if the keyval parser makes
>> > sense for the option.
>>
>> Yes, providing for "keyval without visitor" may be useful.
>>
>> > Or as a first step, just call into a single C
>> > function for any unknown option so that we don't have to split the big
>> > switch block immediately and can instead reduce it incrementally. Then
>> > 'data': 'any' is an intermediate step we can take when a function is too
>> > complicated to be fully QAPIfied in a single commit.
>>
>> Yes, we could have some options defined in the QAPI schema, the
>> remainder in .hx, then stitch the generated code together. Enables
>> moving commands from .hx to the QAPI schema one by one.
>>
>> I'm not sure that's simpler than replacing .hx wholesale.
>
> I didn't even think of .hx any more. So the above refers only to the
> parsing loop in qemu_init(). Moving things from .hx to the schema can
> probably be done mostly independent from converting the implementation
> when you start with 'gen': false.
Once we have the means to define options in the QAPI schema, replacing
.hx wholesale is possible, and probably less painful than moving options
one by one over time.
> Actually, the other thing .hx contains is help texts. I seem to remember
> that you had another half-finished set of patches to display help from
> QAPI documentation?
I toyed with generating "better than nothing" help for the wire format
with a visitor. Doc comments aren't visible there.
Doc comments are basically the reference manual interpolated into schema
code. A way to say "show me the reference manual for -frobnicate" could
be useful. But it's not what -help shows today. The reference manual
is much, much more verbose than -help. If we want to keep -help terse,
we'll need a way to write text for it.
>> > But yes, I guess cracking 1. is what I had in mind: Let QAPI control the
>> > command line parsing and only call out into handwritten code for not yet
>> > converted options. I think this would be a big step forward compared to
>> > the current state of only calling into QAPI for selected options because
>> > it makes it very obvious that new code should use QAPI, and it would
>> > also give us more a visible list of things that still need to be
>> > converted.
>>
>> No argument.
>>
>> I have (dusty) RFC patches.
>
> Time to undust them?
I'd like to.
>> >> >> > What I had in mind was using the schema to generate the necessary code,
>> >> >> > using the generic keyval parser everywhere, and just providing a hook
>> >> >> > where the QDict could be modified after the keyval parser and before the
>> >> >> > visitor. Most command line options would not have any explicit code for
>> >> >> > parsing input, only the declarative schema and the final option handler
>> >> >> > getting a QAPI type (which could actually be the corresponding QMP
>> >> >> > command handler in the common case).
>> >> >> >
>> >> >> > I think these are very different ideas. If we want HMP-like, then all
>> >> >> > the keyval based code paths we've built so far don't make much sense.
>> >> >>
>> >> >> I'm not sure the differences are "very" :)
>> >> >>
>> >> >> I think you start at "human-friendly and machine-friendly should use the
>> >> >> abstract syntax defined in the QAPI schema, except where human-friendly
>> >> >> has to differ, say for backward compatibility".
>> >> >>
>> >> >> This approach considers differences a blemish. The "standard" abstract
>> >> >> syntax (the one defined in the QAPI schema) should be accepted in
>> >> >> addition to the "alternate" one whenever possible, so "modern" use can
>> >> >> avoid the blemishes, and blemishes can be removed once they have fallen
>> >> >> out of use.
>> >> >>
>> >> >> "Alternate" should not be limited to human-friendly. Not limiting keeps
>> >> >> things consistent, which is good, because differences are bad.
>> >> >>
>> >> >> Is that a fair description?
>> >> >
>> >> > Hm, yes and no.
>> >> >
>> >> >
>> >> > I do think that making the overlap between "standard" and "alternate"
>> >> > abstract syntax as big as possible is a good thing because it means less
>> >> > translation has to go on between them, and ultimately the interfaces are
>> >> > more consistent with each other which actually improves the human
>> >> > friendliness for people who get in touch with both syntaxes.
>> >> >
>> >> > The -chardev conversion mostly deals with differences that I do consider
>> >> > a blemish: There is no real reason why options need to have different
>> >> > names on both interfaces, and there is also a lot of nesting in QMP for
>> >> > which there is no real reason.
>> >> >
>> >> > The reason why we need to accept both is compatibility, not usability.
>> >>
>> >> Compatibility requires us to accept both, but each in its place. It
>> >> doesn't actually require us to accept both in either place.
>> >
>> > Yes.
>> >
>> >> > There are also some differences that are justified by friendliness.
>> >> > Having "addr" as a nested struct in QMP just makes sense, and on the
>> >> > command line having it flattened makes more sense.
>> >> >
>> >> > So accepting "path" for "device" in ChardevHostdev is probably a case
>> >> > where switching both QMP and CLI to "modern" use and remove the
>> >> > "blemish" eventually makes sense to me. The nesting for "addr" isn't.
>> >> >
>> >> > We may even add more differences to make things human friendly. My
>> >> > standard for this is whether the difference serves only for
>> >> > compatibility (should go away eventually) or usability (should stay).
>> >>
>> >> In theory, cleaning up QMP is nice. In practice, we don't have a good
>> >> way to rename or move members. When we do, we have to make both old and
>> >> new optional, even when you actually have to specify exactly one. This
>> >> isn't the end of the world, but it is trading one kind of blemish for
>> >> another.
>> >>
>> >> Improving on that would involve a non-trivial extension to
>> >> introspection. Pain right away, gain only when all management
>> >> applications that matter grok it.
>> >>
>> >> I think the realistic assumption is that QMP blemishes stay.
>> >>
>> >> Of course, we'd prefer not to copy existing QMP blemishes into new
>> >> interfaces. That's why we have SocketAddress for new code, and
>> >> SocketAddressLegacy for old code. Workable for simple utility types. I
>> >> don't have a good solution for "bigger" types.
>> >
>> > If this is a common problem and we want to solve it in a declarative
>> > way, presumably a way to define mappings similar to aliases is not
>> > unthinkable. Apart from the effort to implement it, there is no reason
>> > why two types on the external interface couldn't share a single
>> > representation in C, just with different mappings.
>>
>> True.
>>
>> If we use aliases to provide multiple wire formats for a common C type,
>> the common C type can only be the most nested one. I'd prefer the least
>> nested one. We'll live.
>
> This direction would actually have been easier to implement, but it's
> also much less powerful and couldn't have solved the problem I tried to
> solve.
>
> It could be used for flattening simple unions and for local aliases, but
> pulling things like addr.* to the top level would be impossible unless
> you directly embed the fields of SocketAddress into ChardevBackend.
>
>> >> >> I start at "human-friendly syntax should be as friendly as we can make
>> >> >> it, except where we have to compromise, say for backward compatibility".
>> >> >>
>> >> >> This approach embraces differences. Mind, not differences just for
>> >> >> differences sake, only where they help users.
>> >> >>
>> >> >> Additionally accepting the "standard" abstract syntax is useful only
>> >> >> where it helps users.
>> >> >>
>> >> >> "Alternate" should be limited to human-friendly.
>> >> >
>> >> > I think there is a small inconsistency in this reasoning: You say that
>> >> > differences must help users, but then this is not the measuring stick
>> >> > you use in the next paragraph. If it were, you would be asking whether
>> >> > rejecting "standard" abstract syntax helps users, rather than whether
>> >> > adding it does. (Even more so because rejecting it is more work! Not
>> >> > much more work, but it adds a little bit of complexity.)
>> >>
>> >> Additionally accepting "standard" complicates the interface and its
>> >> documentation. Whether this helps users more than it hurts them is not
>> >> obvious.
>> >
>> > Fair. When it's unclear for users, let's consider developers? :-)
>> >
>> > I really don't want to invest a lot of time and effort (and code
>> > complexity) just to get "standard" rejected.
>>
>> I acknowledge the existence of working code :)
>>
>> If we had understood the problem back then as well as we do now, you
>> might have written different code.
>
> s/the problem/the maintainer's preferences/ ;-)
>
> The implicit requirement I wasn't aware of is "there should be only one
> way to express the same thing on each external interface". Aliases go
> against that by their very nature - they provide a second way.
>
>> Aliases are for *adding* "alternate" to "standard" by nature.
>>
>> To get "alternate" *instead* of "standard", something else might be a
>> better fit. "T2 inherits from T1 with the following renames" comes to
>> mind.
>
> Right.
>
>> >> >> > I guess I can go into more details once we have answered these
>> >> >> > fundamental questions.
>> >> >> >
>> >> >> > (We could and should have talked about them and about whether you want
>> >> >> > to have aliases at all a year ago, before going into detailed code
>> >> >> > review and making error messages perfect in code that we might throw
>> >> >> > away anyway...)
>> >> >>
>> >> >> I doubt you could possibly be more right than that!
>> >> >
>> >> > So what questions need to be answered before we can make a decision?
>> >> >
>> >> > You talked a lot about how you like QMP to be different from the command
>> >> > line. So is restricting aliases to the command line enough to make them
>> >> > acceptable? Or do we need larger changes? Should I just throw them away
>> >> > and write translation code for -chardev manually?
>> >>
>> >> Fair question, but I'm out of time for right now. Let's continue
>> >> tomorrow.
>> >
>> > :-)
>>
>> What's the smallest set of aliases sufficient to make your -chardev
>> QAPIfication work?
>
> How do you define "make the QAPIfication work"?
>
> The -chardev conversion adds the following aliases to the schema:
>
> * SocketAddressLegacy: Flatten a simple union (wildcard alias +
> non-local alias)
>
> { 'source': ['data'] },
> { 'name': 'fd', 'source': ['data', 'str'] }
>
> * ChardevFile: Rename local member
>
> { 'name': 'path', 'source': ['out'] }
>
> * ChardevHostdev: Rename local member
>
> { 'name': 'path', 'source': ['device'] }
>
> * ChardevSocket: Flatten embedded struct (wildcard alias)
>
> { 'source': ['addr'] }
>
> * ChardevUdp: Flatten two embedded structs with renaming (wildcard
> alias + non-local alias)
>
> { 'source': ['remote'] },
> { 'name': 'localaddr', 'source': ['local', 'data', 'host'] },
> { 'name': 'localport', 'source': ['local', 'data', 'port'] }
>
> * ChardevSpiceChannel: Rename local member
>
> { 'name': 'name', 'source': ['type'] }
>
> * ChardevSpicePort: Rename local member
>
> { 'name': 'name', 'source': ['fqdn'] }
>
> * ChardevBackend: Flatten a simple union (wildcard alias)
>
> { 'source': ['data'] }
>
> * ChardevOptions: Flatten embedded struct (wildcard alias)
>
> { 'source': ['backend'] }
Aside: most of the renames are due to "reuse" of existing QemuOpts
parameters. I'm sure it has saved "lots" of typing.
> The deeper they are nested in the type hierarchy, especially when unions
> with different variants come into play, the nastier they are to replace
> with C code. The C code stays the simplest if all of the aliases are
> there, and it gets uglier the more of them you leave out.
>
> I don't know your idea of "sufficient", so I'll leave mapping that to a
> scale of sufficiency to you.
Alright let me see what we got.
This is the tree structure with branches not mentioned in aliases
omitted:
ChardevOptions
backend: ChardevBackend
data: ChardevFile
data: ChardevHostdev
data: ChardevSocket
addr: SocketAddressLegacy
data: String
str: str
data: ...
data: ChardevUdp
remote: SocketAddressLegacy
data: String
str: str
data: ...
local: SocketAddressLegacy
data: String
str: str
data: ...
data: ChardevSpiceChannel
data: ChardevSpicePort
data: ...
This is how we map -chardev's argument to the tree structure:
* Always, in qemu_chr_new_from_opts(), qemu_chr_parse_opts(),
qemu_chr_parse_common():
log
- id=id id
- [backend=]T backend.type
- logfile backend.data.logfile
- logappend backend.data.logappend
* When T is file, in qemu_chr_parse_file_out():
- path backend.data.out
- append backend.data.append
Note: there is no way to set backend.data.in.
* When T is serial, parallel, or pipe, in qemu_chr_parse_serial(),
qemu_chr_parse_parallel(), qemu_chr_parse_pipe():
- path backend.data.device
* When T is socket, in qemu_chr_parse_socket()
- delay and nodelay backend.data.nodelay
- server backend.data.server
- telnet backend.data.telnet
- tn3270 backend.data.tn3270
- websocket backend.data.websocket
- wait backend.data.wait
- reconnect backend.data.reconnect
- tls-creds backend.data.tls-creds
- tls-authz backend.data.tls-authz
- host, path, fd backend.data.addr.type
Note: there is no way to set backend.data.addr.type to vsock.
Additionally, when path is present:
- path backend.data.addr.data.path
- abstract backend.data.addr.data.abstract
- tight backend.data.addr.data.tight
Additionally, when host is present:
- host backend.data.addr.data.host
- port backend.data.addr.data.port
- to backend.data.addr.data.to
- ipv4 backend.data.addr.data.ipv4
- ipv6 backend.data.addr.data.ipv6
Note: there is no way to set backend.data.addr.data.numeric,
.keep-alive, .mptcp.
Additionally, when fd is present:
- fd backend.data.addr.data.str
* When T is udp, in qemu_chr_parse_udp():
- host backend.data.remote.data.host
- port backend.data.remote.data.port
- ipv4 backend.data.remote.data.ipv4
- ipv6 backend.data.remote.data.ipv6
- localaddr backend.data.local.data.host
- localport backend.data.local.data.port
Note: there is no way to set backend.data.remote.type and
backend.data.local.type; both can only be inet. There is no way to
set backend.data.{remote,local}.data.to, .numeric, .keep-alive,
.mptcp. There is no way to set backend.data.local.data.ipv4, .ipv6.
* I'm omitting the remaining values of T.
* Parameters that exist for any value of T other than the one given are
silently ignored. Example: -chardev null,id=woot,ipv4=on.
Do you preserve this wart for compatibility's sake?
Observations:
* Your replacement of this mapping code makes the dotted keys
corresponding to the schema available in addition to the traditional
key. Example: backend.data.addr.data.host in addition to host.
* This makes some parameters available that weren't before. Example:
backend.data.addr.data.numeric and numeric. Also
backend.data.local.data.host but not host, because that's already
backend.data.remote.data.host.
* It also creates "ghost" aliases, i.e. keys that don't exist in either
of the two interfaces before. These are artifacts of the alias chain
from traditional key to schema member. Example:
backend.data.addr.host, backend.addr.data.host, backend.addr.host,
data.addr.host, addr.data.host, and addr.host. I think. No, I missed
backend.host. Did I get them all? No idea :)
All this should be spelled out in commit message(s). I didn't peek
ahead to check them.
A different way to skin this cat would be putting the aliases at the
top, i.e. ChardevOptions. I'm aware of your arguments against this.
Let's explore it anyway.
backend backend.type
path backend.data.out
path backend.data.device
* backend.data.*
fd backend.data.addr.data.str
* backend.data.addr.data.*
* backend.data.remote.data.*
localaddr backend.data.local.data.host
localport backend.data.local.data.port
Observations / questions:
* Look ma, no "ghosts"!
* We need "path" twice. They resolve to different branches of the
union. Hmm.
Aliases pointing into union branches give me a queasy feeling. What
if we define an alias just for one branch, but then have it resolve in
an unwanted way in another branch? "Ghost" aliases, I guess.
Perhaps we should attach the aliases to the union branch instead. The
"put them at the top" idea falls apart then.
The issue exists just as much with "chained" use of aliases. But
there, it's just a few more ghosts joining ghost congress.
* How to provide full access to backend.data.local.data.*? Assuming
that's desired.
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-04 14:07 ` Kevin Wolf
2021-10-05 13:49 ` Markus Armbruster
@ 2021-10-11 7:44 ` Markus Armbruster
2021-10-12 14:36 ` Kevin Wolf
1 sibling, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-10-11 7:44 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
[...]
> What I had in mind was using the schema to generate the necessary code,
> using the generic keyval parser everywhere, and just providing a hook
> where the QDict could be modified after the keyval parser and before the
> visitor. Most command line options would not have any explicit code for
> parsing input, only the declarative schema and the final option handler
> getting a QAPI type (which could actually be the corresponding QMP
> command handler in the common case).
A walk to the bakery made me see a problem with transforming the qdict
between keyval parsing and the visitor: error reporting. On closer
investigation, the problem exists even with just aliases.
All experiments performed with your complete QAPIfication at
https://repo.or.cz/qemu/kevin.git qapi-alias-chardev-v4.
Example: flattening leads to suboptimal error
$ qemu-system-x86_64 -chardev udp,id=chr0,port=12345,ipv4=om
qemu-system-x86_64: -chardev udp,id=chr0,port=12345,ipv4=om: Parameter 'backend.data.remote.data.ipv4' expects 'on' or 'off'
We're using "alternate" notation, but the error message barks back in
"standard" notation. It comes from the visitor. While less than
pleasant, it's still understandable, because the "standard" key ends
with the "alternate" key.
Example: renaming leads to confusing error
So far, we rename only members of type 'str', where the visitor can't
fail. Just for illustrating the problem, I'm adding one where it can:
diff --git a/qapi/char.json b/qapi/char.json
index 0e39840d4f..b436d83cbe 100644
--- a/qapi/char.json
+++ b/qapi/char.json
@@ -398,7 +398,8 @@
##
{ 'struct': 'ChardevRingbuf',
'data': { '*size': 'int' },
- 'base': 'ChardevCommon' }
+ 'base': 'ChardevCommon',
+ 'aliases': [ { 'name': 'capacity', 'source': [ 'size' ] } ] }
##
# @ChardevQemuVDAgent:
With this patch:
$ qemu-system-x86_64 -chardev ringbuf,id=chr0,capacity=lots
qemu-system-x86_64: -chardev ringbuf,id=chr0,capacity=lots: Parameter 'backend.data.size' expects integer
The error message fails to connect to the offending key=value.
Example: manual transformation leads to confusion #1
Starting point:
$ qemu-system-x86_64 -chardev udp,id=chr0,port=12345,localaddr=localhost
Works. host defaults to localhost, localport defaults to 0, same as in
git master.
Now "forget" to specify @port:
$ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost
qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost: Parameter 'backend.data.remote.data.port' is missing
Again, the visitor's error message uses "standard" notation.
We obediently do what the error message tells us to do:
$ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345
qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345: Parameters 'backend.*' used inconsistently
Mission accomplished: confusion :)
Example: manual transformation leads to confusion #2
Confusion is possible even without tricking the user into mixing
"standard" and "alternate" explicitly:
$ qemu-system-x86_64 -chardev udp,id=chr0,backend.data.remote.type=udp
qemu-system-x86_64: -chardev udp,id=chr0,backend.data.remote.type=udp: Parameters 'backend.*' used inconsistently
Here, the user tried to stick to "standard", but forgot to specify a
required member. The transformation machinery then "helpfully"
transformed nothing into something, which made the visitor throw up.
Clear error reporting is a critical part of a human-friendly interface.
We have two separate problems with it:
1. The visitor reports errors as if aliases didn't exist
Fixing this is "merely" a matter of tracing back alias applications.
More complexity...
2. The visitor reports errors as if manual transformation didn't exist
Manual transformation can distort the users input beyond recognition.
The visitor reports errors for the transformed input.
To fix this one, we'd have to augment the parse tree so it points
back at the actual user input, and then make the manual
transformations preserve that. Uff!
I'm afraid I need another round of thinking on how to best drag legacy
syntax along when we QAPIfy.
[...]
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-08 10:17 ` Markus Armbruster
@ 2021-10-12 14:00 ` Kevin Wolf
0 siblings, 0 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-10-12 14:00 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 08.10.2021 um 12:17 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
> > Am 07.10.2021 um 13:06 hat Markus Armbruster geschrieben:
> >> What's the smallest set of aliases sufficient to make your -chardev
> >> QAPIfication work?
> >
> > How do you define "make the QAPIfication work"?
> >
> > The -chardev conversion adds the following aliases to the schema:
> >
> > * SocketAddressLegacy: Flatten a simple union (wildcard alias +
> > non-local alias)
> >
> > { 'source': ['data'] },
> > { 'name': 'fd', 'source': ['data', 'str'] }
> >
> > * ChardevFile: Rename local member
> >
> > { 'name': 'path', 'source': ['out'] }
> >
> > * ChardevHostdev: Rename local member
> >
> > { 'name': 'path', 'source': ['device'] }
> >
> > * ChardevSocket: Flatten embedded struct (wildcard alias)
> >
> > { 'source': ['addr'] }
> >
> > * ChardevUdp: Flatten two embedded structs with renaming (wildcard
> > alias + non-local alias)
> >
> > { 'source': ['remote'] },
> > { 'name': 'localaddr', 'source': ['local', 'data', 'host'] },
> > { 'name': 'localport', 'source': ['local', 'data', 'port'] }
> >
> > * ChardevSpiceChannel: Rename local member
> >
> > { 'name': 'name', 'source': ['type'] }
> >
> > * ChardevSpicePort: Rename local member
> >
> > { 'name': 'name', 'source': ['fqdn'] }
> >
> > * ChardevBackend: Flatten a simple union (wildcard alias)
> >
> > { 'source': ['data'] }
> >
> > * ChardevOptions: Flatten embedded struct (wildcard alias)
> >
> > { 'source': ['backend'] }
>
> Aside: most of the renames are due to "reuse" of existing QemuOpts
> parameters. I'm sure it has saved "lots" of typing.
>
> > The deeper they are nested in the type hierarchy, especially when unions
> > with different variants come into play, the nastier they are to replace
> > with C code. The C code stays the simplest if all of the aliases are
> > there, and it gets uglier the more of them you leave out.
> >
> > I don't know your idea of "sufficient", so I'll leave mapping that to a
> > scale of sufficiency to you.
>
> Alright let me see what we got.
>
> This is the tree structure with branches not mentioned in aliases
> omitted:
>
> ChardevOptions
> backend: ChardevBackend
> data: ChardevFile
> data: ChardevHostdev
> data: ChardevSocket
> addr: SocketAddressLegacy
> data: String
> str: str
> data: ...
> data: ChardevUdp
> remote: SocketAddressLegacy
> data: String
> str: str
> data: ...
> local: SocketAddressLegacy
> data: String
> str: str
> data: ...
> data: ChardevSpiceChannel
> data: ChardevSpicePort
> data: ...
>
> This is how we map -chardev's argument to the tree structure:
>
> * Always, in qemu_chr_new_from_opts(), qemu_chr_parse_opts(),
> qemu_chr_parse_common():
> log
> - id=id id
> - [backend=]T backend.type
> - logfile backend.data.logfile
> - logappend backend.data.logappend
>
> * When T is file, in qemu_chr_parse_file_out():
>
> - path backend.data.out
> - append backend.data.append
>
> Note: there is no way to set backend.data.in.
>
> * When T is serial, parallel, or pipe, in qemu_chr_parse_serial(),
> qemu_chr_parse_parallel(), qemu_chr_parse_pipe():
>
> - path backend.data.device
>
> * When T is socket, in qemu_chr_parse_socket()
>
> - delay and nodelay backend.data.nodelay
> - server backend.data.server
> - telnet backend.data.telnet
> - tn3270 backend.data.tn3270
> - websocket backend.data.websocket
> - wait backend.data.wait
> - reconnect backend.data.reconnect
> - tls-creds backend.data.tls-creds
> - tls-authz backend.data.tls-authz
> - host, path, fd backend.data.addr.type
>
> Note: there is no way to set backend.data.addr.type to vsock.
>
> Additionally, when path is present:
>
> - path backend.data.addr.data.path
> - abstract backend.data.addr.data.abstract
> - tight backend.data.addr.data.tight
>
> Additionally, when host is present:
>
> - host backend.data.addr.data.host
> - port backend.data.addr.data.port
> - to backend.data.addr.data.to
> - ipv4 backend.data.addr.data.ipv4
> - ipv6 backend.data.addr.data.ipv6
>
> Note: there is no way to set backend.data.addr.data.numeric,
> .keep-alive, .mptcp.
>
> Additionally, when fd is present:
>
> - fd backend.data.addr.data.str
>
> * When T is udp, in qemu_chr_parse_udp():
>
> - host backend.data.remote.data.host
> - port backend.data.remote.data.port
> - ipv4 backend.data.remote.data.ipv4
> - ipv6 backend.data.remote.data.ipv6
> - localaddr backend.data.local.data.host
> - localport backend.data.local.data.port
>
> Note: there is no way to set backend.data.remote.type and
> backend.data.local.type; both can only be inet. There is no way to
> set backend.data.{remote,local}.data.to, .numeric, .keep-alive,
> .mptcp. There is no way to set backend.data.local.data.ipv4, .ipv6.
>
> * I'm omitting the remaining values of T.
>
> * Parameters that exist for any value of T other than the one given are
> silently ignored. Example: -chardev null,id=woot,ipv4=on.
>
> Do you preserve this wart for compatibility's sake?
No, unless someone can show me some important real life user that would
be affected by it, I think it should be considered a bug and just be
fixed.
If contrary to all expectations users do come and shout at us, we can
consider adding back those silently ignored options that are actually
in use.
> Observations:
>
> * Your replacement of this mapping code makes the dotted keys
> corresponding to the schema available in addition to the traditional
> key. Example: backend.data.addr.data.host in addition to host.
>
> * This makes some parameters available that weren't before. Example:
> backend.data.addr.data.numeric and numeric. Also
> backend.data.local.data.host but not host, because that's already
> backend.data.remote.data.host.
>
> * It also creates "ghost" aliases, i.e. keys that don't exist in either
> of the two interfaces before. These are artifacts of the alias chain
> from traditional key to schema member. Example:
> backend.data.addr.host, backend.addr.data.host, backend.addr.host,
> data.addr.host, addr.data.host, and addr.host. I think. No, I missed
> backend.host. Did I get them all? No idea :)
At least it feels like a quite consistent way of having "ghost"
aliases...
> All this should be spelled out in commit message(s). I didn't peek
> ahead to check them.
>
> A different way to skin this cat would be putting the aliases at the
> top, i.e. ChardevOptions. I'm aware of your arguments against this.
> Let's explore it anyway.
>
> backend backend.type
> path backend.data.out
> path backend.data.device
> * backend.data.*
> fd backend.data.addr.data.str
> * backend.data.addr.data.*
> * backend.data.remote.data.*
> localaddr backend.data.local.data.host
> localport backend.data.local.data.port
>
> Observations / questions:
>
> * Look ma, no "ghosts"!
I assume '*' doesn't mean wildcard aliases like in the current series
(because this would add back some ghosts for nested objects), but an
individual listing of all aliased scalar members?
> * We need "path" twice. They resolve to different branches of the
> union. Hmm.
>
> Aliases pointing into union branches give me a queasy feeling. What
> if we define an alias just for one branch, but then have it resolve in
> an unwanted way in another branch? "Ghost" aliases, I guess.
...compared to this one where they appear only sporadically on name
collisions.
> Perhaps we should attach the aliases to the union branch instead. The
> "put them at the top" idea falls apart then.
>
> The issue exists just as much with "chained" use of aliases. But
> there, it's just a few more ghosts joining ghost congress.
>
> * How to provide full access to backend.data.local.data.*? Assuming
> that's desired.
I think having access to all options is desirable.
My simple approach gives you "local.*". It's considered a ghost by your
listing, but it's actually not because the individual fields are
inaccessible on the top level (they are shadowed by "remote.*").
All of this goes on top of the problems we already knew, like that this
list of aliases is hard to maintain because you don't necessarily think
of updating ChardevOptions when you add something to SocketAddress.
So I still feel like having aliases only on the top level is a dead end.
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-11 7:44 ` Markus Armbruster
@ 2021-10-12 14:36 ` Kevin Wolf
2021-10-13 9:41 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Kevin Wolf @ 2021-10-12 14:36 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 11.10.2021 um 09:44 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
>
> [...]
>
> > What I had in mind was using the schema to generate the necessary code,
> > using the generic keyval parser everywhere, and just providing a hook
> > where the QDict could be modified after the keyval parser and before the
> > visitor. Most command line options would not have any explicit code for
> > parsing input, only the declarative schema and the final option handler
> > getting a QAPI type (which could actually be the corresponding QMP
> > command handler in the common case).
>
> A walk to the bakery made me see a problem with transforming the qdict
> between keyval parsing and the visitor: error reporting. On closer
> investigation, the problem exists even with just aliases.
I already commented on part of this on IRC, but let me reply here as
well.
On general remark is that I would make the same distinction between
aliases for compatibility and usability that I mentioned elsewhere in
this thread.
In the case of compatibility, it's already a concession that we still
accept the option - suboptimal error messages for incorrect command
lines aren't a major concern for me there. Users are supposed to move to
the new syntax anyway.
On the other hand, aliases we employ for usability are cases where we
don't expect humans to move to something else. I think this is mostly
for flattening structures. Here error messages should be good.
> All experiments performed with your complete QAPIfication at
> https://repo.or.cz/qemu/kevin.git qapi-alias-chardev-v4.
>
>
> Example: flattening leads to suboptimal error
>
> $ qemu-system-x86_64 -chardev udp,id=chr0,port=12345,ipv4=om
> qemu-system-x86_64: -chardev udp,id=chr0,port=12345,ipv4=om: Parameter 'backend.data.remote.data.ipv4' expects 'on' or 'off'
>
> We're using "alternate" notation, but the error message barks back in
> "standard" notation. It comes from the visitor. While less than
> pleasant, it's still understandable, because the "standard" key ends
> with the "alternate" key.
This is not a fundamental problem with aliases. The right name for the
option is unambiguous and known to the visitor: It's the name that the
user specified.
With manual QDict modification it becomes a more fundamental problem
because the visitor can't know the original name any more.
>
> Example: renaming leads to confusing error
>
> So far, we rename only members of type 'str', where the visitor can't
> fail. Just for illustrating the problem, I'm adding one where it can:
>
> diff --git a/qapi/char.json b/qapi/char.json
> index 0e39840d4f..b436d83cbe 100644
> --- a/qapi/char.json
> +++ b/qapi/char.json
> @@ -398,7 +398,8 @@
> ##
> { 'struct': 'ChardevRingbuf',
> 'data': { '*size': 'int' },
> - 'base': 'ChardevCommon' }
> + 'base': 'ChardevCommon',
> + 'aliases': [ { 'name': 'capacity', 'source': [ 'size' ] } ] }
>
> ##
> # @ChardevQemuVDAgent:
>
> With this patch:
>
> $ qemu-system-x86_64 -chardev ringbuf,id=chr0,capacity=lots
> qemu-system-x86_64: -chardev ringbuf,id=chr0,capacity=lots: Parameter 'backend.data.size' expects integer
>
> The error message fails to connect to the offending key=value.
Same problem as above. The error message should use the key that the
user actually specified.
>
> Example: manual transformation leads to confusion #1
>
> Starting point:
>
> $ qemu-system-x86_64 -chardev udp,id=chr0,port=12345,localaddr=localhost
>
> Works. host defaults to localhost, localport defaults to 0, same as in
> git master.
>
> Now "forget" to specify @port:
>
> $ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost
> qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost: Parameter 'backend.data.remote.data.port' is missing
>
> Again, the visitor's error message uses "standard" notation.
The output isn't wrong, it's just more verbose than necessary.
Getting this one shortened is a bit harder because the right name is
ambiguous, the user didn't specify anything we can just print back.
Possibly in CLI context, making use of any wildcard aliases would be a
reasonable strategy. This would reduce this one to just 'port'.
> We obediently do what the error message tells us to do:
>
> $ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345
> qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345: Parameters 'backend.*' used inconsistently
>
> Mission accomplished: confusion :)
This one already fails before aliases do their work. The reason is that
the default key for -chardev is 'backend', and QMP and CLI use the same
name 'backend' for two completely different things.
We could rename the default key into 'x-backend' and make it behave the
same as 'backend', then the keyval parser would only fail when you
explicitly wrote '-chardev backend=udp,...' and the problem is more
obvious.
By the way, we have a very similar issue with '-drive file=...', if we
ever want to parse that one with the keyval parser. It can be both a
string for the filename or an object for the configuration of the 'file'
child that many block drivers have.
> Example: manual transformation leads to confusion #2
>
> Confusion is possible even without tricking the user into mixing
> "standard" and "alternate" explicitly:
>
> $ qemu-system-x86_64 -chardev udp,id=chr0,backend.data.remote.type=udp
> qemu-system-x86_64: -chardev udp,id=chr0,backend.data.remote.type=udp: Parameters 'backend.*' used inconsistently
>
> Here, the user tried to stick to "standard", but forgot to specify a
> required member. The transformation machinery then "helpfully"
> transformed nothing into something, which made the visitor throw up.
Not the visitor, but the keyval parser. Same problem as above.
> Clear error reporting is a critical part of a human-friendly interface.
> We have two separate problems with it:
>
> 1. The visitor reports errors as if aliases didn't exist
>
> Fixing this is "merely" a matter of tracing back alias applications.
> More complexity...
>
> 2. The visitor reports errors as if manual transformation didn't exist
>
> Manual transformation can distort the users input beyond recognition.
> The visitor reports errors for the transformed input.
>
> To fix this one, we'd have to augment the parse tree so it points
> back at the actual user input, and then make the manual
> transformations preserve that. Uff!
Manual transformations are hard to write in a way that they give perfect
results. As long as they are only used for legacy syntax and we expect
users to move to a new way to spell things, this might be acceptable for
a transition period until we remove the old syntax.
In other cases, the easiest way is probably to duplicate even more of
the schema and manually make sure that the visitor will accept the input
before we transform it.
The best way to avoid this is providing tools in QAPI that make manual
transformations unnecessary.
> I'm afraid I need another round of thinking on how to best drag legacy
> syntax along when we QAPIfy.
Let me know when you've come to any conclusions (or more things to
discuss).
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-12 14:36 ` Kevin Wolf
@ 2021-10-13 9:41 ` Markus Armbruster
2021-10-13 11:10 ` Markus Armbruster
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-10-13 9:41 UTC (permalink / raw)
To: Kevin Wolf; +Cc: jsnow, qemu-devel
Kevin Wolf <kwolf@redhat.com> writes:
> Am 11.10.2021 um 09:44 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>>
>> [...]
>>
>> > What I had in mind was using the schema to generate the necessary code,
>> > using the generic keyval parser everywhere, and just providing a hook
>> > where the QDict could be modified after the keyval parser and before the
>> > visitor. Most command line options would not have any explicit code for
>> > parsing input, only the declarative schema and the final option handler
>> > getting a QAPI type (which could actually be the corresponding QMP
>> > command handler in the common case).
>>
>> A walk to the bakery made me see a problem with transforming the qdict
>> between keyval parsing and the visitor: error reporting. On closer
>> investigation, the problem exists even with just aliases.
>
> I already commented on part of this on IRC, but let me reply here as
> well.
>
> On general remark is that I would make the same distinction between
> aliases for compatibility and usability that I mentioned elsewhere in
> this thread.
>
> In the case of compatibility, it's already a concession that we still
> accept the option - suboptimal error messages for incorrect command
> lines aren't a major concern for me there. Users are supposed to move to
> the new syntax anyway.
Well, these aren't "incorrect", merely deprecated. Bad UX is still bad
there, but at least it'll "fix" itself when the deprecated part goes
away.
> On the other hand, aliases we employ for usability are cases where we
> don't expect humans to move to something else. I think this is mostly
> for flattening structures. Here error messages should be good.
>
>> All experiments performed with your complete QAPIfication at
>> https://repo.or.cz/qemu/kevin.git qapi-alias-chardev-v4.
>>
>>
>> Example: flattening leads to suboptimal error
>>
>> $ qemu-system-x86_64 -chardev udp,id=chr0,port=12345,ipv4=om
>> qemu-system-x86_64: -chardev udp,id=chr0,port=12345,ipv4=om: Parameter 'backend.data.remote.data.ipv4' expects 'on' or 'off'
>>
>> We're using "alternate" notation, but the error message barks back in
>> "standard" notation. It comes from the visitor. While less than
>> pleasant, it's still understandable, because the "standard" key ends
>> with the "alternate" key.
>
> This is not a fundamental problem with aliases. The right name for the
> option is unambiguous and known to the visitor: It's the name that the
> user specified.
This is "1. The visitor reports errors as if aliases didn't exist"
below.
> With manual QDict modification it becomes a more fundamental problem
> because the visitor can't know the original name any more.
This is "2. The visitor reports errors as if manual transformation
didn't exist" below.
I agree 1 is easier to fix than 2.
>> Example: renaming leads to confusing error
>>
>> So far, we rename only members of type 'str', where the visitor can't
>> fail. Just for illustrating the problem, I'm adding one where it can:
>>
>> diff --git a/qapi/char.json b/qapi/char.json
>> index 0e39840d4f..b436d83cbe 100644
>> --- a/qapi/char.json
>> +++ b/qapi/char.json
>> @@ -398,7 +398,8 @@
>> ##
>> { 'struct': 'ChardevRingbuf',
>> 'data': { '*size': 'int' },
>> - 'base': 'ChardevCommon' }
>> + 'base': 'ChardevCommon',
>> + 'aliases': [ { 'name': 'capacity', 'source': [ 'size' ] } ] }
>>
>> ##
>> # @ChardevQemuVDAgent:
>>
>> With this patch:
>>
>> $ qemu-system-x86_64 -chardev ringbuf,id=chr0,capacity=lots
>> qemu-system-x86_64: -chardev ringbuf,id=chr0,capacity=lots: Parameter 'backend.data.size' expects integer
>>
>> The error message fails to connect to the offending key=value.
>
> Same problem as above. The error message should use the key that the
> user actually specified.
Yes.
>> Example: manual transformation leads to confusion #1
>>
>> Starting point:
>>
>> $ qemu-system-x86_64 -chardev udp,id=chr0,port=12345,localaddr=localhost
>>
>> Works. host defaults to localhost, localport defaults to 0, same as in
>> git master.
>>
>> Now "forget" to specify @port:
>>
>> $ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost
>> qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost: Parameter 'backend.data.remote.data.port' is missing
>>
>> Again, the visitor's error message uses "standard" notation.
>
> The output isn't wrong, it's just more verbose than necessary.
>
> Getting this one shortened is a bit harder because the right name is
> ambiguous, the user didn't specify anything we can just print back.
>
> Possibly in CLI context, making use of any wildcard aliases would be a
> reasonable strategy. This would reduce this one to just 'port'.
Which of the possible keys we use in the error message boils down to
picking a preferred key.
>> We obediently do what the error message tells us to do:
>>
>> $ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345
>> qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345: Parameters 'backend.*' used inconsistently
>>
>> Mission accomplished: confusion :)
>
> This one already fails before aliases do their work. The reason is that
> the default key for -chardev is 'backend', and QMP and CLI use the same
> name 'backend' for two completely different things.
Right. I was confused (and the mission was truly accomplished).
> We could rename the default key into 'x-backend' and make it behave the
> same as 'backend', then the keyval parser would only fail when you
> explicitly wrote '-chardev backend=udp,...' and the problem is more
> obvious.
Technically a compatibility break, but we can hope that the longhand
form we change isn't used.
> By the way, we have a very similar issue with '-drive file=...', if we
> ever want to parse that one with the keyval parser. It can be both a
> string for the filename or an object for the configuration of the 'file'
> child that many block drivers have.
Should I be running for the hills?
>> Example: manual transformation leads to confusion #2
>>
>> Confusion is possible even without tricking the user into mixing
>> "standard" and "alternate" explicitly:
>>
>> $ qemu-system-x86_64 -chardev udp,id=chr0,backend.data.remote.type=udp
>> qemu-system-x86_64: -chardev udp,id=chr0,backend.data.remote.type=udp: Parameters 'backend.*' used inconsistently
>>
>> Here, the user tried to stick to "standard", but forgot to specify a
>> required member. The transformation machinery then "helpfully"
>> transformed nothing into something, which made the visitor throw up.
>
> Not the visitor, but the keyval parser. Same problem as above.
>
>> Clear error reporting is a critical part of a human-friendly interface.
>> We have two separate problems with it:
>>
>> 1. The visitor reports errors as if aliases didn't exist
>>
>> Fixing this is "merely" a matter of tracing back alias applications.
>> More complexity...
>>
>> 2. The visitor reports errors as if manual transformation didn't exist
>>
>> Manual transformation can distort the users input beyond recognition.
>> The visitor reports errors for the transformed input.
>>
>> To fix this one, we'd have to augment the parse tree so it points
>> back at the actual user input, and then make the manual
>> transformations preserve that. Uff!
>
> Manual transformations are hard to write in a way that they give perfect
> results. As long as they are only used for legacy syntax and we expect
> users to move to a new way to spell things, this might be acceptable for
> a transition period until we remove the old syntax.
Valid point.
> In other cases, the easiest way is probably to duplicate even more of
> the schema and manually make sure that the visitor will accept the input
> before we transform it.
>
> The best way to avoid this is providing tools in QAPI that make manual
> transformations unnecessary.
Reducing the amount of handwritten code is good, as long as the price is
reasonable. Complex code fetches a higher price.
I think there are a couple of ways to skin this cat.
0. Common to all ways: specify "standard" in the QAPI schema. This
specifies both the "standard" wire format, its parse tree
(represented as QObject), and the "standard" C interface (C types,
basically).
Generic parsers parse into the parse tree. The appropriate input
visitor validates and converts to C types.
1. Parse "alternate" by any means (QemuOpts, keyval, ad hoc), validate,
then transform for the "standard" C interface. Parsing and
validating can fail, the transformation can't.
Drawbacks:
* We duplicate validation, which is a standing invitation for bugs.
* Longwinded, handwritten transformation code. Less vulnerable to
bugs than validation code, I believe.
2. Parse "alternate" by any means (QemuOpts, keyval, ad hoc), transform
to "standard" parse tree.
Drawbacks:
* Bad error messages from input visitor.
* The handwritten transformation code is more longwinded than with 1.
3. Specify "alternate" in the QAPI schema. Map "alternate" C interface
to the "standard" one.
Drawbacks:
* QAPI schema code duplication
* Longwinded, handwritten mapping code.
This is what we did with SocketAddressLegacy.
4. Extend the QAPI schema language to let us specify non-standard wire
format(s). Use that to get the "alternate" wire format we need.
Drawbacks:
* QAPI schema language and generator become more complex.
* Input visitor(s) become more complex.
* Unless we accept "alternate" everywhere, we need a way to limit it
to where it's actually needed. More complexity.
The concrete proposal on the table is aliases. They are not a
complete solution. To solve the whole problem, we combine them with
2. We accept 4's drawbacks for a (sizable) reduction of 2's.
Additionally, we accept "ghost" aliases.
5. Extend the QAPI schema language to let us specify "alternate" wire
formats for "standard" types
This is like 3, except the schema code is mostly referenced instead
duplicated, and the mapping code is generated. Something like
"derive type T' from T with these members moved / renamed".
Drawbacks
* QAPI schema language and generator become more complex.
* Unlike 4, we don't have working code.
Like 4, this will likely solve only part of the problem.
Which one is the least bad?
If this was just about dragging deprecated interfaces to the end of
their grace period, I'd say "whatever is the least work", and that's
almost certainly whatever we have now, possibly hacked up to use the
appropriate internal interface.
Unfortunately, it is also about providing a friendlier interface to
humans "forever".
With 4 or 5, we invest into infrastructure once, then maintain it
forever, to make solving the problem easier and the result easier to
maintain.
Whether the investment will pay off depends on how big and bad the
problem is. Hard to say. One of the reasons we're looking at -chardev
now is that we believe it's the worst of the bunch. But how big is the
bunch, and how bad are the others in there?
>> I'm afraid I need another round of thinking on how to best drag legacy
>> syntax along when we QAPIfy.
>
> Let me know when you've come to any conclusions (or more things to
> discuss).
Is 3 too awful to contemplate?
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-13 9:41 ` Markus Armbruster
@ 2021-10-13 11:10 ` Markus Armbruster
2021-10-14 9:35 ` Kevin Wolf
0 siblings, 1 reply; 43+ messages in thread
From: Markus Armbruster @ 2021-10-13 11:10 UTC (permalink / raw)
To: Markus Armbruster; +Cc: Kevin Wolf, jsnow, qemu-devel
Markus Armbruster <armbru@redhat.com> writes:
> Kevin Wolf <kwolf@redhat.com> writes:
>
>> Am 11.10.2021 um 09:44 hat Markus Armbruster geschrieben:
>>> Kevin Wolf <kwolf@redhat.com> writes:
>>>
>>> [...]
>>>
>>> > What I had in mind was using the schema to generate the necessary code,
>>> > using the generic keyval parser everywhere, and just providing a hook
>>> > where the QDict could be modified after the keyval parser and before the
>>> > visitor. Most command line options would not have any explicit code for
>>> > parsing input, only the declarative schema and the final option handler
>>> > getting a QAPI type (which could actually be the corresponding QMP
>>> > command handler in the common case).
>>>
>>> A walk to the bakery made me see a problem with transforming the qdict
>>> between keyval parsing and the visitor: error reporting. On closer
>>> investigation, the problem exists even with just aliases.
>>
>> I already commented on part of this on IRC, but let me reply here as
>> well.
>>
>> On general remark is that I would make the same distinction between
>> aliases for compatibility and usability that I mentioned elsewhere in
>> this thread.
>>
>> In the case of compatibility, it's already a concession that we still
>> accept the option - suboptimal error messages for incorrect command
>> lines aren't a major concern for me there. Users are supposed to move to
>> the new syntax anyway.
>
> Well, these aren't "incorrect", merely deprecated. Bad UX is still bad
> there, but at least it'll "fix" itself when the deprecated part goes
> away.
>
>> On the other hand, aliases we employ for usability are cases where we
>> don't expect humans to move to something else. I think this is mostly
>> for flattening structures. Here error messages should be good.
>>
>>> All experiments performed with your complete QAPIfication at
>>> https://repo.or.cz/qemu/kevin.git qapi-alias-chardev-v4.
>>>
>>>
>>> Example: flattening leads to suboptimal error
>>>
>>> $ qemu-system-x86_64 -chardev udp,id=chr0,port=12345,ipv4=om
>>> qemu-system-x86_64: -chardev udp,id=chr0,port=12345,ipv4=om: Parameter 'backend.data.remote.data.ipv4' expects 'on' or 'off'
>>>
>>> We're using "alternate" notation, but the error message barks back in
>>> "standard" notation. It comes from the visitor. While less than
>>> pleasant, it's still understandable, because the "standard" key ends
>>> with the "alternate" key.
>>
>> This is not a fundamental problem with aliases. The right name for the
>> option is unambiguous and known to the visitor: It's the name that the
>> user specified.
>
> This is "1. The visitor reports errors as if aliases didn't exist"
> below.
>
>> With manual QDict modification it becomes a more fundamental problem
>> because the visitor can't know the original name any more.
>
> This is "2. The visitor reports errors as if manual transformation
> didn't exist" below.
>
> I agree 1 is easier to fix than 2.
>
>>> Example: renaming leads to confusing error
>>>
>>> So far, we rename only members of type 'str', where the visitor can't
>>> fail. Just for illustrating the problem, I'm adding one where it can:
>>>
>>> diff --git a/qapi/char.json b/qapi/char.json
>>> index 0e39840d4f..b436d83cbe 100644
>>> --- a/qapi/char.json
>>> +++ b/qapi/char.json
>>> @@ -398,7 +398,8 @@
>>> ##
>>> { 'struct': 'ChardevRingbuf',
>>> 'data': { '*size': 'int' },
>>> - 'base': 'ChardevCommon' }
>>> + 'base': 'ChardevCommon',
>>> + 'aliases': [ { 'name': 'capacity', 'source': [ 'size' ] } ] }
>>>
>>> ##
>>> # @ChardevQemuVDAgent:
>>>
>>> With this patch:
>>>
>>> $ qemu-system-x86_64 -chardev ringbuf,id=chr0,capacity=lots
>>> qemu-system-x86_64: -chardev ringbuf,id=chr0,capacity=lots: Parameter 'backend.data.size' expects integer
>>>
>>> The error message fails to connect to the offending key=value.
>>
>> Same problem as above. The error message should use the key that the
>> user actually specified.
>
> Yes.
>
>>> Example: manual transformation leads to confusion #1
>>>
>>> Starting point:
>>>
>>> $ qemu-system-x86_64 -chardev udp,id=chr0,port=12345,localaddr=localhost
>>>
>>> Works. host defaults to localhost, localport defaults to 0, same as in
>>> git master.
>>>
>>> Now "forget" to specify @port:
>>>
>>> $ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost
>>> qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost: Parameter 'backend.data.remote.data.port' is missing
>>>
>>> Again, the visitor's error message uses "standard" notation.
>>
>> The output isn't wrong, it's just more verbose than necessary.
>>
>> Getting this one shortened is a bit harder because the right name is
>> ambiguous, the user didn't specify anything we can just print back.
>>
>> Possibly in CLI context, making use of any wildcard aliases would be a
>> reasonable strategy. This would reduce this one to just 'port'.
>
> Which of the possible keys we use in the error message boils down to
> picking a preferred key.
>
>>> We obediently do what the error message tells us to do:
>>>
>>> $ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345
>>> qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345: Parameters 'backend.*' used inconsistently
>>>
>>> Mission accomplished: confusion :)
>>
>> This one already fails before aliases do their work. The reason is that
>> the default key for -chardev is 'backend', and QMP and CLI use the same
>> name 'backend' for two completely different things.
>
> Right. I was confused (and the mission was truly accomplished).
>
>> We could rename the default key into 'x-backend' and make it behave the
>> same as 'backend', then the keyval parser would only fail when you
>> explicitly wrote '-chardev backend=udp,...' and the problem is more
>> obvious.
>
> Technically a compatibility break, but we can hope that the longhand
> form we change isn't used.
>
>> By the way, we have a very similar issue with '-drive file=...', if we
>> ever want to parse that one with the keyval parser. It can be both a
>> string for the filename or an object for the configuration of the 'file'
>> child that many block drivers have.
>
> Should I be running for the hills?
>
>>> Example: manual transformation leads to confusion #2
>>>
>>> Confusion is possible even without tricking the user into mixing
>>> "standard" and "alternate" explicitly:
>>>
>>> $ qemu-system-x86_64 -chardev udp,id=chr0,backend.data.remote.type=udp
>>> qemu-system-x86_64: -chardev udp,id=chr0,backend.data.remote.type=udp: Parameters 'backend.*' used inconsistently
>>>
>>> Here, the user tried to stick to "standard", but forgot to specify a
>>> required member. The transformation machinery then "helpfully"
>>> transformed nothing into something, which made the visitor throw up.
>>
>> Not the visitor, but the keyval parser. Same problem as above.
>>
>>> Clear error reporting is a critical part of a human-friendly interface.
>>> We have two separate problems with it:
>>>
>>> 1. The visitor reports errors as if aliases didn't exist
>>>
>>> Fixing this is "merely" a matter of tracing back alias applications.
>>> More complexity...
>>>
>>> 2. The visitor reports errors as if manual transformation didn't exist
>>>
>>> Manual transformation can distort the users input beyond recognition.
>>> The visitor reports errors for the transformed input.
>>>
>>> To fix this one, we'd have to augment the parse tree so it points
>>> back at the actual user input, and then make the manual
>>> transformations preserve that. Uff!
>>
>> Manual transformations are hard to write in a way that they give perfect
>> results. As long as they are only used for legacy syntax and we expect
>> users to move to a new way to spell things, this might be acceptable for
>> a transition period until we remove the old syntax.
>
> Valid point.
>
>> In other cases, the easiest way is probably to duplicate even more of
>> the schema and manually make sure that the visitor will accept the input
>> before we transform it.
>>
>> The best way to avoid this is providing tools in QAPI that make manual
>> transformations unnecessary.
>
> Reducing the amount of handwritten code is good, as long as the price is
> reasonable. Complex code fetches a higher price.
>
> I think there are a couple of ways to skin this cat.
>
> 0. Common to all ways: specify "standard" in the QAPI schema. This
> specifies both the "standard" wire format, its parse tree
> (represented as QObject), and the "standard" C interface (C types,
> basically).
>
> Generic parsers parse into the parse tree. The appropriate input
> visitor validates and converts to C types.
>
> 1. Parse "alternate" by any means (QemuOpts, keyval, ad hoc), validate,
> then transform for the "standard" C interface. Parsing and
> validating can fail, the transformation can't.
>
> Drawbacks:
>
> * We duplicate validation, which is a standing invitation for bugs.
>
> * Longwinded, handwritten transformation code. Less vulnerable to
> bugs than validation code, I believe.
>
> 2. Parse "alternate" by any means (QemuOpts, keyval, ad hoc), transform
> to "standard" parse tree.
>
> Drawbacks:
>
> * Bad error messages from input visitor.
>
> * The handwritten transformation code is more longwinded than with 1.
>
> 3. Specify "alternate" in the QAPI schema. Map "alternate" C interface
> to the "standard" one.
>
> Drawbacks:
>
> * QAPI schema code duplication
>
> * Longwinded, handwritten mapping code.
>
> This is what we did with SocketAddressLegacy.
>
> 4. Extend the QAPI schema language to let us specify non-standard wire
> format(s). Use that to get the "alternate" wire format we need.
>
> Drawbacks:
>
> * QAPI schema language and generator become more complex.
>
> * Input visitor(s) become more complex.
>
> * Unless we accept "alternate" everywhere, we need a way to limit it
> to where it's actually needed. More complexity.
>
> The concrete proposal on the table is aliases. They are not a
> complete solution. To solve the whole problem, we combine them with
> 2. We accept 4's drawbacks for a (sizable) reduction of 2's.
> Additionally, we accept "ghost" aliases.
>
> 5. Extend the QAPI schema language to let us specify "alternate" wire
> formats for "standard" types
>
> This is like 3, except the schema code is mostly referenced instead
> duplicated, and the mapping code is generated. Something like
> "derive type T' from T with these members moved / renamed".
>
> Drawbacks
>
> * QAPI schema language and generator become more complex.
>
> * Unlike 4, we don't have working code.
>
> Like 4, this will likely solve only part of the problem.
I got one more:
6. Stir more sugar into the input visitor we use with keyval_parse():
- Recognze unique suffix of full key. Example: "ipv6" as
abbreviation of "backend.data.remote.data.ipv4".
- Default union type tag when variant members of exactly one branch
are present. Example: when we got "backend.data.addr.data.host",
"backend.data.addr.type" defaults to "inet".
Beware, this is even hairier than it may look. For instance, we want
to expand "host=playground.redhat.com,port=12345" into
backend.type=socket
backend.data.addr.type=inet
backend.data.addr.data.host=playground.redhat.com
backend.data.addr.data.port=12345
and not into
backend.type=udp
backend.data.remote.type=inet
backend.data.remote.data.host=playground.redhat.com
backend.data.remote.data.port=12345
I'm afraid this idea also solves only part of the problem.
> Which one is the least bad?
>
> If this was just about dragging deprecated interfaces to the end of
> their grace period, I'd say "whatever is the least work", and that's
> almost certainly whatever we have now, possibly hacked up to use the
> appropriate internal interface.
>
> Unfortunately, it is also about providing a friendlier interface to
> humans "forever".
>
> With 4 or 5, we invest into infrastructure once, then maintain it
> forever, to make solving the problem easier and the result easier to
> maintain.
>
> Whether the investment will pay off depends on how big and bad the
> problem is. Hard to say. One of the reasons we're looking at -chardev
> now is that we believe it's the worst of the bunch. But how big is the
> bunch, and how bad are the others in there?
>
>>> I'm afraid I need another round of thinking on how to best drag legacy
>>> syntax along when we QAPIfy.
>>
>> Let me know when you've come to any conclusions (or more things to
>> discuss).
>
> Is 3 too awful to contemplate?
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: [PATCH v3 6/6] tests/qapi-schema: Test cases for aliases
2021-10-13 11:10 ` Markus Armbruster
@ 2021-10-14 9:35 ` Kevin Wolf
0 siblings, 0 replies; 43+ messages in thread
From: Kevin Wolf @ 2021-10-14 9:35 UTC (permalink / raw)
To: Markus Armbruster; +Cc: jsnow, qemu-devel
Am 13.10.2021 um 13:10 hat Markus Armbruster geschrieben:
> Markus Armbruster <armbru@redhat.com> writes:
>
> > Kevin Wolf <kwolf@redhat.com> writes:
> >
> >> Am 11.10.2021 um 09:44 hat Markus Armbruster geschrieben:
> >>> Kevin Wolf <kwolf@redhat.com> writes:
> >>>
> >>> [...]
> >>>
> >>> > What I had in mind was using the schema to generate the necessary code,
> >>> > using the generic keyval parser everywhere, and just providing a hook
> >>> > where the QDict could be modified after the keyval parser and before the
> >>> > visitor. Most command line options would not have any explicit code for
> >>> > parsing input, only the declarative schema and the final option handler
> >>> > getting a QAPI type (which could actually be the corresponding QMP
> >>> > command handler in the common case).
> >>>
> >>> A walk to the bakery made me see a problem with transforming the qdict
> >>> between keyval parsing and the visitor: error reporting. On closer
> >>> investigation, the problem exists even with just aliases.
> >>
> >> I already commented on part of this on IRC, but let me reply here as
> >> well.
> >>
> >> On general remark is that I would make the same distinction between
> >> aliases for compatibility and usability that I mentioned elsewhere in
> >> this thread.
> >>
> >> In the case of compatibility, it's already a concession that we still
> >> accept the option - suboptimal error messages for incorrect command
> >> lines aren't a major concern for me there. Users are supposed to move to
> >> the new syntax anyway.
> >
> > Well, these aren't "incorrect", merely deprecated. Bad UX is still bad
> > there, but at least it'll "fix" itself when the deprecated part goes
> > away.
Most of the error messages aren't "incorrect" either, merely suboptimal
and guiding the user towards verbose non-deprecated alternatives.
> >>> We obediently do what the error message tells us to do:
> >>>
> >>> $ qemu-system-x86_64 -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345
> >>> qemu-system-x86_64: -chardev udp,id=chr0,localaddr=localhost,backend.data.remote.data.port=12345: Parameters 'backend.*' used inconsistently
> >>>
> >>> Mission accomplished: confusion :)
> >>
> >> This one already fails before aliases do their work. The reason is that
> >> the default key for -chardev is 'backend', and QMP and CLI use the same
> >> name 'backend' for two completely different things.
> >
> > Right. I was confused (and the mission was truly accomplished).
> >
> >> We could rename the default key into 'x-backend' and make it behave the
> >> same as 'backend', then the keyval parser would only fail when you
> >> explicitly wrote '-chardev backend=udp,...' and the problem is more
> >> obvious.
> >
> > Technically a compatibility break, but we can hope that the longhand
> > form we change isn't used.
No, it's not a compatibility break. Existing command lines can only have
'backend=...', but not 'backend.*=...', so there is no conflict and they
keep working.
> >> By the way, we have a very similar issue with '-drive file=...', if we
> >> ever want to parse that one with the keyval parser. It can be both a
> >> string for the filename or an object for the configuration of the 'file'
> >> child that many block drivers have.
> >
> > Should I be running for the hills?
If that means that I can then just commit whatever feels right to me? :-P
> >>> Example: manual transformation leads to confusion #2
> >>>
> >>> Confusion is possible even without tricking the user into mixing
> >>> "standard" and "alternate" explicitly:
> >>>
> >>> $ qemu-system-x86_64 -chardev udp,id=chr0,backend.data.remote.type=udp
> >>> qemu-system-x86_64: -chardev udp,id=chr0,backend.data.remote.type=udp: Parameters 'backend.*' used inconsistently
> >>>
> >>> Here, the user tried to stick to "standard", but forgot to specify a
> >>> required member. The transformation machinery then "helpfully"
> >>> transformed nothing into something, which made the visitor throw up.
> >>
> >> Not the visitor, but the keyval parser. Same problem as above.
> >>
> >>> Clear error reporting is a critical part of a human-friendly interface.
> >>> We have two separate problems with it:
> >>>
> >>> 1. The visitor reports errors as if aliases didn't exist
> >>>
> >>> Fixing this is "merely" a matter of tracing back alias applications.
> >>> More complexity...
> >>>
> >>> 2. The visitor reports errors as if manual transformation didn't exist
> >>>
> >>> Manual transformation can distort the users input beyond recognition.
> >>> The visitor reports errors for the transformed input.
> >>>
> >>> To fix this one, we'd have to augment the parse tree so it points
> >>> back at the actual user input, and then make the manual
> >>> transformations preserve that. Uff!
> >>
> >> Manual transformations are hard to write in a way that they give perfect
> >> results. As long as they are only used for legacy syntax and we expect
> >> users to move to a new way to spell things, this might be acceptable for
> >> a transition period until we remove the old syntax.
> >
> > Valid point.
> >
> >> In other cases, the easiest way is probably to duplicate even more of
> >> the schema and manually make sure that the visitor will accept the input
> >> before we transform it.
> >>
> >> The best way to avoid this is providing tools in QAPI that make manual
> >> transformations unnecessary.
> >
> > Reducing the amount of handwritten code is good, as long as the price is
> > reasonable. Complex code fetches a higher price.
> >
> > I think there are a couple of ways to skin this cat.
> >
> > 0. Common to all ways: specify "standard" in the QAPI schema. This
> > specifies both the "standard" wire format, its parse tree
> > (represented as QObject), and the "standard" C interface (C types,
> > basically).
> >
> > Generic parsers parse into the parse tree. The appropriate input
> > visitor validates and converts to C types.
> >
> > 1. Parse "alternate" by any means (QemuOpts, keyval, ad hoc), validate,
> > then transform for the "standard" C interface. Parsing and
> > validating can fail, the transformation can't.
> >
> > Drawbacks:
> >
> > * We duplicate validation, which is a standing invitation for bugs.
> >
> > * Longwinded, handwritten transformation code. Less vulnerable to
> > bugs than validation code, I believe.
> >
> > 2. Parse "alternate" by any means (QemuOpts, keyval, ad hoc), transform
> > to "standard" parse tree.
> >
> > Drawbacks:
> >
> > * Bad error messages from input visitor.
> >
> > * The handwritten transformation code is more longwinded than with 1.
> >
> > 3. Specify "alternate" in the QAPI schema. Map "alternate" C interface
> > to the "standard" one.
> >
> > Drawbacks:
> >
> > * QAPI schema code duplication
> >
> > * Longwinded, handwritten mapping code.
> >
> > This is what we did with SocketAddressLegacy.
> >
> > 4. Extend the QAPI schema language to let us specify non-standard wire
> > format(s). Use that to get the "alternate" wire format we need.
> >
> > Drawbacks:
> >
> > * QAPI schema language and generator become more complex.
> >
> > * Input visitor(s) become more complex.
> >
> > * Unless we accept "alternate" everywhere, we need a way to limit it
> > to where it's actually needed. More complexity.
> >
> > The concrete proposal on the table is aliases. They are not a
> > complete solution. To solve the whole problem, we combine them with
> > 2. We accept 4's drawbacks for a (sizable) reduction of 2's.
> > Additionally, we accept "ghost" aliases.
We combine it with 2 to solve these problems:
* Automatically determining the union type for SocketAddressLegacy
* Accepting short-form booleans (deprecated since 6.0, can be dropped)
* Diverging default values between CLI and QMP.
This includes a case in chardev-udp where QMP requires a whole
SocketAddressLegacy or nothing, but CLI accepts specifying only one of
host/port and provides a default for the other one.
* Enable aliases for chardev types (= aliases for enum values)
Solving these in generic QAPI code would probably be possible, but apart
from the short-form booleans the drawbacks of 2 are pretty much
insignificant (especially the error messages part doesn't apply), so it
feels tolerable.
> > 5. Extend the QAPI schema language to let us specify "alternate" wire
> > formats for "standard" types
> >
> > This is like 3, except the schema code is mostly referenced instead
> > duplicated, and the mapping code is generated. Something like
> > "derive type T' from T with these members moved / renamed".
> >
> > Drawbacks
> >
> > * QAPI schema language and generator become more complex.
> >
> > * Unlike 4, we don't have working code.
> >
> > Like 4, this will likely solve only part of the problem.
>
> I got one more:
>
> 6. Stir more sugar into the input visitor we use with keyval_parse():
>
> - Recognze unique suffix of full key. Example: "ipv6" as
> abbreviation of "backend.data.remote.data.ipv4".
Deciding "unique" in the visitor code feels tricky. You don't know what
future code will visit.
The only option I see is that the QAPI generator already compiles a full
list of possible abbreviations for every object type. (Obviously fails
for 'any', but I think this is not a problem.) Ugly.
Maintaining compatibility feels hard. Adding a new member "ipv4" to a
completely different type that might be used in a different union
branch, would make this stop working, probably without anyone noticing.
> - Default union type tag when variant members of exactly one branch
> are present. Example: when we got "backend.data.addr.data.host",
> "backend.data.addr.type" defaults to "inet".
>
> Beware, this is even hairier than it may look. For instance, we want
> to expand "host=playground.redhat.com,port=12345" into
>
> backend.type=socket
> backend.data.addr.type=inet
> backend.data.addr.data.host=playground.redhat.com
> backend.data.addr.data.port=12345
>
> and not into
>
> backend.type=udp
> backend.data.remote.type=inet
> backend.data.remote.data.host=playground.redhat.com
> backend.data.remote.data.port=12345
>
> I'm afraid this idea also solves only part of the problem.
Am I misunderstanding how you intended it to work? I thought it wouldn't
accept host=... at all because it isn't a unique suffix. In which case
it would obviously solve even less of the problem.
> > Which one is the least bad?
> >
> > If this was just about dragging deprecated interfaces to the end of
> > their grace period, I'd say "whatever is the least work", and that's
> > almost certainly whatever we have now, possibly hacked up to use the
> > appropriate internal interface.
> >
> > Unfortunately, it is also about providing a friendlier interface to
> > humans "forever".
> >
> > With 4 or 5, we invest into infrastructure once, then maintain it
> > forever, to make solving the problem easier and the result easier to
> > maintain.
> >
> > Whether the investment will pay off depends on how big and bad the
> > problem is. Hard to say. One of the reasons we're looking at -chardev
> > now is that we believe it's the worst of the bunch. But how big is the
> > bunch, and how bad are the others in there?
> >
> >>> I'm afraid I need another round of thinking on how to best drag legacy
> >>> syntax along when we QAPIfy.
> >>
> >> Let me know when you've come to any conclusions (or more things to
> >> discuss).
> >
> > Is 3 too awful to contemplate?
I don't feel it's worth my while because the result would be only
marginally better than the existing 1.
The short-term alternative is just adding JSON and leaving the rest
alone. Maybe deprecate the old syntax and just break it at some flag day
together with -object and -device. This fixes the compatibility
problems, but leaves the usability problem unaddressed, i.e. it will
result in a very human unfriendly syntax. If that is preferable to
having suboptimal error messages in legacy cases, I'm not sure. But at
least it will be a lot easier for me.
If we do want to address a wider problem, including making CLI more
human friendly (which would possibly not only apply to -chardev, but
also -blockdev and other options if it allows more than one valid syntax
on the command line), I'm willing to work on 4 or 5.
I think mandatory (to avoid ghosts) flattening and local renames can be
solved without touching the visitor like this alias series does, just by
generating different code, like:
if (v.profile == QMP):
visit_Foo(...)
else if (v.profile == CLI):
visit_Foo_members(...)
I don't see yet how to do this for non-local renames (like localaddr ->
local.data.host for charev-udp). Making the alternate syntax mandatory
to avoid ghosts also means that the solution can't be applied to
existing options like -blockdev without breaking compatibility.
Anyway, I've made two attempts at solving a wider problem (first just
flattening simple unions, and then more generically with aliases) and
both got shot down by you. For the third attempt I'd prefer very clear
requirements before I even start because I don't feel like wasting
another year.
Kevin
^ permalink raw reply [flat|nested] 43+ messages in thread
end of thread, other threads:[~2021-10-14 9:41 UTC | newest]
Thread overview: 43+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-12 16:11 [PATCH v3 0/6] qapi: Add support for aliases Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 1/6] qapi: Add interfaces for alias support to Visitor Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 2/6] qapi: Remember alias definitions in qobject-input-visitor Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 3/6] qapi: Simplify full_name_nth() " Kevin Wolf
2021-08-12 16:11 ` [PATCH v3 4/6] qapi: Apply aliases " Kevin Wolf
2021-09-06 15:16 ` Markus Armbruster
2021-09-08 13:01 ` Kevin Wolf
2021-09-14 6:58 ` Markus Armbruster
2021-09-14 9:35 ` Kevin Wolf
2021-09-14 14:24 ` Markus Armbruster
2021-08-12 16:11 ` [PATCH v3 5/6] qapi: Add support for aliases Kevin Wolf
2021-09-06 15:24 ` Markus Armbruster
2021-09-09 16:39 ` Kevin Wolf
2021-09-14 8:42 ` Markus Armbruster
2021-09-14 11:00 ` Markus Armbruster
2021-09-14 14:24 ` Kevin Wolf
2021-09-16 7:49 ` Markus Armbruster
2021-08-12 16:11 ` [PATCH v3 6/6] tests/qapi-schema: Test cases " Kevin Wolf
2021-09-06 15:28 ` Markus Armbruster
2021-09-10 15:04 ` Kevin Wolf
2021-09-14 8:59 ` Markus Armbruster
2021-09-14 10:05 ` Kevin Wolf
2021-09-14 13:29 ` Markus Armbruster
2021-09-15 9:24 ` Kevin Wolf
2021-09-17 8:26 ` Markus Armbruster
2021-09-17 15:03 ` Kevin Wolf
2021-10-02 13:33 ` Markus Armbruster
2021-10-04 14:07 ` Kevin Wolf
2021-10-05 13:49 ` Markus Armbruster
2021-10-05 17:05 ` Kevin Wolf
2021-10-06 13:11 ` Markus Armbruster
2021-10-06 16:36 ` Kevin Wolf
2021-10-07 11:06 ` Markus Armbruster
2021-10-07 16:12 ` Kevin Wolf
2021-10-08 10:17 ` Markus Armbruster
2021-10-12 14:00 ` Kevin Wolf
2021-10-11 7:44 ` Markus Armbruster
2021-10-12 14:36 ` Kevin Wolf
2021-10-13 9:41 ` Markus Armbruster
2021-10-13 11:10 ` Markus Armbruster
2021-10-14 9:35 ` Kevin Wolf
2021-08-24 9:36 ` [PATCH v3 0/6] qapi: Add support " Markus Armbruster
2021-09-06 15:32 ` Markus Armbruster
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.