* [RFC PATCH v1 0/8] qapi: add generator for Golang interface
@ 2022-04-01 22:40 Victor Toso
2022-04-01 22:40 ` [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go Victor Toso
` (10 more replies)
0 siblings, 11 replies; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:40 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
Hi,
Happy 1st April. Not a joke :) /* ugh, took me too long to send */
This series is about adding a generator in scripts/qapi to produce
Go data structures that can be used to communicate with QEMU over
QMP.
* Why Go?
There are quite a few Go projects that interact with QEMU over QMP
and they endup using a mix of different libraries with their own
code.
** Which projects?
The ones I've found so far:
- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu
- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm
- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers
- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt
(let me know if you know others)
* But Why?
I'm particularly interested in 3 out of 4 of the projects above and
only Kubevirt uses libvirt to handle QEMU. That means that every
QEMU releases where a QMP command, event or other data struct is
added, removed or changed, those projects need to check what changed
in QEMU and then address those changes in their projects, if needed.
The idea behind generating Go data structures is that we can keep a
Go module which can have releases that follow QEMU releases.
The project that uses this Go module, only need to bump the module
version and it shall receive all the changes in their own vendored
code base.
* Status
There are a few rough edges to work on but this is usable. The major
thing I forgot to add is handling Error from Commands. It'll be the
first thing I'll work on next week.
If you want to start using this Today you can fetch it in at
https://gitlab.com/victortoso/qapi-go/
There are quite a few tests that I took from the examples in the
qapi schema. Coverage using go's cover tool is giving `28.6% of
statements`
I've uploaded the a static generated godoc output of the above Go
module here:
https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
* License
While the generator (golang.py in this series) is GPL v2, the
generated code needs to be compatible with other Golang projects,
such as the ones mentioned above. My intention is to keep a Go
module with a MIT license.
* Disclaimer to reviewers
This is my first serious python project so there'll be lots of
suggetions that I'll be happy to take and learn from.
Thanks for taking a look, let me know if you have questions, ideas
or suggestions.
Cheers,
Victor
Victor Toso (8):
qapi: golang: Generate qapi's enum types in Go
qapi: golang: Generate qapi's alternate types in Go
qapi: golang: Generate qapi's struct types in Go
qapi: golang: Generate qapi's union types in Go
qapi: golang: Generate qapi's event types in Go
qapi: golang: Generate qapi's command types in Go
qapi: golang: Add CommandResult type to Go
qapi: golang: document skip function visit_array_types
qapi/meson.build | 1 +
scripts/qapi/golang.py | 727 +++++++++++++++++++++++++++++++++++++++++
scripts/qapi/main.py | 2 +
3 files changed, 730 insertions(+)
create mode 100644 scripts/qapi/golang.py
--
2.35.1
^ permalink raw reply [flat|nested] 71+ messages in thread
* [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
@ 2022-04-01 22:40 ` Victor Toso
2022-05-10 10:06 ` Daniel P. Berrangé
2022-04-01 22:40 ` [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate " Victor Toso
` (9 subsequent siblings)
10 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:40 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
This patch handles QAPI enum types and generates its equivalent in Go.
The highlights of this implementation are:
1. For each QAPI enum, we will define an int32 type in Go to be the
assigned type of this specific enum
2. While in the Go codebase we can use the generated enum values, the
specification requires that, on the wire, the enumeration type's
value to be represented by its string name.
For this reason, each Go type that represent's a QAPI enum will be
implementing the Marshaler[0] and Unmarshaler[1] interfaces to
seamless handle QMP's string to Go int32 value and vice-versa.
3. Naming: CamelCase will be used in any identifier that we want to
export [2], which is everything in this patch.
[0] https://pkg.go.dev/encoding/json#Marshaler
[1] https://pkg.go.dev/encoding/json#Unmarshaler
[2] https://go.dev/ref/spec#Exported_identifiers
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
qapi/meson.build | 1 +
scripts/qapi/golang.py | 225 +++++++++++++++++++++++++++++++++++++++++
scripts/qapi/main.py | 2 +
3 files changed, 228 insertions(+)
create mode 100644 scripts/qapi/golang.py
diff --git a/qapi/meson.build b/qapi/meson.build
index 656ef0e039..0951692332 100644
--- a/qapi/meson.build
+++ b/qapi/meson.build
@@ -90,6 +90,7 @@ qapi_nonmodule_outputs = [
'qapi-init-commands.h', 'qapi-init-commands.c',
'qapi-events.h', 'qapi-events.c',
'qapi-emit-events.c', 'qapi-emit-events.h',
+ 'qapibara.go',
]
# First build all sources
diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
new file mode 100644
index 0000000000..070d4cbbae
--- /dev/null
+++ b/scripts/qapi/golang.py
@@ -0,0 +1,225 @@
+"""
+Golang QAPI generator
+"""
+# Copyright (c) 2021 Red Hat Inc.
+#
+# Authors:
+# Victor Toso <victortoso@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+# Just for type hint on self
+from __future__ import annotations
+
+import os
+from typing import Dict, List, Optional
+
+from .schema import (
+ QAPISchema,
+ QAPISchemaVisitor,
+ QAPISchemaEnumMember,
+ QAPISchemaFeature,
+ QAPISchemaIfCond,
+ QAPISchemaObjectType,
+ QAPISchemaObjectTypeMember,
+ QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
+
+class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
+
+ def __init__(self, prefix: str):
+ super().__init__()
+ self.target = {name: "" for name in ["enum"]}
+ self.schema = None
+ self._docmap = {}
+ self.golang_package_name = "qapi"
+
+ def visit_begin(self, schema):
+ self.schema = schema
+
+ # Every Go file needs to reference its package name
+ for target in self.target:
+ self.target[target] = f"package {self.golang_package_name}\n"
+
+ # Iterate once in schema.docs to map doc objects to its name
+ for doc in schema.docs:
+ if doc.symbol is None:
+ continue
+ self._docmap[doc.symbol] = doc
+
+ def visit_end(self):
+ self.schema = None
+
+ def visit_object_type(self: QAPISchemaGenGolangVisitor,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]
+ ) -> None:
+ pass
+
+ def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ variants: QAPISchemaVariants
+ ) -> None:
+ pass
+
+ def visit_enum_type(self: QAPISchemaGenGolangVisitor,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str]
+ ) -> None:
+ doc = self._docmap.get(name, None)
+ doc_struct, doc_fields = qapi_to_golang_struct_docs(doc)
+
+ value = qapi_to_field_name_enum(members[0].name)
+ fields = f"\t{name}{value} {name} = iota\n"
+ for member in members[1:]:
+ field_doc = " " + doc_fields.get(member.name, "") if doc_fields else ""
+ value = qapi_to_field_name_enum(member.name)
+ fields += f"\t{name}{value}{field_doc}\n"
+
+ self.target["enum"] += f'''
+{doc_struct}
+type {name} int32
+const (
+{fields[:-1]}
+)
+{generate_marshal_methods_enum(members)}
+'''
+
+ def visit_array_type(self, name, info, ifcond, element_type):
+ pass
+
+ def visit_command(self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ arg_type: Optional[QAPISchemaObjectType],
+ ret_type: Optional[QAPISchemaType],
+ gen: bool,
+ success_response: bool,
+ boxed: bool,
+ allow_oob: bool,
+ allow_preconfig: bool,
+ coroutine: bool) -> None:
+ pass
+
+ def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+ pass
+
+ def write(self, output_dir: str) -> None:
+ for module_name, content in self.target.items():
+ go_module = module_name + "s.go"
+ go_dir = "go"
+ pathname = os.path.join(output_dir, go_dir, go_module)
+ odir = os.path.dirname(pathname)
+ os.makedirs(odir, exist_ok=True)
+
+ with open(pathname, "w") as outfile:
+ outfile.write(content)
+
+
+def gen_golang(schema: QAPISchema,
+ output_dir: str,
+ prefix: str) -> None:
+ vis = QAPISchemaGenGolangVisitor(prefix)
+ schema.visit(vis)
+ vis.write(output_dir)
+
+def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
+ type = qapi_to_go_type_name(members[0].defined_in, "enum")
+
+ marshal_switch_cases = ""
+ unmarshal_switch_cases = ""
+ for i in range(len(members)):
+ go_type = type + qapi_to_field_name_enum(members[i].name)
+ name = members[i].name
+
+ marshal_switch_cases += f'''
+ case {go_type}:
+ return []byte(`"{name}"`), nil'''
+
+ unmarshal_switch_cases += f'''
+ case "{name}":
+ (*s) = {go_type}'''
+
+ return f'''
+func (s {type}) MarshalJSON() ([]byte, error) {{
+ switch s {{
+{marshal_switch_cases[1:]}
+ default:
+ fmt.Println("Failed to decode {type}", s)
+ }}
+ return nil, errors.New("Failed")
+}}
+
+func (s *{type}) UnmarshalJSON(data []byte) error {{
+ var name string
+
+ if err := json.Unmarshal(data, &name); err != nil {{
+ return err
+ }}
+
+ switch name {{
+{unmarshal_switch_cases[1:]}
+ default:
+ fmt.Println("Failed to decode {type}", *s)
+ }}
+ return nil
+}}
+'''
+
+# Takes the documentation object of a specific type and returns
+# that type's documentation followed by its member's docs.
+def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
+ if doc is None:
+ return "// No documentation available", None
+
+ main = ""
+ if len(doc.body.text) > 0:
+ main = f"// {doc.body.text}".replace("\n", "\n// ")
+
+ for section in doc.sections:
+ # Skip sections that are not useful for Golang consumers
+ if section.name and "TODO" in section.name:
+ continue
+
+ # Small hack to only add // when doc.body.text was empty
+ prefix = "// " if len(main) == 0 else "\n\n"
+ main += f"{prefix}{section.name}: {section.text}".replace("\n", "\n// ")
+
+ fields = {}
+ for key, value in doc.args.items():
+ if len(value.text) > 0:
+ fields[key] = " // " + ' '.join(value.text.replace("\n", " ").split())
+
+ return main, fields
+
+def qapi_to_field_name_enum(name: str) -> str:
+ return name.title().replace("-", "")
+
+def qapi_to_go_type_name(name: str, meta: str) -> str:
+ if name.startswith("q_obj_"):
+ name = name[6:]
+
+ # We want to keep CamelCase for Golang types. We want to avoid removing
+ # already set CameCase names while fixing uppercase ones, eg:
+ # 1) q_obj_SocketAddress_base -> SocketAddressBase
+ # 2) q_obj_WATCHDOG-arg -> WatchdogArg
+ words = [word for word in name.replace("_", "-").split("-")]
+ name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
+ name += ''.join(word.title() for word in words[1:])
+ return name
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
index fc216a53d3..661fb1e091 100644
--- a/scripts/qapi/main.py
+++ b/scripts/qapi/main.py
@@ -15,6 +15,7 @@
from .common import must_match
from .error import QAPIError
from .events import gen_events
+from .golang import gen_golang
from .introspect import gen_introspect
from .schema import QAPISchema
from .types import gen_types
@@ -54,6 +55,7 @@ def generate(schema_file: str,
gen_events(schema, output_dir, prefix)
gen_introspect(schema, output_dir, prefix, unmask)
+ gen_golang(schema, output_dir, prefix)
def main() -> int:
"""
--
2.35.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
2022-04-01 22:40 ` [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go Victor Toso
@ 2022-04-01 22:40 ` Victor Toso
2022-05-10 10:10 ` Daniel P. Berrangé
2022-04-01 22:40 ` [RFC PATCH v1 3/8] qapi: golang: Generate qapi's struct " Victor Toso
` (8 subsequent siblings)
10 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:40 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
This patch handles QAPI alternate types and generates data structures
in Go that handles it.
At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNull
Alternate types are similar to Union but without a discriminator that
can be used to identify the underlying value on the wire. It is needed
to infer it. That can't be easily mapped in Go.
For each Alternate type, we will be using a Any type to hold the
value. 'Any' is an alias type for interface{} (similar to void* in C).
Similarly to the Enum types (see previous commit), we will implement
Marshaler and Unmarshaler interfaces for the Alternate types and in
those MarshalJSON() and UnmarshalJSON() methods is where we are going
to put the logic to read/set alternate's value.
Note that on UnmarshalJSON(), a helper function called StrictDecode()
will be used. This function is the main logic to infer if a given JSON
object fits in a given Go struct. Because we only have 5 alternate
types, it is not hard to validate the unmarshaling logic but we might
need to improve it in the future if Alternate with branches that have
similar fields appear.
Examples:
* BlockdevRef
```go
// Data to set in BlockdevOptions
qcow2 := BlockdevOptionsQcow2{}
// BlockdevRef using a string
qcow2.File = BlockdevRef{Value: "/some/place/my-image"}
opt := BlockdevOptions{}
opt.Driver = BlockdevDriverQcow2
opt.Value = qcow2
b, _ := json.Marshal(data.s)
// string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}`
```
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++-
1 file changed, 155 insertions(+), 2 deletions(-)
diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 070d4cbbae..8be31bd902 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -31,7 +31,8 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
def __init__(self, prefix: str):
super().__init__()
- self.target = {name: "" for name in ["enum"]}
+ self.target = {name: "" for name in ["alternate", "enum", "helper"]}
+ self.objects_seen = {}
self.schema = None
self._docmap = {}
self.golang_package_name = "qapi"
@@ -43,6 +44,10 @@ def visit_begin(self, schema):
for target in self.target:
self.target[target] = f"package {self.golang_package_name}\n"
+ self.target["helper"] += f'''
+ // Alias for go version lower than 1.18
+ type Any = interface{{}}'''
+
# Iterate once in schema.docs to map doc objects to its name
for doc in schema.docs:
if doc.symbol is None:
@@ -52,6 +57,22 @@ def visit_begin(self, schema):
def visit_end(self):
self.schema = None
+ self.target["helper"] += '''
+// Creates a decoder that errors on unknown Fields
+// Returns true if successfully decoded @from string @into type
+// Returns false without error is failed with "unknown field"
+// Returns false with error is a different error was found
+func StrictDecode(into interface{}, from []byte) error {
+ dec := json.NewDecoder(strings.NewReader(string(from)))
+ dec.DisallowUnknownFields()
+
+ if err := dec.Decode(into); err != nil {
+ return err
+ }
+ return nil
+}
+'''
+
def visit_object_type(self: QAPISchemaGenGolangVisitor,
name: str,
info: Optional[QAPISourceInfo],
@@ -70,7 +91,123 @@ def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
features: List[QAPISchemaFeature],
variants: QAPISchemaVariants
) -> None:
- pass
+ assert name not in self.objects_seen
+ self.objects_seen[name] = True
+
+ # Alternate marshal logic
+ #
+ # To avoid programming errors by users of this generated Go module,
+ # we add a runtime check to error out in case the underlying Go type
+ # doesn't not match any of supported types of the Alternate type.
+ #
+ # Also, Golang's json Marshal will include as JSON's object, the
+ # wrapper we use to hold the Go struct (Value Any -> `Value: {...}`)
+ # This would not be an valid QMP message so we workaround it by
+ # calling RemoveValueObject function.
+ doc = self._docmap.get(name, None)
+ doc_struct, doc_fields = qapi_to_golang_struct_docs(doc)
+
+ members_doc = '''// Options are:'''
+ if_supported_types = ""
+ for var in variants.variants:
+ field_doc = doc_fields.get(var.name, "")
+ field_go_type = qapi_schema_type_to_go_type(var.type.name)
+ members_doc += f'''\n// * {var.name} ({field_go_type}):{field_doc[3:]}'''
+
+ if field_go_type == "nil":
+ field_go_type = "*string"
+
+ if_supported_types += f'''typestr != "{field_go_type}" &&\n\t\t'''
+
+ # Alternate unmarshal logic
+ #
+ # With Alternate types, we have to check the JSON data in order to
+ # identify what is the target Go type. So, this is different than an
+ # union which has an identifier that we can check first.
+ # StrictDecode function tries to match the given JSON data to a given
+ # Go type and it'll error in case it doesn´t fit, for instance, when
+ # there were members in the JSON data that had no equivalent in the
+ # target Go type.
+ #
+ # For this reason, the order is important.
+ #
+ # At this moment, the only field that must be checked first is JSON
+ # NULL, which is relevant to a few alternate types. In the future, we
+ # might need to improve the logic to be foolproof between target Go
+ # types that might have a common base (non existing Today).
+ check_type_str = '''
+ // Check for {name}
+ {{
+ var value {go_type}
+ if err := StrictDecode(&value, data); {error_check} {{
+ s.Value = {set_value}
+ return nil
+ }}
+ }}'''
+ reference_checks = ""
+ for var in variants.variants:
+ if var.type.name == "null":
+ # We use a pointer (by referece) to check for JSON's NULL
+ reference_checks += check_type_str.format(
+ name = var.type.name,
+ go_type = "*string",
+ error_check = "err == nil && value == nil",
+ set_value = "nil")
+ break;
+
+ value_checks = ""
+ for var in variants.variants:
+ if var.type.name != "null":
+ go_type = qapi_schema_type_to_go_type(var.type.name)
+ value_checks += check_type_str.format(
+ name = var.type.name,
+ go_type = go_type,
+ error_check = "err == nil",
+ set_value = "value")
+
+ unmarshal_checks = ""
+ if len(reference_checks) > 0 and len(value_checks) > 0:
+ unmarshal_checks = reference_checks[1:] + value_checks
+ else:
+ unmarshal_checks = reference_checks[1:] if len(reference_checks) > 0 else value_checks[1:]
+
+ self.target["alternate"] += f'''
+{doc_struct}
+type {name} struct {{
+{members_doc}
+ Value Any
+}}
+
+func (s {name}) MarshalJSON() ([]byte, error) {{
+ typestr := fmt.Sprintf("%T", s.Value)
+ typestr = typestr[strings.LastIndex(typestr, ".")+1:]
+
+ // Runtime check for supported types
+ if typestr != "<nil>" &&
+{if_supported_types[:-6]} {{
+ return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
+ }}
+
+ b, err := json.Marshal(s.Value);
+ if err != nil {{
+ return nil, err
+ }}
+
+ return b, nil
+}}
+
+func (s *{name}) UnmarshalJSON(data []byte) error {{
+{unmarshal_checks}
+ // Check type to error out nicely
+ {{
+ var value Any
+ if err := json.Unmarshal(data, &value); err != nil {{
+ return err
+ }}
+ return errors.New(fmt.Sprintf("Unsupported type %T (value: %v)", value, value))
+ }}
+}}
+'''
def visit_enum_type(self: QAPISchemaGenGolangVisitor,
name: str,
@@ -208,6 +345,22 @@ def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
return main, fields
+def qapi_schema_type_to_go_type(type: str) -> str:
+ schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool',
+ 'number': 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8',
+ 'int16': 'int16', 'int32': 'int32', 'int64': 'int64', 'uint8': 'uint8',
+ 'uint16': 'uint16', 'uint32': 'uint32', 'uint64': 'uint64',
+ 'any': 'Any', 'QType': 'QType',
+ }
+
+ prefix = ""
+ if type.endswith("List"):
+ prefix = "[]"
+ type = type[:-4]
+
+ type = schema_types_to_go.get(type, type)
+ return prefix + type
+
def qapi_to_field_name_enum(name: str) -> str:
return name.title().replace("-", "")
--
2.35.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [RFC PATCH v1 3/8] qapi: golang: Generate qapi's struct types in Go
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
2022-04-01 22:40 ` [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go Victor Toso
2022-04-01 22:40 ` [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate " Victor Toso
@ 2022-04-01 22:40 ` Victor Toso
2022-04-01 22:41 ` [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union " Victor Toso
` (7 subsequent siblings)
10 siblings, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:40 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
This patch handles QAPI struct types and generates the equivalent
types in Go.
At the time of this writing, it generates 375 structures.
The highlights of this implementation are:
1. Generating an Go struct that requires a @base type, the @base
type is embedded in this Go structure.
Example: See InetSocketAddress with it's base InetSocketAddressBase
2. Differently from previous two types ('enum' and 'alternate'), the
generated QAPI's struct type do not need to implement Marshaler and
Unmarshaler interfaces. This generated structures will naturally
match with JSON Objects.
3. About the Go struct's fields:
i) They can be either by Value or Reference.
ii) Every field that is marked as optional in the QAPI specification
are translated to Reference fields in its Go structure. This design
decision is the most straightforward way to check if a given field
was set or not.
iii) Mandatory fields are always by Value with the exception of QAPI
arrays, which are handled by Reference (to a block of memory) by Go.
iv) All the fields are named with Uppercase due Golang's export
convention.
v) In order to avoid any kind of issues when encoding ordecoding, to
or from JSON, we mark all fields with its @name and, when it is
optional, member, with @omitempty
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang.py | 79 ++++++++++++++++++++++++++++++++++++++++--
1 file changed, 77 insertions(+), 2 deletions(-)
diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 8be31bd902..50e39f8925 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -31,7 +31,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
def __init__(self, prefix: str):
super().__init__()
- self.target = {name: "" for name in ["alternate", "enum", "helper"]}
+ self.target = {name: "" for name in ["alternate", "enum", "helper", "struct"]}
self.objects_seen = {}
self.schema = None
self._docmap = {}
@@ -82,7 +82,31 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor,
members: List[QAPISchemaObjectTypeMember],
variants: Optional[QAPISchemaVariants]
) -> None:
- pass
+ # Do not handle anything besides structs
+ if (name == self.schema.the_empty_object_type.name or
+ not isinstance(name, str) or
+ info.defn_meta not in ["struct"]):
+ return
+
+ assert name not in self.objects_seen
+ self.objects_seen[name] = True
+
+ # visit all inner objects as well, they are not going to be
+ # called by python's generator.
+ if variants:
+ for var in variants.variants:
+ assert isinstance(var.type, QAPISchemaObjectType)
+ self.visit_object_type(self,
+ var.type.name,
+ var.type.info,
+ var.type.ifcond,
+ var.type.base,
+ var.type.local_members,
+ var.type.variants)
+
+ doc = self._docmap.get(info.defn_name, None)
+ self.target[info.defn_meta] += qapi_to_golang_struct(name, doc, info,
+ ifcond, features, base, members, variants)
def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
name: str,
@@ -276,6 +300,14 @@ def gen_golang(schema: QAPISchema,
schema.visit(vis)
vis.write(output_dir)
+# Helper function for boxed or self contained structures.
+def generate_struct_type(type_name, args="", doc_struct="") -> str:
+ args = args if len(args) == 0 else f"\n{args}\n"
+ return f'''
+{doc_struct}
+type {type_name} struct {{{args}}}
+'''
+
def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
type = qapi_to_go_type_name(members[0].defined_in, "enum")
@@ -345,6 +377,46 @@ def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
return main, fields
+# Helper function that is used for most of QAPI types
+def qapi_to_golang_struct(name: str,
+ doc: QAPIDoc,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]) -> str:
+
+ type_name = qapi_to_go_type_name(name, info.defn_meta)
+ doc_struct, doc_fields = qapi_to_golang_struct_docs(doc)
+
+ base_fields = ""
+ if base:
+ base_type_name = qapi_to_go_type_name(base.name, base.meta)
+ base_fields = f"\t// Base type for this struct\n\t{base_type_name}\n"
+
+ own_fields = ""
+ for memb in members:
+ field = qapi_to_field_name(memb.name)
+ member_type = qapi_schema_type_to_go_type(memb.type.name)
+
+ # In Golang, we are using "encoding/json" library to Marshal and Unmarshal between
+ # over-the-wire QMP and Golang struct. The field tag 'omitempty' does not behave as
+ # expected for some types with default values and they only way to "ignore by default"
+ # unset fields is to have them as reference in the Struct.
+ # This way, a *bool and *int can be ignored where a bool or int might have been set.
+ isptr = "*" if memb.optional and member_type[0] not in "*[" else ""
+ optional = ",omitempty" if memb.optional else ""
+ fieldtag = '`json:"{name}{optional}"`'.format(name=memb.name,optional=optional)
+
+ field_doc = doc_fields.get(memb.name, "")
+ own_fields += f"\t{field} {isptr}{member_type}{fieldtag}{field_doc}\n"
+
+ all_fields = base_fields if len(base_fields) > 0 else ""
+ all_fields += own_fields[:-1] if len(own_fields) > 0 else ""
+
+ return generate_struct_type(type_name, all_fields, doc_struct)
+
def qapi_schema_type_to_go_type(type: str) -> str:
schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool',
'number': 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8',
@@ -364,6 +436,9 @@ def qapi_schema_type_to_go_type(type: str) -> str:
def qapi_to_field_name_enum(name: str) -> str:
return name.title().replace("-", "")
+def qapi_to_field_name(name: str) -> str:
+ return name.title().replace("_", "").replace("-", "")
+
def qapi_to_go_type_name(name: str, meta: str) -> str:
if name.startswith("q_obj_"):
name = name[6:]
--
2.35.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
` (2 preceding siblings ...)
2022-04-01 22:40 ` [RFC PATCH v1 3/8] qapi: golang: Generate qapi's struct " Victor Toso
@ 2022-04-01 22:41 ` Victor Toso
2022-05-10 10:26 ` Daniel P. Berrangé
2022-04-01 22:41 ` [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event " Victor Toso
` (6 subsequent siblings)
10 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:41 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
This patch handles QAPI union types and generates the equivalent data
structures and methods in Go to handle it.
At the moment of this writing, it generates 67 structures.
The QAPI union type can be summarized by its common members that are
defined in a @base struct and a @value. The @value type can vary and
depends on @base's field that we call @discriminator. The
@discriminator is always a Enum type.
Golang does not have Unions. The generation of QAPI union type in Go
with this patch, follows similar approach to what is done for QAPI
struct types and QAPI alternate types.
Similarly to Go implementation of QAPI alternate types, we will
implement the Marshaler and Unmarshaler interfaces to seamless decode
from JSON objects to Golang structs and vice versa.
Similarly to Go implementation of QAPI struct types, we will need to
tag @base fields accordingly.
The embedded documentation in Golang's structures and fields are
particularly important here, to help developers know what Types to use
for @value. Runtime checks too.
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang.py | 124 +++++++++++++++++++++++++++++++++++++++--
1 file changed, 119 insertions(+), 5 deletions(-)
diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 50e39f8925..0a1bf430ba 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -31,7 +31,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
def __init__(self, prefix: str):
super().__init__()
- self.target = {name: "" for name in ["alternate", "enum", "helper", "struct"]}
+ self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]}
self.objects_seen = {}
self.schema = None
self._docmap = {}
@@ -82,10 +82,10 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor,
members: List[QAPISchemaObjectTypeMember],
variants: Optional[QAPISchemaVariants]
) -> None:
- # Do not handle anything besides structs
+ # Do not handle anything besides struct and unions.
if (name == self.schema.the_empty_object_type.name or
not isinstance(name, str) or
- info.defn_meta not in ["struct"]):
+ info.defn_meta not in ["struct", "union"]):
return
assert name not in self.objects_seen
@@ -351,6 +351,93 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
}}
'''
+# Marshal methods for Union types
+def generate_marshal_methods(type: str,
+ type_dict: Dict[str, str],
+ discriminator: str = "",
+ base: str = "") -> str:
+ assert base != ""
+ discriminator = "base." + discriminator
+
+ switch_case_format = '''
+ case {name}:
+ value := {case_type}{{}}
+ if err := json.Unmarshal(data, &value); err != nil {{
+ return err
+ }}
+ s.Value = value'''
+
+ if_supported_types = ""
+ added = {}
+ switch_cases = ""
+ for name in sorted(type_dict):
+ case_type = type_dict[name]
+ isptr = "*" if case_type[0] not in "*[" else ""
+ switch_cases += switch_case_format.format(name = name,
+ case_type = case_type)
+ if case_type not in added:
+ if_supported_types += f'''typestr != "{case_type}" &&\n\t\t'''
+ added[case_type] = True
+
+ marshalfn = f'''
+func (s {type}) MarshalJSON() ([]byte, error) {{
+ base, err := json.Marshal(s.{base})
+ if err != nil {{
+ return nil, err
+ }}
+
+ typestr := fmt.Sprintf("%T", s.Value)
+ typestr = typestr[strings.LastIndex(typestr, ".")+1:]
+
+ // "The branches need not cover all possible enum values"
+ // This means that on Marshal, we can safely ignore empty values
+ if typestr == "<nil>" {{
+ return []byte(base), nil
+ }}
+
+ // Runtime check for supported value types
+ if {if_supported_types[:-6]} {{
+ return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
+ }}
+ value, err := json.Marshal(s.Value)
+ if err != nil {{
+ return nil, err
+ }}
+
+ // Workaround to avoid checking s.Value being empty
+ if string(value) == "{{}}" {{
+ return []byte(base), nil
+ }}
+
+ // Removes the last '}}' from base and the first '{{' from value, in order to
+ // return a single JSON object.
+ result := fmt.Sprintf("%s,%s", base[:len(base)-1], value[1:])
+ return []byte(result), nil
+}}
+'''
+ unmarshal_base = f'''
+ var base {base}
+ if err := json.Unmarshal(data, &base); err != nil {{
+ return err
+ }}
+ s.{base} = base
+'''
+ unmarshal_default_warn = f'''
+ default:
+ fmt.Println("Failed to decode {type}", {discriminator})'''
+
+ return f'''{marshalfn}
+func (s *{type}) UnmarshalJSON(data []byte) error {{
+ {unmarshal_base}
+ switch {discriminator} {{
+{switch_cases[1:]}
+ {unmarshal_default_warn}
+ }}
+
+ return nil
+}}
+'''
+
# Takes the documentation object of a specific type and returns
# that type's documentation followed by its member's docs.
def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
@@ -412,10 +499,37 @@ def qapi_to_golang_struct(name: str,
field_doc = doc_fields.get(memb.name, "")
own_fields += f"\t{field} {isptr}{member_type}{fieldtag}{field_doc}\n"
+ union_types = {}
+ variant_fields = ""
+ if variants:
+ variant_fields = f"// Value based on @{variants.tag_member.name}, possible types:"
+ for var in variants.variants:
+ if var.type.is_implicit():
+ continue
+
+ name = variants.tag_member._type_name + var.name.title().replace("-", "")
+ union_types[name] = var.type.name
+ variant_fields += f"\n\t// * {var.type.c_unboxed_type()}"
+
+ variant_fields += f"\n\tValue Any"
+
all_fields = base_fields if len(base_fields) > 0 else ""
all_fields += own_fields[:-1] if len(own_fields) > 0 else ""
-
- return generate_struct_type(type_name, all_fields, doc_struct)
+ all_fields += variant_fields if len(variant_fields) > 0 else ""
+
+ unmarshal_fn = ""
+ if info.defn_meta == "union" and variants is not None:
+ # Union's without variants are the Union's base data structure.
+ # e.g: SchemaInfo's base is SchemainfoBase.
+ discriminator = qapi_to_field_name(variants.tag_member.name)
+ base = qapi_to_go_type_name(variants.tag_member.defined_in,
+ variants.tag_member.info.defn_meta)
+ unmarshal_fn = generate_marshal_methods(type_name,
+ union_types,
+ discriminator = discriminator,
+ base = base_type_name)
+
+ return generate_struct_type(type_name, all_fields, doc_struct) + unmarshal_fn
def qapi_schema_type_to_go_type(type: str) -> str:
schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool',
--
2.35.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event types in Go
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
` (3 preceding siblings ...)
2022-04-01 22:41 ` [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union " Victor Toso
@ 2022-04-01 22:41 ` Victor Toso
2022-05-10 10:42 ` Daniel P. Berrangé
2022-04-01 22:41 ` [RFC PATCH v1 6/8] qapi: golang: Generate qapi's command " Victor Toso
` (5 subsequent siblings)
10 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:41 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
This patch handles QAPI event types and generates data structures in
Go that handles it.
At the moment of this writing, it generates 51 structures (49 events)
In Golang, each event is handled as a Go structure and there is no big
difference, in the Go generated code, between what is a QAPI event
type and what is a QAPI struct.
Each QAPI event has the suffix 'Event' in its Golang data structure
and contains the fields, mandatory and optional, that can be
sent or received.
In addition, there are two structures added to handle QAPI
specification for event types: 'Event' and 'EventBase'.
'EventBase' contains @Name and @Timestamp fields and then 'Event'
extends 'EventBase' with an @Arg field of type 'Any'.
The 'Event' type implements the Unmarshaler to decode the QMP JSON
Object into the correct Golang (event) struct. The goal here is to
facilitate receiving Events.
A TODO for this type is to implement Marshaler for 'Event'. It'll
containt runtime checks to validate before transforming the struct
into a JSON Object.
Example:
```go
qmpMsg := `{
"event" : "MIGRATION",
"timestamp":{
"seconds": 1432121972,
"microseconds": 744001
},
"data":{
"status": "completed"
}
}`
e := Event{}
_ = json.Unmarshal([]byte(qmpMsg), &e)
// e.Name == "MIGRATION"
// e.Arg.(MigrationEvent).Status == MigrationStatusCompleted
```
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++----
1 file changed, 84 insertions(+), 8 deletions(-)
diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 0a1bf430ba..3bb66d07c7 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
def __init__(self, prefix: str):
super().__init__()
- self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]}
+ self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]}
self.objects_seen = {}
self.schema = None
+ self.events = {}
self._docmap = {}
self.golang_package_name = "qapi"
@@ -57,6 +58,24 @@ def visit_begin(self, schema):
def visit_end(self):
self.schema = None
+ # EventBase and Event are not specified in the QAPI schema,
+ # so we need to generate it ourselves.
+ self.target["event"] += '''
+type EventBase struct {
+ Name string `json:"event"`
+ Timestamp struct {
+ Seconds int64 `json:"seconds"`
+ Microseconds int64 `json:"microseconds"`
+ } `json:"timestamp"`
+}
+
+type Event struct {
+ EventBase
+ Arg Any `json:"data,omitempty"`
+}
+'''
+ self.target["event"] += generate_marshal_methods('Event', self.events)
+
self.target["helper"] += '''
// Creates a decoder that errors on unknown Fields
// Returns true if successfully decoded @from string @into type
@@ -279,7 +298,28 @@ def visit_command(self,
pass
def visit_event(self, name, info, ifcond, features, arg_type, boxed):
- pass
+ assert name == info.defn_name
+ type_name = qapi_to_go_type_name(name, info.defn_meta)
+ self.events[name] = type_name
+
+ doc = self._docmap.get(name, None)
+ self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
+ content = ""
+ if self_contained:
+ doc_struct, _ = qapi_to_golang_struct_docs(doc)
+ content = generate_struct_type(type_name, "", doc_struct)
+ else:
+ assert isinstance(arg_type, QAPISchemaObjectType)
+ content = qapi_to_golang_struct(name,
+ doc,
+ arg_type.info,
+ arg_type.ifcond,
+ arg_type.features,
+ arg_type.base,
+ arg_type.members,
+ arg_type.variants)
+
+ self.target["event"] += content
def write(self, output_dir: str) -> None:
for module_name, content in self.target.items():
@@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
}}
'''
-# Marshal methods for Union types
+# Marshal methods for Event and Union types
def generate_marshal_methods(type: str,
type_dict: Dict[str, str],
discriminator: str = "",
base: str = "") -> str:
- assert base != ""
- discriminator = "base." + discriminator
-
- switch_case_format = '''
+ type_is_union = False
+ json_field = ""
+ struct_field = ""
+ if type == "Event":
+ base = type + "Base"
+ discriminator = "base.Name"
+ struct_field = "Arg"
+ json_field = "data"
+ else:
+ assert base != ""
+ discriminator = "base." + discriminator
+ type_is_union = True
+
+ switch_case_format = ""
+ if not type_is_union:
+ switch_case_format = '''
+ case "{name}":
+ tmp := struct {{
+ Value {isptr}{case_type} `json:"{json_field},omitempty"`
+ }}{{}}
+ if err := json.Unmarshal(data, &tmp); err != nil {{
+ return err
+ }}
+ if tmp.Value == nil {{
+ s.{struct_field} = nil
+ }} else {{
+ s.{struct_field} = {isptr}tmp.Value
+ }}'''
+ else:
+ switch_case_format = '''
case {name}:
value := {case_type}{{}}
if err := json.Unmarshal(data, &value); err != nil {{
@@ -374,12 +440,17 @@ def generate_marshal_methods(type: str,
case_type = type_dict[name]
isptr = "*" if case_type[0] not in "*[" else ""
switch_cases += switch_case_format.format(name = name,
+ struct_field = struct_field,
+ json_field = json_field,
+ isptr = isptr,
case_type = case_type)
if case_type not in added:
if_supported_types += f'''typestr != "{case_type}" &&\n\t\t'''
added[case_type] = True
- marshalfn = f'''
+ marshalfn = ""
+ if type_is_union:
+ marshalfn = f'''
func (s {type}) MarshalJSON() ([]byte, error) {{
base, err := json.Marshal(s.{base})
if err != nil {{
@@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str:
words = [word for word in name.replace("_", "-").split("-")]
name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
name += ''.join(word.title() for word in words[1:])
+
+ if meta == "event":
+ name = name[:-3] if name.endswith("Arg") else name
+ name += meta.title()
+
return name
--
2.35.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [RFC PATCH v1 6/8] qapi: golang: Generate qapi's command types in Go
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
` (4 preceding siblings ...)
2022-04-01 22:41 ` [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event " Victor Toso
@ 2022-04-01 22:41 ` Victor Toso
2022-04-01 22:41 ` [RFC PATCH v1 7/8] qapi: golang: Add CommandResult type to Go Victor Toso
` (4 subsequent siblings)
10 siblings, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:41 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
This patch handles QAPI command types and generates data structures in
Go that decodes from QMP JSON Object to Go data structure and vice
versa.
At the time of this writing, it generates 210 structures
(208 commands)
This is very similar to previous commit, that handles QAPI union types
in Go.
Each QAPI command will generate a Go struct with the suffix 'Command'.
Its fields, if any, are the mandatory or optional arguments defined in
the QAPI command.
Simlar to Event, this patch adds two structures to handle QAPI
specification for command types: 'Command' and 'CommandBase'.
'CommandBase' contains @Id and @Name. 'Command' extends 'CommandBase'
with an @Arg field of type 'Any'.
The 'Command' type implements the Unmarshaler to decode QMP JSON
Object into the correct Golang (command) struct.
Marshal example:
```go
cmdArg := SetPasswordCommand{}
cmdArg.Protocol = DisplayProtocolVnc
cmdArg.Password = "secret"
cmd := Command{}
cmd.Name = "set_password"
cmd.Arg = cmdArg
b, _ := json.Marshal(&cmd)
// string(b) == `{"execute":"set_password","arguments":{"protocol":"vnc","password":"secret"}}`
```
Unmarshal example:
```go
qmpCommand := `
{
"execute": "set_password",
"arguments":{
"protocol": "vnc",
"password": "secret"
}
}`
cmd := Command{}
_ = json.Unmarshal([]byte(qmpCommand), &cmd)
// cmd.Name == "set_password"
// cmd1.Arg.(SetPasswordCommand).Protocol == DisplayProtocolVnc
```
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang.py | 49 ++++++++++++++++++++++++++++++++++++++----
1 file changed, 45 insertions(+), 4 deletions(-)
diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 3bb66d07c7..0b9c19babb 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -31,10 +31,11 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
def __init__(self, prefix: str):
super().__init__()
- self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]}
+ self.target = {name: "" for name in ["alternate", "command", "enum", "event", "helper", "struct", "union"]}
self.objects_seen = {}
self.schema = None
self.events = {}
+ self.commands = {}
self._docmap = {}
self.golang_package_name = "qapi"
@@ -76,6 +77,19 @@ def visit_end(self):
'''
self.target["event"] += generate_marshal_methods('Event', self.events)
+ self.target["command"] += '''
+type CommandBase struct {
+ Id string `json:"id,omitempty"`
+ Name string `json:"execute"`
+}
+
+type Command struct {
+ CommandBase
+ Arg Any `json:"arguments,omitempty"`
+}
+'''
+ self.target["command"] += generate_marshal_methods('Command', self.commands)
+
self.target["helper"] += '''
// Creates a decoder that errors on unknown Fields
// Returns true if successfully decoded @from string @into type
@@ -295,7 +309,29 @@ def visit_command(self,
allow_oob: bool,
allow_preconfig: bool,
coroutine: bool) -> None:
- pass
+ assert name == info.defn_name
+ type_name = qapi_to_go_type_name(name, info.defn_meta)
+ self.commands[name] = type_name
+
+ doc = self._docmap.get(name, None)
+ self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
+ content = ""
+ if boxed or self_contained:
+ args = "" if not arg_type else "\n" + arg_type.name
+ doc_struct, _ = qapi_to_golang_struct_docs(doc)
+ content = generate_struct_type(type_name, args, doc_struct)
+ else:
+ assert isinstance(arg_type, QAPISchemaObjectType)
+ content = qapi_to_golang_struct(name,
+ doc,
+ arg_type.info,
+ arg_type.ifcond,
+ arg_type.features,
+ arg_type.base,
+ arg_type.members,
+ arg_type.variants)
+
+ self.target["command"] += content
def visit_event(self, name, info, ifcond, features, arg_type, boxed):
assert name == info.defn_name
@@ -391,7 +427,7 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
}}
'''
-# Marshal methods for Event and Union types
+# Marshal methods for Event, Commad and Union types
def generate_marshal_methods(type: str,
type_dict: Dict[str, str],
discriminator: str = "",
@@ -404,6 +440,11 @@ def generate_marshal_methods(type: str,
discriminator = "base.Name"
struct_field = "Arg"
json_field = "data"
+ elif type == "Command":
+ base = type + "Base"
+ discriminator = "base.Name"
+ struct_field = "Arg"
+ json_field = "arguments"
else:
assert base != ""
discriminator = "base." + discriminator
@@ -636,7 +677,7 @@ def qapi_to_go_type_name(name: str, meta: str) -> str:
name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
name += ''.join(word.title() for word in words[1:])
- if meta == "event":
+ if meta == "event" or meta == "command":
name = name[:-3] if name.endswith("Arg") else name
name += meta.title()
--
2.35.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [RFC PATCH v1 7/8] qapi: golang: Add CommandResult type to Go
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
` (5 preceding siblings ...)
2022-04-01 22:41 ` [RFC PATCH v1 6/8] qapi: golang: Generate qapi's command " Victor Toso
@ 2022-04-01 22:41 ` Victor Toso
2022-04-01 22:41 ` [RFC PATCH v1 8/8] qapi: golang: document skip function visit_array_types Victor Toso
` (3 subsequent siblings)
10 siblings, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:41 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
This patch adds a struct type in Go that will handle return values for
QAPI's command types.
The return value of a Command is, encouraged to be, QAPI's complext
types or an Array of those.
The 'CommandResult' type acts in similar fashion to 'Event' and
'Command', in order to map the right return data structure based on
the issued 'Command'.
This patch also adds a syntax sugar method to 'Command' to return the
'CommandResult' struct to use when receiving the return data.
Example:
```go
cmd := Command{}
cmd.Name = `query-tpm-models`
// bytes, _ := json.Marshal(&cmd)
// send bytes ...
received := `{"return":["tpm-tis","tpm-crb","tpm-spapr"]}`
cmdRet := cmd.GetReturnType()
_ = json.Unmarshal([]byte(received), &cmdRet)
// cmdRet.Value.([]TpmModel)[2] == TpmModelTpmSpapr
```
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang.py | 42 ++++++++++++++++++++++++++++++++++++++++--
1 file changed, 40 insertions(+), 2 deletions(-)
diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 0b9c19babb..5d3395514d 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -36,6 +36,7 @@ def __init__(self, prefix: str):
self.schema = None
self.events = {}
self.commands = {}
+ self.command_results = {}
self._docmap = {}
self.golang_package_name = "qapi"
@@ -90,6 +91,32 @@ def visit_end(self):
'''
self.target["command"] += generate_marshal_methods('Command', self.commands)
+ self.target["command"] += '''
+type CommandResult struct {
+ CommandBase
+ Value Any `json:"return,omitempty"`
+}
+
+func (s Command) GetReturnType() CommandResult {
+ return CommandResult{
+ CommandBase: s.CommandBase,
+ }
+}
+
+// In order to evaluate nil value to empty JSON object
+func (s *CommandResult) MarshalJSON() ([]byte, error) {
+ if s.Value == nil {
+ return []byte(`{"return":{}}`), nil
+ }
+ tmp := struct {
+ Value Any `json:"return"`
+ }{Value: s.Value}
+
+ return json.Marshal(&tmp)
+}
+'''
+ self.target["command"] += generate_marshal_methods('CommandResult', self.command_results)
+
self.target["helper"] += '''
// Creates a decoder that errors on unknown Fields
// Returns true if successfully decoded @from string @into type
@@ -312,6 +339,9 @@ def visit_command(self,
assert name == info.defn_name
type_name = qapi_to_go_type_name(name, info.defn_meta)
self.commands[name] = type_name
+ if ret_type:
+ ret_type_name = qapi_schema_type_to_go_type(ret_type.name)
+ self.command_results[name] = ret_type_name
doc = self._docmap.get(name, None)
self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
@@ -445,6 +475,11 @@ def generate_marshal_methods(type: str,
discriminator = "base.Name"
struct_field = "Arg"
json_field = "arguments"
+ elif type == "CommandResult":
+ base = "CommandBase"
+ discriminator = "s.Name"
+ struct_field = "Value"
+ json_field = "return"
else:
assert base != ""
discriminator = "base." + discriminator
@@ -527,14 +562,17 @@ def generate_marshal_methods(type: str,
return []byte(result), nil
}}
'''
- unmarshal_base = f'''
+ unmarshal_base = ""
+ unmarshal_default_warn = ""
+ if type != "CommandResult":
+ unmarshal_base = f'''
var base {base}
if err := json.Unmarshal(data, &base); err != nil {{
return err
}}
s.{base} = base
'''
- unmarshal_default_warn = f'''
+ unmarshal_default_warn = f'''
default:
fmt.Println("Failed to decode {type}", {discriminator})'''
--
2.35.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [RFC PATCH v1 8/8] qapi: golang: document skip function visit_array_types
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
` (6 preceding siblings ...)
2022-04-01 22:41 ` [RFC PATCH v1 7/8] qapi: golang: Add CommandResult type to Go Victor Toso
@ 2022-04-01 22:41 ` Victor Toso
2022-04-19 18:12 ` [RFC PATCH v1 0/8] qapi: add generator for Golang interface Andrea Bolognani
` (2 subsequent siblings)
10 siblings, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-04-01 22:41 UTC (permalink / raw)
To: qemu-devel; +Cc: John Snow, Eric Blake, Markus Armbruster
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 5d3395514d..9a775d0691 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -321,7 +321,12 @@ def visit_enum_type(self: QAPISchemaGenGolangVisitor,
'''
def visit_array_type(self, name, info, ifcond, element_type):
- pass
+ # TLDR: We don't need to any extra boilerplate in Go to handle Arrays.
+ #
+ # This function is implemented just to be sure that:
+ # 1. Every array type ends with List
+ # 2. Every array type's element is the array type without 'List'
+ assert name.endswith("List") and name[:-4] == element_type.name
def visit_command(self,
name: str,
--
2.35.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
` (7 preceding siblings ...)
2022-04-01 22:41 ` [RFC PATCH v1 8/8] qapi: golang: document skip function visit_array_types Victor Toso
@ 2022-04-19 18:12 ` Andrea Bolognani
2022-04-19 18:42 ` Andrea Bolognani
` (2 more replies)
2022-04-26 11:14 ` Markus Armbruster
2022-05-10 10:47 ` Daniel P. Berrangé
10 siblings, 3 replies; 71+ messages in thread
From: Andrea Bolognani @ 2022-04-19 18:12 UTC (permalink / raw)
To: Victor Toso; +Cc: Eric Blake, John Snow, qemu-devel, Markus Armbruster
[-- Attachment #1: Type: text/plain, Size: 3896 bytes --]
On Sat, Apr 02, 2022 at 12:40:56AM +0200, Victor Toso wrote:
> Thanks for taking a look, let me know if you have questions, ideas
> or suggestions.
Full disclosure: I have only given the actual implementation a very
cursory look so far, and I've focused on the generated Go API
instead.
Overall things look pretty good.
One concern that I have is about naming struct members: things like
SpiceInfo.MouseMode and most others are translated from the QAPI
schema exactly the way you'd expect them, but for example
ChardevCommon.Logappend doesn't look quite right. Of course there's
no way to programmatically figure out what to capitalize, but maybe
there's room for adding this kind of information in the form of
additional annotations or something like that? Same for the various
structs or members that have unexpectedly-capitalized "Tls" or "Vnc"
in them.
To be clear, I don't think the above is a blocker - just something to
be aware of, and think about.
My biggest concern is about the interface offered for commands.
Based on the example you have in the README and how commands are
defined, invoking (a simplified version of) the trace-event-get-state
command would look like
cmd := Command{
Name: "trace-event-get-state",
Arg: TraceEventGetStateCommand{
Name: "qemu_memalign",
},
}
qmp_input, _ := json.Marshal(&cmd)
// qmp_input now contains
// {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
// do something with it
qmp_output :=
([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
ret := cmd.GetReturnType()
_ = json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast
// to a TraceEventInfo struct
First of all, from an application's point of view there are way too
many steps involved: performing this operation should really be as
simple as
ret, _ := qmp.TraceEventGetState("qemu_memalign")
// ret is a TraceEventInfo instance
That's the end state we should be working towards.
Of course that assumes that the "qmp" object knows where the QMP
socket is, knows how to talk the QMP protocol, transparently deals
with serializing and deserializing data... Plus, in some case you
might want to deal with the wire transfer yourself in an
application-specific manner. So it makes sense to have the basic
building blocks available and then build the more ergonomic SDK on
top of that - with only the first part being in scope for this
series.
Even with that in mind, the current interface is IMO problematic
because of its almost complete lack of type safety. Both Command.Arg
and CommandResult.Value are of type Any and CommandBase.Name, which
is used to drive the JSON unmarshal logic as well as ending up on the
wire when executing a command, is just a plain string.
I think the low-level interface should look more like
cmd := TraceEventGetStateCommand{
Name: "qemu_memalign",
}
qmp_input, _ := json.Marshal(&cmd)
// qmp_input looks the same as before
qmp_output :=
([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
ret := TraceEventInfo{}
_ = json.Unmarshal(qmp_output, &ret)
// ret is a TraceEventInfo instance
The advantages over the current implementation is that the compiler
will prevent you from doing something silly like passing the wrong
set of arguments to a commmand, and that the application has to
explicitly spell out what kind of object it expects to get as output.
I'm attaching an incomplete implementation that I used for playing
around. It's obviously too simplistic, but hopefully it will help
illustrate my point.
Dealing with errors and commands that don't have a return value might
require us to have generic CommandResult wrapper after all, but we
should really try as hard as we can to stick to type safe interfaces.
--
Andrea Bolognani / Red Hat / Virtualization
[-- Attachment #2: command.go --]
[-- Type: text/plain, Size: 1369 bytes --]
package main
import (
"encoding/json"
"fmt"
)
type TraceEventGetStateCommand struct {
Name string `json:"name"`
}
func (self *TraceEventGetStateCommand) MarshalJSON() ([]byte, error) {
type Arguments TraceEventGetStateCommand
return json.Marshal(&struct {
Execute string `json:"execute"`
Arguments *Arguments `json:"arguments"`
}{
Execute: "trace-event-get-state",
Arguments: (*Arguments)(self),
})
}
type TraceEventInfo struct {
Name string `json:"name"`
State string `json:"state"`
}
func (self *TraceEventInfo) UnmarshalJSON(data []byte) error {
type Return TraceEventInfo
ret := struct {
Return Return `json:"return"`
}{}
err := json.Unmarshal(data, &ret)
if err != nil {
return err
}
self.Name = ret.Return.Name
self.State = ret.Return.State
return nil
}
func main() {
var err error
var qmp_input []byte
var qmp_output []byte
cmd := TraceEventGetStateCommand{
Name: "qemu_memalign",
}
if qmp_input, err = json.Marshal(&cmd); err != nil {
panic(err)
}
fmt.Printf(" cmd: %v\n", cmd)
fmt.Printf("-> qmp_input: %v\n", string(qmp_input))
qmp_output = ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
ret := TraceEventInfo{}
if err = json.Unmarshal(qmp_output, &ret); err != nil {
panic(err)
}
fmt.Printf("<- qmp_output: %v\n", string(qmp_output))
fmt.Printf(" ret: %v\n", ret)
}
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-19 18:12 ` [RFC PATCH v1 0/8] qapi: add generator for Golang interface Andrea Bolognani
@ 2022-04-19 18:42 ` Andrea Bolognani
2022-04-28 13:50 ` Markus Armbruster
2022-05-09 10:21 ` Victor Toso
2 siblings, 0 replies; 71+ messages in thread
From: Andrea Bolognani @ 2022-04-19 18:42 UTC (permalink / raw)
To: Victor Toso; +Cc: Eric Blake, John Snow, qemu-devel, Markus Armbruster
On Tue, Apr 19, 2022 at 11:12:28AM -0700, Andrea Bolognani wrote:
> Dealing with errors and commands that don't have a return value might
> require us to have generic CommandResult wrapper after all, but we
> should really try as hard as we can to stick to type safe interfaces.
On second thought, this wouldn't actually need to be generic: we
could have something like
type TraceEventGetStateResult struct {
Result TraceEventInfo `json:"return"`
Error *Error `json:"error"`
}
and the caller would check that res.Error is nil before accessing
res.Result.
Commands for which a return value is not expected would just have the
Error part in their corresponding Result struct, and those that can
either return an object or nothing (are there actually any like
that?) could have a pointer as the Result member.
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
` (8 preceding siblings ...)
2022-04-19 18:12 ` [RFC PATCH v1 0/8] qapi: add generator for Golang interface Andrea Bolognani
@ 2022-04-26 11:14 ` Markus Armbruster
2022-05-09 10:52 ` Victor Toso
2022-05-10 8:53 ` Daniel P. Berrangé
2022-05-10 10:47 ` Daniel P. Berrangé
10 siblings, 2 replies; 71+ messages in thread
From: Markus Armbruster @ 2022-04-26 11:14 UTC (permalink / raw)
To: Victor Toso; +Cc: John Snow, Eric Blake, qemu-devel, Marc-André Lureau
Victor Toso <victortoso@redhat.com> writes:
> Hi,
>
> Happy 1st April. Not a joke :) /* ugh, took me too long to send */
>
> This series is about adding a generator in scripts/qapi to produce
> Go data structures that can be used to communicate with QEMU over
> QMP.
>
>
> * Why Go?
>
> There are quite a few Go projects that interact with QEMU over QMP
> and they endup using a mix of different libraries with their own
> code.
>
>
> ** Which projects?
>
> The ones I've found so far:
>
> - podman machine
> https://github.com/containers/podman/tree/main/pkg/machine/qemu
>
> - kata-containers (govmm)
> https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm
>
> - lxd
> https://github.com/lxc/lxd/tree/master/lxd/instance/drivers
>
> - kubevirt (plain json strings)
> https://github.com/kubevirt/kubevirt
>
> (let me know if you know others)
>
>
> * But Why?
>
> I'm particularly interested in 3 out of 4 of the projects above and
> only Kubevirt uses libvirt to handle QEMU. That means that every
> QEMU releases where a QMP command, event or other data struct is
> added, removed or changed, those projects need to check what changed
> in QEMU and then address those changes in their projects, if needed.
>
> The idea behind generating Go data structures is that we can keep a
> Go module which can have releases that follow QEMU releases.
We need to look at "following the QEMU releases" a bit more closely.
Merging your patches gives us the capability to generate a Go interface
to HEAD's version of QMP.
The obvious way for an out-of-tree Go program to use this generated Go
interface is to build with a specific version of it. It can then talk
QMP to any compatible QEMU version.
Compatibility with older QEMUs is not assured: stuff added since is
present on the Go QMP client end, but not on the QEMU QMP server end.
Compatibility with newer QEMUs is subject to our deprecation policy:
In general features are intended to be supported indefinitely once
introduced into QEMU. In the event that a feature needs to be
removed, it will be listed in this section. The feature will remain
functional for the release in which it was deprecated and one
further release. After these two releases, the feature is liable to
be removed.
So, if you stay away from deprecated stuff, you're good for two more
releases at least.
Does this work for the projects you have in mind?
Aside: graceful degradation in case of incompatibility seems desirable.
> The project that uses this Go module, only need to bump the module
> version and it shall receive all the changes in their own vendored
> code base.
Ideally, incompatible changes that affect the Go program show up as
compile errors. Do they?
> * Status
>
> There are a few rough edges to work on but this is usable. The major
> thing I forgot to add is handling Error from Commands. It'll be the
> first thing I'll work on next week.
>
> If you want to start using this Today you can fetch it in at
>
> https://gitlab.com/victortoso/qapi-go/
>
> There are quite a few tests that I took from the examples in the
> qapi schema. Coverage using go's cover tool is giving `28.6% of
> statements`
>
> I've uploaded the a static generated godoc output of the above Go
> module here:
>
> https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
>
>
> * License
>
> While the generator (golang.py in this series) is GPL v2, the
I'd make it v2+, just to express my displeasure with the decision to
make the initial QAPI generator v2 only for no good reason at all.
> generated code needs to be compatible with other Golang projects,
> such as the ones mentioned above. My intention is to keep a Go
> module with a MIT license.
Meh. Can't be helped, I guess.
> * Disclaimer to reviewers
>
> This is my first serious python project so there'll be lots of
> suggetions that I'll be happy to take and learn from.
>
>
> Thanks for taking a look, let me know if you have questions, ideas
> or suggestions.
>
> Cheers,
> Victor
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-19 18:12 ` [RFC PATCH v1 0/8] qapi: add generator for Golang interface Andrea Bolognani
2022-04-19 18:42 ` Andrea Bolognani
@ 2022-04-28 13:50 ` Markus Armbruster
2022-04-29 13:15 ` Andrea Bolognani
2022-05-09 10:21 ` Victor Toso
2 siblings, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-04-28 13:50 UTC (permalink / raw)
To: Andrea Bolognani
Cc: John Snow, Victor Toso, qemu-devel, Eric Blake, Markus Armbruster
Andrea Bolognani <abologna@redhat.com> writes:
> On Sat, Apr 02, 2022 at 12:40:56AM +0200, Victor Toso wrote:
>> Thanks for taking a look, let me know if you have questions, ideas
>> or suggestions.
>
> Full disclosure: I have only given the actual implementation a very
> cursory look so far, and I've focused on the generated Go API
> instead.
>
> Overall things look pretty good.
>
> One concern that I have is about naming struct members: things like
> SpiceInfo.MouseMode and most others are translated from the QAPI
> schema exactly the way you'd expect them, but for example
> ChardevCommon.Logappend doesn't look quite right.
It doesn't look quite right in the QAPI schema, either: @logappend. If
it was @log-append, as it should, then it would get translated to
LogAppend, I guess.
Fixing up style isn't a code generator's job.
> Of course there's
> no way to programmatically figure out what to capitalize,
Some case conversions are straightforward enough. For instance, the C
code generator generates qapi_event_send_some_event() for event
SOME_EVENT, and inclusion guard macro FILE_NAME_H for module file name
file-name.json. No magic involved.
Conversion from lower-case-with-dashes to CamelCase doesn't have to be
magic, either. You just have to accept garbage-in (like missing dashes)
will give you garbage-out. I wouldn't care too much;
CamelCaseIsAnIllegibleMessAnyway.
Conversion from CamelCase is always trouble, though. There is one
instance so far: generating C enumeration constants. We want TYPE_NAME
+ '_' + MEMBER_NAME, where TYPE_NAME is the enumeration type's name
converted from CamelCase to UPPER_CASE_WITH_UNDERSCORES, and MEMBER_NAME
is the member name converted from lower-case-with-dashes to
UPPER_CASE_WITH_UNDERSCORES.
camel_to_upper() tries, but the result is often unappealing, surprising,
or both.
> but maybe
> there's room for adding this kind of information in the form of
> additional annotations or something like that?
We did for enumeration types: 'prefix' overrides the TYPE_NAME prefix.
I fear this was a mistake.
> Same for the various
> structs or members that have unexpectedly-capitalized "Tls" or "Vnc"
> in them.
Examples?
> To be clear, I don't think the above is a blocker - just something to
> be aware of, and think about.
Yup.
[...]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-28 13:50 ` Markus Armbruster
@ 2022-04-29 13:15 ` Andrea Bolognani
2022-05-02 7:21 ` Markus Armbruster
0 siblings, 1 reply; 71+ messages in thread
From: Andrea Bolognani @ 2022-04-29 13:15 UTC (permalink / raw)
To: Markus Armbruster; +Cc: John Snow, Victor Toso, qemu-devel, Eric Blake
On Thu, Apr 28, 2022 at 03:50:55PM +0200, Markus Armbruster wrote:
> Andrea Bolognani <abologna@redhat.com> writes:
> > One concern that I have is about naming struct members: things like
> > SpiceInfo.MouseMode and most others are translated from the QAPI
> > schema exactly the way you'd expect them, but for example
> > ChardevCommon.Logappend doesn't look quite right.
>
> It doesn't look quite right in the QAPI schema, either: @logappend. If
> it was @log-append, as it should, then it would get translated to
> LogAppend, I guess.
>
> Fixing up style isn't a code generator's job.
I agree that the generator shouldn't take too many liberties when
translating names, and specifically should never attempt to figure
out that @logappend should have been @log-append instead.
What I was thinking of was more along the lines of, can we change the
schema so that the proper name is available to the generator without
breaking the wire protocol? Maybe something like
##
# ChardevCommon:
#
# @logappend (rename @log-append): ...
##
That way the generator would have access to both information, and
would thus be able to generate
type ChardevCommon struct {
LogAppend *bool `json:"logappend,omitempty"`
}
The wire protocol would still retain the unappealing name, but at
least client libraries could hide the uglyness from users.
> > Same for the various
> > structs or members that have unexpectedly-capitalized "Tls" or "Vnc"
> > in them.
>
> Examples?
A perfect one is TlsCredsProperties, whose endpoint member is of type
QCryptoTLSCredsEndpoint.
On the VNC front, we have SetPasswordOptionsVnc but also
DisplayReloadOptionsVNC.
There's plenty more, but this should be illustrative enough already.
Capitalization of these acronyms is inconsistent across the schema,
with one of the two forms disagreeing with Go naming expectations.
In this case we might be able to just change the schema without
introducing backwards compatibility issues, though? Type names are
not actually transmitted on the wire IIUC.
> > but maybe
> > there's room for adding this kind of information in the form of
> > additional annotations or something like that?
>
> We did for enumeration types: 'prefix' overrides the TYPE_NAME prefix.
> I fear this was a mistake.
This might be an oversimplification, but I think we might be able to
solve all of these issues with a single annotation in the form
namespace:word-MLA-other-words
So for example QCryptoTLSCredsEndpoint would be annotated with
q:crypto-TLS-creds-endpoint
and each generator would have enough information to produce
identifiers that fit into the corresponding language, such as
qcrypto_tls_creds_endpoint
CryptoTlsCredsEndpoint
or whatever else.
Of course such annotations would only be necessary to deal with
identifiers that are not already following the expected naming
conventions and when MLAs are involved.
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-29 13:15 ` Andrea Bolognani
@ 2022-05-02 7:21 ` Markus Armbruster
2022-05-02 9:04 ` Andrea Bolognani
0 siblings, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-05-02 7:21 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Kevin Wolf, John Snow, Victor Toso, qemu-devel, Eric Blake
Andrea Bolognani <abologna@redhat.com> writes:
> On Thu, Apr 28, 2022 at 03:50:55PM +0200, Markus Armbruster wrote:
>> Andrea Bolognani <abologna@redhat.com> writes:
>> > One concern that I have is about naming struct members: things like
>> > SpiceInfo.MouseMode and most others are translated from the QAPI
>> > schema exactly the way you'd expect them, but for example
>> > ChardevCommon.Logappend doesn't look quite right.
>>
>> It doesn't look quite right in the QAPI schema, either: @logappend. If
>> it was @log-append, as it should, then it would get translated to
>> LogAppend, I guess.
>>
>> Fixing up style isn't a code generator's job.
>
> I agree that the generator shouldn't take too many liberties when
> translating names, and specifically should never attempt to figure
> out that @logappend should have been @log-append instead.
>
> What I was thinking of was more along the lines of, can we change the
> schema so that the proper name is available to the generator without
> breaking the wire protocol? Maybe something like
>
> ##
> # ChardevCommon:
> #
> # @logappend (rename @log-append): ...
> ##
>
> That way the generator would have access to both information, and
> would thus be able to generate
>
> type ChardevCommon struct {
> LogAppend *bool `json:"logappend,omitempty"`
> }
>
> The wire protocol would still retain the unappealing name, but at
> least client libraries could hide the uglyness from users.
At the price of mild inconsistency between the library interface and
QMP.
We could clean up QMP if we care, keeping around the old names for
compatibility. See also Kevin's work on QAPI aliases. Which is much
more ambitious, though.
>> > Same for the various
>> > structs or members that have unexpectedly-capitalized "Tls" or "Vnc"
>> > in them.
>>
>> Examples?
>
> A perfect one is TlsCredsProperties, whose endpoint member is of type
> QCryptoTLSCredsEndpoint.
>
> On the VNC front, we have SetPasswordOptionsVnc but also
> DisplayReloadOptionsVNC.
>
> There's plenty more, but this should be illustrative enough already.
> Capitalization of these acronyms is inconsistent across the schema,
Common issue with camel-cased compounds containing acronyms, because
either way is ugly.
> with one of the two forms disagreeing with Go naming expectations.
Pardon my ignorance: What are Go's expectations?
> In this case we might be able to just change the schema without
> introducing backwards compatibility issues, though? Type names are
> not actually transmitted on the wire IIUC.
Correct!
>> > but maybe
>> > there's room for adding this kind of information in the form of
>> > additional annotations or something like that?
>>
>> We did for enumeration types: 'prefix' overrides the TYPE_NAME prefix.
>> I fear this was a mistake.
>
> This might be an oversimplification, but I think we might be able to
> solve all of these issues with a single annotation in the form
>
> namespace:word-MLA-other-words
>
> So for example QCryptoTLSCredsEndpoint would be annotated with
>
> q:crypto-TLS-creds-endpoint
>
> and each generator would have enough information to produce
> identifiers that fit into the corresponding language, such as
>
> qcrypto_tls_creds_endpoint
> CryptoTlsCredsEndpoint
>
> or whatever else.
>
> Of course such annotations would only be necessary to deal with
> identifiers that are not already following the expected naming
> conventions and when MLAs are involved.
Pardon my ignorance some more: what are MLAs?
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-02 7:21 ` Markus Armbruster
@ 2022-05-02 9:04 ` Andrea Bolognani
2022-05-02 11:46 ` Markus Armbruster
0 siblings, 1 reply; 71+ messages in thread
From: Andrea Bolognani @ 2022-05-02 9:04 UTC (permalink / raw)
To: Markus Armbruster
Cc: Kevin Wolf, John Snow, Victor Toso, qemu-devel, Eric Blake
On Mon, May 02, 2022 at 09:21:36AM +0200, Markus Armbruster wrote:
> Andrea Bolognani <abologna@redhat.com> writes:
> > The wire protocol would still retain the unappealing name, but at
> > least client libraries could hide the uglyness from users.
>
> At the price of mild inconsistency between the library interface and
> QMP.
That's fine, and in fact it already happens all the time when QAPI
names (log-append) are translated to C identifiers (log_append).
> We could clean up QMP if we care, keeping around the old names for
> compatibility. See also Kevin's work on QAPI aliases. Which is much
> more ambitious, though.
I wasn't aware of that effort. Personally I'm always in favor of
cleaning up inconsistencies, so I am automatically a fan :)
That said, the idea of exposing a sub-par Go API until such massive
undertaking can be completed is not terribly appealing. And it would
not address every facet of the issue (see below).
> > Capitalization of these acronyms is inconsistent across the schema,
>
> Common issue with camel-cased compounds containing acronyms, because
> either way is ugly.
Agreed :) But consistent ugliness is still preferable to inconsistent
ugliness.
> > with one of the two forms disagreeing with Go naming expectations.
>
> Pardon my ignorance: What are Go's expectations?
Acronyms are usually all upper case:
https://pkg.go.dev/net/http#ParseHTTPVersion
https://pkg.go.dev/net/http#ProxyURL
https://pkg.go.dev/crypto/tls#NewLRUClientSessionCache
The same seems to be true of Python:
https://docs.python.org/3/library/http.html#http.HTTPStatus
https://docs.python.org/3/library/urllib.error.html#urllib.error.URLError
https://docs.python.org/3/library/xmlrpc.server.html#xmlrpc.server.SimpleXMLRPCServer
Rust, on the other hand, seems to prefer only capitalizing the first
letter of a word, no matter if it's an acronym:
https://doc.rust-lang.org/std/net/struct.TcpStream.html
https://doc.rust-lang.org/std/net/struct.Ipv4Addr.html
https://doc.rust-lang.org/std/ffi/struct.OsString.html
Whether different naming conventions are used for types, functions
and struct members is also language-dependent.
> > In this case we might be able to just change the schema without
> > introducing backwards compatibility issues, though? Type names are
> > not actually transmitted on the wire IIUC.
>
> Correct!
That's great, but even if we decided to change all type names so that
the schema is internally consistent and follows a naming convention
that's reasonable for C, Go and Python, we'd still leave the Rust
interface looking weird... There's no one-size-fits-all name,
unfortunately.
> > Of course such annotations would only be necessary to deal with
> > identifiers that are not already following the expected naming
> > conventions and when MLAs are involved.
>
> Pardon my ignorance some more: what are MLAs?
Multi Letter Acronyms. Which are actually just called "acronyms" I
guess? O:-)
The point is that, if we want to provide a language interface that
feels natural, we need a way to mark parts of a QAPI symbol's name in
a way that makes it possible for the generator to know they're
acronyms and treat them in an appropriate, language-specific manner.
The obvious way to implement this would be with an annotation along
the lines of the one I proposed. Other ideas?
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-02 9:04 ` Andrea Bolognani
@ 2022-05-02 11:46 ` Markus Armbruster
2022-05-02 14:01 ` Andrea Bolognani
0 siblings, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-05-02 11:46 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Kevin Wolf, John Snow, Victor Toso, qemu-devel, Eric Blake
Andrea Bolognani <abologna@redhat.com> writes:
> On Mon, May 02, 2022 at 09:21:36AM +0200, Markus Armbruster wrote:
>> Andrea Bolognani <abologna@redhat.com> writes:
>> > The wire protocol would still retain the unappealing name, but at
>> > least client libraries could hide the uglyness from users.
>>
>> At the price of mild inconsistency between the library interface and
>> QMP.
>
> That's fine, and in fact it already happens all the time when QAPI
> names (log-append) are translated to C identifiers (log_append).
There's a difference between trivial translations like "replace '-' by
'_'" and arbitrary replacement like the one for enumeration constants
involving 'prefix'.
>> We could clean up QMP if we care, keeping around the old names for
>> compatibility. See also Kevin's work on QAPI aliases. Which is much
>> more ambitious, though.
>
> I wasn't aware of that effort. Personally I'm always in favor of
> cleaning up inconsistencies, so I am automatically a fan :)
>
> That said, the idea of exposing a sub-par Go API until such massive
> undertaking can be completed is not terribly appealing.
Point.
> And it would
> not address every facet of the issue (see below).
>
>> > Capitalization of these acronyms is inconsistent across the schema,
>>
>> Common issue with camel-cased compounds containing acronyms, because
>> either way is ugly.
>
> Agreed :) But consistent ugliness is still preferable to inconsistent
> ugliness.
True.
>> > with one of the two forms disagreeing with Go naming expectations.
>>
>> Pardon my ignorance: What are Go's expectations?
>
> Acronyms are usually all upper case:
>
> https://pkg.go.dev/net/http#ParseHTTPVersion
> https://pkg.go.dev/net/http#ProxyURL
> https://pkg.go.dev/crypto/tls#NewLRUClientSessionCache
>
> The same seems to be true of Python:
>
> https://docs.python.org/3/library/http.html#http.HTTPStatus
> https://docs.python.org/3/library/urllib.error.html#urllib.error.URLError
> https://docs.python.org/3/library/xmlrpc.server.html#xmlrpc.server.SimpleXMLRPCServer
>
> Rust, on the other hand, seems to prefer only capitalizing the first
> letter of a word, no matter if it's an acronym:
>
> https://doc.rust-lang.org/std/net/struct.TcpStream.html
> https://doc.rust-lang.org/std/net/struct.Ipv4Addr.html
> https://doc.rust-lang.org/std/ffi/struct.OsString.html
Another strange game where the only winning move is not to play.
> Whether different naming conventions are used for types, functions
> and struct members is also language-dependent.
Yes.
>> > In this case we might be able to just change the schema without
>> > introducing backwards compatibility issues, though? Type names are
>> > not actually transmitted on the wire IIUC.
>>
>> Correct!
>
> That's great, but even if we decided to change all type names so that
> the schema is internally consistent and follows a naming convention
> that's reasonable for C, Go and Python, we'd still leave the Rust
> interface looking weird... There's no one-size-fits-all name,
> unfortunately.
C will remain the primary customer for quite a while. It doesn't come
with universally accepted naming conventions, so we made up our own. I
think we have some wiggle room there. We could, for instance, decide to
clean up the current inconsistent capitalization of acronyms in the QAPI
schema to either style, TCPStream or TcpStream.
Your point remains: some names will still look "weird" in some possible
target languages.
>> > Of course such annotations would only be necessary to deal with
>> > identifiers that are not already following the expected naming
>> > conventions and when MLAs are involved.
>>
>> Pardon my ignorance some more: what are MLAs?
>
> Multi Letter Acronyms. Which are actually just called "acronyms" I
> guess? O:-)
Well played!
> The point is that, if we want to provide a language interface that
> feels natural, we need a way to mark parts of a QAPI symbol's name in
> a way that makes it possible for the generator to know they're
> acronyms and treat them in an appropriate, language-specific manner.
It's not just acronyms. Consider IAmALittleTeapot. If you can assume
that only beginning of words are capitalized, even for acronyms, you can
split this into words without trouble. You can't recover correct case,
though: "i am a little teapot" is wrong.
"Split before capital letter" falls apart when you have characters that
cannot be capitalized: Point3d.
Camel case is hopeless.
> The obvious way to implement this would be with an annotation along
> the lines of the one I proposed. Other ideas?
I'm afraid having the schema spell out names in multiple naming
conventions could be onerous. How many names will need it? Times how
many naming conventions?
Another issue: the fancier the translation from schema name to
language-specific name gets, the harder it becomes to find one from the
other.
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-02 11:46 ` Markus Armbruster
@ 2022-05-02 14:01 ` Andrea Bolognani
2022-05-03 7:57 ` Markus Armbruster
` (2 more replies)
0 siblings, 3 replies; 71+ messages in thread
From: Andrea Bolognani @ 2022-05-02 14:01 UTC (permalink / raw)
To: Markus Armbruster
Cc: Kevin Wolf, John Snow, Victor Toso, qemu-devel, Eric Blake
On Mon, May 02, 2022 at 01:46:23PM +0200, Markus Armbruster wrote:
> Andrea Bolognani <abologna@redhat.com> writes:
> >> > The wire protocol would still retain the unappealing name, but at
> >> > least client libraries could hide the uglyness from users.
> >>
> >> At the price of mild inconsistency between the library interface and
> >> QMP.
> >
> > That's fine, and in fact it already happens all the time when QAPI
> > names (log-append) are translated to C identifiers (log_append).
>
> There's a difference between trivial translations like "replace '-' by
> '_'" and arbitrary replacement like the one for enumeration constants
> involving 'prefix'.
Fair enough.
I still feel that 1) users of a language SDK will ideally not need to
look at the QAPI schema or wire chatter too often and 2) even when
that ends up being necessary, figuring out that LogAppend and
logappend are the same thing is not going to be an unreasonable
hurdle, especially when the status quo would be to work with
Logappend instead.
> > The point is that, if we want to provide a language interface that
> > feels natural, we need a way to mark parts of a QAPI symbol's name in
> > a way that makes it possible for the generator to know they're
> > acronyms and treat them in an appropriate, language-specific manner.
>
> It's not just acronyms. Consider IAmALittleTeapot. If you can assume
> that only beginning of words are capitalized, even for acronyms, you can
> split this into words without trouble. You can't recover correct case,
> though: "i am a little teapot" is wrong.
Is there any scenario in which we would care though? We're in the
business of translating identifiers from one machine representation
to another, so once it has been split up correctly into the words
that compose it (which in your example above it has) then we don't
really care about anything else unless acronyms are involved.
In other words, we can obtain the list of words "i am a little
teapot" programmatically both from IAmALittleTeapot and
i-am-a-little-teapot, and in both cases we can then generate
IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or i_am_a_little_teapot or
whatever is appropriate for the context and target language, but the
fact that in a proper English sentence "I" would have to be
capitalized doesn't really enter the picture.
> "Split before capital letter" falls apart when you have characters that
> cannot be capitalized: Point3d.
>
> Camel case is hopeless.
I would argue that it works quite well for most scenarios, but there
are some corner cases where it's clearly not good enough. If we can
define a way to clue in the generator about "Point3d" having to be
interpreted as "point 3d" and "VNCProps" as "vnc props", then we are
golden. That wouldn't be necessary for simple cases that are already
handled correctly.
A more radical idea would be to start using dash-notation for types
too. That'd remove the word splitting issue altogether, at the cost
of the schema being (possibly) harder to read and more distanced from
the generated code.
You'd still only be able to generate VncProps from vnc-props though.
> > The obvious way to implement this would be with an annotation along
> > the lines of the one I proposed. Other ideas?
>
> I'm afraid having the schema spell out names in multiple naming
> conventions could be onerous. How many names will need it?
I don't have hard data on this. I could try extracting it, but that
might end up being a bigger job than I had anticipated.
My guess is that the number of cases where the naive algorithm can't
split words correctly is relatively small compared to the size of the
entire QAPI schema. Fair warning: I have made incorrect guesses in
the past ;)
> Times how many naming conventions?
Yeah, I don't think requiring all possible permutations to be spelled
out in the schema is the way to go. That's exactly why my proposal
was to offer a way to inject the semantic information that the parser
can't figure out itself.
Once you have a way to inform the generator that "VNCProps" is made
of the two words "vnc" and "props", and that "vnc" is an acronym,
then it can generate an identifier appropriate for the target
language without having to spell out anywhere that such an identifier
would be VNCProps for Go and VncProps for Rust.
By the way, while looking around I realized that we also have to take
into account things like D-Bus: the QAPI type ChardevDBus, for
example, would probably translate verbatim to Go but have to be
changed to ChardevDbus for Rust. Fun :)
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Words are always separated by dashes; "regular" words are entirely
lowercase, while the presence of even a single uppercase letter in a
word denotes the fact that its case should be preserved when the
naming conventions of the target language allow that.
> Another issue: the fancier the translation from schema name to
> language-specific name gets, the harder it becomes to find one from the
> other.
That's true, but at least to me the trade-off feels reasonable.
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-02 14:01 ` Andrea Bolognani
@ 2022-05-03 7:57 ` Markus Armbruster
2022-05-03 9:40 ` Andrea Bolognani
2022-05-09 18:53 ` Victor Toso
2022-05-10 9:52 ` Daniel P. Berrangé
2 siblings, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-05-03 7:57 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Victor Toso, qemu-devel, John Snow, Eric Blake, Kevin Wolf
Andrea Bolognani <abologna@redhat.com> writes:
> On Mon, May 02, 2022 at 01:46:23PM +0200, Markus Armbruster wrote:
>> Andrea Bolognani <abologna@redhat.com> writes:
>> >> > The wire protocol would still retain the unappealing name, but at
>> >> > least client libraries could hide the uglyness from users.
>> >>
>> >> At the price of mild inconsistency between the library interface and
>> >> QMP.
>> >
>> > That's fine, and in fact it already happens all the time when QAPI
>> > names (log-append) are translated to C identifiers (log_append).
>>
>> There's a difference between trivial translations like "replace '-' by
>> '_'" and arbitrary replacement like the one for enumeration constants
>> involving 'prefix'.
>
> Fair enough.
>
> I still feel that 1) users of a language SDK will ideally not need to
> look at the QAPI schema or wire chatter too often
I think the most likely point of contact is the QEMU QMP Reference
Manual.
> even when
> that ends up being necessary, figuring out that LogAppend and
> logappend are the same thing is not going to be an unreasonable
> hurdle, especially when the status quo would be to work with
> Logappend instead.
Yes, it's "mild inconsistency", hardly an unreasonable hurdle. I think
it gets in the way mostly when searching documentation. Differences in
case are mostly harmless, just use case-insensitive search. Use of '_'
vs '-' would also be harmless (just do the replacement), if the use of
'-' in the schema was consistent. Sadly, it's not, and that's already a
perennial low-level annoyance.
My point is: a name override feature like the one you propose needs to
be used with discipline and restraint. Adds to reviewers' mental load.
Needs to be worth it. I'm not saying it isn't, I'm just pointing out a
cost.
>> > The point is that, if we want to provide a language interface that
>> > feels natural, we need a way to mark parts of a QAPI symbol's name in
>> > a way that makes it possible for the generator to know they're
>> > acronyms and treat them in an appropriate, language-specific manner.
>>
>> It's not just acronyms. Consider IAmALittleTeapot. If you can assume
>> that only beginning of words are capitalized, even for acronyms, you can
>> split this into words without trouble. You can't recover correct case,
>> though: "i am a little teapot" is wrong.
>
> Is there any scenario in which we would care though? We're in the
> business of translating identifiers from one machine representation
> to another, so once it has been split up correctly into the words
> that compose it (which in your example above it has) then we don't
> really care about anything else unless acronyms are involved.
>
> In other words, we can obtain the list of words "i am a little
> teapot" programmatically both from IAmALittleTeapot and
> i-am-a-little-teapot, and in both cases we can then generate
> IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or i_am_a_little_teapot or
> whatever is appropriate for the context and target language, but the
> fact that in a proper English sentence "I" would have to be
> capitalized doesn't really enter the picture.
My point is that conversion from CamelCase has two sub-problems:
splitting words and recovering case. Splitting words is easy when
exactly the beginning of words is capitalized. Recovering case is
guesswork. Most English words are all lower case, but some start with a
capital letter, and acronyms are all caps.
Wild idea: assume all lower case, but keep a list of exceptions.
>> "Split before capital letter" falls apart when you have characters that
>> cannot be capitalized: Point3d.
>>
>> Camel case is hopeless.
>
> I would argue that it works quite well for most scenarios, but there
> are some corner cases where it's clearly not good enough. If we can
> define a way to clue in the generator about "Point3d" having to be
> interpreted as "point 3d" and "VNCProps" as "vnc props", then we are
> golden. That wouldn't be necessary for simple cases that are already
> handled correctly.
Hyphenization rules? *Cough* *cough*
> A more radical idea would be to start using dash-notation for types
> too. That'd remove the word splitting issue altogether, at the cost
> of the schema being (possibly) harder to read and more distanced from
> the generated code.
Yes.
> You'd still only be able to generate VncProps from vnc-props though.
>
>> > The obvious way to implement this would be with an annotation along
>> > the lines of the one I proposed. Other ideas?
>>
>> I'm afraid having the schema spell out names in multiple naming
>> conventions could be onerous. How many names will need it?
>
> I don't have hard data on this. I could try extracting it, but that
> might end up being a bigger job than I had anticipated.
I figure extracting is easier for me than for you. But let's have a
closer look at the job at hand first.
The QAPI schema language uses three naming styles:
* lower-case-with-hyphens for command and member names
Many names use upper case and '_'. See pragma command-name-exceptions
and member-name-exceptions.
Some (many?) names lack separators between words (example: logappend).
* UPPER_CASE_WITH_UNDERSCORE for event names
* CamelCase for type names
Capitalization of words is inconsistent in places (example: VncInfo
vs. DisplayReloadOptionsVNC).
What style conversions will we need for Go? Any other conversions come
to mind?
What problems do these conversions have?
> My guess is that the number of cases where the naive algorithm can't
> split words correctly is relatively small compared to the size of the
> entire QAPI schema. Fair warning: I have made incorrect guesses in
> the past ;)
>
>> Times how many naming conventions?
>
> Yeah, I don't think requiring all possible permutations to be spelled
> out in the schema is the way to go. That's exactly why my proposal
> was to offer a way to inject the semantic information that the parser
> can't figure out itself.
>
> Once you have a way to inform the generator that "VNCProps" is made
> of the two words "vnc" and "props", and that "vnc" is an acronym,
> then it can generate an identifier appropriate for the target
> language without having to spell out anywhere that such an identifier
> would be VNCProps for Go and VncProps for Rust.
>
> By the way, while looking around I realized that we also have to take
> into account things like D-Bus: the QAPI type ChardevDBus, for
> example, would probably translate verbatim to Go but have to be
> changed to ChardevDbus for Rust. Fun :)
>
> Revised proposal for the annotation:
>
> ns:word-WORD-WoRD-123Word
>
> Words are always separated by dashes; "regular" words are entirely
> lowercase, while the presence of even a single uppercase letter in a
> word denotes the fact that its case should be preserved when the
> naming conventions of the target language allow that.
Is a word always capitalized the same for a single target language? Or
could capitalization depend on context?
>> Another issue: the fancier the translation from schema name to
>> language-specific name gets, the harder it becomes to find one from the
>> other.
>
> That's true, but at least to me the trade-off feels reasonable.
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-03 7:57 ` Markus Armbruster
@ 2022-05-03 9:40 ` Andrea Bolognani
2022-05-03 11:04 ` Kevin Wolf
` (2 more replies)
0 siblings, 3 replies; 71+ messages in thread
From: Andrea Bolognani @ 2022-05-03 9:40 UTC (permalink / raw)
To: Markus Armbruster
Cc: Victor Toso, qemu-devel, John Snow, Eric Blake, Kevin Wolf
On Tue, May 03, 2022 at 09:57:27AM +0200, Markus Armbruster wrote:
> Andrea Bolognani <abologna@redhat.com> writes:
> > I still feel that 1) users of a language SDK will ideally not need to
> > look at the QAPI schema or wire chatter too often
>
> I think the most likely point of contact is the QEMU QMP Reference
> Manual.
Note that there isn't anything preventing us from including the
original QAPI name in the documentation for the corresponding Go
symbol, or even a link to the reference manual.
So we could make jumping from the Go API documentation, which is what
a Go programmer will be looking at most of the time, to the QMP
documentation pretty much effortless.
> My point is: a name override feature like the one you propose needs to
> be used with discipline and restraint. Adds to reviewers' mental load.
> Needs to be worth it. I'm not saying it isn't, I'm just pointing out a
> cost.
Yeah, I get that.
Note that I'm not suggesting it should be possible for a name to be
completely overridden - I just want to make it possible for a human
to provide the name parsing algorithm solutions to those problems it
can't figure out on its own.
We could prevent that feature from being misused by verifying that
the symbol the annotation is attached to can be derived from the list
of words provided. That way, something like
# SOMEName (completely-DIFFERENT-name)
would be rejected and we would avoid misuse.
> Wild idea: assume all lower case, but keep a list of exceptions.
That could actually work reasonably well for QEMU because we only
need to handle correctly what's in the schema, not arbitrary input.
There's always the risk of the list of exceptions getting out of sync
with the needs of the schema, but there's similarly no guarantee that
annotations are going to be introduced when they are necessary, so
it's mostly a wash.
The only slight advantage of the annotation approach would be that it
might be easier to notice it being missing because it's close to the
name it refers to, while the list of exceptions is tucked away in a
script far away from it.
> The QAPI schema language uses three naming styles:
>
> * lower-case-with-hyphens for command and member names
>
> Many names use upper case and '_'. See pragma command-name-exceptions
> and member-name-exceptions.
Looking at the output generated by Victor's WIP script, it looks like
these are already handled as nicely as those that don't fall under
any exception.
> Some (many?) names lack separators between words (example: logappend).
>
> * UPPER_CASE_WITH_UNDERSCORE for event names
>
> * CamelCase for type names
>
> Capitalization of words is inconsistent in places (example: VncInfo
> vs. DisplayReloadOptionsVNC).
>
> What style conversions will we need for Go? Any other conversions come
> to mind?
>
> What problems do these conversions have?
Go uses CamelCase for pretty much everything: types, methods,
constants...
There's one slight wrinkle, in that the case of the first letter
decides whether it's going to be a PublicName or a privateName. We
can't do anything about that, but it shouldn't really affect us
that much because we'll want all QAPI names to be public.
So the issues preventing us from producing a "perfect" Go API are
1. inconsistent capitalization in type names
-> could be addressed by simply changing the schema, as type
names do not travel on the wire
2. missing dashes in certain command/member names
-> leads to Incorrectcamelcase. Kevin's work is supposed to
address this
3. inability to know which parts of a lower-case-name or
UPPER_CASE_NAME are acronyms or are otherwise supposed to be
capitalized in a specific way
-> leads to WeirdVncAndDbusCapitalization. There's currently no
way, either implemented or planned, to avoid this
In addition to these I'm also thinking that QKeyCode and all the
QCrypto stuff should probably lose their prefixes.
Note that 3 shouldn't be an issue for Rust and addressing 1 would
actually make things worse for that language, because at the moment
at least *some* of the types follow its expected naming rules :)
> > Revised proposal for the annotation:
> >
> > ns:word-WORD-WoRD-123Word
> >
> > Words are always separated by dashes; "regular" words are entirely
> > lowercase, while the presence of even a single uppercase letter in a
> > word denotes the fact that its case should be preserved when the
> > naming conventions of the target language allow that.
>
> Is a word always capitalized the same for a single target language? Or
> could capitalization depend on context?
I'm not aware of any language that would adopt more than a single
style of capitalization, outside of course the obvious
lower_case_name or UPPER_CASE_NAME scenarios where the original
capitalization stops being relevant.
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-03 9:40 ` Andrea Bolognani
@ 2022-05-03 11:04 ` Kevin Wolf
2022-05-10 9:55 ` Daniel P. Berrangé
2022-05-11 6:15 ` Markus Armbruster
2 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-05-03 11:04 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Markus Armbruster, Victor Toso, qemu-devel, John Snow, Eric Blake
Am 03.05.2022 um 11:40 hat Andrea Bolognani geschrieben:
> So the issues preventing us from producing a "perfect" Go API are
>
> 1. inconsistent capitalization in type names
>
> -> could be addressed by simply changing the schema, as type
> names do not travel on the wire
>
> 2. missing dashes in certain command/member names
>
> -> leads to Incorrectcamelcase. Kevin's work is supposed to
> address this
I am surprised that Markus pointed you to my aliases work because the
conclusion after reviewing my last attempt was that he doesn't want the
feature in the proposed form and he wasn't sure if he wants it at all.
To be honest, I considered this one dead.
Even if magically a solution appeared that everyone agreed to, don't
hold your breath, I'm working on different things now and not planning
to get back to QAPI stuff before 7.1 (and preferably not before meeting
Markus in person and discussing the design before I spend the time to
produce another thing that will never be merged). I am hoping to get
back to QAPI later, but I was mostly planning with continuing the QOM
integration, not aliases.
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-19 18:12 ` [RFC PATCH v1 0/8] qapi: add generator for Golang interface Andrea Bolognani
2022-04-19 18:42 ` Andrea Bolognani
2022-04-28 13:50 ` Markus Armbruster
@ 2022-05-09 10:21 ` Victor Toso
2022-05-10 17:37 ` Andrea Bolognani
2 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-09 10:21 UTC (permalink / raw)
To: Andrea Bolognani; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
[-- Attachment #1: Type: text/plain, Size: 10087 bytes --]
Hi!
Sorry for taking some time to reply.
On Tue, Apr 19, 2022 at 11:12:28AM -0700, Andrea Bolognani wrote:
> On Sat, Apr 02, 2022 at 12:40:56AM +0200, Victor Toso wrote:
> > Thanks for taking a look, let me know if you have questions, ideas
> > or suggestions.
>
> Full disclosure: I have only given the actual implementation a very
> cursory look so far, and I've focused on the generated Go API
> instead.
>
> Overall things look pretty good.
Glad to hear.
> One concern that I have is about naming struct members: things like
> SpiceInfo.MouseMode and most others are translated from the QAPI
> schema exactly the way you'd expect them, but for example
> ChardevCommon.Logappend doesn't look quite right. Of course there's
> no way to programmatically figure out what to capitalize, but maybe
> there's room for adding this kind of information in the form of
> additional annotations or something like that? Same for the various
> structs or members that have unexpectedly-capitalized "Tls" or "Vnc"
> in them.
>
> To be clear, I don't think the above is a blocker - just something to
> be aware of, and think about.
There was a good discussion around this with Markus so I don't
want to break it in another thread.
I'm happy that you have found those inconsistencies. I'll reply
on the other thread about it but I don't mind working towards
fixing it, either at code generator level or at QAPI level.
> My biggest concern is about the interface offered for commands.
>
> Based on the example you have in the README and how commands are
> defined, invoking (a simplified version of) the trace-event-get-state
> command would look like
>
> cmd := Command{
> Name: "trace-event-get-state",
> Arg: TraceEventGetStateCommand{
> Name: "qemu_memalign",
> },
> }
> qmp_input, _ := json.Marshal(&cmd)
> // qmp_input now contains
> // {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
> // do something with it
>
> qmp_output :=
> ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
> ret := cmd.GetReturnType()
> _ = json.Unmarshal(qmp_output, &ret)
> // ret is a CommandResult instance whose Value member can be cast
> // to a TraceEventInfo struct
>
> First of all, from an application's point of view there are way too
> many steps involved:
It can actually get worse. I've used a lot of nested struct to
define a Base type for a given Type. In Go, If you try to
initialize a Type that has a nested Struct, you'll need to use
the nested struct Type as field name and this is too verbose.
See https://github.com/golang/go/issues/29438 (merged with:
https://github.com/golang/go/issues/12854)
The main reason that I kept it is because it maps very well with
the over-the-wire protocol.
> performing this operation should really be as
> simple as
>
> ret, _ := qmp.TraceEventGetState("qemu_memalign")
> // ret is a TraceEventInfo instance
>
> That's the end state we should be working towards.
>
> Of course that assumes that the "qmp" object knows where the
> QMP socket is, knows how to talk the QMP protocol,
> transparently deals with serializing and deserializing data...
> Plus, in some case you might want to deal with the wire
> transfer yourself in an application-specific manner. So it
> makes sense to have the basic building blocks available and
> then build the more ergonomic SDK on top of that - with only
> the first part being in scope for this series.
Right. Indeed, I thought a bit about what I want to fit into the
code generator that will reside in QEMU and what we might want to
develop on top of that.
The goal for this series really is generating the data types that
can be converted to/from QMP messages.
I completely agree with the message below: Type validation is
important at this stage.
> Even with that in mind, the current interface is IMO
> problematic because of its almost complete lack of type safety.
> Both Command.Arg and CommandResult.Value are of type Any and
> CommandBase.Name, which is used to drive the JSON unmarshal
> logic as well as ending up on the wire when executing a
> command, is just a plain string.
>
> I think the low-level interface should look more like
>
> cmd := TraceEventGetStateCommand{
> Name: "qemu_memalign",
> }
> qmp_input, _ := json.Marshal(&cmd)
> // qmp_input looks the same as before
That isn't too hard to implement and I've started with this
design at first. Each QAPI Command can implement a method Name()
which returns the over-the-wire name for that Command.
I'm not yet sure if this is preferable over some other syntactic
sugar function that might be generated (this series) or the next
layer that will be on top of this.
But I agree with you that it should be improved before reaching
actual Applications.
> qmp_output :=
> ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
> ret := TraceEventInfo{}
> _ = json.Unmarshal(qmp_output, &ret)
> // ret is a TraceEventInfo instance
>
> The advantages over the current implementation is that the compiler
> will prevent you from doing something silly like passing the wrong
> set of arguments to a commmand, and that the application has to
> explicitly spell out what kind of object it expects to get as output.
I think that, if we know all types that we can have at QAPI spec,
the process of marshalling and unmarshalling should verify it.
So, even if we don't change the expected interface as suggested,
that work needs to be done. For some types, I've already did it,
like for Unions and Alternate types.
Example: https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/unions.go#L28
This union type can have 4 values for the Any interface type. The
code generator documents it to help user's out.
| type SocketAddressLegacy struct {
| // Base type for this struct
| SocketAddressLegacyBase
| // Value based on @type, possible types:
| // * InetSocketAddressWrapper
| // * UnixSocketAddressWrapper
| // * VsockSocketAddressWrapper
| // * StringWrapper
| Value Any
| }
On the Marshal function, I used Sprintf as a way to fetch Value's
type. There are other alternatives but to the cost of adding
other deps.
| func (s SocketAddressLegacy) MarshalJSON() ([]byte, error) {
| base, err := json.Marshal(s.SocketAddressLegacyBase)
| if err != nil {
| return nil, err
| }
|
| typestr := fmt.Sprintf("%T", s.Value)
| typestr =
| typestr[strings.LastIndex(typestr, ".")+1:]
...
| // "The branches need not cover all possible enum values"
| // This means that on Marshal, we can safely ignore empty values
| if typestr == "<nil>" {
| return []byte(base), nil
| }
And then we have some Runtime checks to be sure to avoid the
scenario mismatching Value's type.
| // Runtime check for supported value types
| if typestr != "StringWrapper" &&
| typestr != "InetSocketAddressWrapper" &&
| typestr != "UnixSocketAddressWrapper" &&
| typestr != "VsockSocketAddressWrapper" {
| return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
| }
| value, err := json.Marshal(s.Value)
| if err != nil {
| return nil, err
| }
With Alternate type, extra care was need on Unmarshal as we don't
know the underlying type without looking at the message we
received. That's the only reason of StrictDecode() helper
function.
I'm just pointing out with above examples that I agree with you
with Type safety. It is hard to infer everything at compile-time
so we need some Runtime checks. Having some nicer APIs will
definitely help and improve developer experience too.
> I'm attaching an incomplete implementation that I used for
> playing around. It's obviously too simplistic, but hopefully it
> will help illustrate my point.
>
> Dealing with errors and commands that don't have a return value
> might require us to have generic CommandResult wrapper after
> all, but we should really try as hard as we can to stick to
> type safe interfaces.
Agree. Many thanks again, for the review, suggestions and
discussions.
Cheers,
Victor
> --
> Andrea Bolognani / Red Hat / Virtualization
> package main
>
> import (
> "encoding/json"
> "fmt"
> )
>
> type TraceEventGetStateCommand struct {
> Name string `json:"name"`
> }
>
> func (self *TraceEventGetStateCommand) MarshalJSON() ([]byte, error) {
> type Arguments TraceEventGetStateCommand
> return json.Marshal(&struct {
> Execute string `json:"execute"`
> Arguments *Arguments `json:"arguments"`
> }{
> Execute: "trace-event-get-state",
> Arguments: (*Arguments)(self),
> })
> }
>
> type TraceEventInfo struct {
> Name string `json:"name"`
> State string `json:"state"`
> }
>
> func (self *TraceEventInfo) UnmarshalJSON(data []byte) error {
> type Return TraceEventInfo
> ret := struct {
> Return Return `json:"return"`
> }{}
> err := json.Unmarshal(data, &ret)
> if err != nil {
> return err
> }
> self.Name = ret.Return.Name
> self.State = ret.Return.State
> return nil
> }
>
> func main() {
> var err error
> var qmp_input []byte
> var qmp_output []byte
>
> cmd := TraceEventGetStateCommand{
> Name: "qemu_memalign",
> }
> if qmp_input, err = json.Marshal(&cmd); err != nil {
> panic(err)
> }
> fmt.Printf(" cmd: %v\n", cmd)
> fmt.Printf("-> qmp_input: %v\n", string(qmp_input))
>
> qmp_output = ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
> ret := TraceEventInfo{}
> if err = json.Unmarshal(qmp_output, &ret); err != nil {
> panic(err)
> }
> fmt.Printf("<- qmp_output: %v\n", string(qmp_output))
> fmt.Printf(" ret: %v\n", ret)
> }
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-26 11:14 ` Markus Armbruster
@ 2022-05-09 10:52 ` Victor Toso
2022-05-10 8:53 ` Daniel P. Berrangé
1 sibling, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-05-09 10:52 UTC (permalink / raw)
To: Markus Armbruster
Cc: qemu-devel, Eric Blake, John Snow, Marc-André Lureau
[-- Attachment #1: Type: text/plain, Size: 6274 bytes --]
Hi,
Thanks for the quick review Markus. Sorry for taking quite a bit
of time to get back to you.
On Tue, Apr 26, 2022 at 01:14:28PM +0200, Markus Armbruster wrote:
> Victor Toso <victortoso@redhat.com> writes:
>
> > Hi,
> >
> > Happy 1st April. Not a joke :) /* ugh, took me too long to send */
> >
> > This series is about adding a generator in scripts/qapi to produce
> > Go data structures that can be used to communicate with QEMU over
> > QMP.
> >
> >
> > * Why Go?
> >
> > There are quite a few Go projects that interact with QEMU over QMP
> > and they endup using a mix of different libraries with their own
> > code.
> >
> >
> > ** Which projects?
> >
> > The ones I've found so far:
> >
> > - podman machine
> > https://github.com/containers/podman/tree/main/pkg/machine/qemu
> >
> > - kata-containers (govmm)
> > https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm
> >
> > - lxd
> > https://github.com/lxc/lxd/tree/master/lxd/instance/drivers
> >
> > - kubevirt (plain json strings)
> > https://github.com/kubevirt/kubevirt
> >
> > (let me know if you know others)
> >
> >
> > * But Why?
> >
> > I'm particularly interested in 3 out of 4 of the projects above and
> > only Kubevirt uses libvirt to handle QEMU. That means that every
> > QEMU releases where a QMP command, event or other data struct is
> > added, removed or changed, those projects need to check what changed
> > in QEMU and then address those changes in their projects, if needed.
> >
> > The idea behind generating Go data structures is that we can keep a
> > Go module which can have releases that follow QEMU releases.
>
> We need to look at "following the QEMU releases" a bit more
> closely.
>
> Merging your patches gives us the capability to generate a Go
> interface to HEAD's version of QMP.
Right, just to put it up here, it should be expected that the
qapi-go project is to have releases that match QEMU's release.
> The obvious way for an out-of-tree Go program to use this
> generated Go interface is to build with a specific version of
> it. It can then talk QMP to any compatible QEMU version.
>
> Compatibility with older QEMUs is not assured: stuff added
> since is present on the Go QMP client end, but not on the QEMU
> QMP server end.
>
> Compatibility with newer QEMUs is subject to our deprecation
> policy:
>
> In general features are intended to be supported
> indefinitely once introduced into QEMU. In the event that
> a feature needs to be removed, it will be listed in this
> section. The feature will remain functional for the
> release in which it was deprecated and one further release.
> After these two releases, the feature is liable to be
> removed.
>
> So, if you stay away from deprecated stuff, you're good for two
> more releases at least.
>
> Does this work for the projects you have in mind?
It depends on how the project will be using qapi-go, so I can't
say for them all.
There are projects that will be targeting specific QEMU version
(e.g: Kubevirt, Kata containers) and for those, I think they
don't mind only bumping qapi-go when they plan to change the
target QEMU version or perhaps to keep separated binary versions
for a limited amount of time (just my guess).
Some other projects like Podman, will likely be talking with the
running version of QEMU they have in that host. The possibilities
are quite broad here.
> Aside: graceful degradation in case of incompatibility seems
> desirable.
I agree. I haven't thought much on how to handle those scenarios
yet and suggestions are more than welcomed.
I know that, those projects are already hardcoding commands and
expected return types by hand. My first goal is to provide well
defined types, QAPI/QMP compliant, with easy to reach
documentation as provided by QAPI docs.
I expect that, step by step, we can improve things all around but
I don't expect it to be done all at once.
> > The project that uses this Go module, only need to bump the
> > module version and it shall receive all the changes in their
> > own vendored code base.
>
> Ideally, incompatible changes that affect the Go program show
> up as compile errors. Do they?
It depends. A field/type that they were using but is removed, for
sure a compile-time error.
What about a new mandatory field? If you run the unit tests in
qapi-go, there will be failures but it wouldn't be compile time
error. If user doesn't define a mandatory field in a Go struct,
the default value is used for the over-the-wire message.
Perhaps some tooling can be developed to help users check that
something they are using has changed. I'll look into it.
> > * Status
> >
> > There are a few rough edges to work on but this is usable. The major
> > thing I forgot to add is handling Error from Commands. It'll be the
> > first thing I'll work on next week.
> >
> > If you want to start using this Today you can fetch it in at
> >
> > https://gitlab.com/victortoso/qapi-go/
> >
> > There are quite a few tests that I took from the examples in the
> > qapi schema. Coverage using go's cover tool is giving `28.6% of
> > statements`
> >
> > I've uploaded the a static generated godoc output of the above Go
> > module here:
> >
> > https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
> >
> >
> > * License
> >
> > While the generator (golang.py in this series) is GPL v2, the
>
> I'd make it v2+, just to express my displeasure with the
> decision to make the initial QAPI generator v2 only for no good
> reason at all.
>
> > generated code needs to be compatible with other Golang projects,
> > such as the ones mentioned above. My intention is to keep a Go
> > module with a MIT license.
>
> Meh. Can't be helped, I guess.
:)
> > * Disclaimer to reviewers
> >
> > This is my first serious python project so there'll be lots of
> > suggetions that I'll be happy to take and learn from.
> >
> >
> > Thanks for taking a look, let me know if you have questions, ideas
> > or suggestions.
> >
> > Cheers,
> > Victor
Thanks again,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-02 14:01 ` Andrea Bolognani
2022-05-03 7:57 ` Markus Armbruster
@ 2022-05-09 18:53 ` Victor Toso
2022-05-10 8:06 ` Markus Armbruster
2022-05-10 9:52 ` Daniel P. Berrangé
2 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-09 18:53 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Markus Armbruster, qemu-devel, John Snow, Eric Blake, Kevin Wolf
[-- Attachment #1: Type: text/plain, Size: 6584 bytes --]
Hi,
On Mon, May 02, 2022 at 10:01:41AM -0400, Andrea Bolognani wrote:
> On Mon, May 02, 2022 at 01:46:23PM +0200, Markus Armbruster wrote:
> > Andrea Bolognani <abologna@redhat.com> writes:
> > >> > The wire protocol would still retain the unappealing
> > >> > name, but at least client libraries could hide the
> > >> > uglyness from users.
> > >>
> > >> At the price of mild inconsistency between the library
> > >> interface and QMP.
> > >
> > > That's fine, and in fact it already happens all the time
> > > when QAPI names (log-append) are translated to C
> > > identifiers (log_append).
> >
> > There's a difference between trivial translations like
> > "replace '-' by '_'" and arbitrary replacement like the one
> > for enumeration constants involving 'prefix'.
>
> Fair enough.
>
> I still feel that 1) users of a language SDK will ideally not
> need to look at the QAPI schema or wire chatter too often and
That should be the preference, yes.
> 2) even when that ends up being necessary, figuring out that
> LogAppend and logappend are the same thing is not going to be
> an unreasonable hurdle, especially when the status quo would be
> to work with Logappend instead.
If user really needs to leave their ecosystem in order to check
an alias, I hope we are already considering this a corner case.
Still, if we are thinking about multiple languages communicating
with QEMU, it is reasonable to consider the necessity of some
docs page where they can easily grep/search for the types, alias,
examples, etc. IMHO, this should be a long term goal and not a
blocker... I can volunteer on working on that later.
> > > The point is that, if we want to provide a language
> > > interface that feels natural, we need a way to mark parts
> > > of a QAPI symbol's name in a way that makes it possible for
> > > the generator to know they're acronyms and treat them in an
> > > appropriate, language-specific manner.
> >
> > It's not just acronyms. Consider IAmALittleTeapot. If you
> > can assume that only beginning of words are capitalized, even
> > for acronyms, you can split this into words without trouble.
> > You can't recover correct case, though: "i am a little
> > teapot" is wrong.
>
> Is there any scenario in which we would care though? We're in
> the business of translating identifiers from one machine
> representation to another, so once it has been split up
> correctly into the words that compose it (which in your example
> above it has) then we don't really care about anything else
> unless acronyms are involved.
>
> In other words, we can obtain the list of words "i am a little
> teapot" programmatically both from IAmALittleTeapot and
> i-am-a-little-teapot, and in both cases we can then generate
> IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or
> i_am_a_little_teapot or whatever is appropriate for the context
> and target language, but the fact that in a proper English
> sentence "I" would have to be capitalized doesn't really enter
> the picture.
>
> > "Split before capital letter" falls apart when you have
> > characters that cannot be capitalized: Point3d.
> >
> > Camel case is hopeless.
>
> I would argue that it works quite well for most scenarios, but
> there are some corner cases where it's clearly not good enough.
> If we can define a way to clue in the generator about "Point3d"
> having to be interpreted as "point 3d" and "VNCProps" as "vnc
> props", then we are golden. That wouldn't be necessary for
> simple cases that are already handled correctly.
>
> A more radical idea would be to start using dash-notation for
> types too. That'd remove the word splitting issue altogether,
> at the cost of the schema being (possibly) harder to read and
> more distanced from the generated code.
>
> You'd still only be able to generate VncProps from vnc-props
> though.
>
> > > The obvious way to implement this would be with an
> > > annotation along the lines of the one I proposed. Other
> > > ideas?
> >
> > I'm afraid having the schema spell out names in multiple
> > naming conventions could be onerous. How many names will
> > need it?
>
> I don't have hard data on this. I could try extracting it, but
> that might end up being a bigger job than I had anticipated.
The only way to know is by checking /o\
I'll give it a shot.
> My guess is that the number of cases where the naive algorithm
> can't split words correctly is relatively small compared to the
> size of the entire QAPI schema. Fair warning: I have made
> incorrect guesses in the past ;)
>
> > Times how many naming conventions?
>
> Yeah, I don't think requiring all possible permutations to be
> spelled out in the schema is the way to go. That's exactly why
> my proposal was to offer a way to inject the semantic
> information that the parser can't figure out itself.
>
> Once you have a way to inform the generator that "VNCProps" is
> made of the two words "vnc" and "props", and that "vnc" is an
> acronym, then it can generate an identifier appropriate for the
> target language without having to spell out anywhere that such
> an identifier would be VNCProps for Go and VncProps for Rust.
>
> By the way, while looking around I realized that we also have
> to take into account things like D-Bus: the QAPI type
> ChardevDBus, for example, would probably translate verbatim to
> Go but have to be changed to ChardevDbus for Rust. Fun :)
>
> Revised proposal for the annotation:
>
> ns:word-WORD-WoRD-123Word
I really like it.
> Words are always separated by dashes; "regular" words are
> entirely lowercase, while the presence of even a single
> uppercase letter in a word denotes the fact that its case
> should be preserved when the naming conventions of the target
> language allow that.
>
> > Another issue: the fancier the translation from schema name
> > to language-specific name gets, the harder it becomes to find
> > one from the other.
>
> That's true, but at least to me the trade-off feels reasonable.
I don't quite get the argument why it gets harder to find. We can
simply provide the actual name as reference in the generated
documentation, no?
As Kevin already pointed out that he is not planning to work on
the Alias (reference below), any other idea besides the Andrea's
annotation suggestion? While this is not a blocker, I agree it
would be nice to be consistent.
https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-09 18:53 ` Victor Toso
@ 2022-05-10 8:06 ` Markus Armbruster
2022-05-10 11:48 ` Victor Toso
0 siblings, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-05-10 8:06 UTC (permalink / raw)
To: Victor Toso
Cc: Andrea Bolognani, qemu-devel, John Snow, Eric Blake, Kevin Wolf
Victor Toso <victortoso@redhat.com> writes:
> Hi,
>
> On Mon, May 02, 2022 at 10:01:41AM -0400, Andrea Bolognani wrote:
>> On Mon, May 02, 2022 at 01:46:23PM +0200, Markus Armbruster wrote:
>> > Andrea Bolognani <abologna@redhat.com> writes:
>> > >> > The wire protocol would still retain the unappealing
>> > >> > name, but at least client libraries could hide the
>> > >> > uglyness from users.
>> > >>
>> > >> At the price of mild inconsistency between the library
>> > >> interface and QMP.
>> > >
>> > > That's fine, and in fact it already happens all the time
>> > > when QAPI names (log-append) are translated to C
>> > > identifiers (log_append).
>> >
>> > There's a difference between trivial translations like
>> > "replace '-' by '_'" and arbitrary replacement like the one
>> > for enumeration constants involving 'prefix'.
>>
>> Fair enough.
>>
>> I still feel that 1) users of a language SDK will ideally not
>> need to look at the QAPI schema or wire chatter too often and
>
> That should be the preference, yes.
>
>> 2) even when that ends up being necessary, figuring out that
>> LogAppend and logappend are the same thing is not going to be
>> an unreasonable hurdle, especially when the status quo would be
>> to work with Logappend instead.
>
> If user really needs to leave their ecosystem in order to check
> an alias, I hope we are already considering this a corner case.
> Still, if we are thinking about multiple languages communicating
> with QEMU, it is reasonable to consider the necessity of some
> docs page where they can easily grep/search for the types, alias,
> examples, etc. IMHO, this should be a long term goal and not a
> blocker... I can volunteer on working on that later.
>
>> > > The point is that, if we want to provide a language
>> > > interface that feels natural, we need a way to mark parts
>> > > of a QAPI symbol's name in a way that makes it possible for
>> > > the generator to know they're acronyms and treat them in an
>> > > appropriate, language-specific manner.
>> >
>> > It's not just acronyms. Consider IAmALittleTeapot. If you
>> > can assume that only beginning of words are capitalized, even
>> > for acronyms, you can split this into words without trouble.
>> > You can't recover correct case, though: "i am a little
>> > teapot" is wrong.
>>
>> Is there any scenario in which we would care though? We're in
>> the business of translating identifiers from one machine
>> representation to another, so once it has been split up
>> correctly into the words that compose it (which in your example
>> above it has) then we don't really care about anything else
>> unless acronyms are involved.
>>
>> In other words, we can obtain the list of words "i am a little
>> teapot" programmatically both from IAmALittleTeapot and
>> i-am-a-little-teapot, and in both cases we can then generate
>> IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or
>> i_am_a_little_teapot or whatever is appropriate for the context
>> and target language, but the fact that in a proper English
>> sentence "I" would have to be capitalized doesn't really enter
>> the picture.
>>
>> > "Split before capital letter" falls apart when you have
>> > characters that cannot be capitalized: Point3d.
>> >
>> > Camel case is hopeless.
>>
>> I would argue that it works quite well for most scenarios, but
>> there are some corner cases where it's clearly not good enough.
>> If we can define a way to clue in the generator about "Point3d"
>> having to be interpreted as "point 3d" and "VNCProps" as "vnc
>> props", then we are golden. That wouldn't be necessary for
>> simple cases that are already handled correctly.
>>
>> A more radical idea would be to start using dash-notation for
>> types too. That'd remove the word splitting issue altogether,
>> at the cost of the schema being (possibly) harder to read and
>> more distanced from the generated code.
>>
>> You'd still only be able to generate VncProps from vnc-props
>> though.
>>
>> > > The obvious way to implement this would be with an
>> > > annotation along the lines of the one I proposed. Other
>> > > ideas?
>> >
>> > I'm afraid having the schema spell out names in multiple
>> > naming conventions could be onerous. How many names will
>> > need it?
>>
>> I don't have hard data on this. I could try extracting it, but
>> that might end up being a bigger job than I had anticipated.
>
> The only way to know is by checking /o\
> I'll give it a shot.
I append a quick hack to find names in a QAPI schema, and sort them into
buckets "lc" (lower-with-hyphen), "uc" (UPPER_WITH_UNDERSCORE), "cc"
(CamelCase where words are obvious), and "mc" (everything else, mostly
CamelCase where words aren't obvious).
Note that this ignores naming rule violations such as upper case enum
member EAX.
>> My guess is that the number of cases where the naive algorithm
>> can't split words correctly is relatively small compared to the
>> size of the entire QAPI schema. Fair warning: I have made
>> incorrect guesses in the past ;)
>>
>> > Times how many naming conventions?
>>
>> Yeah, I don't think requiring all possible permutations to be
>> spelled out in the schema is the way to go. That's exactly why
>> my proposal was to offer a way to inject the semantic
>> information that the parser can't figure out itself.
>>
>> Once you have a way to inform the generator that "VNCProps" is
>> made of the two words "vnc" and "props", and that "vnc" is an
>> acronym, then it can generate an identifier appropriate for the
>> target language without having to spell out anywhere that such
>> an identifier would be VNCProps for Go and VncProps for Rust.
>>
>> By the way, while looking around I realized that we also have
>> to take into account things like D-Bus: the QAPI type
>> ChardevDBus, for example, would probably translate verbatim to
>> Go but have to be changed to ChardevDbus for Rust. Fun :)
>>
>> Revised proposal for the annotation:
>>
>> ns:word-WORD-WoRD-123Word
>
> I really like it.
>
>> Words are always separated by dashes; "regular" words are
>> entirely lowercase, while the presence of even a single
>> uppercase letter in a word denotes the fact that its case
>> should be preserved when the naming conventions of the target
>> language allow that.
>>
>> > Another issue: the fancier the translation from schema name
>> > to language-specific name gets, the harder it becomes to find
>> > one from the other.
>>
>> That's true, but at least to me the trade-off feels reasonable.
>
> I don't quite get the argument why it gets harder to find. We can
> simply provide the actual name as reference in the generated
> documentation, no?
Predictable name transformation can save me detours through
documentation. Being able to guess the Go bindings just from the QMP
Reference Manual can be nice: it lets me write Go code with just the QMP
Reference manual in view. Being able to guess the name in the QAPI
schema from the name in the Go bindings can also be nice: it lets me
look it up in the QMP Reference manual without jumping through the
bindings documentation.
Or do you envisage a Go bindings manual that fully replaces the QMP
Reference Manual for developers writing Go?
> As Kevin already pointed out that he is not planning to work on
> the Alias (reference below), any other idea besides the Andrea's
> annotation suggestion? While this is not a blocker, I agree it
> would be nice to be consistent.
>
> https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html
Let's first try to get a handle on how many schema names are
problematic.
$ git-diff
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index 5a1782b57e..804b8ab455 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -94,6 +94,7 @@ def check_name_str(name: str, info: QAPISourceInfo, source: str) -> str:
"""
# Reserve the entire 'q_' namespace for c_name(), and for 'q_empty'
# and 'q_obj_*' implicit type names.
+ print("###", name)
match = valid_name.match(name)
if not match or c_name(name, False).startswith('q_'):
raise QAPISemError(info, "%s has an invalid name" % source)
$ cd bld
$ python3 ../scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sort
cc Abort
cc AbortWrapper
cc AcpiTableOptions
cc ActionCompletionMode
cc AddfdInfo
cc AnnounceParameters
cc AudioFormat
cc Audiodev
cc AudiodevAlsaOptions
cc AudiodevAlsaPerDirectionOptions
cc AudiodevCoreaudioOptions
cc AudiodevCoreaudioPerDirectionOptions
cc AudiodevDriver
cc AudiodevDsoundOptions
cc AudiodevGenericOptions
cc AudiodevJackOptions
cc AudiodevJackPerDirectionOptions
cc AudiodevOssOptions
cc AudiodevOssPerDirectionOptions
cc AudiodevPaOptions
cc AudiodevPaPerDirectionOptions
cc AudiodevPerDirectionOptions
cc AudiodevSdlOptions
cc AudiodevSdlPerDirectionOptions
cc AudiodevWavOptions
cc AuthZListFileProperties
cc AuthZListProperties
cc AuthZPAMProperties
cc AuthZSimpleProperties
cc BackupCommon
cc BackupPerf
cc BalloonInfo
cc BiosAtaTranslation
cc BitmapMigrationBitmapAlias
cc BitmapMigrationBitmapAliasTransform
cc BitmapMigrationNodeAlias
cc BitmapSyncMode
cc BlkdebugEvent
cc BlkdebugIOType
cc BlkdebugInjectErrorOptions
cc BlkdebugSetStateOptions
cc BlockDeviceInfo
cc BlockDeviceIoStatus
cc BlockDeviceStats
cc BlockDeviceTimedStats
cc BlockDirtyBitmap
cc BlockDirtyBitmapAdd
cc BlockDirtyBitmapAddWrapper
cc BlockDirtyBitmapMerge
cc BlockDirtyBitmapMergeWrapper
cc BlockDirtyBitmapOrStr
cc BlockDirtyBitmapSha256
cc BlockDirtyBitmapWrapper
cc BlockDirtyInfo
cc BlockErrorAction
cc BlockExportInfo
cc BlockExportOptions
cc BlockExportOptionsFuse
cc BlockExportOptionsNbd
cc BlockExportOptionsNbdBase
cc BlockExportOptionsVhostUserBlk
cc BlockExportRemoveMode
cc BlockExportType
cc BlockIOThrottle
cc BlockInfo
cc BlockJobInfo
cc BlockLatencyHistogramInfo
cc BlockMeasureInfo
cc BlockPermission
cc BlockStats
cc BlockStatsSpecific
cc BlockStatsSpecificFile
cc BlockStatsSpecificNvme
cc BlockdevAioOptions
cc BlockdevAmendOptions
cc BlockdevAmendOptionsLUKS
cc BlockdevAmendOptionsQcow2
cc BlockdevBackup
cc BlockdevBackupWrapper
cc BlockdevCacheInfo
cc BlockdevCacheOptions
cc BlockdevChangeReadOnlyMode
cc BlockdevCreateOptions
cc BlockdevCreateOptionsFile
cc BlockdevCreateOptionsGluster
cc BlockdevCreateOptionsLUKS
cc BlockdevCreateOptionsNfs
cc BlockdevCreateOptionsParallels
cc BlockdevCreateOptionsQcow
cc BlockdevCreateOptionsQcow2
cc BlockdevCreateOptionsQed
cc BlockdevCreateOptionsRbd
cc BlockdevCreateOptionsSsh
cc BlockdevCreateOptionsVdi
cc BlockdevCreateOptionsVhdx
cc BlockdevCreateOptionsVmdk
cc BlockdevCreateOptionsVpc
cc BlockdevDetectZeroesOptions
cc BlockdevDiscardOptions
cc BlockdevDriver
cc BlockdevOnError
cc BlockdevOptions
cc BlockdevOptionsBlkdebug
cc BlockdevOptionsBlklogwrites
cc BlockdevOptionsBlkreplay
cc BlockdevOptionsBlkverify
cc BlockdevOptionsCbw
cc BlockdevOptionsCor
cc BlockdevOptionsCurlBase
cc BlockdevOptionsCurlFtp
cc BlockdevOptionsCurlFtps
cc BlockdevOptionsCurlHttp
cc BlockdevOptionsCurlHttps
cc BlockdevOptionsFile
cc BlockdevOptionsGenericCOWFormat
cc BlockdevOptionsGenericFormat
cc BlockdevOptionsGluster
cc BlockdevOptionsIscsi
cc BlockdevOptionsLUKS
cc BlockdevOptionsNVMe
cc BlockdevOptionsNbd
cc BlockdevOptionsNfs
cc BlockdevOptionsNull
cc BlockdevOptionsPreallocate
cc BlockdevOptionsQcow
cc BlockdevOptionsQcow2
cc BlockdevOptionsQuorum
cc BlockdevOptionsRaw
cc BlockdevOptionsRbd
cc BlockdevOptionsReplication
cc BlockdevOptionsSsh
cc BlockdevOptionsThrottle
cc BlockdevOptionsVVFAT
cc BlockdevQcow2Encryption
cc BlockdevQcow2EncryptionFormat
cc BlockdevQcow2Version
cc BlockdevQcowEncryption
cc BlockdevQcowEncryptionFormat
cc BlockdevRef
cc BlockdevRefOrNull
cc BlockdevSnapshot
cc BlockdevSnapshotInternal
cc BlockdevSnapshotInternalWrapper
cc BlockdevSnapshotSync
cc BlockdevSnapshotSyncWrapper
cc BlockdevSnapshotWrapper
cc BlockdevVhdxSubformat
cc BlockdevVmdkAdapterType
cc BlockdevVmdkSubformat
cc BlockdevVpcSubformat
cc CanHostSocketcanProperties
cc ChardevBackend
cc ChardevBackendInfo
cc ChardevBackendKind
cc ChardevCommon
cc ChardevCommonWrapper
cc ChardevDBus
cc ChardevDBusWrapper
cc ChardevFile
cc ChardevFileWrapper
cc ChardevHostdev
cc ChardevHostdevWrapper
cc ChardevInfo
cc ChardevMux
cc ChardevMuxWrapper
cc ChardevQemuVDAgent
cc ChardevQemuVDAgentWrapper
cc ChardevReturn
cc ChardevRingbuf
cc ChardevRingbufWrapper
cc ChardevSocket
cc ChardevSocketWrapper
cc ChardevSpiceChannel
cc ChardevSpiceChannelWrapper
cc ChardevSpicePort
cc ChardevSpicePortWrapper
cc ChardevStdio
cc ChardevStdioWrapper
cc ChardevUdp
cc ChardevUdpWrapper
cc ChardevVC
cc ChardevVCWrapper
cc ColoCompareProperties
cc CommandInfo
cc CommandLineOptionInfo
cc CommandLineParameterInfo
cc CommandLineParameterType
cc CommandNotFound
cc CompatPolicy
cc CompatPolicyInput
cc CompatPolicyOutput
cc CompressionStats
cc CpuDefinitionInfo
cc CpuInfoFast
cc CpuInfoS390
cc CpuInstanceProperties
cc CpuModelBaselineInfo
cc CpuModelCompareInfo
cc CpuModelCompareResult
cc CpuModelExpansionInfo
cc CpuModelExpansionType
cc CpuModelInfo
cc CpuS390State
cc CryptodevBackendProperties
cc CryptodevVhostUserProperties
cc CurrentMachineParams
cc DataFormat
cc DeviceNotActive
cc DeviceNotFound
cc DirtyRateInfo
cc DirtyRateMeasureMode
cc DirtyRateStatus
cc DirtyRateVcpu
cc DisplayCocoa
cc DisplayCurses
cc DisplayDBus
cc DisplayEGLHeadless
cc DisplayGLMode
cc DisplayGTK
cc DisplayOptions
cc DisplayProtocol
cc DisplayReloadOptions
cc DisplayReloadOptionsVNC
cc DisplayReloadType
cc DisplayType
cc DisplayUpdateOptions
cc DisplayUpdateOptionsVNC
cc DisplayUpdateType
cc DriveBackup
cc DriveBackupWrapper
cc DriveMirror
cc DummyForceArrays
cc DumpGuestMemoryCapability
cc DumpGuestMemoryFormat
cc DumpQueryResult
cc DumpStatus
cc EventLoopBaseProperties
cc ExpirePasswordOptions
cc ExpirePasswordOptionsVnc
cc FailoverStatus
cc FdsetFdInfo
cc FdsetInfo
cc FilterBufferProperties
cc FilterDumpProperties
cc FilterMirrorProperties
cc FilterRedirectorProperties
cc FilterRewriterProperties
cc FloppyDriveType
cc FuseExportAllowOther
cc GenericError
cc GrabToggleKeys
cc GuestPanicAction
cc GuestPanicInformation
cc GuestPanicInformationHyperV
cc GuestPanicInformationS390
cc GuestPanicInformationType
cc GuidInfo
cc HmatCacheAssociativity
cc HmatCacheWritePolicy
cc HmatLBDataType
cc HmatLBMemoryHierarchy
cc HostMemPolicy
cc HotpluggableCPU
cc HumanReadableText
cc ImageCheck
cc ImageFormat
cc ImageInfo
cc ImageInfoSpecific
cc ImageInfoSpecificKind
cc ImageInfoSpecificLUKSWrapper
cc ImageInfoSpecificQCow2
cc ImageInfoSpecificQCow2Encryption
cc ImageInfoSpecificQCow2EncryptionBase
cc ImageInfoSpecificQCow2Wrapper
cc ImageInfoSpecificRbd
cc ImageInfoSpecificRbdWrapper
cc ImageInfoSpecificVmdk
cc ImageInfoSpecificVmdkWrapper
cc InetSocketAddress
cc InetSocketAddressBase
cc InetSocketAddressWrapper
cc InputAxis
cc InputBarrierProperties
cc InputBtnEvent
cc InputBtnEventWrapper
cc InputButton
cc InputEvent
cc InputEventKind
cc InputKeyEvent
cc InputKeyEventWrapper
cc InputLinuxProperties
cc InputMoveEvent
cc InputMoveEventWrapper
cc IntWrapper
cc IoOperationType
cc IothreadProperties
cc IscsiHeaderDigest
cc IscsiTransport
cc JobInfo
cc JobStatus
cc JobType
cc JobVerb
cc KeyValue
cc KeyValueKind
cc KvmInfo
cc LostTickPolicy
cc MachineInfo
cc MainLoopProperties
cc MapEntry
cc Memdev
cc MemoryBackendEpcProperties
cc MemoryBackendFileProperties
cc MemoryBackendMemfdProperties
cc MemoryBackendProperties
cc MemoryDeviceInfo
cc MemoryDeviceInfoKind
cc MemoryFailureAction
cc MemoryFailureFlags
cc MemoryFailureRecipient
cc MemoryInfo
cc MigrateSetParameters
cc MigrationCapability
cc MigrationCapabilityStatus
cc MigrationInfo
cc MigrationParameter
cc MigrationParameters
cc MigrationStats
cc MigrationStatus
cc MirrorCopyMode
cc MirrorSyncMode
cc MonitorMode
cc MonitorOptions
cc MouseInfo
cc MultiFDCompression
cc NameInfo
cc NbdServerAddOptions
cc NbdServerOptions
cc NetClientDriver
cc NetFilterDirection
cc NetLegacyNicOptions
cc Netdev
cc NetdevBridgeOptions
cc NetdevHubPortOptions
cc NetdevL2TPv3Options
cc NetdevNetmapOptions
cc NetdevSocketOptions
cc NetdevTapOptions
cc NetdevUserOptions
cc NetdevVdeOptions
cc NetdevVhostUserOptions
cc NetdevVhostVDPAOptions
cc NetfilterInsert
cc NetfilterProperties
cc NetworkAddressFamily
cc NewImageMode
cc NumaCpuOptions
cc NumaDistOptions
cc NumaHmatCacheOptions
cc NumaHmatLBOptions
cc NumaNodeOptions
cc NumaOptions
cc NumaOptionsType
cc ObjectOptions
cc ObjectPropertyInfo
cc ObjectType
cc ObjectTypeInfo
cc OffAutoPCIBAR
cc OnOffAuto
cc OnOffSplit
cc PanicAction
cc PciBridgeInfo
cc PciBusInfo
cc PciDeviceClass
cc PciDeviceId
cc PciDeviceInfo
cc PciInfo
cc PciMemoryRange
cc PciMemoryRegion
cc PrManagerHelperProperties
cc PreallocMode
cc QapiErrorClass
cc Qcow2BitmapInfo
cc Qcow2BitmapInfoFlags
cc Qcow2CompressionType
cc Qcow2OverlapCheckFlags
cc Qcow2OverlapCheckMode
cc Qcow2OverlapChecks
cc QtestProperties
cc QuorumOpType
cc QuorumReadPattern
cc RbdAuthMode
cc RbdEncryptionCreateOptions
cc RbdEncryptionCreateOptionsLUKS
cc RbdEncryptionCreateOptionsLUKS2
cc RbdEncryptionCreateOptionsLUKSBase
cc RbdEncryptionOptions
cc RbdEncryptionOptionsLUKS
cc RbdEncryptionOptionsLUKS2
cc RbdEncryptionOptionsLUKSBase
cc RbdImageEncryptionFormat
cc RebootAction
cc RemoteObjectProperties
cc ReplayInfo
cc ReplayMode
cc ReplicationMode
cc ReplicationStatus
cc RngEgdProperties
cc RngProperties
cc RngRandomProperties
cc RockerOfDpaFlow
cc RockerOfDpaFlowAction
cc RockerOfDpaFlowKey
cc RockerOfDpaFlowMask
cc RockerOfDpaGroup
cc RockerPort
cc RockerPortAutoneg
cc RockerPortDuplex
cc RockerSwitch
cc RunState
cc RxFilterInfo
cc RxState
cc SchemaInfo
cc SchemaInfoAlternate
cc SchemaInfoAlternateMember
cc SchemaInfoArray
cc SchemaInfoBuiltin
cc SchemaInfoCommand
cc SchemaInfoEnum
cc SchemaInfoEnumMember
cc SchemaInfoEvent
cc SchemaInfoObject
cc SchemaInfoObjectMember
cc SchemaInfoObjectVariant
cc SchemaMetaType
cc SecretCommonProperties
cc SecretKeyringProperties
cc SecretProperties
cc SetPasswordAction
cc SetPasswordOptions
cc SetPasswordOptionsVnc
cc SevAttestationReport
cc SevCapability
cc SevGuestProperties
cc SevInfo
cc SevLaunchMeasureInfo
cc SevState
cc SgxEPC
cc SgxEPCDeviceInfo
cc SgxEPCDeviceInfoWrapper
cc SgxEPCProperties
cc ShutdownAction
cc ShutdownCause
cc SmbiosEntryPointType
cc SnapshotInfo
cc SocketAddress
cc SocketAddressLegacy
cc SocketAddressType
cc SpiceBasicInfo
cc SpiceChannel
cc SpiceInfo
cc SpiceQueryMouseMode
cc SpiceServerInfo
cc SshHostKeyCheck
cc SshHostKeyCheckHashType
cc SshHostKeyCheckMode
cc SshHostKeyHash
cc StatusInfo
cc StrOrNull
cc String
cc StringWrapper
cc SysEmuTarget
cc TargetInfo
cc ThrottleGroupProperties
cc ThrottleLimits
cc TlsCredsAnonProperties
cc TlsCredsProperties
cc TlsCredsPskProperties
cc TlsCredsX509Properties
cc TpmModel
cc TpmType
cc TpmTypeOptions
cc TraceEventInfo
cc TraceEventState
cc TransactionAction
cc TransactionActionKind
cc TransactionProperties
cc UnixSocketAddress
cc UnixSocketAddressWrapper
cc UuidInfo
cc VersionInfo
cc VersionTriple
cc VfioStats
cc VirtioMEMDeviceInfo
cc VirtioMEMDeviceInfoWrapper
cc VirtioPMEMDeviceInfo
cc VirtioPMEMDeviceInfoWrapper
cc VncBasicInfo
cc VncClientInfo
cc VncInfo
cc VncInfo2
cc VncPrimaryAuth
cc VncServerInfo
cc VncServerInfo2
cc VncVencryptSubAuth
cc VsockSocketAddress
cc VsockSocketAddressWrapper
cc WatchdogAction
cc YankInstance
cc YankInstanceBlockNode
cc YankInstanceChardev
cc YankInstanceType
lc a
lc aarch64
lc abort
lc aborting
lc abs
lc absolute
lc absolute-paths
lc abstract
lc accept
lc access-bandwidth
lc access-latency
lc action
lc action-required
lc actions
lc active
lc active-l1
lc active-l2
lc actual
lc actual-size
lc adapter-type
lc add-fd
lc addr
lc address
lc addresses
lc aes
lc aes-128
lc aes-192
lc aes-256
lc again
lc aio
lc aio-max-batch
lc alias
lc alias-of
lc align
lc aligned-accesses
lc all
lc allocated-clusters
lc allocation-depth
lc allow
lc allow-oob
lc allow-other
lc allow-write-only-overlay
lc alpha
lc alsa
lc alt
lc alt-alt
lc alternate
lc always
lc amend
lc amount-exceeded
lc announce-initial
lc announce-max
lc announce-rounds
lc announce-self
lc announce-step
lc api-major
lc api-minor
lc apostrophe
lc append
lc arch
lc arg-type
lc arg1
lc arg2
lc arg3
lc arg4
lc arg5
lc arm
lc array
lc associativity
lc asterisk
lc audiodev
lc audiomute
lc audionext
lc audioplay
lc audioprev
lc audiostop
lc auth
lc auth-client-required
lc authz-list
lc authz-listfile
lc authz-pam
lc authz-simple
lc auto
lc auto-converge
lc auto-dismiss
lc auto-finalize
lc auto-read-only
lc autoneg
lc avr
lc axis
lc b
lc backend
lc background
lc background-snapshot
lc backing
lc backing-file
lc backing-filename
lc backing-filename-format
lc backing-fmt
lc backing-image
lc backslash
lc backspace
lc backup
lc balloon
lc bandwidth
lc bar
lc bar0
lc bar1
lc bar2
lc bar3
lc bar4
lc bar5
lc base
lc base-memory
lc base-node
lc base64
lc before
lc begin
lc behind
lc bind
lc bins
lc bitmap
lc bitmap-directory
lc bitmap-mode
lc bitmaps
lc blk
lc blkdebug
lc blklogwrites
lc blkreplay
lc blkverify
lc block
lc block-backend
lc block-bitmap-mapping
lc block-commit
lc block-dirty-bitmap-add
lc block-dirty-bitmap-clear
lc block-dirty-bitmap-disable
lc block-dirty-bitmap-enable
lc block-dirty-bitmap-merge
lc block-dirty-bitmap-remove
lc block-driver
lc block-export-add
lc block-export-del
lc block-incremental
lc block-job
lc block-job-cancel
lc block-job-complete
lc block-job-dismiss
lc block-job-finalize
lc block-job-pause
lc block-job-resume
lc block-job-set-speed
lc block-latency-histogram-set
lc block-node
lc block-set-write-threshold
lc block-size
lc block-state-zero
lc block-status
lc block-stream
lc blockdev-add
lc blockdev-backup
lc blockdev-change-medium
lc blockdev-close-tray
lc blockdev-create
lc blockdev-del
lc blockdev-insert-medium
lc blockdev-mirror
lc blockdev-open-tray
lc blockdev-remove-medium
lc blockdev-reopen
lc blockdev-snapshot
lc blockdev-snapshot-delete-internal-sync
lc blockdev-snapshot-internal-sync
lc blockdev-snapshot-sync
lc blocked-reasons
lc bochs
lc boolean
lc bootfile
lc bottom
lc boundaries
lc boundaries-flush
lc boundaries-read
lc boundaries-write
lc bps
lc bps-read
lc bps-read-max
lc bps-read-max-length
lc bps-total
lc bps-total-max
lc bps-total-max-length
lc bps-write
lc bps-write-max
lc bps-write-max-length
lc br
lc braille
lc bridge
lc broadcast-allowed
lc btn
lc buf-size
lc buffer-count
lc buffer-length
lc build-id
lc builtin
lc bus
lc buslogic
lc busy
lc busy-rate
lc button
lc bytes
lc c
lc cache
lc cache-clean-interval
lc cache-miss
lc cache-miss-rate
lc cache-size
lc cached
lc calc-dirty-rate
lc calc-time
lc calculator
lc can-bus
lc can-host-socketcan
lc canbus
lc cancel
lc cancel-path
lc cancelled
lc cancelling
lc capabilities
lc capability
lc case
lc cast5-128
lc cbc
lc cbitpos
lc cephx
lc cert-chain
lc cert-subject
lc change-backing-file
lc change-vnc-password
lc channel-id
lc channel-type
lc channels
lc chardev
lc chardev-add
lc chardev-change
lc chardev-remove
lc chardev-send-break
lc charset
lc check-errors
lc check-stop
lc checkpoint-ready
lc checkpoint-reply
lc checkpoint-request
lc child
lc children
lc cid
lc cipher-alg
lc cipher-mode
lc class
lc client
lc client-name
lc clients
lc clipboard
lc cloop
lc closefd
lc cluster-id
lc cluster-size
lc clusters
lc cocoa
lc colo
lc colo-compare
lc cols
lc comma
lc command
lc command-line
lc commit
lc compat
lc compiled-version
lc complete
lc completed
lc completion-errors
lc completion-mode
lc complex
lc compose
lc compress
lc compress-level
lc compress-threads
lc compress-wait-thread
lc compressed
lc compressed-clusters
lc compressed-size
lc compression
lc compression-rate
lc compression-type
lc computer
lc concluded
lc conf
lc config
lc connect
lc connect-ports
lc connected
lc connection-id
lc consistent-read
lc console
lc constant
lc cont
lc control
lc cookie
lc cookie-secret
lc cookie64
lc copy
lc copy-before-write
lc copy-mode
lc copy-on-read
lc core
lc core-id
lc coreaudio
lc cores
lc corrupt
lc corruptions
lc corruptions-fixed
lc count
lc counter
lc cpu
lc cpu-index
lc cpu-max
lc cpu-state
lc cpu-throttle-increment
lc cpu-throttle-initial
lc cpu-throttle-percentage
lc cpu-throttle-tailslow
lc cpu0-id
lc cpuid-input-eax
lc cpuid-input-ecx
lc cpuid-register
lc cpus
lc crash
lc crc32c
lc crc32c-none
lc create
lc create-type
lc created
lc cris
lc cryptodev-backend
lc cryptodev-backend-builtin
lc cryptodev-vhost-user
lc ctr
lc ctrl
lc ctrl-ctrl
lc ctrl-scrolllock
lc current
lc current-progress
lc curses
lc cut
lc d
lc d0
lc d1
lc d12
lc d120
lc d144
lc d16
lc d2
lc d288
lc d3
lc d32
lc d3des
lc d4
lc d5
lc d6
lc d64
lc d7
lc d8
lc d9
lc data
lc data-file
lc data-file-raw
lc data-type
lc date-nsec
lc date-sec
lc dbus
lc dbus-vmstate
lc debug
lc decompress-threads
lc default
lc default-cpu-type
lc default-ram-id
lc default-value
lc definition
lc delay
lc delete
lc deny
lc deprecated
lc deprecated-input
lc deprecated-output
lc depth
lc des
lc desc
lc description
lc detach
lc detect-zeroes
lc dev
lc device
lc device-id
lc device-list-properties
lc devices
lc devid
lc devname
lc dh-cert-file
lc dhcpstart
lc die-id
lc dies
lc dimm
lc dir
lc direct
lc dirty-bitmap
lc dirty-bitmaps
lc dirty-flag
lc dirty-pages-rate
lc dirty-rate
lc dirty-ring
lc dirty-sync-count
lc disabled
lc disabled-wait
lc discard
lc discard-bytes-ok
lc discard-data
lc discard-nb-failed
lc discard-nb-ok
lc disconnect
lc disk
lc dismiss
lc display
lc display-reload
lc display-update
lc dist
lc dmg
lc dns
lc dnssearch
lc domainname
lc dot
lc down
lc downscript
lc downtime
lc downtime-bytes
lc downtime-limit
lc drive-backup
lc drive-mirror
lc driver
lc driver-specific
lc drop-cache
lc drv
lc dsound
lc dsp-policy
lc dst
lc dstport
lc dump
lc dump-guest-memory
lc dump-skeys
lc duplex
lc duplicate
lc dynamic
lc dynamic-auto-read-only
lc e
lc ecb
lc edges
lc egl-headless
lc eject
lc element-type
lc elf
lc emulated
lc emulator
lc enable
lc enabled
lc encoding-rate
lc encrypt
lc encrypted
lc encryption-format
lc end
lc endpoint
lc enospc
lc enum
lc equal
lc errno
lc error
lc error-desc
lc es
lc esc
lc essiv
lc eth-dst
lc eth-src
lc eth-type
lc evdev
lc event
lc events
lc exact
lc exact-name
lc exclusive
lc existing
lc expected-downtime
lc export
lc extended-l2
lc extent-size-hint
lc extents
lc external
lc extint-loop
lc extra
lc f
lc f1
lc f10
lc f11
lc f12
lc f2
lc f3
lc f32
lc f4
lc f5
lc f6
lc f7
lc f8
lc f9
lc fail
lc failed
lc failover
lc falloc
lc family
lc fat-type
lc fatal
lc fd
lc fdname
lc fds
lc fdset-id
lc features
lc fifo
lc file
lc filename
lc filter-buffer
lc filter-dump
lc filter-mirror
lc filter-node-name
lc filter-redirector
lc filter-replay
lc filter-rewriter
lc finalize
lc find
lc finish-migrate
lc first-level
lc fixed
lc fixed-iothread
lc fixed-settings
lc flags
lc flat
lc flc
lc floppy
lc flush
lc force
lc force-share
lc force-size
lc format
lc format-specific
lc formats
lc fqdn
lc fragmented-clusters
lc frequency
lc front
lc frontend-open
lc ftp
lc ftps
lc full
lc full-backing-filename
lc full-grab
lc full-screen
lc fully-allocated
lc function
lc fuse
lc g
lc getfd
lc gid-status
lc gl
lc glob
lc gluster
lc goto-tbl
lc gpa
lc grab-on-hover
lc grab-toggle
lc granularity
lc group
lc group-id
lc group-ids
lc grouped
lc growable
lc gtk
lc guest
lc guest-panic
lc guest-panicked
lc guest-reset
lc guest-shutdown
lc guestfwd
lc guid
lc h
lc half
lc handle
lc hard
lc hash
lc hash-alg
lc head
lc header-digest
lc height
lc help
lc helper
lc henkan
lc hide
lc hierarchy
lc hiragana
lc hits
lc hmat-cache
lc hmat-lb
lc hold-time
lc home
lc host
lc host-error
lc host-key-check
lc host-nodes
lc host-qmp-quit
lc host-qmp-system-reset
lc host-signal
lc host-ui
lc hostfwd
lc hostname
lc hotpluggable
lc hotpluggable-cpus
lc hotplugged
lc hppa
lc http
lc https
lc hubid
lc hubport
lc hugetlb
lc hugetlbsize
lc human-monitor-command
lc human-readable-text
lc hwversion
lc hyper-v
lc hypervisor
lc i
lc i386
lc icount
lc id
lc id-list
lc ide
lc identical
lc identity
lc if
lc ifname
lc ignore
lc ignore-unavailable
lc image
lc image-end-offset
lc image-node-name
lc immediately
lc implements
lc in
lc in-pport
lc in-use
lc inactive
lc inactive-l1
lc inactive-l2
lc inc
lc incompatible
lc inconsistent
lc incremental
lc indev
lc index
lc individual
lc inet
lc info
lc initial
lc initiator
lc initiator-name
lc inject
lc inject-error
lc inject-nmi
lc inmigrate
lc input-barrier
lc input-linux
lc input-send-event
lc insert
lc inserted
lc instances
lc int
lc interface-id
lc interfaces
lc interleave
lc internal-error
lc interval
lc io-error
lc io-status
lc iops
lc iops-read
lc iops-read-max
lc iops-read-max-length
lc iops-size
lc iops-total
lc iops-total-max
lc iops-total-max-length
lc iops-write
lc iops-write-max
lc iops-write-max-length
lc iothread
lc iotype
lc ip
lc ip-dst
lc ip-proto
lc ip-tos
lc ipv4
lc ipv6
lc ipv6-dns
lc ipv6-host
lc ipv6-prefix
lc ipv6-prefixlen
lc irq
lc is-default
lc iscsi
lc iser
lc iter-time
lc iters
lc iv
lc ivgen-alg
lc ivgen-hash-alg
lc j
lc jack
lc job-cancel
lc job-complete
lc job-dismiss
lc job-finalize
lc job-id
lc job-pause
lc job-resume
lc json-cli
lc json-cli-hotplug
lc json-type
lc k
lc katakanahiragana
lc kdump-lzo
lc kdump-snappy
lc kdump-zlib
lc keep
lc keep-alive
lc kernel
lc kernel-hashes
lc key
lc key-offset
lc key-secret
lc keyid
lc keys
lc keyslot
lc l
lc l2-cache-entry-size
lc l2-cache-size
lc l2tpv3
lc label
lc lang1
lc lang2
lc large
lc last-mode
lc late-block-activate
lc latency
lc latency-ns
lc launch-secret
lc launch-update
lc lazy-refcounts
lc lba
lc leaks
lc leaks-fixed
lc left
lc left-command-key
lc len
lc length
lc less
lc level
lc lf
lc limit
lc limits
lc line
lc link-up
lc listen
lc live
lc load
lc loaded
lc local
lc localaddr
lc location
lc locked
lc locking
lc log
lc log-append
lc log-sector-size
lc log-size
lc log-super-update-interval
lc logappend
lc logfile
lc logical-block-size
lc lsilogic
lc luks
lc luks2
lc lun
lc m
lc m68k
lc macaddr
lc mail
lc main-header
lc main-loop
lc main-mac
lc major
lc mask
lc master-key-iters
lc match
lc max
lc max-bandwidth
lc max-chunk
lc max-connections
lc max-cpu-throttle
lc max-discard
lc max-postcopy-bandwidth
lc max-size
lc max-transfer
lc max-workers
lc max-write-zero
lc maxcpus
lc maxlen
lc mbps
lc mcast
lc md5
lc measured
lc measuring
lc mediaselect
lc mem
lc mem-path
lc memaddr
lc members
lc memdev
lc memory
lc memory-backend-epc
lc memory-backend-file
lc memory-backend-memfd
lc memory-backend-ram
lc memsave
lc menu
lc merge
lc meta-meta
lc meta-type
lc metadata
lc micro
lc microblaze
lc microblazeel
lc middle
lc migrate
lc migrate-continue
lc migrate-incoming
lc migrate-pause
lc migrate-recover
lc migrate-set-capabilities
lc migrate-set-parameters
lc migrate-start-postcopy
lc migrated
lc migration
lc migration-safe
lc minor
lc minus
lc mips
lc mips64
lc mips64el
lc mipsel
lc mirror
lc mixing-engine
lc mnonce
lc mode
lc model
lc modela
lc modelb
lc mountpoint
lc mouse
lc mouse-mode
lc mptcp
lc msg
lc msmouse
lc muhenkan
lc multicast
lc multicast-overflow
lc multicast-table
lc multifd
lc multifd-bytes
lc multifd-channels
lc multifd-compression
lc multifd-zlib-level
lc multifd-zstd-level
lc mux
lc n
lc name
lc namespace
lc native
lc nbd
lc nbd-server-add
lc nbd-server-remove
lc nbd-server-start
lc nbd-server-stop
lc net
lc netdev
lc netmap
lc never
lc new-secret
lc new-vlan-id
lc nfs
lc nic
lc nios2
lc no-flush
lc nocow
lc node
lc node-id
lc node-name
lc nodeid
lc nodelay
lc nodes
lc none
lc none-crc32c
lc normal
lc normal-bytes
lc nospace
lc null
lc null-aio
lc null-co
lc num-queues
lc numa-mem-supported
lc number
lc numeric
lc nvdimm
lc nvme
lc o
lc object
lc object-add
lc object-del
lc off
lc offset
lc ok
lc old-secret
lc on
lc on-error
lc on-source-error
lc on-success
lc on-target-error
lc once
lc oob
lc opaque
lc open
lc open-timeout
lc opened
lc operating
lc operation
lc opint-loop
lc opt-discard
lc opt-write-zero
lc option
lc options
lc or1k
lc oss
lc out
lc out-pport
lc outdev
lc overflow
lc overlap-check
lc overlay
lc p
lc p2p
lc pa
lc package
lc packet-header
lc page-cache-size
lc page-sampling
lc page-size
lc pages
lc pages-per-second
lc paging
lc panic
lc parallel
lc parallels
lc parameters
lc parent
lc parent-cid
lc pass
lc pass-discard-other
lc pass-discard-request
lc pass-discard-snapshot
lc passthrough
lc password
lc password-secret
lc passwordid
lc paste
lc path
lc pause
lc pause-before-switchover
lc paused
lc payload-offset
lc pdh
lc pef-guest
lc pending
lc period-length
lc perm
lc persistent
lc pgdn
lc pgmint-loop
lc pgup
lc pincounter
lc pipe
lc plain
lc plain64
lc play
lc plugged-memory
lc pmem
lc pmemsave
lc png
lc policy
lc poll-grow
lc poll-max-ns
lc poll-shrink
lc poll-us
lc pool
lc pop-vlan
lc port
lc portal
lc ports
lc position
lc postcopy-active
lc postcopy-blocktime
lc postcopy-bytes
lc postcopy-paused
lc postcopy-ram
lc postcopy-recover
lc postcopy-requests
lc postcopy-vcpu-blocktime
lc postmigrate
lc power
lc poweroff
lc ppc
lc ppc64
lc ppm
lc pport
lc pr-manager
lc pr-manager-helper
lc pre-switchover
lc prealloc
lc prealloc-align
lc prealloc-size
lc prealloc-threads
lc preallocate
lc preallocation
lc precopy-bytes
lc preferred
lc prefetch
lc prelaunch
lc present
lc pretty
lc primary
lc print
lc priority
lc processing
lc promiscuous
lc properties
lc property
lc props
lc protocol
lc proxy-password-secret
lc proxy-username
lc psw-addr
lc psw-mask
lc pty
lc pwritev
lc q
lc qcode
lc qcow
lc qcow2
lc qdev
lc qed
lc qemu
lc qemu-vdagent
lc qom-get
lc qom-list
lc qom-list-properties
lc qom-list-types
lc qom-path
lc qom-set
lc qom-type
lc qtest
lc query-acpi-ospm-status
lc query-balloon
lc query-block
lc query-block-exports
lc query-block-jobs
lc query-blockstats
lc query-chardev
lc query-chardev-backends
lc query-colo-status
lc query-command-line-options
lc query-commands
lc query-cpu-definitions
lc query-cpu-model-baseline
lc query-cpu-model-comparison
lc query-cpu-model-expansion
lc query-cpus-fast
lc query-current-machine
lc query-dirty-rate
lc query-display-options
lc query-dump
lc query-dump-guest-memory-capability
lc query-fdsets
lc query-gic-capabilities
lc query-hotpluggable-cpus
lc query-iothreads
lc query-jobs
lc query-kvm
lc query-machines
lc query-memdev
lc query-memory-devices
lc query-memory-size-summary
lc query-mice
lc query-migrate
lc query-migrate-capabilities
lc query-migrate-parameters
lc query-name
lc query-named-block-nodes
lc query-nodes
lc query-pci
lc query-pr-managers
lc query-qmp-schema
lc query-replay
lc query-rocker
lc query-rocker-of-dpa-flows
lc query-rocker-of-dpa-groups
lc query-rocker-ports
lc query-rx-filter
lc query-sev
lc query-sev-attestation-report
lc query-sev-capabilities
lc query-sev-launch-measure
lc query-sgx
lc query-sgx-capabilities
lc query-spice
lc query-status
lc query-target
lc query-tpm
lc query-tpm-models
lc query-tpm-types
lc query-uuid
lc query-version
lc query-vm-generation-id
lc query-vnc
lc query-vnc-servers
lc query-xen-replication-status
lc query-yank
lc queue
lc queues
lc quit
lc quorum
lc r
lc ra2
lc ra2ne
lc ram
lc raw
lc rbd
lc rdma-pin-all
lc read
lc read-bandwidth
lc read-latency
lc read-only
lc read-only-mode
lc read-pattern
lc read-write
lc read-zeroes
lc readahead
lc readahead-size
lc readline
lc readonly
lc ready
lc reason
lc reboot
lc receive-update
lc rechs
lc recipient
lc reconnect
lc reconnect-delay
lc record
lc recording
lc recursive
lc reduced-phys-bits
lc refcount-bits
lc refcount-block
lc refcount-cache-size
lc refcount-table
lc reference
lc refresh
lc regions
lc reject
lc rel
lc relaunch
lc release-ram
lc remaining
lc remote
lc removable
lc remove-fd
lc rendernode
lc repeat
lc replaces
lc replay-break
lc replay-delete-break
lc replay-seek
lc replication
lc report
lc request
lc requested-size
lc require
lc required
lc reserve
lc reset
lc resize
lc responsible-properties
lc restore-vm
lc restrict
lc result
lc resume
lc ret
lc ret-type
lc retain
lc return-path
lc rev
lc rewrite-corrupted
lc right
lc ringbuf
lc ringbuf-read
lc ringbuf-write
lc ripemd160
lc riscv32
lc riscv64
lc rng-builtin
lc rng-egd
lc rng-random
lc ro
lc rounds
lc rows
lc rtc-reset-reinjection
lc rules
lc run
lc running
lc rw
lc rx
lc rxcookie
lc rxsession
lc s
lc s16
lc s32
lc s390
lc s390-pv-guest
lc s390x
lc s8
lc safe
lc sample-pages
lc sanity-check
lc sasl
lc save-vm
lc savevm-monitor-nodes
lc screendump
lc script
lc scrolllock
lc sdl
lc seal
lc second-level
lc secondary
lc secret
lc section-size
lc sections
lc sector
lc sector-num
lc sectors-count
lc semicolon
lc send-key
lc send-update
lc serial
lc serpent-128
lc serpent-192
lc serpent-256
lc server
lc server-name
lc service
lc session-file
lc set-action
lc set-eth-dst
lc set-eth-src
lc set-numa-node
lc set-speed
lc set-state
lc set-vlan-id
lc setup
lc setup-time
lc sev-device
lc sev-guest
lc sev-inject-launch-secret
lc sgx
lc sgx-epc
lc sgx1
lc sgx2
lc sh4
lc sh4eb
lc sha1
lc sha224
lc sha256
lc sha384
lc sha512
lc share
lc shared-perm
lc shift
lc shift-shift
lc show-cursor
lc shutdown
lc shutting-down
lc side
lc sig
lc signal
lc singlestep
lc size
lc skipauth
lc skipped
lc slash
lc sleep
lc slew
lc slot
lc slot-type
lc slots
lc smb
lc smbserver
lc snapshot
lc snapshot-access
lc snapshot-delete
lc snapshot-file
lc snapshot-load
lc snapshot-node-name
lc snapshot-save
lc snapshot-table
lc snapshots
lc sndbuf
lc sock
lc socket
lc socket-address
lc socket-id
lc sockets
lc source
lc sparc
lc sparc64
lc spc
lc speed
lc spice
lc spice-app
lc spiceport
lc spicevmc
lc split
lc src
lc srcport
lc ssh
lc sslverify
lc standby
lc start
lc start-server
lc start-time
lc state
lc static
lc stats
lc status
lc stdio
lc step
lc stop
lc stopped
lc str
lc stream
lc stream-name
lc string
lc stripes
lc subformat
lc subnet-prefix
lc subordinate
lc subset
lc subsystem
lc subsystem-reset
lc subsystem-vendor
lc superset
lc suspended
lc swap-opt-cmd
lc sync
lc sysrq
lc t
lc tab
lc table-size
lc tag
lc take-child-perms
lc tap
lc target
lc tbl-id
lc tcp
lc tcp-syn-count
lc telnet
lc template
lc test
lc testdev
lc tftp
lc tftp-server-name
lc third-level
lc thread-id
lc thread-pool-max
lc thread-pool-min
lc threads
lc threshold
lc throttle
lc throttle-group
lc throttle-trigger-threshold
lc tight
lc time
lc timeout
lc timer-period
lc tls
lc tls-authz
lc tls-certs
lc tls-cipher-suites
lc tls-creds
lc tls-creds-anon
lc tls-creds-psk
lc tls-creds-x509
lc tls-hostname
lc tls-none
lc tls-plain
lc tls-port
lc tls-sasl
lc tls-vnc
lc tn3270
lc to
lc toolsversion
lc top
lc top-id
lc top-node
lc total
lc total-clusters
lc total-progress
lc total-time
lc tpm-crb
lc tpm-spapr
lc tpm-tis
lc trace-event-get-state
lc trace-event-set-state
lc transaction
lc transferred
lc transform
lc transport
lc tray-open
lc tricore
lc try-mmap
lc try-poll
lc ttl-check
lc tunnel-id
lc tunnel-lport
lc twofish-128
lc twofish-192
lc twofish-256
lc tx
lc txcookie
lc txsession
lc type
lc typename
lc u
lc u16
lc u32
lc u8
lc udp
lc ultra
lc unaligned-accesses
lc unavailable
lc unavailable-features
lc undefined
lc undo
lc unicast
lc unicast-overflow
lc unicast-table
lc uninit
lc uninitialized
lc unix
lc unknown
lc unmap
lc unmapped
lc unshare-child-perms
lc unstable
lc unstable-input
lc unstable-output
lc unstarted
lc unused
lc up
lc uri
lc url
lc use-copy-range
lc user
lc username
lc utf8
lc uuid
lc v
lc v2
lc v3
lc val
lc validate-uuid
lc value
lc values
lc variants
lc vc
lc vcpu
lc vcpu-dirty-rate
lc vcpus-count
lc vde
lc vdi
lc vectors
lc vencrypt
lc vendor
lc verify-peer
lc version
lc vfio
lc vhdx
lc vhost
lc vhost-user
lc vhost-user-blk
lc vhost-vdpa
lc vhostdev
lc vhostfd
lc vhostfds
lc vhostforce
lc virtio-mem
lc virtio-pmem
lc virtual-size
lc vlan
lc vlan-id
lc vlan-table
lc vm-clock-nsec
lc vm-clock-sec
lc vm-state-size
lc vmdk
lc vmstate
lc vmstate-loaded
lc vmstate-received
lc vmstate-send
lc vmstate-size
lc vnc
lc voices
lc volume
lc volumedown
lc volumeup
lc vote-threshold
lc vpc
lc vsock
lc vvfat
lc w
lc wait
lc wait-unplug
lc waiting
lc wake
lc wakeup-suspend-support
lc watchdog
lc watchdog-set-action
lc wav
lc wctablet
lc websocket
lc wheel-down
lc wheel-left
lc wheel-right
lc wheel-up
lc width
lc win-dmp
lc window-close
lc writable
lc write
lc write-back
lc write-bandwidth
lc write-blocking
lc write-latency
lc write-threshold
lc write-through
lc write-unchanged
lc write-zeroes
lc writeback
lc writethrough
lc x
lc x-blockdev-amend
lc x-blockdev-change
lc x-blockdev-set-iothread
lc x-bps-read
lc x-bps-read-max
lc x-bps-read-max-length
lc x-bps-total
lc x-bps-total-max
lc x-bps-total-max-length
lc x-bps-write
lc x-bps-write-max
lc x-bps-write-max-length
lc x-check-cache-dropped
lc x-checkpoint-delay
lc x-colo
lc x-colo-lost-heartbeat
lc x-debug-block-dirty-bitmap-sha256
lc x-debug-query-block-graph
lc x-dirty-bitmap
lc x-exit-preconfig
lc x-ignore-shared
lc x-iops-read
lc x-iops-read-max
lc x-iops-read-max-length
lc x-iops-size
lc x-iops-total
lc x-iops-total-max
lc x-iops-total-max-length
lc x-iops-write
lc x-iops-write-max
lc x-iops-write-max-length
lc x-origin
lc x-perf
lc x-query-irq
lc x-query-jit
lc x-query-numa
lc x-query-opcount
lc x-query-profile
lc x-query-ramblock
lc x-query-rdma
lc x-query-roms
lc x-query-usb
lc x-remote-object
lc x-use-canonical-path-for-ramblock-id
lc x509-none
lc x509-plain
lc x509-sasl
lc x509-vnc
lc xbzrle
lc xbzrle-cache
lc xbzrle-cache-size
lc xen-colo-do-checkpoint
lc xen-load-devices-state
lc xen-save-devices-state
lc xen-set-global-dirty-log
lc xen-set-replication
lc xtensa
lc xtensaeb
lc xts
lc y
lc y-origin
lc yank
lc yen
lc z
lc zero
lc zero-blocks
lc zeroed-grain
lc zlib
lc zoom-to-fit
lc zstd
mc ACPIOSTInfo
mc ACPISlotType
mc COLOExitReason
mc COLOMessage
mc COLOMode
mc COLOStatus
mc DBusVMStateProperties
mc GICCapability
mc IOThreadInfo
mc JSONType
mc KVMMissingCap
mc NFSServer
mc NFSTransport
mc PCDIMMDeviceInfo
mc PCDIMMDeviceInfoWrapper
mc PCIELinkSpeed
mc PCIELinkWidth
mc PRManagerInfo
mc QAuthZListFormat
mc QAuthZListPolicy
mc QAuthZListRule
mc QCryptoBlockAmendOptions
mc QCryptoBlockAmendOptionsLUKS
mc QCryptoBlockCreateOptions
mc QCryptoBlockCreateOptionsLUKS
mc QCryptoBlockFormat
mc QCryptoBlockInfo
mc QCryptoBlockInfoBase
mc QCryptoBlockInfoLUKS
mc QCryptoBlockInfoLUKSSlot
mc QCryptoBlockLUKSKeyslotState
mc QCryptoBlockOpenOptions
mc QCryptoBlockOptionsBase
mc QCryptoBlockOptionsLUKS
mc QCryptoBlockOptionsQCow
mc QCryptoCipherAlgorithm
mc QCryptoCipherMode
mc QCryptoHashAlgorithm
mc QCryptoIVGenAlgorithm
mc QCryptoSecretFormat
mc QCryptoTLSCredsEndpoint
mc QKeyCode
mc QKeyCodeWrapper
mc QMPCapability
mc S390CrashReason
mc SGXEPCSection
mc SGXInfo
mc SMPConfiguration
mc TPMEmulatorOptions
mc TPMEmulatorOptionsWrapper
mc TPMInfo
mc TPMPassthroughOptions
mc TPMPassthroughOptionsWrapper
mc X86CPUFeatureWordInfo
mc X86CPURegister32
mc XBZRLECacheStats
mc XDbgBlockGraph
mc XDbgBlockGraphEdge
mc XDbgBlockGraphNode
mc XDbgBlockGraphNodeType
mc ac_back
mc ac_bookmarks
mc ac_forward
mc ac_home
mc ac_refresh
mc account_failed
mc account_invalid
mc add_client
mc alt_r
mc asl_compiler_id
mc asl_compiler_rev
mc avg_flush_latency_ns
mc avg_rd_latency_ns
mc avg_rd_queue_depth
mc avg_wr_latency_ns
mc avg_wr_queue_depth
mc backing_file
mc backing_file_depth
mc block_resize
mc block_set_io_throttle
mc bps_max
mc bps_max_length
mc bps_rd
mc bps_rd_max
mc bps_rd_max_length
mc bps_wr
mc bps_wr_max
mc bps_wr_max_length
mc bracket_left
mc bracket_right
mc caps_lock
mc class_info
mc client_migrate_info
mc cluster_alloc
mc cluster_alloc_bytes
mc cluster_alloc_space
mc cluster_free
mc compare_timeout
mc cor_write
mc cow_read
mc cow_write
mc ctrl_r
mc d2_5
mc detect_zeroes
mc device_add
mc device_del
mc empty_image_prepare
mc expire_password
mc expired_scan_cycle
mc failed_flush_operations
mc failed_rd_operations
mc failed_unmap_operations
mc failed_wr_operations
mc flush_latency_histogram
mc flush_operations
mc flush_to_disk
mc flush_to_os
mc flush_total_time_ns
mc grab_all
mc grave_accent
mc host_cdrom
mc host_device
mc idle_time_ns
mc interval_length
mc invalid_flush_operations
mc invalid_rd_operations
mc invalid_unmap_operations
mc invalid_wr_operations
mc io_range
mc io_uring
mc iops_max
mc iops_max_length
mc iops_rd
mc iops_rd_max
mc iops_rd_max_length
mc iops_size
mc iops_wr
mc iops_wr_max
mc iops_wr_max_length
mc irq_pin
mc known_hosts
mc kp_0
mc kp_1
mc kp_2
mc kp_3
mc kp_4
mc kp_5
mc kp_6
mc kp_7
mc kp_8
mc kp_9
mc kp_add
mc kp_comma
mc kp_decimal
mc kp_divide
mc kp_enter
mc kp_equals
mc kp_multiply
mc kp_subtract
mc l1_grow_activate_table
mc l1_grow_alloc_table
mc l1_grow_write_table
mc l1_shrink_free_l2_clusters
mc l1_shrink_write_table
mc l1_update
mc l2_alloc_cow_read
mc l2_alloc_write
mc l2_load
mc l2_update
mc l2_update_compressed
mc legacyESX
mc max_flush_latency_ns
mc max_queue_size
mc max_rd_latency_ns
mc max_wr_latency_ns
mc mem_type_64
mc memory_range
mc meta_l
mc meta_r
mc migrate_cancel
mc min_flush_latency_ns
mc min_rd_latency_ns
mc min_wr_latency_ns
mc monolithicFlat
mc monolithicSparse
mc netdev_add
mc netdev_del
mc new_state
mc notify_dev
mc num_lock
mc oem_id
mc oem_rev
mc oem_table_id
mc pci_bridge
mc prefetchable_range
mc primary_in
mc pwritev_done
mc pwritev_rmw_after_head
mc pwritev_rmw_after_tail
mc pwritev_rmw_head
mc pwritev_rmw_tail
mc pwritev_zero
mc qdev_id
mc qmp_capabilities
mc rd_bytes
mc rd_latency_histogram
mc rd_merged
mc rd_operations
mc rd_total_time_ns
mc read_aio
mc read_backing_aio
mc read_compressed
mc refblock_alloc
mc refblock_alloc_hookup
mc refblock_alloc_switch_table
mc refblock_alloc_write
mc refblock_alloc_write_blocks
mc refblock_alloc_write_table
mc refblock_load
mc refblock_update
mc refblock_update_part
mc reftable_grow
mc reftable_load
mc reftable_update
mc sasl_username
mc scroll_lock
mc secondary_in
mc secret_keyring
mc set_link
mc set_password
mc shift_r
mc streamOptimized
mc system_powerdown
mc system_reset
mc system_wakeup
mc timed_stats
mc tray_open
mc twoGbMaxExtentFlat
mc twoGbMaxExtentSparse
mc unmap_bytes
mc unmap_merged
mc unmap_operations
mc unmap_total_time_ns
mc vmstate_load
mc vmstate_save
mc vnet_hdr
mc vnet_hdr_support
mc wr_bytes
mc wr_highest_offset
mc wr_latency_histogram
mc wr_merged
mc wr_operations
mc wr_total_time_ns
mc write_aio
mc write_compressed
mc write_threshold
mc x509_dname
mc x86_64
uc ACPI_DEVICE_OST
uc BALLOON_CHANGE
uc BLOCK_EXPORT_DELETED
uc BLOCK_IMAGE_CORRUPTED
uc BLOCK_IO_ERROR
uc BLOCK_JOB_CANCELLED
uc BLOCK_JOB_COMPLETED
uc BLOCK_JOB_ERROR
uc BLOCK_JOB_PENDING
uc BLOCK_JOB_READY
uc BLOCK_WRITE_THRESHOLD
uc COLO_EXIT
uc CPU
uc DEVICE_DELETED
uc DEVICE_TRAY_MOVED
uc DEVICE_UNPLUG_GUEST_ERROR
uc DIMM
uc DUMP_COMPLETED
uc EAX
uc EBP
uc EBX
uc ECX
uc EDI
uc EDX
uc ESI
uc ESP
uc FAILOVER_NEGOTIATED
uc GUEST_CRASHLOADED
uc GUEST_PANICKED
uc JOB_STATUS_CHANGE
uc MEMORY_DEVICE_SIZE_CHANGE
uc MEMORY_FAILURE
uc MEM_UNPLUG_ERROR
uc MIGRATION
uc MIGRATION_PASS
uc NIC_RX_FILTER_CHANGED
uc POWERDOWN
uc PR_MANAGER_STATUS_CHANGED
uc QUORUM_FAILURE
uc QUORUM_REPORT_BAD
uc RDMA_GID_STATUS_CHANGED
uc RESET
uc RESUME
uc RTC_CHANGE
uc SHUTDOWN
uc SPICE_CONNECTED
uc SPICE_DISCONNECTED
uc SPICE_INITIALIZED
uc SPICE_MIGRATE_COMPLETED
uc STOP
uc SUSPEND
uc SUSPEND_DISK
uc UNPLUG_PRIMARY
uc UUID
uc VNC_CONNECTED
uc VNC_DISCONNECTED
uc VNC_INITIALIZED
uc VSERPORT_CHANGE
uc WAKEUP
uc WATCHDOG
^ permalink raw reply related [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-26 11:14 ` Markus Armbruster
2022-05-09 10:52 ` Victor Toso
@ 2022-05-10 8:53 ` Daniel P. Berrangé
2022-05-10 9:06 ` Victor Toso
2022-05-10 12:02 ` Markus Armbruster
1 sibling, 2 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 8:53 UTC (permalink / raw)
To: Markus Armbruster
Cc: Victor Toso, John Snow, Eric Blake, qemu-devel, Marc-André Lureau
On Tue, Apr 26, 2022 at 01:14:28PM +0200, Markus Armbruster wrote:
> Victor Toso <victortoso@redhat.com> writes:
>
> > Hi,
> >
> > Happy 1st April. Not a joke :) /* ugh, took me too long to send */
> >
> > This series is about adding a generator in scripts/qapi to produce
> > Go data structures that can be used to communicate with QEMU over
> > QMP.
> >
> >
> > * Why Go?
> >
> > There are quite a few Go projects that interact with QEMU over QMP
> > and they endup using a mix of different libraries with their own
> > code.
> >
> >
> > ** Which projects?
> >
> > The ones I've found so far:
> >
> > - podman machine
> > https://github.com/containers/podman/tree/main/pkg/machine/qemu
> >
> > - kata-containers (govmm)
> > https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm
> >
> > - lxd
> > https://github.com/lxc/lxd/tree/master/lxd/instance/drivers
> >
> > - kubevirt (plain json strings)
> > https://github.com/kubevirt/kubevirt
> >
> > (let me know if you know others)
> >
> >
> > * But Why?
> >
> > I'm particularly interested in 3 out of 4 of the projects above and
> > only Kubevirt uses libvirt to handle QEMU. That means that every
> > QEMU releases where a QMP command, event or other data struct is
> > added, removed or changed, those projects need to check what changed
> > in QEMU and then address those changes in their projects, if needed.
> >
> > The idea behind generating Go data structures is that we can keep a
> > Go module which can have releases that follow QEMU releases.
>
> We need to look at "following the QEMU releases" a bit more closely.
>
> Merging your patches gives us the capability to generate a Go interface
> to HEAD's version of QMP.
>
> The obvious way for an out-of-tree Go program to use this generated Go
> interface is to build with a specific version of it. It can then talk
> QMP to any compatible QEMU version.
>
> Compatibility with older QEMUs is not assured: stuff added since is
> present on the Go QMP client end, but not on the QEMU QMP server end.
>
> Compatibility with newer QEMUs is subject to our deprecation policy:
>
> In general features are intended to be supported indefinitely once
> introduced into QEMU. In the event that a feature needs to be
> removed, it will be listed in this section. The feature will remain
> functional for the release in which it was deprecated and one
> further release. After these two releases, the feature is liable to
> be removed.
>
> So, if you stay away from deprecated stuff, you're good for two more
> releases at least.
>
> Does this work for the projects you have in mind?
It might work for some projects, but in the general case I find it pretty
unappealing as a restriction. Mixing and matching new QEMU with old libvirt,
or vica-verca has been an incredibly common thing todo when both developing
and perhaps more importantly debugging problems. For example I have one
libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x
update, which spans a great many QEMU releases.
I like the idea of auto-generating clients from the QAPI schema, and
would like it if we were able to use this kind of approach on the libvirt
side, but for that we need to be more flexible in version matching.
Our current approach to deprecation features and subsequently removing
them from the QAPI schema works fine when the QAPI schema is only used
internally by QEMU, not when we we expand usage of QAPI to external
applications.
I think we need to figure out a way to make the QAPI schema itself be
append only, while still allowing QEMU to deprecation & remove features.
For a minimum viable use case, this doesn't feel all that difficult, as
conceptually instead of deleting the field from QAPI, we just need to
annotate it to say when it was deleted from the QEMU side. The QAPI
generator for internal QEMU usage, can omit any fields annotated as
deleted in QAPI schema. The QAPI generator for external app usage,
can (optionally) be told to include deleted fields ranging back to
a given version number. So apps can chooses what degree of compat
they wish to retain.
Apps that wish to have version compat, would of course need to write
their code to be aware of which fields they need to seend for which
QEMU version.
> > * Status
> >
> > There are a few rough edges to work on but this is usable. The major
> > thing I forgot to add is handling Error from Commands. It'll be the
> > first thing I'll work on next week.
> >
> > If you want to start using this Today you can fetch it in at
> >
> > https://gitlab.com/victortoso/qapi-go/
> >
> > There are quite a few tests that I took from the examples in the
> > qapi schema. Coverage using go's cover tool is giving `28.6% of
> > statements`
> >
> > I've uploaded the a static generated godoc output of the above Go
> > module here:
> >
> > https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
> >
> >
> > * License
> >
> > While the generator (golang.py in this series) is GPL v2, the
>
> I'd make it v2+, just to express my displeasure with the decision to
> make the initial QAPI generator v2 only for no good reason at all.
Our policy is that all new code should be v2+ anyway, unless it was
clearly derived from some pre-existing v2-only code. Upto Victor to
say whether the golang.py is considered clean, or was copy+paste
in any parts from existin v2-only code
> > generated code needs to be compatible with other Golang projects,
> > such as the ones mentioned above. My intention is to keep a Go
> > module with a MIT license.
>
> Meh. Can't be helped, I guess.
This does make me wonder though whether the license of the QAPI input
files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is
the Go code to be considered a derived work of the QAPI JSON files. I'm
not finding a clearly articulated POV on this question so far.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 8:53 ` Daniel P. Berrangé
@ 2022-05-10 9:06 ` Victor Toso
2022-05-10 9:17 ` Daniel P. Berrangé
2022-05-10 12:02 ` Markus Armbruster
1 sibling, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-10 9:06 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Markus Armbruster, John Snow, Eric Blake, qemu-devel,
Marc-André Lureau
[-- Attachment #1.1: Type: text/plain, Size: 1784 bytes --]
Hi,
On Tue, May 10, 2022 at 09:53:05AM +0100, Daniel P. Berrangé wrote:
> > > * License
> > >
> > > While the generator (golang.py in this series) is GPL v2, the
> >
> > I'd make it v2+, just to express my displeasure with the decision to
> > make the initial QAPI generator v2 only for no good reason at all.
>
> Our policy is that all new code should be v2+ anyway, unless it
> was clearly derived from some pre-existing v2-only code. Upto
> Victor to say whether the golang.py is considered clean, or was
> copy+paste in any parts from existin v2-only code
Should be fine to consider it v2+, the generator.
> > > generated code needs to be compatible with other Golang projects,
> > > such as the ones mentioned above. My intention is to keep a Go
> > > module with a MIT license.
> >
> > Meh. Can't be helped, I guess.
>
> This does make me wonder though whether the license of the QAPI
> input files has a bearing on the Go (or other $LANGUAGE) ouput
> files. eg is the Go code to be considered a derived work of the
> QAPI JSON files.
GPL does not enforce that the generated code to be GPL [0] unless
the generator copies GPL code to the output. My only concern has
been the fact that I am copying the documentation of QAPI
specification to the Go package in order to have data structures,
commands, etc. with the information provided by the
specification.
[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
> I'm not finding a clearly articulated POV on this question so
> far.
I don't find it trivial either but I've accepted that the Go data
structures are fine based on [0] and the documentation can be
split from the Go module (sadly!) if someone finds it to be a
legal issue.
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 9:06 ` Victor Toso
@ 2022-05-10 9:17 ` Daniel P. Berrangé
2022-05-10 9:32 ` Daniel P. Berrangé
0 siblings, 1 reply; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 9:17 UTC (permalink / raw)
To: Victor Toso
Cc: Markus Armbruster, John Snow, Eric Blake, qemu-devel,
Marc-André Lureau
On Tue, May 10, 2022 at 11:06:39AM +0200, Victor Toso wrote:
> Hi,
>
> On Tue, May 10, 2022 at 09:53:05AM +0100, Daniel P. Berrangé wrote:
> > > > * License
> > > >
> > > > While the generator (golang.py in this series) is GPL v2, the
> > >
> > > I'd make it v2+, just to express my displeasure with the decision to
> > > make the initial QAPI generator v2 only for no good reason at all.
> >
> > Our policy is that all new code should be v2+ anyway, unless it
> > was clearly derived from some pre-existing v2-only code. Upto
> > Victor to say whether the golang.py is considered clean, or was
> > copy+paste in any parts from existin v2-only code
>
> Should be fine to consider it v2+, the generator.
>
> > > > generated code needs to be compatible with other Golang projects,
> > > > such as the ones mentioned above. My intention is to keep a Go
> > > > module with a MIT license.
> > >
> > > Meh. Can't be helped, I guess.
> >
> > This does make me wonder though whether the license of the QAPI
> > input files has a bearing on the Go (or other $LANGUAGE) ouput
> > files. eg is the Go code to be considered a derived work of the
> > QAPI JSON files.
>
> GPL does not enforce that the generated code to be GPL [0] unless
> the generator copies GPL code to the output. My only concern has
> been the fact that I am copying the documentation of QAPI
> specification to the Go package in order to have data structures,
> commands, etc. with the information provided by the
> specification.
>
> [0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
>
> > I'm not finding a clearly articulated POV on this question so
> > far.
>
> I don't find it trivial either but I've accepted that the Go data
> structures are fine based on [0] and the documentation can be
> split from the Go module (sadly!) if someone finds it to be a
> legal issue.
Ah well that link above is actually reasonably clear:
"More generally, when a program translates its input into
some other form, the copyright status of the output inherits
that of the input it was generated from. "
In our case the input is the QAPI JSON, and we're translating that
into Golang. That could be read as meaning our Golang code has to
be GPLv2+ licensed just as with the QAPI, not merely the docs.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 9:17 ` Daniel P. Berrangé
@ 2022-05-10 9:32 ` Daniel P. Berrangé
2022-05-10 10:50 ` Victor Toso
0 siblings, 1 reply; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 9:32 UTC (permalink / raw)
To: Victor Toso, Markus Armbruster, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
On Tue, May 10, 2022 at 10:18:15AM +0100, Daniel P. Berrangé wrote:
> On Tue, May 10, 2022 at 11:06:39AM +0200, Victor Toso wrote:
> > Hi,
> >
> > On Tue, May 10, 2022 at 09:53:05AM +0100, Daniel P. Berrangé wrote:
> > > > > * License
> > > > >
> > > > > While the generator (golang.py in this series) is GPL v2, the
> > > >
> > > > I'd make it v2+, just to express my displeasure with the decision to
> > > > make the initial QAPI generator v2 only for no good reason at all.
> > >
> > > Our policy is that all new code should be v2+ anyway, unless it
> > > was clearly derived from some pre-existing v2-only code. Upto
> > > Victor to say whether the golang.py is considered clean, or was
> > > copy+paste in any parts from existin v2-only code
> >
> > Should be fine to consider it v2+, the generator.
> >
> > > > > generated code needs to be compatible with other Golang projects,
> > > > > such as the ones mentioned above. My intention is to keep a Go
> > > > > module with a MIT license.
> > > >
> > > > Meh. Can't be helped, I guess.
> > >
> > > This does make me wonder though whether the license of the QAPI
> > > input files has a bearing on the Go (or other $LANGUAGE) ouput
> > > files. eg is the Go code to be considered a derived work of the
> > > QAPI JSON files.
> >
> > GPL does not enforce that the generated code to be GPL [0] unless
> > the generator copies GPL code to the output. My only concern has
> > been the fact that I am copying the documentation of QAPI
> > specification to the Go package in order to have data structures,
> > commands, etc. with the information provided by the
> > specification.
> >
> > [0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
> >
> > > I'm not finding a clearly articulated POV on this question so
> > > far.
> >
> > I don't find it trivial either but I've accepted that the Go data
> > structures are fine based on [0] and the documentation can be
> > split from the Go module (sadly!) if someone finds it to be a
> > legal issue.
>
> Ah well that link above is actually reasonably clear:
>
> "More generally, when a program translates its input into
> some other form, the copyright status of the output inherits
> that of the input it was generated from. "
>
> In our case the input is the QAPI JSON, and we're translating that
> into Golang. That could be read as meaning our Golang code has to
> be GPLv2+ licensed just as with the QAPI, not merely the docs.
Oh but I'm forgetting that QAPI JSON can be said to be our public API
interface, and so while the docs text would be GPLv2+, we can claim
fair use for the interface definition in the generator and pick an
arbitrary output license.
We could likely deal with the docs problem by not copying the docs
directly into the generated code, instead link to the published
docs on qemu.org. This would require us to improve our current docs
generated anchor generation. ie currently docs link for say the
struct 'NumaNodeOptions' potentially changes every time we generate
it
https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416
we would need to that be something stable, ie
https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
Then the generated Go can just do
// See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
thus avoiding any copyright complication
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-02 14:01 ` Andrea Bolognani
2022-05-03 7:57 ` Markus Armbruster
2022-05-09 18:53 ` Victor Toso
@ 2022-05-10 9:52 ` Daniel P. Berrangé
2022-05-10 15:25 ` Andrea Bolognani
2022-05-11 13:45 ` Markus Armbruster
2 siblings, 2 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 9:52 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Markus Armbruster, Kevin Wolf, John Snow, Victor Toso,
qemu-devel, Eric Blake
On Mon, May 02, 2022 at 10:01:41AM -0400, Andrea Bolognani wrote:
> > Times how many naming conventions?
>
> Yeah, I don't think requiring all possible permutations to be spelled
> out in the schema is the way to go. That's exactly why my proposal
> was to offer a way to inject the semantic information that the parser
> can't figure out itself.
>
> Once you have a way to inform the generator that "VNCProps" is made
> of the two words "vnc" and "props", and that "vnc" is an acronym,
> then it can generate an identifier appropriate for the target
> language without having to spell out anywhere that such an identifier
> would be VNCProps for Go and VncProps for Rust.
>
> By the way, while looking around I realized that we also have to take
> into account things like D-Bus: the QAPI type ChardevDBus, for
> example, would probably translate verbatim to Go but have to be
> changed to ChardevDbus for Rust. Fun :)
The hardest one of all is probably
QAuthZListPolicy
which must be split 'QAuthZ' + 'List' + 'Policy' - the trailing
uppercase ruins all heuristics you can come up with, as there's no
viable way to distinguish that scenario from 'ChardevDBus' which
is 'Chardev' + 'DBus' not 'ChardevD' + 'Bus'
> Revised proposal for the annotation:
>
> ns:word-WORD-WoRD-123Word
Ugly, but we should only need this in the fairly niche scenarios,
so not too pain ful to add a handful of these:
Essentially if have the schema use CamelCase with UPPERCASE
acronyms, and declare two rules:
1. Split on every boundary from lower to upper
2. Acronym detection if there's a sequence of 3 uppercase
letters, then split before the last uppercase.
then we'll do the right thing with the vast majority of cases:
ChardevSocket:
Rule 1: Chardev + Socket
Rule 2: Chardev + Socket
VNCProps:
Rule 1: VNCProps
Rule 2: VNC + Props
ChardevDBus
Rule 1: Chardev + DBus
Rule 2: Chardev + DBus
and fail on
QAuthZListPolicy
Rule 1: QAuth + ZList + Policy
Rule 2: QAuth + ZList + Policy
so only the last case needs ns:QAuthZ-List-Policy annotation, whcih
doesn't feel like a big burden
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-03 9:40 ` Andrea Bolognani
2022-05-03 11:04 ` Kevin Wolf
@ 2022-05-10 9:55 ` Daniel P. Berrangé
2022-05-11 6:15 ` Markus Armbruster
2 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 9:55 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Markus Armbruster, Victor Toso, qemu-devel, John Snow,
Eric Blake, Kevin Wolf
On Tue, May 03, 2022 at 02:40:14AM -0700, Andrea Bolognani wrote:
> On Tue, May 03, 2022 at 09:57:27AM +0200, Markus Armbruster wrote:
> > Andrea Bolognani <abologna@redhat.com> writes:
> > > I still feel that 1) users of a language SDK will ideally not need to
> > > look at the QAPI schema or wire chatter too often
> >
> > I think the most likely point of contact is the QEMU QMP Reference
> > Manual.
>
> Note that there isn't anything preventing us from including the
> original QAPI name in the documentation for the corresponding Go
> symbol, or even a link to the reference manual.
>
> So we could make jumping from the Go API documentation, which is what
> a Go programmer will be looking at most of the time, to the QMP
> documentation pretty much effortless.
>
> > My point is: a name override feature like the one you propose needs to
> > be used with discipline and restraint. Adds to reviewers' mental load.
> > Needs to be worth it. I'm not saying it isn't, I'm just pointing out a
> > cost.
>
> Yeah, I get that.
>
> Note that I'm not suggesting it should be possible for a name to be
> completely overridden - I just want to make it possible for a human
> to provide the name parsing algorithm solutions to those problems it
> can't figure out on its own.
>
> We could prevent that feature from being misused by verifying that
> the symbol the annotation is attached to can be derived from the list
> of words provided. That way, something like
>
> # SOMEName (completely-DIFFERENT-name)
>
> would be rejected and we would avoid misuse.
>
> > Wild idea: assume all lower case, but keep a list of exceptions.
>
> That could actually work reasonably well for QEMU because we only
> need to handle correctly what's in the schema, not arbitrary input.
>
> There's always the risk of the list of exceptions getting out of sync
> with the needs of the schema, but there's similarly no guarantee that
> annotations are going to be introduced when they are necessary, so
> it's mostly a wash.
>
> The only slight advantage of the annotation approach would be that it
> might be easier to notice it being missing because it's close to the
> name it refers to, while the list of exceptions is tucked away in a
> script far away from it.
>
> > The QAPI schema language uses three naming styles:
> >
> > * lower-case-with-hyphens for command and member names
> >
> > Many names use upper case and '_'. See pragma command-name-exceptions
> > and member-name-exceptions.
>
> Looking at the output generated by Victor's WIP script, it looks like
> these are already handled as nicely as those that don't fall under
> any exception.
>
> > Some (many?) names lack separators between words (example: logappend).
> >
> > * UPPER_CASE_WITH_UNDERSCORE for event names
> >
> > * CamelCase for type names
> >
> > Capitalization of words is inconsistent in places (example: VncInfo
> > vs. DisplayReloadOptionsVNC).
> >
> > What style conversions will we need for Go? Any other conversions come
> > to mind?
> >
> > What problems do these conversions have?
>
> Go uses CamelCase for pretty much everything: types, methods,
> constants...
>
> There's one slight wrinkle, in that the case of the first letter
> decides whether it's going to be a PublicName or a privateName. We
> can't do anything about that, but it shouldn't really affect us
> that much because we'll want all QAPI names to be public.
>
> So the issues preventing us from producing a "perfect" Go API are
>
> 1. inconsistent capitalization in type names
>
> -> could be addressed by simply changing the schema, as type
> names do not travel on the wire
>
> 2. missing dashes in certain command/member names
>
> -> leads to Incorrectcamelcase. Kevin's work is supposed to
> address this
>
> 3. inability to know which parts of a lower-case-name or
> UPPER_CASE_NAME are acronyms or are otherwise supposed to be
> capitalized in a specific way
>
> -> leads to WeirdVncAndDbusCapitalization. There's currently no
> way, either implemented or planned, to avoid this
>
> In addition to these I'm also thinking that QKeyCode and all the
> QCrypto stuff should probably lose their prefixes.
At the C level, those prefixes are pretty important to avoid
clashing with symbols defined by system headers and/or the
external crypto library headers, as the unprefixed names are
too generic. All non-C languages of course have proper
namespacing
> > > Revised proposal for the annotation:
> > >
> > > ns:word-WORD-WoRD-123Word
> > >
> > > Words are always separated by dashes; "regular" words are entirely
> > > lowercase, while the presence of even a single uppercase letter in a
> > > word denotes the fact that its case should be preserved when the
> > > naming conventions of the target language allow that.
> >
> > Is a word always capitalized the same for a single target language? Or
> > could capitalization depend on context?
>
> I'm not aware of any language that would adopt more than a single
> style of capitalization, outside of course the obvious
> lower_case_name or UPPER_CASE_NAME scenarios where the original
> capitalization stops being relevant.
As long as the capitalization we use in the schema can provides sufficient
info to detect word splitting and acronyms, we can cope with any output
language naming rules that I know of.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
2022-04-01 22:40 ` [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go Victor Toso
@ 2022-05-10 10:06 ` Daniel P. Berrangé
2022-05-10 11:15 ` Victor Toso
0 siblings, 1 reply; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 10:06 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Sat, Apr 02, 2022 at 12:40:57AM +0200, Victor Toso wrote:
> This patch handles QAPI enum types and generates its equivalent in Go.
>
> The highlights of this implementation are:
>
> 1. For each QAPI enum, we will define an int32 type in Go to be the
> assigned type of this specific enum
I think there's a potential case to be made for considering
representation of enums as strings in Golang, as it more
gracefully degrades.
Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.
This implementation has
type ShutdownCause int32
const (
ShutdownCauseNone ShutdownCause = iota
ShutdownCauseHostError // An error prevents further use of guest
ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
ShutdownCauseHostUi // Reaction to a UI event, like window close
ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
)
and
type ShutdownEvent struct {
Guest bool `json:"guest"` // If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
Reason ShutdownCause `json:"reason"` // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
}
Consider that the application is compiled against the QAPI generated
API from QEMU 7.0. The application is believed to run happily against
7.1, because app doesn't need any of the functionality added in 7.1
QAPI. But consider that QEMU 7.1 had introduced a new shutdown reason
value.
The application may want to know that the guest has shutdown, but does
NOT care about the reason for the shutdown.
Since the ShutdownReason is implemented as an int though, the unmarshalling
for the reason field is going to fail.
If the enums were just represented as strings, then we can gracefully
accept any new enum value that arrives in future. The application can
thus also still log the shutdown reason string, even though it was not
a value explicitly known to the generated API.
>
> 2. While in the Go codebase we can use the generated enum values, the
> specification requires that, on the wire, the enumeration type's
> value to be represented by its string name.
>
> For this reason, each Go type that represent's a QAPI enum will be
> implementing the Marshaler[0] and Unmarshaler[1] interfaces to
> seamless handle QMP's string to Go int32 value and vice-versa.
>
> 3. Naming: CamelCase will be used in any identifier that we want to
> export [2], which is everything in this patch.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go
2022-04-01 22:40 ` [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate " Victor Toso
@ 2022-05-10 10:10 ` Daniel P. Berrangé
2022-05-10 11:21 ` Victor Toso
0 siblings, 1 reply; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 10:10 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Sat, Apr 02, 2022 at 12:40:58AM +0200, Victor Toso wrote:
> This patch handles QAPI alternate types and generates data structures
> in Go that handles it.
>
> At this moment, there are 5 alternates in qemu/qapi, they are:
> * BlockDirtyBitmapMergeSource
> * Qcow2OverlapChecks
> * BlockdevRef
> * BlockdevRefOrNull
> * StrOrNull
>
> Alternate types are similar to Union but without a discriminator that
> can be used to identify the underlying value on the wire. It is needed
> to infer it. That can't be easily mapped in Go.
I don't buy that. Given this example:
type BlockdevRef struct {
// Options are:
// * definition (BlockdevOptions): defines a new block device inline
// * reference (string): references the ID of an existing block device
Value Any
}
What is the problem with having this Go struct:
type BlockdevRef struct {
Definition *BlockdevOptions
Reference *string
}
when deserializing from JSON, we know exactly which one of these two
fields to populate. The programmer consuming this can look at which
field is non-nil.
When serializing to JSON, we serialize which ever field is non-nil.
If both fields are non-nil that's a programmer bug. Either ignore it
and only serialize the first non-nil field, or raise an error.
>
> For each Alternate type, we will be using a Any type to hold the
> value. 'Any' is an alias type for interface{} (similar to void* in C).
>
> Similarly to the Enum types (see previous commit), we will implement
> Marshaler and Unmarshaler interfaces for the Alternate types and in
> those MarshalJSON() and UnmarshalJSON() methods is where we are going
> to put the logic to read/set alternate's value.
>
> Note that on UnmarshalJSON(), a helper function called StrictDecode()
> will be used. This function is the main logic to infer if a given JSON
> object fits in a given Go struct. Because we only have 5 alternate
> types, it is not hard to validate the unmarshaling logic but we might
> need to improve it in the future if Alternate with branches that have
> similar fields appear.
>
> Examples:
> * BlockdevRef
> ```go
> // Data to set in BlockdevOptions
> qcow2 := BlockdevOptionsQcow2{}
> // BlockdevRef using a string
> qcow2.File = BlockdevRef{Value: "/some/place/my-image"}
> opt := BlockdevOptions{}
> opt.Driver = BlockdevDriverQcow2
> opt.Value = qcow2
>
> b, _ := json.Marshal(data.s)
> // string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}`
> ```
>
> Signed-off-by: Victor Toso <victortoso@redhat.com>
> ---
> scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 155 insertions(+), 2 deletions(-)
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go
2022-04-01 22:41 ` [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union " Victor Toso
@ 2022-05-10 10:26 ` Daniel P. Berrangé
2022-05-10 11:32 ` Victor Toso
0 siblings, 1 reply; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 10:26 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Sat, Apr 02, 2022 at 12:41:00AM +0200, Victor Toso wrote:
> This patch handles QAPI union types and generates the equivalent data
> structures and methods in Go to handle it.
>
> At the moment of this writing, it generates 67 structures.
>
> The QAPI union type can be summarized by its common members that are
> defined in a @base struct and a @value. The @value type can vary and
> depends on @base's field that we call @discriminator. The
> @discriminator is always a Enum type.
>
> Golang does not have Unions. The generation of QAPI union type in Go
> with this patch, follows similar approach to what is done for QAPI
> struct types and QAPI alternate types.
The common way to approach unions in Go is to just use a struct
where each union case is an optional field, and declare that only
one field must ever be set. ie
type SocketAddressLegacy struct {
// Value based on @type, possible types:
Inet *InetSocketAddressWrapper
Unix *UnixSocketAddressWrapper
VSock *VsockSocketAddressWrapper
FD *StringWrapper
}
When deserializing from JSON we populate exactly one of the
optional fields.
When serializing to JSON process the first field that is
non-nil.
Note, you don't actually need to include the discriminator
as a field at all, since it is implicitly determined by
whichever case is non-nil. Introducing the discriminator
as a field just provides the possibility for the programmer
to make inconsistent settings, for no gain.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event types in Go
2022-04-01 22:41 ` [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event " Victor Toso
@ 2022-05-10 10:42 ` Daniel P. Berrangé
2022-05-10 11:38 ` Victor Toso
0 siblings, 1 reply; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 10:42 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Sat, Apr 02, 2022 at 12:41:01AM +0200, Victor Toso wrote:
> This patch handles QAPI event types and generates data structures in
> Go that handles it.
>
> At the moment of this writing, it generates 51 structures (49 events)
>
> In Golang, each event is handled as a Go structure and there is no big
> difference, in the Go generated code, between what is a QAPI event
> type and what is a QAPI struct.
>
> Each QAPI event has the suffix 'Event' in its Golang data structure
> and contains the fields, mandatory and optional, that can be
> sent or received.
>
> In addition, there are two structures added to handle QAPI
> specification for event types: 'Event' and 'EventBase'.
>
> 'EventBase' contains @Name and @Timestamp fields and then 'Event'
> extends 'EventBase' with an @Arg field of type 'Any'.
Again, I don't think we should need to use an Any type here.
Rather than
type EventBase struct {
Name string `json:"event"`
Timestamp struct {
Seconds int64 `json:"seconds"`
Microseconds int64 `json:"microseconds"`
} `json:"timestamp"`
}
type Event struct {
EventBase
Arg Any `json:"data,omitempty"`
}
type ShutdownEvent struct {
Guest bool `json:"guest"`
Reason ShutdownCause `json:"reason"`
}
I think we should just embed EventBase directly in each specific
event eg
type Event struct {
Name string `json:"event"`
Timestamp struct {
Seconds int64 `json:"seconds"`
Microseconds int64 `json:"microseconds"`
} `json:"timestamp"`
}
type ShutdownEvent struct {
Event Event
Guest bool `json:"guest"`
Reason ShutdownCause `json:"reason"`
}
Or perhaps better still, use an interface
type Event interface {
GetName() string
GetTimestamp() string
}
type Timestamp struct {
Seconds int64 `json:"seconds"`
Microseconds int64 `json:"microseconds"`
}
type ShutdownEvent struct {
Timestamp Timestamp `json:"timestamp"`
Guest bool `json:"guest"`
Reason ShutdownCause `json:"reason"`
}
func (ev *ShutdownEvent) GetName() string {
return "SHUTDOWN"
}
That way you can define public APIs taking 'Event' as a type,
and impls of the events can be passed directly in/out.
Similar comment for the Command type.
>
> The 'Event' type implements the Unmarshaler to decode the QMP JSON
> Object into the correct Golang (event) struct. The goal here is to
> facilitate receiving Events.
>
> A TODO for this type is to implement Marshaler for 'Event'. It'll
> containt runtime checks to validate before transforming the struct
> into a JSON Object.
>
> Example:
> ```go
> qmpMsg := `{
> "event" : "MIGRATION",
> "timestamp":{
> "seconds": 1432121972,
> "microseconds": 744001
> },
> "data":{
> "status": "completed"
> }
> }`
>
> e := Event{}
> _ = json.Unmarshal([]byte(qmpMsg), &e)
> // e.Name == "MIGRATION"
> // e.Arg.(MigrationEvent).Status == MigrationStatusCompleted
> ```
>
> Signed-off-by: Victor Toso <victortoso@redhat.com>
> ---
> scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++----
> 1 file changed, 84 insertions(+), 8 deletions(-)
>
> diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> index 0a1bf430ba..3bb66d07c7 100644
> --- a/scripts/qapi/golang.py
> +++ b/scripts/qapi/golang.py
> @@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
>
> def __init__(self, prefix: str):
> super().__init__()
> - self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]}
> + self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]}
> self.objects_seen = {}
> self.schema = None
> + self.events = {}
> self._docmap = {}
> self.golang_package_name = "qapi"
>
> @@ -57,6 +58,24 @@ def visit_begin(self, schema):
> def visit_end(self):
> self.schema = None
>
> + # EventBase and Event are not specified in the QAPI schema,
> + # so we need to generate it ourselves.
> + self.target["event"] += '''
> +type EventBase struct {
> + Name string `json:"event"`
> + Timestamp struct {
> + Seconds int64 `json:"seconds"`
> + Microseconds int64 `json:"microseconds"`
> + } `json:"timestamp"`
> +}
> +
> +type Event struct {
> + EventBase
> + Arg Any `json:"data,omitempty"`
> +}
> +'''
> + self.target["event"] += generate_marshal_methods('Event', self.events)
> +
> self.target["helper"] += '''
> // Creates a decoder that errors on unknown Fields
> // Returns true if successfully decoded @from string @into type
> @@ -279,7 +298,28 @@ def visit_command(self,
> pass
>
> def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> - pass
> + assert name == info.defn_name
> + type_name = qapi_to_go_type_name(name, info.defn_meta)
> + self.events[name] = type_name
> +
> + doc = self._docmap.get(name, None)
> + self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
> + content = ""
> + if self_contained:
> + doc_struct, _ = qapi_to_golang_struct_docs(doc)
> + content = generate_struct_type(type_name, "", doc_struct)
> + else:
> + assert isinstance(arg_type, QAPISchemaObjectType)
> + content = qapi_to_golang_struct(name,
> + doc,
> + arg_type.info,
> + arg_type.ifcond,
> + arg_type.features,
> + arg_type.base,
> + arg_type.members,
> + arg_type.variants)
> +
> + self.target["event"] += content
>
> def write(self, output_dir: str) -> None:
> for module_name, content in self.target.items():
> @@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
> }}
> '''
>
> -# Marshal methods for Union types
> +# Marshal methods for Event and Union types
> def generate_marshal_methods(type: str,
> type_dict: Dict[str, str],
> discriminator: str = "",
> base: str = "") -> str:
> - assert base != ""
> - discriminator = "base." + discriminator
> -
> - switch_case_format = '''
> + type_is_union = False
> + json_field = ""
> + struct_field = ""
> + if type == "Event":
> + base = type + "Base"
> + discriminator = "base.Name"
> + struct_field = "Arg"
> + json_field = "data"
> + else:
> + assert base != ""
> + discriminator = "base." + discriminator
> + type_is_union = True
> +
> + switch_case_format = ""
> + if not type_is_union:
> + switch_case_format = '''
> + case "{name}":
> + tmp := struct {{
> + Value {isptr}{case_type} `json:"{json_field},omitempty"`
> + }}{{}}
> + if err := json.Unmarshal(data, &tmp); err != nil {{
> + return err
> + }}
> + if tmp.Value == nil {{
> + s.{struct_field} = nil
> + }} else {{
> + s.{struct_field} = {isptr}tmp.Value
> + }}'''
> + else:
> + switch_case_format = '''
> case {name}:
> value := {case_type}{{}}
> if err := json.Unmarshal(data, &value); err != nil {{
> @@ -374,12 +440,17 @@ def generate_marshal_methods(type: str,
> case_type = type_dict[name]
> isptr = "*" if case_type[0] not in "*[" else ""
> switch_cases += switch_case_format.format(name = name,
> + struct_field = struct_field,
> + json_field = json_field,
> + isptr = isptr,
> case_type = case_type)
> if case_type not in added:
> if_supported_types += f'''typestr != "{case_type}" &&\n\t\t'''
> added[case_type] = True
>
> - marshalfn = f'''
> + marshalfn = ""
> + if type_is_union:
> + marshalfn = f'''
> func (s {type}) MarshalJSON() ([]byte, error) {{
> base, err := json.Marshal(s.{base})
> if err != nil {{
> @@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str:
> words = [word for word in name.replace("_", "-").split("-")]
> name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
> name += ''.join(word.title() for word in words[1:])
> +
> + if meta == "event":
> + name = name[:-3] if name.endswith("Arg") else name
> + name += meta.title()
> +
> return name
> --
> 2.35.1
>
>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
` (9 preceding siblings ...)
2022-04-26 11:14 ` Markus Armbruster
@ 2022-05-10 10:47 ` Daniel P. Berrangé
10 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 10:47 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Sat, Apr 02, 2022 at 12:40:56AM +0200, Victor Toso wrote:
> * Status
>
> There are a few rough edges to work on but this is usable. The major
> thing I forgot to add is handling Error from Commands. It'll be the
> first thing I'll work on next week.
>
> If you want to start using this Today you can fetch it in at
>
> https://gitlab.com/victortoso/qapi-go/
Looking at this my main concern is that there is way too much use
of the 'Any' type. The valid Golang types that can be used with
any of the 'Any' fields are merely expresssed as comments in the
code. I think this needs changing so that the Golang types are
directly expressed in code.
I think there are perhaps only 2-3 cases where the 'Any' type
is genuinely neccessary due to the QAPI schema having an unbounded
set of possible types - SchemaInfoObjectMember, ObjectPropertyInfo
and QomSetCommand
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 9:32 ` Daniel P. Berrangé
@ 2022-05-10 10:50 ` Victor Toso
2022-05-10 10:57 ` Daniel P. Berrangé
0 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-10 10:50 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Markus Armbruster, John Snow, Eric Blake, qemu-devel,
Marc-André Lureau
[-- Attachment #1: Type: text/plain, Size: 3865 bytes --]
Hi,
On Tue, May 10, 2022 at 10:32:49AM +0100, Daniel P. Berrangé wrote:
> On Tue, May 10, 2022 at 10:18:15AM +0100, Daniel P. Berrangé wrote:
> > On Tue, May 10, 2022 at 11:06:39AM +0200, Victor Toso wrote:
> > > > > > generated code needs to be compatible with other Golang projects,
> > > > > > such as the ones mentioned above. My intention is to keep a Go
> > > > > > module with a MIT license.
> > > > >
> > > > > Meh. Can't be helped, I guess.
> > > >
> > > > This does make me wonder though whether the license of the QAPI
> > > > input files has a bearing on the Go (or other $LANGUAGE) ouput
> > > > files. eg is the Go code to be considered a derived work of the
> > > > QAPI JSON files.
> > >
> > > GPL does not enforce that the generated code to be GPL [0] unless
> > > the generator copies GPL code to the output. My only concern has
> > > been the fact that I am copying the documentation of QAPI
> > > specification to the Go package in order to have data structures,
> > > commands, etc. with the information provided by the
> > > specification.
> > >
> > > [0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
> > >
> > > > I'm not finding a clearly articulated POV on this question so
> > > > far.
> > >
> > > I don't find it trivial either but I've accepted that the Go data
> > > structures are fine based on [0] and the documentation can be
> > > split from the Go module (sadly!) if someone finds it to be a
> > > legal issue.
> >
> > Ah well that link above is actually reasonably clear:
> >
> > "More generally, when a program translates its input into
> > some other form, the copyright status of the output inherits
> > that of the input it was generated from. "
> >
> > In our case the input is the QAPI JSON, and we're translating that
> > into Golang. That could be read as meaning our Golang code has to
> > be GPLv2+ licensed just as with the QAPI, not merely the docs.
>
> Oh but I'm forgetting that QAPI JSON can be said to be our
> public API interface, and so while the docs text would be
> GPLv2+, we can claim fair use for the interface definition in
> the generator and pick an arbitrary output license.
Still, it explicit says in the section "In what cases is the
output of a GPL program covered by the GPL too?" is " Only when
the program copies part of itself into the output".
https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#WhatCaseIsOutputGPL
So, to my understating, even if we are consuming a GPLv2+ spec
with a GPLv2+ generator, the output does not need to be GPLv2+
too, unless we are *copying* parts of the input/generator into
the output - which is the case for the documentation only.
I'll raise this again with the my company's legal team to be
sure.
> We could likely deal with the docs problem by not copying the
> docs directly into the generated code, instead link to the
> published docs on qemu.org. This would require us to improve
> our current docs generated anchor generation. ie currently docs
> link for say the struct 'NumaNodeOptions' potentially changes
> every time we generate it
>
> https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416
>
> we would need to that be something stable, ie
>
> https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
>
> Then the generated Go can just do
>
> // See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
>
> thus avoiding any copyright complication
Yes, but it would be quite sad solution. Documentation in Go is
bounded to the module source code and we would be making people
jump thorough links here.
I mean, if that's what we need to do, okay.
I'll keep thinking about alternatives.
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 10:50 ` Victor Toso
@ 2022-05-10 10:57 ` Daniel P. Berrangé
0 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 10:57 UTC (permalink / raw)
To: Victor Toso
Cc: Markus Armbruster, John Snow, Eric Blake, qemu-devel,
Marc-André Lureau
On Tue, May 10, 2022 at 12:50:47PM +0200, Victor Toso wrote:
> Hi,
>
> On Tue, May 10, 2022 at 10:32:49AM +0100, Daniel P. Berrangé wrote:
> > On Tue, May 10, 2022 at 10:18:15AM +0100, Daniel P. Berrangé wrote:
> > > On Tue, May 10, 2022 at 11:06:39AM +0200, Victor Toso wrote:
> > > > > > > generated code needs to be compatible with other Golang projects,
> > > > > > > such as the ones mentioned above. My intention is to keep a Go
> > > > > > > module with a MIT license.
> > > > > >
> > > > > > Meh. Can't be helped, I guess.
> > > > >
> > > > > This does make me wonder though whether the license of the QAPI
> > > > > input files has a bearing on the Go (or other $LANGUAGE) ouput
> > > > > files. eg is the Go code to be considered a derived work of the
> > > > > QAPI JSON files.
> > > >
> > > > GPL does not enforce that the generated code to be GPL [0] unless
> > > > the generator copies GPL code to the output. My only concern has
> > > > been the fact that I am copying the documentation of QAPI
> > > > specification to the Go package in order to have data structures,
> > > > commands, etc. with the information provided by the
> > > > specification.
> > > >
> > > > [0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
> > > >
> > > > > I'm not finding a clearly articulated POV on this question so
> > > > > far.
> > > >
> > > > I don't find it trivial either but I've accepted that the Go data
> > > > structures are fine based on [0] and the documentation can be
> > > > split from the Go module (sadly!) if someone finds it to be a
> > > > legal issue.
> > >
> > > Ah well that link above is actually reasonably clear:
> > >
> > > "More generally, when a program translates its input into
> > > some other form, the copyright status of the output inherits
> > > that of the input it was generated from. "
> > >
> > > In our case the input is the QAPI JSON, and we're translating that
> > > into Golang. That could be read as meaning our Golang code has to
> > > be GPLv2+ licensed just as with the QAPI, not merely the docs.
> >
> > Oh but I'm forgetting that QAPI JSON can be said to be our
> > public API interface, and so while the docs text would be
> > GPLv2+, we can claim fair use for the interface definition in
> > the generator and pick an arbitrary output license.
>
> Still, it explicit says in the section "In what cases is the
> output of a GPL program covered by the GPL too?" is " Only when
> the program copies part of itself into the output".
>
> https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#WhatCaseIsOutputGPL
>
> So, to my understating, even if we are consuming a GPLv2+ spec
> with a GPLv2+ generator, the output does not need to be GPLv2+
> too, unless we are *copying* parts of the input/generator into
> the output - which is the case for the documentation only.
>
> I'll raise this again with the my company's legal team to be
> sure.
>
> > We could likely deal with the docs problem by not copying the
> > docs directly into the generated code, instead link to the
> > published docs on qemu.org. This would require us to improve
> > our current docs generated anchor generation. ie currently docs
> > link for say the struct 'NumaNodeOptions' potentially changes
> > every time we generate it
> >
> > https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416
> >
> > we would need to that be something stable, ie
> >
> > https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
> >
> > Then the generated Go can just do
> >
> > // See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
> >
> > thus avoiding any copyright complication
>
> Yes, but it would be quite sad solution. Documentation in Go is
> bounded to the module source code and we would be making people
> jump thorough links here.
>
> I mean, if that's what we need to do, okay.
It isn't the end of the world IMHO, as people are typically browsing
docs from the online docs site, so full details are only a click away.
This is what I did in libvirt Go APIs for example
https://pkg.go.dev/libvirt.org/go/libvirt#Domain.SnapshotLookupByName
I feel the biggest impact for developers is actually the quality of
the docs that exist. Time invested in better QAPI docs will have more
impact on developers, than trying to eliminate the need to follow one
extra hyperlink.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
2022-05-10 10:06 ` Daniel P. Berrangé
@ 2022-05-10 11:15 ` Victor Toso
2022-05-10 11:19 ` Daniel P. Berrangé
0 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-10 11:15 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
[-- Attachment #1: Type: text/plain, Size: 5630 bytes --]
Hi,
On Tue, May 10, 2022 at 11:06:16AM +0100, Daniel P. Berrangé wrote:
> On Sat, Apr 02, 2022 at 12:40:57AM +0200, Victor Toso wrote:
> > This patch handles QAPI enum types and generates its equivalent in Go.
> >
> > The highlights of this implementation are:
> >
> > 1. For each QAPI enum, we will define an int32 type in Go to be the
> > assigned type of this specific enum
>
> I think there's a potential case to be made for considering
> representation of enums as strings in Golang, as it more
> gracefully degrades.
>
> Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.
>
> This implementation has
>
> type ShutdownCause int32
>
> const (
> ShutdownCauseNone ShutdownCause = iota
> ShutdownCauseHostError // An error prevents further use of guest
> ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
> ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
> ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
> ShutdownCauseHostUi // Reaction to a UI event, like window close
> ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
> ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
> ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
> ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
> )
>
> and
>
> type ShutdownEvent struct {
> Guest bool `json:"guest"` // If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
> Reason ShutdownCause `json:"reason"` // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
> }
>
> Consider that the application is compiled against the QAPI
> generated API from QEMU 7.0. The application is believed to run
> happily against 7.1, because app doesn't need any of the
> functionality added in 7.1 QAPI. But consider that QEMU 7.1
> had introduced a new shutdown reason value.
>
> The application may want to know that the guest has shutdown,
> but does NOT care about the reason for the shutdown.
>
> Since the ShutdownReason is implemented as an int though, the
> unmarshalling for the reason field is going to fail.
Marshalling does error if you try to convert an int that is not
in the range of the enum type.
Unmarshalling should not error in this case, but the field ends
up not being set which defaults to 0 (in this case,
ShutdownCauseNone). That could mislead the *actual* reason but
not without a warning which is expected in this case, imho.
(I know is not an actual warning, just a Println, but it'll be
fixed in the next iteration)
| func (s *ShutdownCause) UnmarshalJSON(data []byte) error {
| var name string
|
| if err := json.Unmarshal(data, &name); err != nil {
| return err
| }
|
| switch name {
| case "none":
| (*s) = ShutdownCauseNone
| case "host-error":
| (*s) = ShutdownCauseHostError
| case "host-qmp-quit":
| (*s) = ShutdownCauseHostQmpQuit
| case "host-qmp-system-reset":
| (*s) = ShutdownCauseHostQmpSystemReset
| case "host-signal":
| (*s) = ShutdownCauseHostSignal
| case "host-ui":
| (*s) = ShutdownCauseHostUi
| case "guest-shutdown":
| (*s) = ShutdownCauseGuestShutdown
| case "guest-reset":
| (*s) = ShutdownCauseGuestReset
| case "guest-panic":
| (*s) = ShutdownCauseGuestPanic
| case "subsystem-reset":
| (*s) = ShutdownCauseSubsystemReset
| default:
| fmt.Println("Failed to decode ShutdownCause", *s)
| }
| return nil
| }
> If the enums were just represented as strings, then we can
> gracefully accept any new enum value that arrives in future.
> The application can thus also still log the shutdown reason
> string, even though it was not a value explicitly known to the
> generated API.
As a string, the warning should still exist and default value of
"" or nil for ptr would apply. IMHO, between string + warning and
int + warning, I'd still go with int here.
That's a design decision to be made. All in all, I don't know
much about the changes in QEMU/QAPI per version to take this
decision, so I'll rely on you and the list for this, not just for
enums but for the other types too.
> > 2. While in the Go codebase we can use the generated enum values, the
> > specification requires that, on the wire, the enumeration type's
> > value to be represented by its string name.
> >
> > For this reason, each Go type that represent's a QAPI enum will be
> > implementing the Marshaler[0] and Unmarshaler[1] interfaces to
> > seamless handle QMP's string to Go int32 value and vice-versa.
> >
> > 3. Naming: CamelCase will be used in any identifier that we want to
> > export [2], which is everything in this patch.
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
2022-05-10 11:15 ` Victor Toso
@ 2022-05-10 11:19 ` Daniel P. Berrangé
2022-05-10 11:28 ` Victor Toso
0 siblings, 1 reply; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 11:19 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Tue, May 10, 2022 at 01:15:32PM +0200, Victor Toso wrote:
> Hi,
>
> On Tue, May 10, 2022 at 11:06:16AM +0100, Daniel P. Berrangé wrote:
> > On Sat, Apr 02, 2022 at 12:40:57AM +0200, Victor Toso wrote:
> > > This patch handles QAPI enum types and generates its equivalent in Go.
> > >
> > > The highlights of this implementation are:
> > >
> > > 1. For each QAPI enum, we will define an int32 type in Go to be the
> > > assigned type of this specific enum
> >
> > I think there's a potential case to be made for considering
> > representation of enums as strings in Golang, as it more
> > gracefully degrades.
> >
> > Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.
> >
> > This implementation has
> >
> > type ShutdownCause int32
> >
> > const (
> > ShutdownCauseNone ShutdownCause = iota
> > ShutdownCauseHostError // An error prevents further use of guest
> > ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
> > ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
> > ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
> > ShutdownCauseHostUi // Reaction to a UI event, like window close
> > ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
> > ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
> > ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
> > ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
> > )
> >
> > and
> >
> > type ShutdownEvent struct {
> > Guest bool `json:"guest"` // If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
> > Reason ShutdownCause `json:"reason"` // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
> > }
> >
> > Consider that the application is compiled against the QAPI
> > generated API from QEMU 7.0. The application is believed to run
> > happily against 7.1, because app doesn't need any of the
> > functionality added in 7.1 QAPI. But consider that QEMU 7.1
> > had introduced a new shutdown reason value.
> >
> > The application may want to know that the guest has shutdown,
> > but does NOT care about the reason for the shutdown.
> >
> > Since the ShutdownReason is implemented as an int though, the
> > unmarshalling for the reason field is going to fail.
>
> Marshalling does error if you try to convert an int that is not
> in the range of the enum type.
>
> Unmarshalling should not error in this case, but the field ends
> up not being set which defaults to 0 (in this case,
> ShutdownCauseNone). That could mislead the *actual* reason but
> not without a warning which is expected in this case, imho.
>
> (I know is not an actual warning, just a Println, but it'll be
> fixed in the next iteration)
I don't thinnk we should be emitting warnings/printlns at all
in this case though. The app should be able to consume the
enum without needing a warning. If the app wants to validate
against a specific set of constants, it can decide to emit
a warning if there was a case it can't handle. We shouldn't
emit warnings/errors in the unmarshalling step though,as it
isn't the right place to decide on such policy.
>
> | func (s *ShutdownCause) UnmarshalJSON(data []byte) error {
> | var name string
> |
> | if err := json.Unmarshal(data, &name); err != nil {
> | return err
> | }
> |
> | switch name {
> | case "none":
> | (*s) = ShutdownCauseNone
> | case "host-error":
> | (*s) = ShutdownCauseHostError
> | case "host-qmp-quit":
> | (*s) = ShutdownCauseHostQmpQuit
> | case "host-qmp-system-reset":
> | (*s) = ShutdownCauseHostQmpSystemReset
> | case "host-signal":
> | (*s) = ShutdownCauseHostSignal
> | case "host-ui":
> | (*s) = ShutdownCauseHostUi
> | case "guest-shutdown":
> | (*s) = ShutdownCauseGuestShutdown
> | case "guest-reset":
> | (*s) = ShutdownCauseGuestReset
> | case "guest-panic":
> | (*s) = ShutdownCauseGuestPanic
> | case "subsystem-reset":
> | (*s) = ShutdownCauseSubsystemReset
> | default:
> | fmt.Println("Failed to decode ShutdownCause", *s)
> | }
> | return nil
> | }
>
> > If the enums were just represented as strings, then we can
> > gracefully accept any new enum value that arrives in future.
> > The application can thus also still log the shutdown reason
> > string, even though it was not a value explicitly known to the
> > generated API.
>
> As a string, the warning should still exist and default value of
> "" or nil for ptr would apply. IMHO, between string + warning and
> int + warning, I'd still go with int here.
>
> That's a design decision to be made. All in all, I don't know
> much about the changes in QEMU/QAPI per version to take this
> decision, so I'll rely on you and the list for this, not just for
> enums but for the other types too.
Essentially every release of QEMU will have QAPI changes. Most of
the time these are append-only changes. ie a new struct, new command,
new field, new enum value. Sometimes there will be removals due
to deprecation, though note my other reply saying that we really
ought to stop doing removals from the schema, and instead just
annotate when a field stops being used.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go
2022-05-10 10:10 ` Daniel P. Berrangé
@ 2022-05-10 11:21 ` Victor Toso
2022-05-10 11:28 ` Daniel P. Berrangé
0 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-10 11:21 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
[-- Attachment #1: Type: text/plain, Size: 3516 bytes --]
Hi,
On Tue, May 10, 2022 at 11:10:45AM +0100, Daniel P. Berrangé wrote:
> On Sat, Apr 02, 2022 at 12:40:58AM +0200, Victor Toso wrote:
> > This patch handles QAPI alternate types and generates data
> > structures in Go that handles it.
> >
> > At this moment, there are 5 alternates in qemu/qapi, they are:
> > * BlockDirtyBitmapMergeSource
> > * Qcow2OverlapChecks
> > * BlockdevRef
> > * BlockdevRefOrNull
> > * StrOrNull
> >
> > Alternate types are similar to Union but without a
> > discriminator that can be used to identify the underlying
> > value on the wire. It is needed to infer it. That can't be
> > easily mapped in Go.
>
> I don't buy that. Given this example:
>
> type BlockdevRef struct {
> // Options are:
> // * definition (BlockdevOptions): defines a new block device inline
> // * reference (string): references the ID of an existing block device
> Value Any
> }
>
> What is the problem with having this Go struct:
>
> type BlockdevRef struct {
> Definition *BlockdevOptions
> Reference *string
> }
... this is better.
> when deserializing from JSON, we know exactly which one of
> these two fields to populate. The programmer consuming this can
> look at which field is non-nil.
>
> When serializing to JSON, we serialize which ever field is
> non-nil.
>
> If both fields are non-nil that's a programmer bug. Either
> ignore it and only serialize the first non-nil field, or raise
> an error.
It would be a programmer bug if they set a Value of a type not
allowed by Type's spec, but it would be a *runtime* error. Your
suggestion is more type safe.
Thanks.
>
> >
> > For each Alternate type, we will be using a Any type to hold the
> > value. 'Any' is an alias type for interface{} (similar to void* in C).
> >
> > Similarly to the Enum types (see previous commit), we will implement
> > Marshaler and Unmarshaler interfaces for the Alternate types and in
> > those MarshalJSON() and UnmarshalJSON() methods is where we are going
> > to put the logic to read/set alternate's value.
> >
> > Note that on UnmarshalJSON(), a helper function called StrictDecode()
> > will be used. This function is the main logic to infer if a given JSON
> > object fits in a given Go struct. Because we only have 5 alternate
> > types, it is not hard to validate the unmarshaling logic but we might
> > need to improve it in the future if Alternate with branches that have
> > similar fields appear.
> >
> > Examples:
> > * BlockdevRef
> > ```go
> > // Data to set in BlockdevOptions
> > qcow2 := BlockdevOptionsQcow2{}
> > // BlockdevRef using a string
> > qcow2.File = BlockdevRef{Value: "/some/place/my-image"}
> > opt := BlockdevOptions{}
> > opt.Driver = BlockdevDriverQcow2
> > opt.Value = qcow2
> >
> > b, _ := json.Marshal(data.s)
> > // string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}`
> > ```
> >
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> > scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++-
> > 1 file changed, 155 insertions(+), 2 deletions(-)
>
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
2022-05-10 11:19 ` Daniel P. Berrangé
@ 2022-05-10 11:28 ` Victor Toso
2022-05-10 11:39 ` Daniel P. Berrangé
0 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-10 11:28 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
[-- Attachment #1: Type: text/plain, Size: 3659 bytes --]
Hi,
On Tue, May 10, 2022 at 12:19:57PM +0100, Daniel P. Berrangé wrote:
> > Marshalling does error if you try to convert an int that is not
> > in the range of the enum type.
> >
> > Unmarshalling should not error in this case, but the field ends
> > up not being set which defaults to 0 (in this case,
> > ShutdownCauseNone). That could mislead the *actual* reason but
> > not without a warning which is expected in this case, imho.
> >
> > (I know is not an actual warning, just a Println, but it'll be
> > fixed in the next iteration)
>
> I don't thinnk we should be emitting warnings/printlns at all
> in this case though. The app should be able to consume the enum
> without needing a warning. If the app wants to validate
> against a specific set of constants, it can decide to emit a
> warning if there was a case it can't handle. We shouldn't emit
> warnings/errors in the unmarshalling step though,as it isn't
> the right place to decide on such policy.
I think it is useful to know that, a binary compiled to qapi-go
7.0 but running with qemu 7.1 would have some mismatches. It
could help detect issues (e.g: Why my program doesn't know/show
the reason for shutdown?).
So, some sort of --verbose option for this level should exist.
> > | func (s *ShutdownCause) UnmarshalJSON(data []byte) error {
> > | var name string
> > |
> > | if err := json.Unmarshal(data, &name); err != nil {
> > | return err
> > | }
> > |
> > | switch name {
> > | case "none":
> > | (*s) = ShutdownCauseNone
> > | case "host-error":
> > | (*s) = ShutdownCauseHostError
> > | case "host-qmp-quit":
> > | (*s) = ShutdownCauseHostQmpQuit
> > | case "host-qmp-system-reset":
> > | (*s) = ShutdownCauseHostQmpSystemReset
> > | case "host-signal":
> > | (*s) = ShutdownCauseHostSignal
> > | case "host-ui":
> > | (*s) = ShutdownCauseHostUi
> > | case "guest-shutdown":
> > | (*s) = ShutdownCauseGuestShutdown
> > | case "guest-reset":
> > | (*s) = ShutdownCauseGuestReset
> > | case "guest-panic":
> > | (*s) = ShutdownCauseGuestPanic
> > | case "subsystem-reset":
> > | (*s) = ShutdownCauseSubsystemReset
> > | default:
> > | fmt.Println("Failed to decode ShutdownCause", *s)
> > | }
> > | return nil
> > | }
> >
> > > If the enums were just represented as strings, then we can
> > > gracefully accept any new enum value that arrives in future.
> > > The application can thus also still log the shutdown reason
> > > string, even though it was not a value explicitly known to the
> > > generated API.
> >
> > As a string, the warning should still exist and default value of
> > "" or nil for ptr would apply. IMHO, between string + warning and
> > int + warning, I'd still go with int here.
> >
> > That's a design decision to be made. All in all, I don't know
> > much about the changes in QEMU/QAPI per version to take this
> > decision, so I'll rely on you and the list for this, not just for
> > enums but for the other types too.
>
> Essentially every release of QEMU will have QAPI changes. Most of
> the time these are append-only changes. ie a new struct, new command,
> new field, new enum value. Sometimes there will be removals due
> to deprecation, though note my other reply saying that we really
> ought to stop doing removals from the schema, and instead just
> annotate when a field stops being used.
Ok, thanks.
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go
2022-05-10 11:21 ` Victor Toso
@ 2022-05-10 11:28 ` Daniel P. Berrangé
0 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 11:28 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Tue, May 10, 2022 at 01:21:29PM +0200, Victor Toso wrote:
> Hi,
>
> On Tue, May 10, 2022 at 11:10:45AM +0100, Daniel P. Berrangé wrote:
> > On Sat, Apr 02, 2022 at 12:40:58AM +0200, Victor Toso wrote:
> > > This patch handles QAPI alternate types and generates data
> > > structures in Go that handles it.
> > >
> > > At this moment, there are 5 alternates in qemu/qapi, they are:
> > > * BlockDirtyBitmapMergeSource
> > > * Qcow2OverlapChecks
> > > * BlockdevRef
> > > * BlockdevRefOrNull
> > > * StrOrNull
> > >
> > > Alternate types are similar to Union but without a
> > > discriminator that can be used to identify the underlying
> > > value on the wire. It is needed to infer it. That can't be
> > > easily mapped in Go.
> >
> > I don't buy that. Given this example:
> >
> > type BlockdevRef struct {
> > // Options are:
> > // * definition (BlockdevOptions): defines a new block device inline
> > // * reference (string): references the ID of an existing block device
> > Value Any
> > }
> >
> > What is the problem with having this Go struct:
> >
> > type BlockdevRef struct {
> > Definition *BlockdevOptions
> > Reference *string
> > }
>
> ... this is better.
>
> > when deserializing from JSON, we know exactly which one of
> > these two fields to populate. The programmer consuming this can
> > look at which field is non-nil.
> >
> > When serializing to JSON, we serialize which ever field is
> > non-nil.
> >
> > If both fields are non-nil that's a programmer bug. Either
> > ignore it and only serialize the first non-nil field, or raise
> > an error.
>
> It would be a programmer bug if they set a Value of a type not
> allowed by Type's spec, but it would be a *runtime* error. Your
> suggestion is more type safe.
Yep, essentially I'm saying I want the code to enable type
checking to be done at compile time instead of runtime,
whenever it is possible to achieve that from a technical POV.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go
2022-05-10 10:26 ` Daniel P. Berrangé
@ 2022-05-10 11:32 ` Victor Toso
2022-05-10 11:42 ` Daniel P. Berrangé
0 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-10 11:32 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
[-- Attachment #1: Type: text/plain, Size: 1969 bytes --]
Hi,
On Tue, May 10, 2022 at 11:26:56AM +0100, Daniel P. Berrangé wrote:
> On Sat, Apr 02, 2022 at 12:41:00AM +0200, Victor Toso wrote:
> > This patch handles QAPI union types and generates the equivalent data
> > structures and methods in Go to handle it.
> >
> > At the moment of this writing, it generates 67 structures.
> >
> > The QAPI union type can be summarized by its common members that are
> > defined in a @base struct and a @value. The @value type can vary and
> > depends on @base's field that we call @discriminator. The
> > @discriminator is always a Enum type.
> >
> > Golang does not have Unions. The generation of QAPI union type in Go
> > with this patch, follows similar approach to what is done for QAPI
> > struct types and QAPI alternate types.
>
> The common way to approach unions in Go is to just use a struct
> where each union case is an optional field, and declare that
> only one field must ever be set. ie
>
> type SocketAddressLegacy struct {
> // Value based on @type, possible types:
> Inet *InetSocketAddressWrapper
> Unix *UnixSocketAddressWrapper
> VSock *VsockSocketAddressWrapper
> FD *StringWrapper
> }
Like Alternates, I like this better.
> When deserializing from JSON we populate exactly one of the
> optional fields.
>
> When serializing to JSON process the first field that is
> non-nil.
>
> Note, you don't actually need to include the discriminator as a
> field at all, since it is implicitly determined by whichever
> case is non-nil. Introducing the discriminator as a field just
> provides the possibility for the programmer to make
> inconsistent settings, for no gain.
Sounds reasonable. We still need to implement Marshal/Unmarshal
for unknow types (e.g: a new Type for SocketAddressLegacy was
introduced in 7.1 and we should be able to know that current
qapi-go version can't understand it).
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event types in Go
2022-05-10 10:42 ` Daniel P. Berrangé
@ 2022-05-10 11:38 ` Victor Toso
0 siblings, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-05-10 11:38 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
[-- Attachment #1: Type: text/plain, Size: 10510 bytes --]
Hi,
On Tue, May 10, 2022 at 11:42:10AM +0100, Daniel P. Berrangé wrote:
> On Sat, Apr 02, 2022 at 12:41:01AM +0200, Victor Toso wrote:
> > This patch handles QAPI event types and generates data structures in
> > Go that handles it.
> >
> > At the moment of this writing, it generates 51 structures (49 events)
> >
> > In Golang, each event is handled as a Go structure and there is no big
> > difference, in the Go generated code, between what is a QAPI event
> > type and what is a QAPI struct.
> >
> > Each QAPI event has the suffix 'Event' in its Golang data structure
> > and contains the fields, mandatory and optional, that can be
> > sent or received.
> >
> > In addition, there are two structures added to handle QAPI
> > specification for event types: 'Event' and 'EventBase'.
> >
> > 'EventBase' contains @Name and @Timestamp fields and then 'Event'
> > extends 'EventBase' with an @Arg field of type 'Any'.
>
> Again, I don't think we should need to use an Any type here.
>
> Rather than
>
> type EventBase struct {
> Name string `json:"event"`
> Timestamp struct {
> Seconds int64 `json:"seconds"`
> Microseconds int64 `json:"microseconds"`
> } `json:"timestamp"`
> }
>
> type Event struct {
> EventBase
> Arg Any `json:"data,omitempty"`
> }
>
> type ShutdownEvent struct {
> Guest bool `json:"guest"`
> Reason ShutdownCause `json:"reason"`
> }
>
>
> I think we should just embed EventBase directly in each specific
> event eg
>
>
> type Event struct {
> Name string `json:"event"`
> Timestamp struct {
> Seconds int64 `json:"seconds"`
> Microseconds int64 `json:"microseconds"`
> } `json:"timestamp"`
> }
>
> type ShutdownEvent struct {
> Event Event
> Guest bool `json:"guest"`
> Reason ShutdownCause `json:"reason"`
> }
>
>
> Or perhaps better still, use an interface
>
> type Event interface {
> GetName() string
> GetTimestamp() string
> }
>
> type Timestamp struct {
> Seconds int64 `json:"seconds"`
> Microseconds int64 `json:"microseconds"`
> }
>
> type ShutdownEvent struct {
> Timestamp Timestamp `json:"timestamp"`
> Guest bool `json:"guest"`
> Reason ShutdownCause `json:"reason"`
> }
>
> func (ev *ShutdownEvent) GetName() string {
> return "SHUTDOWN"
> }
>
> That way you can define public APIs taking 'Event' as a type,
> and impls of the events can be passed directly in/out.
>
> Similar comment for the Command type.
Yeah, if we are removing Any for Unions and Alternates, we can
sure do the same for Events and Commands.
> > The 'Event' type implements the Unmarshaler to decode the QMP JSON
> > Object into the correct Golang (event) struct. The goal here is to
> > facilitate receiving Events.
> >
> > A TODO for this type is to implement Marshaler for 'Event'. It'll
> > containt runtime checks to validate before transforming the struct
> > into a JSON Object.
> >
> > Example:
> > ```go
> > qmpMsg := `{
> > "event" : "MIGRATION",
> > "timestamp":{
> > "seconds": 1432121972,
> > "microseconds": 744001
> > },
> > "data":{
> > "status": "completed"
> > }
> > }`
> >
> > e := Event{}
> > _ = json.Unmarshal([]byte(qmpMsg), &e)
> > // e.Name == "MIGRATION"
> > // e.Arg.(MigrationEvent).Status == MigrationStatusCompleted
> > ```
> >
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> > scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++----
> > 1 file changed, 84 insertions(+), 8 deletions(-)
> >
> > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > index 0a1bf430ba..3bb66d07c7 100644
> > --- a/scripts/qapi/golang.py
> > +++ b/scripts/qapi/golang.py
> > @@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> >
> > def __init__(self, prefix: str):
> > super().__init__()
> > - self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]}
> > + self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]}
> > self.objects_seen = {}
> > self.schema = None
> > + self.events = {}
> > self._docmap = {}
> > self.golang_package_name = "qapi"
> >
> > @@ -57,6 +58,24 @@ def visit_begin(self, schema):
> > def visit_end(self):
> > self.schema = None
> >
> > + # EventBase and Event are not specified in the QAPI schema,
> > + # so we need to generate it ourselves.
> > + self.target["event"] += '''
> > +type EventBase struct {
> > + Name string `json:"event"`
> > + Timestamp struct {
> > + Seconds int64 `json:"seconds"`
> > + Microseconds int64 `json:"microseconds"`
> > + } `json:"timestamp"`
> > +}
> > +
> > +type Event struct {
> > + EventBase
> > + Arg Any `json:"data,omitempty"`
> > +}
> > +'''
> > + self.target["event"] += generate_marshal_methods('Event', self.events)
> > +
> > self.target["helper"] += '''
> > // Creates a decoder that errors on unknown Fields
> > // Returns true if successfully decoded @from string @into type
> > @@ -279,7 +298,28 @@ def visit_command(self,
> > pass
> >
> > def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> > - pass
> > + assert name == info.defn_name
> > + type_name = qapi_to_go_type_name(name, info.defn_meta)
> > + self.events[name] = type_name
> > +
> > + doc = self._docmap.get(name, None)
> > + self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
> > + content = ""
> > + if self_contained:
> > + doc_struct, _ = qapi_to_golang_struct_docs(doc)
> > + content = generate_struct_type(type_name, "", doc_struct)
> > + else:
> > + assert isinstance(arg_type, QAPISchemaObjectType)
> > + content = qapi_to_golang_struct(name,
> > + doc,
> > + arg_type.info,
> > + arg_type.ifcond,
> > + arg_type.features,
> > + arg_type.base,
> > + arg_type.members,
> > + arg_type.variants)
> > +
> > + self.target["event"] += content
> >
> > def write(self, output_dir: str) -> None:
> > for module_name, content in self.target.items():
> > @@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
> > }}
> > '''
> >
> > -# Marshal methods for Union types
> > +# Marshal methods for Event and Union types
> > def generate_marshal_methods(type: str,
> > type_dict: Dict[str, str],
> > discriminator: str = "",
> > base: str = "") -> str:
> > - assert base != ""
> > - discriminator = "base." + discriminator
> > -
> > - switch_case_format = '''
> > + type_is_union = False
> > + json_field = ""
> > + struct_field = ""
> > + if type == "Event":
> > + base = type + "Base"
> > + discriminator = "base.Name"
> > + struct_field = "Arg"
> > + json_field = "data"
> > + else:
> > + assert base != ""
> > + discriminator = "base." + discriminator
> > + type_is_union = True
> > +
> > + switch_case_format = ""
> > + if not type_is_union:
> > + switch_case_format = '''
> > + case "{name}":
> > + tmp := struct {{
> > + Value {isptr}{case_type} `json:"{json_field},omitempty"`
> > + }}{{}}
> > + if err := json.Unmarshal(data, &tmp); err != nil {{
> > + return err
> > + }}
> > + if tmp.Value == nil {{
> > + s.{struct_field} = nil
> > + }} else {{
> > + s.{struct_field} = {isptr}tmp.Value
> > + }}'''
> > + else:
> > + switch_case_format = '''
> > case {name}:
> > value := {case_type}{{}}
> > if err := json.Unmarshal(data, &value); err != nil {{
> > @@ -374,12 +440,17 @@ def generate_marshal_methods(type: str,
> > case_type = type_dict[name]
> > isptr = "*" if case_type[0] not in "*[" else ""
> > switch_cases += switch_case_format.format(name = name,
> > + struct_field = struct_field,
> > + json_field = json_field,
> > + isptr = isptr,
> > case_type = case_type)
> > if case_type not in added:
> > if_supported_types += f'''typestr != "{case_type}" &&\n\t\t'''
> > added[case_type] = True
> >
> > - marshalfn = f'''
> > + marshalfn = ""
> > + if type_is_union:
> > + marshalfn = f'''
> > func (s {type}) MarshalJSON() ([]byte, error) {{
> > base, err := json.Marshal(s.{base})
> > if err != nil {{
> > @@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str:
> > words = [word for word in name.replace("_", "-").split("-")]
> > name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
> > name += ''.join(word.title() for word in words[1:])
> > +
> > + if meta == "event":
> > + name = name[:-3] if name.endswith("Arg") else name
> > + name += meta.title()
> > +
> > return name
> > --
> > 2.35.1
> >
> >
>
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
2022-05-10 11:28 ` Victor Toso
@ 2022-05-10 11:39 ` Daniel P. Berrangé
0 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 11:39 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Tue, May 10, 2022 at 01:28:39PM +0200, Victor Toso wrote:
> Hi,
>
> On Tue, May 10, 2022 at 12:19:57PM +0100, Daniel P. Berrangé wrote:
> > > Marshalling does error if you try to convert an int that is not
> > > in the range of the enum type.
> > >
> > > Unmarshalling should not error in this case, but the field ends
> > > up not being set which defaults to 0 (in this case,
> > > ShutdownCauseNone). That could mislead the *actual* reason but
> > > not without a warning which is expected in this case, imho.
> > >
> > > (I know is not an actual warning, just a Println, but it'll be
> > > fixed in the next iteration)
> >
> > I don't thinnk we should be emitting warnings/printlns at all
> > in this case though. The app should be able to consume the enum
> > without needing a warning. If the app wants to validate
> > against a specific set of constants, it can decide to emit a
> > warning if there was a case it can't handle. We shouldn't emit
> > warnings/errors in the unmarshalling step though,as it isn't
> > the right place to decide on such policy.
>
> I think it is useful to know that, a binary compiled to qapi-go
> 7.0 but running with qemu 7.1 would have some mismatches. It
> could help detect issues (e.g: Why my program doesn't know/show
> the reason for shutdown?).
If apps are actually looking at the shutdown reasons, it is
trivial for the app themselves to log any unexpected reasons.
They should in fact do that regardless, because if someone
re-compiles the app against a new version of the QEMU Go API,
there may be new ShutdownCause's defined that their codes does
not handle. There's no compile time check from Go side that
their code has handled all ShutdownCause values, since there's
no struct enum type in Go. This again pushes me to towards
the view that the enums should just remain as strings, the use
of integers gives no functional benefit AFAICT, given the lack
of a real enum type in Go.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go
2022-05-10 11:32 ` Victor Toso
@ 2022-05-10 11:42 ` Daniel P. Berrangé
0 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 11:42 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Tue, May 10, 2022 at 01:32:08PM +0200, Victor Toso wrote:
> Hi,
>
> On Tue, May 10, 2022 at 11:26:56AM +0100, Daniel P. Berrangé wrote:
> > On Sat, Apr 02, 2022 at 12:41:00AM +0200, Victor Toso wrote:
> > > This patch handles QAPI union types and generates the equivalent data
> > > structures and methods in Go to handle it.
> > >
> > > At the moment of this writing, it generates 67 structures.
> > >
> > > The QAPI union type can be summarized by its common members that are
> > > defined in a @base struct and a @value. The @value type can vary and
> > > depends on @base's field that we call @discriminator. The
> > > @discriminator is always a Enum type.
> > >
> > > Golang does not have Unions. The generation of QAPI union type in Go
> > > with this patch, follows similar approach to what is done for QAPI
> > > struct types and QAPI alternate types.
> >
> > The common way to approach unions in Go is to just use a struct
> > where each union case is an optional field, and declare that
> > only one field must ever be set. ie
> >
> > type SocketAddressLegacy struct {
> > // Value based on @type, possible types:
> > Inet *InetSocketAddressWrapper
> > Unix *UnixSocketAddressWrapper
> > VSock *VsockSocketAddressWrapper
> > FD *StringWrapper
> > }
>
> Like Alternates, I like this better.
>
> > When deserializing from JSON we populate exactly one of the
> > optional fields.
> >
> > When serializing to JSON process the first field that is
> > non-nil.
> >
> > Note, you don't actually need to include the discriminator as a
> > field at all, since it is implicitly determined by whichever
> > case is non-nil. Introducing the discriminator as a field just
> > provides the possibility for the programmer to make
> > inconsistent settings, for no gain.
>
> Sounds reasonable. We still need to implement Marshal/Unmarshal
> for unknow types (e.g: a new Type for SocketAddressLegacy was
> introduced in 7.1 and we should be able to know that current
> qapi-go version can't understand it).
If there's a new type seen on the wire then, the easy
option is to just not deserialize it at all. Return
a SocketAddressLegacy struct with all fields nil, or
perhaps return a full 'error', since this is likely
to be significant functionalproblem for the application.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 8:06 ` Markus Armbruster
@ 2022-05-10 11:48 ` Victor Toso
0 siblings, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-05-10 11:48 UTC (permalink / raw)
To: Markus Armbruster
Cc: Andrea Bolognani, qemu-devel, John Snow, Eric Blake, Kevin Wolf
[-- Attachment #1: Type: text/plain, Size: 49174 bytes --]
Hi,
On Tue, May 10, 2022 at 10:06:34AM +0200, Markus Armbruster wrote:
> Victor Toso <victortoso@redhat.com> writes:
> >> That's true, but at least to me the trade-off feels reasonable.
> >
> > I don't quite get the argument why it gets harder to find. We can
> > simply provide the actual name as reference in the generated
> > documentation, no?
>
> Predictable name transformation can save me detours through
> documentation. Being able to guess the Go bindings just from the QMP
> Reference Manual can be nice: it lets me write Go code with just the QMP
> Reference manual in view. Being able to guess the name in the QAPI
> schema from the name in the Go bindings can also be nice: it lets me
> look it up in the QMP Reference manual without jumping through the
> bindings documentation.
>
> Or do you envisage a Go bindings manual that fully replaces the
> QMP Reference Manual for developers writing Go?
When the language specifies a way of documenting the code and
provides tools to generate documentation (e.g: html format) from
the source code, it is common that IDEs would use this to provide
documentation from a given library in a simple way.
Links break that. But we might not have a choice anyway due the
fact that copying GPLv2+ could potentially be a problem to
license the Go module as something else.
> > As Kevin already pointed out that he is not planning to work on
> > the Alias (reference below), any other idea besides the Andrea's
> > annotation suggestion? While this is not a blocker, I agree it
> > would be nice to be consistent.
> >
> > https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html
>
> Let's first try to get a handle on how many schema names are
> problematic.
Thanks for this. I'll reply again when I went through it (if no
one else beats me to it).
Cheers,
Victor
> $ git-diff
> diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
> index 5a1782b57e..804b8ab455 100644
> --- a/scripts/qapi/expr.py
> +++ b/scripts/qapi/expr.py
> @@ -94,6 +94,7 @@ def check_name_str(name: str, info: QAPISourceInfo, source: str) -> str:
> """
> # Reserve the entire 'q_' namespace for c_name(), and for 'q_empty'
> # and 'q_obj_*' implicit type names.
> + print("###", name)
> match = valid_name.match(name)
> if not match or c_name(name, False).startswith('q_'):
> raise QAPISemError(info, "%s has an invalid name" % source)
> $ cd bld
> $ python3 ../scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sort
> cc Abort
> cc AbortWrapper
> cc AcpiTableOptions
> cc ActionCompletionMode
> cc AddfdInfo
> cc AnnounceParameters
> cc AudioFormat
> cc Audiodev
> cc AudiodevAlsaOptions
> cc AudiodevAlsaPerDirectionOptions
> cc AudiodevCoreaudioOptions
> cc AudiodevCoreaudioPerDirectionOptions
> cc AudiodevDriver
> cc AudiodevDsoundOptions
> cc AudiodevGenericOptions
> cc AudiodevJackOptions
> cc AudiodevJackPerDirectionOptions
> cc AudiodevOssOptions
> cc AudiodevOssPerDirectionOptions
> cc AudiodevPaOptions
> cc AudiodevPaPerDirectionOptions
> cc AudiodevPerDirectionOptions
> cc AudiodevSdlOptions
> cc AudiodevSdlPerDirectionOptions
> cc AudiodevWavOptions
> cc AuthZListFileProperties
> cc AuthZListProperties
> cc AuthZPAMProperties
> cc AuthZSimpleProperties
> cc BackupCommon
> cc BackupPerf
> cc BalloonInfo
> cc BiosAtaTranslation
> cc BitmapMigrationBitmapAlias
> cc BitmapMigrationBitmapAliasTransform
> cc BitmapMigrationNodeAlias
> cc BitmapSyncMode
> cc BlkdebugEvent
> cc BlkdebugIOType
> cc BlkdebugInjectErrorOptions
> cc BlkdebugSetStateOptions
> cc BlockDeviceInfo
> cc BlockDeviceIoStatus
> cc BlockDeviceStats
> cc BlockDeviceTimedStats
> cc BlockDirtyBitmap
> cc BlockDirtyBitmapAdd
> cc BlockDirtyBitmapAddWrapper
> cc BlockDirtyBitmapMerge
> cc BlockDirtyBitmapMergeWrapper
> cc BlockDirtyBitmapOrStr
> cc BlockDirtyBitmapSha256
> cc BlockDirtyBitmapWrapper
> cc BlockDirtyInfo
> cc BlockErrorAction
> cc BlockExportInfo
> cc BlockExportOptions
> cc BlockExportOptionsFuse
> cc BlockExportOptionsNbd
> cc BlockExportOptionsNbdBase
> cc BlockExportOptionsVhostUserBlk
> cc BlockExportRemoveMode
> cc BlockExportType
> cc BlockIOThrottle
> cc BlockInfo
> cc BlockJobInfo
> cc BlockLatencyHistogramInfo
> cc BlockMeasureInfo
> cc BlockPermission
> cc BlockStats
> cc BlockStatsSpecific
> cc BlockStatsSpecificFile
> cc BlockStatsSpecificNvme
> cc BlockdevAioOptions
> cc BlockdevAmendOptions
> cc BlockdevAmendOptionsLUKS
> cc BlockdevAmendOptionsQcow2
> cc BlockdevBackup
> cc BlockdevBackupWrapper
> cc BlockdevCacheInfo
> cc BlockdevCacheOptions
> cc BlockdevChangeReadOnlyMode
> cc BlockdevCreateOptions
> cc BlockdevCreateOptionsFile
> cc BlockdevCreateOptionsGluster
> cc BlockdevCreateOptionsLUKS
> cc BlockdevCreateOptionsNfs
> cc BlockdevCreateOptionsParallels
> cc BlockdevCreateOptionsQcow
> cc BlockdevCreateOptionsQcow2
> cc BlockdevCreateOptionsQed
> cc BlockdevCreateOptionsRbd
> cc BlockdevCreateOptionsSsh
> cc BlockdevCreateOptionsVdi
> cc BlockdevCreateOptionsVhdx
> cc BlockdevCreateOptionsVmdk
> cc BlockdevCreateOptionsVpc
> cc BlockdevDetectZeroesOptions
> cc BlockdevDiscardOptions
> cc BlockdevDriver
> cc BlockdevOnError
> cc BlockdevOptions
> cc BlockdevOptionsBlkdebug
> cc BlockdevOptionsBlklogwrites
> cc BlockdevOptionsBlkreplay
> cc BlockdevOptionsBlkverify
> cc BlockdevOptionsCbw
> cc BlockdevOptionsCor
> cc BlockdevOptionsCurlBase
> cc BlockdevOptionsCurlFtp
> cc BlockdevOptionsCurlFtps
> cc BlockdevOptionsCurlHttp
> cc BlockdevOptionsCurlHttps
> cc BlockdevOptionsFile
> cc BlockdevOptionsGenericCOWFormat
> cc BlockdevOptionsGenericFormat
> cc BlockdevOptionsGluster
> cc BlockdevOptionsIscsi
> cc BlockdevOptionsLUKS
> cc BlockdevOptionsNVMe
> cc BlockdevOptionsNbd
> cc BlockdevOptionsNfs
> cc BlockdevOptionsNull
> cc BlockdevOptionsPreallocate
> cc BlockdevOptionsQcow
> cc BlockdevOptionsQcow2
> cc BlockdevOptionsQuorum
> cc BlockdevOptionsRaw
> cc BlockdevOptionsRbd
> cc BlockdevOptionsReplication
> cc BlockdevOptionsSsh
> cc BlockdevOptionsThrottle
> cc BlockdevOptionsVVFAT
> cc BlockdevQcow2Encryption
> cc BlockdevQcow2EncryptionFormat
> cc BlockdevQcow2Version
> cc BlockdevQcowEncryption
> cc BlockdevQcowEncryptionFormat
> cc BlockdevRef
> cc BlockdevRefOrNull
> cc BlockdevSnapshot
> cc BlockdevSnapshotInternal
> cc BlockdevSnapshotInternalWrapper
> cc BlockdevSnapshotSync
> cc BlockdevSnapshotSyncWrapper
> cc BlockdevSnapshotWrapper
> cc BlockdevVhdxSubformat
> cc BlockdevVmdkAdapterType
> cc BlockdevVmdkSubformat
> cc BlockdevVpcSubformat
> cc CanHostSocketcanProperties
> cc ChardevBackend
> cc ChardevBackendInfo
> cc ChardevBackendKind
> cc ChardevCommon
> cc ChardevCommonWrapper
> cc ChardevDBus
> cc ChardevDBusWrapper
> cc ChardevFile
> cc ChardevFileWrapper
> cc ChardevHostdev
> cc ChardevHostdevWrapper
> cc ChardevInfo
> cc ChardevMux
> cc ChardevMuxWrapper
> cc ChardevQemuVDAgent
> cc ChardevQemuVDAgentWrapper
> cc ChardevReturn
> cc ChardevRingbuf
> cc ChardevRingbufWrapper
> cc ChardevSocket
> cc ChardevSocketWrapper
> cc ChardevSpiceChannel
> cc ChardevSpiceChannelWrapper
> cc ChardevSpicePort
> cc ChardevSpicePortWrapper
> cc ChardevStdio
> cc ChardevStdioWrapper
> cc ChardevUdp
> cc ChardevUdpWrapper
> cc ChardevVC
> cc ChardevVCWrapper
> cc ColoCompareProperties
> cc CommandInfo
> cc CommandLineOptionInfo
> cc CommandLineParameterInfo
> cc CommandLineParameterType
> cc CommandNotFound
> cc CompatPolicy
> cc CompatPolicyInput
> cc CompatPolicyOutput
> cc CompressionStats
> cc CpuDefinitionInfo
> cc CpuInfoFast
> cc CpuInfoS390
> cc CpuInstanceProperties
> cc CpuModelBaselineInfo
> cc CpuModelCompareInfo
> cc CpuModelCompareResult
> cc CpuModelExpansionInfo
> cc CpuModelExpansionType
> cc CpuModelInfo
> cc CpuS390State
> cc CryptodevBackendProperties
> cc CryptodevVhostUserProperties
> cc CurrentMachineParams
> cc DataFormat
> cc DeviceNotActive
> cc DeviceNotFound
> cc DirtyRateInfo
> cc DirtyRateMeasureMode
> cc DirtyRateStatus
> cc DirtyRateVcpu
> cc DisplayCocoa
> cc DisplayCurses
> cc DisplayDBus
> cc DisplayEGLHeadless
> cc DisplayGLMode
> cc DisplayGTK
> cc DisplayOptions
> cc DisplayProtocol
> cc DisplayReloadOptions
> cc DisplayReloadOptionsVNC
> cc DisplayReloadType
> cc DisplayType
> cc DisplayUpdateOptions
> cc DisplayUpdateOptionsVNC
> cc DisplayUpdateType
> cc DriveBackup
> cc DriveBackupWrapper
> cc DriveMirror
> cc DummyForceArrays
> cc DumpGuestMemoryCapability
> cc DumpGuestMemoryFormat
> cc DumpQueryResult
> cc DumpStatus
> cc EventLoopBaseProperties
> cc ExpirePasswordOptions
> cc ExpirePasswordOptionsVnc
> cc FailoverStatus
> cc FdsetFdInfo
> cc FdsetInfo
> cc FilterBufferProperties
> cc FilterDumpProperties
> cc FilterMirrorProperties
> cc FilterRedirectorProperties
> cc FilterRewriterProperties
> cc FloppyDriveType
> cc FuseExportAllowOther
> cc GenericError
> cc GrabToggleKeys
> cc GuestPanicAction
> cc GuestPanicInformation
> cc GuestPanicInformationHyperV
> cc GuestPanicInformationS390
> cc GuestPanicInformationType
> cc GuidInfo
> cc HmatCacheAssociativity
> cc HmatCacheWritePolicy
> cc HmatLBDataType
> cc HmatLBMemoryHierarchy
> cc HostMemPolicy
> cc HotpluggableCPU
> cc HumanReadableText
> cc ImageCheck
> cc ImageFormat
> cc ImageInfo
> cc ImageInfoSpecific
> cc ImageInfoSpecificKind
> cc ImageInfoSpecificLUKSWrapper
> cc ImageInfoSpecificQCow2
> cc ImageInfoSpecificQCow2Encryption
> cc ImageInfoSpecificQCow2EncryptionBase
> cc ImageInfoSpecificQCow2Wrapper
> cc ImageInfoSpecificRbd
> cc ImageInfoSpecificRbdWrapper
> cc ImageInfoSpecificVmdk
> cc ImageInfoSpecificVmdkWrapper
> cc InetSocketAddress
> cc InetSocketAddressBase
> cc InetSocketAddressWrapper
> cc InputAxis
> cc InputBarrierProperties
> cc InputBtnEvent
> cc InputBtnEventWrapper
> cc InputButton
> cc InputEvent
> cc InputEventKind
> cc InputKeyEvent
> cc InputKeyEventWrapper
> cc InputLinuxProperties
> cc InputMoveEvent
> cc InputMoveEventWrapper
> cc IntWrapper
> cc IoOperationType
> cc IothreadProperties
> cc IscsiHeaderDigest
> cc IscsiTransport
> cc JobInfo
> cc JobStatus
> cc JobType
> cc JobVerb
> cc KeyValue
> cc KeyValueKind
> cc KvmInfo
> cc LostTickPolicy
> cc MachineInfo
> cc MainLoopProperties
> cc MapEntry
> cc Memdev
> cc MemoryBackendEpcProperties
> cc MemoryBackendFileProperties
> cc MemoryBackendMemfdProperties
> cc MemoryBackendProperties
> cc MemoryDeviceInfo
> cc MemoryDeviceInfoKind
> cc MemoryFailureAction
> cc MemoryFailureFlags
> cc MemoryFailureRecipient
> cc MemoryInfo
> cc MigrateSetParameters
> cc MigrationCapability
> cc MigrationCapabilityStatus
> cc MigrationInfo
> cc MigrationParameter
> cc MigrationParameters
> cc MigrationStats
> cc MigrationStatus
> cc MirrorCopyMode
> cc MirrorSyncMode
> cc MonitorMode
> cc MonitorOptions
> cc MouseInfo
> cc MultiFDCompression
> cc NameInfo
> cc NbdServerAddOptions
> cc NbdServerOptions
> cc NetClientDriver
> cc NetFilterDirection
> cc NetLegacyNicOptions
> cc Netdev
> cc NetdevBridgeOptions
> cc NetdevHubPortOptions
> cc NetdevL2TPv3Options
> cc NetdevNetmapOptions
> cc NetdevSocketOptions
> cc NetdevTapOptions
> cc NetdevUserOptions
> cc NetdevVdeOptions
> cc NetdevVhostUserOptions
> cc NetdevVhostVDPAOptions
> cc NetfilterInsert
> cc NetfilterProperties
> cc NetworkAddressFamily
> cc NewImageMode
> cc NumaCpuOptions
> cc NumaDistOptions
> cc NumaHmatCacheOptions
> cc NumaHmatLBOptions
> cc NumaNodeOptions
> cc NumaOptions
> cc NumaOptionsType
> cc ObjectOptions
> cc ObjectPropertyInfo
> cc ObjectType
> cc ObjectTypeInfo
> cc OffAutoPCIBAR
> cc OnOffAuto
> cc OnOffSplit
> cc PanicAction
> cc PciBridgeInfo
> cc PciBusInfo
> cc PciDeviceClass
> cc PciDeviceId
> cc PciDeviceInfo
> cc PciInfo
> cc PciMemoryRange
> cc PciMemoryRegion
> cc PrManagerHelperProperties
> cc PreallocMode
> cc QapiErrorClass
> cc Qcow2BitmapInfo
> cc Qcow2BitmapInfoFlags
> cc Qcow2CompressionType
> cc Qcow2OverlapCheckFlags
> cc Qcow2OverlapCheckMode
> cc Qcow2OverlapChecks
> cc QtestProperties
> cc QuorumOpType
> cc QuorumReadPattern
> cc RbdAuthMode
> cc RbdEncryptionCreateOptions
> cc RbdEncryptionCreateOptionsLUKS
> cc RbdEncryptionCreateOptionsLUKS2
> cc RbdEncryptionCreateOptionsLUKSBase
> cc RbdEncryptionOptions
> cc RbdEncryptionOptionsLUKS
> cc RbdEncryptionOptionsLUKS2
> cc RbdEncryptionOptionsLUKSBase
> cc RbdImageEncryptionFormat
> cc RebootAction
> cc RemoteObjectProperties
> cc ReplayInfo
> cc ReplayMode
> cc ReplicationMode
> cc ReplicationStatus
> cc RngEgdProperties
> cc RngProperties
> cc RngRandomProperties
> cc RockerOfDpaFlow
> cc RockerOfDpaFlowAction
> cc RockerOfDpaFlowKey
> cc RockerOfDpaFlowMask
> cc RockerOfDpaGroup
> cc RockerPort
> cc RockerPortAutoneg
> cc RockerPortDuplex
> cc RockerSwitch
> cc RunState
> cc RxFilterInfo
> cc RxState
> cc SchemaInfo
> cc SchemaInfoAlternate
> cc SchemaInfoAlternateMember
> cc SchemaInfoArray
> cc SchemaInfoBuiltin
> cc SchemaInfoCommand
> cc SchemaInfoEnum
> cc SchemaInfoEnumMember
> cc SchemaInfoEvent
> cc SchemaInfoObject
> cc SchemaInfoObjectMember
> cc SchemaInfoObjectVariant
> cc SchemaMetaType
> cc SecretCommonProperties
> cc SecretKeyringProperties
> cc SecretProperties
> cc SetPasswordAction
> cc SetPasswordOptions
> cc SetPasswordOptionsVnc
> cc SevAttestationReport
> cc SevCapability
> cc SevGuestProperties
> cc SevInfo
> cc SevLaunchMeasureInfo
> cc SevState
> cc SgxEPC
> cc SgxEPCDeviceInfo
> cc SgxEPCDeviceInfoWrapper
> cc SgxEPCProperties
> cc ShutdownAction
> cc ShutdownCause
> cc SmbiosEntryPointType
> cc SnapshotInfo
> cc SocketAddress
> cc SocketAddressLegacy
> cc SocketAddressType
> cc SpiceBasicInfo
> cc SpiceChannel
> cc SpiceInfo
> cc SpiceQueryMouseMode
> cc SpiceServerInfo
> cc SshHostKeyCheck
> cc SshHostKeyCheckHashType
> cc SshHostKeyCheckMode
> cc SshHostKeyHash
> cc StatusInfo
> cc StrOrNull
> cc String
> cc StringWrapper
> cc SysEmuTarget
> cc TargetInfo
> cc ThrottleGroupProperties
> cc ThrottleLimits
> cc TlsCredsAnonProperties
> cc TlsCredsProperties
> cc TlsCredsPskProperties
> cc TlsCredsX509Properties
> cc TpmModel
> cc TpmType
> cc TpmTypeOptions
> cc TraceEventInfo
> cc TraceEventState
> cc TransactionAction
> cc TransactionActionKind
> cc TransactionProperties
> cc UnixSocketAddress
> cc UnixSocketAddressWrapper
> cc UuidInfo
> cc VersionInfo
> cc VersionTriple
> cc VfioStats
> cc VirtioMEMDeviceInfo
> cc VirtioMEMDeviceInfoWrapper
> cc VirtioPMEMDeviceInfo
> cc VirtioPMEMDeviceInfoWrapper
> cc VncBasicInfo
> cc VncClientInfo
> cc VncInfo
> cc VncInfo2
> cc VncPrimaryAuth
> cc VncServerInfo
> cc VncServerInfo2
> cc VncVencryptSubAuth
> cc VsockSocketAddress
> cc VsockSocketAddressWrapper
> cc WatchdogAction
> cc YankInstance
> cc YankInstanceBlockNode
> cc YankInstanceChardev
> cc YankInstanceType
> lc a
> lc aarch64
> lc abort
> lc aborting
> lc abs
> lc absolute
> lc absolute-paths
> lc abstract
> lc accept
> lc access-bandwidth
> lc access-latency
> lc action
> lc action-required
> lc actions
> lc active
> lc active-l1
> lc active-l2
> lc actual
> lc actual-size
> lc adapter-type
> lc add-fd
> lc addr
> lc address
> lc addresses
> lc aes
> lc aes-128
> lc aes-192
> lc aes-256
> lc again
> lc aio
> lc aio-max-batch
> lc alias
> lc alias-of
> lc align
> lc aligned-accesses
> lc all
> lc allocated-clusters
> lc allocation-depth
> lc allow
> lc allow-oob
> lc allow-other
> lc allow-write-only-overlay
> lc alpha
> lc alsa
> lc alt
> lc alt-alt
> lc alternate
> lc always
> lc amend
> lc amount-exceeded
> lc announce-initial
> lc announce-max
> lc announce-rounds
> lc announce-self
> lc announce-step
> lc api-major
> lc api-minor
> lc apostrophe
> lc append
> lc arch
> lc arg-type
> lc arg1
> lc arg2
> lc arg3
> lc arg4
> lc arg5
> lc arm
> lc array
> lc associativity
> lc asterisk
> lc audiodev
> lc audiomute
> lc audionext
> lc audioplay
> lc audioprev
> lc audiostop
> lc auth
> lc auth-client-required
> lc authz-list
> lc authz-listfile
> lc authz-pam
> lc authz-simple
> lc auto
> lc auto-converge
> lc auto-dismiss
> lc auto-finalize
> lc auto-read-only
> lc autoneg
> lc avr
> lc axis
> lc b
> lc backend
> lc background
> lc background-snapshot
> lc backing
> lc backing-file
> lc backing-filename
> lc backing-filename-format
> lc backing-fmt
> lc backing-image
> lc backslash
> lc backspace
> lc backup
> lc balloon
> lc bandwidth
> lc bar
> lc bar0
> lc bar1
> lc bar2
> lc bar3
> lc bar4
> lc bar5
> lc base
> lc base-memory
> lc base-node
> lc base64
> lc before
> lc begin
> lc behind
> lc bind
> lc bins
> lc bitmap
> lc bitmap-directory
> lc bitmap-mode
> lc bitmaps
> lc blk
> lc blkdebug
> lc blklogwrites
> lc blkreplay
> lc blkverify
> lc block
> lc block-backend
> lc block-bitmap-mapping
> lc block-commit
> lc block-dirty-bitmap-add
> lc block-dirty-bitmap-clear
> lc block-dirty-bitmap-disable
> lc block-dirty-bitmap-enable
> lc block-dirty-bitmap-merge
> lc block-dirty-bitmap-remove
> lc block-driver
> lc block-export-add
> lc block-export-del
> lc block-incremental
> lc block-job
> lc block-job-cancel
> lc block-job-complete
> lc block-job-dismiss
> lc block-job-finalize
> lc block-job-pause
> lc block-job-resume
> lc block-job-set-speed
> lc block-latency-histogram-set
> lc block-node
> lc block-set-write-threshold
> lc block-size
> lc block-state-zero
> lc block-status
> lc block-stream
> lc blockdev-add
> lc blockdev-backup
> lc blockdev-change-medium
> lc blockdev-close-tray
> lc blockdev-create
> lc blockdev-del
> lc blockdev-insert-medium
> lc blockdev-mirror
> lc blockdev-open-tray
> lc blockdev-remove-medium
> lc blockdev-reopen
> lc blockdev-snapshot
> lc blockdev-snapshot-delete-internal-sync
> lc blockdev-snapshot-internal-sync
> lc blockdev-snapshot-sync
> lc blocked-reasons
> lc bochs
> lc boolean
> lc bootfile
> lc bottom
> lc boundaries
> lc boundaries-flush
> lc boundaries-read
> lc boundaries-write
> lc bps
> lc bps-read
> lc bps-read-max
> lc bps-read-max-length
> lc bps-total
> lc bps-total-max
> lc bps-total-max-length
> lc bps-write
> lc bps-write-max
> lc bps-write-max-length
> lc br
> lc braille
> lc bridge
> lc broadcast-allowed
> lc btn
> lc buf-size
> lc buffer-count
> lc buffer-length
> lc build-id
> lc builtin
> lc bus
> lc buslogic
> lc busy
> lc busy-rate
> lc button
> lc bytes
> lc c
> lc cache
> lc cache-clean-interval
> lc cache-miss
> lc cache-miss-rate
> lc cache-size
> lc cached
> lc calc-dirty-rate
> lc calc-time
> lc calculator
> lc can-bus
> lc can-host-socketcan
> lc canbus
> lc cancel
> lc cancel-path
> lc cancelled
> lc cancelling
> lc capabilities
> lc capability
> lc case
> lc cast5-128
> lc cbc
> lc cbitpos
> lc cephx
> lc cert-chain
> lc cert-subject
> lc change-backing-file
> lc change-vnc-password
> lc channel-id
> lc channel-type
> lc channels
> lc chardev
> lc chardev-add
> lc chardev-change
> lc chardev-remove
> lc chardev-send-break
> lc charset
> lc check-errors
> lc check-stop
> lc checkpoint-ready
> lc checkpoint-reply
> lc checkpoint-request
> lc child
> lc children
> lc cid
> lc cipher-alg
> lc cipher-mode
> lc class
> lc client
> lc client-name
> lc clients
> lc clipboard
> lc cloop
> lc closefd
> lc cluster-id
> lc cluster-size
> lc clusters
> lc cocoa
> lc colo
> lc colo-compare
> lc cols
> lc comma
> lc command
> lc command-line
> lc commit
> lc compat
> lc compiled-version
> lc complete
> lc completed
> lc completion-errors
> lc completion-mode
> lc complex
> lc compose
> lc compress
> lc compress-level
> lc compress-threads
> lc compress-wait-thread
> lc compressed
> lc compressed-clusters
> lc compressed-size
> lc compression
> lc compression-rate
> lc compression-type
> lc computer
> lc concluded
> lc conf
> lc config
> lc connect
> lc connect-ports
> lc connected
> lc connection-id
> lc consistent-read
> lc console
> lc constant
> lc cont
> lc control
> lc cookie
> lc cookie-secret
> lc cookie64
> lc copy
> lc copy-before-write
> lc copy-mode
> lc copy-on-read
> lc core
> lc core-id
> lc coreaudio
> lc cores
> lc corrupt
> lc corruptions
> lc corruptions-fixed
> lc count
> lc counter
> lc cpu
> lc cpu-index
> lc cpu-max
> lc cpu-state
> lc cpu-throttle-increment
> lc cpu-throttle-initial
> lc cpu-throttle-percentage
> lc cpu-throttle-tailslow
> lc cpu0-id
> lc cpuid-input-eax
> lc cpuid-input-ecx
> lc cpuid-register
> lc cpus
> lc crash
> lc crc32c
> lc crc32c-none
> lc create
> lc create-type
> lc created
> lc cris
> lc cryptodev-backend
> lc cryptodev-backend-builtin
> lc cryptodev-vhost-user
> lc ctr
> lc ctrl
> lc ctrl-ctrl
> lc ctrl-scrolllock
> lc current
> lc current-progress
> lc curses
> lc cut
> lc d
> lc d0
> lc d1
> lc d12
> lc d120
> lc d144
> lc d16
> lc d2
> lc d288
> lc d3
> lc d32
> lc d3des
> lc d4
> lc d5
> lc d6
> lc d64
> lc d7
> lc d8
> lc d9
> lc data
> lc data-file
> lc data-file-raw
> lc data-type
> lc date-nsec
> lc date-sec
> lc dbus
> lc dbus-vmstate
> lc debug
> lc decompress-threads
> lc default
> lc default-cpu-type
> lc default-ram-id
> lc default-value
> lc definition
> lc delay
> lc delete
> lc deny
> lc deprecated
> lc deprecated-input
> lc deprecated-output
> lc depth
> lc des
> lc desc
> lc description
> lc detach
> lc detect-zeroes
> lc dev
> lc device
> lc device-id
> lc device-list-properties
> lc devices
> lc devid
> lc devname
> lc dh-cert-file
> lc dhcpstart
> lc die-id
> lc dies
> lc dimm
> lc dir
> lc direct
> lc dirty-bitmap
> lc dirty-bitmaps
> lc dirty-flag
> lc dirty-pages-rate
> lc dirty-rate
> lc dirty-ring
> lc dirty-sync-count
> lc disabled
> lc disabled-wait
> lc discard
> lc discard-bytes-ok
> lc discard-data
> lc discard-nb-failed
> lc discard-nb-ok
> lc disconnect
> lc disk
> lc dismiss
> lc display
> lc display-reload
> lc display-update
> lc dist
> lc dmg
> lc dns
> lc dnssearch
> lc domainname
> lc dot
> lc down
> lc downscript
> lc downtime
> lc downtime-bytes
> lc downtime-limit
> lc drive-backup
> lc drive-mirror
> lc driver
> lc driver-specific
> lc drop-cache
> lc drv
> lc dsound
> lc dsp-policy
> lc dst
> lc dstport
> lc dump
> lc dump-guest-memory
> lc dump-skeys
> lc duplex
> lc duplicate
> lc dynamic
> lc dynamic-auto-read-only
> lc e
> lc ecb
> lc edges
> lc egl-headless
> lc eject
> lc element-type
> lc elf
> lc emulated
> lc emulator
> lc enable
> lc enabled
> lc encoding-rate
> lc encrypt
> lc encrypted
> lc encryption-format
> lc end
> lc endpoint
> lc enospc
> lc enum
> lc equal
> lc errno
> lc error
> lc error-desc
> lc es
> lc esc
> lc essiv
> lc eth-dst
> lc eth-src
> lc eth-type
> lc evdev
> lc event
> lc events
> lc exact
> lc exact-name
> lc exclusive
> lc existing
> lc expected-downtime
> lc export
> lc extended-l2
> lc extent-size-hint
> lc extents
> lc external
> lc extint-loop
> lc extra
> lc f
> lc f1
> lc f10
> lc f11
> lc f12
> lc f2
> lc f3
> lc f32
> lc f4
> lc f5
> lc f6
> lc f7
> lc f8
> lc f9
> lc fail
> lc failed
> lc failover
> lc falloc
> lc family
> lc fat-type
> lc fatal
> lc fd
> lc fdname
> lc fds
> lc fdset-id
> lc features
> lc fifo
> lc file
> lc filename
> lc filter-buffer
> lc filter-dump
> lc filter-mirror
> lc filter-node-name
> lc filter-redirector
> lc filter-replay
> lc filter-rewriter
> lc finalize
> lc find
> lc finish-migrate
> lc first-level
> lc fixed
> lc fixed-iothread
> lc fixed-settings
> lc flags
> lc flat
> lc flc
> lc floppy
> lc flush
> lc force
> lc force-share
> lc force-size
> lc format
> lc format-specific
> lc formats
> lc fqdn
> lc fragmented-clusters
> lc frequency
> lc front
> lc frontend-open
> lc ftp
> lc ftps
> lc full
> lc full-backing-filename
> lc full-grab
> lc full-screen
> lc fully-allocated
> lc function
> lc fuse
> lc g
> lc getfd
> lc gid-status
> lc gl
> lc glob
> lc gluster
> lc goto-tbl
> lc gpa
> lc grab-on-hover
> lc grab-toggle
> lc granularity
> lc group
> lc group-id
> lc group-ids
> lc grouped
> lc growable
> lc gtk
> lc guest
> lc guest-panic
> lc guest-panicked
> lc guest-reset
> lc guest-shutdown
> lc guestfwd
> lc guid
> lc h
> lc half
> lc handle
> lc hard
> lc hash
> lc hash-alg
> lc head
> lc header-digest
> lc height
> lc help
> lc helper
> lc henkan
> lc hide
> lc hierarchy
> lc hiragana
> lc hits
> lc hmat-cache
> lc hmat-lb
> lc hold-time
> lc home
> lc host
> lc host-error
> lc host-key-check
> lc host-nodes
> lc host-qmp-quit
> lc host-qmp-system-reset
> lc host-signal
> lc host-ui
> lc hostfwd
> lc hostname
> lc hotpluggable
> lc hotpluggable-cpus
> lc hotplugged
> lc hppa
> lc http
> lc https
> lc hubid
> lc hubport
> lc hugetlb
> lc hugetlbsize
> lc human-monitor-command
> lc human-readable-text
> lc hwversion
> lc hyper-v
> lc hypervisor
> lc i
> lc i386
> lc icount
> lc id
> lc id-list
> lc ide
> lc identical
> lc identity
> lc if
> lc ifname
> lc ignore
> lc ignore-unavailable
> lc image
> lc image-end-offset
> lc image-node-name
> lc immediately
> lc implements
> lc in
> lc in-pport
> lc in-use
> lc inactive
> lc inactive-l1
> lc inactive-l2
> lc inc
> lc incompatible
> lc inconsistent
> lc incremental
> lc indev
> lc index
> lc individual
> lc inet
> lc info
> lc initial
> lc initiator
> lc initiator-name
> lc inject
> lc inject-error
> lc inject-nmi
> lc inmigrate
> lc input-barrier
> lc input-linux
> lc input-send-event
> lc insert
> lc inserted
> lc instances
> lc int
> lc interface-id
> lc interfaces
> lc interleave
> lc internal-error
> lc interval
> lc io-error
> lc io-status
> lc iops
> lc iops-read
> lc iops-read-max
> lc iops-read-max-length
> lc iops-size
> lc iops-total
> lc iops-total-max
> lc iops-total-max-length
> lc iops-write
> lc iops-write-max
> lc iops-write-max-length
> lc iothread
> lc iotype
> lc ip
> lc ip-dst
> lc ip-proto
> lc ip-tos
> lc ipv4
> lc ipv6
> lc ipv6-dns
> lc ipv6-host
> lc ipv6-prefix
> lc ipv6-prefixlen
> lc irq
> lc is-default
> lc iscsi
> lc iser
> lc iter-time
> lc iters
> lc iv
> lc ivgen-alg
> lc ivgen-hash-alg
> lc j
> lc jack
> lc job-cancel
> lc job-complete
> lc job-dismiss
> lc job-finalize
> lc job-id
> lc job-pause
> lc job-resume
> lc json-cli
> lc json-cli-hotplug
> lc json-type
> lc k
> lc katakanahiragana
> lc kdump-lzo
> lc kdump-snappy
> lc kdump-zlib
> lc keep
> lc keep-alive
> lc kernel
> lc kernel-hashes
> lc key
> lc key-offset
> lc key-secret
> lc keyid
> lc keys
> lc keyslot
> lc l
> lc l2-cache-entry-size
> lc l2-cache-size
> lc l2tpv3
> lc label
> lc lang1
> lc lang2
> lc large
> lc last-mode
> lc late-block-activate
> lc latency
> lc latency-ns
> lc launch-secret
> lc launch-update
> lc lazy-refcounts
> lc lba
> lc leaks
> lc leaks-fixed
> lc left
> lc left-command-key
> lc len
> lc length
> lc less
> lc level
> lc lf
> lc limit
> lc limits
> lc line
> lc link-up
> lc listen
> lc live
> lc load
> lc loaded
> lc local
> lc localaddr
> lc location
> lc locked
> lc locking
> lc log
> lc log-append
> lc log-sector-size
> lc log-size
> lc log-super-update-interval
> lc logappend
> lc logfile
> lc logical-block-size
> lc lsilogic
> lc luks
> lc luks2
> lc lun
> lc m
> lc m68k
> lc macaddr
> lc mail
> lc main-header
> lc main-loop
> lc main-mac
> lc major
> lc mask
> lc master-key-iters
> lc match
> lc max
> lc max-bandwidth
> lc max-chunk
> lc max-connections
> lc max-cpu-throttle
> lc max-discard
> lc max-postcopy-bandwidth
> lc max-size
> lc max-transfer
> lc max-workers
> lc max-write-zero
> lc maxcpus
> lc maxlen
> lc mbps
> lc mcast
> lc md5
> lc measured
> lc measuring
> lc mediaselect
> lc mem
> lc mem-path
> lc memaddr
> lc members
> lc memdev
> lc memory
> lc memory-backend-epc
> lc memory-backend-file
> lc memory-backend-memfd
> lc memory-backend-ram
> lc memsave
> lc menu
> lc merge
> lc meta-meta
> lc meta-type
> lc metadata
> lc micro
> lc microblaze
> lc microblazeel
> lc middle
> lc migrate
> lc migrate-continue
> lc migrate-incoming
> lc migrate-pause
> lc migrate-recover
> lc migrate-set-capabilities
> lc migrate-set-parameters
> lc migrate-start-postcopy
> lc migrated
> lc migration
> lc migration-safe
> lc minor
> lc minus
> lc mips
> lc mips64
> lc mips64el
> lc mipsel
> lc mirror
> lc mixing-engine
> lc mnonce
> lc mode
> lc model
> lc modela
> lc modelb
> lc mountpoint
> lc mouse
> lc mouse-mode
> lc mptcp
> lc msg
> lc msmouse
> lc muhenkan
> lc multicast
> lc multicast-overflow
> lc multicast-table
> lc multifd
> lc multifd-bytes
> lc multifd-channels
> lc multifd-compression
> lc multifd-zlib-level
> lc multifd-zstd-level
> lc mux
> lc n
> lc name
> lc namespace
> lc native
> lc nbd
> lc nbd-server-add
> lc nbd-server-remove
> lc nbd-server-start
> lc nbd-server-stop
> lc net
> lc netdev
> lc netmap
> lc never
> lc new-secret
> lc new-vlan-id
> lc nfs
> lc nic
> lc nios2
> lc no-flush
> lc nocow
> lc node
> lc node-id
> lc node-name
> lc nodeid
> lc nodelay
> lc nodes
> lc none
> lc none-crc32c
> lc normal
> lc normal-bytes
> lc nospace
> lc null
> lc null-aio
> lc null-co
> lc num-queues
> lc numa-mem-supported
> lc number
> lc numeric
> lc nvdimm
> lc nvme
> lc o
> lc object
> lc object-add
> lc object-del
> lc off
> lc offset
> lc ok
> lc old-secret
> lc on
> lc on-error
> lc on-source-error
> lc on-success
> lc on-target-error
> lc once
> lc oob
> lc opaque
> lc open
> lc open-timeout
> lc opened
> lc operating
> lc operation
> lc opint-loop
> lc opt-discard
> lc opt-write-zero
> lc option
> lc options
> lc or1k
> lc oss
> lc out
> lc out-pport
> lc outdev
> lc overflow
> lc overlap-check
> lc overlay
> lc p
> lc p2p
> lc pa
> lc package
> lc packet-header
> lc page-cache-size
> lc page-sampling
> lc page-size
> lc pages
> lc pages-per-second
> lc paging
> lc panic
> lc parallel
> lc parallels
> lc parameters
> lc parent
> lc parent-cid
> lc pass
> lc pass-discard-other
> lc pass-discard-request
> lc pass-discard-snapshot
> lc passthrough
> lc password
> lc password-secret
> lc passwordid
> lc paste
> lc path
> lc pause
> lc pause-before-switchover
> lc paused
> lc payload-offset
> lc pdh
> lc pef-guest
> lc pending
> lc period-length
> lc perm
> lc persistent
> lc pgdn
> lc pgmint-loop
> lc pgup
> lc pincounter
> lc pipe
> lc plain
> lc plain64
> lc play
> lc plugged-memory
> lc pmem
> lc pmemsave
> lc png
> lc policy
> lc poll-grow
> lc poll-max-ns
> lc poll-shrink
> lc poll-us
> lc pool
> lc pop-vlan
> lc port
> lc portal
> lc ports
> lc position
> lc postcopy-active
> lc postcopy-blocktime
> lc postcopy-bytes
> lc postcopy-paused
> lc postcopy-ram
> lc postcopy-recover
> lc postcopy-requests
> lc postcopy-vcpu-blocktime
> lc postmigrate
> lc power
> lc poweroff
> lc ppc
> lc ppc64
> lc ppm
> lc pport
> lc pr-manager
> lc pr-manager-helper
> lc pre-switchover
> lc prealloc
> lc prealloc-align
> lc prealloc-size
> lc prealloc-threads
> lc preallocate
> lc preallocation
> lc precopy-bytes
> lc preferred
> lc prefetch
> lc prelaunch
> lc present
> lc pretty
> lc primary
> lc print
> lc priority
> lc processing
> lc promiscuous
> lc properties
> lc property
> lc props
> lc protocol
> lc proxy-password-secret
> lc proxy-username
> lc psw-addr
> lc psw-mask
> lc pty
> lc pwritev
> lc q
> lc qcode
> lc qcow
> lc qcow2
> lc qdev
> lc qed
> lc qemu
> lc qemu-vdagent
> lc qom-get
> lc qom-list
> lc qom-list-properties
> lc qom-list-types
> lc qom-path
> lc qom-set
> lc qom-type
> lc qtest
> lc query-acpi-ospm-status
> lc query-balloon
> lc query-block
> lc query-block-exports
> lc query-block-jobs
> lc query-blockstats
> lc query-chardev
> lc query-chardev-backends
> lc query-colo-status
> lc query-command-line-options
> lc query-commands
> lc query-cpu-definitions
> lc query-cpu-model-baseline
> lc query-cpu-model-comparison
> lc query-cpu-model-expansion
> lc query-cpus-fast
> lc query-current-machine
> lc query-dirty-rate
> lc query-display-options
> lc query-dump
> lc query-dump-guest-memory-capability
> lc query-fdsets
> lc query-gic-capabilities
> lc query-hotpluggable-cpus
> lc query-iothreads
> lc query-jobs
> lc query-kvm
> lc query-machines
> lc query-memdev
> lc query-memory-devices
> lc query-memory-size-summary
> lc query-mice
> lc query-migrate
> lc query-migrate-capabilities
> lc query-migrate-parameters
> lc query-name
> lc query-named-block-nodes
> lc query-nodes
> lc query-pci
> lc query-pr-managers
> lc query-qmp-schema
> lc query-replay
> lc query-rocker
> lc query-rocker-of-dpa-flows
> lc query-rocker-of-dpa-groups
> lc query-rocker-ports
> lc query-rx-filter
> lc query-sev
> lc query-sev-attestation-report
> lc query-sev-capabilities
> lc query-sev-launch-measure
> lc query-sgx
> lc query-sgx-capabilities
> lc query-spice
> lc query-status
> lc query-target
> lc query-tpm
> lc query-tpm-models
> lc query-tpm-types
> lc query-uuid
> lc query-version
> lc query-vm-generation-id
> lc query-vnc
> lc query-vnc-servers
> lc query-xen-replication-status
> lc query-yank
> lc queue
> lc queues
> lc quit
> lc quorum
> lc r
> lc ra2
> lc ra2ne
> lc ram
> lc raw
> lc rbd
> lc rdma-pin-all
> lc read
> lc read-bandwidth
> lc read-latency
> lc read-only
> lc read-only-mode
> lc read-pattern
> lc read-write
> lc read-zeroes
> lc readahead
> lc readahead-size
> lc readline
> lc readonly
> lc ready
> lc reason
> lc reboot
> lc receive-update
> lc rechs
> lc recipient
> lc reconnect
> lc reconnect-delay
> lc record
> lc recording
> lc recursive
> lc reduced-phys-bits
> lc refcount-bits
> lc refcount-block
> lc refcount-cache-size
> lc refcount-table
> lc reference
> lc refresh
> lc regions
> lc reject
> lc rel
> lc relaunch
> lc release-ram
> lc remaining
> lc remote
> lc removable
> lc remove-fd
> lc rendernode
> lc repeat
> lc replaces
> lc replay-break
> lc replay-delete-break
> lc replay-seek
> lc replication
> lc report
> lc request
> lc requested-size
> lc require
> lc required
> lc reserve
> lc reset
> lc resize
> lc responsible-properties
> lc restore-vm
> lc restrict
> lc result
> lc resume
> lc ret
> lc ret-type
> lc retain
> lc return-path
> lc rev
> lc rewrite-corrupted
> lc right
> lc ringbuf
> lc ringbuf-read
> lc ringbuf-write
> lc ripemd160
> lc riscv32
> lc riscv64
> lc rng-builtin
> lc rng-egd
> lc rng-random
> lc ro
> lc rounds
> lc rows
> lc rtc-reset-reinjection
> lc rules
> lc run
> lc running
> lc rw
> lc rx
> lc rxcookie
> lc rxsession
> lc s
> lc s16
> lc s32
> lc s390
> lc s390-pv-guest
> lc s390x
> lc s8
> lc safe
> lc sample-pages
> lc sanity-check
> lc sasl
> lc save-vm
> lc savevm-monitor-nodes
> lc screendump
> lc script
> lc scrolllock
> lc sdl
> lc seal
> lc second-level
> lc secondary
> lc secret
> lc section-size
> lc sections
> lc sector
> lc sector-num
> lc sectors-count
> lc semicolon
> lc send-key
> lc send-update
> lc serial
> lc serpent-128
> lc serpent-192
> lc serpent-256
> lc server
> lc server-name
> lc service
> lc session-file
> lc set-action
> lc set-eth-dst
> lc set-eth-src
> lc set-numa-node
> lc set-speed
> lc set-state
> lc set-vlan-id
> lc setup
> lc setup-time
> lc sev-device
> lc sev-guest
> lc sev-inject-launch-secret
> lc sgx
> lc sgx-epc
> lc sgx1
> lc sgx2
> lc sh4
> lc sh4eb
> lc sha1
> lc sha224
> lc sha256
> lc sha384
> lc sha512
> lc share
> lc shared-perm
> lc shift
> lc shift-shift
> lc show-cursor
> lc shutdown
> lc shutting-down
> lc side
> lc sig
> lc signal
> lc singlestep
> lc size
> lc skipauth
> lc skipped
> lc slash
> lc sleep
> lc slew
> lc slot
> lc slot-type
> lc slots
> lc smb
> lc smbserver
> lc snapshot
> lc snapshot-access
> lc snapshot-delete
> lc snapshot-file
> lc snapshot-load
> lc snapshot-node-name
> lc snapshot-save
> lc snapshot-table
> lc snapshots
> lc sndbuf
> lc sock
> lc socket
> lc socket-address
> lc socket-id
> lc sockets
> lc source
> lc sparc
> lc sparc64
> lc spc
> lc speed
> lc spice
> lc spice-app
> lc spiceport
> lc spicevmc
> lc split
> lc src
> lc srcport
> lc ssh
> lc sslverify
> lc standby
> lc start
> lc start-server
> lc start-time
> lc state
> lc static
> lc stats
> lc status
> lc stdio
> lc step
> lc stop
> lc stopped
> lc str
> lc stream
> lc stream-name
> lc string
> lc stripes
> lc subformat
> lc subnet-prefix
> lc subordinate
> lc subset
> lc subsystem
> lc subsystem-reset
> lc subsystem-vendor
> lc superset
> lc suspended
> lc swap-opt-cmd
> lc sync
> lc sysrq
> lc t
> lc tab
> lc table-size
> lc tag
> lc take-child-perms
> lc tap
> lc target
> lc tbl-id
> lc tcp
> lc tcp-syn-count
> lc telnet
> lc template
> lc test
> lc testdev
> lc tftp
> lc tftp-server-name
> lc third-level
> lc thread-id
> lc thread-pool-max
> lc thread-pool-min
> lc threads
> lc threshold
> lc throttle
> lc throttle-group
> lc throttle-trigger-threshold
> lc tight
> lc time
> lc timeout
> lc timer-period
> lc tls
> lc tls-authz
> lc tls-certs
> lc tls-cipher-suites
> lc tls-creds
> lc tls-creds-anon
> lc tls-creds-psk
> lc tls-creds-x509
> lc tls-hostname
> lc tls-none
> lc tls-plain
> lc tls-port
> lc tls-sasl
> lc tls-vnc
> lc tn3270
> lc to
> lc toolsversion
> lc top
> lc top-id
> lc top-node
> lc total
> lc total-clusters
> lc total-progress
> lc total-time
> lc tpm-crb
> lc tpm-spapr
> lc tpm-tis
> lc trace-event-get-state
> lc trace-event-set-state
> lc transaction
> lc transferred
> lc transform
> lc transport
> lc tray-open
> lc tricore
> lc try-mmap
> lc try-poll
> lc ttl-check
> lc tunnel-id
> lc tunnel-lport
> lc twofish-128
> lc twofish-192
> lc twofish-256
> lc tx
> lc txcookie
> lc txsession
> lc type
> lc typename
> lc u
> lc u16
> lc u32
> lc u8
> lc udp
> lc ultra
> lc unaligned-accesses
> lc unavailable
> lc unavailable-features
> lc undefined
> lc undo
> lc unicast
> lc unicast-overflow
> lc unicast-table
> lc uninit
> lc uninitialized
> lc unix
> lc unknown
> lc unmap
> lc unmapped
> lc unshare-child-perms
> lc unstable
> lc unstable-input
> lc unstable-output
> lc unstarted
> lc unused
> lc up
> lc uri
> lc url
> lc use-copy-range
> lc user
> lc username
> lc utf8
> lc uuid
> lc v
> lc v2
> lc v3
> lc val
> lc validate-uuid
> lc value
> lc values
> lc variants
> lc vc
> lc vcpu
> lc vcpu-dirty-rate
> lc vcpus-count
> lc vde
> lc vdi
> lc vectors
> lc vencrypt
> lc vendor
> lc verify-peer
> lc version
> lc vfio
> lc vhdx
> lc vhost
> lc vhost-user
> lc vhost-user-blk
> lc vhost-vdpa
> lc vhostdev
> lc vhostfd
> lc vhostfds
> lc vhostforce
> lc virtio-mem
> lc virtio-pmem
> lc virtual-size
> lc vlan
> lc vlan-id
> lc vlan-table
> lc vm-clock-nsec
> lc vm-clock-sec
> lc vm-state-size
> lc vmdk
> lc vmstate
> lc vmstate-loaded
> lc vmstate-received
> lc vmstate-send
> lc vmstate-size
> lc vnc
> lc voices
> lc volume
> lc volumedown
> lc volumeup
> lc vote-threshold
> lc vpc
> lc vsock
> lc vvfat
> lc w
> lc wait
> lc wait-unplug
> lc waiting
> lc wake
> lc wakeup-suspend-support
> lc watchdog
> lc watchdog-set-action
> lc wav
> lc wctablet
> lc websocket
> lc wheel-down
> lc wheel-left
> lc wheel-right
> lc wheel-up
> lc width
> lc win-dmp
> lc window-close
> lc writable
> lc write
> lc write-back
> lc write-bandwidth
> lc write-blocking
> lc write-latency
> lc write-threshold
> lc write-through
> lc write-unchanged
> lc write-zeroes
> lc writeback
> lc writethrough
> lc x
> lc x-blockdev-amend
> lc x-blockdev-change
> lc x-blockdev-set-iothread
> lc x-bps-read
> lc x-bps-read-max
> lc x-bps-read-max-length
> lc x-bps-total
> lc x-bps-total-max
> lc x-bps-total-max-length
> lc x-bps-write
> lc x-bps-write-max
> lc x-bps-write-max-length
> lc x-check-cache-dropped
> lc x-checkpoint-delay
> lc x-colo
> lc x-colo-lost-heartbeat
> lc x-debug-block-dirty-bitmap-sha256
> lc x-debug-query-block-graph
> lc x-dirty-bitmap
> lc x-exit-preconfig
> lc x-ignore-shared
> lc x-iops-read
> lc x-iops-read-max
> lc x-iops-read-max-length
> lc x-iops-size
> lc x-iops-total
> lc x-iops-total-max
> lc x-iops-total-max-length
> lc x-iops-write
> lc x-iops-write-max
> lc x-iops-write-max-length
> lc x-origin
> lc x-perf
> lc x-query-irq
> lc x-query-jit
> lc x-query-numa
> lc x-query-opcount
> lc x-query-profile
> lc x-query-ramblock
> lc x-query-rdma
> lc x-query-roms
> lc x-query-usb
> lc x-remote-object
> lc x-use-canonical-path-for-ramblock-id
> lc x509-none
> lc x509-plain
> lc x509-sasl
> lc x509-vnc
> lc xbzrle
> lc xbzrle-cache
> lc xbzrle-cache-size
> lc xen-colo-do-checkpoint
> lc xen-load-devices-state
> lc xen-save-devices-state
> lc xen-set-global-dirty-log
> lc xen-set-replication
> lc xtensa
> lc xtensaeb
> lc xts
> lc y
> lc y-origin
> lc yank
> lc yen
> lc z
> lc zero
> lc zero-blocks
> lc zeroed-grain
> lc zlib
> lc zoom-to-fit
> lc zstd
> mc ACPIOSTInfo
> mc ACPISlotType
> mc COLOExitReason
> mc COLOMessage
> mc COLOMode
> mc COLOStatus
> mc DBusVMStateProperties
> mc GICCapability
> mc IOThreadInfo
> mc JSONType
> mc KVMMissingCap
> mc NFSServer
> mc NFSTransport
> mc PCDIMMDeviceInfo
> mc PCDIMMDeviceInfoWrapper
> mc PCIELinkSpeed
> mc PCIELinkWidth
> mc PRManagerInfo
> mc QAuthZListFormat
> mc QAuthZListPolicy
> mc QAuthZListRule
> mc QCryptoBlockAmendOptions
> mc QCryptoBlockAmendOptionsLUKS
> mc QCryptoBlockCreateOptions
> mc QCryptoBlockCreateOptionsLUKS
> mc QCryptoBlockFormat
> mc QCryptoBlockInfo
> mc QCryptoBlockInfoBase
> mc QCryptoBlockInfoLUKS
> mc QCryptoBlockInfoLUKSSlot
> mc QCryptoBlockLUKSKeyslotState
> mc QCryptoBlockOpenOptions
> mc QCryptoBlockOptionsBase
> mc QCryptoBlockOptionsLUKS
> mc QCryptoBlockOptionsQCow
> mc QCryptoCipherAlgorithm
> mc QCryptoCipherMode
> mc QCryptoHashAlgorithm
> mc QCryptoIVGenAlgorithm
> mc QCryptoSecretFormat
> mc QCryptoTLSCredsEndpoint
> mc QKeyCode
> mc QKeyCodeWrapper
> mc QMPCapability
> mc S390CrashReason
> mc SGXEPCSection
> mc SGXInfo
> mc SMPConfiguration
> mc TPMEmulatorOptions
> mc TPMEmulatorOptionsWrapper
> mc TPMInfo
> mc TPMPassthroughOptions
> mc TPMPassthroughOptionsWrapper
> mc X86CPUFeatureWordInfo
> mc X86CPURegister32
> mc XBZRLECacheStats
> mc XDbgBlockGraph
> mc XDbgBlockGraphEdge
> mc XDbgBlockGraphNode
> mc XDbgBlockGraphNodeType
> mc ac_back
> mc ac_bookmarks
> mc ac_forward
> mc ac_home
> mc ac_refresh
> mc account_failed
> mc account_invalid
> mc add_client
> mc alt_r
> mc asl_compiler_id
> mc asl_compiler_rev
> mc avg_flush_latency_ns
> mc avg_rd_latency_ns
> mc avg_rd_queue_depth
> mc avg_wr_latency_ns
> mc avg_wr_queue_depth
> mc backing_file
> mc backing_file_depth
> mc block_resize
> mc block_set_io_throttle
> mc bps_max
> mc bps_max_length
> mc bps_rd
> mc bps_rd_max
> mc bps_rd_max_length
> mc bps_wr
> mc bps_wr_max
> mc bps_wr_max_length
> mc bracket_left
> mc bracket_right
> mc caps_lock
> mc class_info
> mc client_migrate_info
> mc cluster_alloc
> mc cluster_alloc_bytes
> mc cluster_alloc_space
> mc cluster_free
> mc compare_timeout
> mc cor_write
> mc cow_read
> mc cow_write
> mc ctrl_r
> mc d2_5
> mc detect_zeroes
> mc device_add
> mc device_del
> mc empty_image_prepare
> mc expire_password
> mc expired_scan_cycle
> mc failed_flush_operations
> mc failed_rd_operations
> mc failed_unmap_operations
> mc failed_wr_operations
> mc flush_latency_histogram
> mc flush_operations
> mc flush_to_disk
> mc flush_to_os
> mc flush_total_time_ns
> mc grab_all
> mc grave_accent
> mc host_cdrom
> mc host_device
> mc idle_time_ns
> mc interval_length
> mc invalid_flush_operations
> mc invalid_rd_operations
> mc invalid_unmap_operations
> mc invalid_wr_operations
> mc io_range
> mc io_uring
> mc iops_max
> mc iops_max_length
> mc iops_rd
> mc iops_rd_max
> mc iops_rd_max_length
> mc iops_size
> mc iops_wr
> mc iops_wr_max
> mc iops_wr_max_length
> mc irq_pin
> mc known_hosts
> mc kp_0
> mc kp_1
> mc kp_2
> mc kp_3
> mc kp_4
> mc kp_5
> mc kp_6
> mc kp_7
> mc kp_8
> mc kp_9
> mc kp_add
> mc kp_comma
> mc kp_decimal
> mc kp_divide
> mc kp_enter
> mc kp_equals
> mc kp_multiply
> mc kp_subtract
> mc l1_grow_activate_table
> mc l1_grow_alloc_table
> mc l1_grow_write_table
> mc l1_shrink_free_l2_clusters
> mc l1_shrink_write_table
> mc l1_update
> mc l2_alloc_cow_read
> mc l2_alloc_write
> mc l2_load
> mc l2_update
> mc l2_update_compressed
> mc legacyESX
> mc max_flush_latency_ns
> mc max_queue_size
> mc max_rd_latency_ns
> mc max_wr_latency_ns
> mc mem_type_64
> mc memory_range
> mc meta_l
> mc meta_r
> mc migrate_cancel
> mc min_flush_latency_ns
> mc min_rd_latency_ns
> mc min_wr_latency_ns
> mc monolithicFlat
> mc monolithicSparse
> mc netdev_add
> mc netdev_del
> mc new_state
> mc notify_dev
> mc num_lock
> mc oem_id
> mc oem_rev
> mc oem_table_id
> mc pci_bridge
> mc prefetchable_range
> mc primary_in
> mc pwritev_done
> mc pwritev_rmw_after_head
> mc pwritev_rmw_after_tail
> mc pwritev_rmw_head
> mc pwritev_rmw_tail
> mc pwritev_zero
> mc qdev_id
> mc qmp_capabilities
> mc rd_bytes
> mc rd_latency_histogram
> mc rd_merged
> mc rd_operations
> mc rd_total_time_ns
> mc read_aio
> mc read_backing_aio
> mc read_compressed
> mc refblock_alloc
> mc refblock_alloc_hookup
> mc refblock_alloc_switch_table
> mc refblock_alloc_write
> mc refblock_alloc_write_blocks
> mc refblock_alloc_write_table
> mc refblock_load
> mc refblock_update
> mc refblock_update_part
> mc reftable_grow
> mc reftable_load
> mc reftable_update
> mc sasl_username
> mc scroll_lock
> mc secondary_in
> mc secret_keyring
> mc set_link
> mc set_password
> mc shift_r
> mc streamOptimized
> mc system_powerdown
> mc system_reset
> mc system_wakeup
> mc timed_stats
> mc tray_open
> mc twoGbMaxExtentFlat
> mc twoGbMaxExtentSparse
> mc unmap_bytes
> mc unmap_merged
> mc unmap_operations
> mc unmap_total_time_ns
> mc vmstate_load
> mc vmstate_save
> mc vnet_hdr
> mc vnet_hdr_support
> mc wr_bytes
> mc wr_highest_offset
> mc wr_latency_histogram
> mc wr_merged
> mc wr_operations
> mc wr_total_time_ns
> mc write_aio
> mc write_compressed
> mc write_threshold
> mc x509_dname
> mc x86_64
> uc ACPI_DEVICE_OST
> uc BALLOON_CHANGE
> uc BLOCK_EXPORT_DELETED
> uc BLOCK_IMAGE_CORRUPTED
> uc BLOCK_IO_ERROR
> uc BLOCK_JOB_CANCELLED
> uc BLOCK_JOB_COMPLETED
> uc BLOCK_JOB_ERROR
> uc BLOCK_JOB_PENDING
> uc BLOCK_JOB_READY
> uc BLOCK_WRITE_THRESHOLD
> uc COLO_EXIT
> uc CPU
> uc DEVICE_DELETED
> uc DEVICE_TRAY_MOVED
> uc DEVICE_UNPLUG_GUEST_ERROR
> uc DIMM
> uc DUMP_COMPLETED
> uc EAX
> uc EBP
> uc EBX
> uc ECX
> uc EDI
> uc EDX
> uc ESI
> uc ESP
> uc FAILOVER_NEGOTIATED
> uc GUEST_CRASHLOADED
> uc GUEST_PANICKED
> uc JOB_STATUS_CHANGE
> uc MEMORY_DEVICE_SIZE_CHANGE
> uc MEMORY_FAILURE
> uc MEM_UNPLUG_ERROR
> uc MIGRATION
> uc MIGRATION_PASS
> uc NIC_RX_FILTER_CHANGED
> uc POWERDOWN
> uc PR_MANAGER_STATUS_CHANGED
> uc QUORUM_FAILURE
> uc QUORUM_REPORT_BAD
> uc RDMA_GID_STATUS_CHANGED
> uc RESET
> uc RESUME
> uc RTC_CHANGE
> uc SHUTDOWN
> uc SPICE_CONNECTED
> uc SPICE_DISCONNECTED
> uc SPICE_INITIALIZED
> uc SPICE_MIGRATE_COMPLETED
> uc STOP
> uc SUSPEND
> uc SUSPEND_DISK
> uc UNPLUG_PRIMARY
> uc UUID
> uc VNC_CONNECTED
> uc VNC_DISCONNECTED
> uc VNC_INITIALIZED
> uc VSERPORT_CHANGE
> uc WAKEUP
> uc WATCHDOG
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 8:53 ` Daniel P. Berrangé
2022-05-10 9:06 ` Victor Toso
@ 2022-05-10 12:02 ` Markus Armbruster
2022-05-10 12:34 ` Daniel P. Berrangé
1 sibling, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-05-10 12:02 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Victor Toso, John Snow, Eric Blake, qemu-devel, Marc-André Lureau
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Tue, Apr 26, 2022 at 01:14:28PM +0200, Markus Armbruster wrote:
>> Victor Toso <victortoso@redhat.com> writes:
>>
>> > Hi,
>> >
>> > Happy 1st April. Not a joke :) /* ugh, took me too long to send */
>> >
>> > This series is about adding a generator in scripts/qapi to produce
>> > Go data structures that can be used to communicate with QEMU over
>> > QMP.
>> >
>> >
>> > * Why Go?
>> >
>> > There are quite a few Go projects that interact with QEMU over QMP
>> > and they endup using a mix of different libraries with their own
>> > code.
>> >
>> >
>> > ** Which projects?
>> >
>> > The ones I've found so far:
>> >
>> > - podman machine
>> > https://github.com/containers/podman/tree/main/pkg/machine/qemu
>> >
>> > - kata-containers (govmm)
>> > https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm
>> >
>> > - lxd
>> > https://github.com/lxc/lxd/tree/master/lxd/instance/drivers
>> >
>> > - kubevirt (plain json strings)
>> > https://github.com/kubevirt/kubevirt
>> >
>> > (let me know if you know others)
>> >
>> >
>> > * But Why?
>> >
>> > I'm particularly interested in 3 out of 4 of the projects above and
>> > only Kubevirt uses libvirt to handle QEMU. That means that every
>> > QEMU releases where a QMP command, event or other data struct is
>> > added, removed or changed, those projects need to check what changed
>> > in QEMU and then address those changes in their projects, if needed.
>> >
>> > The idea behind generating Go data structures is that we can keep a
>> > Go module which can have releases that follow QEMU releases.
>>
>> We need to look at "following the QEMU releases" a bit more closely.
>>
>> Merging your patches gives us the capability to generate a Go interface
>> to HEAD's version of QMP.
>>
>> The obvious way for an out-of-tree Go program to use this generated Go
>> interface is to build with a specific version of it. It can then talk
>> QMP to any compatible QEMU version.
>>
>> Compatibility with older QEMUs is not assured: stuff added since is
>> present on the Go QMP client end, but not on the QEMU QMP server end.
>>
>> Compatibility with newer QEMUs is subject to our deprecation policy:
>>
>> In general features are intended to be supported indefinitely once
>> introduced into QEMU. In the event that a feature needs to be
>> removed, it will be listed in this section. The feature will remain
>> functional for the release in which it was deprecated and one
>> further release. After these two releases, the feature is liable to
>> be removed.
>>
>> So, if you stay away from deprecated stuff, you're good for two more
>> releases at least.
>>
>> Does this work for the projects you have in mind?
>
> It might work for some projects, but in the general case I find it pretty
> unappealing as a restriction. Mixing and matching new QEMU with old libvirt,
> or vica-verca has been an incredibly common thing todo when both developing
> and perhaps more importantly debugging problems. For example I have one
> libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x
> update, which spans a great many QEMU releases.
I'd like to propose that for compatibility with a wide range of QEMU
versions, you use or reinvent libvirt.
> I like the idea of auto-generating clients from the QAPI schema, and
> would like it if we were able to use this kind of approach on the libvirt
> side, but for that we need to be more flexible in version matching.
>
> Our current approach to deprecation features and subsequently removing
> them from the QAPI schema works fine when the QAPI schema is only used
> internally by QEMU, not when we we expand usage of QAPI to external
> applications.
>
> I think we need to figure out a way to make the QAPI schema itself be
> append only, while still allowing QEMU to deprecation & remove features.
This is going to get complicated fast.
> For a minimum viable use case, this doesn't feel all that difficult, as
> conceptually instead of deleting the field from QAPI, we just need to
> annotate it to say when it was deleted from the QEMU side. The QAPI
> generator for internal QEMU usage, can omit any fields annotated as
> deleted in QAPI schema. The QAPI generator for external app usage,
> can (optionally) be told to include deleted fields ranging back to
> a given version number. So apps can chooses what degree of compat
> they wish to retain.
Consider this evolution of command block_resize
* Initially, it has a mandatory argument @device[*].
* An alternative way to specify the command's object emerges: new
argument @node-name. Both old @device and new @node-name become
optional, and exactly one of them must be specified. This is commit
3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."
* At some future date, the old way gets deprecated: argument @device
acquires feature @deprecated.
* Still later, the old way gets removed: @device is deleted, and
@node-name becomes mandatory.
What is the proper version-spanning interface?
I figure it's both arguments optional, must specify the right one for
the version of QEMU actually in use. This spans versions, but it fails
to abstract from them.
Note that it's not enough to replace "delete member" by "mark member
deleted in <version>". You also have to keep full history for "is it
optional". And for types, because those can evolve compatibly, too,
e.g. from struct to flat union, or from string to alternate of string
and something else. What is the proper version-spanning interface in
all the possible cases?
> Apps that wish to have version compat, would of course need to write
> their code to be aware of which fields they need to seend for which
> QEMU version.
At which point we're reinventing libvirt.
>> > * Status
>> >
>> > There are a few rough edges to work on but this is usable. The major
>> > thing I forgot to add is handling Error from Commands. It'll be the
>> > first thing I'll work on next week.
>> >
>> > If you want to start using this Today you can fetch it in at
>> >
>> > https://gitlab.com/victortoso/qapi-go/
>> >
>> > There are quite a few tests that I took from the examples in the
>> > qapi schema. Coverage using go's cover tool is giving `28.6% of
>> > statements`
>> >
>> > I've uploaded the a static generated godoc output of the above Go
>> > module here:
>> >
>> > https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
>> >
>> >
>> > * License
>> >
>> > While the generator (golang.py in this series) is GPL v2, the
>>
>> I'd make it v2+, just to express my displeasure with the decision to
>> make the initial QAPI generator v2 only for no good reason at all.
>
> Our policy is that all new code should be v2+ anyway, unless it was
> clearly derived from some pre-existing v2-only code. Upto Victor to
> say whether the golang.py is considered clean, or was copy+paste
> in any parts from existin v2-only code
Makes sense.
>> > generated code needs to be compatible with other Golang projects,
>> > such as the ones mentioned above. My intention is to keep a Go
>> > module with a MIT license.
>>
>> Meh. Can't be helped, I guess.
>
> This does make me wonder though whether the license of the QAPI input
> files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is
> the Go code to be considered a derived work of the QAPI JSON files. I'm
> not finding a clearly articulated POV on this question so far.
Oww. You're right.
The safe and easy answer is "same license as the generator code".
Anything else is either not safe or not easy, I'm afraid.
[*] Because everyhing in QEMU must be called either "device" or
"driver". It's the law!
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 12:02 ` Markus Armbruster
@ 2022-05-10 12:34 ` Daniel P. Berrangé
2022-05-10 12:51 ` Daniel P. Berrangé
2022-05-11 14:17 ` Markus Armbruster
0 siblings, 2 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 12:34 UTC (permalink / raw)
To: Markus Armbruster
Cc: Victor Toso, John Snow, Eric Blake, qemu-devel, Marc-André Lureau
On Tue, May 10, 2022 at 02:02:56PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
>
> > On Tue, Apr 26, 2022 at 01:14:28PM +0200, Markus Armbruster wrote:
> >> We need to look at "following the QEMU releases" a bit more closely.
> >>
> >> Merging your patches gives us the capability to generate a Go interface
> >> to HEAD's version of QMP.
> >>
> >> The obvious way for an out-of-tree Go program to use this generated Go
> >> interface is to build with a specific version of it. It can then talk
> >> QMP to any compatible QEMU version.
> >>
> >> Compatibility with older QEMUs is not assured: stuff added since is
> >> present on the Go QMP client end, but not on the QEMU QMP server end.
> >>
> >> Compatibility with newer QEMUs is subject to our deprecation policy:
> >>
> >> In general features are intended to be supported indefinitely once
> >> introduced into QEMU. In the event that a feature needs to be
> >> removed, it will be listed in this section. The feature will remain
> >> functional for the release in which it was deprecated and one
> >> further release. After these two releases, the feature is liable to
> >> be removed.
> >>
> >> So, if you stay away from deprecated stuff, you're good for two more
> >> releases at least.
> >>
> >> Does this work for the projects you have in mind?
> >
> > It might work for some projects, but in the general case I find it pretty
> > unappealing as a restriction. Mixing and matching new QEMU with old libvirt,
> > or vica-verca has been an incredibly common thing todo when both developing
> > and perhaps more importantly debugging problems. For example I have one
> > libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x
> > update, which spans a great many QEMU releases.
>
> I'd like to propose that for compatibility with a wide range of QEMU
> versions, you use or reinvent libvirt.
Implicit in that statement though is that libvirt will not be able
to make use of the QAPI code generator as proposed though. If we are
designing something to make our application consumer's lives easier,
but we exclude such a major application, is our solution actually
a good one.
> > For a minimum viable use case, this doesn't feel all that difficult, as
> > conceptually instead of deleting the field from QAPI, we just need to
> > annotate it to say when it was deleted from the QEMU side. The QAPI
> > generator for internal QEMU usage, can omit any fields annotated as
> > deleted in QAPI schema. The QAPI generator for external app usage,
> > can (optionally) be told to include deleted fields ranging back to
> > a given version number. So apps can chooses what degree of compat
> > they wish to retain.
>
> Consider this evolution of command block_resize
To help us understand, I'll illustrate some possible interfaces
in both Go and Python, since that covers dynamic and static
languages
> * Initially, it has a mandatory argument @device[*].
Python definition:
def block_resize(device, size)
Caller:
block_resize('dev0', 1*GiB)
Golang definition
type BlockResizeCommand struct {
Device string
Size int
}
Caller
cmd := &BlockResizeCommand{
Device: "dev0",
Size: 1 * GiB,
}
> * An alternative way to specify the command's object emerges: new
> argument @node-name. Both old @device and new @node-name become
> optional, and exactly one of them must be specified. This is commit
> 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."
Python definition. Tricky, as non-optional params must be before
optional params, but size is naturally the last arg. One option
is to pointlessly mark 'size' as optional
def block_resize(device=None, node_name=None, size=None)
Caller
block_resize(device="dev0", size=1*GiB)
block_resize(node_name="devnode0", size=1*GiB)
In golang definition
type BlockResizeArguments struct {
Device string
NodeName string
Size int
}
Caller choice of
cmd := &BlockResizeCommand{
Device: "dev0",
Size: 1 * GiB,
}
cmd := &BlockResizeCommand{
NodeName: "devnode0",
Size: 1 * GiB,
}
Neither case can easily prevent passing Device and NodeName
at same time.
> * At some future date, the old way gets deprecated: argument @device
> acquires feature @deprecated.
Ok, no change needed to the APIs in either case. Possibly have
code emit a warning if a deprecated field is set.
> * Still later, the old way gets removed: @device is deleted, and
> @node-name becomes mandatory.
Again no change needed to APIs, but QEMU will throw back an
error if the wrong one is used.
> What is the proper version-spanning interface?
>
> I figure it's both arguments optional, must specify the right one for
> the version of QEMU actually in use. This spans versions, but it fails
> to abstract from them.
Yep, I think that's inevitable in this scenario. THe plus side
is that apps that want to span versions can do so. The downside
is that apps that don't want smarts to span version, may loose
compile time warnings about use of the now deleted field.
I suggested the code generator have an option to say what level
of compat to use for generated code, so that apps can request an
API without compat, which will result in compile errors. This
though assumes every consumer app is embedding their own
generated copy of the code. Not neccessarily desirable.
At the C level you can play games with __deprecated__ to get
compile time warnings in some cases.
#define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56
causes QEMU to get compile time warnings (or errors) if it
attempts to use a API feature deprecated in 2.56, even if
the API exists in the header & library.
> Note that it's not enough to replace "delete member" by "mark member
> deleted in <version>". You also have to keep full history for "is it
> optional". And for types, because those can evolve compatibly, too,
> e.g. from struct to flat union, or from string to alternate of string
> and something else. What is the proper version-spanning interface in
> all the possible cases?
I've not thought through all possible scenarios, but there may end
up being restrictions, such that changes that were previously possible
may have to be forbidden.
One example, in the past we could do deprecate a field 'foo', then
delete 'foo' and then some time re-introduce 'foo' with a completely
different type. That would not be possible if we wanted to maintain
compat, but in this example that's probably a good thing, as it'll
be super confusing to have the same field name change type like that
over time. Easier to just use a different name.
So the question to me is not whether all our previous changes are
still possible, but whether enough of the typwes of change are
possible, such that we can cope with the ongoing maint in a
reasonable way. I don't think we've explored the possibility enough
to say one way or the other.
> > Apps that wish to have version compat, would of course need to write
> > their code to be aware of which fields they need to seend for which
> > QEMU version.
>
> At which point we're reinventing libvirt.
The premise of the code generators is that there are applications
that want to consume QEMU that are not libvirt. With this line of
reasoning we could easily say that all such applications should
just use libvirt and then we don't need to provide any of these
code generators. The fact that we're considering these code
generators though, says that we're accepting there are valid use
cases that don't want to use libvirt for whatever reasons. It is
reasonable that some of those applications may wish to target
a wide range of QEMU versions, just like libvirt does.
It is also reasonable to say that libvirt would be better off if
it could auto-generate a client API for QEMU too, instead of
writing it by hand from a human reading the QAPI
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 12:34 ` Daniel P. Berrangé
@ 2022-05-10 12:51 ` Daniel P. Berrangé
2022-05-11 14:17 ` Markus Armbruster
2022-05-11 15:38 ` Andrea Bolognani
2022-05-11 14:17 ` Markus Armbruster
1 sibling, 2 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 12:51 UTC (permalink / raw)
To: Markus Armbruster, Victor Toso, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
On Tue, May 10, 2022 at 01:34:03PM +0100, Daniel P. Berrangé wrote:
> On Tue, May 10, 2022 at 02:02:56PM +0200, Markus Armbruster wrote:
> > > For a minimum viable use case, this doesn't feel all that difficult, as
> > > conceptually instead of deleting the field from QAPI, we just need to
> > > annotate it to say when it was deleted from the QEMU side. The QAPI
> > > generator for internal QEMU usage, can omit any fields annotated as
> > > deleted in QAPI schema. The QAPI generator for external app usage,
> > > can (optionally) be told to include deleted fields ranging back to
> > > a given version number. So apps can chooses what degree of compat
> > > they wish to retain.
> >
> > Consider this evolution of command block_resize
>
> To help us understand, I'll illustrate some possible interfaces
> in both Go and Python, since that covers dynamic and static
> languages
>
> > * Initially, it has a mandatory argument @device[*].
>
> Python definition:
>
> def block_resize(device, size)
>
> Caller:
>
> block_resize('dev0', 1*GiB)
>
>
> Golang definition
>
> type BlockResizeCommand struct {
> Device string
> Size int
> }
>
> Caller
>
> cmd := &BlockResizeCommand{
> Device: "dev0",
> Size: 1 * GiB,
> }
>
> > * An alternative way to specify the command's object emerges: new
> > argument @node-name. Both old @device and new @node-name become
> > optional, and exactly one of them must be specified. This is commit
> > 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."
>
> Python definition. Tricky, as non-optional params must be before
> optional params, but size is naturally the last arg. One option
> is to pointlessly mark 'size' as optional
>
> def block_resize(device=None, node_name=None, size=None)
>
> Caller
>
> block_resize(device="dev0", size=1*GiB)
> block_resize(node_name="devnode0", size=1*GiB)
>
>
> In golang definition
>
> type BlockResizeArguments struct {
> Device string
> NodeName string
> Size int
> }
>
> Caller choice of
>
> cmd := &BlockResizeCommand{
> Device: "dev0",
> Size: 1 * GiB,
> }
>
> cmd := &BlockResizeCommand{
> NodeName: "devnode0",
> Size: 1 * GiB,
> }
>
>
> Neither case can easily prevent passing Device and NodeName
> at same time.
>
> > * At some future date, the old way gets deprecated: argument @device
> > acquires feature @deprecated.
>
> Ok, no change needed to the APIs in either case. Possibly have
> code emit a warning if a deprecated field is set.
>
> > * Still later, the old way gets removed: @device is deleted, and
> > @node-name becomes mandatory.
>
> Again no change needed to APIs, but QEMU will throw back an
> error if the wrong one is used.
>
> > What is the proper version-spanning interface?
> >
> > I figure it's both arguments optional, must specify the right one for
> > the version of QEMU actually in use. This spans versions, but it fails
> > to abstract from them.
>
> Yep, I think that's inevitable in this scenario. THe plus side
> is that apps that want to span versions can do so. The downside
> is that apps that don't want smarts to span version, may loose
> compile time warnings about use of the now deleted field.
Having said that, a different way to approach the problem is to expose
the versioning directly in the generated code.
Consider a QAPI with versioning info about the fields
{ 'command': 'block_resize',
'since': '5.0.0',
'data': { 'device': ['type': 'str', 'until': '5.2.0' ],
'*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ],
'*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ],
'node-name': ['type': 'str', 'since': '7.0.0' ],
'size': 'int' } }
Meaning
* Introduced in 5.0.0, with 'device' mandatory
* In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
* In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatory
Now consider the Go structs
In 5.0.0 we can generate:
type BlockResizeArguments struct {
V500 *BlockResizeArguments500
}
type BlockResizeArgumentsV1 struct {
Device string
Size int
}
app can use
dev := "dev0"
cmd := BlockResizeArguments{
V500: &BlockResizeArguments500{
Device: dev,
Size: 1 * GiB
}
}
In 5.2.0 we can now generate
type BlockResizeArguments struct {
V500 *BlockResizeArgumentsV500
V520 *BlockResizeArgumentsV520
}
type BlockResizeArgumentsV500 struct {
Device string
Size int
}
type BlockResizeArgumentsV520 struct {
Device *string
NodeName *string
Size int
}
App can use the same as before, or switch to one of
dev := "dev0"
cmd := BlockResizeArguments{
V520: &BlockResizeArguments520{
Device: &dev,
Size: 1 * GiB
}
}
or
node := "nodedev0"
cmd := BlockResizeArguments{
V520: &BlockResizeArguments520{
NodeName: &node,
Size: 1 * GiB
}
}
In 7.0.0 we can now generate
type BlockResizeArguments struct {
V500 *BlockResizeArgumentsV500
V520 *BlockResizeArgumentsV520
V700 *BlockResizeArgumentsV700
}
type BlockResizeArgumentsV500 struct {
Device string
Size int
}
type BlockResizeArgumentsV520 struct {
Device *string
NodeName *string
Size int
}
type BlockResizeArgumentsV700 struct {
NodeName string
Size int
}
App can use the same as before, or switch to
node := "nodedev0"
cmd := BlockResizeArguments{
V700: &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
}
This kind of per-command/type versioning is not uncommon when defining API
protocols/interfaces.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 9:52 ` Daniel P. Berrangé
@ 2022-05-10 15:25 ` Andrea Bolognani
2022-05-11 13:45 ` Markus Armbruster
1 sibling, 0 replies; 71+ messages in thread
From: Andrea Bolognani @ 2022-05-10 15:25 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Markus Armbruster, Kevin Wolf, John Snow, Victor Toso,
qemu-devel, Eric Blake
On Tue, May 10, 2022 at 10:52:34AM +0100, Daniel P. Berrangé wrote:
> On Mon, May 02, 2022 at 10:01:41AM -0400, Andrea Bolognani wrote:
> > Revised proposal for the annotation:
> >
> > ns:word-WORD-WoRD-123Word
>
> Ugly, but we should only need this in the fairly niche scenarios,
> so not too pain ful to add a handful of these:
>
> Essentially if have the schema use CamelCase with UPPERCASE
> acronyms, and declare two rules:
>
> 1. Split on every boundary from lower to upper
> 2. Acronym detection if there's a sequence of 3 uppercase
> letters, then split before the last uppercase.
That should cover most of the type names, but we're still going to
need to help the parser out when it comes to detecting acronyms in
other contexts, such as all instances of the word "VNC" below:
{ 'enum': 'DisplayProtocol',
'data': [ 'vnc', ... ] }
{ 'command': 'query-vnc-servers', ... }
{ 'event': 'VNC_DISCONNECTED', ... }
> QAuthZListPolicy
>
> Rule 1: QAuth + ZList + Policy
> Rule 2: QAuth + ZList + Policy
>
> so only the last case needs ns:QAuthZ-List-Policy annotation, whcih
> doesn't feel like a big burden
Note that in my proposal the ns: part would be used exactly for cases
like this one to separate the namespace part which, as you said in
your other reply, needs to be preserved when generating C code but
can be safely dropped when targeting a language that has actual
namespace support. So the annotation would look like
Q:AuthZ-List-Policy
in this specific case. The ns: part would be optional, as a namespace
is not embedded in most of the names.
It's also interesting to see how "AuthZ" is capitalized in various Go
modules that implement the concept:
https://pkg.go.dev/search?limit=50&m=symbol&q=authz
Most use "Authz" rather than "AuthZ". If we get to the point where
the only bit of weirdness is how we spell this specific word, I think
I'll be able to live with it :)
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-09 10:21 ` Victor Toso
@ 2022-05-10 17:37 ` Andrea Bolognani
2022-05-10 18:02 ` Daniel P. Berrangé
0 siblings, 1 reply; 71+ messages in thread
From: Andrea Bolognani @ 2022-05-10 17:37 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Mon, May 09, 2022 at 12:21:10PM +0200, Victor Toso wrote:
> On Tue, Apr 19, 2022 at 11:12:28AM -0700, Andrea Bolognani wrote:
> > Based on the example you have in the README and how commands are
> > defined, invoking (a simplified version of) the trace-event-get-state
> > command would look like
> >
> > cmd := Command{
> > Name: "trace-event-get-state",
> > Arg: TraceEventGetStateCommand{
> > Name: "qemu_memalign",
> > },
> > }
> > qmp_input, _ := json.Marshal(&cmd)
> > // qmp_input now contains
> > // {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
> > // do something with it
> >
> > qmp_output :=
> > ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
> > ret := cmd.GetReturnType()
> > _ = json.Unmarshal(qmp_output, &ret)
> > // ret is a CommandResult instance whose Value member can be cast
> > // to a TraceEventInfo struct
> >
> > First of all, from an application's point of view there are way too
> > many steps involved:
>
> It can actually get worse. I've used a lot of nested struct to
> define a Base type for a given Type. In Go, If you try to
> initialize a Type that has a nested Struct, you'll need to use
> the nested struct Type as field name and this is too verbose.
>
> See https://github.com/golang/go/issues/29438 (merged with:
> https://github.com/golang/go/issues/12854)
>
> The main reason that I kept it is because it maps very well with
> the over-the-wire protocol.
Right, I had not realized how bad things really were :)
I've noticed the use of base types and while I didn't bring it up in
my initial message because the other concerns seemed of much higher
importance, I actually wondered whether we need to expose them to
users of the Go SDK.
I think we should flatten things. That's what happens with the C
generator already, for example in
struct InetSocketAddress {
/* Members inherited from InetSocketAddressBase: */
char *host;
char *port;
/* Own members: */
bool has_numeric;
bool numeric;
/* ... */
};
This representation mirrors the wire protocol perfectly, so I see no
reason not to adopt it. Am I missing something?
> > performing this operation should really be as
> > simple as
> >
> > ret, _ := qmp.TraceEventGetState("qemu_memalign")
> > // ret is a TraceEventInfo instance
> >
> > That's the end state we should be working towards.
> >
> > Of course that assumes that the "qmp" object knows where the
> > QMP socket is, knows how to talk the QMP protocol,
> > transparently deals with serializing and deserializing data...
> > Plus, in some case you might want to deal with the wire
> > transfer yourself in an application-specific manner. So it
> > makes sense to have the basic building blocks available and
> > then build the more ergonomic SDK on top of that - with only
> > the first part being in scope for this series.
>
> Right. Indeed, I thought a bit about what I want to fit into the
> code generator that will reside in QEMU and what we might want to
> develop on top of that.
>
> The goal for this series really is generating the data types that
> can be converted to/from QMP messages.
That's perfectly fine, and in fact I believe that splitting the whole
endeavor into three parts - QMP protocol implementation, QAPI types
serialization/deserialization, and a high-level SDK that gives easy
access to the previous two - is the best approach.
> > qmp_output :=
> > ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
> > ret := TraceEventInfo{}
> > _ = json.Unmarshal(qmp_output, &ret)
> > // ret is a TraceEventInfo instance
> >
> > The advantages over the current implementation is that the compiler
> > will prevent you from doing something silly like passing the wrong
> > set of arguments to a commmand, and that the application has to
> > explicitly spell out what kind of object it expects to get as output.
>
> I think that, if we know all types that we can have at QAPI spec,
> the process of marshalling and unmarshalling should verify it.
> So, even if we don't change the expected interface as suggested,
> that work needs to be done. For some types, I've already did it,
> like for Unions and Alternate types.
>
> Example: https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/unions.go#L28
>
> This union type can have 4 values for the Any interface type. The
> code generator documents it to help user's out.
>
> | type SocketAddressLegacy struct {
> | // Base type for this struct
> | SocketAddressLegacyBase
> | // Value based on @type, possible types:
> | // * InetSocketAddressWrapper
> | // * UnixSocketAddressWrapper
> | // * VsockSocketAddressWrapper
> | // * StringWrapper
> | Value Any
> | }
>
> On the Marshal function, I used Sprintf as a way to fetch Value's
> type. There are other alternatives but to the cost of adding
> other deps.
>
> | func (s SocketAddressLegacy) MarshalJSON() ([]byte, error) {
> | base, err := json.Marshal(s.SocketAddressLegacyBase)
> | if err != nil {
> | return nil, err
> | }
> |
> | typestr := fmt.Sprintf("%T", s.Value)
> | typestr =
> | typestr[strings.LastIndex(typestr, ".")+1:]
>
> ...
>
> | // "The branches need not cover all possible enum values"
> | // This means that on Marshal, we can safely ignore empty values
> | if typestr == "<nil>" {
> | return []byte(base), nil
> | }
>
> And then we have some Runtime checks to be sure to avoid the
> scenario mismatching Value's type.
>
> | // Runtime check for supported value types
> | if typestr != "StringWrapper" &&
> | typestr != "InetSocketAddressWrapper" &&
> | typestr != "UnixSocketAddressWrapper" &&
> | typestr != "VsockSocketAddressWrapper" {
> | return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
> | }
> | value, err := json.Marshal(s.Value)
> | if err != nil {
> | return nil, err
> | }
>
> With Alternate type, extra care was need on Unmarshal as we don't
> know the underlying type without looking at the message we
> received. That's the only reason of StrictDecode() helper
> function.
>
> I'm just pointing out with above examples that I agree with you
> with Type safety. It is hard to infer everything at compile-time
> so we need some Runtime checks. Having some nicer APIs will
> definitely help and improve developer experience too.
I agree that in some cases build time validation is simply not
possible, but this doesn't seem to be one of those cases.
For unmarshaling? Sure, especially if we keep in mind the forward
compatibility scenarios that Dan has raised elsewhere in the thread,
and that to be completely honest I haven't gotten around to fully
digesting yet.
But when it comes to marshaling we know ahead of time that the value
is going to be one of a handful of types, and we should get the
compiler to spot violations of this assumption for us. Using "Any"
prevents that.
If you look at libvirt-go-xml-module, alternates are handled with
things like
type DomainInterfaceSource struct {
Ethernet *DomainInterfaceSourceEthernet `xml:"-"`
Server *DomainInterfaceSourceServer `xml:"-"`
// ...
}
type DomainInterfaceSourceEthernet struct {
IP []DomainInterfaceIP `xml:"ip"`
Route []DomainInterfaceRoute `xml:"route"`
}
type DomainInterfaceSourceServer struct {
Address string `xml:"address,attr,omitempty"`
Port uint `xml:"port,attr,omitempty"`
Local *DomainInterfaceSourceLocal `xml:"local"`
}
and (un)marshaling is done like
func (a *DomainInterfaceSource) MarshalXML(e *xml.Encoder, start
xml.StartElement) error {
if a.User != nil {
return nil
} else if a.Ethernet != nil {
if len(a.Ethernet.IP) > 0 && len(a.Ethernet.Route) > 0 {
return e.EncodeElement(a.Ethernet, start)
}
return nil
} else if // ...
}
func (a *DomainInterfaceSource) UnmarshalXML(d *xml.Decoder, start
xml.StartElement) error {
if a.User != nil {
return d.DecodeElement(a.User, &start)
} else if a.Ethernet != nil {
return d.DecodeElement(a.Ethernet, &start)
} else if // ...
}
When using these data structures, you'll then do something like
iface := DomainInterface{
Source: &DomainInterfaceSource{
Network: &DomainInterfaceNetwork{
Network: "default",
},
},
}
instead of
iface := DomainInterface{
Source: DomainInterfaceSource{
Type: "network",
Value: DomainInterfaceNetwork{
Network: "default",
},
},
}
which is more compact and, crucially, allows the compiler to catch
many typing issues ahead of time. I think this approach would work
really well for QAPI too.
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 17:37 ` Andrea Bolognani
@ 2022-05-10 18:02 ` Daniel P. Berrangé
0 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-10 18:02 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Victor Toso, qemu-devel, John Snow, Eric Blake, Markus Armbruster
On Tue, May 10, 2022 at 01:37:50PM -0400, Andrea Bolognani wrote:
> On Mon, May 09, 2022 at 12:21:10PM +0200, Victor Toso wrote:
> > On Tue, Apr 19, 2022 at 11:12:28AM -0700, Andrea Bolognani wrote:
> > > Based on the example you have in the README and how commands are
> > > defined, invoking (a simplified version of) the trace-event-get-state
> > > command would look like
> > >
> > > cmd := Command{
> > > Name: "trace-event-get-state",
> > > Arg: TraceEventGetStateCommand{
> > > Name: "qemu_memalign",
> > > },
> > > }
Note there is clear redundancy here between 'Name' and the struct
type. IMHO the better approach would be
cmd := TraceEventGetStateCommand{
Name: 'qemu_memalign'
}
and have 'Command' simply as an interface that TraceEventGetStateCommand
implements. I don't think the interface would need more than a
Marshal and Demarshal method, which serialize the 'Arg' and then add
the Name field explicitly.
> > > qmp_input, _ := json.Marshal(&cmd)
> > > // qmp_input now contains
> > > // {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
> > > // do something with it
> > >
> > > qmp_output :=
> > > ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
> > > ret := cmd.GetReturnType()
> > > _ = json.Unmarshal(qmp_output, &ret)
> > > // ret is a CommandResult instance whose Value member can be cast
> > > // to a TraceEventInfo struct
> > >
> > > First of all, from an application's point of view there are way too
> > > many steps involved:
> >
> > It can actually get worse. I've used a lot of nested struct to
> > define a Base type for a given Type. In Go, If you try to
> > initialize a Type that has a nested Struct, you'll need to use
> > the nested struct Type as field name and this is too verbose.
> >
> > See https://github.com/golang/go/issues/29438 (merged with:
> > https://github.com/golang/go/issues/12854)
> >
> > The main reason that I kept it is because it maps very well with
> > the over-the-wire protocol.
>
> Right, I had not realized how bad things really were :)
>
> I've noticed the use of base types and while I didn't bring it up in
> my initial message because the other concerns seemed of much higher
> importance, I actually wondered whether we need to expose them to
> users of the Go SDK.
>
> I think we should flatten things. That's what happens with the C
> generator already, for example in
>
> struct InetSocketAddress {
> /* Members inherited from InetSocketAddressBase: */
> char *host;
> char *port;
> /* Own members: */
> bool has_numeric;
> bool numeric;
> /* ... */
> };
>
> This representation mirrors the wire protocol perfectly, so I see no
> reason not to adopt it. Am I missing something?
The main reason not to flatten is if you have scenarios where it is
useful to work against the base type directly. I'm not sure that we
have such a need though.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-03 9:40 ` Andrea Bolognani
2022-05-03 11:04 ` Kevin Wolf
2022-05-10 9:55 ` Daniel P. Berrangé
@ 2022-05-11 6:15 ` Markus Armbruster
2 siblings, 0 replies; 71+ messages in thread
From: Markus Armbruster @ 2022-05-11 6:15 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Victor Toso, qemu-devel, John Snow, Eric Blake, Kevin Wolf
Andrea Bolognani <abologna@redhat.com> writes:
> On Tue, May 03, 2022 at 09:57:27AM +0200, Markus Armbruster wrote:
>> Andrea Bolognani <abologna@redhat.com> writes:
>> > I still feel that 1) users of a language SDK will ideally not need to
>> > look at the QAPI schema or wire chatter too often
>>
>> I think the most likely point of contact is the QEMU QMP Reference
>> Manual.
>
> Note that there isn't anything preventing us from including the
> original QAPI name in the documentation for the corresponding Go
> symbol, or even a link to the reference manual.
>
> So we could make jumping from the Go API documentation, which is what
> a Go programmer will be looking at most of the time, to the QMP
> documentation pretty much effortless.
>
>> My point is: a name override feature like the one you propose needs to
>> be used with discipline and restraint. Adds to reviewers' mental load.
>> Needs to be worth it. I'm not saying it isn't, I'm just pointing out a
>> cost.
>
> Yeah, I get that.
>
> Note that I'm not suggesting it should be possible for a name to be
> completely overridden - I just want to make it possible for a human
> to provide the name parsing algorithm solutions to those problems it
> can't figure out on its own.
>
> We could prevent that feature from being misused by verifying that
> the symbol the annotation is attached to can be derived from the list
> of words provided. That way, something like
>
> # SOMEName (completely-DIFFERENT-name)
>
> would be rejected and we would avoid misuse.
Possibly as simple as "down-case both names and drop the funny
characters, result must be the same".
>> Wild idea: assume all lower case, but keep a list of exceptions.
>
> That could actually work reasonably well for QEMU because we only
> need to handle correctly what's in the schema, not arbitrary input.
>
> There's always the risk of the list of exceptions getting out of sync
> with the needs of the schema, but there's similarly no guarantee that
> annotations are going to be introduced when they are necessary, so
> it's mostly a wash.
>
> The only slight advantage of the annotation approach would be that it
> might be easier to notice it being missing because it's close to the
> name it refers to, while the list of exceptions is tucked away in a
> script far away from it.
We'd put it in qapi/pragma.json, I guess.
>> The QAPI schema language uses three naming styles:
>>
>> * lower-case-with-hyphens for command and member names
>>
>> Many names use upper case and '_'. See pragma command-name-exceptions
>> and member-name-exceptions.
>
> Looking at the output generated by Victor's WIP script, it looks like
> these are already handled as nicely as those that don't fall under
> any exception.
>
>> Some (many?) names lack separators between words (example: logappend).
How many would be good to know.
Ad hoc hackery to find names, filter out camels (because word splitting
is too hard there), split into words, look up words in a word list:
$ for i in `/usr/bin/python3 /work/armbru/qemu/scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [a-z0-9_-]+$/ { print lu; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sed '/^mc\|^cc/d;s/^.. //;s/[^A-Za-z0-9]/\n/g' | tr A-Z a-z | sort -u`; do grep -q "^$i$" /usr/share/dict/words || echo $i; done
420 lines. How many arguably lack separators between words? Wild guess
based on glancing at the output sideways: some 50.
>> * UPPER_CASE_WITH_UNDERSCORE for event names
>>
>> * CamelCase for type names
>>
>> Capitalization of words is inconsistent in places (example: VncInfo
>> vs. DisplayReloadOptionsVNC).
>>
>> What style conversions will we need for Go? Any other conversions come
>> to mind?
>>
>> What problems do these conversions have?
>
> Go uses CamelCase for pretty much everything: types, methods,
> constants...
>
> There's one slight wrinkle, in that the case of the first letter
> decides whether it's going to be a PublicName or a privateName. We
> can't do anything about that, but it shouldn't really affect us
> that much because we'll want all QAPI names to be public.
>
> So the issues preventing us from producing a "perfect" Go API are
>
> 1. inconsistent capitalization in type names
>
> -> could be addressed by simply changing the schema, as type
> names do not travel on the wire
At the price of some churn in C code.
Perhaps more consistent capitalization could be regarded as a slight
improvement on its own. We need to see (a good sample of) the changes
to judge.
> 2. missing dashes in certain command/member names
>
> -> leads to Incorrectcamelcase.
Names with words run together are arguably no uglier in CamelCase (Go)
than in lower_case_with_underscores (C).
> Kevin's work is supposed to
> address this
Except it's stuck.
Perhaps Kevin and I can get it moving again.
Perhaps we can try to extract a local alias feature that can be grown
into the more ambitious aliases Kevin wants (if we can solve the
issues).
> 3. inability to know which parts of a lower-case-name or
> UPPER_CASE_NAME are acronyms or are otherwise supposed to be
> capitalized in a specific way
>
> -> leads to WeirdVncAndDbusCapitalization. There's currently no
> way, either implemented or planned, to avoid this
A list of words with special capitalization needs[*]?
VNC is an acronym, some languagues want VNC in camels, some Vnc.
DBus is an abbreviation, some languages want DBus in camels, some Dbus.
> In addition to these I'm also thinking that QKeyCode and all the
> QCrypto stuff should probably lose their prefixes.
As Daniel pointed out, schema names sometimes have prefixes because we
need the generated C identifiers to have prefixes.
If we hate these prefixes enough, we can try to limit them to C
identifiers.
> Note that 3 shouldn't be an issue for Rust and addressing 1 would
> actually make things worse for that language, because at the moment
> at least *some* of the types follow its expected naming rules :)
Solving Go problems by creating Rust problems doesn't feel like a good
move to me.
>> > Revised proposal for the annotation:
>> >
>> > ns:word-WORD-WoRD-123Word
>> >
>> > Words are always separated by dashes; "regular" words are entirely
>> > lowercase, while the presence of even a single uppercase letter in a
>> > word denotes the fact that its case should be preserved when the
>> > naming conventions of the target language allow that.
>>
>> Is a word always capitalized the same for a single target language? Or
>> could capitalization depend on context?
>
> I'm not aware of any language that would adopt more than a single
> style of capitalization, outside of course the obvious
> lower_case_name or UPPER_CASE_NAME scenarios where the original
> capitalization stops being relevant.
Makes sense.
[*] Sounds like crony capitalism, doesn't it :)
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 9:52 ` Daniel P. Berrangé
2022-05-10 15:25 ` Andrea Bolognani
@ 2022-05-11 13:45 ` Markus Armbruster
1 sibling, 0 replies; 71+ messages in thread
From: Markus Armbruster @ 2022-05-11 13:45 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Andrea Bolognani, Kevin Wolf, John Snow, Victor Toso, qemu-devel,
Eric Blake
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Mon, May 02, 2022 at 10:01:41AM -0400, Andrea Bolognani wrote:
>> > Times how many naming conventions?
>>
>> Yeah, I don't think requiring all possible permutations to be spelled
>> out in the schema is the way to go. That's exactly why my proposal
>> was to offer a way to inject the semantic information that the parser
>> can't figure out itself.
>>
>> Once you have a way to inform the generator that "VNCProps" is made
>> of the two words "vnc" and "props", and that "vnc" is an acronym,
>> then it can generate an identifier appropriate for the target
>> language without having to spell out anywhere that such an identifier
>> would be VNCProps for Go and VncProps for Rust.
>>
>> By the way, while looking around I realized that we also have to take
>> into account things like D-Bus: the QAPI type ChardevDBus, for
>> example, would probably translate verbatim to Go but have to be
>> changed to ChardevDbus for Rust. Fun :)
For what it's worth, I prefer Rust's style, because I hate downcasing
tails of abbreviations a lot less than I hate WORDSRUNTOGETHER.
> The hardest one of all is probably
>
> QAuthZListPolicy
>
> which must be split 'QAuthZ' + 'List' + 'Policy' - the trailing
> uppercase ruins all heuristics you can come up with, as there's no
> viable way to distinguish that scenario from 'ChardevDBus' which
> is 'Chardev' + 'DBus' not 'ChardevD' + 'Bus'
>
>> Revised proposal for the annotation:
>>
>> ns:word-WORD-WoRD-123Word
>
> Ugly, but we should only need this in the fairly niche scenarios,
> so not too pain ful to add a handful of these:
>
> Essentially if have the schema use CamelCase with UPPERCASE
> acronyms, and declare two rules:
>
> 1. Split on every boundary from lower to upper
> 2. Acronym detection if there's a sequence of 3 uppercase
> letters, then split before the last uppercase.
>
> then we'll do the right thing with the vast majority of cases:
>
> ChardevSocket:
> Rule 1: Chardev + Socket
> Rule 2: Chardev + Socket
Rule 2 isn't used here.
>
> VNCProps:
> Rule 1: VNCProps
> Rule 2: VNC + Props
Rule 2 is used here.
For Rust-style VncProps, rule 2 is again unused.
>
> ChardevDBus
Note: "DBus" is not an (upper-case) acronym. It's a proper name grown
from an abbreviation of "desktop bus".
> Rule 1: Chardev + DBus
> Rule 2: Chardev + DBus
Rule 2 isn't used here, either.
> and fail on
>
> QAuthZListPolicy
>
> Rule 1: QAuth + ZList + Policy
> Rule 2: QAuth + ZList + Policy
>
> so only the last case needs ns:QAuthZ-List-Policy annotation, whcih
> doesn't feel like a big burden
I think there are two interesting sub-problems within the larger problem
of mapping QAPI names to "nice" target language identifiers.
1. Splitting words run together (mostly camels, but also mistakes like
logappend).
2. Adjust case in words for the target language
For 1., we can rely on word separators '-', '_' and change from lower to
upper case. Cases where that doesn't suffice remain.
Adopting Go-style camels for QAPI will add to them, but something like
your rule 2 above should mitigate.
What about this simple greedy algorithm:
funny_words = [QAuthZ, ...]
tail = name
words = []
while tail != "":
if tail starts with any of the words in funny_words:
next_word = that member of funny_words
else:
next_word = tail up to next separator or change from lower
to upper case
append next_word to tail
tail = remainder of tail with leading separator stripped
For 2., adjusting to all lower case (like "vnc", "props", "dbus"), all
upper case (like "VNC", "PROPS", "DBUS"), and capitalized (like "Vnc",
"Props", "Dbus") is trivial. Adjusting to capitalized except for funny
words (like "VNC", "Props", "DBus") is not. Need a list of funny words,
I guess.
Perhaps a single list of funnies would do for 1. and for 2.
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 12:34 ` Daniel P. Berrangé
2022-05-10 12:51 ` Daniel P. Berrangé
@ 2022-05-11 14:17 ` Markus Armbruster
2022-05-18 8:55 ` Victor Toso
1 sibling, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-05-11 14:17 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Victor Toso, John Snow, Eric Blake, qemu-devel, Marc-André Lureau
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Tue, May 10, 2022 at 02:02:56PM +0200, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>
>> > On Tue, Apr 26, 2022 at 01:14:28PM +0200, Markus Armbruster wrote:
>> >> We need to look at "following the QEMU releases" a bit more closely.
>> >>
>> >> Merging your patches gives us the capability to generate a Go interface
>> >> to HEAD's version of QMP.
>> >>
>> >> The obvious way for an out-of-tree Go program to use this generated Go
>> >> interface is to build with a specific version of it. It can then talk
>> >> QMP to any compatible QEMU version.
>> >>
>> >> Compatibility with older QEMUs is not assured: stuff added since is
>> >> present on the Go QMP client end, but not on the QEMU QMP server end.
>> >>
>> >> Compatibility with newer QEMUs is subject to our deprecation policy:
>> >>
>> >> In general features are intended to be supported indefinitely once
>> >> introduced into QEMU. In the event that a feature needs to be
>> >> removed, it will be listed in this section. The feature will remain
>> >> functional for the release in which it was deprecated and one
>> >> further release. After these two releases, the feature is liable to
>> >> be removed.
>> >>
>> >> So, if you stay away from deprecated stuff, you're good for two more
>> >> releases at least.
>> >>
>> >> Does this work for the projects you have in mind?
>> >
>> > It might work for some projects, but in the general case I find it pretty
>> > unappealing as a restriction. Mixing and matching new QEMU with old libvirt,
>> > or vica-verca has been an incredibly common thing todo when both developing
>> > and perhaps more importantly debugging problems. For example I have one
>> > libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x
>> > update, which spans a great many QEMU releases.
>>
>> I'd like to propose that for compatibility with a wide range of QEMU
>> versions, you use or reinvent libvirt.
>
> Implicit in that statement though is that libvirt will not be able
> to make use of the QAPI code generator as proposed though. If we are
> designing something to make our application consumer's lives easier,
> but we exclude such a major application, is our solution actually
> a good one.
A solution for a narrow problem we can actually deliver and maintain is
better than a solution for a wider problem we can't.
>> > For a minimum viable use case, this doesn't feel all that difficult, as
>> > conceptually instead of deleting the field from QAPI, we just need to
>> > annotate it to say when it was deleted from the QEMU side. The QAPI
>> > generator for internal QEMU usage, can omit any fields annotated as
>> > deleted in QAPI schema. The QAPI generator for external app usage,
>> > can (optionally) be told to include deleted fields ranging back to
>> > a given version number. So apps can chooses what degree of compat
>> > they wish to retain.
>>
>> Consider this evolution of command block_resize
>
> To help us understand, I'll illustrate some possible interfaces
> in both Go and Python, since that covers dynamic and static
> languages
>
>> * Initially, it has a mandatory argument @device[*].
>
> Python definition:
>
> def block_resize(device, size)
>
> Caller:
>
> block_resize('dev0', 1*GiB)
>
>
> Golang definition
>
> type BlockResizeCommand struct {
> Device string
> Size int
> }
>
> Caller
>
> cmd := &BlockResizeCommand{
> Device: "dev0",
> Size: 1 * GiB,
> }
>
>> * An alternative way to specify the command's object emerges: new
>> argument @node-name. Both old @device and new @node-name become
>> optional, and exactly one of them must be specified. This is commit
>> 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."
>
> Python definition. Tricky, as non-optional params must be before
> optional params, but size is naturally the last arg. One option
> is to pointlessly mark 'size' as optional
>
> def block_resize(device=None, node_name=None, size=None)
Who needs compile-time checking anyway.
Back to serious. Keyword arguments might be a better match for Python
bindings.
> Caller
>
> block_resize(device="dev0", size=1*GiB)
> block_resize(node_name="devnode0", size=1*GiB)
>
>
> In golang definition
>
> type BlockResizeArguments struct {
> Device string
> NodeName string
> Size int
> }
>
> Caller choice of
>
> cmd := &BlockResizeCommand{
> Device: "dev0",
> Size: 1 * GiB,
> }
>
> cmd := &BlockResizeCommand{
> NodeName: "devnode0",
> Size: 1 * GiB,
> }
Note that the Go bindings you sketched effectively use (poor man's)
keyword arguments.
> Neither case can easily prevent passing Device and NodeName
> at same time.
That defect lies at the schema's feet.
>> * At some future date, the old way gets deprecated: argument @device
>> acquires feature @deprecated.
>
> Ok, no change needed to the APIs in either case. Possibly have
> code emit a warning if a deprecated field is set.
>
>> * Still later, the old way gets removed: @device is deleted, and
>> @node-name becomes mandatory.
>
> Again no change needed to APIs, but QEMU will throw back an
> error if the wrong one is used.
>
>> What is the proper version-spanning interface?
>>
>> I figure it's both arguments optional, must specify the right one for
>> the version of QEMU actually in use. This spans versions, but it fails
>> to abstract from them.
>
> Yep, I think that's inevitable in this scenario. THe plus side
> is that apps that want to span versions can do so. The downside
> is that apps that don't want smarts to span version, may loose
> compile time warnings about use of the now deleted field.
The version-spanning interface will arguably be a bad interface for any
version.
> I suggested the code generator have an option to say what level
> of compat to use for generated code, so that apps can request an
> API without compat, which will result in compile errors. This
> though assumes every consumer app is embedding their own
> generated copy of the code. Not neccessarily desirable.
>
> At the C level you can play games with __deprecated__ to get
> compile time warnings in some cases.
>
> #define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56
>
> causes QEMU to get compile time warnings (or errors) if it
> attempts to use a API feature deprecated in 2.56, even if
> the API exists in the header & library.
>
>
>> Note that it's not enough to replace "delete member" by "mark member
>> deleted in <version>". You also have to keep full history for "is it
>> optional". And for types, because those can evolve compatibly, too,
>> e.g. from struct to flat union, or from string to alternate of string
>> and something else. What is the proper version-spanning interface in
>> all the possible cases?
>
> I've not thought through all possible scenarios, but there may end
> up being restrictions, such that changes that were previously possible
> may have to be forbidden.
"There may be restrictions" is not exactly a confidence-inspring design
assumption. We need a reasonably dependable idea on what exactly we're
intending to sacrifice.
> One example, in the past we could do deprecate a field 'foo', then
> delete 'foo' and then some time re-introduce 'foo' with a completely
> different type. That would not be possible if we wanted to maintain
> compat, but in this example that's probably a good thing, as it'll
> be super confusing to have the same field name change type like that
> over time. Easier to just use a different name.
>
> So the question to me is not whether all our previous changes are
> still possible, but whether enough of the typwes of change are
> possible, such that we can cope with the ongoing maint in a
> reasonable way. I don't think we've explored the possibility enough
> to say one way or the other.
>
>> > Apps that wish to have version compat, would of course need to write
>> > their code to be aware of which fields they need to seend for which
>> > QEMU version.
>>
>> At which point we're reinventing libvirt.
>
> The premise of the code generators is that there are applications
> that want to consume QEMU that are not libvirt. With this line of
> reasoning we could easily say that all such applications should
> just use libvirt and then we don't need to provide any of these
> code generators. The fact that we're considering these code
> generators though, says that we're accepting there are valid use
> cases that don't want to use libvirt for whatever reasons.
Can't resist any longer: "What has libvirt ever done for us?"
https://www.youtube.com/watch?v=Qc7HmhrgTuQ
> It is
> reasonable that some of those applications may wish to target
> a wide range of QEMU versions, just like libvirt does.
At which point they're comitting to reinventing the relevant parts of
libvirt.
I'd expect marshalling and umarshalling QMP to be among the smaller
sub-problems then. It may look relatively large at first, because it's
among the first ones to be solved. More so when you hand-wave away the
more interesting ones of *abstraction* until they bite you in the
posterior.
> It is also reasonable to say that libvirt would be better off if
> it could auto-generate a client API for QEMU too, instead of
> writing it by hand from a human reading the QAPI
>
> With regards,
> Daniel
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 12:51 ` Daniel P. Berrangé
@ 2022-05-11 14:17 ` Markus Armbruster
2022-05-11 14:41 ` Daniel P. Berrangé
2022-05-11 15:38 ` Andrea Bolognani
1 sibling, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-05-11 14:17 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Victor Toso, John Snow, Eric Blake, qemu-devel, Marc-André Lureau
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Tue, May 10, 2022 at 01:34:03PM +0100, Daniel P. Berrangé wrote:
>> On Tue, May 10, 2022 at 02:02:56PM +0200, Markus Armbruster wrote:
>> > > For a minimum viable use case, this doesn't feel all that difficult, as
>> > > conceptually instead of deleting the field from QAPI, we just need to
>> > > annotate it to say when it was deleted from the QEMU side. The QAPI
>> > > generator for internal QEMU usage, can omit any fields annotated as
>> > > deleted in QAPI schema. The QAPI generator for external app usage,
>> > > can (optionally) be told to include deleted fields ranging back to
>> > > a given version number. So apps can chooses what degree of compat
>> > > they wish to retain.
>> >
>> > Consider this evolution of command block_resize
>>
>> To help us understand, I'll illustrate some possible interfaces
>> in both Go and Python, since that covers dynamic and static
>> languages
>>
>> > * Initially, it has a mandatory argument @device[*].
>>
>> Python definition:
>>
>> def block_resize(device, size)
>>
>> Caller:
>>
>> block_resize('dev0', 1*GiB)
>>
>>
>> Golang definition
>>
>> type BlockResizeCommand struct {
>> Device string
>> Size int
>> }
>>
>> Caller
>>
>> cmd := &BlockResizeCommand{
>> Device: "dev0",
>> Size: 1 * GiB,
>> }
>>
>> > * An alternative way to specify the command's object emerges: new
>> > argument @node-name. Both old @device and new @node-name become
>> > optional, and exactly one of them must be specified. This is commit
>> > 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."
>>
>> Python definition. Tricky, as non-optional params must be before
>> optional params, but size is naturally the last arg. One option
>> is to pointlessly mark 'size' as optional
>>
>> def block_resize(device=None, node_name=None, size=None)
>>
>> Caller
>>
>> block_resize(device="dev0", size=1*GiB)
>> block_resize(node_name="devnode0", size=1*GiB)
>>
>>
>> In golang definition
>>
>> type BlockResizeArguments struct {
>> Device string
>> NodeName string
>> Size int
>> }
>>
>> Caller choice of
>>
>> cmd := &BlockResizeCommand{
>> Device: "dev0",
>> Size: 1 * GiB,
>> }
>>
>> cmd := &BlockResizeCommand{
>> NodeName: "devnode0",
>> Size: 1 * GiB,
>> }
>>
>>
>> Neither case can easily prevent passing Device and NodeName
>> at same time.
>>
>> > * At some future date, the old way gets deprecated: argument @device
>> > acquires feature @deprecated.
>>
>> Ok, no change needed to the APIs in either case. Possibly have
>> code emit a warning if a deprecated field is set.
>>
>> > * Still later, the old way gets removed: @device is deleted, and
>> > @node-name becomes mandatory.
>>
>> Again no change needed to APIs, but QEMU will throw back an
>> error if the wrong one is used.
>>
>> > What is the proper version-spanning interface?
>> >
>> > I figure it's both arguments optional, must specify the right one for
>> > the version of QEMU actually in use. This spans versions, but it fails
>> > to abstract from them.
>>
>> Yep, I think that's inevitable in this scenario. THe plus side
>> is that apps that want to span versions can do so. The downside
>> is that apps that don't want smarts to span version, may loose
>> compile time warnings about use of the now deleted field.
>
> Having said that, a different way to approach the problem is to expose
> the versioning directly in the generated code.
>
> Consider a QAPI with versioning info about the fields
>
> { 'command': 'block_resize',
> 'since': '5.0.0',
> 'data': { 'device': ['type': 'str', 'until': '5.2.0' ],
> '*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ],
> '*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ],
> 'node-name': ['type': 'str', 'since': '7.0.0' ],
> 'size': 'int' } }
>
> Meaning
>
> * Introduced in 5.0.0, with 'device' mandatory
> * In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
> * In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatory
>
> Now consider the Go structs
>
> In 5.0.0 we can generate:
>
> type BlockResizeArguments struct {
> V500 *BlockResizeArguments500
> }
>
> type BlockResizeArgumentsV1 struct {
> Device string
> Size int
> }
>
> app can use
>
> dev := "dev0"
> cmd := BlockResizeArguments{
> V500: &BlockResizeArguments500{
> Device: dev,
> Size: 1 * GiB
> }
> }
>
>
> In 5.2.0 we can now generate
>
> type BlockResizeArguments struct {
> V500 *BlockResizeArgumentsV500
> V520 *BlockResizeArgumentsV520
> }
>
> type BlockResizeArgumentsV500 struct {
> Device string
> Size int
> }
>
> type BlockResizeArgumentsV520 struct {
> Device *string
> NodeName *string
> Size int
> }
>
>
> App can use the same as before, or switch to one of
>
> dev := "dev0"
> cmd := BlockResizeArguments{
> V520: &BlockResizeArguments520{
> Device: &dev,
> Size: 1 * GiB
> }
> }
>
> or
>
> node := "nodedev0"
> cmd := BlockResizeArguments{
> V520: &BlockResizeArguments520{
> NodeName: &node,
> Size: 1 * GiB
> }
> }
>
>
>
> In 7.0.0 we can now generate
>
>
> type BlockResizeArguments struct {
> V500 *BlockResizeArgumentsV500
> V520 *BlockResizeArgumentsV520
> V700 *BlockResizeArgumentsV700
> }
>
> type BlockResizeArgumentsV500 struct {
> Device string
> Size int
> }
>
> type BlockResizeArgumentsV520 struct {
> Device *string
> NodeName *string
> Size int
> }
>
> type BlockResizeArgumentsV700 struct {
> NodeName string
> Size int
> }
>
>
>
> App can use the same as before, or switch to
>
> node := "nodedev0"
> cmd := BlockResizeArguments{
> V700: &BlockResizeArguments700{
> NodeName: node,
> Size: 1 * GiB
> }
> }
>
>
> This kind of per-command/type versioning is not uncommon when defining API
> protocols/interfaces.
1. At every release, put a copy of the QAPI schema in the freezer.
2. For every copy, generate Go types with a suitable name suffix.
Collect all the name stems.
3. For each name stem, define a Go struct that contains all the suffixed
Go types with that stem.
Look Ma, no cluttering the QAPI schema with a full history! Also no
complicating the schema language to provide the means for that.
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-11 14:17 ` Markus Armbruster
@ 2022-05-11 14:41 ` Daniel P. Berrangé
0 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-11 14:41 UTC (permalink / raw)
To: Markus Armbruster
Cc: Victor Toso, John Snow, Eric Blake, qemu-devel, Marc-André Lureau
On Wed, May 11, 2022 at 04:17:43PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
>
> > On Tue, May 10, 2022 at 01:34:03PM +0100, Daniel P. Berrangé wrote:
> > Having said that, a different way to approach the problem is to expose
> > the versioning directly in the generated code.
> >
> > Consider a QAPI with versioning info about the fields
> >
> > { 'command': 'block_resize',
> > 'since': '5.0.0',
> > 'data': { 'device': ['type': 'str', 'until': '5.2.0' ],
> > '*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ],
> > '*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ],
> > 'node-name': ['type': 'str', 'since': '7.0.0' ],
> > 'size': 'int' } }
> >
> > Meaning
> >
> > * Introduced in 5.0.0, with 'device' mandatory
> > * In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
> > * In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatory
> >
> > Now consider the Go structs
> >
> > In 5.0.0 we can generate:
> >
> > type BlockResizeArguments struct {
> > V500 *BlockResizeArguments500
> > }
> >
> > type BlockResizeArgumentsV1 struct {
> > Device string
> > Size int
> > }
> >
> > app can use
> >
> > dev := "dev0"
> > cmd := BlockResizeArguments{
> > V500: &BlockResizeArguments500{
> > Device: dev,
> > Size: 1 * GiB
> > }
> > }
> >
> >
> > In 5.2.0 we can now generate
> >
> > type BlockResizeArguments struct {
> > V500 *BlockResizeArgumentsV500
> > V520 *BlockResizeArgumentsV520
> > }
> >
> > type BlockResizeArgumentsV500 struct {
> > Device string
> > Size int
> > }
> >
> > type BlockResizeArgumentsV520 struct {
> > Device *string
> > NodeName *string
> > Size int
> > }
> >
> >
> > App can use the same as before, or switch to one of
> >
> > dev := "dev0"
> > cmd := BlockResizeArguments{
> > V520: &BlockResizeArguments520{
> > Device: &dev,
> > Size: 1 * GiB
> > }
> > }
> >
> > or
> >
> > node := "nodedev0"
> > cmd := BlockResizeArguments{
> > V520: &BlockResizeArguments520{
> > NodeName: &node,
> > Size: 1 * GiB
> > }
> > }
> >
> >
> >
> > In 7.0.0 we can now generate
> >
> >
> > type BlockResizeArguments struct {
> > V500 *BlockResizeArgumentsV500
> > V520 *BlockResizeArgumentsV520
> > V700 *BlockResizeArgumentsV700
> > }
> >
> > type BlockResizeArgumentsV500 struct {
> > Device string
> > Size int
> > }
> >
> > type BlockResizeArgumentsV520 struct {
> > Device *string
> > NodeName *string
> > Size int
> > }
> >
> > type BlockResizeArgumentsV700 struct {
> > NodeName string
> > Size int
> > }
> >
> >
> >
> > App can use the same as before, or switch to
> >
> > node := "nodedev0"
> > cmd := BlockResizeArguments{
> > V700: &BlockResizeArguments700{
> > NodeName: node,
> > Size: 1 * GiB
> > }
> > }
> >
> >
> > This kind of per-command/type versioning is not uncommon when defining API
> > protocols/interfaces.
>
> 1. At every release, put a copy of the QAPI schema in the freezer.
>
> 2. For every copy, generate Go types with a suitable name suffix.
> Collect all the name stems.
>
> 3. For each name stem, define a Go struct that contains all the suffixed
> Go types with that stem.
>
> Look Ma, no cluttering the QAPI schema with a full history! Also no
> complicating the schema language to provide the means for that.
That could indeed be a viable approach. Puts a little more work on the
code generator to match up the types, but probably isn't too hard.
Incidentally, we've intentionally not exposed type names in the QAPI
introspection in QMP. With code generators, when the generated code
type names driven from QAPI schema, there's likely going to be an
expectation that type names in QAPI have some kind of stability rules.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-10 12:51 ` Daniel P. Berrangé
2022-05-11 14:17 ` Markus Armbruster
@ 2022-05-11 15:38 ` Andrea Bolognani
2022-05-11 15:50 ` Daniel P. Berrangé
1 sibling, 1 reply; 71+ messages in thread
From: Andrea Bolognani @ 2022-05-11 15:38 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Markus Armbruster, Victor Toso, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
On Tue, May 10, 2022 at 01:51:05PM +0100, Daniel P. Berrangé wrote:
> In 7.0.0 we can now generate
>
> type BlockResizeArguments struct {
> V500 *BlockResizeArgumentsV500
> V520 *BlockResizeArgumentsV520
> V700 *BlockResizeArgumentsV700
> }
>
> type BlockResizeArgumentsV500 struct {
> Device string
> Size int
> }
>
> type BlockResizeArgumentsV520 struct {
> Device *string
> NodeName *string
> Size int
> }
>
> type BlockResizeArgumentsV700 struct {
> NodeName string
> Size int
> }
>
> App can use the same as before, or switch to
>
> node := "nodedev0"
> cmd := BlockResizeArguments{
> V700: &BlockResizeArguments700{
> NodeName: node,
> Size: 1 * GiB
> }
> }
This honestly looks pretty unwieldy.
If the application already knows it's targeting a specific version of
the QEMU API, which for the above code to make any sense it will have
to, couldn't it do something like
import qemu .../qemu/v700
at the beginning of the file and then use regular old
cmd := qemu.BlockResizeArguments{
NodeName: nodeName,
Size: size,
}
instead?
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-11 15:38 ` Andrea Bolognani
@ 2022-05-11 15:50 ` Daniel P. Berrangé
2022-05-11 16:22 ` Andrea Bolognani
2022-05-18 8:10 ` Victor Toso
0 siblings, 2 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-11 15:50 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Markus Armbruster, Victor Toso, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
On Wed, May 11, 2022 at 08:38:04AM -0700, Andrea Bolognani wrote:
> On Tue, May 10, 2022 at 01:51:05PM +0100, Daniel P. Berrangé wrote:
> > In 7.0.0 we can now generate
> >
> > type BlockResizeArguments struct {
> > V500 *BlockResizeArgumentsV500
> > V520 *BlockResizeArgumentsV520
> > V700 *BlockResizeArgumentsV700
> > }
> >
> > type BlockResizeArgumentsV500 struct {
> > Device string
> > Size int
> > }
> >
> > type BlockResizeArgumentsV520 struct {
> > Device *string
> > NodeName *string
> > Size int
> > }
> >
> > type BlockResizeArgumentsV700 struct {
> > NodeName string
> > Size int
> > }
> >
> > App can use the same as before, or switch to
> >
> > node := "nodedev0"
> > cmd := BlockResizeArguments{
> > V700: &BlockResizeArguments700{
> > NodeName: node,
> > Size: 1 * GiB
> > }
> > }
>
> This honestly looks pretty unwieldy.
It isn't all that more verbose than without the versions - just
a single struct wrapper.
>
> If the application already knows it's targeting a specific version of
> the QEMU API, which for the above code to make any sense it will have
> to, couldn't it do something like
>
> import qemu .../qemu/v700
>
> at the beginning of the file and then use regular old
>
> cmd := qemu.BlockResizeArguments{
> NodeName: nodeName,
> Size: size,
> }
>
> instead?
This would lead to a situation where every struct is duplicated
for every version, even though 90% of the time they'll be identical
across multiple versions. This is not very ammenable to the desire
to be able to dynamically choose per-command which version you
want based on which version of QEMU you're connected to.
ie
var cmd Command
if qmp.HasVersion(qemu.Version(7, 0, 0)) {
cmd = BlockResizeArguments{
V700: &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
}
} else {
cmd = BlockResizeArguments{
V520: &BlockResizeArguments520{
Device: dev,
Size: 1 * GiB
}
}
}
And of course the HasVersion check is going to be different
for each command that matters.
Having said that, this perhaps shows the nested structs are
overkill. We could have
var cmd Command
if qmp.HasVersion(qemu.Version(7, 0, 0)) {
cmd = &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
} else {
cmd = &BlockResizeArguments520{
Device: dev,
Size: 1 * GiB
}
}
If there was some need for common handling of the different versioned
variants, we could still have a 'BlockResizeArguments' that has a field
per version, as an optional thing. Or have a BlockResizeArguments
interface, implemented by each version </hand-wavey>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-11 15:50 ` Daniel P. Berrangé
@ 2022-05-11 16:22 ` Andrea Bolognani
2022-05-11 16:32 ` Daniel P. Berrangé
2022-05-18 8:10 ` Victor Toso
1 sibling, 1 reply; 71+ messages in thread
From: Andrea Bolognani @ 2022-05-11 16:22 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Markus Armbruster, Victor Toso, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
On Wed, May 11, 2022 at 04:50:35PM +0100, Daniel P. Berrangé wrote:
> This would lead to a situation where every struct is duplicated
> for every version, even though 90% of the time they'll be identical
> across multiple versions. This is not very ammenable to the desire
> to be able to dynamically choose per-command which version you
> want based on which version of QEMU you're connected to.
>
> ie
>
> var cmd Command
> if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> cmd = BlockResizeArguments{
> V700: &BlockResizeArguments700{
> NodeName: node,
> Size: 1 * GiB
> }
> }
> } else {
> cmd = BlockResizeArguments{
> V520: &BlockResizeArguments520{
> Device: dev,
> Size: 1 * GiB
> }
> }
> }
>
> And of course the HasVersion check is going to be different
> for each command that matters.
>
> Having said that, this perhaps shows the nested structs are
> overkill. We could have
>
> var cmd Command
> if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> cmd = &BlockResizeArguments700{
> NodeName: node,
> Size: 1 * GiB
> }
> } else {
> cmd = &BlockResizeArguments520{
> Device: dev,
> Size: 1 * GiB
> }
> }
Right, making the decision at the import level would require adding a
level of indirection and make this kind of dynamic logic awkward.
We shouldn't force users to sprinkle version numbers everywhere
though, especially since different commands will change at different
points in time. It should be possible to do something like
if !qmp.HasAPI(600) {
panic("update QEMU")
}
cmd := &BlockResizeArguments600{ // really BlockResizeArguments520
Device: device,
Size: size,
}
or even
if !qmp.HasAPI(qmp.API.Latest) {
panic("update QEMU")
}
cmd := &BlockResizeArguments{
NodeName: nodeName,
Size: size,
}
Should be easy enough to achieve with type aliases.
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-11 16:22 ` Andrea Bolognani
@ 2022-05-11 16:32 ` Daniel P. Berrangé
0 siblings, 0 replies; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-11 16:32 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Markus Armbruster, Victor Toso, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
On Wed, May 11, 2022 at 09:22:36AM -0700, Andrea Bolognani wrote:
> On Wed, May 11, 2022 at 04:50:35PM +0100, Daniel P. Berrangé wrote:
> > This would lead to a situation where every struct is duplicated
> > for every version, even though 90% of the time they'll be identical
> > across multiple versions. This is not very ammenable to the desire
> > to be able to dynamically choose per-command which version you
> > want based on which version of QEMU you're connected to.
> >
> > ie
> >
> > var cmd Command
> > if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> > cmd = BlockResizeArguments{
> > V700: &BlockResizeArguments700{
> > NodeName: node,
> > Size: 1 * GiB
> > }
> > }
> > } else {
> > cmd = BlockResizeArguments{
> > V520: &BlockResizeArguments520{
> > Device: dev,
> > Size: 1 * GiB
> > }
> > }
> > }
> >
> > And of course the HasVersion check is going to be different
> > for each command that matters.
> >
> > Having said that, this perhaps shows the nested structs are
> > overkill. We could have
> >
> > var cmd Command
> > if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> > cmd = &BlockResizeArguments700{
> > NodeName: node,
> > Size: 1 * GiB
> > }
> > } else {
> > cmd = &BlockResizeArguments520{
> > Device: dev,
> > Size: 1 * GiB
> > }
> > }
>
> Right, making the decision at the import level would require adding a
> level of indirection and make this kind of dynamic logic awkward.
>
> We shouldn't force users to sprinkle version numbers everywhere
> though, especially since different commands will change at different
> points in time. It should be possible to do something like
>
> if !qmp.HasAPI(600) {
> panic("update QEMU")
> }
>
> cmd := &BlockResizeArguments600{ // really BlockResizeArguments520
> Device: device,
> Size: size,
> }
>
> or even
>
> if !qmp.HasAPI(qmp.API.Latest) {
> panic("update QEMU")
> }
>
> cmd := &BlockResizeArguments{
> NodeName: nodeName,
> Size: size,
> }
>
> Should be easy enough to achieve with type aliases.
I guess we would have a single package which does
typedef BlockResizeArguments520 BlockResizeArguments;
...for each type...
The interaction with API versioning will be tedious though. For the
versioned structs we'll be forever back compatible, due to the
append-only nature of the versioned struct approach. For the type
aliases, we'll be forever breaking compat as at least 1 struct
alias is likely to need changing every QEMU release.
Might suggest having 2 distinct go modules. A base module which
is versioned as a stable API with semver (forever v1.xxx), and
an add-on module that depends on base module, which is versioned
as an unstable API with semver (forever v0.xxx)
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-11 15:50 ` Daniel P. Berrangé
2022-05-11 16:22 ` Andrea Bolognani
@ 2022-05-18 8:10 ` Victor Toso
2022-05-18 8:51 ` Daniel P. Berrangé
1 sibling, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-18 8:10 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Andrea Bolognani, Markus Armbruster, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
[-- Attachment #1: Type: text/plain, Size: 3754 bytes --]
Hi,
On Wed, May 11, 2022 at 04:50:35PM +0100, Daniel P. Berrangé wrote:
> On Wed, May 11, 2022 at 08:38:04AM -0700, Andrea Bolognani wrote:
> > On Tue, May 10, 2022 at 01:51:05PM +0100, Daniel P. Berrangé wrote:
> > > In 7.0.0 we can now generate
> > >
> > > type BlockResizeArguments struct {
> > > V500 *BlockResizeArgumentsV500
> > > V520 *BlockResizeArgumentsV520
> > > V700 *BlockResizeArgumentsV700
> > > }
> > >
> > > type BlockResizeArgumentsV500 struct {
> > > Device string
> > > Size int
> > > }
> > >
> > > type BlockResizeArgumentsV520 struct {
> > > Device *string
> > > NodeName *string
> > > Size int
> > > }
> > >
> > > type BlockResizeArgumentsV700 struct {
> > > NodeName string
> > > Size int
> > > }
> > >
> > > App can use the same as before, or switch to
> > >
> > > node := "nodedev0"
> > > cmd := BlockResizeArguments{
> > > V700: &BlockResizeArguments700{
> > > NodeName: node,
> > > Size: 1 * GiB
> > > }
> > > }
> >
> > This honestly looks pretty unwieldy.
>
> It isn't all that more verbose than without the versions - just
> a single struct wrapper.
>
> >
> > If the application already knows it's targeting a specific version of
> > the QEMU API, which for the above code to make any sense it will have
> > to, couldn't it do something like
> >
> > import qemu .../qemu/v700
> >
> > at the beginning of the file and then use regular old
> >
> > cmd := qemu.BlockResizeArguments{
> > NodeName: nodeName,
> > Size: size,
> > }
> >
> > instead?
>
> This would lead to a situation where every struct is duplicated
> for every version, even though 90% of the time they'll be identical
> across multiple versions. This is not very ammenable to the desire
> to be able to dynamically choose per-command which version you
> want based on which version of QEMU you're connected to.
>
> ie
>
>
> var cmd Command
> if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> cmd = BlockResizeArguments{
> V700: &BlockResizeArguments700{
> NodeName: node,
> Size: 1 * GiB
> }
> }
> } else {
> cmd = BlockResizeArguments{
> V520: &BlockResizeArguments520{
> Device: dev,
> Size: 1 * GiB
> }
> }
> }
>
> And of course the HasVersion check is going to be different
> for each command that matters.
>
> Having said that, this perhaps shows the nested structs are
> overkill. We could have
>
>
> var cmd Command
> if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> cmd = &BlockResizeArguments700{
> NodeName: node,
> Size: 1 * GiB
> }
> } else {
> cmd = &BlockResizeArguments520{
> Device: dev,
> Size: 1 * GiB
> }
> }
The else block would be wrong in versions above 7.0.0 where
block_resize changed. There will be a need to know for a specific
Type if we are covered with latest qemu/qapi-go or not. Not yet
sure how to address that, likely we will need to keep the
information that something has been added/changed/removed per
version per Type in qapi-go...
Still, I think the above proposal is a good compromise to make..
> If there was some need for common handling of the different versioned
> variants, we could still have a 'BlockResizeArguments' that has a field
> per version, as an optional thing. Or have a BlockResizeArguments
> interface, implemented by each version </hand-wavey>
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-18 8:10 ` Victor Toso
@ 2022-05-18 8:51 ` Daniel P. Berrangé
2022-05-18 9:01 ` Victor Toso
0 siblings, 1 reply; 71+ messages in thread
From: Daniel P. Berrangé @ 2022-05-18 8:51 UTC (permalink / raw)
To: Victor Toso
Cc: Andrea Bolognani, Markus Armbruster, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
On Wed, May 18, 2022 at 10:10:48AM +0200, Victor Toso wrote:
> Hi,
>
> On Wed, May 11, 2022 at 04:50:35PM +0100, Daniel P. Berrangé wrote:
> > On Wed, May 11, 2022 at 08:38:04AM -0700, Andrea Bolognani wrote:
> > > On Tue, May 10, 2022 at 01:51:05PM +0100, Daniel P. Berrangé wrote:
> > > > In 7.0.0 we can now generate
> > > >
> > > > type BlockResizeArguments struct {
> > > > V500 *BlockResizeArgumentsV500
> > > > V520 *BlockResizeArgumentsV520
> > > > V700 *BlockResizeArgumentsV700
> > > > }
> > > >
> > > > type BlockResizeArgumentsV500 struct {
> > > > Device string
> > > > Size int
> > > > }
> > > >
> > > > type BlockResizeArgumentsV520 struct {
> > > > Device *string
> > > > NodeName *string
> > > > Size int
> > > > }
> > > >
> > > > type BlockResizeArgumentsV700 struct {
> > > > NodeName string
> > > > Size int
> > > > }
> > > >
> > > > App can use the same as before, or switch to
> > > >
> > > > node := "nodedev0"
> > > > cmd := BlockResizeArguments{
> > > > V700: &BlockResizeArguments700{
> > > > NodeName: node,
> > > > Size: 1 * GiB
> > > > }
> > > > }
> > >
> > > This honestly looks pretty unwieldy.
> >
> > It isn't all that more verbose than without the versions - just
> > a single struct wrapper.
> >
> > >
> > > If the application already knows it's targeting a specific version of
> > > the QEMU API, which for the above code to make any sense it will have
> > > to, couldn't it do something like
> > >
> > > import qemu .../qemu/v700
> > >
> > > at the beginning of the file and then use regular old
> > >
> > > cmd := qemu.BlockResizeArguments{
> > > NodeName: nodeName,
> > > Size: size,
> > > }
> > >
> > > instead?
> >
> > This would lead to a situation where every struct is duplicated
> > for every version, even though 90% of the time they'll be identical
> > across multiple versions. This is not very ammenable to the desire
> > to be able to dynamically choose per-command which version you
> > want based on which version of QEMU you're connected to.
> >
> > ie
> >
> >
> > var cmd Command
> > if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> > cmd = BlockResizeArguments{
> > V700: &BlockResizeArguments700{
> > NodeName: node,
> > Size: 1 * GiB
> > }
> > }
> > } else {
> > cmd = BlockResizeArguments{
> > V520: &BlockResizeArguments520{
> > Device: dev,
> > Size: 1 * GiB
> > }
> > }
> > }
> >
> > And of course the HasVersion check is going to be different
> > for each command that matters.
> >
> > Having said that, this perhaps shows the nested structs are
> > overkill. We could have
> >
> >
> > var cmd Command
> > if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> > cmd = &BlockResizeArguments700{
> > NodeName: node,
> > Size: 1 * GiB
> > }
> > } else {
> > cmd = &BlockResizeArguments520{
> > Device: dev,
> > Size: 1 * GiB
> > }
> > }
>
> The else block would be wrong in versions above 7.0.0 where
> block_resize changed. There will be a need to know for a specific
> Type if we are covered with latest qemu/qapi-go or not. Not yet
> sure how to address that, likely we will need to keep the
> information that something has been added/changed/removed per
> version per Type in qapi-go...
I put that in the "nice to have" category. No application can
predict the future, and nor do they really need to try in
general.
If the application code was written when the newest QEMU was
7.1.0, and the above code is correct for QEMU <= 7.1.0, then
that's good enough. If the BlockResizeArguments struct changed
in a later QEMU version 8.0.0, that doesn't matter at the point
the app code was written.
Much of the time the changes are back compatible, ie just adding
a new field, and so everything will still work fine if the app
carries on using BlockResizeArguments700, despite a new
BlockResizeArguments800 arriving with a new field.
Only in the cases where a field was removed or changed in a
non-compatible manner would an app have problems, and QEMU will
happily report an error at runtime if the app sends something
incompatible.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-11 14:17 ` Markus Armbruster
@ 2022-05-18 8:55 ` Victor Toso
2022-05-18 12:30 ` Markus Armbruster
0 siblings, 1 reply; 71+ messages in thread
From: Victor Toso @ 2022-05-18 8:55 UTC (permalink / raw)
To: Markus Armbruster
Cc: Daniel P. Berrangé,
John Snow, Eric Blake, qemu-devel, Marc-André Lureau
[-- Attachment #1: Type: text/plain, Size: 7150 bytes --]
Hi,
On Wed, May 11, 2022 at 04:17:35PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> > Caller
> >
> > block_resize(device="dev0", size=1*GiB)
> > block_resize(node_name="devnode0", size=1*GiB)
> >
> >
> > In golang definition
> >
> > type BlockResizeArguments struct {
> > Device string
> > NodeName string
> > Size int
> > }
> >
> > Caller choice of
> >
> > cmd := &BlockResizeCommand{
> > Device: "dev0",
> > Size: 1 * GiB,
> > }
> >
> > cmd := &BlockResizeCommand{
> > NodeName: "devnode0",
> > Size: 1 * GiB,
> > }
>
> Note that the Go bindings you sketched effectively use (poor
> man's) keyword arguments.
>
> > Neither case can easily prevent passing Device and NodeName
> > at same time.
>
> That defect lies at the schema's feet.
Right. The schema does not provide any metadata to explicit say
that only @device or @node-name should be used, correct?
This would be important to differentiate of a simple 'adding a
new optional argument' plus 'making this other argument
optional'.
> >> * At some future date, the old way gets deprecated: argument @device
> >> acquires feature @deprecated.
> >
> > Ok, no change needed to the APIs in either case. Possibly have
> > code emit a warning if a deprecated field is set.
> >
> >> * Still later, the old way gets removed: @device is deleted, and
> >> @node-name becomes mandatory.
> >
> > Again no change needed to APIs, but QEMU will throw back an
> > error if the wrong one is used.
> >
> >> What is the proper version-spanning interface?
> >>
> >> I figure it's both arguments optional, must specify the right one for
> >> the version of QEMU actually in use. This spans versions, but it fails
> >> to abstract from them.
> >
> > Yep, I think that's inevitable in this scenario. THe plus side
> > is that apps that want to span versions can do so. The downside
> > is that apps that don't want smarts to span version, may loose
> > compile time warnings about use of the now deleted field.
>
> The version-spanning interface will arguably be a bad interface for any
> version.
>
> > I suggested the code generator have an option to say what level
> > of compat to use for generated code, so that apps can request an
> > API without compat, which will result in compile errors. This
> > though assumes every consumer app is embedding their own
> > generated copy of the code. Not neccessarily desirable.
> >
> > At the C level you can play games with __deprecated__ to get
> > compile time warnings in some cases.
> >
> > #define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56
> >
> > causes QEMU to get compile time warnings (or errors) if it
> > attempts to use a API feature deprecated in 2.56, even if
> > the API exists in the header & library.
> >
> >
> >> Note that it's not enough to replace "delete member" by "mark member
> >> deleted in <version>". You also have to keep full history for "is it
> >> optional". And for types, because those can evolve compatibly, too,
> >> e.g. from struct to flat union, or from string to alternate of string
> >> and something else. What is the proper version-spanning interface in
> >> all the possible cases?
> >
> > I've not thought through all possible scenarios, but there may end
> > up being restrictions, such that changes that were previously possible
> > may have to be forbidden.
>
> "There may be restrictions" is not exactly a confidence-inspring design
> assumption. We need a reasonably dependable idea on what exactly we're
> intending to sacrifice.
I can't help much here but I guess we can evolve QAPI schema as
we move forward. Adding metadata that helps document changes to
the benefit of giving code generators tools to provide a way to
work with those QAPI changes seems desirable, no?
> > One example, in the past we could do deprecate a field 'foo', then
> > delete 'foo' and then some time re-introduce 'foo' with a completely
> > different type. That would not be possible if we wanted to maintain
> > compat, but in this example that's probably a good thing, as it'll
> > be super confusing to have the same field name change type like that
> > over time. Easier to just use a different name.
> >
> > So the question to me is not whether all our previous changes are
> > still possible, but whether enough of the typwes of change are
> > possible, such that we can cope with the ongoing maint in a
> > reasonable way. I don't think we've explored the possibility enough
> > to say one way or the other.
> >
> >> > Apps that wish to have version compat, would of course need to write
> >> > their code to be aware of which fields they need to seend for which
> >> > QEMU version.
> >>
> >> At which point we're reinventing libvirt.
IMHO, at this moment, qapi-go is targeting communicating with
QEMU and handling multiple QEMU versions seems reasonable to me.
Perhaps libvirt can use qapi-go in the future or other generated
interface. That would be cool.
> > The premise of the code generators is that there are applications
> > that want to consume QEMU that are not libvirt. With this line of
> > reasoning we could easily say that all such applications should
> > just use libvirt and then we don't need to provide any of these
> > code generators. The fact that we're considering these code
> > generators though, says that we're accepting there are valid use
> > cases that don't want to use libvirt for whatever reasons.
>
> Can't resist any longer: "What has libvirt ever done for us?"
> https://www.youtube.com/watch?v=Qc7HmhrgTuQ
>
> > It is
> > reasonable that some of those applications may wish to target
> > a wide range of QEMU versions, just like libvirt does.
>
> At which point they're comitting to reinventing the relevant parts of
> libvirt.
>
> I'd expect marshalling and umarshalling QMP to be among the
> smaller sub-problems then. It may look relatively large at
> first, because it's among the first ones to be solved. More so
> when you hand-wave away the more interesting ones of
> *abstraction* until they bite you in the posterior.
I might have missed it but I don't see unsolvable problems.
1) We decide if we want a Golang interface that can communicate
with multiple QEMU versions or not;
2) We discuss how that Golang interface would look like;
3) We discuss what is needed in QEMU/QAPI to achieve (2)
4) We work on QEMU/QAPI to address (3)
5) We move on with qapi-go proposal
I see only benefits with this project, plus the fact we already
have Golang projects doing their own code to communicate with
QEMU and I do believe we will make their lives easier.
> > It is also reasonable to say that libvirt would be better off
> > if it could auto-generate a client API for QEMU too, instead
> > of writing it by hand from a human reading the QAPI
> >
> > With regards,
> > Daniel
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-18 8:51 ` Daniel P. Berrangé
@ 2022-05-18 9:01 ` Victor Toso
0 siblings, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-05-18 9:01 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Andrea Bolognani, Markus Armbruster, John Snow, Eric Blake,
qemu-devel, Marc-André Lureau
[-- Attachment #1: Type: text/plain, Size: 5279 bytes --]
HI,
On Wed, May 18, 2022 at 09:51:56AM +0100, Daniel P. Berrangé wrote:
> On Wed, May 18, 2022 at 10:10:48AM +0200, Victor Toso wrote:
> > Hi,
> >
> > On Wed, May 11, 2022 at 04:50:35PM +0100, Daniel P. Berrangé wrote:
> > > On Wed, May 11, 2022 at 08:38:04AM -0700, Andrea Bolognani wrote:
> > > > On Tue, May 10, 2022 at 01:51:05PM +0100, Daniel P. Berrangé wrote:
> > > > > In 7.0.0 we can now generate
> > > > >
> > > > > type BlockResizeArguments struct {
> > > > > V500 *BlockResizeArgumentsV500
> > > > > V520 *BlockResizeArgumentsV520
> > > > > V700 *BlockResizeArgumentsV700
> > > > > }
> > > > >
> > > > > type BlockResizeArgumentsV500 struct {
> > > > > Device string
> > > > > Size int
> > > > > }
> > > > >
> > > > > type BlockResizeArgumentsV520 struct {
> > > > > Device *string
> > > > > NodeName *string
> > > > > Size int
> > > > > }
> > > > >
> > > > > type BlockResizeArgumentsV700 struct {
> > > > > NodeName string
> > > > > Size int
> > > > > }
> > > > >
> > > > > App can use the same as before, or switch to
> > > > >
> > > > > node := "nodedev0"
> > > > > cmd := BlockResizeArguments{
> > > > > V700: &BlockResizeArguments700{
> > > > > NodeName: node,
> > > > > Size: 1 * GiB
> > > > > }
> > > > > }
> > > >
> > > > This honestly looks pretty unwieldy.
> > >
> > > It isn't all that more verbose than without the versions - just
> > > a single struct wrapper.
> > >
> > > >
> > > > If the application already knows it's targeting a specific version of
> > > > the QEMU API, which for the above code to make any sense it will have
> > > > to, couldn't it do something like
> > > >
> > > > import qemu .../qemu/v700
> > > >
> > > > at the beginning of the file and then use regular old
> > > >
> > > > cmd := qemu.BlockResizeArguments{
> > > > NodeName: nodeName,
> > > > Size: size,
> > > > }
> > > >
> > > > instead?
> > >
> > > This would lead to a situation where every struct is duplicated
> > > for every version, even though 90% of the time they'll be identical
> > > across multiple versions. This is not very ammenable to the desire
> > > to be able to dynamically choose per-command which version you
> > > want based on which version of QEMU you're connected to.
> > >
> > > ie
> > >
> > >
> > > var cmd Command
> > > if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> > > cmd = BlockResizeArguments{
> > > V700: &BlockResizeArguments700{
> > > NodeName: node,
> > > Size: 1 * GiB
> > > }
> > > }
> > > } else {
> > > cmd = BlockResizeArguments{
> > > V520: &BlockResizeArguments520{
> > > Device: dev,
> > > Size: 1 * GiB
> > > }
> > > }
> > > }
> > >
> > > And of course the HasVersion check is going to be different
> > > for each command that matters.
> > >
> > > Having said that, this perhaps shows the nested structs are
> > > overkill. We could have
> > >
> > >
> > > var cmd Command
> > > if qmp.HasVersion(qemu.Version(7, 0, 0)) {
> > > cmd = &BlockResizeArguments700{
> > > NodeName: node,
> > > Size: 1 * GiB
> > > }
> > > } else {
> > > cmd = &BlockResizeArguments520{
> > > Device: dev,
> > > Size: 1 * GiB
> > > }
> > > }
> >
> > The else block would be wrong in versions above 7.0.0 where
> > block_resize changed. There will be a need to know for a specific
> > Type if we are covered with latest qemu/qapi-go or not. Not yet
> > sure how to address that, likely we will need to keep the
> > information that something has been added/changed/removed per
> > version per Type in qapi-go...
>
> I put that in the "nice to have" category. No application can
> predict the future, and nor do they really need to try in
> general.
>
> If the application code was written when the newest QEMU was
> 7.1.0, and the above code is correct for QEMU <= 7.1.0, then
> that's good enough. If the BlockResizeArguments struct changed
> in a later QEMU version 8.0.0, that doesn't matter at the point
> the app code was written.
I'm not thinking at runtime, I'm thinking at compile time.
I update $myproject's qpai-go to 8.0.0 and get a warnings that
some types would not work with 8.0.0 (e.g: because there is a new
BlockResizeArguments800)
> Much of the time the changes are back compatible, ie just adding
> a new field, and so everything will still work fine if the app
> carries on using BlockResizeArguments700, despite a new
> BlockResizeArguments800 arriving with a new field.
>
> Only in the cases where a field was removed or changed in a
> non-compatible manner would an app have problems, and QEMU will
> happily report an error at runtime if the app sends something
> incompatible.
Yeah, runtime error is fine but if we can provide some extra
checks at the time someone wants to move qapi-go from 7.2.0 to
8.0.0, that would be great.
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-18 8:55 ` Victor Toso
@ 2022-05-18 12:30 ` Markus Armbruster
2022-05-25 13:49 ` Andrea Bolognani
0 siblings, 1 reply; 71+ messages in thread
From: Markus Armbruster @ 2022-05-18 12:30 UTC (permalink / raw)
To: Victor Toso
Cc: Daniel P. Berrangé,
John Snow, Eric Blake, qemu-devel, Marc-André Lureau
Victor Toso <victortoso@redhat.com> writes:
> Hi,
>
> On Wed, May 11, 2022 at 04:17:35PM +0200, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> > Caller
>> >
>> > block_resize(device="dev0", size=1*GiB)
>> > block_resize(node_name="devnode0", size=1*GiB)
>> >
>> >
>> > In golang definition
>> >
>> > type BlockResizeArguments struct {
>> > Device string
>> > NodeName string
>> > Size int
>> > }
>> >
>> > Caller choice of
>> >
>> > cmd := &BlockResizeCommand{
>> > Device: "dev0",
>> > Size: 1 * GiB,
>> > }
>> >
>> > cmd := &BlockResizeCommand{
>> > NodeName: "devnode0",
>> > Size: 1 * GiB,
>> > }
>>
>> Note that the Go bindings you sketched effectively use (poor
>> man's) keyword arguments.
>>
>> > Neither case can easily prevent passing Device and NodeName
>> > at same time.
>>
>> That defect lies at the schema's feet.
>
> Right. The schema does not provide any metadata to explicit say
> that only @device or @node-name should be used, correct?
Correct.
The existing means to express either / or are tagged unions and
alternates.
Tagged unions require an explicit tag member.
Alternates select the alternative by the type of the value.
We don't have anything that selects by member name.
> This would be important to differentiate of a simple 'adding a
> new optional argument' plus 'making this other argument
> optional'.
We also don't have means to express "this integer must be a power of
two", or "this string must name a block backend", or any number of
semantic constraints.
We have to draw the line somewhere. Schema language complexity needs to
earn its keep.
>> >> * At some future date, the old way gets deprecated: argument @device
>> >> acquires feature @deprecated.
>> >
>> > Ok, no change needed to the APIs in either case. Possibly have
>> > code emit a warning if a deprecated field is set.
>> >
>> >> * Still later, the old way gets removed: @device is deleted, and
>> >> @node-name becomes mandatory.
>> >
>> > Again no change needed to APIs, but QEMU will throw back an
>> > error if the wrong one is used.
>> >
>> >> What is the proper version-spanning interface?
>> >>
>> >> I figure it's both arguments optional, must specify the right one for
>> >> the version of QEMU actually in use. This spans versions, but it fails
>> >> to abstract from them.
>> >
>> > Yep, I think that's inevitable in this scenario. THe plus side
>> > is that apps that want to span versions can do so. The downside
>> > is that apps that don't want smarts to span version, may loose
>> > compile time warnings about use of the now deleted field.
>>
>> The version-spanning interface will arguably be a bad interface for any
>> version.
>>
>> > I suggested the code generator have an option to say what level
>> > of compat to use for generated code, so that apps can request an
>> > API without compat, which will result in compile errors. This
>> > though assumes every consumer app is embedding their own
>> > generated copy of the code. Not neccessarily desirable.
>> >
>> > At the C level you can play games with __deprecated__ to get
>> > compile time warnings in some cases.
>> >
>> > #define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56
>> >
>> > causes QEMU to get compile time warnings (or errors) if it
>> > attempts to use a API feature deprecated in 2.56, even if
>> > the API exists in the header & library.
>> >
>> >
>> >> Note that it's not enough to replace "delete member" by "mark member
>> >> deleted in <version>". You also have to keep full history for "is it
>> >> optional". And for types, because those can evolve compatibly, too,
>> >> e.g. from struct to flat union, or from string to alternate of string
>> >> and something else. What is the proper version-spanning interface in
>> >> all the possible cases?
>> >
>> > I've not thought through all possible scenarios, but there may end
>> > up being restrictions, such that changes that were previously possible
>> > may have to be forbidden.
>>
>> "There may be restrictions" is not exactly a confidence-inspring design
>> assumption. We need a reasonably dependable idea on what exactly we're
>> intending to sacrifice.
>
> I can't help much here but I guess we can evolve QAPI schema as
> we move forward. Adding metadata that helps document changes to
> the benefit of giving code generators tools to provide a way to
> work with those QAPI changes seems desirable, no?
QMP comes with a certain compatibility promise. Keeping the promise has
been expensive, but that's okay; a weaker promise would have been
differently expensive. It's a compromise that has emerged over a long
time.
I fail to see why QEMU-provided QMP bindings should come with a stronger
promise than QEMU-provided QMP.
>> > One example, in the past we could do deprecate a field 'foo', then
>> > delete 'foo' and then some time re-introduce 'foo' with a completely
>> > different type. That would not be possible if we wanted to maintain
>> > compat, but in this example that's probably a good thing, as it'll
>> > be super confusing to have the same field name change type like that
>> > over time. Easier to just use a different name.
>> >
>> > So the question to me is not whether all our previous changes are
>> > still possible, but whether enough of the typwes of change are
>> > possible, such that we can cope with the ongoing maint in a
>> > reasonable way. I don't think we've explored the possibility enough
>> > to say one way or the other.
>> >
>> >> > Apps that wish to have version compat, would of course need to write
>> >> > their code to be aware of which fields they need to seend for which
>> >> > QEMU version.
>> >>
>> >> At which point we're reinventing libvirt.
>
> IMHO, at this moment, qapi-go is targeting communicating with
> QEMU and handling multiple QEMU versions seems reasonable to me.
It's targeting communicating in *QMP*. QMP is designed to support
communicating with a range of QEMU versions. Full compatibility is
promised for a narrow range. Outside that range, graceful degradation.
*If* you want to widen the full compatibility range, do it in *QMP*. Or
do it on top of QEMU, e.g. in libvirt.
> Perhaps libvirt can use qapi-go in the future or other generated
> interface. That would be cool.
"Would be cool" and a dollar buys you a cup of bad coffee.
Is it a good use of our limited resources?
How much will it delay delivery of Go bindings compared to less
ambitious version?
>> > The premise of the code generators is that there are applications
>> > that want to consume QEMU that are not libvirt. With this line of
>> > reasoning we could easily say that all such applications should
>> > just use libvirt and then we don't need to provide any of these
>> > code generators. The fact that we're considering these code
>> > generators though, says that we're accepting there are valid use
>> > cases that don't want to use libvirt for whatever reasons.
>>
>> Can't resist any longer: "What has libvirt ever done for us?"
>> https://www.youtube.com/watch?v=Qc7HmhrgTuQ
>>
>> > It is
>> > reasonable that some of those applications may wish to target
>> > a wide range of QEMU versions, just like libvirt does.
>>
>> At which point they're comitting to reinventing the relevant parts of
>> libvirt.
>>
>> I'd expect marshalling and umarshalling QMP to be among the
>> smaller sub-problems then. It may look relatively large at
>> first, because it's among the first ones to be solved. More so
>> when you hand-wave away the more interesting ones of
>> *abstraction* until they bite you in the posterior.
>
> I might have missed it but I don't see unsolvable problems.
>
> 1) We decide if we want a Golang interface that can communicate
> with multiple QEMU versions or not;
> 2) We discuss how that Golang interface would look like;
> 3) We discuss what is needed in QEMU/QAPI to achieve (2)
> 4) We work on QEMU/QAPI to address (3)
> 5) We move on with qapi-go proposal
This reinvents significant parts of libvirt in QEMU.
> I see only benefits with this project, plus the fact we already
> have Golang projects doing their own code to communicate with
> QEMU and I do believe we will make their lives easier.
Perhaps my idea of practical Go usage is hopelessly naive, but here goes
anyway.
Go project vendors current Go bindings (because they vendor everything).
Works with current QEMU, plus a range of past and future QEMUs.
Eventually something breaks with a then current QEMU. Go project
updates vendored Go bindings to current. Compiler points out everything
they need fixing (this assumes the type system is sufficiently
expressive, and bindings use it properly).
Good enough for these Golang projects?
Or asked in a different way: what exactly are the compatibility
requirements of these projects?
Please keep it to requirements *actual* projects have. Providing for
additional, non-trivial requirements hypothetical projects might have
is, in my reasoned opinion, folly we cannot afford. YAGNI until
demonstrated otherwise.
If different projects have different requirements, then QEMU having to
provide for all of them does not follow. Nothing absolves us from
weighing benefits vs. costs and risks.
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-18 12:30 ` Markus Armbruster
@ 2022-05-25 13:49 ` Andrea Bolognani
2022-05-25 14:10 ` Markus Armbruster
2022-06-01 13:53 ` Victor Toso
0 siblings, 2 replies; 71+ messages in thread
From: Andrea Bolognani @ 2022-05-25 13:49 UTC (permalink / raw)
To: Markus Armbruster
Cc: Victor Toso, Daniel P. Berrangé,
John Snow, Eric Blake, qemu-devel, Marc-André Lureau
On Wed, May 18, 2022 at 02:30:11PM +0200, Markus Armbruster wrote:
> Victor Toso <victortoso@redhat.com> writes:
> > IMHO, at this moment, qapi-go is targeting communicating with
> > QEMU and handling multiple QEMU versions seems reasonable to me.
>
> It's targeting communicating in *QMP*. QMP is designed to support
> communicating with a range of QEMU versions. Full compatibility is
> promised for a narrow range. Outside that range, graceful degradation.
>
> *If* you want to widen the full compatibility range, do it in *QMP*. Or
> do it on top of QEMU, e.g. in libvirt.
>
> > Perhaps libvirt can use qapi-go in the future or other generated
> > interface. That would be cool.
>
> "Would be cool" and a dollar buys you a cup of bad coffee.
>
> Is it a good use of our limited resources?
>
> How much will it delay delivery of Go bindings compared to less
> ambitious version?
Yeah, this thread has basically branched to cover three topics:
1. what an MVP Go interface for QMP should look like;
2. how to make sure said interface uses pretty names;
3. how to make it work across multiple QEMU versions.
All of these are important in the long run, but as far as I'm
concerned only 1. is an actual blocker to making progress.
If we get to the point where we can generate a reasonably complete
and well-typed Go interface that can be used to communicate with a
single version of QEMU, we should just plaster EXPERIMENTAL all over
it and get it merged.
Basically get the MVP done and then iterate over it in-tree rather
than trying to get everything perfect from the start.
Sounds reasonable?
--
Andrea Bolognani / Red Hat / Virtualization
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-25 13:49 ` Andrea Bolognani
@ 2022-05-25 14:10 ` Markus Armbruster
2022-06-01 13:53 ` Victor Toso
1 sibling, 0 replies; 71+ messages in thread
From: Markus Armbruster @ 2022-05-25 14:10 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Victor Toso, Daniel P. Berrangé,
John Snow, Eric Blake, qemu-devel, Marc-André Lureau
Andrea Bolognani <abologna@redhat.com> writes:
> On Wed, May 18, 2022 at 02:30:11PM +0200, Markus Armbruster wrote:
>> Victor Toso <victortoso@redhat.com> writes:
>> > IMHO, at this moment, qapi-go is targeting communicating with
>> > QEMU and handling multiple QEMU versions seems reasonable to me.
>>
>> It's targeting communicating in *QMP*. QMP is designed to support
>> communicating with a range of QEMU versions. Full compatibility is
>> promised for a narrow range. Outside that range, graceful degradation.
>>
>> *If* you want to widen the full compatibility range, do it in *QMP*. Or
>> do it on top of QEMU, e.g. in libvirt.
>>
>> > Perhaps libvirt can use qapi-go in the future or other generated
>> > interface. That would be cool.
>>
>> "Would be cool" and a dollar buys you a cup of bad coffee.
>>
>> Is it a good use of our limited resources?
>>
>> How much will it delay delivery of Go bindings compared to less
>> ambitious version?
>
> Yeah, this thread has basically branched to cover three topics:
>
> 1. what an MVP Go interface for QMP should look like;
> 2. how to make sure said interface uses pretty names;
> 3. how to make it work across multiple QEMU versions.
>
> All of these are important in the long run, but as far as I'm
> concerned only 1. is an actual blocker to making progress.
>
> If we get to the point where we can generate a reasonably complete
> and well-typed Go interface that can be used to communicate with a
> single version of QEMU, we should just plaster EXPERIMENTAL all over
> it and get it merged.
>
> Basically get the MVP done and then iterate over it in-tree rather
> than trying to get everything perfect from the start.
>
> Sounds reasonable?
Yes, with an undogmatic interpretation of "minimally viable". Doing
more should be okay when it doesn't complicate things outside the
Go-generating backend.
Exploring how to generate Go bindings that make good use of static
typing will be interesting enough. Aiming for wide compatibility in
addition risks delay and/or failure. Exploding heads, too.
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface
2022-05-25 13:49 ` Andrea Bolognani
2022-05-25 14:10 ` Markus Armbruster
@ 2022-06-01 13:53 ` Victor Toso
1 sibling, 0 replies; 71+ messages in thread
From: Victor Toso @ 2022-06-01 13:53 UTC (permalink / raw)
To: Andrea Bolognani
Cc: Markus Armbruster, Daniel P. Berrangé,
John Snow, Eric Blake, qemu-devel, Marc-André Lureau
[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]
Hi,
On Wed, May 25, 2022 at 08:49:19AM -0500, Andrea Bolognani wrote:
> On Wed, May 18, 2022 at 02:30:11PM +0200, Markus Armbruster wrote:
> > Victor Toso <victortoso@redhat.com> writes:
> > > IMHO, at this moment, qapi-go is targeting communicating with
> > > QEMU and handling multiple QEMU versions seems reasonable to me.
> >
> > It's targeting communicating in *QMP*. QMP is designed to support
> > communicating with a range of QEMU versions. Full compatibility is
> > promised for a narrow range. Outside that range, graceful degradation.
> >
> > *If* you want to widen the full compatibility range, do it in *QMP*. Or
> > do it on top of QEMU, e.g. in libvirt.
> >
> > > Perhaps libvirt can use qapi-go in the future or other generated
> > > interface. That would be cool.
> >
> > "Would be cool" and a dollar buys you a cup of bad coffee.
> >
> > Is it a good use of our limited resources?
> >
> > How much will it delay delivery of Go bindings compared to less
> > ambitious version?
>
> Yeah, this thread has basically branched to cover three topics:
>
> 1. what an MVP Go interface for QMP should look like;
> 2. how to make sure said interface uses pretty names;
> 3. how to make it work across multiple QEMU versions.
>
> All of these are important in the long run, but as far as I'm
> concerned only 1. is an actual blocker to making progress.
I agree although (1) and (3) are holding hands a bit.
> If we get to the point where we can generate a reasonably
> complete and well-typed Go interface that can be used to
> communicate with a single version of QEMU, we should just
> plaster EXPERIMENTAL all over it and get it merged.
>
> Basically get the MVP done and then iterate over it in-tree
> rather than trying to get everything perfect from the start.
>
> Sounds reasonable?
Yep. The whole discussion has been great as to clarify
limitations and possible goals but not aiming to get it perfect
all at once seems reasonable.
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
end of thread, other threads:[~2022-06-01 13:54 UTC | newest]
Thread overview: 71+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-01 22:40 [RFC PATCH v1 0/8] qapi: add generator for Golang interface Victor Toso
2022-04-01 22:40 ` [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go Victor Toso
2022-05-10 10:06 ` Daniel P. Berrangé
2022-05-10 11:15 ` Victor Toso
2022-05-10 11:19 ` Daniel P. Berrangé
2022-05-10 11:28 ` Victor Toso
2022-05-10 11:39 ` Daniel P. Berrangé
2022-04-01 22:40 ` [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate " Victor Toso
2022-05-10 10:10 ` Daniel P. Berrangé
2022-05-10 11:21 ` Victor Toso
2022-05-10 11:28 ` Daniel P. Berrangé
2022-04-01 22:40 ` [RFC PATCH v1 3/8] qapi: golang: Generate qapi's struct " Victor Toso
2022-04-01 22:41 ` [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union " Victor Toso
2022-05-10 10:26 ` Daniel P. Berrangé
2022-05-10 11:32 ` Victor Toso
2022-05-10 11:42 ` Daniel P. Berrangé
2022-04-01 22:41 ` [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event " Victor Toso
2022-05-10 10:42 ` Daniel P. Berrangé
2022-05-10 11:38 ` Victor Toso
2022-04-01 22:41 ` [RFC PATCH v1 6/8] qapi: golang: Generate qapi's command " Victor Toso
2022-04-01 22:41 ` [RFC PATCH v1 7/8] qapi: golang: Add CommandResult type to Go Victor Toso
2022-04-01 22:41 ` [RFC PATCH v1 8/8] qapi: golang: document skip function visit_array_types Victor Toso
2022-04-19 18:12 ` [RFC PATCH v1 0/8] qapi: add generator for Golang interface Andrea Bolognani
2022-04-19 18:42 ` Andrea Bolognani
2022-04-28 13:50 ` Markus Armbruster
2022-04-29 13:15 ` Andrea Bolognani
2022-05-02 7:21 ` Markus Armbruster
2022-05-02 9:04 ` Andrea Bolognani
2022-05-02 11:46 ` Markus Armbruster
2022-05-02 14:01 ` Andrea Bolognani
2022-05-03 7:57 ` Markus Armbruster
2022-05-03 9:40 ` Andrea Bolognani
2022-05-03 11:04 ` Kevin Wolf
2022-05-10 9:55 ` Daniel P. Berrangé
2022-05-11 6:15 ` Markus Armbruster
2022-05-09 18:53 ` Victor Toso
2022-05-10 8:06 ` Markus Armbruster
2022-05-10 11:48 ` Victor Toso
2022-05-10 9:52 ` Daniel P. Berrangé
2022-05-10 15:25 ` Andrea Bolognani
2022-05-11 13:45 ` Markus Armbruster
2022-05-09 10:21 ` Victor Toso
2022-05-10 17:37 ` Andrea Bolognani
2022-05-10 18:02 ` Daniel P. Berrangé
2022-04-26 11:14 ` Markus Armbruster
2022-05-09 10:52 ` Victor Toso
2022-05-10 8:53 ` Daniel P. Berrangé
2022-05-10 9:06 ` Victor Toso
2022-05-10 9:17 ` Daniel P. Berrangé
2022-05-10 9:32 ` Daniel P. Berrangé
2022-05-10 10:50 ` Victor Toso
2022-05-10 10:57 ` Daniel P. Berrangé
2022-05-10 12:02 ` Markus Armbruster
2022-05-10 12:34 ` Daniel P. Berrangé
2022-05-10 12:51 ` Daniel P. Berrangé
2022-05-11 14:17 ` Markus Armbruster
2022-05-11 14:41 ` Daniel P. Berrangé
2022-05-11 15:38 ` Andrea Bolognani
2022-05-11 15:50 ` Daniel P. Berrangé
2022-05-11 16:22 ` Andrea Bolognani
2022-05-11 16:32 ` Daniel P. Berrangé
2022-05-18 8:10 ` Victor Toso
2022-05-18 8:51 ` Daniel P. Berrangé
2022-05-18 9:01 ` Victor Toso
2022-05-11 14:17 ` Markus Armbruster
2022-05-18 8:55 ` Victor Toso
2022-05-18 12:30 ` Markus Armbruster
2022-05-25 13:49 ` Andrea Bolognani
2022-05-25 14:10 ` Markus Armbruster
2022-06-01 13:53 ` Victor Toso
2022-05-10 10:47 ` Daniel P. Berrangé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.