qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples
@ 2021-09-07 12:19 marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 01/32] RFC: docs: add supported host CPUs section marcandre.lureau
                   ` (32 more replies)
  0 siblings, 33 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Hi,

Among the QEMU developers, there is a desire to use Rust. (see previous
thread from Stefan "Why QEMU should move from C to Rust", the rust-vmm
related projects and other experiments).

Thanks to our QAPI type system and the associate code generator, it is possible
to create Rust bindings for the generated C types (also called FFI binding) and
functions (rust-bindgen could probably do a similar job, but it is unnecessary
given the FFI generator is quite straightforward).

More interesting (but more complicated) is to expose a Rust API, and provide
convenient conversions FFI<->Rust. Taking inspiration from glib-rs binding, I
implemented a simplified version of the FromGlib/ToGlib traits, with simpler
ownership model, sufficient for QAPI needs.

The usage is as such:

- from_qemu_none(ptr: *const sys::P) -> T
  Return a Rust type T for a const ffi pointer P.

- from_qemu_full(ptr: *mut sys::P) -> T
  Return a Rust type T for a ffi pointer P, taking ownership.

- T::to_qemu_none() -> Stash<P>
  Returns a borrowed ffi pointer P (using a Stash to destroy "glue"
  storage data, if any).

- T::to_qemu_full() -> P
  Returns a ffi pointer P. (P resources are leaked/passed to C/ffi)

With the Rust QAPI bindings, we can implement QMP callbacks in idiomatic Rust.
With the translation traits, we can also bind manually more types and implement
other functions (QOM is my next target).

Since Rust doesn't cover all the platforms QEMU supports (see first patch and
https://doc.rust-lang.org/nightly/rustc/platform-support.html), the Rust code
should be optional. Nevertheless, it is desirable to write sensitive code in
Rust first, and a C version if the code/feature is not optional and not specific
to a platform where Rust is Tier-1 or 2.

v3 (RFC):
 - rebased (was waiting for qapi 'if' conditions to be merged)
 - now covers the various QEMU QAPI schemas (QEMU schema is left out of this
   series, but test schema is the complete)
 - numerous improvements, patch reorgnisation, cleanups etc

v2 (PoC):
 - split the original patch in smaller patches and more digestable form
 - dropped the DBus interface experiment from this series
 - various build-sys improvements, new configure options --with-rust(-target)
 - various attempts at better meson integration, finally satisfied enough with the
   current solution, which handles getting link flags from Rust sanely.
   (more meson stuff to come to handle config-host/features mapping etc).
 - rebased, QGA QMP now uses unions, added support for it.
 - start a common crate (which re-surfaced issues with foreign types and traits,
   worked around with a NewPtr wrapper)
 - explicit errors when ifcond are presents (after various unsucessful attempts,
   I will try to tackle it in v3)
 - mingw cross compilation support
 - some attempts to add it to CI
 - actually implement {get,set}-vcpus
 - vendor the external crates

Marc-André Lureau (32):
  RFC: docs: add supported host CPUs section
  build-sys: add HAVE_IPPROTO_MPTCP
  scripts/qapi: teach c_param_type() to return const argument type
  glib-compat: add G_SIZEOF_MEMBER
  scripts/qapi: add QAPISchemaVisitor.visit_module_end
  scripts/qapi: add a CABI module
  scripts/qapi: generate CABI dump for C types
  tests: build qapi-cabi (C ABI dump)
  build-sys: add i686 cpu target
  build-sys: add --with-rust{-target} & basic build infrastructure
  build-sys: add a cargo-wrapper script
  rust: provide a common crate for QEMU
  rust: use vendored-sources
  scripts/qapi: add QAPISchemaIfCond.rsgen()
  scripts/qapi: strip trailing whitespaces
  scripts/qapi: add Rust FFI bindings generation
  scripts/qapi: learn to generate ABI dump for Rust FFI
  tests: generate Rust bindings
  tests: check Rust and C CABI diffs
  scripts/qapi: generate high-level Rust bindings
  tests/rust: build a common library, checking bindings compile
  qga: build qapi-cabi binary (ABI from C)
  qga/rust: build and link an empty static library
  qga/rust: generate QGA QAPI types FFI bindings
  qga/rust: build a qga-cabi-rs executable (ABI from Rust)
  qga/rust: check the Rust C binding
  qga/rust: build high-level Rust QAPI types
  qga/rust: implement get-host-name in Rust (example)
  qga/rust: implement {get,set}-vcpus in Rust (example)
  tests/vm: add Rust to FreeBSD VM
  tests/vm: bump fedora VM to f32
  tests/vm: add Rust to Fedora

 docs/about/build-platforms.rst |  28 +
 configure                      |  32 +-
 meson.build                    |  49 +-
 qapi/sockets.json              |   2 +-
 include/glib-compat.h          |   7 +
 include/qemu/osdep.h           |  10 -
 io/dns-resolver.c              |   2 +-
 qga/commands-posix.c           | 152 ------
 qga/commands-win32.c           |  71 ---
 qga/commands.c                 |  34 +-
 qga/qapi-cabi.c                |   9 +
 tests/qapi-cabi.c              |   9 +
 tests/unit/test-qga.c          |   4 +
 util/oslib-posix.c             |  35 --
 util/oslib-win32.c             |  13 -
 util/qemu-sockets.c            |   6 +-
 .cargo/config.toml.in          |   5 +
 .cargo/meson.build             |   5 +
 .gitignore                     |   1 +
 .gitmodules                    |   4 +
 Cargo.lock                     | 119 ++++
 Cargo.toml                     |   6 +
 meson_options.txt              |   5 +
 qga/Cargo.toml                 |  24 +
 qga/lib.rs                     |   5 +
 qga/meson.build                |  64 ++-
 qga/qapi-cabi.rs               |   6 +
 qga/qapi.rs                    |   6 +
 qga/qapi_ffi.rs                |   8 +
 qga/qmp/hostname.rs            |   9 +
 qga/qmp/mod.rs                 |  28 +
 qga/qmp/vcpus.rs               | 161 ++++++
 rust/common/Cargo.toml         |  11 +
 rust/common/src/error.rs       | 113 ++++
 rust/common/src/ffi.rs         |  93 ++++
 rust/common/src/lib.rs         |  21 +
 rust/common/src/qemu.rs        | 101 ++++
 rust/common/src/qmp.rs         |   0
 rust/common/src/translate.rs   | 482 ++++++++++++++++
 rust/vendored                  |   1 +
 scripts/archive-source.sh      |   2 +-
 scripts/cargo_wrapper.py       | 183 +++++++
 scripts/configh_to_cfg.py      |  44 ++
 scripts/diff_commands.py       |  40 ++
 scripts/qapi/cabi.py           | 187 +++++++
 scripts/qapi/common.py         |  16 +
 scripts/qapi/gen.py            |   6 +-
 scripts/qapi/main.py           |  25 +-
 scripts/qapi/rs.py             | 262 +++++++++
 scripts/qapi/rs_ffi.py         | 446 +++++++++++++++
 scripts/qapi/rs_types.py       | 966 +++++++++++++++++++++++++++++++++
 scripts/qapi/schema.py         |  23 +-
 scripts/qapi/types.py          |  58 +-
 tests/Cargo.toml               |  17 +
 tests/lib.rs                   |   2 +
 tests/meson.build              |  72 ++-
 tests/qapi-cabi.rs             |   5 +
 tests/qapi.rs                  |  11 +
 tests/qapi_ffi.rs              |   8 +
 tests/vm/fedora                |  12 +-
 tests/vm/freebsd               |   1 +
 61 files changed, 3794 insertions(+), 333 deletions(-)
 create mode 100644 qga/qapi-cabi.c
 create mode 100644 tests/qapi-cabi.c
 create mode 100644 .cargo/config.toml.in
 create mode 100644 .cargo/meson.build
 create mode 100644 Cargo.lock
 create mode 100644 Cargo.toml
 create mode 100644 qga/Cargo.toml
 create mode 100644 qga/lib.rs
 create mode 100644 qga/qapi-cabi.rs
 create mode 100644 qga/qapi.rs
 create mode 100644 qga/qapi_ffi.rs
 create mode 100644 qga/qmp/hostname.rs
 create mode 100644 qga/qmp/mod.rs
 create mode 100644 qga/qmp/vcpus.rs
 create mode 100644 rust/common/Cargo.toml
 create mode 100644 rust/common/src/error.rs
 create mode 100644 rust/common/src/ffi.rs
 create mode 100644 rust/common/src/lib.rs
 create mode 100644 rust/common/src/qemu.rs
 create mode 100644 rust/common/src/qmp.rs
 create mode 100644 rust/common/src/translate.rs
 create mode 160000 rust/vendored
 create mode 100644 scripts/cargo_wrapper.py
 create mode 100644 scripts/configh_to_cfg.py
 create mode 100644 scripts/diff_commands.py
 create mode 100644 scripts/qapi/cabi.py
 create mode 100644 scripts/qapi/rs.py
 create mode 100644 scripts/qapi/rs_ffi.py
 create mode 100644 scripts/qapi/rs_types.py
 create mode 100644 tests/Cargo.toml
 create mode 100644 tests/lib.rs
 create mode 100644 tests/qapi-cabi.rs
 create mode 100644 tests/qapi.rs
 create mode 100644 tests/qapi_ffi.rs

-- 
2.33.0.113.g6c40894d24




^ permalink raw reply	[flat|nested] 74+ messages in thread

* [RFC v3 01/32] RFC: docs: add supported host CPUs section
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:33   ` Peter Maydell
  2021-09-07 12:19 ` [RFC v3 02/32] build-sys: add HAVE_IPPROTO_MPTCP marcandre.lureau
                   ` (31 subsequent siblings)
  32 siblings, 1 reply; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

I was looking for such documentation, but couldn't find it.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 docs/about/build-platforms.rst | 28 ++++++++++++++++++++++++++++
 meson.build                    |  2 +-
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
index 692323609e..bfe90e574e 100644
--- a/docs/about/build-platforms.rst
+++ b/docs/about/build-platforms.rst
@@ -29,6 +29,34 @@ The `Repology`_ site is a useful resource to identify
 currently shipped versions of software in various operating systems,
 though it does not cover all distros listed below.
 
+Supported host CPUs
+-------------------
+
+Those host CPUs have a native TCG backend and are regularly tested:
+
+  .. list-table::
+   :header-rows: 1
+
+   * - CPU Family
+     - Accelerators
+   * - ARM
+     - kvm, xen
+   * - MIPS
+     - kvm
+   * - PPC
+     - kvm
+   * - RISC-V
+     -
+   * - s390x
+     - kvm
+   * - SPARC
+     -
+   * - x86
+     - kvm, xen, hax, hvf (64 bit only), nvmm, whpx (64 bit only)
+
+Other architectures are not actively maintained. They use the slow and
+experimental TCG interpreter. They may be removed in future releases.
+
 Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
 -----------------------------------------
 
diff --git a/meson.build b/meson.build
index 7e58e6279b..9e43c9b311 100644
--- a/meson.build
+++ b/meson.build
@@ -78,7 +78,7 @@ endif
 
 accelerator_targets = { 'CONFIG_KVM': kvm_targets }
 if cpu in ['x86', 'x86_64', 'arm', 'aarch64']
-  # i368 emulator provides xenpv machine type for multiple architectures
+  # i386 emulator provides xenpv machine type for multiple architectures
   accelerator_targets += {
     'CONFIG_XEN': ['i386-softmmu', 'x86_64-softmmu'],
   }
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 02/32] build-sys: add HAVE_IPPROTO_MPTCP
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 01/32] RFC: docs: add supported host CPUs section marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 12:01   ` Markus Armbruster
  2021-09-13 13:02   ` Paolo Bonzini
  2021-09-07 12:19 ` [RFC v3 03/32] scripts/qapi: teach c_param_type() to return const argument type marcandre.lureau
                   ` (30 subsequent siblings)
  32 siblings, 2 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

The QAPI schema shouldn't rely on C system headers #define, but on
configure-time project #define, so we can express the build condition in
a C-independent way.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 meson.build         | 2 ++
 qapi/sockets.json   | 2 +-
 io/dns-resolver.c   | 2 +-
 util/qemu-sockets.c | 6 +++---
 4 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/meson.build b/meson.build
index 9e43c9b311..6e871af4d0 100644
--- a/meson.build
+++ b/meson.build
@@ -1367,6 +1367,8 @@ config_host_data.set('HAVE_OPTRESET',
                      cc.has_header_symbol('getopt.h', 'optreset'))
 config_host_data.set('HAVE_UTMPX',
                      cc.has_header_symbol('utmpx.h', 'struct utmpx'))
+config_host_data.set('HAVE_IPPROTO_MPTCP',
+                     cc.has_header_symbol('netinet/in.h', 'IPPROTO_MPTCP'))
 
 # has_member
 config_host_data.set('HAVE_SIGEV_NOTIFY_THREAD_ID',
diff --git a/qapi/sockets.json b/qapi/sockets.json
index 7866dc27d6..c9101f937f 100644
--- a/qapi/sockets.json
+++ b/qapi/sockets.json
@@ -69,7 +69,7 @@
     '*ipv4': 'bool',
     '*ipv6': 'bool',
     '*keep-alive': 'bool',
-    '*mptcp': { 'type': 'bool', 'if': 'IPPROTO_MPTCP' } } }
+    '*mptcp': { 'type': 'bool', 'if': 'HAVE_IPPROTO_MPTCP' } } }
 
 ##
 # @UnixSocketAddress:
diff --git a/io/dns-resolver.c b/io/dns-resolver.c
index a5946a93bf..53b0e8407a 100644
--- a/io/dns-resolver.c
+++ b/io/dns-resolver.c
@@ -122,7 +122,7 @@ static int qio_dns_resolver_lookup_sync_inet(QIODNSResolver *resolver,
             .ipv4 = iaddr->ipv4,
             .has_ipv6 = iaddr->has_ipv6,
             .ipv6 = iaddr->ipv6,
-#ifdef IPPROTO_MPTCP
+#ifdef HAVE_IPPROTO_MPTCP
             .has_mptcp = iaddr->has_mptcp,
             .mptcp = iaddr->mptcp,
 #endif
diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c
index f2f3676d1f..02eb2f3d34 100644
--- a/util/qemu-sockets.c
+++ b/util/qemu-sockets.c
@@ -278,7 +278,7 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
 
     /* create socket + bind/listen */
     for (e = res; e != NULL; e = e->ai_next) {
-#ifdef IPPROTO_MPTCP
+#ifdef HAVE_IPPROTO_MPTCP
         if (saddr->has_mptcp && saddr->mptcp) {
             e->ai_protocol = IPPROTO_MPTCP;
         }
@@ -462,7 +462,7 @@ int inet_connect_saddr(InetSocketAddress *saddr, Error **errp)
         error_free(local_err);
         local_err = NULL;
 
-#ifdef IPPROTO_MPTCP
+#ifdef HAVE_IPPROTO_MPTCP
         if (saddr->has_mptcp && saddr->mptcp) {
             e->ai_protocol = IPPROTO_MPTCP;
         }
@@ -699,7 +699,7 @@ int inet_parse(InetSocketAddress *addr, const char *str, Error **errp)
         }
         addr->has_keep_alive = true;
     }
-#ifdef IPPROTO_MPTCP
+#ifdef HAVE_IPPROTO_MPTCP
     begin = strstr(optstr, ",mptcp");
     if (begin) {
         if (inet_parse_flag("mptcp", begin + strlen(",mptcp"),
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 03/32] scripts/qapi: teach c_param_type() to return const argument type
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 01/32] RFC: docs: add supported host CPUs section marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 02/32] build-sys: add HAVE_IPPROTO_MPTCP marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 12:10   ` Markus Armbruster
  2021-09-07 12:19 ` [RFC v3 04/32] glib-compat: add G_SIZEOF_MEMBER marcandre.lureau
                   ` (29 subsequent siblings)
  32 siblings, 1 reply; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

The argument isn't owned by the callee, so it better be const.
But a lot of code in QEMU rely on non-const arguments to tweak it (steal
values etc).

Since Rust types / bindings are derived from the C version, we have to
be more accurate there to do correct ownership in the bindings.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi/schema.py | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 3d72c7dfc9..1f6301c394 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -226,8 +226,15 @@ def c_type(self):
         pass
 
     # Return the C type to be used in a parameter list.
-    def c_param_type(self):
-        return self.c_type()
+    #
+    # The argument should be considered const, since no ownership is given to
+    # the callee, but qemu C code frequently tweaks it. Set const=True for a
+    # stricter declaration.
+    def c_param_type(self, const: bool = False):
+        c_type = self.c_type()
+        if const and c_type.endswith(POINTER_SUFFIX):
+            c_type = 'const ' + c_type
+        return c_type
 
     # Return the C type to be used where we suppress boxing.
     def c_unboxed_type(self):
@@ -280,10 +287,10 @@ def c_name(self):
     def c_type(self):
         return self._c_type_name
 
-    def c_param_type(self):
+    def c_param_type(self, const: bool = False):
         if self.name == 'str':
             return 'const ' + self._c_type_name
-        return self._c_type_name
+        return super().c_param_type(const)
 
     def json_type(self):
         return self._json_type_name
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 04/32] glib-compat: add G_SIZEOF_MEMBER
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (2 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 03/32] scripts/qapi: teach c_param_type() to return const argument type marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 12:16   ` Markus Armbruster
  2021-09-07 12:19 ` [RFC v3 05/32] scripts/qapi: add QAPISchemaVisitor.visit_module_end marcandre.lureau
                   ` (28 subsequent siblings)
  32 siblings, 1 reply; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

The macro was introduced in 2.64. Let's redefine it, to allow its usage
with older versions and silence the deprecation warning.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 include/glib-compat.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/include/glib-compat.h b/include/glib-compat.h
index 9e95c888f5..e278e09d1d 100644
--- a/include/glib-compat.h
+++ b/include/glib-compat.h
@@ -110,4 +110,11 @@ qemu_g_test_slow(void)
 
 #pragma GCC diagnostic pop
 
+/* introduced in 2.64 */
+#ifdef G_SIZEOF_MEMBER
+#undef G_SIZEOF_MEMBER
+#endif
+
+#define G_SIZEOF_MEMBER(type, member) sizeof(((type *)0)->member)
+
 #endif
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 05/32] scripts/qapi: add QAPISchemaVisitor.visit_module_end
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (3 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 04/32] glib-compat: add G_SIZEOF_MEMBER marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 12:26   ` Markus Armbruster
  2021-09-07 12:19 ` [RFC v3 06/32] scripts/qapi: add a CABI module marcandre.lureau
                   ` (27 subsequent siblings)
  32 siblings, 1 reply; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Used in following patches to generate code after visiting a module.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi/schema.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 1f6301c394..6455a8f425 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -128,6 +128,9 @@ def visit_end(self):
     def visit_module(self, name):
         pass
 
+    def visit_module_end(self, name) -> None:
+        pass
+
     def visit_needed(self, entity):
         # Default to visiting everything
         return True
@@ -207,6 +210,7 @@ def visit(self, visitor):
         for entity in self._entity_list:
             if visitor.visit_needed(entity):
                 entity.visit(visitor)
+        visitor.visit_module_end(self.name)
 
 
 class QAPISchemaInclude(QAPISchemaEntity):
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 06/32] scripts/qapi: add a CABI module
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (4 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 05/32] scripts/qapi: add QAPISchemaVisitor.visit_module_end marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 07/32] scripts/qapi: generate CABI dump for C types marcandre.lureau
                   ` (26 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

This module will help to build C ABI tests for both Rust and C QAPI
generated types.

Currently, it will simply print basic struct layout (size and offset of
members) and the enum size and maximum values. By diffing the output, it
will help to ensure that the Rust definition is compatible with the C
version, including with configuration specifics.

Example output:

GuestDeviceAddress struct: sizeof=16
 type member: sizeof=4 offset=0

GuestDeviceAddressKind enum: sizeof=4
 max=1

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 meson.build          |   1 +
 scripts/qapi/cabi.py | 137 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 138 insertions(+)
 create mode 100644 scripts/qapi/cabi.py

diff --git a/meson.build b/meson.build
index 6e871af4d0..cbf3ce19ae 100644
--- a/meson.build
+++ b/meson.build
@@ -1966,6 +1966,7 @@ hxtool = find_program('scripts/hxtool')
 shaderinclude = find_program('scripts/shaderinclude.pl')
 qapi_gen = find_program('scripts/qapi-gen.py')
 qapi_gen_depends = [ meson.source_root() / 'scripts/qapi/__init__.py',
+                     meson.source_root() / 'scripts/qapi/cabi.py',
                      meson.source_root() / 'scripts/qapi/commands.py',
                      meson.source_root() / 'scripts/qapi/common.py',
                      meson.source_root() / 'scripts/qapi/error.py',
diff --git a/scripts/qapi/cabi.py b/scripts/qapi/cabi.py
new file mode 100644
index 0000000000..f33680477f
--- /dev/null
+++ b/scripts/qapi/cabi.py
@@ -0,0 +1,137 @@
+#
+# QAPI helper library
+# C ABI verification test generator
+#
+# pylint: disable=too-few-public-methods
+
+from typing import List, Optional
+
+from .common import c_enum_const, c_name, mcgen
+from .schema import (
+    QAPISchemaEnumMember,
+    QAPISchemaIfCond,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaVariants,
+)
+
+
+class CABI:
+    def __init__(self, name: str, ifcond: QAPISchemaIfCond):
+        self.name = name
+        self.ifcond = ifcond
+
+    def gen_c(self) -> str:
+        raise NotImplementedError()
+
+
+class CABIEnum(CABI):
+    def __init__(
+        self,
+        name: str,
+        ifcond: QAPISchemaIfCond,
+        members: List[QAPISchemaEnumMember],
+        prefix: Optional[str] = None,
+    ):
+        super().__init__(name, ifcond)
+        self.members = members
+        self.prefix = prefix
+
+    def gen_c(self) -> str:
+        last = c_enum_const(self.name, "_MAX", self.prefix)
+        ret = self.ifcond.gen_if()
+        ret += mcgen("""
+    printf("%(name)s enum: sizeof=%%zu\\n", sizeof(%(cname)s));
+    printf(" max=%%d\\n", %(last)s);
+    printf("\\n");
+""", name=self.name, cname=c_name(self.name), last=last)
+        ret += self.ifcond.gen_endif()
+        return ret
+
+
+class CABIStruct(CABI):
+    def __init__(self, name: str, ifcond: QAPISchemaIfCond):
+        super().__init__(name, ifcond)
+        self.members: List[CABIStructMember] = []
+
+    def add_members(self, members: List[QAPISchemaObjectTypeMember]) -> None:
+        for memb in members:
+            if memb.optional:
+                self.add_member(memb.name, memb.ifcond, "has_")
+            self.add_member(memb.name, memb.ifcond)
+
+    def add_variants(self, variants: QAPISchemaVariants) -> None:
+        for var in variants.variants:
+            if var.type.name == "q_empty":
+                continue
+            self.add_member(var.name, var.ifcond, "u.")
+
+    def add_member(self, member: str,
+                   ifcond: Optional[QAPISchemaIfCond] = None,
+                   prefix: str = '') -> None:
+        self.members.append(CABIStructMember(self, member, ifcond, prefix))
+
+    def gen_c(self) -> str:
+        ret = self.ifcond.gen_if()
+        ret += mcgen("""
+    printf("%(name)s struct: sizeof=%%zu\\n", sizeof(%(name)s));
+""", name=self.name)
+        for member in self.members:
+            ret += member.gen_c()
+        ret += mcgen("""
+    printf("\\n");
+""")
+        ret += self.ifcond.gen_endif()
+        return ret
+
+
+class CABIStructMember:
+    def __init__(self, struct: CABIStruct, name: str,
+                 ifcond: Optional[QAPISchemaIfCond] = None,
+                 prefix: str = ''):
+        self.struct = struct
+        self.name = name
+        self.ifcond = ifcond or QAPISchemaIfCond()
+        self.prefix = prefix
+
+    def gen_c(self) -> str:
+        ret = self.ifcond.gen_if()
+        cmember = self.prefix + c_name(self.name)
+        ret += mcgen("""
+    printf(" %(member)s member: sizeof=%%zu offset=%%zu\\n",
+            G_SIZEOF_MEMBER(struct %(sname)s, %(cmember)s),
+            offsetof(struct %(sname)s, %(cmember)s));
+""", member=self.name, sname=self.struct.name, cmember=cmember)
+        ret += self.ifcond.gen_endif()
+        return ret
+
+
+def gen_object_cabi(
+    name: str,
+    ifcond: QAPISchemaIfCond,
+    base: Optional[QAPISchemaObjectType],
+    members: List[QAPISchemaObjectTypeMember],
+    variants: Optional[QAPISchemaVariants],
+) -> List[CABI]:
+    if name == 'q_empty':
+        return []
+    ret = []
+    for var in variants.variants if variants else ():
+        obj = var.type
+        if not isinstance(obj, QAPISchemaObjectType):
+            continue
+        ret.extend(
+            gen_object_cabi(
+                obj.name, obj.ifcond, obj.base, obj.local_members, obj.variants
+            )
+        )
+    cabi = CABIStruct(c_name(name), ifcond)
+    if base:
+        cabi.add_members(base.members)
+    cabi.add_members(members)
+    if variants:
+        cabi.add_variants(variants)
+    if (not base or base.is_empty()) and not members and not variants:
+        cabi.add_member('qapi_dummy_for_empty_struct')
+    ret.append(cabi)
+    return ret
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 07/32] scripts/qapi: generate CABI dump for C types
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (5 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 06/32] scripts/qapi: add a CABI module marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 08/32] tests: build qapi-cabi (C ABI dump) marcandre.lureau
                   ` (25 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

With type units, generate the C ABI dump functions (as blocks protected
with QAPI_CABI define). The top-level function is qapi_cabi(), and it
calls the sub-modules dump functions.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi/types.py | 58 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 831294fe42..7ac3645d18 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,8 +13,15 @@
 # See the COPYING file in the top-level directory.
 """
 
-from typing import List, Optional
+from pathlib import Path
+from typing import (
+    Dict,
+    List,
+    Optional,
+    Set,
+)
 
+from .cabi import CABI, CABIEnum, gen_object_cabi
 from .common import c_enum_const, c_name, mcgen
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import (
@@ -22,6 +29,7 @@
     QAPISchemaEnumMember,
     QAPISchemaFeature,
     QAPISchemaIfCond,
+    QAPISchemaModule,
     QAPISchemaObjectType,
     QAPISchemaObjectTypeMember,
     QAPISchemaType,
@@ -265,6 +273,13 @@ def __init__(self, prefix: str):
         super().__init__(
             prefix, 'qapi-types', ' * Schema-defined QAPI types',
             ' * Built-in QAPI types', __doc__)
+        self._cabi_functions: List[str] = []
+        self._cabi_functions_called: Set[str] = set()
+        self._cabi: Dict[str, CABI] = {}
+
+    def _cabi_add(self, cabis: List[CABI]) -> None:
+        for cabi in cabis:
+            self._cabi.setdefault(cabi.name, cabi)
 
     def _begin_builtin_module(self) -> None:
         self._genc.preamble_add(mcgen('''
@@ -295,6 +310,43 @@ def visit_begin(self, schema: QAPISchema) -> None:
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
 
+    def _get_qapi_cabi_fn(self, name: str) -> str:
+        fn_name = 'qapi_cabi'
+        if QAPISchemaModule.is_builtin_module(name):
+            fn_name += '_builtin'
+        elif name != self._main_module:
+            name = Path(name).stem
+            fn_name += '_' + c_name(name)
+        return fn_name
+
+    def visit_include(self, name: str, info: Optional[QAPISourceInfo]) -> None:
+        super().visit_include(name, info)
+        cabi_fn = self._get_qapi_cabi_fn(name)
+        if cabi_fn not in self._cabi_functions_called:
+            self._cabi_functions.append(cabi_fn)
+
+    def visit_module_end(self, name: str) -> None:
+        cabi_gen = "".join(f'    {fn}();\n' for fn in self._cabi_functions)
+        self._cabi_functions_called |= set(self._cabi_functions)
+        self._cabi_functions = []
+        cabi_gen += "".join([c.gen_c() for _, c in sorted(self._cabi.items())])
+        self._cabi = {}
+        fn_name = self._get_qapi_cabi_fn(name)
+        self._genh.add(mcgen('''
+
+#ifdef QAPI_CABI
+void %(fn_name)s(void);
+#endif
+''', fn_name=fn_name))
+        self._genc.add(mcgen('''
+
+#ifdef QAPI_CABI
+void %(fn_name)s(void) {
+%(cabi_gen)s
+}
+#endif
+''', fn_name=fn_name, cabi_gen=cabi_gen))
+
     def _gen_type_cleanup(self, name: str) -> None:
         self._genh.add(gen_type_cleanup_decl(name))
         self._genc.add(gen_type_cleanup(name))
@@ -309,6 +361,7 @@ def visit_enum_type(self,
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.preamble_add(gen_enum(name, members, prefix))
             self._genc.add(gen_enum_lookup(name, members, prefix))
+        self._cabi_add([CABIEnum(name, ifcond, members, prefix)])
 
     def visit_array_type(self,
                          name: str,
@@ -334,6 +387,7 @@ def visit_object_type(self,
         with ifcontext(ifcond, self._genh):
             self._genh.preamble_add(gen_fwd_object_or_array(name))
         self._genh.add(gen_object(name, ifcond, base, members, variants))
+        self._cabi_add(gen_object_cabi(name, ifcond, base, members, variants))
         with ifcontext(ifcond, self._genh, self._genc):
             if base and not base.is_implicit():
                 self._genh.add(gen_upcast(name, base))
@@ -353,6 +407,8 @@ def visit_alternate_type(self,
             self._genh.preamble_add(gen_fwd_object_or_array(name))
         self._genh.add(gen_object(name, ifcond, None,
                                   [variants.tag_member], variants))
+        self._cabi_add(gen_object_cabi(name, ifcond, None,
+                                       [variants.tag_member], variants))
         with ifcontext(ifcond, self._genh, self._genc):
             self._gen_type_cleanup(name)
 
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 08/32] tests: build qapi-cabi (C ABI dump)
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (6 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 07/32] scripts/qapi: generate CABI dump for C types marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 09/32] build-sys: add i686 cpu target marcandre.lureau
                   ` (24 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/qapi-cabi.c |  9 +++++++++
 tests/meson.build | 13 +++++++++++--
 2 files changed, 20 insertions(+), 2 deletions(-)
 create mode 100644 tests/qapi-cabi.c

diff --git a/tests/qapi-cabi.c b/tests/qapi-cabi.c
new file mode 100644
index 0000000000..2b94edb4ab
--- /dev/null
+++ b/tests/qapi-cabi.c
@@ -0,0 +1,9 @@
+#include "qemu/osdep.h"
+
+#include "test-qapi-types.h"
+
+int main(int argc, char *argv[])
+{
+    qapi_cabi();
+    return 0;
+}
diff --git a/tests/meson.build b/tests/meson.build
index 55a7b08275..be95223d62 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -54,8 +54,17 @@ foreach o: test_qapi_files.to_list() + test_qapi_files_extra.to_list()
   test_qapi_sources += o
 endforeach
 
-libtestqapi = static_library('testqapi', sources: [genh, test_qapi_sources])
-testqapi = declare_dependency(link_with: libtestqapi, sources: [genh, test_qapi_headers])
+libtestqapi = static_library('testqapi',
+                             sources: [genh, test_qapi_sources],
+                             c_args: ['-DQAPI_CABI'])
+testqapi = declare_dependency(link_with: libtestqapi,
+                              dependencies: [qemuutil],
+                              sources: [genh, test_qapi_headers])
+
+executable('qapi-cabi',
+           files('qapi-cabi.c'),
+           dependencies: testqapi,
+           c_args: ['-DQAPI_CABI'])
 
 test_deps = {
   'test-qht-par': qht_bench,
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 09/32] build-sys: add i686 cpu target
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (7 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 08/32] tests: build qapi-cabi (C ABI dump) marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 13:45   ` Peter Maydell
  2021-09-07 12:19 ` [RFC v3 10/32] build-sys: add --with-rust{-target} & basic build infrastructure marcandre.lureau
                   ` (23 subsequent siblings)
  32 siblings, 1 reply; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Rust does not have i386 targets, so distinguish when target cpu is i686.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 configure | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/configure b/configure
index 8adf2127c3..48ff2837d9 100755
--- a/configure
+++ b/configure
@@ -617,6 +617,8 @@ esac
 if test ! -z "$cpu" ; then
   # command line argument
   :
+elif check_define __i686__ ; then
+  cpu="i686"
 elif check_define __i386__ ; then
   cpu="i386"
 elif check_define __x86_64__ ; then
@@ -672,9 +674,12 @@ case "$cpu" in
   ppc64le)
     ARCH="ppc64"
   ;;
-  i386|i486|i586|i686|i86pc|BePC)
+  i386|i486|i586|i86pc|BePC)
     cpu="i386"
   ;;
+  i686)
+    ARCH="i386"
+  ;;
   x86_64|amd64)
     cpu="x86_64"
   ;;
@@ -1673,7 +1678,7 @@ case "$cpu" in
            CPU_CFLAGS="-m64"
            QEMU_LDFLAGS="-m64 $QEMU_LDFLAGS"
            ;;
-    i386)
+    i386|i686)
            CPU_CFLAGS="-m32"
            QEMU_LDFLAGS="-m32 $QEMU_LDFLAGS"
            ;;
@@ -4396,7 +4401,7 @@ QEMU_GA_MSI_MINGW_DLL_PATH="$($pkg_config --variable=prefix glib-2.0)/bin"
 
 # Mac OS X ships with a broken assembler
 roms=
-if { test "$cpu" = "i386" || test "$cpu" = "x86_64"; } && \
+if { test "$cpu" = "i386" || test "$cpu" = "i686" || test "$cpu" = "x86_64"; } && \
         test "$targetos" != "Darwin" && test "$targetos" != "SunOS" && \
         test "$targetos" != "Haiku" && test "$softmmu" = yes ; then
     # Different host OS linkers have different ideas about the name of the ELF
@@ -4962,7 +4967,7 @@ fi
 if test "$linux" = "yes" ; then
   mkdir -p linux-headers
   case "$cpu" in
-  i386|x86_64|x32)
+  i386|i686|x86_64|x32)
     linux_arch=x86
     ;;
   ppc|ppc64|ppc64le)
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 10/32] build-sys: add --with-rust{-target} & basic build infrastructure
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (8 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 09/32] build-sys: add i686 cpu target marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 14:00   ` Peter Maydell
  2021-09-07 12:19 ` [RFC v3 11/32] build-sys: add a cargo-wrapper script marcandre.lureau
                   ` (22 subsequent siblings)
  32 siblings, 1 reply; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Add the build-sys infrastructure to optionally build Rust code.
Introduce a top-level workspace, so various sub-projects (libraries,
executables etc) can be developed together, sharing the dependencies and
output directory.

If not Tier 1 (aarch64 and x86), all of the host architecture QEMU
supports should be Tier 2:
https://doc.rust-lang.org/nightly/rustc/platform-support.html

Rust is generally available on various distributions (thanks to Firefox,
I suppose). If not, it can usually be installed with rustup.

configure will enable Rust support automatically if cargo is present.
Rust support can be disabled --without-rust. When detecting windows
cross building, it will use the $cpu-pc-windows-gnu target by
default (more default mappings could be added over time). This can be
changed with --with-rust-target=RTT.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 configure         | 11 +++++++++++
 meson.build       | 27 +++++++++++++++++++++++++++
 .gitignore        |  1 +
 Cargo.toml        |  2 ++
 meson_options.txt |  5 +++++
 5 files changed, 46 insertions(+)
 create mode 100644 Cargo.toml

diff --git a/configure b/configure
index 48ff2837d9..470b90543f 100755
--- a/configure
+++ b/configure
@@ -445,6 +445,8 @@ fuse="auto"
 fuse_lseek="auto"
 multiprocess="auto"
 slirp_smbd="$default_feature"
+with_rust="auto"
+with_rust_target=""
 
 malloc_trim="auto"
 gio="$default_feature"
@@ -1586,6 +1588,12 @@ for opt do
   ;;
   --disable-slirp-smbd) slirp_smbd=no
   ;;
+  --with-rust) with_rust=enabled
+  ;;
+  --without-rust) with_rust=disabled
+  ;;
+  --with-rust-target=*) with_rust_target="$optarg"
+  ;;
   *)
       echo "ERROR: unknown option $opt"
       echo "Try '$0 --help' for more information"
@@ -1773,6 +1781,8 @@ Advanced options (experts only):
   --extra-ldflags=LDFLAGS  append extra linker flags LDFLAGS
   --cross-cc-ARCH=CC       use compiler when building ARCH guest test cases
   --cross-cc-flags-ARCH=   use compiler flags when building ARCH guest tests
+  --with-rust              enable Rust compilation
+  --with-rust-target=RTT   use the given Rust target triple
   --make=MAKE              use specified make [$make]
   --python=PYTHON          use specified python [$python]
   --sphinx-build=SPHINX    use specified sphinx-build [$sphinx_build]
@@ -5199,6 +5209,7 @@ if test "$skip_meson" = no; then
         -Db_coverage=$(if test "$gcov" = yes; then echo true; else echo false; fi) \
         -Db_lto=$lto -Dcfi=$cfi -Dcfi_debug=$cfi_debug \
         -Dmalloc=$malloc -Dmalloc_trim=$malloc_trim -Dsparse=$sparse \
+        -Dwith_rust=$with_rust -Dwith_rust_target=$with_rust_target \
         -Dkvm=$kvm -Dhax=$hax -Dwhpx=$whpx -Dhvf=$hvf -Dnvmm=$nvmm \
         -Dxen=$xen -Dxen_pci_passthrough=$xen_pci_passthrough -Dtcg=$tcg \
         -Dcocoa=$cocoa -Dgtk=$gtk -Dmpath=$mpath -Dsdl=$sdl -Dsdl_image=$sdl_image \
diff --git a/meson.build b/meson.build
index cbf3ce19ae..63b61a2bc8 100644
--- a/meson.build
+++ b/meson.build
@@ -108,6 +108,29 @@ endif
 
 bzip2 = find_program('bzip2', required: install_edk2_blobs)
 
+cargo = find_program('cargo', required: get_option('with_rust'))
+with_rust = cargo.found()
+
+if with_rust
+  rust_target_triple = get_option('with_rust_target')
+  if meson.is_cross_build()
+    # more default target mappings may be added over time
+    if rust_target_triple == '' and targetos == 'windows'
+      rust_target_triple = host_machine.cpu() + '-pc-windows-gnu'
+    endif
+    if rust_target_triple == ''
+      error('cross-compiling, but no Rust target-triple defined.')
+    endif
+  endif
+endif
+config_host_data.set('CONFIG_WITH_RUST', with_rust)
+
+if get_option('optimization') in ['0', '1', 'g']
+  rs_build_type = 'debug'
+else
+  rs_build_type = 'release'
+endif
+
 ##################
 # Compiler flags #
 ##################
@@ -2916,6 +2939,10 @@ if targetos == 'windows'
     summary_info += {'Windows SDK':   config_host['WIN_SDK']}
   endif
 endif
+summary_info += {'Rust support':      with_rust}
+if with_rust and rust_target_triple != ''
+  summary_info += {'Rust target':     rust_target_triple}
+endif
 summary_info += {'CFLAGS':            ' '.join(get_option('c_args')
                                                + ['-O' + get_option('optimization')]
                                                + (get_option('debug') ? ['-g'] : []))}
diff --git a/.gitignore b/.gitignore
index eb2553026c..78715bc7c4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
 /GNUmakefile
 /build/
+/target/
 *.pyc
 .sdk
 .stgit-*
diff --git a/Cargo.toml b/Cargo.toml
new file mode 100644
index 0000000000..c4b464ff15
--- /dev/null
+++ b/Cargo.toml
@@ -0,0 +1,2 @@
+[workspace]
+members = []
diff --git a/meson_options.txt b/meson_options.txt
index a9a9b8f4c6..fe9f90634b 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -155,3 +155,8 @@ option('slirp', type: 'combo', value: 'auto',
 option('fdt', type: 'combo', value: 'auto',
        choices: ['disabled', 'enabled', 'auto', 'system', 'internal'],
        description: 'Whether and how to find the libfdt library')
+
+option('with_rust', type: 'feature', value: 'auto',
+       description: 'Enable Rust support')
+option('with_rust_target', type : 'string', value: '',
+       description: 'Rust target triple')
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 11/32] build-sys: add a cargo-wrapper script
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (9 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 10/32] build-sys: add --with-rust{-target} & basic build infrastructure marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 12/32] rust: provide a common crate for QEMU marcandre.lureau
                   ` (21 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Introduce a script to help calling cargo from Rust.

This script will help to invoke cargo in different ways. For now, it
supports building static libraries and binaries. It sets up a common
environment, run the compiler with config-host options (config-target to
be added later when needed) and grep the compile output for the linker
flags. Finally it copies the built artefact to the expected meson output
directory, and create a file with the linker arguments.

cargo is the most convenient way to build Rust code, with various
crate dependencies, tests, and has features that meson lacks in general
for Rust.

Trying to convert projects to meson automatically is an option I
considered seriously (see for ex https://github.com/badboy/bygge for
ninja conversion), but the complexity of the task makes this endeavour
out of scope at this point. External dependencies would suffer various
issues that would be hard to implement and get accepted.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 meson.build               |   4 +
 scripts/cargo_wrapper.py  | 182 ++++++++++++++++++++++++++++++++++++++
 scripts/configh_to_cfg.py |  44 +++++++++
 3 files changed, 230 insertions(+)
 create mode 100644 scripts/cargo_wrapper.py
 create mode 100644 scripts/configh_to_cfg.py

diff --git a/meson.build b/meson.build
index 63b61a2bc8..a21c70d77f 100644
--- a/meson.build
+++ b/meson.build
@@ -110,6 +110,10 @@ bzip2 = find_program('bzip2', required: install_edk2_blobs)
 
 cargo = find_program('cargo', required: get_option('with_rust'))
 with_rust = cargo.found()
+cargo_wrapper = [
+  find_program('scripts/cargo_wrapper.py'),
+  '--configh', meson.current_build_dir() / 'config-host.h'
+]
 
 if with_rust
   rust_target_triple = get_option('with_rust_target')
diff --git a/scripts/cargo_wrapper.py b/scripts/cargo_wrapper.py
new file mode 100644
index 0000000000..75518e8c02
--- /dev/null
+++ b/scripts/cargo_wrapper.py
@@ -0,0 +1,182 @@
+#!/usr/bin/env python3
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Marc-André Lureau <marcandre.lureau@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import argparse
+import configparser
+import distutils.file_util
+import json
+import logging
+import os
+import os.path
+import re
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any, Dict, List, Tuple
+
+from configh_to_cfg import generate_cfg
+
+
+def get_rustc_target_spec(target_triple: str) -> Any:
+    cmd = [
+        "rustc",
+        "+nightly",
+        "-Z",
+        "unstable-options",
+        "--print",
+        "target-spec-json",
+    ]
+    if target_triple:
+        cmd += ["--target", target_triple]
+
+    out = subprocess.check_output(cmd)
+    return json.loads(out)
+
+
+def get_exe_suffix(target_triple: str) -> str:
+    try:
+        # this will fail if nightly is not installed
+        spec = get_rustc_target_spec(target_triple)
+        return spec.get("exe-suffix", "")
+    except: # pylint: disable=W0702
+        # let's implement a simple fallback
+        if 'windows' in target_triple:
+            return '.exe'
+        return ''
+
+
+def get_cargo_target_dir(args: argparse.Namespace) -> str:
+    # avoid conflict with qemu "target" directory
+    return os.path.join(args.build_dir, "rs-target")
+
+
+def get_manifest_path(args: argparse.Namespace) -> str:
+    return os.path.join(args.src_dir, "Cargo.toml")
+
+
+def get_cargo_rustc(
+    cargo_rustc_args: List[str], args: argparse.Namespace
+) -> Tuple[Dict[str, Any], List[str]]:
+    cfg = [c for h in args.configh for c in generate_cfg(h)]
+    target_dir = get_cargo_target_dir(args)
+    manifest_path = get_manifest_path(args)
+
+    env = {}
+    env["MESON_CURRENT_BUILD_DIR"] = args.build_dir
+    env["MESON_BUILD_ROOT"] = args.build_root
+    env["WINAPI_NO_BUNDLED_LIBRARIES"] = "1"
+    cargo_cmd = [
+        "cargo",
+        "rustc",
+        "--target-dir",
+        target_dir,
+        "--manifest-path",
+        manifest_path,
+    ]
+    cargo_cmd += cargo_rustc_args
+    if args.target_triple:
+        cargo_cmd += ["--target", args.target_triple]
+    if args.build_type == "release":
+        cargo_cmd += ["--release"]
+    cargo_cmd += ["--"] + cfg + args.EXTRA
+
+    return (env, cargo_cmd)
+
+
+def run_cargo(env: Dict[str, Any], cargo_cmd: List[str]) -> str:
+    envlog = " ".join(["{}={}".format(k, v) for k, v in env.items()])
+    cmdlog = " ".join(cargo_cmd)
+    logging.debug("Running %s %s", envlog, cmdlog)
+    try:
+        out = subprocess.check_output(
+            cargo_cmd,
+            env=dict(os.environ, **env),
+            stderr=subprocess.STDOUT,
+            universal_newlines=True,
+        )
+    except subprocess.CalledProcessError as err:
+        print("Environment: " + envlog)
+        print("Command: " + cmdlog)
+        print(err.output)
+        sys.exit(1)
+
+    return out
+
+
+def build_lib(args: argparse.Namespace) -> None:
+    logging.debug('build-lib')
+    target_dir = get_cargo_target_dir(args)
+    manifest_path = get_manifest_path(args)
+    # let's pretend it's an INI file to avoid extra toml dependency
+    config = configparser.ConfigParser()
+    config.read(manifest_path)
+    package_name = config["package"]["name"].strip('"').replace('-', '_')
+    liba = os.path.join(
+        target_dir, args.target_triple, args.build_type, "lib" + package_name + ".a"
+    )
+    libargs = os.path.join(args.build_dir, "lib" + package_name + ".args")
+
+    env, cargo_cmd = get_cargo_rustc(["--lib"], args)
+    cargo_cmd += ["--print", "native-static-libs"]
+    out = run_cargo(env, cargo_cmd)
+    native_static_libs = re.search(r"native-static-libs:(.*)", out)
+    link_args = native_static_libs.group(1)
+    with open(libargs, "w") as file:
+        print(link_args, file=file)
+    logging.debug("cp %s %s", liba, args.build_dir)
+    distutils.file_util.copy_file(liba, args.build_dir, update=True)
+
+
+def build_bin(args: argparse.Namespace) -> None:
+    logging.debug('build-bin')
+    env, cargo_cmd = get_cargo_rustc(["--bin", args.bin], args)
+    exe_suffix = get_exe_suffix(args.target_triple)
+    run_cargo(env, cargo_cmd)
+    target_dir = get_cargo_target_dir(args)
+    path = os.path.join(
+        target_dir, args.target_triple, args.build_type, args.bin + exe_suffix
+    )
+    dest = args.build_dir
+    if args.rename:
+        dest = Path(dest) / args.rename
+    logging.debug("cp %s %s", path, dest)
+    distutils.file_util.copy_file(path, dest, update=True)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-v", "--verbose", action='store_true')
+    parser.add_argument("--configh", action='append', default=[])
+    parser.add_argument("build_dir")
+    parser.add_argument("src_dir")
+    parser.add_argument("build_root")
+    parser.add_argument("build_type")
+    parser.add_argument("target_triple")
+    subparsers = parser.add_subparsers()
+
+    buildlib = subparsers.add_parser("build-lib")
+    buildlib.add_argument("EXTRA", nargs="*")
+    buildlib.set_defaults(func=build_lib)
+
+    buildbin = subparsers.add_parser("build-bin")
+    buildbin.add_argument("--rename")
+    buildbin.add_argument("bin")
+    buildbin.add_argument("EXTRA", nargs="*")
+    buildbin.set_defaults(func=build_bin)
+
+    args = parser.parse_args()
+    if args.verbose:
+        logging.basicConfig(level=logging.DEBUG)
+    logging.debug('args: %s', args)
+
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/scripts/configh_to_cfg.py b/scripts/configh_to_cfg.py
new file mode 100644
index 0000000000..b8f0c2da9a
--- /dev/null
+++ b/scripts/configh_to_cfg.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Marc-André Lureau <marcandre.lureau@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import argparse
+from typing import List
+
+
+def cfg_name(name: str) -> str:
+    if name.startswith("CONFIG_") or name.startswith("TARGET_") or name.startswith("HAVE_"):
+        return name
+    return ""
+
+
+def generate_cfg(header: str) -> List[str]:
+    with open(header, encoding="utf-8") as cfg:
+        config = [l.split()[1:] for l in cfg if l.startswith("#define")]
+
+    cfg_list = []
+    for cfg in config:
+        name = cfg_name(cfg[0])
+        if not name:
+            continue
+        if len(cfg) >= 2 and cfg[1] != "1":
+            continue
+        cfg_list.append("--cfg")
+        cfg_list.append(name)
+    return cfg_list
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("HEADER")
+    args = parser.parse_args()
+    print(" ".join(generate_cfg(args.HEADER)))
+
+
+if __name__ == "__main__":
+    main()
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 12/32] rust: provide a common crate for QEMU
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (10 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 11/32] build-sys: add a cargo-wrapper script marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-10  1:18   ` Alistair Francis
  2021-09-13 17:11   ` Paolo Bonzini
  2021-09-07 12:19 ` [RFC v3 13/32] rust: use vendored-sources marcandre.lureau
                   ` (20 subsequent siblings)
  32 siblings, 2 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

This crates provides common bindings and facilities for QEMU C API
shared by various projects.

Most importantly, it defines the conversion traits used to convert from
C to Rust types. Those traits are largely adapted from glib-rs, since
those have proved to be very flexible, and should guide us to bind
further QEMU types such as QOM. If glib-rs becomes a dependency, we
should consider adopting glib translate traits. For QAPI, we need a
smaller subset.

Cargo.lock is checked-in, as QEMU produces end-of-chain binaries, and it
is desirable to track the exact set of packages that are involved in
managed builds.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 Cargo.lock                   |  63 +++++
 Cargo.toml                   |   4 +-
 rust/common/Cargo.toml       |  11 +
 rust/common/src/error.rs     | 113 ++++++++
 rust/common/src/ffi.rs       |  93 +++++++
 rust/common/src/lib.rs       |  21 ++
 rust/common/src/qemu.rs      | 101 ++++++++
 rust/common/src/qmp.rs       |   0
 rust/common/src/translate.rs | 482 +++++++++++++++++++++++++++++++++++
 9 files changed, 887 insertions(+), 1 deletion(-)
 create mode 100644 Cargo.lock
 create mode 100644 rust/common/Cargo.toml
 create mode 100644 rust/common/src/error.rs
 create mode 100644 rust/common/src/ffi.rs
 create mode 100644 rust/common/src/lib.rs
 create mode 100644 rust/common/src/qemu.rs
 create mode 100644 rust/common/src/qmp.rs
 create mode 100644 rust/common/src/translate.rs

diff --git a/Cargo.lock b/Cargo.lock
new file mode 100644
index 0000000000..8dc2dd9da7
--- /dev/null
+++ b/Cargo.lock
@@ -0,0 +1,63 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 3
+
+[[package]]
+name = "autocfg"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a"
+
+[[package]]
+name = "bitflags"
+version = "1.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
+
+[[package]]
+name = "cc"
+version = "1.0.70"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d26a6ce4b6a484fa3edb70f7efa6fc430fd2b87285fe8b84304fd0936faa0dc0"
+
+[[package]]
+name = "cfg-if"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+
+[[package]]
+name = "common"
+version = "0.1.0"
+dependencies = [
+ "libc",
+ "nix",
+]
+
+[[package]]
+name = "libc"
+version = "0.2.101"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3cb00336871be5ed2c8ed44b60ae9959dc5b9f08539422ed43f09e34ecaeba21"
+
+[[package]]
+name = "memoffset"
+version = "0.6.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "59accc507f1338036a0477ef61afdae33cde60840f4dfe481319ce3ad116ddf9"
+dependencies = [
+ "autocfg",
+]
+
+[[package]]
+name = "nix"
+version = "0.20.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "df8e5e343312e7fbeb2a52139114e9e702991ef9c2aea6817ff2440b35647d56"
+dependencies = [
+ "bitflags",
+ "cc",
+ "cfg-if",
+ "libc",
+ "memoffset",
+]
diff --git a/Cargo.toml b/Cargo.toml
index c4b464ff15..14131eed3c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,2 +1,4 @@
 [workspace]
-members = []
+members = [
+  "rust/common",
+]
diff --git a/rust/common/Cargo.toml b/rust/common/Cargo.toml
new file mode 100644
index 0000000000..6c240447f3
--- /dev/null
+++ b/rust/common/Cargo.toml
@@ -0,0 +1,11 @@
+[package]
+name = "common"
+version = "0.1.0"
+edition = "2018"
+publish = false
+
+[dependencies]
+libc = "0.2.92"
+
+[target."cfg(unix)".dependencies]
+nix = "0.20.0"
diff --git a/rust/common/src/error.rs b/rust/common/src/error.rs
new file mode 100644
index 0000000000..f166ac42ea
--- /dev/null
+++ b/rust/common/src/error.rs
@@ -0,0 +1,113 @@
+use std::{self, ffi::CString, fmt, io, ptr};
+
+use crate::translate::*;
+use crate::{ffi, qemu};
+
+/// Common error type for QEMU and related projects.
+#[derive(Debug)]
+pub enum Error {
+    /// A generic error with file and line location.
+    FailedAt(String, &'static str, u32),
+    /// An IO error.
+    Io(io::Error),
+    #[cfg(unix)]
+    /// A nix error.
+    Nix(nix::Error),
+}
+
+/// Alias for a `Result` with the error type for QEMU.
+pub type Result<T> = std::result::Result<T, Error>;
+
+impl Error {
+    fn message(&self) -> String {
+        use Error::*;
+        match self {
+            FailedAt(msg, _, _) => msg.into(),
+            Io(io) => format!("IO error: {}", io),
+            #[cfg(unix)]
+            Nix(nix) => format!("Nix error: {}", nix),
+        }
+    }
+
+    fn location(&self) -> Option<(&'static str, u32)> {
+        use Error::*;
+        match self {
+            FailedAt(_, file, line) => Some((file, *line)),
+            Io(_) => None,
+            #[cfg(unix)]
+            Nix(_) => None,
+        }
+    }
+}
+
+impl fmt::Display for Error {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        use Error::*;
+        match self {
+            FailedAt(msg, file, line) => write!(f, "{} ({}:{})", msg, file, line),
+            _ => write!(f, "{}", self.message()),
+        }
+    }
+}
+
+impl From<io::Error> for Error {
+    fn from(val: io::Error) -> Self {
+        Error::Io(val)
+    }
+}
+
+#[cfg(unix)]
+impl From<nix::Error> for Error {
+    fn from(val: nix::Error) -> Self {
+        Error::Nix(val)
+    }
+}
+
+impl QemuPtrDefault for Error {
+    type QemuType = *mut ffi::Error;
+}
+
+impl<'a> ToQemuPtr<'a, *mut ffi::Error> for Error {
+    type Storage = qemu::CError;
+
+    fn to_qemu_none(&'a self) -> Stash<'a, *mut ffi::Error, Self> {
+        let err = self.to_qemu_full();
+
+        Stash(err, unsafe { from_qemu_full(err) })
+    }
+
+    fn to_qemu_full(&self) -> *mut ffi::Error {
+        let cmsg =
+            CString::new(self.message()).expect("ToQemuPtr<Error>: unexpected '\0' character");
+        let mut csrc = CString::new("").unwrap();
+        let (src, line) = self.location().map_or((ptr::null(), 0_i32), |loc| {
+            csrc = CString::new(loc.0).expect("ToQemuPtr<Error>:: unexpected '\0' character");
+            (csrc.as_ptr() as *const libc::c_char, loc.1 as i32)
+        });
+        let func = ptr::null();
+
+        let mut err: *mut ffi::Error = ptr::null_mut();
+        unsafe {
+            ffi::error_setg_internal(
+                &mut err as *mut *mut _,
+                src,
+                line,
+                func,
+                cmsg.as_ptr() as *const libc::c_char,
+            );
+            err
+        }
+    }
+}
+
+/// Convenience macro to build a [`Error::FailedAt`] error.
+///
+/// Returns a `Result::Err` with the file:line location.
+/// (the error can then be converted to a QEMU `ffi::Error`)
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! err {
+    ($msg:expr) => {
+        Err(Error::FailedAt($msg.into(), file!(), line!()))
+    };
+}
diff --git a/rust/common/src/ffi.rs b/rust/common/src/ffi.rs
new file mode 100644
index 0000000000..82818d503a
--- /dev/null
+++ b/rust/common/src/ffi.rs
@@ -0,0 +1,93 @@
+//! Bindings to the raw low-level C API commonly provided by QEMU projects.
+//!
+//! Manual bindings to C API availabe when linking QEMU projects.
+//! It includes minimal glib allocation functions too, since it's the default
+//! allocator used by QEMU, and we don't depend on glib-rs crate yet).
+//!
+//! Higher-level Rust-friendly bindings are provided by different modules.
+
+use libc::{c_char, c_void, size_t};
+
+extern "C" {
+    pub fn g_malloc0(n_bytes: size_t) -> *mut c_void;
+    pub fn g_free(ptr: *mut c_void);
+    pub fn g_strndup(str: *const c_char, n: size_t) -> *mut c_char;
+}
+
+#[repr(C)]
+pub struct QObject(c_void);
+
+impl ::std::fmt::Debug for QObject {
+    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
+        f.debug_struct(&format!("QObject @ {:?}", self as *const _))
+            .finish()
+    }
+}
+
+#[repr(C)]
+pub struct QNull(c_void);
+
+impl ::std::fmt::Debug for QNull {
+    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
+        f.debug_struct(&format!("QNull @ {:?}", self as *const _))
+            .finish()
+    }
+}
+
+#[repr(C)]
+pub struct Error(c_void);
+
+impl ::std::fmt::Debug for Error {
+    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
+        f.debug_struct(&format!("Error @ {:?}", self as *const _))
+            .finish()
+    }
+}
+
+extern "C" {
+    pub fn error_setg_internal(
+        errp: *mut *mut Error,
+        src: *const c_char,
+        line: i32,
+        func: *const c_char,
+        fmt: *const c_char,
+        ...
+    );
+    pub fn error_get_pretty(err: *const Error) -> *const c_char;
+    pub fn error_free(err: *mut Error);
+}
+
+/// Wrap a QMP hanlder.
+#[macro_export]
+macro_rules! qmp {
+    // the basic return value variant
+    ($e:expr, $errp:ident, $errval:expr) => {{
+        assert!(!$errp.is_null());
+        unsafe {
+            *$errp = std::ptr::null_mut();
+        }
+
+        match $e {
+            Ok(val) => val,
+            Err(err) => unsafe {
+                *$errp = err.to_qemu_full();
+                $errval
+            },
+        }
+    }};
+    // the ptr return value variant
+    ($e:expr, $errp:ident) => {{
+        assert!(!$errp.is_null());
+        unsafe {
+            *$errp = std::ptr::null_mut();
+        }
+
+        match $e {
+            Ok(val) => val.to_qemu_full().into(),
+            Err(err) => unsafe {
+                *$errp = err.to_qemu_full();
+                std::ptr::null_mut()
+            },
+        }
+    }};
+}
diff --git a/rust/common/src/lib.rs b/rust/common/src/lib.rs
new file mode 100644
index 0000000000..4de826bc2e
--- /dev/null
+++ b/rust/common/src/lib.rs
@@ -0,0 +1,21 @@
+//! Common code for QEMU
+//!
+//! This crates provides common bindings and facilities for QEMU C API shared by
+//! various projects. Most importantly, it defines the conversion traits used to
+//! convert from C to Rust types. Those traits are largely adapted from glib-rs,
+//! since those have prooven to be very flexible, and should guide us to bind
+//! further QEMU types such as QOM. If glib-rs becomes a dependency, we should
+//! consider adopting glib translate traits. For QAPI, we need a smaller subset.
+
+pub use libc;
+
+mod error;
+pub use error::*;
+
+mod qemu;
+pub use qemu::*;
+
+mod translate;
+pub use translate::*;
+
+pub mod ffi;
diff --git a/rust/common/src/qemu.rs b/rust/common/src/qemu.rs
new file mode 100644
index 0000000000..dd01c6d92d
--- /dev/null
+++ b/rust/common/src/qemu.rs
@@ -0,0 +1,101 @@
+use std::{ffi::CStr, ptr, str};
+
+use crate::{ffi, translate};
+use translate::{FromQemuPtrFull, FromQemuPtrNone, QemuPtrDefault, Stash, ToQemuPtr};
+
+/// A type representing an owned C QEMU Error.
+pub struct CError(ptr::NonNull<ffi::Error>);
+
+impl translate::FromQemuPtrFull<*mut ffi::Error> for CError {
+    unsafe fn from_qemu_full(ptr: *mut ffi::Error) -> Self {
+        assert!(!ptr.is_null());
+        Self(ptr::NonNull::new_unchecked(ptr))
+    }
+}
+
+impl CError {
+    pub fn pretty(&self) -> &str {
+        unsafe {
+            let pretty = ffi::error_get_pretty(self.0.as_ptr());
+            let bytes = CStr::from_ptr(pretty).to_bytes();
+            str::from_utf8(bytes)
+                .unwrap_or_else(|err| str::from_utf8(&bytes[..err.valid_up_to()]).unwrap())
+        }
+    }
+}
+
+impl Drop for CError {
+    fn drop(&mut self) {
+        unsafe { ffi::error_free(self.0.as_ptr()) }
+    }
+}
+
+/// QObject (JSON object)
+#[derive(Clone, Debug)]
+pub struct QObject;
+
+impl QemuPtrDefault for QObject {
+    type QemuType = *mut ffi::QObject;
+}
+
+impl FromQemuPtrFull<*mut ffi::QObject> for QObject {
+    #[inline]
+    unsafe fn from_qemu_full(_ffi: *mut ffi::QObject) -> Self {
+        unimplemented!()
+    }
+}
+
+impl FromQemuPtrNone<*const ffi::QObject> for QObject {
+    #[inline]
+    unsafe fn from_qemu_none(_ffi: *const ffi::QObject) -> Self {
+        unimplemented!()
+    }
+}
+
+impl<'a> ToQemuPtr<'a, *mut ffi::QObject> for QObject {
+    type Storage = ();
+
+    #[inline]
+    fn to_qemu_none(&self) -> Stash<'a, *mut ffi::QObject, QObject> {
+        unimplemented!()
+    }
+    #[inline]
+    fn to_qemu_full(&self) -> *mut ffi::QObject {
+        unimplemented!()
+    }
+}
+
+/// QNull (JSON null)
+#[derive(Clone, Debug)]
+pub struct QNull;
+
+impl QemuPtrDefault for QNull {
+    type QemuType = *mut ffi::QNull;
+}
+
+impl FromQemuPtrFull<*mut ffi::QObject> for QNull {
+    #[inline]
+    unsafe fn from_qemu_full(_ffi: *mut ffi::QObject) -> Self {
+        unimplemented!()
+    }
+}
+
+impl FromQemuPtrNone<*const ffi::QObject> for QNull {
+    #[inline]
+    unsafe fn from_qemu_none(_ffi: *const ffi::QObject) -> Self {
+        unimplemented!()
+    }
+}
+
+impl<'a> ToQemuPtr<'a, *mut ffi::QNull> for QNull {
+    type Storage = ();
+
+    #[inline]
+    fn to_qemu_none(&self) -> Stash<'a, *mut ffi::QNull, QNull> {
+        unimplemented!()
+    }
+    #[inline]
+    fn to_qemu_full(&self) -> *mut ffi::QNull {
+        unimplemented!()
+    }
+}
diff --git a/rust/common/src/qmp.rs b/rust/common/src/qmp.rs
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/rust/common/src/translate.rs b/rust/common/src/translate.rs
new file mode 100644
index 0000000000..315e14fa25
--- /dev/null
+++ b/rust/common/src/translate.rs
@@ -0,0 +1,482 @@
+// largely adapted from glib-rs
+// we don't depend on glib-rs as this brings a lot more code that we may not need
+// and also because there are issues with the conversion traits for our ffi::*mut.
+use libc::{c_char, size_t};
+use std::ffi::{CStr, CString};
+use std::ptr;
+
+use crate::ffi;
+
+/// A pointer.
+pub trait Ptr: Copy + 'static {
+    fn is_null(&self) -> bool;
+    fn from<X>(ptr: *mut X) -> Self;
+    fn to<X>(self) -> *mut X;
+}
+
+impl<T: 'static> Ptr for *const T {
+    #[inline]
+    fn is_null(&self) -> bool {
+        (*self).is_null()
+    }
+
+    #[inline]
+    fn from<X>(ptr: *mut X) -> *const T {
+        ptr as *const T
+    }
+
+    #[inline]
+    fn to<X>(self) -> *mut X {
+        self as *mut X
+    }
+}
+
+impl<T: 'static> Ptr for *mut T {
+    #[inline]
+    fn is_null(&self) -> bool {
+        (*self).is_null()
+    }
+
+    #[inline]
+    fn from<X>(ptr: *mut X) -> *mut T {
+        ptr as *mut T
+    }
+
+    #[inline]
+    fn to<X>(self) -> *mut X {
+        self as *mut X
+    }
+}
+
+/// Macro to declare a `NewPtr` struct.
+///
+/// A macro to declare a newtype for pointers, to workaround that *T are not
+/// defined in our binding crates, and allow foreign traits implementations.
+/// (this is used by qapi-gen bindings)
+#[allow(unused_macros)]
+#[macro_export]
+#[doc(hidden)]
+macro_rules! new_ptr {
+    () => {
+        #[derive(Copy, Clone)]
+        pub struct NewPtr<P: Ptr>(pub P);
+
+        impl<P: Ptr> Ptr for NewPtr<P> {
+            #[inline]
+            fn is_null(&self) -> bool {
+                self.0.is_null()
+            }
+
+            #[inline]
+            fn from<X>(ptr: *mut X) -> Self {
+                NewPtr(P::from(ptr))
+            }
+
+            #[inline]
+            fn to<X>(self) -> *mut X {
+                self.0.to()
+            }
+        }
+    };
+}
+
+/// Provides the default pointer type to be used in some container conversions.
+///
+/// It's `*mut c_char` for `String`, `*mut ffi::GuestInfo` for `GuestInfo`...
+pub trait QemuPtrDefault {
+    type QemuType: Ptr;
+}
+
+impl QemuPtrDefault for String {
+    type QemuType = *mut c_char;
+}
+
+/// A Stash contains the temporary storage and a pointer into it.
+///
+/// The pointer is valid for the lifetime of the `Stash`. As the lifetime of the
+/// `Stash` returned from `to_qemu_none` is at least the enclosing statement,
+/// you can avoid explicitly binding the stash in most cases and just take the
+/// pointer out of it:
+///
+/// ```ignore
+///     pub fn set_device_name(&self, name: &str) {
+///         unsafe {
+///             ffi::qemu_device_set_name(self.pointer, name.to_qemu_none().0)
+///         }
+///     }
+/// ```
+pub struct Stash<'a, P: Copy, T: ?Sized + ToQemuPtr<'a, P>>(
+    pub P,
+    pub <T as ToQemuPtr<'a, P>>::Storage,
+);
+
+/// Translate to a pointer.
+pub trait ToQemuPtr<'a, P: Copy> {
+    type Storage;
+
+    /// The pointer in the `Stash` is only valid for the lifetime of the `Stash`.
+    fn to_qemu_none(&'a self) -> Stash<'a, P, Self>;
+
+    /// Transfer the ownership to the ffi.
+    fn to_qemu_full(&self) -> P {
+        unimplemented!();
+    }
+}
+
+impl<'a, P: Ptr, T: ToQemuPtr<'a, P>> ToQemuPtr<'a, P> for Option<T> {
+    type Storage = Option<<T as ToQemuPtr<'a, P>>::Storage>;
+
+    #[inline]
+    fn to_qemu_none(&'a self) -> Stash<'a, P, Option<T>> {
+        self.as_ref()
+            .map_or(Stash(Ptr::from::<()>(ptr::null_mut()), None), |s| {
+                let s = s.to_qemu_none();
+                Stash(s.0, Some(s.1))
+            })
+    }
+
+    #[inline]
+    fn to_qemu_full(&self) -> P {
+        self.as_ref()
+            .map_or(Ptr::from::<()>(ptr::null_mut()), ToQemuPtr::to_qemu_full)
+    }
+}
+
+impl<'a, P: Ptr, T: ToQemuPtr<'a, P>> ToQemuPtr<'a, P> for Box<T> {
+    type Storage = <T as ToQemuPtr<'a, P>>::Storage;
+
+    #[inline]
+    fn to_qemu_none(&'a self) -> Stash<'a, P, Box<T>> {
+        let s = self.as_ref().to_qemu_none();
+        Stash(s.0, s.1)
+    }
+
+    #[inline]
+    fn to_qemu_full(&self) -> P {
+        ToQemuPtr::to_qemu_full(self.as_ref())
+    }
+}
+
+impl<'a> ToQemuPtr<'a, *mut c_char> for String {
+    type Storage = CString;
+
+    #[inline]
+    fn to_qemu_none(&self) -> Stash<'a, *mut c_char, String> {
+        let tmp = CString::new(&self[..])
+            .expect("String::ToQemuPtr<*mut c_char>: unexpected '\0' character");
+        Stash(tmp.as_ptr() as *mut c_char, tmp)
+    }
+
+    #[inline]
+    fn to_qemu_full(&self) -> *mut c_char {
+        unsafe { ffi::g_strndup(self.as_ptr() as *const c_char, self.len() as size_t) }
+    }
+}
+
+/// Translate from a pointer type, without taking ownership.
+pub trait FromQemuPtrNone<P: Ptr>: Sized {
+    /// # Safety
+    ///
+    /// `ptr` must be a valid pointer. It is not referenced after the call.
+    unsafe fn from_qemu_none(ptr: P) -> Self;
+}
+
+/// Translate from a pointer type, taking ownership.
+pub trait FromQemuPtrFull<P: Ptr>: Sized {
+    /// # Safety
+    ///
+    /// `ptr` must be a valid pointer. Ownership is transferred.
+    unsafe fn from_qemu_full(ptr: P) -> Self;
+}
+
+/// See [`FromQemuPtrNone`](trait.FromQemuPtrNone.html).
+#[inline]
+#[allow(clippy::missing_safety_doc)]
+pub unsafe fn from_qemu_none<P: Ptr, T: FromQemuPtrNone<P>>(ptr: P) -> T {
+    FromQemuPtrNone::from_qemu_none(ptr)
+}
+
+/// See [`FromQemuPtrFull`](trait.FromQemuPtrFull.html).
+#[inline]
+#[allow(clippy::missing_safety_doc)]
+pub unsafe fn from_qemu_full<P: Ptr, T: FromQemuPtrFull<P>>(ptr: P) -> T {
+    FromQemuPtrFull::from_qemu_full(ptr)
+}
+
+impl<P: Ptr, T: FromQemuPtrNone<P>> FromQemuPtrNone<P> for Option<T> {
+    #[inline]
+    unsafe fn from_qemu_none(ptr: P) -> Option<T> {
+        if ptr.is_null() {
+            None
+        } else {
+            Some(from_qemu_none(ptr))
+        }
+    }
+}
+
+impl<P: Ptr, T: FromQemuPtrFull<P>> FromQemuPtrFull<P> for Option<T> {
+    #[inline]
+    unsafe fn from_qemu_full(ptr: P) -> Option<T> {
+        if ptr.is_null() {
+            None
+        } else {
+            Some(from_qemu_full(ptr))
+        }
+    }
+}
+
+impl FromQemuPtrNone<*const c_char> for String {
+    #[inline]
+    unsafe fn from_qemu_none(ptr: *const c_char) -> Self {
+        assert!(!ptr.is_null());
+        String::from_utf8_lossy(CStr::from_ptr(ptr).to_bytes()).into_owned()
+    }
+}
+
+impl FromQemuPtrFull<*mut c_char> for String {
+    #[inline]
+    unsafe fn from_qemu_full(ptr: *mut c_char) -> Self {
+        let res = from_qemu_none(ptr as *const _);
+        ffi::g_free(ptr as *mut _);
+        res
+    }
+}
+
+#[doc(hidden)]
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! vec_ffi_wrapper {
+    ($ffi:ident) => {
+        #[allow(non_camel_case_types)]
+        pub struct $ffi(*mut qapi_ffi::$ffi);
+
+        impl Drop for $ffi {
+            fn drop(&mut self) {
+                let mut list = self.0;
+                unsafe {
+                    while !list.is_null() {
+                        let next = (*list).next;
+                        Box::from_raw(list);
+                        list = next;
+                    }
+                }
+            }
+        }
+
+        impl From<NewPtr<*mut qapi_ffi::$ffi>> for *mut qapi_ffi::$ffi {
+            fn from(p: NewPtr<*mut qapi_ffi::$ffi>) -> Self {
+                p.0
+            }
+        }
+    };
+}
+
+#[doc(hidden)]
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! impl_vec_scalars_to_qemu {
+    ($rs:ty, $ffi:ident) => {
+        impl<'a> ToQemuPtr<'a, NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
+            type Storage = $ffi;
+
+            #[inline]
+            fn to_qemu_none(&self) -> Stash<NewPtr<*mut qapi_ffi::$ffi>, Self> {
+                let mut list: *mut qapi_ffi::$ffi = std::ptr::null_mut();
+                for value in self.iter().rev() {
+                    let b = Box::new(qapi_ffi::$ffi {
+                        next: list,
+                        value: *value,
+                    });
+                    list = Box::into_raw(b);
+                }
+                Stash(NewPtr(list), $ffi(list))
+            }
+
+            #[inline]
+            fn to_qemu_full(&self) -> NewPtr<*mut qapi_ffi::$ffi> {
+                let mut list: *mut qapi_ffi::$ffi = std::ptr::null_mut();
+                unsafe {
+                    for value in self.iter().rev() {
+                        let l = ffi::g_malloc0(std::mem::size_of::<qapi_ffi::$ffi>())
+                            as *mut qapi_ffi::$ffi;
+                        (*l).next = list;
+                        (*l).value = *value;
+                        list = l;
+                    }
+                }
+                NewPtr(list)
+            }
+        }
+    };
+}
+
+#[doc(hidden)]
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! impl_vec_scalars_from_qemu {
+    ($rs:ty, $ffi:ident, $free_ffi:ident) => {
+        impl FromQemuPtrFull<NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
+            #[inline]
+            unsafe fn from_qemu_full(ffi: NewPtr<*mut qapi_ffi::$ffi>) -> Self {
+                let ret = from_qemu_none(NewPtr(ffi.0 as *const _));
+                qapi_ffi::$free_ffi(ffi.0);
+                ret
+            }
+        }
+
+        impl FromQemuPtrNone<NewPtr<*const qapi_ffi::$ffi>> for Vec<$rs> {
+            #[inline]
+            unsafe fn from_qemu_none(ffi: NewPtr<*const qapi_ffi::$ffi>) -> Self {
+                let mut ret = vec![];
+                let mut it = ffi.0;
+                while !it.is_null() {
+                    let e = &*it;
+                    ret.push(e.value);
+                    it = e.next;
+                }
+                ret
+            }
+        }
+    };
+}
+
+#[doc(hidden)]
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! impl_vec_to_qemu {
+    ($rs:ty, $ffi:ident) => {
+        // impl doesn't use only types from inside the current crate
+        // impl QemuPtrDefault for Vec<$rs> {
+        //     type QemuType = NewPtr<*mut qapi_ffi::$ffi>;
+        // }
+
+        impl<'a> ToQemuPtr<'a, NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
+            type Storage = ($ffi, Vec<Stash<'a, <$rs as QemuPtrDefault>::QemuType, $rs>>);
+
+            #[inline]
+            fn to_qemu_none(&self) -> Stash<NewPtr<*mut qapi_ffi::$ffi>, Self> {
+                let stash_vec: Vec<_> = self.iter().rev().map(ToQemuPtr::to_qemu_none).collect();
+                let mut list: *mut qapi_ffi::$ffi = std::ptr::null_mut();
+                for stash in &stash_vec {
+                    let b = Box::new(qapi_ffi::$ffi {
+                        next: list,
+                        value: Ptr::to(stash.0),
+                    });
+                    list = Box::into_raw(b);
+                }
+                Stash(NewPtr(list), ($ffi(list), stash_vec))
+            }
+
+            #[inline]
+            fn to_qemu_full(&self) -> NewPtr<*mut qapi_ffi::$ffi> {
+                let v: Vec<_> = self.iter().rev().map(ToQemuPtr::to_qemu_full).collect();
+                let mut list: *mut qapi_ffi::$ffi = std::ptr::null_mut();
+                unsafe {
+                    for val in v {
+                        let l = ffi::g_malloc0(std::mem::size_of::<qapi_ffi::$ffi>())
+                            as *mut qapi_ffi::$ffi;
+                        (*l).next = list;
+                        (*l).value = val;
+                        list = l;
+                    }
+                }
+                NewPtr(list)
+            }
+        }
+    };
+}
+
+#[doc(hidden)]
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! impl_vec_from_qemu {
+    ($rs:ty, $ffi:ident, $free_ffi:ident) => {
+        impl FromQemuPtrFull<NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
+            #[inline]
+            unsafe fn from_qemu_full(ffi: NewPtr<*mut qapi_ffi::$ffi>) -> Self {
+                let ret = from_qemu_none(NewPtr(ffi.0 as *const _));
+                qapi_ffi::$free_ffi(ffi.0);
+                ret
+            }
+        }
+
+        impl FromQemuPtrNone<NewPtr<*const qapi_ffi::$ffi>> for Vec<$rs> {
+            #[inline]
+            unsafe fn from_qemu_none(ffi: NewPtr<*const qapi_ffi::$ffi>) -> Self {
+                let mut ret = vec![];
+                let mut it = ffi.0;
+                while !it.is_null() {
+                    let e = &*it;
+                    ret.push(from_qemu_none(e.value as *const _));
+                    it = e.next;
+                }
+                ret
+            }
+        }
+    };
+}
+
+/// A macro to help the implementation of `Vec<T>` translations.
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! vec_type {
+    (Vec<$rs:ty>, $ffi:ident, $free_ffi:ident, 0) => {
+        vec_ffi_wrapper!($ffi);
+        impl_vec_from_qemu!($rs, $ffi, $free_ffi);
+        impl_vec_to_qemu!($rs, $ffi);
+    };
+    (Vec<$rs:ty>, $ffi:ident, $free_ffi:ident, 1) => {
+        vec_ffi_wrapper!($ffi);
+        impl_vec_scalars_from_qemu!($rs, $ffi, $free_ffi);
+        impl_vec_scalars_to_qemu!($rs, $ffi);
+    };
+}
+
+/// A macro to implement [`ToQemuPtr`] as boxed scalars
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! impl_to_qemu_scalar_boxed {
+    ($ty:ty) => {
+        impl<'a> ToQemuPtr<'a, *mut $ty> for $ty {
+            type Storage = Box<$ty>;
+
+            fn to_qemu_none(&'a self) -> Stash<'a, *mut $ty, Self> {
+                let mut box_ = Box::new(*self);
+                Stash(&mut *box_, box_)
+            }
+
+            fn to_qemu_full(&self) -> *mut $ty {
+                unsafe {
+                    let ptr = ffi::g_malloc0(std::mem::size_of::<$ty>()) as *mut _;
+                    *ptr = *self;
+                    ptr
+                }
+            }
+        }
+    };
+}
+
+/// A macro to implement [`FromQemuPtrNone`] for scalar pointers.
+#[allow(unused_macros)]
+#[macro_export]
+macro_rules! impl_from_qemu_none_scalar {
+    ($ty:ty) => {
+        impl FromQemuPtrNone<*const $ty> for $ty {
+            unsafe fn from_qemu_none(ptr: *const $ty) -> Self {
+                *ptr
+            }
+        }
+    };
+}
+
+macro_rules! impl_scalar_boxed {
+    ($($t:ident)*) => (
+        $(
+            impl_to_qemu_scalar_boxed!($t);
+            impl_from_qemu_none_scalar!($t);
+        )*
+    )
+}
+
+// the only built-in used so far, feel free to add more as needed
+impl_scalar_boxed!(bool i64 f64);
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 13/32] rust: use vendored-sources
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (11 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 12/32] rust: provide a common crate for QEMU marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 15:38   ` Ian Jackson
  2021-09-09 16:02   ` Peter Maydell
  2021-09-07 12:19 ` [RFC v3 14/32] scripts/qapi: add QAPISchemaIfCond.rsgen() marcandre.lureau
                   ` (19 subsequent siblings)
  32 siblings, 2 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Most likely, QEMU will want tighter control over the sources, rather
than relying on crates.io downloading, use a git submodule with all the
dependencies. However, cargo --offline was added in 1.36.

"cargo vendor" helps gathering and updating the dependencies.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 configure                 | 8 ++++++++
 meson.build               | 7 ++++++-
 .cargo/config.toml.in     | 5 +++++
 .cargo/meson.build        | 5 +++++
 .gitmodules               | 4 ++++
 rust/vendored             | 1 +
 scripts/archive-source.sh | 2 +-
 scripts/cargo_wrapper.py  | 1 +
 8 files changed, 31 insertions(+), 2 deletions(-)
 create mode 100644 .cargo/config.toml.in
 create mode 100644 .cargo/meson.build
 create mode 160000 rust/vendored

diff --git a/configure b/configure
index 470b90543f..82a94ab93b 100755
--- a/configure
+++ b/configure
@@ -2068,6 +2068,14 @@ if test -z "$ninja"; then
     fi
 fi
 
+case "$with_rust" in
+  auto|enabled)
+    if test -e "${source_path}/.git"; then
+      git_submodules="${git_submodules} rust/vendored"
+    fi
+    ;;
+esac
+
 # Check that the C compiler works. Doing this here before testing
 # the host CPU ensures that we had a valid CC to autodetect the
 # $cpu var (and we should bail right here if that's not the case).
diff --git a/meson.build b/meson.build
index a21c70d77f..29d218d35a 100644
--- a/meson.build
+++ b/meson.build
@@ -108,7 +108,11 @@ endif
 
 bzip2 = find_program('bzip2', required: install_edk2_blobs)
 
-cargo = find_program('cargo', required: get_option('with_rust'))
+cargo = find_program('cargo',
+                     required: get_option('with_rust'),
+                     # require --offline support (1.36), but fixed (1.42)
+                     version: '>= 1.42')
+
 with_rust = cargo.found()
 cargo_wrapper = [
   find_program('scripts/cargo_wrapper.py'),
@@ -126,6 +130,7 @@ if with_rust
       error('cross-compiling, but no Rust target-triple defined.')
     endif
   endif
+  subdir('.cargo')
 endif
 config_host_data.set('CONFIG_WITH_RUST', with_rust)
 
diff --git a/.cargo/config.toml.in b/.cargo/config.toml.in
new file mode 100644
index 0000000000..d1531aa52a
--- /dev/null
+++ b/.cargo/config.toml.in
@@ -0,0 +1,5 @@
+[source.crates-io]
+replace-with = "vendored-sources"
+
+[source.vendored-sources]
+directory = "@vendored_directory@"
diff --git a/.cargo/meson.build b/.cargo/meson.build
new file mode 100644
index 0000000000..4e7c296ab0
--- /dev/null
+++ b/.cargo/meson.build
@@ -0,0 +1,5 @@
+configure_file(output: 'config.toml',
+               input: 'config.toml.in',
+               configuration: {
+                   'vendored_directory': meson.source_root() / 'rust/vendored'
+               })
diff --git a/.gitmodules b/.gitmodules
index 08b1b48a09..f767a4f386 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -64,3 +64,7 @@
 [submodule "roms/vbootrom"]
 	path = roms/vbootrom
 	url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "rust/vendored"]
+	path = rust/vendored
+	#url = https://gitlab.com/qemu-project/qemu-rust-vendored.git
+	url = https://github.com/elmarco/qemu-rust-vendored.git
diff --git a/rust/vendored b/rust/vendored
new file mode 160000
index 0000000000..7077bbbd11
--- /dev/null
+++ b/rust/vendored
@@ -0,0 +1 @@
+Subproject commit 7077bbbd11a67d60062a9483f996113a349a4ca1
diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index c6169db69f..03afcee8b9 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -26,7 +26,7 @@ sub_file="${sub_tdir}/submodule.tar"
 # independent of what the developer currently has initialized
 # in their checkout, because the build environment is completely
 # different to the host OS.
-submodules="dtc slirp meson ui/keycodemapdb"
+submodules="dtc slirp meson ui/keycodemapdb rust/vendored"
 submodules="$submodules tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
 sub_deinit=""
 
diff --git a/scripts/cargo_wrapper.py b/scripts/cargo_wrapper.py
index 75518e8c02..c07f51494b 100644
--- a/scripts/cargo_wrapper.py
+++ b/scripts/cargo_wrapper.py
@@ -78,6 +78,7 @@ def get_cargo_rustc(
         target_dir,
         "--manifest-path",
         manifest_path,
+        "--offline",
     ]
     cargo_cmd += cargo_rustc_args
     if args.target_triple:
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 14/32] scripts/qapi: add QAPISchemaIfCond.rsgen()
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (12 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 13/32] rust: use vendored-sources marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 12:33   ` Markus Armbruster
  2021-09-07 12:19 ` [RFC v3 15/32] scripts/qapi: strip trailing whitespaces marcandre.lureau
                   ` (18 subsequent siblings)
  32 siblings, 1 reply; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Generate Rust #[cfg(...)] guards from QAPI 'if' conditions.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi/common.py | 16 ++++++++++++++++
 scripts/qapi/schema.py |  4 ++++
 2 files changed, 20 insertions(+)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 5f8f76e5b2..6d22c66391 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -201,6 +201,22 @@ def guardend(name: str) -> str:
                  name=c_fname(name).upper())
 
 
+def rsgen_ifcond(ifcond: Union[str, Dict[str, Any]]) -> str:
+
+    def cfg(ifcond: Union[str, Dict[str, Any]]):
+        if isinstance(ifcond, str):
+            return ifcond
+        if isinstance(ifcond, list):
+            return ', '.join([cfg(c) for c in ifcond])
+        oper, operands = next(iter(ifcond.items()))
+        operands = cfg(operands)
+        return f'{oper}({operands})'
+
+    if not ifcond:
+        return ''
+    return '#[cfg(%s)]' % cfg(ifcond)
+
+
 def gen_ifcond(ifcond: Optional[Union[str, Dict[str, Any]]],
                cond_fmt: str, not_fmt: str,
                all_operator: str, any_operator: str) -> str:
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 6455a8f425..c61f35e13f 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -26,6 +26,7 @@
     docgen_ifcond,
     gen_endif,
     gen_if,
+    rsgen_ifcond,
 )
 from .error import QAPIError, QAPISemError, QAPISourceError
 from .expr import check_exprs
@@ -48,6 +49,9 @@ def gen_endif(self):
     def docgen(self):
         return docgen_ifcond(self.ifcond)
 
+    def rsgen(self):
+        return rsgen_ifcond(self.ifcond)
+
     def is_present(self):
         return bool(self.ifcond)
 
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 15/32] scripts/qapi: strip trailing whitespaces
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (13 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 14/32] scripts/qapi: add QAPISchemaIfCond.rsgen() marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 16/32] scripts/qapi: add Rust FFI bindings generation marcandre.lureau
                   ` (17 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

This help workaround a rustfmt issue.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi/gen.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index ab26d5c937..fa23f43808 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -50,7 +50,11 @@ def add(self, text: str) -> None:
         self._body += text
 
     def get_content(self) -> str:
-        return self._top() + self._preamble + self._body + self._bottom()
+        content = self._top() + self._preamble + self._body + self._bottom()
+        # delete trailing white-spaces (working around
+        # https://github.com/rust-lang/rustfmt/issues/4248)
+        content = re.sub(r'\s+$', '\n', content, 0, re.M)
+        return content
 
     def _top(self) -> str:
         # pylint: disable=no-self-use
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 16/32] scripts/qapi: add Rust FFI bindings generation
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (14 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 15/32] scripts/qapi: strip trailing whitespaces marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 17/32] scripts/qapi: learn to generate ABI dump for Rust FFI marcandre.lureau
                   ` (16 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Generate the C QAPI types in Rust, with a few common niceties to
Debug/Clone/Copy the Rust type.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 meson.build            |   4 +-
 scripts/qapi/main.py   |  23 ++-
 scripts/qapi/rs.py     | 170 ++++++++++++++++
 scripts/qapi/rs_ffi.py | 446 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 635 insertions(+), 8 deletions(-)
 create mode 100644 scripts/qapi/rs.py
 create mode 100644 scripts/qapi/rs_ffi.py

diff --git a/meson.build b/meson.build
index 29d218d35a..b45f409eb4 100644
--- a/meson.build
+++ b/meson.build
@@ -2012,7 +2012,9 @@ qapi_gen_depends = [ meson.source_root() / 'scripts/qapi/__init__.py',
                      meson.source_root() / 'scripts/qapi/types.py',
                      meson.source_root() / 'scripts/qapi/visit.py',
                      meson.source_root() / 'scripts/qapi/common.py',
-                     meson.source_root() / 'scripts/qapi-gen.py'
+                     meson.source_root() / 'scripts/qapi/rs.py',
+                     meson.source_root() / 'scripts/qapi/rs_ffi.py',
+                     meson.source_root() / 'scripts/qapi-gen.py',
 ]
 
 tracetool = [
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
index f2ea6e0ce4..deba72ee4e 100644
--- a/scripts/qapi/main.py
+++ b/scripts/qapi/main.py
@@ -16,6 +16,7 @@
 from .error import QAPIError
 from .events import gen_events
 from .introspect import gen_introspect
+from .rs_ffi import gen_rs_ffitypes
 from .schema import QAPISchema
 from .types import gen_types
 from .visit import gen_visit
@@ -32,7 +33,8 @@ def generate(schema_file: str,
              output_dir: str,
              prefix: str,
              unmask: bool = False,
-             builtins: bool = False) -> None:
+             builtins: bool = False,
+             rust: bool = False) -> None:
     """
     Generate C code for the given schema into the target directory.
 
@@ -41,17 +43,21 @@ def generate(schema_file: str,
     :param prefix: Optional C-code prefix for symbol names.
     :param unmask: Expose non-ABI names through introspection?
     :param builtins: Generate code for built-in types?
+    :param rust: Generate Rust binding code?
 
     :raise QAPIError: On failures.
     """
     assert invalid_prefix_char(prefix) is None
 
     schema = QAPISchema(schema_file)
-    gen_types(schema, output_dir, prefix, builtins)
-    gen_visit(schema, output_dir, prefix, builtins)
-    gen_commands(schema, output_dir, prefix)
-    gen_events(schema, output_dir, prefix)
-    gen_introspect(schema, output_dir, prefix, unmask)
+    if rust:
+        gen_rs_ffitypes(schema, output_dir, prefix)
+    else:
+        gen_types(schema, output_dir, prefix, builtins)
+        gen_visit(schema, output_dir, prefix, builtins)
+        gen_commands(schema, output_dir, prefix)
+        gen_events(schema, output_dir, prefix)
+        gen_introspect(schema, output_dir, prefix, unmask)
 
 
 def main() -> int:
@@ -74,6 +80,8 @@ def main() -> int:
     parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                         dest='unmask',
                         help="expose non-ABI names in introspection")
+    parser.add_argument('-r', '--rust', action='store_true',
+                        help="generate Rust binding code")
     parser.add_argument('schema', action='store')
     args = parser.parse_args()
 
@@ -88,7 +96,8 @@ def main() -> int:
                  output_dir=args.output_dir,
                  prefix=args.prefix,
                  unmask=args.unmask,
-                 builtins=args.builtins)
+                 builtins=args.builtins,
+                 rust=args.rust)
     except QAPIError as err:
         print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
         return 1
diff --git a/scripts/qapi/rs.py b/scripts/qapi/rs.py
new file mode 100644
index 0000000000..be42329fa4
--- /dev/null
+++ b/scripts/qapi/rs.py
@@ -0,0 +1,170 @@
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+"""
+QAPI Rust generator
+"""
+
+import os
+import re
+import subprocess
+from typing import NamedTuple, Optional
+
+from .common import POINTER_SUFFIX
+from .gen import QAPIGen
+from .schema import QAPISchemaModule, QAPISchemaVisitor
+
+
+# see to_snake_case() below
+snake_case = re.compile(r'((?<=[a-z0-9])[A-Z]|(?!^)[A-Z](?=[a-z]))')
+
+
+rs_name_trans = str.maketrans('.-', '__')
+
+
+# Map @name to a valid Rust identifier.
+# If @protect, avoid returning certain ticklish identifiers (like
+# keywords) by prepending raw identifier prefix 'r#'.
+def rs_name(name: str, protect: bool = True) -> str:
+    name = name.translate(rs_name_trans)
+    if name[0].isnumeric():
+        name = '_' + name
+    if not protect:
+        return name
+    # based from the list:
+    # https://doc.rust-lang.org/reference/keywords.html
+    if name in ('Self', 'abstract', 'as', 'async',
+                'await', 'become', 'box', 'break',
+                'const', 'continue', 'crate', 'do',
+                'dyn', 'else', 'enum', 'extern',
+                'false', 'final', 'fn', 'for',
+                'if', 'impl', 'in', 'let',
+                'loop', 'macro', 'match', 'mod',
+                'move', 'mut', 'override', 'priv',
+                'pub', 'ref', 'return', 'self',
+                'static', 'struct', 'super', 'trait',
+                'true', 'try', 'type', 'typeof',
+                'union', 'unsafe', 'unsized', 'use',
+                'virtual', 'where', 'while', 'yield'):
+        name = 'r#' + name
+    # avoid some clashes with the standard library
+    if name in ('String',):
+        name = 'Qapi' + name
+
+    return name
+
+
+class CType(NamedTuple):
+    is_pointer: bool
+    is_const: bool
+    is_list: bool
+    c_type: str
+
+
+def rs_ctype_parse(c_type: str) -> CType:
+    is_pointer = False
+    if c_type.endswith(POINTER_SUFFIX):
+        is_pointer = True
+        c_type = c_type[:-len(POINTER_SUFFIX)]
+    is_list = c_type.endswith('List')
+    is_const = False
+    if c_type.startswith('const '):
+        is_const = True
+        c_type = c_type[6:]
+
+    c_type = rs_name(c_type)
+    return CType(is_pointer, is_const, is_list, c_type)
+
+
+def rs_ffitype(c_type: str, ffi_ns: str = 'qapi_ffi::',
+               list_as_newp: bool = False) -> str:
+    (is_pointer, is_const, is_list, c_type) = rs_ctype_parse(c_type)
+    c_type = rs_name(c_type)
+    to_rs = {
+        'bool': 'bool',
+        'char': 'libc::c_char',  # for clarity (i8)
+        'double': 'f64',
+        'int': 'i64',
+        'int16': 'i16',
+        'int16_t': 'i16',
+        'int32': 'i32',
+        'int32_t': 'i32',
+        'int64': 'i64',
+        'int64_t': 'i64',
+        'int8': 'i8',
+        'int8_t': 'i8',
+        'uint16': 'u16',
+        'uint16_t': 'u16',
+        'uint32': 'u32',
+        'uint32_t': 'u32',
+        'uint64': 'u64',
+        'uint64_t': 'u64',
+        'uint8': 'u8',
+        'uint8_t': 'u8',
+        'QObject': 'common::ffi::QObject',
+        'QNull': 'common::ffi::QNull',
+    }
+
+    ret = ''
+    if is_const and is_pointer:
+        ret += '*const '
+    elif is_pointer:
+        ret += '*mut '
+    if c_type in to_rs:
+        ret += to_rs[c_type]
+    else:
+        ret += ffi_ns + c_type
+
+    if is_list and list_as_newp:
+        ret = 'NewPtr<{}>'.format(ret)
+
+    return ret
+
+
+def to_camel_case(value: str) -> str:
+    # special case for last enum value
+    if value == '_MAX':
+        return value
+    raw_id = False
+    if value.startswith('r#'):
+        raw_id = True
+        value = value[2:]
+    value = ''.join('_' + word if word[0].isdigit()
+                    else word[:1].upper() + word[1:]
+                    for word in filter(None, re.split("[-_]+", value)))
+    if raw_id:
+        return 'r#' + value
+    return value
+
+
+def to_snake_case(value: str) -> str:
+    return snake_case.sub(r'_\1', value).lower()
+
+
+class QAPIGenRs(QAPIGen):
+    pass
+
+
+class QAPISchemaRsVisitor(QAPISchemaVisitor):
+
+    def __init__(self, prefix: str, what: str):
+        super().__init__()
+        self._prefix = prefix
+        self._what = what
+        self._gen = QAPIGenRs(self._prefix + self._what + '.rs')
+        self._main_module: Optional[str] = None
+
+    def visit_module(self, name: Optional[str]) -> None:
+        if name is None:
+            return
+        if QAPISchemaModule.is_user_module(name):
+            if self._main_module is None:
+                self._main_module = name
+
+    def write(self, output_dir: str) -> None:
+        self._gen.write(output_dir)
+
+        pathname = os.path.join(output_dir, self._gen.fname)
+        try:
+            subprocess.check_call(['rustfmt', pathname])
+        except FileNotFoundError:
+            pass
diff --git a/scripts/qapi/rs_ffi.py b/scripts/qapi/rs_ffi.py
new file mode 100644
index 0000000000..59e834b36a
--- /dev/null
+++ b/scripts/qapi/rs_ffi.py
@@ -0,0 +1,446 @@
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+"""
+QAPI Rust sys/ffi generator
+"""
+
+from pathlib import Path
+from typing import (
+    Dict,
+    List,
+    Optional,
+    Set,
+)
+
+from .cabi import CABI, CABIEnum, gen_object_cabi
+from .common import mcgen
+from .rs import (
+    QAPISchemaRsVisitor,
+    rs_ffitype,
+    rs_name,
+    to_camel_case,
+    to_snake_case,
+)
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaIfCond,
+    QAPISchemaModule,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaType,
+    QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
+
+
+objects_seen = set()
+
+
+def gen_rs_ffi_enum(name: str,
+                    ifcond: QAPISchemaIfCond,
+                    members: List[QAPISchemaEnumMember]) -> str:
+    # append automatically generated _max value
+    enum_members = members + [QAPISchemaEnumMember('_MAX', None)]
+
+    ret = mcgen('''
+
+%(cfg)s
+#[derive(Copy, Clone, Debug, PartialEq, Eq)]
+#[repr(C)]
+pub enum %(rs_name)s {
+''',
+                cfg=ifcond.rsgen(),
+                rs_name=rs_name(name))
+
+    for member in enum_members:
+        ret += mcgen('''
+    %(cfg)s
+    %(c_enum)s,
+''',
+                     cfg=member.ifcond.rsgen(),
+                     c_enum=to_camel_case(rs_name(member.name)))
+    # picked the first, since that's what malloc0 does
+    # but arguably could use _MAX instead, or a qapi annotation
+    default = to_camel_case(rs_name(enum_members[0].name))
+    ret += mcgen('''
+}
+
+%(cfg)s
+impl Default for %(rs_name)s {
+    #[inline]
+    fn default() -> %(rs_name)s {
+        Self::%(default)s
+    }
+}
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name),
+                 default=default)
+    return ret
+
+
+def gen_rs_ffi_struct_members(members:
+                              List[QAPISchemaObjectTypeMember]) -> str:
+    ret = ''
+    for memb in members:
+        if memb.optional:
+            ret += mcgen('''
+    %(cfg)s
+    pub has_%(rs_name)s: bool,
+''',
+                         cfg=memb.ifcond.rsgen(),
+                         rs_name=to_snake_case(rs_name(memb.name,
+                                                       protect=False)))
+        ret += mcgen('''
+    %(cfg)s
+    pub %(rs_name)s: %(rs_ffitype)s,
+''',
+                     cfg=memb.ifcond.rsgen(),
+                     rs_ffitype=rs_ffitype(memb.type.c_type(), ''),
+                     rs_name=to_snake_case(rs_name(memb.name)))
+    return ret
+
+
+def gen_rs_ffi_free(name: str, ifcond: QAPISchemaIfCond) -> str:
+    name = rs_name(name, protect=False)
+    typ = rs_name(name)
+    return mcgen('''
+
+%(cfg)s
+extern "C" {
+    pub fn qapi_free_%(name)s(obj: *mut %(ty)s);
+}
+''', cfg=ifcond.rsgen(), name=name, ty=typ)
+
+
+def gen_rs_ffi_variants(name: str,
+                        ifcond: QAPISchemaIfCond,
+                        variants: Optional[QAPISchemaVariants]) -> str:
+    ret = mcgen('''
+
+%(cfg)s
+#[repr(C)]
+#[derive(Copy, Clone)]
+pub union %(rs_name)s { /* union tag is @%(tag_name)s */
+''',
+                cfg=ifcond.rsgen(),
+                tag_name=rs_name(variants.tag_member.name),
+                rs_name=rs_name(name))
+
+    empty = True
+    for var in variants.variants:
+        if var.type.name == 'q_empty':
+            continue
+        empty = False
+        ret += mcgen('''
+    %(cfg)s
+    pub %(rs_name)s: %(rs_ffitype)s,
+''',
+                     cfg=var.ifcond.rsgen(),
+                     rs_ffitype=rs_ffitype(var.type.c_unboxed_type(), ''),
+                     rs_name=rs_name(var.name))
+
+    ret += mcgen('''
+    pub qapi_dummy: QapiDummy,
+''')
+
+    ret += mcgen('''
+}
+
+%(cfg)s
+impl ::std::fmt::Debug for %(rs_name)s {
+    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
+        f.debug_struct(&format!("%(rs_name)s @ {:?}", self as *const _))
+            .finish()
+    }
+}
+''', cfg=ifcond.rsgen(), rs_name=rs_name(name))
+
+    if empty:
+        ret += mcgen('''
+
+%(cfg)s
+impl ::std::default::Default for %(rs_name)s {
+    fn default() -> Self {
+        Self { qapi_dummy: QapiDummy }
+    }
+}
+''', cfg=ifcond.rsgen(), rs_name=rs_name(name))
+    return ret
+
+
+def gen_rs_ffi_object(name: str,
+                      ifcond: QAPISchemaIfCond,
+                      base: Optional[QAPISchemaObjectType],
+                      members: List[QAPISchemaObjectTypeMember],
+                      variants: Optional[QAPISchemaVariants]) -> str:
+    if name in objects_seen:
+        return ''
+
+    ret = ''
+    objects_seen.add(name)
+    unionty = name + 'Union'
+    if variants:
+        for var in variants.variants:
+            if isinstance(var.type, QAPISchemaObjectType):
+                ret += gen_rs_ffi_object(var.type.name,
+                                         var.type.ifcond,
+                                         var.type.base,
+                                         var.type.local_members,
+                                         var.type.variants)
+        ret += gen_rs_ffi_variants(unionty, ifcond, variants)
+
+    ret += gen_rs_ffi_free(name, ifcond)
+    ret += mcgen('''
+
+%(cfg)s
+#[repr(C)]
+#[derive(Copy, Clone, Debug)]
+pub struct %(rs_name)s {
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name))
+
+    if base:
+        if not base.is_implicit():
+            ret += mcgen('''
+    // Members inherited:
+''')
+        ret += gen_rs_ffi_struct_members(base.members)
+        if not base.is_implicit():
+            ret += mcgen('''
+    // Own members:
+''')
+
+    ret += gen_rs_ffi_struct_members(members)
+    if variants:
+        ret += mcgen('''
+        pub u: %(unionty)s
+''', unionty=rs_name(unionty))
+
+    empty = False
+    # for compatibility with C ABI
+    if (not base or base.is_empty()) and not members and not variants:
+        empty = True
+        ret += mcgen('''
+    pub qapi_dummy_for_empty_struct: u8,
+''')
+    ret += mcgen('''
+}
+''')
+
+    if empty:
+        ret += mcgen('''
+
+%(cfg)s
+impl Default for %(rs_name)s {
+    fn default() -> Self {
+        Self { qapi_dummy_for_empty_struct: 0 }
+    }
+}
+''',
+                     cfg=ifcond.rsgen(),
+                     rs_name=rs_name(name))
+
+    return ret
+
+
+def gen_rs_ffi_variant(name: str,
+                       ifcond: QAPISchemaIfCond,
+                       variants: Optional[QAPISchemaVariants]) -> str:
+    if name in objects_seen:
+        return ''
+
+    objects_seen.add(name)
+
+    ret = ''
+    gen_variants = ''
+    for var in variants.variants:
+        if var.type.name == 'q_empty':
+            continue
+        typ = rs_ffitype(var.type.c_unboxed_type(), '')
+        gen_variants += mcgen('''
+    %(cfg)s
+    pub %(mem_name)s: %(rs_ffitype)s,
+''',
+                              cfg=var.ifcond.rsgen(),
+                              rs_ffitype=typ,
+                              mem_name=rs_name(var.name))
+    if not gen_variants:
+        ret += mcgen('''
+   impl ::std::default::Default for %(rs_name)sUnion {
+       fn default() -> Self {
+           Self { qapi_dummy: QapiDummy }
+       }
+   }
+''')
+    gen_variants += mcgen('''
+    pub qapi_dummy: QapiDummy,
+''')
+
+    ret += gen_rs_ffi_free(name, ifcond)
+    ret += mcgen('''
+
+%(cfg)s
+#[repr(C)]
+#[derive(Copy,Clone)]
+pub union %(rs_name)sUnion {
+    %(variants)s
+}
+
+%(cfg)s
+impl ::std::fmt::Debug for %(rs_name)sUnion {
+    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
+        f.debug_struct(&format!("%(rs_name)sUnion @ {:?}", self as *const _))
+            .finish()
+    }
+}
+
+%(cfg)s
+#[repr(C)]
+#[derive(Copy, Clone, Debug)]
+pub struct %(rs_name)s {
+    pub %(tag)s: QType,
+    pub u: %(rs_name)sUnion,
+}
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name),
+                 tag=rs_name(variants.tag_member.name),
+                 variants=gen_variants)
+    return ret
+
+
+def gen_rs_ffi_array(name: str,
+                     ifcond: QAPISchemaIfCond,
+                     element_type: QAPISchemaType) -> str:
+    ret = mcgen('''
+
+%(cfg)s
+#[repr(C)]
+#[derive(Copy, Clone)]
+pub struct %(rs_name)s {
+    pub next: *mut %(rs_name)s,
+    pub value: %(rs_ffitype)s,
+}
+
+%(cfg)s
+impl ::std::fmt::Debug for %(rs_name)s {
+    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
+        f.debug_struct(&format!("%(rs_name)s @ {:?}", self as *const _))
+            .finish()
+    }
+}
+''',
+                cfg=ifcond.rsgen(),
+                rs_name=rs_name(name),
+                rs_ffitype=rs_ffitype(element_type.c_type(), ''))
+    ret += gen_rs_ffi_free(name, ifcond)
+    return ret
+
+
+class QAPISchemaGenRsFFITypeVisitor(QAPISchemaRsVisitor):
+
+    def __init__(self, prefix: str):
+        super().__init__(prefix, 'qapi-ffi-types')
+        self._cabi: Dict[str, CABI] = {}
+        self._cabi_functions: List[str] = []
+        self._cabi_functions_called: Set[str] = set()
+
+    def _cabi_add(self, cabis: List[CABI]) -> None:
+        for cabi in cabis:
+            self._cabi.setdefault(cabi.name, cabi)
+
+    def visit_begin(self, schema: QAPISchema) -> None:
+        # gen_object() is recursive, ensure it doesn't visit the empty type
+        objects_seen.add(schema.the_empty_object_type.name)
+        self._gen.preamble_add(
+            mcgen('''
+// generated by qapi-gen, DO NOT EDIT
+
+#[repr(C)]
+#[derive(Copy, Clone, Debug)]
+pub struct QapiDummy;
+'''))
+
+    def _get_qapi_cabi_fn(self, name: str) -> str:
+        fn_name = 'cabi'
+        if QAPISchemaModule.is_builtin_module(name):
+            fn_name += '_builtin'
+        elif name != self._main_module:
+            name = Path(name).stem
+            fn_name += '_' + rs_name(name, False)
+        return fn_name
+
+    def visit_include(self, name: str, info: Optional[QAPISourceInfo]) -> None:
+        super().visit_include(name, info)
+        cabi_fn = self._get_qapi_cabi_fn(name)
+        if cabi_fn not in self._cabi_functions_called:
+            self._cabi_functions.append(cabi_fn)
+
+    def visit_module_end(self, name: str) -> None:
+        cabi_gen = "".join(f'    {fn}();\n' for fn in self._cabi_functions)
+        self._cabi_functions_called |= set(self._cabi_functions)
+        self._cabi_functions = []
+        cabi_gen += "".join(
+            [c.gen_rs() for _, c in sorted(self._cabi.items())]
+        )
+        self._cabi = {}
+        fn_name = self._get_qapi_cabi_fn(name)
+        self._gen.add(mcgen('''
+#[cfg(QAPI_CABI)]
+pub(crate) fn %(fn_name)s() {
+%(cabi_gen)s
+}
+''', fn_name=fn_name, cabi_gen=cabi_gen))
+
+    def visit_enum_type(self,
+                        name: str,
+                        info: Optional[QAPISourceInfo],
+                        ifcond: QAPISchemaIfCond,
+                        features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]) -> None:
+        self._gen.add(gen_rs_ffi_enum(name, ifcond, members))
+        self._cabi_add([CABIEnum(name, ifcond, members, prefix)])
+
+    def visit_array_type(self,
+                         name: str,
+                         info: Optional[QAPISourceInfo],
+                         ifcond: QAPISchemaIfCond,
+                         element_type: QAPISchemaType) -> None:
+        self._gen.add(gen_rs_ffi_array(name, ifcond, element_type))
+
+    def visit_object_type(self,
+                          name: str,
+                          info: Optional[QAPISourceInfo],
+                          ifcond: QAPISchemaIfCond,
+                          features: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> None:
+        # Nothing to do for the special empty builtin
+        if name == 'q_empty':
+            return
+        self._gen.add(gen_rs_ffi_object(name, ifcond, base, members, variants))
+        self._cabi_add(gen_object_cabi(name, ifcond, base, members, variants))
+
+    def visit_alternate_type(self,
+                             name: str,
+                             info: QAPISourceInfo,
+                             ifcond: QAPISchemaIfCond,
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants) -> None:
+        self._gen.add(gen_rs_ffi_variant(name, ifcond, variants))
+        self._cabi_add(gen_object_cabi(name, ifcond, None,
+                                       [variants.tag_member], variants))
+
+
+def gen_rs_ffitypes(schema: QAPISchema,
+                    output_dir: str,
+                    prefix: str) -> None:
+    vis = QAPISchemaGenRsFFITypeVisitor(prefix)
+    schema.visit(vis)
+    vis.write(output_dir)
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 17/32] scripts/qapi: learn to generate ABI dump for Rust FFI
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (15 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 16/32] scripts/qapi: add Rust FFI bindings generation marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 18/32] tests: generate Rust bindings marcandre.lureau
                   ` (15 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi/cabi.py | 50 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/scripts/qapi/cabi.py b/scripts/qapi/cabi.py
index f33680477f..29c6d5144d 100644
--- a/scripts/qapi/cabi.py
+++ b/scripts/qapi/cabi.py
@@ -7,6 +7,7 @@
 from typing import List, Optional
 
 from .common import c_enum_const, c_name, mcgen
+from .rs import rs_name, to_snake_case
 from .schema import (
     QAPISchemaEnumMember,
     QAPISchemaIfCond,
@@ -24,6 +25,9 @@ def __init__(self, name: str, ifcond: QAPISchemaIfCond):
     def gen_c(self) -> str:
         raise NotImplementedError()
 
+    def gen_rs(self) -> str:
+        raise NotImplementedError()
+
 
 class CABIEnum(CABI):
     def __init__(
@@ -48,6 +52,19 @@ def gen_c(self) -> str:
         ret += self.ifcond.gen_endif()
         return ret
 
+    def gen_rs(self) -> str:
+        return mcgen("""
+%(cfg)s
+{
+    println!("%(name)s enum: sizeof={}", ::std::mem::size_of::<%(rs_name)s>());
+    println!(" max={}", %(rs_name)s::_MAX as u32);
+    println!();
+}
+""",
+                     name=self.name,
+                     rs_name=rs_name(self.name),
+                     cfg=self.ifcond.rsgen())
+
 
 class CABIStruct(CABI):
     def __init__(self, name: str, ifcond: QAPISchemaIfCond):
@@ -84,6 +101,21 @@ def gen_c(self) -> str:
         ret += self.ifcond.gen_endif()
         return ret
 
+    def gen_rs(self) -> str:
+        ret = mcgen("""
+%(cfg)s
+{
+    println!("%(name)s struct: sizeof={}",
+        ::std::mem::size_of::<%(rs_name)s>());
+""", name=self.name, rs_name=rs_name(self.name), cfg=self.ifcond.rsgen())
+        for member in self.members:
+            ret += member.gen_rs()
+        ret += mcgen("""
+    println!();
+}
+""")
+        return ret
+
 
 class CABIStructMember:
     def __init__(self, struct: CABIStruct, name: str,
@@ -105,6 +137,24 @@ def gen_c(self) -> str:
         ret += self.ifcond.gen_endif()
         return ret
 
+    def gen_rs(self) -> str:
+        ret = ''
+        if self.ifcond:
+            ret += self.ifcond.rsgen()
+        protect = not self.prefix or self.prefix[-1] == '.'
+        rsmember = self.prefix + to_snake_case(rs_name(self.name, protect))
+        ret += mcgen("""
+unsafe {
+    println!(" %(member)s member: sizeof={} offset={}",
+        ::std::mem::size_of_val(
+            &(*::std::ptr::null::<%(sname)s>()).%(rsmember)s
+        ),
+        &(*(::std::ptr::null::<%(sname)s>())).%(rsmember)s as *const _ as usize
+    );
+}
+""", member=self.name, sname=rs_name(self.struct.name), rsmember=rsmember)
+        return ret
+
 
 def gen_object_cabi(
     name: str,
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 18/32] tests: generate Rust bindings
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (16 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 17/32] scripts/qapi: learn to generate ABI dump for Rust FFI marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 19/32] tests: check Rust and C CABI diffs marcandre.lureau
                   ` (14 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/meson.build | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/tests/meson.build b/tests/meson.build
index be95223d62..7292fe20df 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -31,15 +31,29 @@ test_qapi_outputs = [
   'test-qapi-visit.h',
 ]
 
+test_qapi_inputs = files(
+  'qapi-schema/qapi-schema-test.json',
+  'qapi-schema/include/sub-module.json',
+  'qapi-schema/sub-sub-module.json'
+)
+
 test_qapi_files = custom_target('Test QAPI files',
                                 output: test_qapi_outputs,
-                                input: files('qapi-schema/qapi-schema-test.json',
-                                             'qapi-schema/include/sub-module.json',
-                                             'qapi-schema/sub-sub-module.json'),
+                                input: test_qapi_inputs,
                                 command: [ qapi_gen, '-o', meson.current_build_dir(),
                                            '-b', '-p', 'test-', '@INPUT0@' ],
                                 depend_files: qapi_gen_depends)
 
+if with_rust
+  test_qapi_rs_outputs = ['test-qapi-ffi-types.rs']
+  test_qapi_rs = custom_target('Test QAPI Rust binding',
+                               output: test_qapi_rs_outputs,
+                               input: test_qapi_inputs,
+                               command: [ qapi_gen, '-o', meson.current_build_dir(),
+                                          '-r', '-b', '-p', 'test-', '@INPUT0@' ],
+                               depend_files: qapi_gen_depends)
+endif
+
 # meson doesn't like generated output in other directories
 # perhaps change qapi_gen to replace / with _, like Meson itself does?
 subdir('include')
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 19/32] tests: check Rust and C CABI diffs
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (17 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 18/32] tests: generate Rust bindings marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 20/32] scripts/qapi: generate high-level Rust bindings marcandre.lureau
                   ` (13 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 meson.build              |  3 +++
 Cargo.lock               |  7 +++++++
 Cargo.toml               |  1 +
 scripts/diff_commands.py | 40 ++++++++++++++++++++++++++++++++++++++++
 tests/Cargo.toml         | 13 +++++++++++++
 tests/meson.build        | 29 +++++++++++++++++++++++++----
 tests/qapi-cabi.rs       |  5 +++++
 tests/qapi_ffi.rs        |  8 ++++++++
 8 files changed, 102 insertions(+), 4 deletions(-)
 create mode 100644 scripts/diff_commands.py
 create mode 100644 tests/Cargo.toml
 create mode 100644 tests/qapi-cabi.rs
 create mode 100644 tests/qapi_ffi.rs

diff --git a/meson.build b/meson.build
index b45f409eb4..74e90059c2 100644
--- a/meson.build
+++ b/meson.build
@@ -118,6 +118,7 @@ cargo_wrapper = [
   find_program('scripts/cargo_wrapper.py'),
   '--configh', meson.current_build_dir() / 'config-host.h'
 ]
+diff_commands = find_program('scripts/diff_commands.py')
 
 if with_rust
   rust_target_triple = get_option('with_rust_target')
@@ -212,7 +213,9 @@ iokit = []
 emulator_link_args = []
 nvmm =not_found
 hvf = not_found
+exe_suffix = ''
 if targetos == 'windows'
+  exe_suffix = '.exe'
   socket = cc.find_library('ws2_32')
   winmm = cc.find_library('winmm')
 
diff --git a/Cargo.lock b/Cargo.lock
index 8dc2dd9da7..f2cd0ee96e 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -61,3 +61,10 @@ dependencies = [
  "libc",
  "memoffset",
 ]
+
+[[package]]
+name = "qemu-tests"
+version = "0.0.1"
+dependencies = [
+ "common",
+]
diff --git a/Cargo.toml b/Cargo.toml
index 14131eed3c..f4a078e62d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,4 +1,5 @@
 [workspace]
 members = [
   "rust/common",
+  "tests",
 ]
diff --git a/scripts/diff_commands.py b/scripts/diff_commands.py
new file mode 100644
index 0000000000..eecc03dd76
--- /dev/null
+++ b/scripts/diff_commands.py
@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Marc-André Lureau <marcandre.lureau@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import argparse
+import difflib
+import subprocess
+import sys
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("EXE1")
+    parser.add_argument("EXE2")
+    args = parser.parse_args()
+
+    exe1_out = subprocess.check_output(args.EXE1, universal_newlines=True)
+    exe2_out = subprocess.check_output(args.EXE2, universal_newlines=True)
+    out_diff = difflib.unified_diff(
+        exe1_out.splitlines(True),
+        exe2_out.splitlines(True),
+        fromfile=args.EXE1,
+        tofile=args.EXE2,
+    )
+    has_diff = False
+    for line in out_diff:
+        has_diff = True
+        sys.stdout.write(line)
+
+    if has_diff:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tests/Cargo.toml b/tests/Cargo.toml
new file mode 100644
index 0000000000..7a4f6060b1
--- /dev/null
+++ b/tests/Cargo.toml
@@ -0,0 +1,13 @@
+[package]
+name = "qemu-tests"
+version = "0.0.1"
+edition = "2018"
+publish = false
+
+[dependencies]
+common = { path = "../rust/common" }
+
+[[bin]]
+name = "qapi-cabi-rs"
+path = "qapi-cabi.rs"
+doc = false
diff --git a/tests/meson.build b/tests/meson.build
index 7292fe20df..f9af42caba 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -52,6 +52,19 @@ if with_rust
                                command: [ qapi_gen, '-o', meson.current_build_dir(),
                                           '-r', '-b', '-p', 'test-', '@INPUT0@' ],
                                depend_files: qapi_gen_depends)
+  qapi_cabi_rs = custom_target('qapi-cabi-rs',
+                               output: ['qapi-cabi-rs' + exe_suffix],
+                               build_always_stale: true,
+                               depends: [test_qapi_rs],
+                               command: [cargo_wrapper,
+                                         meson.current_build_dir(),
+                                         meson.current_source_dir(),
+                                         meson.build_root(),
+                                         rs_build_type,
+                                         rust_target_triple,
+                                         'build-bin',
+                                         'qapi-cabi-rs',
+                                         '--', '--cfg', 'QAPI_CABI'])
 endif
 
 # meson doesn't like generated output in other directories
@@ -75,10 +88,18 @@ testqapi = declare_dependency(link_with: libtestqapi,
                               dependencies: [qemuutil],
                               sources: [genh, test_qapi_headers])
 
-executable('qapi-cabi',
-           files('qapi-cabi.c'),
-           dependencies: testqapi,
-           c_args: ['-DQAPI_CABI'])
+qapi_cabi = executable('qapi-cabi',
+                       files('qapi-cabi.c'),
+                       dependencies: testqapi,
+                       c_args: ['-DQAPI_CABI'])
+
+if with_rust
+  test('Test QAPI CABI diff',
+       diff_commands,
+       args: [qapi_cabi.full_path(), qapi_cabi_rs.full_path()],
+       depends: [qapi_cabi, qapi_cabi_rs],
+       suite: ['qapi'])
+endif
 
 test_deps = {
   'test-qht-par': qht_bench,
diff --git a/tests/qapi-cabi.rs b/tests/qapi-cabi.rs
new file mode 100644
index 0000000000..0b4b99cc78
--- /dev/null
+++ b/tests/qapi-cabi.rs
@@ -0,0 +1,5 @@
+mod qapi_ffi;
+
+fn main() {
+    qapi_ffi::cabi()
+}
diff --git a/tests/qapi_ffi.rs b/tests/qapi_ffi.rs
new file mode 100644
index 0000000000..d50a02efbe
--- /dev/null
+++ b/tests/qapi_ffi.rs
@@ -0,0 +1,8 @@
+#![allow(dead_code)]
+
+use common::libc;
+
+include!(concat!(
+    env!("MESON_BUILD_ROOT"),
+    "/tests/test-qapi-ffi-types.rs"
+));
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 20/32] scripts/qapi: generate high-level Rust bindings
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (18 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 19/32] tests: check Rust and C CABI diffs marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 21/32] tests/rust: build a common library, checking bindings compile marcandre.lureau
                   ` (12 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Generate high-level idiomatic Rust code for the QAPI types, with to/from
translations for the C FFI.

- char* is mapped to String, scalars to there corresponding Rust types

- enums are simply aliased from FFI

- has_foo/foo members are mapped to Option<T>

- lists are represented as Vec<T>

- structures have Rust versions, with To/From FFI conversions

- alternate are represented as Rust enum

- unions are represented in a similar way as in C: a struct S with a "u"
  member (since S may have extra 'base' fields). However, the discriminant
  isn't a member of S, since Rust enum already include it.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 meson.build              |   1 +
 scripts/qapi/main.py     |   2 +
 scripts/qapi/rs.py       |  94 +++-
 scripts/qapi/rs_types.py | 966 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 1062 insertions(+), 1 deletion(-)
 create mode 100644 scripts/qapi/rs_types.py

diff --git a/meson.build b/meson.build
index 74e90059c2..8e12a4dd70 100644
--- a/meson.build
+++ b/meson.build
@@ -2017,6 +2017,7 @@ qapi_gen_depends = [ meson.source_root() / 'scripts/qapi/__init__.py',
                      meson.source_root() / 'scripts/qapi/common.py',
                      meson.source_root() / 'scripts/qapi/rs.py',
                      meson.source_root() / 'scripts/qapi/rs_ffi.py',
+                     meson.source_root() / 'scripts/qapi/rs_types.py',
                      meson.source_root() / 'scripts/qapi-gen.py',
 ]
 
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
index deba72ee4e..9756c0c35d 100644
--- a/scripts/qapi/main.py
+++ b/scripts/qapi/main.py
@@ -17,6 +17,7 @@
 from .events import gen_events
 from .introspect import gen_introspect
 from .rs_ffi import gen_rs_ffitypes
+from .rs_types import gen_rs_types
 from .schema import QAPISchema
 from .types import gen_types
 from .visit import gen_visit
@@ -52,6 +53,7 @@ def generate(schema_file: str,
     schema = QAPISchema(schema_file)
     if rust:
         gen_rs_ffitypes(schema, output_dir, prefix)
+        gen_rs_types(schema, output_dir, prefix)
     else:
         gen_types(schema, output_dir, prefix, builtins)
         gen_visit(schema, output_dir, prefix, builtins)
diff --git a/scripts/qapi/rs.py b/scripts/qapi/rs.py
index be42329fa4..b53930eab2 100644
--- a/scripts/qapi/rs.py
+++ b/scripts/qapi/rs.py
@@ -9,7 +9,7 @@
 import subprocess
 from typing import NamedTuple, Optional
 
-from .common import POINTER_SUFFIX
+from .common import POINTER_SUFFIX, mcgen
 from .gen import QAPIGen
 from .schema import QAPISchemaModule, QAPISchemaVisitor
 
@@ -53,6 +53,64 @@ def rs_name(name: str, protect: bool = True) -> str:
     return name
 
 
+def rs_type(c_type: str,
+            qapi_ns: Optional[str] = 'qapi::',
+            optional: Optional[bool] = False,
+            box: bool = False) -> str:
+    (is_pointer, _, is_list, c_type) = rs_ctype_parse(c_type)
+    # accepts QAPI types ('any', 'str', ...) as we translate
+    # qapiList to Rust FFI types here.
+    to_rs = {
+        'any': 'QObject',
+        'bool': 'bool',
+        'char': 'i8',
+        'double': 'f64',
+        'int': 'i64',
+        'int16': 'i16',
+        'int16_t': 'i16',
+        'int32': 'i32',
+        'int32_t': 'i32',
+        'int64': 'i64',
+        'int64_t': 'i64',
+        'int8': 'i8',
+        'int8_t': 'i8',
+        'null': 'QNull',
+        'number': 'f64',
+        'size': 'u64',
+        'str': 'String',
+        'uint16': 'u16',
+        'uint16_t': 'u16',
+        'uint32': 'u32',
+        'uint32_t': 'u32',
+        'uint64': 'u64',
+        'uint64_t': 'u64',
+        'uint8': 'u8',
+        'uint8_t': 'u8',
+        'String': 'QapiString',
+    }
+    if is_pointer:
+        to_rs.update({
+            'char': 'String',
+        })
+
+    if is_list:
+        c_type = c_type[:-4]
+
+    to_rs = to_rs.get(c_type)
+    if to_rs:
+        ret = to_rs
+    else:
+        ret = qapi_ns + c_type
+
+    if is_list:
+        ret = 'Vec<%s>' % ret
+    elif is_pointer and not to_rs and box:
+        ret = 'Box<%s>' % ret
+    if optional:
+        ret = 'Option<%s>' % ret
+    return ret
+
+
 class CType(NamedTuple):
     is_pointer: bool
     is_const: bool
@@ -140,6 +198,40 @@ def to_snake_case(value: str) -> str:
     return snake_case.sub(r'_\1', value).lower()
 
 
+def to_qemu_none(c_type: str, name: str) -> str:
+    (is_pointer, _, is_list, _) = rs_ctype_parse(c_type)
+
+    if is_pointer:
+        if c_type == 'char':
+            return mcgen('''
+    let %(name)s_ = CString::new(%(name)s).unwrap();
+    let %(name)s = %(name)s_.as_ptr();
+''', name=name)
+        if is_list:
+            return mcgen('''
+    let %(name)s_ = NewPtr(%(name)s).to_qemu_none();
+    let %(name)s = %(name)s_.0.0;
+''', name=name)
+        return mcgen('''
+    let %(name)s_ = %(name)s.to_qemu_none();
+    let %(name)s = %(name)s_.0;
+''', name=name)
+    return ''
+
+
+def from_qemu(var_name: str, c_type: str, full: Optional[bool] = False) -> str:
+    (is_pointer, _, is_list, c_type) = rs_ctype_parse(c_type)
+    ptr = '{} as *{} _'.format(var_name, 'mut' if full else 'const')
+    if is_list:
+        ptr = 'NewPtr({})'.format(ptr)
+    if is_pointer:
+        ret = 'from_qemu_{}({})'.format('full' if full else 'none', ptr)
+        if c_type != 'char' and not is_list:
+            ret = 'Box::new(%s)' % ret
+        return ret
+    return var_name
+
+
 class QAPIGenRs(QAPIGen):
     pass
 
diff --git a/scripts/qapi/rs_types.py b/scripts/qapi/rs_types.py
new file mode 100644
index 0000000000..eb9877a0de
--- /dev/null
+++ b/scripts/qapi/rs_types.py
@@ -0,0 +1,966 @@
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+"""
+QAPI Rust types generator
+"""
+
+from typing import List, Optional
+
+from .common import POINTER_SUFFIX, mcgen
+from .rs import (
+    QAPISchemaRsVisitor,
+    from_qemu,
+    rs_ctype_parse,
+    rs_ffitype,
+    rs_name,
+    rs_type,
+    to_camel_case,
+    to_snake_case,
+)
+from .schema import (
+    QAPISchema,
+    QAPISchemaEnumMember,
+    QAPISchemaEnumType,
+    QAPISchemaFeature,
+    QAPISchemaIfCond,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaType,
+    QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
+
+
+objects_seen = set()
+
+
+def gen_rs_variants_to_tag(name: str,
+                           ifcond: QAPISchemaIfCond,
+                           variants: Optional[QAPISchemaVariants]) -> str:
+    ret = mcgen('''
+
+%(cfg)s
+impl From<&%(rs_name)sVariant> for %(tag)s {
+    fn from(e: &%(rs_name)sVariant) -> Self {
+        match e {
+    ''',
+                cfg=ifcond.rsgen(),
+                rs_name=rs_name(name),
+                tag=rs_type(variants.tag_member.type.c_type(), ''))
+
+    for var in variants.variants:
+        type_name = var.type.name
+        var_name = to_camel_case(rs_name(var.name))
+        patt = '(_)'
+        if type_name == 'q_empty':
+            patt = ''
+        ret += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariant::%(var_name)s%(patt)s => Self::%(var_name)s,
+''',
+                     cfg=var.ifcond.rsgen(),
+                     rs_name=rs_name(name),
+                     var_name=var_name,
+                     patt=patt)
+
+    ret += mcgen('''
+        }
+    }
+}
+''')
+    return ret
+
+
+def variants_to_qemu_inner(name: str,
+                           variants: Optional[QAPISchemaVariants]) -> str:
+    members = ''
+    none_arms = ''
+    full_arms = ''
+    lifetime = ''
+    for var in variants.variants:
+        var_name = to_camel_case(rs_name(var.name))
+        type_name = var.type.name
+        if type_name == 'q_empty':
+            members += mcgen('''
+    %(cfg)s
+    %(var_name)s,
+''',
+                             cfg=var.ifcond.rsgen(),
+                             var_name=var_name)
+            none_arms += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariant::%(var_name)s => {
+        (std::ptr::null_mut(),
+         %(rs_name)sVariantStorage::%(var_name)s)
+    },
+''',
+                               cfg=var.ifcond.rsgen(),
+                               rs_name=rs_name(name),
+                               var_name=var_name)
+            full_arms += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariant::%(var_name)s => {
+        std::ptr::null_mut()
+    }
+''',
+                               cfg=var.ifcond.rsgen(),
+                               rs_name=rs_name(name),
+                               var_name=var_name)
+            continue
+        c_type = var.type.c_unboxed_type()
+        if type_name.endswith('-wrapper'):
+            wrap = list(var.type.members)[0]
+            type_name = wrap.type.name
+            c_type = wrap.type.c_unboxed_type()
+
+        lifetime = "<'a>"
+        (_, _, is_list, ffitype) = rs_ctype_parse(c_type)
+        ffitype = rs_ffitype(ffitype)
+        ptr_ty = 'NewPtr<*mut %s>' % ffitype if is_list else '*mut ' + ffitype
+        stash_ty = ': Stash<%s, _>' % ptr_ty if is_list else ''
+
+        members += mcgen('''
+    %(cfg)s
+    %(var_name)s(<%(rs_type)s as ToQemuPtr<'a, %(ptr_ty)s>>::Storage),
+''',
+                         cfg=var.ifcond.rsgen(),
+                         var_name=var_name,
+                         rs_type=rs_type(c_type, ''),
+                         ptr_ty=ptr_ty)
+        none_arms += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariant::%(var_name)s(v) => {
+        let stash_%(stash_ty)s = v.to_qemu_none();
+        (stash_.0.to() as *mut std::ffi::c_void,
+         %(rs_name)sVariantStorage::%(var_name)s(stash_.1))
+    },
+''',
+                           cfg=var.ifcond.rsgen(),
+                           rs_name=rs_name(name),
+                           var_name=var_name,
+                           stash_ty=stash_ty)
+        ptr_ty = ': %s' % ptr_ty if is_list else ''
+        full_arms += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariant::%(var_name)s(v) => {
+        let ptr%(ptr_ty)s = v.to_qemu_full();
+        ptr.to() as *mut std::ffi::c_void
+    },
+''',
+                           cfg=var.ifcond.rsgen(),
+                           rs_name=rs_name(name),
+                           var_name=var_name,
+                           ptr_ty=ptr_ty)
+    return (members, none_arms, full_arms, lifetime)
+
+
+def gen_rs_variants_to_qemu(name: str,
+                            ifcond: QAPISchemaIfCond,
+                            variants: Optional[QAPISchemaVariants]) -> str:
+    (members, none_arms, full_arms, lifetime) = \
+        variants_to_qemu_inner(name, variants)
+    return mcgen('''
+
+%(cfg)s
+impl QemuPtrDefault for %(rs_name)sVariant {
+    type QemuType = *mut std::ffi::c_void;
+}
+
+%(cfg)s
+pub enum %(rs_name)sVariantStorage%(lt)s {
+    %(members)s
+}
+
+%(cfg)s
+impl<'a> ToQemuPtr<'a, *mut std::ffi::c_void> for %(rs_name)sVariant {
+    type Storage = %(rs_name)sVariantStorage%(lt)s;
+
+    #[inline]
+    fn to_qemu_none(&'a self)
+    -> Stash<'a, *mut std::ffi::c_void, %(rs_name)sVariant> {
+        let (ptr_, cenum_) = match self {
+             %(none_arms)s
+        };
+
+        Stash(ptr_, cenum_)
+    }
+
+    #[inline]
+    fn to_qemu_full(&self) -> *mut std::ffi::c_void {
+        match self {
+            %(full_arms)s
+        }
+    }
+}
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name),
+                 lt=lifetime,
+                 members=members,
+                 none_arms=none_arms,
+                 full_arms=full_arms)
+
+
+def gen_rs_variants(name: str,
+                    ifcond: QAPISchemaIfCond,
+                    variants: Optional[QAPISchemaVariants]) -> str:
+    ret = mcgen('''
+
+%(cfg)s
+#[derive(Clone,Debug)]
+pub enum %(rs_name)sVariant {
+''',
+                cfg=ifcond.rsgen(),
+                rs_name=rs_name(name))
+
+    for var in variants.variants:
+        type_name = var.type.name
+        var_name = to_camel_case(rs_name(var.name, False))
+        if type_name == 'q_empty':
+            ret += mcgen('''
+    %(cfg)s
+    %(var_name)s,
+''',
+                         cfg=var.ifcond.rsgen(),
+                         var_name=var_name)
+        else:
+            c_type = var.type.c_unboxed_type()
+            if c_type.endswith('_wrapper'):
+                c_type = c_type[6:-8]  # remove q_obj*-wrapper
+            ret += mcgen('''
+    %(cfg)s
+    %(var_name)s(%(rs_type)s),
+''',
+                         cfg=var.ifcond.rsgen(),
+                         var_name=var_name,
+                         rs_type=rs_type(c_type, ''))
+
+    ret += mcgen('''
+}
+''')
+
+    ret += gen_rs_variants_to_tag(name, ifcond, variants)
+    # implement ToQemu trait for the storage handling
+    # no need for gen_rs_variants_from_qemu() however
+    ret += gen_rs_variants_to_qemu(name, ifcond, variants)
+
+    return ret
+
+
+def gen_rs_object_to_qemu(name: str,
+                          ifcond: QAPISchemaIfCond,
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> str:
+    storage = []
+    stash = []
+    ffi_memb = []
+    memb_none = ''
+    memb_full = ''
+    if base:
+        members = list(base.members) + members
+    for memb in members:
+        if variants and variants.tag_member.name == memb.name:
+            continue
+        memb_name = to_snake_case(rs_name(memb.name))
+        c_type = memb.type.c_type()
+        (is_pointer, _, is_list, _) = rs_ctype_parse(c_type)
+        if is_pointer:
+            if memb.ifcond.is_present():
+                raise NotImplementedError("missing support for condition here")
+            typ = rs_type(memb.type.c_type(),
+                          optional=memb.optional,
+                          qapi_ns='',
+                          box=True)
+            styp = rs_ffitype(memb.type.c_type(), list_as_newp=True)
+            storage.append("Stash<'a, %s, %s>" % (styp, typ))
+        if memb.optional:
+            has_memb_name = 'has_%s' % rs_name(memb.name, protect=False)
+            ffi_memb.append(f"{memb.ifcond.rsgen()} {has_memb_name}")
+            has_memb = mcgen('''
+    %(cfg)s
+    let %(has_memb_name)s = self.%(memb_name)s.is_some();
+''',
+                             cfg=memb.ifcond.rsgen(),
+                             memb_name=memb_name,
+                             has_memb_name=has_memb_name)
+            memb_none += has_memb
+            memb_full += has_memb
+
+        if is_pointer:
+            stash_name = '{}_stash_'.format(memb_name)
+            stash.append(stash_name)
+            var = 'NewPtr(%s)' % memb_name if is_list else memb_name
+            memb_none += mcgen('''
+    let %(stash_name)s = self.%(memb_name)s.to_qemu_none();
+    let %(var)s = %(stash_name)s.0;
+''', stash_name=stash_name, memb_name=memb_name, var=var)
+            memb_full += mcgen('''
+    let %(var)s = self.%(memb_name)s.to_qemu_full();
+''', memb_name=memb_name, var=var)
+        else:
+            unwrap = ''
+            if memb.optional:
+                unwrap = '.unwrap_or_default()'
+            assign = mcgen('''
+    %(cfg)s
+    let %(memb_name)s = self.%(memb_name)s%(unwrap)s;
+''',
+                           cfg=memb.ifcond.rsgen(),
+                           memb_name=memb_name,
+                           unwrap=unwrap)
+            memb_none += assign
+            memb_full += assign
+
+        ffi_memb.append(f"{memb.ifcond.rsgen()} {memb_name}")
+
+    if variants:
+        tag_name = rs_name(variants.tag_member.name)
+        ffi_memb.append(tag_name)
+        ffi_memb.append('u')
+        voidp = '*mut std::ffi::c_void'
+        storage.append("Stash<'a, %s, %sVariant>" % (voidp, rs_name(name)))
+        tag = mcgen('''
+    let %(tag_name)s = (&self.u).into();
+''', tag_name=tag_name)
+        memb_none += tag
+        memb_full += tag
+        arms_none = ''
+        arms_full = ''
+        for variant in variants.variants:
+            typ = variant.type
+            if typ.name == 'q_empty':
+                arms_none += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariantStorage::%(kind)s => qapi_ffi::%(rs_name)sUnion {
+        qapi_dummy: qapi_ffi::QapiDummy,
+    },''',
+                                   cfg=variant.ifcond.rsgen(),
+                                   rs_name=rs_name(name),
+                                   kind=to_camel_case(rs_name(variant.name)))
+                arms_full += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariant::%(kind)s => qapi_ffi::%(rs_name)sUnion {
+        qapi_dummy: qapi_ffi::QapiDummy,
+    },''',
+                                   cfg=variant.ifcond.rsgen(),
+                                   rs_name=rs_name(name),
+                                   kind=to_camel_case(rs_name(variant.name)))
+            else:
+                if typ.name.endswith('-wrapper'):
+                    wrap_ty = list(typ.members)[0].type.c_type()
+                    ptr = wrap_ty.endswith(POINTER_SUFFIX)
+                    val = (
+                        rs_ffitype(variant.type.c_unboxed_type()) +
+                        ' { data: u_stash_.0.to() as *mut _ }' if ptr else
+                        ' { data: unsafe { *(u_stash_.0.to() as *const _) } }'
+                    )
+                else:
+                    val = '*_s.0'
+                arms_none += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariantStorage::%(kind)s(ref _s) => qapi_ffi::%(rs_name)sUnion {
+        %(var_name)s: %(val)s,
+    },''',
+                                   cfg=variant.ifcond.rsgen(),
+                                   rs_name=rs_name(name),
+                                   kind=to_camel_case(rs_name(variant.name)),
+                                   var_name=rs_name(variant.name),
+                                   val=val)
+                arms_full += mcgen('''
+    %(cfg)s
+    %(rs_name)sVariant::%(kind)s(_) => qapi_ffi::%(rs_name)sUnion {
+        %(var_name)s: *(u_ptr_.to() as *const _),
+    },''',
+                                   cfg=variant.ifcond.rsgen(),
+                                   rs_name=rs_name(name),
+                                   kind=to_camel_case(rs_name(variant.name)),
+                                   var_name=rs_name(variant.name))
+        memb_none += mcgen('''
+    let u_stash_ = self.u.to_qemu_none();
+    let u = match u_stash_.1 {
+        %(arms)s
+    };
+''', arms=arms_none)
+        stash.append('u_stash_')
+        memb_full += mcgen('''
+    let u_ptr_ = self.u.to_qemu_full();
+    let u = match self.u {
+        %(arms)s
+    };
+    ffi::g_free(u_ptr_);
+''', arms=arms_full)
+
+    if not ffi_memb:
+        ffi_memb = ['qapi_dummy_for_empty_struct: 0']
+
+    return mcgen('''
+
+%(cfg)s
+impl QemuPtrDefault for %(rs_name)s {
+    type QemuType = *mut qapi_ffi::%(rs_name)s;
+}
+
+%(cfg)s
+impl<'a> ToQemuPtr<'a, *mut qapi_ffi::%(rs_name)s> for %(rs_name)s {
+    type Storage = (Box<qapi_ffi::%(rs_name)s>, %(storage)s);
+
+    #[inline]
+    fn to_qemu_none(&'a self)
+    -> Stash<'a, *mut qapi_ffi::%(rs_name)s, %(rs_name)s> {
+        %(memb_none)s
+        let mut box_ = Box::new(qapi_ffi::%(rs_name)s { %(ffi_memb)s });
+
+        Stash(&mut *box_, (box_, %(stash)s))
+    }
+
+    #[inline]
+    fn to_qemu_full(&self) -> *mut qapi_ffi::%(rs_name)s {
+        unsafe {
+            %(memb_full)s
+            let ptr = ffi::g_malloc0(
+                std::mem::size_of::<%(rs_name)s>()) as *mut _;
+            *ptr = qapi_ffi::%(rs_name)s { %(ffi_memb)s };
+            ptr
+        }
+    }
+}
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name),
+                 storage=', '.join(storage),
+                 ffi_memb=', '.join(ffi_memb),
+                 memb_none=memb_none,
+                 memb_full=memb_full,
+                 stash=', '.join(stash))
+
+
+def gen_rs_members(members: List[QAPISchemaObjectTypeMember],
+                   exclude: List[str] = None) -> str:
+    exclude = exclude or []
+    return [f"{m.ifcond.rsgen()} {to_snake_case(rs_name(m.name))}"
+            for m in members if m.name not in exclude]
+
+
+def gen_rs_object_from_qemu(name: str,
+                            ifcond: QAPISchemaIfCond,
+                            base: Optional[QAPISchemaObjectType],
+                            members: List[QAPISchemaObjectTypeMember],
+                            variants: Optional[QAPISchemaVariants]) -> str:
+    exclude = [variants.tag_member.name] if variants else []
+    memb_names = []
+    if base:
+        names = gen_rs_members(base.members, exclude)
+        memb_names.extend(names)
+    names = gen_rs_members(members, exclude)
+    memb_names.extend(names)
+
+    ret = mcgen('''
+
+%(cfg)s
+impl FromQemuPtrFull<*mut qapi_ffi::%(rs_name)s> for %(rs_name)s {
+    unsafe fn from_qemu_full(ffi: *mut qapi_ffi::%(rs_name)s) -> Self {
+        let ret = from_qemu_none(ffi as *const _);
+        qapi_ffi::qapi_free_%(name)s(ffi);
+        ret
+    }
+}
+
+%(cfg)s
+impl FromQemuPtrNone<*const qapi_ffi::%(rs_name)s> for %(rs_name)s {
+    unsafe fn from_qemu_none(ffi: *const qapi_ffi::%(rs_name)s) -> Self {
+        let _ffi = &*ffi;
+''',
+                cfg=ifcond.rsgen(),
+                name=rs_name(name, protect=False),
+                rs_name=rs_name(name))
+
+    if base:
+        members = list(base.members) + members
+
+    tag_member = variants.tag_member if variants else None
+    for memb in members:
+        if memb == tag_member:
+            continue
+        memb_name = rs_name(memb.name)
+        val = from_qemu('_ffi.' + to_snake_case(memb_name), memb.type.c_type())
+        if memb.optional:
+            val = mcgen('''{
+            if _ffi.has_%(memb_name)s {
+                Some(%(val)s)
+            } else {
+                None
+            }
+}''',
+                        memb_name=rs_name(memb.name, protect=False),
+                        val=val)
+
+        ret += mcgen('''
+        %(cfg)s
+        let %(snake_memb_name)s = %(val)s;
+''',
+                     cfg=memb.ifcond.rsgen(),
+                     snake_memb_name=to_snake_case(memb_name),
+                     memb_name=memb_name,
+                     val=val)
+
+    if variants:
+        arms = ''
+        assert isinstance(variants.tag_member.type, QAPISchemaEnumType)
+        for variant in variants.variants:
+            typ = variant.type
+            if typ.name == 'q_empty':
+                memb = ''
+            else:
+                ptr = True
+                is_list = False
+                memb = to_snake_case(rs_name(variant.name))
+                if typ.name.endswith('-wrapper'):
+                    memb = '_ffi.u.%s.data' % memb
+                    wrap_ty = list(typ.members)[0].type.c_type()
+                    (ptr, _, is_list, _) = rs_ctype_parse(wrap_ty)
+                else:
+                    memb = '&_ffi.u.%s' % memb
+                if ptr:
+                    memb = '%s as *const _' % memb
+                    if is_list:
+                        memb = 'NewPtr(%s)' % memb
+                    memb = 'from_qemu_none(%s)' % memb
+                memb = '(%s)' % memb
+            arms += mcgen('''
+%(cfg)s
+%(enum)s::%(variant)s => { %(rs_name)sVariant::%(variant)s%(memb)s },
+''',
+                          cfg=variant.ifcond.rsgen(),
+                          enum=rs_name(variants.tag_member.type.name),
+                          memb=memb,
+                          variant=to_camel_case(rs_name(variant.name)),
+                          rs_name=rs_name(name))
+        ret += mcgen('''
+        let u = match _ffi.%(tag)s {
+            %(arms)s
+            _ => panic!("Variant with invalid tag"),
+        };
+''',
+                     tag=rs_name(variants.tag_member.name),
+                     arms=arms)
+        memb_names.append('u')
+
+    ret += mcgen('''
+            Self { %(memb_names)s }
+        }
+}
+''',
+                 memb_names=', '.join(memb_names))
+    return ret
+
+
+def gen_struct_members(members: List[QAPISchemaObjectTypeMember]) -> str:
+    ret = ''
+    for memb in members:
+        typ = rs_type(memb.type.c_type(), '', optional=memb.optional, box=True)
+        ret += mcgen('''
+    %(cfg)s
+    pub %(rs_name)s: %(rs_type)s,
+''',
+                     cfg=memb.ifcond.rsgen(),
+                     rs_type=typ,
+                     rs_name=to_snake_case(rs_name(memb.name)))
+    return ret
+
+
+def gen_rs_object(name: str,
+                  ifcond: QAPISchemaIfCond,
+                  base: Optional[QAPISchemaObjectType],
+                  members: List[QAPISchemaObjectTypeMember],
+                  variants: Optional[QAPISchemaVariants]) -> str:
+    if name in objects_seen:
+        return ''
+
+    if variants:
+        members = [m for m in members
+                   if m.name != variants.tag_member.name]
+
+    ret = ''
+    objects_seen.add(name)
+
+    if variants:
+        ret += gen_rs_variants(name, ifcond, variants)
+
+    ret += mcgen('''
+
+%(cfg)s
+#[derive(Clone, Debug)]
+pub struct %(rs_name)s {
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name))
+
+    if base:
+        if not base.is_implicit():
+            ret += mcgen('''
+    // Members inherited:
+''',
+                         c_name=base.c_name())
+        base_members = base.members
+        if variants:
+            base_members = [m for m in base.members
+                            if m.name != variants.tag_member.name]
+        ret += gen_struct_members(base_members)
+        if not base.is_implicit():
+            ret += mcgen('''
+    // Own members:
+''')
+
+    ret += gen_struct_members(members)
+
+    if variants:
+        ret += mcgen('''
+    pub u: %(rs_type)sVariant,
+''', rs_type=rs_name(name))
+    ret += mcgen('''
+}
+''')
+
+    ret += gen_rs_object_from_qemu(name, ifcond, base, members, variants)
+    ret += gen_rs_object_to_qemu(name, ifcond, base, members, variants)
+    return ret
+
+
+def gen_rs_alternate_from_qemu(name: str,
+                               ifcond: QAPISchemaIfCond,
+                               variants: Optional[QAPISchemaVariants]) -> str:
+    arms = ''
+    for var in variants.variants:
+        qtype = to_camel_case(var.type.alternate_qtype()[6:].lower())
+        ptr = var.type.c_unboxed_type().endswith(POINTER_SUFFIX)
+        memb = 'ffi.u.%s' % rs_name(var.name)
+        if not ptr:
+            memb = '&' + memb
+        arms += mcgen('''
+        %(cfg)s
+        QType::%(qtype)s => {
+            Self::%(kind)s(from_qemu_none(%(memb)s as *const _))
+        }
+''',
+                      qtype=qtype,
+                      cfg=var.ifcond.rsgen(),
+                      kind=to_camel_case(rs_name(var.name)),
+                      memb=memb)
+
+    ret = mcgen('''
+
+%(cfg)s
+impl FromQemuPtrFull<*mut qapi_ffi::%(rs_name)s> for %(rs_name)s {
+    unsafe fn from_qemu_full(ffi: *mut qapi_ffi::%(rs_name)s) -> Self {
+        let ret = from_qemu_none(ffi as *const _);
+        qapi_ffi::qapi_free_%(name)s(ffi);
+        ret
+    }
+}
+
+%(cfg)s
+impl FromQemuPtrNone<*const qapi_ffi::%(rs_name)s> for %(rs_name)s {
+    unsafe fn from_qemu_none(ffi: *const qapi_ffi::%(rs_name)s) -> Self {
+        let ffi = &*ffi;
+
+        match ffi.r#type {
+            %(arms)s
+            _ => panic!()
+        }
+    }
+}
+''',
+                cfg=ifcond.rsgen(),
+                name=rs_name(name, protect=False),
+                rs_name=rs_name(name),
+                arms=arms)
+    return ret
+
+
+def gen_rs_alternate_to_qemu(name: str,
+                             ifcond: QAPISchemaIfCond,
+                             variants: Optional[QAPISchemaVariants],
+                             lifetime: str) -> str:
+    arms_none = ''
+    arms_full = ''
+    for var in variants.variants:
+        if var.type.name == 'q_empty':
+            continue
+        ptr = var.type.c_unboxed_type().endswith(POINTER_SUFFIX)
+        val = 'val.0' if ptr else 'unsafe { *(val.0.to() as *const _) }'
+        stor = '(val.1)' if var.type.c_type().endswith(POINTER_SUFFIX) else ''
+        qtype = var.type.alternate_qtype()[6:].lower()
+        arms_none += mcgen('''
+        %(cfg)s
+        Self::%(memb_name)s(val) => {
+            let val = val.to_qemu_none();
+            (
+                QType::%(qtype)s,
+                qapi_ffi::%(rs_name)sUnion { %(ffi_memb_name)s: %(val)s },
+                %(rs_name)sStorage::%(memb_name)s%(stor)s
+            )
+        }
+''',
+                           rs_name=rs_name(name),
+                           cfg=var.ifcond.rsgen(),
+                           memb_name=to_camel_case(rs_name(var.name)),
+                           ffi_memb_name=rs_name(var.name),
+                           qtype=to_camel_case(qtype),
+                           val=val,
+                           stor=stor)
+        val = 'val' if ptr else '*val'
+        free = '' if ptr else 'ffi::g_free(val as *mut _);'
+        arms_full += mcgen('''
+        %(cfg)s
+        Self::%(memb_name)s(val) => {
+            let val = val.to_qemu_full();
+            let ret = (QType::%(qtype)s, qapi_ffi::%(rs_name)sUnion {
+                %(ffi_memb_name)s: %(val)s
+            } );
+            %(free)s
+            ret
+        }
+''',
+                           rs_name=rs_name(name),
+                           cfg=var.ifcond.rsgen(),
+                           memb_name=to_camel_case(rs_name(var.name)),
+                           ffi_memb_name=rs_name(var.name),
+                           qtype=to_camel_case(qtype),
+                           val=val,
+                           free=free)
+
+    memb_none = mcgen('''
+    let (r#type, u, stor) = match self {
+        %(arms_none)s
+    };
+''', arms_none=arms_none)
+    memb_full = mcgen('''
+    let (r#type, u) = match self {
+        %(arms_full)s
+    };
+''', arms_full=arms_full)
+    ffi_memb = ['r#type', 'u']
+    return mcgen('''
+
+%(cfg)s
+impl QemuPtrDefault for %(rs_name)s {
+    type QemuType = *mut qapi_ffi::%(rs_name)s;
+}
+
+%(cfg)s
+impl<'a> ToQemuPtr<'a, *mut qapi_ffi::%(rs_name)s> for %(rs_name)s {
+    // Additional boxing of storage needed due to recursive types
+    type Storage = (Box<qapi_ffi::%(rs_name)s>, Box<%(rs_name)sStorage%(lt)s>);
+
+    #[inline]
+    fn to_qemu_none(&'a self)
+    -> Stash<'a, *mut qapi_ffi::%(rs_name)s, %(rs_name)s> {
+        %(memb_none)s
+        let mut box_ = Box::new(qapi_ffi::%(rs_name)s { %(ffi_memb)s });
+
+        Stash(&mut *box_, (box_, Box::new(stor)))
+    }
+
+    #[inline]
+    fn to_qemu_full(&self) -> *mut qapi_ffi::%(rs_name)s {
+        unsafe {
+            %(memb_full)s
+            let ptr = ffi::g_malloc0(
+                std::mem::size_of::<%(rs_name)s>()) as *mut _;
+            *ptr = qapi_ffi::%(rs_name)s { %(ffi_memb)s };
+            ptr
+        }
+    }
+}
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name),
+                 lt=lifetime,
+                 ffi_memb=', '.join(ffi_memb),
+                 memb_none=memb_none,
+                 memb_full=memb_full)
+
+
+def gen_rs_alternate(name: str,
+                     ifcond: QAPISchemaIfCond,
+                     variants: Optional[QAPISchemaVariants]) -> str:
+    if name in objects_seen:
+        return ''
+
+    ret = ''
+    objects_seen.add(name)
+
+    ret += mcgen('''
+%(cfg)s
+#[derive(Clone, Debug)]
+pub enum %(rs_name)s {
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name))
+
+    for var in variants.variants:
+        if var.type.name == 'q_empty':
+            continue
+        ret += mcgen('''
+        %(cfg)s
+        %(mem_name)s(%(rs_type)s),
+''',
+                     cfg=var.ifcond.rsgen(),
+                     rs_type=rs_type(var.type.c_unboxed_type(), ''),
+                     mem_name=to_camel_case(rs_name(var.name)))
+
+    membs = ''
+    lifetime = ''
+    for var in variants.variants:
+        var_name = to_camel_case(rs_name(var.name))
+        type_name = var.type.name
+        if type_name == 'q_empty':
+            continue
+        if not var.type.c_type().endswith(POINTER_SUFFIX):
+            membs += mcgen('''
+    %(cfg)s
+    %(var_name)s,
+''',
+                           cfg=var.ifcond.rsgen(),
+                           var_name=var_name)
+        else:
+            lifetime = "<'a>"
+            c_type = var.type.c_type()
+            ptr_ty = rs_ffitype(c_type)
+            membs += mcgen('''
+    %(cfg)s
+    %(var_name)s(<%(rs_type)s as ToQemuPtr<'a, %(ptr_ty)s>>::Storage),
+''',
+                           cfg=var.ifcond.rsgen(),
+                           var_name=var_name,
+                           rs_type=rs_type(c_type, ''),
+                           ptr_ty=ptr_ty)
+    ret += mcgen('''
+}
+
+%(cfg)s
+pub enum %(rs_name)sStorage%(lt)s {
+    %(membs)s
+}
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name),
+                 lt=lifetime,
+                 membs=membs)
+
+    ret += gen_rs_alternate_from_qemu(name, ifcond, variants)
+    ret += gen_rs_alternate_to_qemu(name, ifcond, variants, lifetime)
+
+    return ret
+
+
+def gen_rs_enum(name: str, ifcond: QAPISchemaIfCond) -> str:
+    return mcgen('''
+
+%(cfg)s
+pub type %(rs_name)s = qapi_ffi::%(ffi_name)s;
+
+%(cfg)s
+impl_to_qemu_scalar_boxed!(%(rs_name)s);
+
+%(cfg)s
+impl_from_qemu_none_scalar!(%(rs_name)s);
+''',
+                 cfg=ifcond.rsgen(),
+                 rs_name=rs_name(name),
+                 ffi_name=rs_name(name))
+
+
+class QAPISchemaGenRsTypeVisitor(QAPISchemaRsVisitor):
+
+    def __init__(self, prefix: str) -> None:
+        super().__init__(prefix, 'qapi-types')
+
+    def visit_begin(self, schema: QAPISchema) -> None:
+        # don't visit the empty type
+        objects_seen.add(schema.the_empty_object_type.name)
+        self._gen.preamble_add(
+            mcgen('''
+// generated by qapi-gen, DO NOT EDIT
+
+use common::{QNull, QObject};
+use crate::qapi_ffi;
+
+'''))
+
+    def visit_array_type(self,
+                         name: str,
+                         info: Optional[QAPISourceInfo],
+                         ifcond: QAPISchemaIfCond,
+                         element_type: QAPISchemaType) -> None:
+        typ = rs_type(name, qapi_ns='')
+        scalar = False
+        if name[:-4] in {'number',
+                         'int',
+                         'int8',
+                         'int16',
+                         'int32',
+                         'int64',
+                         'uint8',
+                         'uint16',
+                         'uint32',
+                         'uint64',
+                         'size',
+                         'bool'}:
+            scalar = True
+        if isinstance(element_type, QAPISchemaEnumType):
+            scalar = True
+
+        self._gen.add(mcgen('''
+%(cfg)s
+mod %(mod)s_module {
+    use super::*;
+
+    vec_type!(%(rs)s, %(ffi)s, qapi_free_%(name)s, %(scalar)i);
+}
+
+%(cfg)s
+pub use %(mod)s_module::*;
+''',
+                            cfg=ifcond.rsgen(),
+                            name=rs_name(name, protect=False),
+                            mod=rs_name(name).lower(),
+                            ffi=rs_name(name),
+                            rs=typ,
+                            scalar=scalar))
+
+    def visit_object_type(self,
+                          name: str,
+                          info: Optional[QAPISourceInfo],
+                          ifcond: QAPISchemaIfCond,
+                          features: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> None:
+        if name.startswith('q_'):
+            return
+        self._gen.add(gen_rs_object(name, ifcond, base, members, variants))
+
+    def visit_enum_type(self,
+                        name: str,
+                        info: Optional[QAPISourceInfo],
+                        ifcond: QAPISchemaIfCond,
+                        features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]) -> None:
+        self._gen.add(gen_rs_enum(name, ifcond))
+
+    def visit_alternate_type(self,
+                             name: str,
+                             info: QAPISourceInfo,
+                             ifcond: QAPISchemaIfCond,
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants) -> None:
+        self._gen.add(gen_rs_alternate(name, ifcond, variants))
+
+
+def gen_rs_types(schema: QAPISchema, output_dir: str, prefix: str) -> None:
+    vis = QAPISchemaGenRsTypeVisitor(prefix)
+    schema.visit(vis)
+    vis.write(output_dir)
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 21/32] tests/rust: build a common library, checking bindings compile
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (19 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 20/32] scripts/qapi: generate high-level Rust bindings marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 22/32] qga: build qapi-cabi binary (ABI from C) marcandre.lureau
                   ` (11 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Meson doesn't integrate very smoothly with Cargo. Use the cargo-wrapper
script as a custom_target() always stale to build the Rust code. The
"build-lib" command will produce a static library in the meson expected
output directory, as well as link flags that must be employed to do the
final link.

Those link flags can't be queried during configure time (Cargo doesn't
have a user-queriable configure step), so we pass them to the linker
thanks to @file argument support at build time.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/Cargo.toml  |  4 ++++
 tests/lib.rs      |  2 ++
 tests/meson.build | 20 +++++++++++++++++++-
 tests/qapi.rs     | 11 +++++++++++
 4 files changed, 36 insertions(+), 1 deletion(-)
 create mode 100644 tests/lib.rs
 create mode 100644 tests/qapi.rs

diff --git a/tests/Cargo.toml b/tests/Cargo.toml
index 7a4f6060b1..8a014dff89 100644
--- a/tests/Cargo.toml
+++ b/tests/Cargo.toml
@@ -7,6 +7,10 @@ publish = false
 [dependencies]
 common = { path = "../rust/common" }
 
+[lib]
+path = "lib.rs"
+crate-type = ["staticlib"]
+
 [[bin]]
 name = "qapi-cabi-rs"
 path = "qapi-cabi.rs"
diff --git a/tests/lib.rs b/tests/lib.rs
new file mode 100644
index 0000000000..e6fdf60a55
--- /dev/null
+++ b/tests/lib.rs
@@ -0,0 +1,2 @@
+mod qapi_ffi;
+mod qapi;
diff --git a/tests/meson.build b/tests/meson.build
index f9af42caba..09aa2bdf55 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -45,7 +45,10 @@ test_qapi_files = custom_target('Test QAPI files',
                                 depend_files: qapi_gen_depends)
 
 if with_rust
-  test_qapi_rs_outputs = ['test-qapi-ffi-types.rs']
+  test_qapi_rs_outputs = [
+    'test-qapi-ffi-types.rs',
+    'test-qapi-types.rs',
+  ]
   test_qapi_rs = custom_target('Test QAPI Rust binding',
                                output: test_qapi_rs_outputs,
                                input: test_qapi_inputs,
@@ -65,6 +68,21 @@ if with_rust
                                          'build-bin',
                                          'qapi-cabi-rs',
                                          '--', '--cfg', 'QAPI_CABI'])
+  libtest_rs = custom_target('Test Rust library',
+                             build_by_default: true,
+                             output: ['libqemu_tests.args', 'libqemu_tests.a'],
+                             build_always_stale: true,
+                             depends: [test_qapi_rs],
+                             command: [cargo_wrapper,
+                                       meson.current_build_dir(),
+                                       meson.current_source_dir(),
+                                       meson.build_root(),
+                                       rs_build_type,
+                                       rust_target_triple,
+                                       'build-lib'])
+  libtest_rs = declare_dependency(
+    link_args: '@' + libtest_rs[0].full_path(),
+    sources: libtest_rs)
 endif
 
 # meson doesn't like generated output in other directories
diff --git a/tests/qapi.rs b/tests/qapi.rs
new file mode 100644
index 0000000000..93e3e714e7
--- /dev/null
+++ b/tests/qapi.rs
@@ -0,0 +1,11 @@
+#![allow(dead_code)]
+#![allow(non_camel_case_types)]
+
+use common::*;
+
+new_ptr!();
+
+include!(concat!(
+    env!("MESON_BUILD_ROOT"),
+    "/tests/test-qapi-types.rs"
+));
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 22/32] qga: build qapi-cabi binary (ABI from C)
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (20 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 21/32] tests/rust: build a common library, checking bindings compile marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 23/32] qga/rust: build and link an empty static library marcandre.lureau
                   ` (10 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Build a binary to dump the QAPI ABI (from C code). Ex:

$ qga/qapi-cabi
GuestAgentCommandInfo struct: sizeof=16
 name member: sizeof=8 offset=0
 enabled member: sizeof=1 offset=8
 success_response member: sizeof=1 offset=9
...

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 qga/qapi-cabi.c |  9 +++++++++
 qga/meson.build | 12 ++++++++++++
 2 files changed, 21 insertions(+)
 create mode 100644 qga/qapi-cabi.c

diff --git a/qga/qapi-cabi.c b/qga/qapi-cabi.c
new file mode 100644
index 0000000000..0704e70869
--- /dev/null
+++ b/qga/qapi-cabi.c
@@ -0,0 +1,9 @@
+#include "qemu/osdep.h"
+
+#include "qga-qapi-types.h"
+
+int main(int argc, const char *argv[])
+{
+    qapi_cabi();
+    return 0;
+}
diff --git a/qga/meson.build b/qga/meson.build
index cfb1fbc085..1b050d8c53 100644
--- a/qga/meson.build
+++ b/qga/meson.build
@@ -21,6 +21,18 @@ qga_qapi_files = custom_target('QGA QAPI files',
                                command: [ qapi_gen, '-o', 'qga', '-p', 'qga-', '@INPUT0@' ],
                                depend_files: qapi_gen_depends)
 
+i = 0
+srcs = [files('qapi-cabi.c')]
+foreach output: qga_qapi_outputs
+  if output.startswith('qga-qapi-types') or output.startswith('qga-qapi-visit')
+    srcs += qga_qapi_files[i]
+  endif
+  i += 1
+endforeach
+qga_qapi_cabi = executable('qapi-cabi', srcs,
+                           dependencies: [qemuutil],
+                           c_args: ['-DQAPI_CABI'])
+
 qga_ss = ss.source_set()
 qga_ss.add(qga_qapi_files.to_list())
 qga_ss.add(files(
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 23/32] qga/rust: build and link an empty static library
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (21 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 22/32] qga: build qapi-cabi binary (ABI from C) marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 24/32] qga/rust: generate QGA QAPI types FFI bindings marcandre.lureau
                   ` (9 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 Cargo.lock      |  4 ++++
 Cargo.toml      |  1 +
 qga/Cargo.toml  |  9 +++++++++
 qga/lib.rs      |  0
 qga/meson.build | 20 +++++++++++++++++++-
 5 files changed, 33 insertions(+), 1 deletion(-)
 create mode 100644 qga/Cargo.toml
 create mode 100644 qga/lib.rs

diff --git a/Cargo.lock b/Cargo.lock
index f2cd0ee96e..2f5b202605 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -68,3 +68,7 @@ version = "0.0.1"
 dependencies = [
  "common",
 ]
+
+[[package]]
+name = "qga"
+version = "0.1.0"
diff --git a/Cargo.toml b/Cargo.toml
index f4a078e62d..3db0646413 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,5 +1,6 @@
 [workspace]
 members = [
+  "qga",
   "rust/common",
   "tests",
 ]
diff --git a/qga/Cargo.toml b/qga/Cargo.toml
new file mode 100644
index 0000000000..a3bffd1fec
--- /dev/null
+++ b/qga/Cargo.toml
@@ -0,0 +1,9 @@
+[package]
+name = "qga"
+version = "0.1.0"
+edition = "2018"
+publish = false
+
+[lib]
+path = "lib.rs"
+crate-type = ["staticlib"]
diff --git a/qga/lib.rs b/qga/lib.rs
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/qga/meson.build b/qga/meson.build
index 1b050d8c53..d272892ced 100644
--- a/qga/meson.build
+++ b/qga/meson.build
@@ -54,9 +54,27 @@ qga_ss.add(when: 'CONFIG_WIN32', if_true: files(
 
 qga_ss = qga_ss.apply(config_host, strict: false)
 
+qga_rs = declare_dependency()
+if with_rust
+  cargo_qga = custom_target('cargo-qga',
+                            build_by_default: true,
+                            output: ['libqga.args', 'libqga.a'],
+                            build_always_stale: true,
+                            command: [cargo_wrapper,
+                                      meson.current_build_dir(),
+                                      meson.current_source_dir(),
+                                      meson.build_root(),
+                                      rs_build_type,
+                                      rust_target_triple,
+                                      'build-lib'])
+  qga_rs = declare_dependency(
+    link_args: '@' + cargo_qga[0].full_path(),
+    sources: cargo_qga)
+endif
+
 qga = executable('qemu-ga', qga_ss.sources(),
                  link_args: config_host['LIBS_QGA'].split(),
-                 dependencies: [qemuutil, libudev],
+                 dependencies: [qemuutil, libudev, qga_rs],
                  install: true)
 all_qga = [qga]
 
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 24/32] qga/rust: generate QGA QAPI types FFI bindings
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (22 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 23/32] qga/rust: build and link an empty static library marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 25/32] qga/rust: build a qga-cabi-rs executable (ABI from Rust) marcandre.lureau
                   ` (8 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Use qapi-gen to generate low-level C FFI bindings for QAPI types,
include them to the build.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 Cargo.lock      |  3 +++
 qga/Cargo.toml  |  3 +++
 qga/lib.rs      |  1 +
 qga/meson.build | 11 +++++++++++
 qga/qapi_ffi.rs |  8 ++++++++
 5 files changed, 26 insertions(+)
 create mode 100644 qga/qapi_ffi.rs

diff --git a/Cargo.lock b/Cargo.lock
index 2f5b202605..ad5bb47762 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -72,3 +72,6 @@ dependencies = [
 [[package]]
 name = "qga"
 version = "0.1.0"
+dependencies = [
+ "common",
+]
diff --git a/qga/Cargo.toml b/qga/Cargo.toml
index a3bffd1fec..807acc947e 100644
--- a/qga/Cargo.toml
+++ b/qga/Cargo.toml
@@ -4,6 +4,9 @@ version = "0.1.0"
 edition = "2018"
 publish = false
 
+[dependencies]
+common = { path = "../rust/common" }
+
 [lib]
 path = "lib.rs"
 crate-type = ["staticlib"]
diff --git a/qga/lib.rs b/qga/lib.rs
index e69de29bb2..17cf43a5e9 100644
--- a/qga/lib.rs
+++ b/qga/lib.rs
@@ -0,0 +1 @@
+mod qapi_ffi;
diff --git a/qga/meson.build b/qga/meson.build
index d272892ced..88d59bef0c 100644
--- a/qga/meson.build
+++ b/qga/meson.build
@@ -56,10 +56,21 @@ qga_ss = qga_ss.apply(config_host, strict: false)
 
 qga_rs = declare_dependency()
 if with_rust
+  qga_qapi_rs_outputs = [
+    'qga-qapi-ffi-types.rs',
+  ]
+
+  qapi_gen_rs_files = custom_target('QGA QAPI Rust bindings',
+                                    output: qga_qapi_rs_outputs,
+                                    input: 'qapi-schema.json',
+                                    command: [ qapi_gen, '-r', '-o', 'qga', '-p', 'qga-', '@INPUT0@' ],
+                                    depend_files: qapi_gen_depends)
+
   cargo_qga = custom_target('cargo-qga',
                             build_by_default: true,
                             output: ['libqga.args', 'libqga.a'],
                             build_always_stale: true,
+                            depends: [qapi_gen_rs_files],
                             command: [cargo_wrapper,
                                       meson.current_build_dir(),
                                       meson.current_source_dir(),
diff --git a/qga/qapi_ffi.rs b/qga/qapi_ffi.rs
new file mode 100644
index 0000000000..1f4ade9318
--- /dev/null
+++ b/qga/qapi_ffi.rs
@@ -0,0 +1,8 @@
+#![allow(dead_code)]
+
+use common::libc;
+
+include!(concat!(
+    env!("MESON_BUILD_ROOT"),
+    "/qga/qga-qapi-ffi-types.rs"
+));
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 25/32] qga/rust: build a qga-cabi-rs executable (ABI from Rust)
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (23 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 24/32] qga/rust: generate QGA QAPI types FFI bindings marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 26/32] qga/rust: check the Rust C binding marcandre.lureau
                   ` (7 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 qga/Cargo.toml   |  5 +++++
 qga/meson.build  | 15 +++++++++++++++
 qga/qapi-cabi.rs |  6 ++++++
 3 files changed, 26 insertions(+)
 create mode 100644 qga/qapi-cabi.rs

diff --git a/qga/Cargo.toml b/qga/Cargo.toml
index 807acc947e..d262b847fa 100644
--- a/qga/Cargo.toml
+++ b/qga/Cargo.toml
@@ -10,3 +10,8 @@ common = { path = "../rust/common" }
 [lib]
 path = "lib.rs"
 crate-type = ["staticlib"]
+
+[[bin]]
+name = "qapi-cabi-rs"
+path = "qapi-cabi.rs"
+doc = false
diff --git a/qga/meson.build b/qga/meson.build
index 88d59bef0c..fd840f5c73 100644
--- a/qga/meson.build
+++ b/qga/meson.build
@@ -81,6 +81,21 @@ if with_rust
   qga_rs = declare_dependency(
     link_args: '@' + cargo_qga[0].full_path(),
     sources: cargo_qga)
+
+  cargo_qga_cabi = custom_target('qga-qapi-cabi-rs',
+                                 output: ['qapi-cabi-rs' + exe_suffix],
+                                 build_always_stale: true,
+                                 depends: [qapi_gen_rs_files],
+                                 command: [cargo_wrapper,
+                                           meson.current_build_dir(),
+                                           meson.current_source_dir(),
+                                           meson.build_root(),
+                                           rs_build_type,
+                                           rust_target_triple,
+                                           'build-bin',
+                                           'qapi-cabi-rs',
+                                           '--', '--cfg', 'QAPI_CABI'])
+
 endif
 
 qga = executable('qemu-ga', qga_ss.sources(),
diff --git a/qga/qapi-cabi.rs b/qga/qapi-cabi.rs
new file mode 100644
index 0000000000..d98a819ef7
--- /dev/null
+++ b/qga/qapi-cabi.rs
@@ -0,0 +1,6 @@
+pub use common::{err, Error, Result};
+mod qapi_ffi;
+
+fn main() {
+    qapi_ffi::cabi()
+}
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 26/32] qga/rust: check the Rust C binding
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (24 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 25/32] qga/rust: build a qga-cabi-rs executable (ABI from Rust) marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 27/32] qga/rust: build high-level Rust QAPI types marcandre.lureau
                   ` (6 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Compare the matching C and the Rust C ABI FFI dumps.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 qga/meson.build | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/qga/meson.build b/qga/meson.build
index fd840f5c73..f2064c1874 100644
--- a/qga/meson.build
+++ b/qga/meson.build
@@ -96,6 +96,11 @@ if with_rust
                                            'qapi-cabi-rs',
                                            '--', '--cfg', 'QAPI_CABI'])
 
+  test('QGA CABI',
+       diff_commands,
+       args: [qga_qapi_cabi.full_path(), cargo_qga_cabi.full_path()],
+       depends: [qga_qapi_cabi, cargo_qga_cabi],
+       suite: ['qga'])
 endif
 
 qga = executable('qemu-ga', qga_ss.sources(),
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 27/32] qga/rust: build high-level Rust QAPI types
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (25 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 26/32] qga/rust: check the Rust C binding marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 28/32] qga/rust: implement get-host-name in Rust (example) marcandre.lureau
                   ` (5 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 qga/lib.rs      | 1 +
 qga/meson.build | 1 +
 qga/qapi.rs     | 6 ++++++
 3 files changed, 8 insertions(+)
 create mode 100644 qga/qapi.rs

diff --git a/qga/lib.rs b/qga/lib.rs
index 17cf43a5e9..7f62788ff6 100644
--- a/qga/lib.rs
+++ b/qga/lib.rs
@@ -1 +1,2 @@
+mod qapi;
 mod qapi_ffi;
diff --git a/qga/meson.build b/qga/meson.build
index f2064c1874..16775bc905 100644
--- a/qga/meson.build
+++ b/qga/meson.build
@@ -58,6 +58,7 @@ qga_rs = declare_dependency()
 if with_rust
   qga_qapi_rs_outputs = [
     'qga-qapi-ffi-types.rs',
+    'qga-qapi-types.rs',
   ]
 
   qapi_gen_rs_files = custom_target('QGA QAPI Rust bindings',
diff --git a/qga/qapi.rs b/qga/qapi.rs
new file mode 100644
index 0000000000..e4b9113300
--- /dev/null
+++ b/qga/qapi.rs
@@ -0,0 +1,6 @@
+#![allow(dead_code)]
+use common::*;
+
+new_ptr!();
+
+include!(concat!(env!("MESON_BUILD_ROOT"), "/qga/qga-qapi-types.rs"));
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 28/32] qga/rust: implement get-host-name in Rust (example)
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (26 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 27/32] qga/rust: build high-level Rust QAPI types marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 29/32] qga/rust: implement {get,set}-vcpus " marcandre.lureau
                   ` (4 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Use the "hostname" crate (https://github.com/svartalf/hostname)

(notice the wrong error message in our win32 implementation)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 include/qemu/osdep.h  | 10 ----------
 qga/commands.c        | 20 ++++----------------
 tests/unit/test-qga.c |  2 ++
 util/oslib-posix.c    | 35 -----------------------------------
 util/oslib-win32.c    | 13 -------------
 Cargo.lock            | 40 ++++++++++++++++++++++++++++++++++++++++
 qga/Cargo.toml        |  1 +
 qga/lib.rs            |  3 +++
 qga/qmp/hostname.rs   |  9 +++++++++
 qga/qmp/mod.rs        | 10 ++++++++++
 10 files changed, 69 insertions(+), 74 deletions(-)
 create mode 100644 qga/qmp/hostname.rs
 create mode 100644 qga/qmp/mod.rs

diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index 60718fc342..ca8f3465d2 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -740,16 +740,6 @@ static inline void qemu_reset_optind(void)
 #endif
 }
 
-/**
- * qemu_get_host_name:
- * @errp: Error object
- *
- * Operating system agnostic way of querying host name.
- *
- * Returns allocated hostname (caller should free), NULL on failure.
- */
-char *qemu_get_host_name(Error **errp);
-
 /**
  * qemu_get_host_physmem:
  *
diff --git a/qga/commands.c b/qga/commands.c
index 80501e4a73..117c219ac4 100644
--- a/qga/commands.c
+++ b/qga/commands.c
@@ -508,25 +508,13 @@ int ga_parse_whence(GuestFileWhence *whence, Error **errp)
     return -1;
 }
 
+#ifndef CONFIG_WITH_RUST
 GuestHostName *qmp_guest_get_host_name(Error **errp)
 {
-    GuestHostName *result = NULL;
-    g_autofree char *hostname = qemu_get_host_name(errp);
-
-    /*
-     * We want to avoid using g_get_host_name() because that
-     * caches the result and we wouldn't reflect changes in the
-     * host name.
-     */
-
-    if (!hostname) {
-        hostname = g_strdup("localhost");
-    }
-
-    result = g_new0(GuestHostName, 1);
-    result->host_name = g_steal_pointer(&hostname);
-    return result;
+    error_setg(errp, QERR_UNSUPPORTED);
+    return NULL;
 }
+#endif
 
 GuestTimezone *qmp_guest_get_timezone(Error **errp)
 {
diff --git a/tests/unit/test-qga.c b/tests/unit/test-qga.c
index 5cb140d1b5..a22ab8c82c 100644
--- a/tests/unit/test-qga.c
+++ b/tests/unit/test-qga.c
@@ -864,6 +864,7 @@ static void test_qga_guest_exec_invalid(gconstpointer fix)
 
 static void test_qga_guest_get_host_name(gconstpointer fix)
 {
+#ifdef CONFIG_WITH_RUST
     const TestFixture *fixture = fix;
     QDict *ret, *val;
 
@@ -875,6 +876,7 @@ static void test_qga_guest_get_host_name(gconstpointer fix)
     g_assert(qdict_haskey(val, "host-name"));
 
     qobject_unref(ret);
+#endif
 }
 
 static void test_qga_guest_get_timezone(gconstpointer fix)
diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index e8bdb02e1d..1498e0b7f5 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -809,41 +809,6 @@ void sigaction_invoke(struct sigaction *action,
     action->sa_sigaction(info->ssi_signo, &si, NULL);
 }
 
-#ifndef HOST_NAME_MAX
-# ifdef _POSIX_HOST_NAME_MAX
-#  define HOST_NAME_MAX _POSIX_HOST_NAME_MAX
-# else
-#  define HOST_NAME_MAX 255
-# endif
-#endif
-
-char *qemu_get_host_name(Error **errp)
-{
-    long len = -1;
-    g_autofree char *hostname = NULL;
-
-#ifdef _SC_HOST_NAME_MAX
-    len = sysconf(_SC_HOST_NAME_MAX);
-#endif /* _SC_HOST_NAME_MAX */
-
-    if (len < 0) {
-        len = HOST_NAME_MAX;
-    }
-
-    /* Unfortunately, gethostname() below does not guarantee a
-     * NULL terminated string. Therefore, allocate one byte more
-     * to be sure. */
-    hostname = g_new0(char, len + 1);
-
-    if (gethostname(hostname, len) < 0) {
-        error_setg_errno(errp, errno,
-                         "cannot get hostname");
-        return NULL;
-    }
-
-    return g_steal_pointer(&hostname);
-}
-
 size_t qemu_get_host_physmem(void)
 {
 #ifdef _SC_PHYS_PAGES
diff --git a/util/oslib-win32.c b/util/oslib-win32.c
index af559ef339..575f697815 100644
--- a/util/oslib-win32.c
+++ b/util/oslib-win32.c
@@ -629,19 +629,6 @@ bool qemu_write_pidfile(const char *filename, Error **errp)
     return true;
 }
 
-char *qemu_get_host_name(Error **errp)
-{
-    wchar_t tmp[MAX_COMPUTERNAME_LENGTH + 1];
-    DWORD size = G_N_ELEMENTS(tmp);
-
-    if (GetComputerNameW(tmp, &size) == 0) {
-        error_setg_win32(errp, GetLastError(), "failed close handle");
-        return NULL;
-    }
-
-    return g_utf16_to_utf8(tmp, size, NULL, NULL, NULL);
-}
-
 size_t qemu_get_host_physmem(void)
 {
     MEMORYSTATUSEX statex;
diff --git a/Cargo.lock b/Cargo.lock
index ad5bb47762..8752dbf2ac 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -34,12 +34,29 @@ dependencies = [
  "nix",
 ]
 
+[[package]]
+name = "hostname"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3c731c3e10504cc8ed35cfe2f1db4c9274c3d35fa486e3b31df46f068ef3e867"
+dependencies = [
+ "libc",
+ "match_cfg",
+ "winapi",
+]
+
 [[package]]
 name = "libc"
 version = "0.2.101"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "3cb00336871be5ed2c8ed44b60ae9959dc5b9f08539422ed43f09e34ecaeba21"
 
+[[package]]
+name = "match_cfg"
+version = "0.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ffbee8634e0d45d258acb448e7eaab3fce7a0a467395d4d9f228e3c1f01fb2e4"
+
 [[package]]
 name = "memoffset"
 version = "0.6.4"
@@ -74,4 +91,27 @@ name = "qga"
 version = "0.1.0"
 dependencies = [
  "common",
+ "hostname",
 ]
+
+[[package]]
+name = "winapi"
+version = "0.3.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
+dependencies = [
+ "winapi-i686-pc-windows-gnu",
+ "winapi-x86_64-pc-windows-gnu",
+]
+
+[[package]]
+name = "winapi-i686-pc-windows-gnu"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
+
+[[package]]
+name = "winapi-x86_64-pc-windows-gnu"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
diff --git a/qga/Cargo.toml b/qga/Cargo.toml
index d262b847fa..387af59a30 100644
--- a/qga/Cargo.toml
+++ b/qga/Cargo.toml
@@ -6,6 +6,7 @@ publish = false
 
 [dependencies]
 common = { path = "../rust/common" }
+hostname = "0.3.1"
 
 [lib]
 path = "lib.rs"
diff --git a/qga/lib.rs b/qga/lib.rs
index 7f62788ff6..e51e211006 100644
--- a/qga/lib.rs
+++ b/qga/lib.rs
@@ -1,2 +1,5 @@
+pub use common::{err, libc, Error, Result};
+
 mod qapi;
 mod qapi_ffi;
+mod qmp;
diff --git a/qga/qmp/hostname.rs b/qga/qmp/hostname.rs
new file mode 100644
index 0000000000..c3eb1f6fd2
--- /dev/null
+++ b/qga/qmp/hostname.rs
@@ -0,0 +1,9 @@
+use crate::*;
+
+pub(crate) fn get() -> Result<qapi::GuestHostName> {
+    let host_name = hostname::get()?
+        .into_string()
+        .or_else(|_| err!("Invalid hostname"))?;
+
+    Ok(qapi::GuestHostName { host_name })
+}
diff --git a/qga/qmp/mod.rs b/qga/qmp/mod.rs
new file mode 100644
index 0000000000..c192e4247d
--- /dev/null
+++ b/qga/qmp/mod.rs
@@ -0,0 +1,10 @@
+use common::*;
+
+use crate::qapi_ffi;
+
+mod hostname;
+
+#[no_mangle]
+extern "C" fn qmp_guest_get_host_name(errp: *mut *mut ffi::Error) -> *mut qapi_ffi::GuestHostName {
+    qmp!(hostname::get(), errp)
+}
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 29/32] qga/rust: implement {get,set}-vcpus in Rust (example)
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (27 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 28/32] qga/rust: implement get-host-name in Rust (example) marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 30/32] tests/vm: add Rust to FreeBSD VM marcandre.lureau
                   ` (3 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

This is a rewrite of the C version (using the nix & winapi crates).

The main difference is that Rust doesn't let you mix const/mut logic,
the way transfer_vcpu in C does. The Rust version does introduce some
duplication, but is also more strict and can prevent mistakes.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 qga/commands-posix.c  | 152 ---------------------------------------
 qga/commands-win32.c  |  71 -------------------
 qga/commands.c        |  14 ++++
 tests/unit/test-qga.c |   2 +
 Cargo.lock            |   2 +
 qga/Cargo.toml        |   6 ++
 qga/qmp/mod.rs        |  20 +++++-
 qga/qmp/vcpus.rs      | 161 ++++++++++++++++++++++++++++++++++++++++++
 8 files changed, 204 insertions(+), 224 deletions(-)
 create mode 100644 qga/qmp/vcpus.rs

diff --git a/qga/commands-posix.c b/qga/commands-posix.c
index 75dbaab68e..97ca70e762 100644
--- a/qga/commands-posix.c
+++ b/qga/commands-posix.c
@@ -2371,146 +2371,6 @@ error:
     return NULL;
 }
 
-/* Transfer online/offline status between @vcpu and the guest system.
- *
- * On input either @errp or *@errp must be NULL.
- *
- * In system-to-@vcpu direction, the following @vcpu fields are accessed:
- * - R: vcpu->logical_id
- * - W: vcpu->online
- * - W: vcpu->can_offline
- *
- * In @vcpu-to-system direction, the following @vcpu fields are accessed:
- * - R: vcpu->logical_id
- * - R: vcpu->online
- *
- * Written members remain unmodified on error.
- */
-static void transfer_vcpu(GuestLogicalProcessor *vcpu, bool sys2vcpu,
-                          char *dirpath, Error **errp)
-{
-    int fd;
-    int res;
-    int dirfd;
-    static const char fn[] = "online";
-
-    dirfd = open(dirpath, O_RDONLY | O_DIRECTORY);
-    if (dirfd == -1) {
-        error_setg_errno(errp, errno, "open(\"%s\")", dirpath);
-        return;
-    }
-
-    fd = openat(dirfd, fn, sys2vcpu ? O_RDONLY : O_RDWR);
-    if (fd == -1) {
-        if (errno != ENOENT) {
-            error_setg_errno(errp, errno, "open(\"%s/%s\")", dirpath, fn);
-        } else if (sys2vcpu) {
-            vcpu->online = true;
-            vcpu->can_offline = false;
-        } else if (!vcpu->online) {
-            error_setg(errp, "logical processor #%" PRId64 " can't be "
-                       "offlined", vcpu->logical_id);
-        } /* otherwise pretend successful re-onlining */
-    } else {
-        unsigned char status;
-
-        res = pread(fd, &status, 1, 0);
-        if (res == -1) {
-            error_setg_errno(errp, errno, "pread(\"%s/%s\")", dirpath, fn);
-        } else if (res == 0) {
-            error_setg(errp, "pread(\"%s/%s\"): unexpected EOF", dirpath,
-                       fn);
-        } else if (sys2vcpu) {
-            vcpu->online = (status != '0');
-            vcpu->can_offline = true;
-        } else if (vcpu->online != (status != '0')) {
-            status = '0' + vcpu->online;
-            if (pwrite(fd, &status, 1, 0) == -1) {
-                error_setg_errno(errp, errno, "pwrite(\"%s/%s\")", dirpath,
-                                 fn);
-            }
-        } /* otherwise pretend successful re-(on|off)-lining */
-
-        res = close(fd);
-        g_assert(res == 0);
-    }
-
-    res = close(dirfd);
-    g_assert(res == 0);
-}
-
-GuestLogicalProcessorList *qmp_guest_get_vcpus(Error **errp)
-{
-    GuestLogicalProcessorList *head, **tail;
-    const char *cpu_dir = "/sys/devices/system/cpu";
-    const gchar *line;
-    g_autoptr(GDir) cpu_gdir = NULL;
-    Error *local_err = NULL;
-
-    head = NULL;
-    tail = &head;
-    cpu_gdir = g_dir_open(cpu_dir, 0, NULL);
-
-    if (cpu_gdir == NULL) {
-        error_setg_errno(errp, errno, "failed to list entries: %s", cpu_dir);
-        return NULL;
-    }
-
-    while (local_err == NULL && (line = g_dir_read_name(cpu_gdir)) != NULL) {
-        GuestLogicalProcessor *vcpu;
-        int64_t id;
-        if (sscanf(line, "cpu%" PRId64, &id)) {
-            g_autofree char *path = g_strdup_printf("/sys/devices/system/cpu/"
-                                                    "cpu%" PRId64 "/", id);
-            vcpu = g_malloc0(sizeof *vcpu);
-            vcpu->logical_id = id;
-            vcpu->has_can_offline = true; /* lolspeak ftw */
-            transfer_vcpu(vcpu, true, path, &local_err);
-            QAPI_LIST_APPEND(tail, vcpu);
-        }
-    }
-
-    if (local_err == NULL) {
-        /* there's no guest with zero VCPUs */
-        g_assert(head != NULL);
-        return head;
-    }
-
-    qapi_free_GuestLogicalProcessorList(head);
-    error_propagate(errp, local_err);
-    return NULL;
-}
-
-int64_t qmp_guest_set_vcpus(GuestLogicalProcessorList *vcpus, Error **errp)
-{
-    int64_t processed;
-    Error *local_err = NULL;
-
-    processed = 0;
-    while (vcpus != NULL) {
-        char *path = g_strdup_printf("/sys/devices/system/cpu/cpu%" PRId64 "/",
-                                     vcpus->value->logical_id);
-
-        transfer_vcpu(vcpus->value, false, path, &local_err);
-        g_free(path);
-        if (local_err != NULL) {
-            break;
-        }
-        ++processed;
-        vcpus = vcpus->next;
-    }
-
-    if (local_err != NULL) {
-        if (processed == 0) {
-            error_propagate(errp, local_err);
-        } else {
-            error_free(local_err);
-        }
-    }
-
-    return processed;
-}
-
 void qmp_guest_set_user_password(const char *username,
                                  const char *password,
                                  bool crypted,
@@ -2946,18 +2806,6 @@ GuestNetworkInterfaceList *qmp_guest_network_get_interfaces(Error **errp)
     return NULL;
 }
 
-GuestLogicalProcessorList *qmp_guest_get_vcpus(Error **errp)
-{
-    error_setg(errp, QERR_UNSUPPORTED);
-    return NULL;
-}
-
-int64_t qmp_guest_set_vcpus(GuestLogicalProcessorList *vcpus, Error **errp)
-{
-    error_setg(errp, QERR_UNSUPPORTED);
-    return -1;
-}
-
 void qmp_guest_set_user_password(const char *username,
                                  const char *password,
                                  bool crypted,
diff --git a/qga/commands-win32.c b/qga/commands-win32.c
index 4e84afd83b..33a3fd7218 100644
--- a/qga/commands-win32.c
+++ b/qga/commands-win32.c
@@ -1816,77 +1816,6 @@ void qmp_guest_set_time(bool has_time, int64_t time_ns, Error **errp)
     }
 }
 
-GuestLogicalProcessorList *qmp_guest_get_vcpus(Error **errp)
-{
-    PSYSTEM_LOGICAL_PROCESSOR_INFORMATION pslpi, ptr;
-    DWORD length;
-    GuestLogicalProcessorList *head, **tail;
-    Error *local_err = NULL;
-    int64_t current;
-
-    ptr = pslpi = NULL;
-    length = 0;
-    current = 0;
-    head = NULL;
-    tail = &head;
-
-    if ((GetLogicalProcessorInformation(pslpi, &length) == FALSE) &&
-        (GetLastError() == ERROR_INSUFFICIENT_BUFFER) &&
-        (length > sizeof(SYSTEM_LOGICAL_PROCESSOR_INFORMATION))) {
-        ptr = pslpi = g_malloc0(length);
-        if (GetLogicalProcessorInformation(pslpi, &length) == FALSE) {
-            error_setg(&local_err, "Failed to get processor information: %d",
-                       (int)GetLastError());
-        }
-    } else {
-        error_setg(&local_err,
-                   "Failed to get processor information buffer length: %d",
-                   (int)GetLastError());
-    }
-
-    while ((local_err == NULL) && (length > 0)) {
-        if (pslpi->Relationship == RelationProcessorCore) {
-            ULONG_PTR cpu_bits = pslpi->ProcessorMask;
-
-            while (cpu_bits > 0) {
-                if (!!(cpu_bits & 1)) {
-                    GuestLogicalProcessor *vcpu;
-
-                    vcpu = g_malloc0(sizeof *vcpu);
-                    vcpu->logical_id = current++;
-                    vcpu->online = true;
-                    vcpu->has_can_offline = true;
-
-                    QAPI_LIST_APPEND(tail, vcpu);
-                }
-                cpu_bits >>= 1;
-            }
-        }
-        length -= sizeof(SYSTEM_LOGICAL_PROCESSOR_INFORMATION);
-        pslpi++; /* next entry */
-    }
-
-    g_free(ptr);
-
-    if (local_err == NULL) {
-        if (head != NULL) {
-            return head;
-        }
-        /* there's no guest with zero VCPUs */
-        error_setg(&local_err, "Guest reported zero VCPUs");
-    }
-
-    qapi_free_GuestLogicalProcessorList(head);
-    error_propagate(errp, local_err);
-    return NULL;
-}
-
-int64_t qmp_guest_set_vcpus(GuestLogicalProcessorList *vcpus, Error **errp)
-{
-    error_setg(errp, QERR_UNSUPPORTED);
-    return -1;
-}
-
 static gchar *
 get_net_error_message(gint error)
 {
diff --git a/qga/commands.c b/qga/commands.c
index 117c219ac4..d5ad904480 100644
--- a/qga/commands.c
+++ b/qga/commands.c
@@ -573,3 +573,17 @@ GuestFileRead *qmp_guest_file_read(int64_t handle, bool has_count,
 
     return read_data;
 }
+
+#ifndef CONFIG_WITH_RUST
+GuestLogicalProcessorList *qmp_guest_get_vcpus(Error **errp)
+{
+    error_setg(errp, QERR_UNSUPPORTED);
+    return NULL;
+}
+
+int64_t qmp_guest_set_vcpus(GuestLogicalProcessorList *vcpus, Error **errp)
+{
+    error_setg(errp, QERR_UNSUPPORTED);
+    return 0;
+}
+#endif
diff --git a/tests/unit/test-qga.c b/tests/unit/test-qga.c
index a22ab8c82c..47fd8eb717 100644
--- a/tests/unit/test-qga.c
+++ b/tests/unit/test-qga.c
@@ -308,6 +308,7 @@ static void test_qga_info(gconstpointer fix)
 
 static void test_qga_get_vcpus(gconstpointer fix)
 {
+#ifdef CONFIG_WITH_RUST
     const TestFixture *fixture = fix;
     QDict *ret;
     QList *list;
@@ -324,6 +325,7 @@ static void test_qga_get_vcpus(gconstpointer fix)
     g_assert(qdict_haskey(qobject_to(QDict, entry->value), "logical-id"));
 
     qobject_unref(ret);
+#endif
 }
 
 static void test_qga_get_fsinfo(gconstpointer fix)
diff --git a/Cargo.lock b/Cargo.lock
index 8752dbf2ac..f3d2631865 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -92,6 +92,8 @@ version = "0.1.0"
 dependencies = [
  "common",
  "hostname",
+ "nix",
+ "winapi",
 ]
 
 [[package]]
diff --git a/qga/Cargo.toml b/qga/Cargo.toml
index 387af59a30..0661b8b7c6 100644
--- a/qga/Cargo.toml
+++ b/qga/Cargo.toml
@@ -8,6 +8,12 @@ publish = false
 common = { path = "../rust/common" }
 hostname = "0.3.1"
 
+[target."cfg(unix)".dependencies]
+nix = "0.20.0"
+
+[target."cfg(windows)".dependencies]
+winapi = { version = "0.3.9", features = ["sysinfoapi", "winnt"] }
+
 [lib]
 path = "lib.rs"
 crate-type = ["staticlib"]
diff --git a/qga/qmp/mod.rs b/qga/qmp/mod.rs
index c192e4247d..5b4f17a07d 100644
--- a/qga/qmp/mod.rs
+++ b/qga/qmp/mod.rs
@@ -1,6 +1,6 @@
 use common::*;
 
-use crate::qapi_ffi;
+use crate::{qapi::NewPtr, qapi_ffi};
 
 mod hostname;
 
@@ -8,3 +8,21 @@ mod hostname;
 extern "C" fn qmp_guest_get_host_name(errp: *mut *mut ffi::Error) -> *mut qapi_ffi::GuestHostName {
     qmp!(hostname::get(), errp)
 }
+
+mod vcpus;
+
+#[no_mangle]
+extern "C" fn qmp_guest_get_vcpus(
+    errp: *mut *mut ffi::Error,
+) -> *mut qapi_ffi::GuestLogicalProcessorList {
+    qmp!(vcpus::get(), errp)
+}
+
+#[no_mangle]
+extern "C" fn qmp_guest_set_vcpus(
+    vcpus: *const qapi_ffi::GuestLogicalProcessorList,
+    errp: *mut *mut ffi::Error,
+) -> libc::c_longlong {
+    let vcpus = unsafe { from_qemu_none(NewPtr(vcpus)) };
+    qmp!(vcpus::set(vcpus), errp, -1)
+}
diff --git a/qga/qmp/vcpus.rs b/qga/qmp/vcpus.rs
new file mode 100644
index 0000000000..f86838355e
--- /dev/null
+++ b/qga/qmp/vcpus.rs
@@ -0,0 +1,161 @@
+#[cfg(unix)]
+use std::fs::OpenOptions;
+#[cfg(unix)]
+use std::io::ErrorKind;
+#[cfg(unix)]
+use std::os::unix::fs::FileExt;
+
+#[cfg(windows)]
+use winapi::um::{sysinfoapi, winnt};
+
+use crate::*;
+
+#[cfg(target_os = "linux")]
+fn get_sysfs_cpu_path(id: i64) -> String {
+    format!("/sys/devices/system/cpu/cpu{}", id)
+}
+
+#[cfg(target_os = "linux")]
+fn set_vcpu(vcpu: &qapi::GuestLogicalProcessor) -> Result<()> {
+    let path = get_sysfs_cpu_path(vcpu.logical_id);
+    std::fs::metadata(&path)?;
+
+    let path = format!("{}/online", path);
+    match OpenOptions::new().read(true).write(true).open(&path) {
+        Ok(file) => {
+            let mut buf = [0u8; 1];
+            file.read_exact_at(&mut buf, 0)?;
+            let online = buf[0] != 0;
+            if vcpu.online != online {
+                buf[0] = if vcpu.online { b'1' } else { b'0' };
+                file.write_all_at(&buf, 0)?;
+            }
+        }
+        Err(e) => {
+            if e.kind() != ErrorKind::NotFound {
+                return Err(e.into());
+            } else if !vcpu.online {
+                return err!(format!(
+                    "logical processor #{} can't be offlined",
+                    vcpu.logical_id
+                ));
+            }
+        }
+    }
+
+    Ok(())
+}
+
+#[cfg(not(target_os = "linux"))]
+fn set_vcpu(_vcpu: &qapi::GuestLogicalProcessor) -> Result<()> {
+    err!("unimplemented")
+}
+
+pub(crate) fn set(vcpus: Vec<qapi::GuestLogicalProcessor>) -> Result<i64> {
+    let mut processed = 0;
+
+    for vcpu in &vcpus {
+        if let Err(e) = set_vcpu(vcpu) {
+            if processed != 0 {
+                break;
+            }
+            return Err(e);
+        }
+
+        processed += 1;
+    }
+
+    Ok(processed)
+}
+
+#[cfg(target_os = "linux")]
+pub(crate) fn get() -> Result<Vec<qapi::GuestLogicalProcessor>> {
+    use nix::unistd::sysconf;
+
+    let mut vcpus = vec![];
+    let nproc_conf = match sysconf(unsafe { std::mem::transmute(libc::_SC_NPROCESSORS_CONF) })? {
+        Some(nproc) => nproc,
+        None => {
+            return err!("Indefinite number of processors.");
+        }
+    };
+
+    for logical_id in 0..nproc_conf {
+        let path = get_sysfs_cpu_path(logical_id);
+        if std::fs::metadata(&path).is_err() {
+            continue;
+        }
+
+        let path = format!("{}/online", path);
+        let (online, can_offline) = match OpenOptions::new().read(true).open(&path) {
+            Ok(file) => {
+                let mut buf = [0u8; 1];
+                file.read_exact_at(&mut buf, 0)?;
+                (buf[0] != 0, Some(true))
+            }
+            Err(e) => {
+                if e.kind() != ErrorKind::NotFound {
+                    return Err(e.into());
+                }
+                (true, Some(false))
+            }
+        };
+
+        vcpus.push(qapi::GuestLogicalProcessor {
+            logical_id,
+            online,
+            can_offline,
+        });
+    }
+
+    Ok(vcpus)
+}
+
+#[cfg(target_os = "windows")]
+fn get_logical_processor_info() -> Result<Vec<winnt::SYSTEM_LOGICAL_PROCESSOR_INFORMATION>> {
+    unsafe {
+        let mut needed_size = 0;
+        sysinfoapi::GetLogicalProcessorInformation(std::ptr::null_mut(), &mut needed_size);
+        let struct_size = std::mem::size_of::<winnt::SYSTEM_LOGICAL_PROCESSOR_INFORMATION>() as u32;
+        if needed_size == 0 || needed_size < struct_size || needed_size % struct_size != 0 {
+            return err!("Failed to get processor information");
+        }
+
+        let nstruct = needed_size / struct_size;
+        let mut buf = Vec::with_capacity(nstruct as usize);
+        let result = sysinfoapi::GetLogicalProcessorInformation(buf.as_mut_ptr(), &mut needed_size);
+        if result == 0 {
+            return err!("Failed to get processor information");
+        }
+
+        let nstruct = needed_size / struct_size;
+        buf.set_len(nstruct as usize);
+        Ok(buf)
+    }
+}
+
+#[cfg(target_os = "windows")]
+pub(crate) fn get() -> Result<Vec<qapi::GuestLogicalProcessor>> {
+    let mut vcpus = vec![];
+
+    get_logical_processor_info()?.iter().map(|info| {
+        for _ in 0..info.ProcessorMask.count_ones() {
+            vcpus.push(qapi::GuestLogicalProcessor {
+                logical_id: vcpus.len() as i64,
+                online: true,
+                can_offline: Some(false),
+            });
+        }
+    });
+
+    if vcpus.is_empty() {
+        return err!("Guest reported zero VCPUs");
+    }
+
+    Ok(vcpus)
+}
+
+#[cfg(not(any(target_os = "linux", target_os = "windows")))]
+pub(crate) fn get() -> Result<Vec<qapi::GuestLogicalProcessor>> {
+    err!("unimplemented")
+}
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 30/32] tests/vm: add Rust to FreeBSD VM
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (28 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 29/32] qga/rust: implement {get,set}-vcpus " marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 31/32] tests/vm: bump fedora VM to f32 marcandre.lureau
                   ` (2 subsequent siblings)
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Test Rust support on FreeBSD.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/vm/freebsd | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/vm/freebsd b/tests/vm/freebsd
index 6e20e84322..62c7adbe3c 100755
--- a/tests/vm/freebsd
+++ b/tests/vm/freebsd
@@ -38,6 +38,7 @@ class FreeBSDVM(basevm.BaseVM):
         "bzip2",
         "python37",
         "ninja",
+        "rust",
 
         # gnu tools
         "bash",
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 31/32] tests/vm: bump fedora VM to f32
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (29 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 30/32] tests/vm: add Rust to FreeBSD VM marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-07 12:19 ` [RFC v3 32/32] tests/vm: add Rust to Fedora marcandre.lureau
  2021-09-08 13:22 ` [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples Markus Armbruster
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

The Rust version shipped in Fedora 30 is < 1.36, and would not let the
build work with --offline. The one in Fedora 31 has some issues with
--offline (https://github.com/rust-lang/cargo/issues/7582)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/vm/fedora | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/tests/vm/fedora b/tests/vm/fedora
index b977efe4a2..03abc8ebe3 100755
--- a/tests/vm/fedora
+++ b/tests/vm/fedora
@@ -23,11 +23,11 @@ class FedoraVM(basevm.BaseVM):
     name = "fedora"
     arch = "x86_64"
 
-    base = "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/"
-    link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
+    base = "https://archives.fedoraproject.org/pub/fedora/linux/releases/32/"
+    link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-32-1.6.iso"
     repo = base + "Server/x86_64/os/"
     full = base + "Everything/x86_64/os/"
-    csum = "5e4eac4566d8c572bfb3bcf54b7d6c82006ec3c6c882a2c9235c6d3494d7b100"
+    csum = "7f4afd2a26c718f9f15e4bbfd9c2e8849f81036d2a82a4e81fa4a313a833da9c"
     size = "20G"
     pkgs = [
         # tools
@@ -106,7 +106,7 @@ class FedoraVM(basevm.BaseVM):
         self.console_wait_send("1) [ ] Standard Part",     "1\n")
         self.console_wait_send("1) [x] Standard Part",     "c\n")
 
-        self.console_wait_send("7) [!] Root password",     "7\n")
+        self.console_wait_send("7) [x] Root password",     "7\n")
         self.console_wait("Password:")
         self.console_send("%s\n" % self._config["root_pass"])
         self.console_wait("Password (confirm):")
@@ -116,7 +116,6 @@ class FedoraVM(basevm.BaseVM):
         self.console_wait_send("1) [ ] Create user",       "1\n")
         self.console_wait_send("3) User name",             "3\n")
         self.console_wait_send("ENTER:", "%s\n" % self._config["guest_user"])
-        self.console_wait_send("4) [ ] Use password",      "4\n")
         self.console_wait_send("5) Password",              "5\n")
         self.console_wait("Password:")
         self.console_send("%s\n" % self._config["guest_pass"])
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* [RFC v3 32/32] tests/vm: add Rust to Fedora
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (30 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 31/32] tests/vm: bump fedora VM to f32 marcandre.lureau
@ 2021-09-07 12:19 ` marcandre.lureau
  2021-09-08 13:22 ` [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples Markus Armbruster
  32 siblings, 0 replies; 74+ messages in thread
From: marcandre.lureau @ 2021-09-07 12:19 UTC (permalink / raw)
  To: qemu-devel; +Cc: pbonzini, berrange, armbru, stefanha, Marc-André Lureau

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Test the Rust support on Fedora.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/vm/fedora | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/vm/fedora b/tests/vm/fedora
index 03abc8ebe3..70347e3fa2 100755
--- a/tests/vm/fedora
+++ b/tests/vm/fedora
@@ -34,6 +34,9 @@ class FedoraVM(basevm.BaseVM):
         'git-core',
         'gcc', 'binutils', 'make', 'ninja-build',
 
+        # Rust
+        'cargo', 'rust',
+
         # perl
         'perl-Test-Harness',
 
-- 
2.33.0.113.g6c40894d24



^ permalink raw reply related	[flat|nested] 74+ messages in thread

* Re: [RFC v3 01/32] RFC: docs: add supported host CPUs section
  2021-09-07 12:19 ` [RFC v3 01/32] RFC: docs: add supported host CPUs section marcandre.lureau
@ 2021-09-07 12:33   ` Peter Maydell
  2021-09-13 11:32     ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Peter Maydell @ 2021-09-07 12:33 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU Developers,
	Stefan Hajnoczi, Markus Armbruster

On Tue, 7 Sept 2021 at 13:23, <marcandre.lureau@redhat.com> wrote:
>
> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> I was looking for such documentation, but couldn't find it.

Yes; this is definitely something we should document, and in
the build-platforms doc is as good a place as any.

> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  docs/about/build-platforms.rst | 28 ++++++++++++++++++++++++++++
>  meson.build                    |  2 +-
>  2 files changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 692323609e..bfe90e574e 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -29,6 +29,34 @@ The `Repology`_ site is a useful resource to identify
>  currently shipped versions of software in various operating systems,
>  though it does not cover all distros listed below.
>
> +Supported host CPUs
> +-------------------
> +
> +Those host CPUs have a native TCG backend and are regularly tested:

This is a list of host architectures, not CPUs.

> +  .. list-table::
> +   :header-rows: 1
> +
> +   * - CPU Family
> +     - Accelerators
> +   * - ARM

The correct capitalization these days is "Arm", by the way :-)

You also should split 64-bit and 32-bit Arm; we support
KVM on 64-bit but not 32-bit.

> +     - kvm, xen
> +   * - MIPS
> +     - kvm
> +   * - PPC
> +     - kvm
> +   * - RISC-V
> +     -
> +   * - s390x
> +     - kvm
> +   * - SPARC
> +     -
> +   * - x86
> +     - kvm, xen, hax, hvf (64 bit only), nvmm, whpx (64 bit only)
> +
> +Other architectures are not actively maintained. They use the slow and
> +experimental TCG interpreter. They may be removed in future releases.

This seems to be conflating TCG and the TCG interpreter.
We should just list which architectures we support (proper)
TCG for, and say that everything else is unsupported
(not mentioning the TCG interpreter at all; using it is
pretty much always a mistake IMHO).

The table also seems to me to be a bit confusing, because
the introductory text suggests it's a list of the TCG
support for each architecture, but the table itself lists
only the non-TCG accelerators. I think we should just list
all the accelerators supported for each host architecture.

Perhaps we should also (eventually) have somewhere some text
describing each accelerator in more detail, though probably
not in this file. A docs/system/accels.rst that described all
the accelerators with a paragraph or so for each, maybe ?

-- PMM


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 02/32] build-sys: add HAVE_IPPROTO_MPTCP
  2021-09-07 12:19 ` [RFC v3 02/32] build-sys: add HAVE_IPPROTO_MPTCP marcandre.lureau
@ 2021-09-08 12:01   ` Markus Armbruster
  2021-09-13 13:02   ` Paolo Bonzini
  1 sibling, 0 replies; 74+ messages in thread
From: Markus Armbruster @ 2021-09-08 12:01 UTC (permalink / raw)
  To: marcandre.lureau; +Cc: pbonzini, berrange, qemu-devel, stefanha

marcandre.lureau@redhat.com writes:

> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> The QAPI schema shouldn't rely on C system headers #define, but on
> configure-time project #define, so we can express the build condition in
> a C-independent way.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>

Reviewed-by: Markus Armbruster <armbru@redhat.com>



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 03/32] scripts/qapi: teach c_param_type() to return const argument type
  2021-09-07 12:19 ` [RFC v3 03/32] scripts/qapi: teach c_param_type() to return const argument type marcandre.lureau
@ 2021-09-08 12:10   ` Markus Armbruster
  2021-09-08 14:33     ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Markus Armbruster @ 2021-09-08 12:10 UTC (permalink / raw)
  To: marcandre.lureau; +Cc: pbonzini, berrange, qemu-devel, stefanha

marcandre.lureau@redhat.com writes:

> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> The argument isn't owned by the callee, so it better be const.
> But a lot of code in QEMU rely on non-const arguments to tweak it (steal
> values etc).
>
> Since Rust types / bindings are derived from the C version, we have to
> be more accurate there to do correct ownership in the bindings.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  scripts/qapi/schema.py | 15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 3d72c7dfc9..1f6301c394 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -226,8 +226,15 @@ def c_type(self):
>          pass
>  
>      # Return the C type to be used in a parameter list.
> -    def c_param_type(self):
> -        return self.c_type()
> +    #
> +    # The argument should be considered const, since no ownership is given to
> +    # the callee, but qemu C code frequently tweaks it. Set const=True for a
> +    # stricter declaration.

This comment makes sense only if you're familiar with Rust, where "may
change" is actually tied to ownership.

However, I can't see a use of .c_param_type(True).  Sure you need this
patch in this series?

> +    def c_param_type(self, const: bool = False):
> +        c_type = self.c_type()
> +        if const and c_type.endswith(POINTER_SUFFIX):
> +            c_type = 'const ' + c_type
> +        return c_type
>  
>      # Return the C type to be used where we suppress boxing.
>      def c_unboxed_type(self):
> @@ -280,10 +287,10 @@ def c_name(self):
>      def c_type(self):
>          return self._c_type_name
>  
> -    def c_param_type(self):
> +    def c_param_type(self, const: bool = False):
>          if self.name == 'str':
>              return 'const ' + self._c_type_name
> -        return self._c_type_name
> +        return super().c_param_type(const)

Would

       def c_param_type(self, const: bool = False):
           return super().c_param_type(const or self.name == 'str')

do?

>  
>      def json_type(self):
>          return self._json_type_name



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 04/32] glib-compat: add G_SIZEOF_MEMBER
  2021-09-07 12:19 ` [RFC v3 04/32] glib-compat: add G_SIZEOF_MEMBER marcandre.lureau
@ 2021-09-08 12:16   ` Markus Armbruster
  2021-09-08 13:49     ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Markus Armbruster @ 2021-09-08 12:16 UTC (permalink / raw)
  To: marcandre.lureau; +Cc: pbonzini, berrange, qemu-devel, stefanha

marcandre.lureau@redhat.com writes:

> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> The macro was introduced in 2.64. Let's redefine it, to allow its usage
> with older versions and silence the deprecation warning.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  include/glib-compat.h | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/include/glib-compat.h b/include/glib-compat.h
> index 9e95c888f5..e278e09d1d 100644
> --- a/include/glib-compat.h
> +++ b/include/glib-compat.h
> @@ -110,4 +110,11 @@ qemu_g_test_slow(void)
>  
>  #pragma GCC diagnostic pop
>  
> +/* introduced in 2.64 */
> +#ifdef G_SIZEOF_MEMBER
> +#undef G_SIZEOF_MEMBER
> +#endif
> +
> +#define G_SIZEOF_MEMBER(type, member) sizeof(((type *)0)->member)
> +
>  #endif

What's wrong with

   #if !GLIB_VERSION(2, 64, 0)
   #define G_SIZEOF_MEMBER(type, member) sizeof(((type *)0)->member)
   #endif

?



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 05/32] scripts/qapi: add QAPISchemaVisitor.visit_module_end
  2021-09-07 12:19 ` [RFC v3 05/32] scripts/qapi: add QAPISchemaVisitor.visit_module_end marcandre.lureau
@ 2021-09-08 12:26   ` Markus Armbruster
  0 siblings, 0 replies; 74+ messages in thread
From: Markus Armbruster @ 2021-09-08 12:26 UTC (permalink / raw)
  To: marcandre.lureau; +Cc: pbonzini, berrange, qemu-devel, stefanha

marcandre.lureau@redhat.com writes:

> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> Used in following patches to generate code after visiting a module.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  scripts/qapi/schema.py | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 1f6301c394..6455a8f425 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -128,6 +128,9 @@ def visit_end(self):
>      def visit_module(self, name):
>          pass
>  
> +    def visit_module_end(self, name) -> None:
> +        pass
> +
>      def visit_needed(self, entity):
>          # Default to visiting everything
>          return True
> @@ -207,6 +210,7 @@ def visit(self, visitor):
>          for entity in self._entity_list:
>              if visitor.visit_needed(entity):
>                  entity.visit(visitor)
> +        visitor.visit_module_end(self.name)
>  
>  
>  class QAPISchemaInclude(QAPISchemaEntity):

QAPISchema.visit(vis) now calls:

    vis.visit_begin
    for each module:
        vis.visit_module(module.name)
        for each entity in module
            entity.visit(vis)
        vis.visit_module_end(module.name)
    vis.visit_end

The .visit_module_end() isn't strictly necessary (we could do its work
in .visit_module() and .visit_end()).  But it's probably simpler this
way.

Let's rename .visit_module() to .visit_module_begin(), for symmetry with
.visit_begin(), .visit_end().



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 14/32] scripts/qapi: add QAPISchemaIfCond.rsgen()
  2021-09-07 12:19 ` [RFC v3 14/32] scripts/qapi: add QAPISchemaIfCond.rsgen() marcandre.lureau
@ 2021-09-08 12:33   ` Markus Armbruster
  2021-09-08 14:06     ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Markus Armbruster @ 2021-09-08 12:33 UTC (permalink / raw)
  To: marcandre.lureau; +Cc: pbonzini, berrange, qemu-devel, stefanha

marcandre.lureau@redhat.com writes:

> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> Generate Rust #[cfg(...)] guards from QAPI 'if' conditions.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  scripts/qapi/common.py | 16 ++++++++++++++++
>  scripts/qapi/schema.py |  4 ++++
>  2 files changed, 20 insertions(+)
>
> diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> index 5f8f76e5b2..6d22c66391 100644
> --- a/scripts/qapi/common.py
> +++ b/scripts/qapi/common.py
> @@ -201,6 +201,22 @@ def guardend(name: str) -> str:
>                   name=c_fname(name).upper())
>  
>  
> +def rsgen_ifcond(ifcond: Union[str, Dict[str, Any]]) -> str:
> +
> +    def cfg(ifcond: Union[str, Dict[str, Any]]):
> +        if isinstance(ifcond, str):
> +            return ifcond
> +        if isinstance(ifcond, list):
> +            return ', '.join([cfg(c) for c in ifcond])
> +        oper, operands = next(iter(ifcond.items()))
> +        operands = cfg(operands)
> +        return f'{oper}({operands})'
> +
> +    if not ifcond:
> +        return ''
> +    return '#[cfg(%s)]' % cfg(ifcond)
> +
> +
>  def gen_ifcond(ifcond: Optional[Union[str, Dict[str, Any]]],
>                 cond_fmt: str, not_fmt: str,
>                 all_operator: str, any_operator: str) -> str:

Can we generalize gen_ifcond() to work for rsgen_ifcond(), too?

> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 6455a8f425..c61f35e13f 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -26,6 +26,7 @@
>      docgen_ifcond,
>      gen_endif,
>      gen_if,
> +    rsgen_ifcond,
>  )
>  from .error import QAPIError, QAPISemError, QAPISourceError
>  from .expr import check_exprs
> @@ -48,6 +49,9 @@ def gen_endif(self):
>      def docgen(self):
>          return docgen_ifcond(self.ifcond)
>  
> +    def rsgen(self):
> +        return rsgen_ifcond(self.ifcond)
> +
>      def is_present(self):
>          return bool(self.ifcond)



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples
  2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
                   ` (31 preceding siblings ...)
  2021-09-07 12:19 ` [RFC v3 32/32] tests/vm: add Rust to Fedora marcandre.lureau
@ 2021-09-08 13:22 ` Markus Armbruster
  2021-09-08 13:55   ` Marc-André Lureau
  32 siblings, 1 reply; 74+ messages in thread
From: Markus Armbruster @ 2021-09-08 13:22 UTC (permalink / raw)
  To: marcandre.lureau; +Cc: pbonzini, berrange, qemu-devel, stefanha, armbru

Build fails for me:

make: Entering directory '/work/armbru/qemu/bld'
config-host.mak is out-of-date, running configure
  GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc capstone slirp
fatal: remote error: upload-pack: not our ref 7077bbbd11a67d60062a9483f996113a349a4ca1
Fetched in submodule path 'rust/vendored', but it did not contain 7077bbbd11a67d60062a9483f996113a349a4ca1. Direct fetching of that commit failed.
/work/armbru/qemu/scripts/git-submodule.sh: failed to update modules

Unable to automatically checkout GIT submodules ' ui/keycodemapdb meson rust/vendored tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc capstone slirp'.
If you require use of an alternative GIT binary (for example to
enable use of a transparent proxy), then please specify it by
running configure by with the '--with-git' argument. e.g.

 $ ./configure --with-git='tsocks git'

Alternatively you may disable automatic GIT submodule checkout
with:

 $ ./configure --with-git-submodules=validate

and then manually update submodules prior to running make, with:

 $ scripts/git-submodule.sh update  ui/keycodemapdb meson rust/vendored tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc capstone slirp

make: *** No rule to make target 'config-host.mak', needed by 'meson.stamp'.
make: Failed to remake makefile 'config-host.mak'.
make: Failed to remake makefile 'Makefile'.
make: Target 'all' not remade because of errors.
make: Leaving directory '/work/armbru/qemu/bld'



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 09/32] build-sys: add i686 cpu target
  2021-09-07 12:19 ` [RFC v3 09/32] build-sys: add i686 cpu target marcandre.lureau
@ 2021-09-08 13:45   ` Peter Maydell
  0 siblings, 0 replies; 74+ messages in thread
From: Peter Maydell @ 2021-09-08 13:45 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU Developers,
	Stefan Hajnoczi, Markus Armbruster

On Tue, 7 Sept 2021 at 13:37, <marcandre.lureau@redhat.com> wrote:
>
> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> Rust does not have i386 targets, so distinguish when target cpu is i686.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  configure | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)
>
> diff --git a/configure b/configure
> index 8adf2127c3..48ff2837d9 100755
> --- a/configure
> +++ b/configure
> @@ -617,6 +617,8 @@ esac
>  if test ! -z "$cpu" ; then
>    # command line argument
>    :
> +elif check_define __i686__ ; then
> +  cpu="i686"
>  elif check_define __i386__ ; then
>    cpu="i386"

This isn't what the 'cpu' variable is attempting to distinguish.
We care only about "is this the 32-bit Intel CPU family?", which
we for convenience name "i386". We do not attempt (at least not in
configure) to distinguish between different sub-flavours or
versions of that 32-bit architecture. (Similarly, 'arm' just
means "32-bit arm", and doesn't distinguish v5, v6, v7 or whatever.)

As it happens the existing C codebase won't work on a classic
i386 -- for instance the cpu_get_host_ticks() function
for i386 host is implemented as the 'rdtsc' insn, which
didn't come in until the pentium.

For native compilation, I guess we should just assume that
whatever rustc targets by default is the right thing.
For cross compilation we probably need some mechanism for
the user to tell us what the right rust cross target is,
the same way we currently have them pass a --cross-prefix
(which typically looks like a target-triple, but where the
cpu part of the triple isn't necessarily the same thing as
the 'cpu' variable configure is using.

thanks
-- PMM


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 04/32] glib-compat: add G_SIZEOF_MEMBER
  2021-09-08 12:16   ` Markus Armbruster
@ 2021-09-08 13:49     ` Marc-André Lureau
  0 siblings, 0 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 13:49 UTC (permalink / raw)
  To: Markus Armbruster
  Cc: Bonzini, Paolo, P. Berrange, Daniel, qemu-devel, Stefan Hajnoczi

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

Hi

On Wed, Sep 8, 2021 at 4:16 PM Markus Armbruster <armbru@redhat.com> wrote:

> marcandre.lureau@redhat.com writes:
>
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > The macro was introduced in 2.64. Let's redefine it, to allow its usage
> > with older versions and silence the deprecation warning.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  include/glib-compat.h | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/include/glib-compat.h b/include/glib-compat.h
> > index 9e95c888f5..e278e09d1d 100644
> > --- a/include/glib-compat.h
> > +++ b/include/glib-compat.h
> > @@ -110,4 +110,11 @@ qemu_g_test_slow(void)
> >
> >  #pragma GCC diagnostic pop
> >
> > +/* introduced in 2.64 */
> > +#ifdef G_SIZEOF_MEMBER
> > +#undef G_SIZEOF_MEMBER
> > +#endif
> > +
> > +#define G_SIZEOF_MEMBER(type, member) sizeof(((type *)0)->member)
> > +
> >  #endif
>
> What's wrong with
>
>    #if !GLIB_VERSION(2, 64, 0)
>    #define G_SIZEOF_MEMBER(type, member) sizeof(((type *)0)->member)
>    #endif
>
>
You would get "deprecation" warnings with glib >= 2.64.

[-- Attachment #2: Type: text/html, Size: 1904 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples
  2021-09-08 13:22 ` [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples Markus Armbruster
@ 2021-09-08 13:55   ` Marc-André Lureau
  2021-09-09 10:31     ` Markus Armbruster
  0 siblings, 1 reply; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 13:55 UTC (permalink / raw)
  To: Markus Armbruster
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Hi

On Wed, Sep 8, 2021 at 5:23 PM Markus Armbruster <armbru@redhat.com> wrote:

> Build fails for me:
>
> make: Entering directory '/work/armbru/qemu/bld'
> config-host.mak is out-of-date, running configure
>   GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
> tests/fp/berkeley-softfloat-3 dtc capstone slirp
> fatal: remote error: upload-pack: not our ref
> 7077bbbd11a67d60062a9483f996113a349a4ca1
>

Looks like I didn't update the repository on github. done

I will update .gitmodule to use the repo on gitlab instead (gitlab.com:
marcandre.lureau/qemu-rust-vendored.git)

-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 1058 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 10/32] build-sys: add --with-rust{-target} & basic build infrastructure
  2021-09-07 12:19 ` [RFC v3 10/32] build-sys: add --with-rust{-target} & basic build infrastructure marcandre.lureau
@ 2021-09-08 14:00   ` Peter Maydell
  2021-09-08 14:21     ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Peter Maydell @ 2021-09-08 14:00 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU Developers,
	Stefan Hajnoczi, Markus Armbruster

On Tue, 7 Sept 2021 at 13:41, <marcandre.lureau@redhat.com> wrote:
>
> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> Add the build-sys infrastructure to optionally build Rust code.
> Introduce a top-level workspace, so various sub-projects (libraries,
> executables etc) can be developed together, sharing the dependencies and
> output directory.
>
> If not Tier 1 (aarch64 and x86), all of the host architecture QEMU
> supports should be Tier 2:
> https://doc.rust-lang.org/nightly/rustc/platform-support.html

I don't think this is quite true -- for instance the riscv
TCG backend supports 32-bit, but the riscv 32 bit linux rust
port is down in Tier 3. But the major stuff seems to be in Tier
1 or 2.


> +if with_rust
> +  rust_target_triple = get_option('with_rust_target')
> +  if meson.is_cross_build()
> +    # more default target mappings may be added over time
> +    if rust_target_triple == '' and targetos == 'windows'
> +      rust_target_triple = host_machine.cpu() + '-pc-windows-gnu'
> +    endif
> +    if rust_target_triple == ''
> +      error('cross-compiling, but no Rust target-triple defined.')
> +    endif
> +  endif
> +endif

Not a huge fan of this... I think we would be better to always
require the user to specify the rust target triple explicitly.


> diff --git a/.gitignore b/.gitignore
> index eb2553026c..78715bc7c4 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -1,5 +1,6 @@
>  /GNUmakefile
>  /build/
> +/target/

What's this for ?

>  *.pyc
>  .sdk
>  .stgit-*

thanks
-- PMM


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 14/32] scripts/qapi: add QAPISchemaIfCond.rsgen()
  2021-09-08 12:33   ` Markus Armbruster
@ 2021-09-08 14:06     ` Marc-André Lureau
  0 siblings, 0 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 14:06 UTC (permalink / raw)
  To: Markus Armbruster
  Cc: Bonzini, Paolo, P. Berrange, Daniel, qemu-devel, Stefan Hajnoczi

[-- Attachment #1: Type: text/plain, Size: 2541 bytes --]

Hi

On Wed, Sep 8, 2021 at 4:33 PM Markus Armbruster <armbru@redhat.com> wrote:

> marcandre.lureau@redhat.com writes:
>
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Generate Rust #[cfg(...)] guards from QAPI 'if' conditions.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  scripts/qapi/common.py | 16 ++++++++++++++++
> >  scripts/qapi/schema.py |  4 ++++
> >  2 files changed, 20 insertions(+)
> >
> > diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
> > index 5f8f76e5b2..6d22c66391 100644
> > --- a/scripts/qapi/common.py
> > +++ b/scripts/qapi/common.py
> > @@ -201,6 +201,22 @@ def guardend(name: str) -> str:
> >                   name=c_fname(name).upper())
> >
> >
> > +def rsgen_ifcond(ifcond: Union[str, Dict[str, Any]]) -> str:
> > +
> > +    def cfg(ifcond: Union[str, Dict[str, Any]]):
> > +        if isinstance(ifcond, str):
> > +            return ifcond
> > +        if isinstance(ifcond, list):
> > +            return ', '.join([cfg(c) for c in ifcond])
> > +        oper, operands = next(iter(ifcond.items()))
> > +        operands = cfg(operands)
> > +        return f'{oper}({operands})'
> > +
> > +    if not ifcond:
> > +        return ''
> > +    return '#[cfg(%s)]' % cfg(ifcond)
> > +
> > +
> >  def gen_ifcond(ifcond: Optional[Union[str, Dict[str, Any]]],
> >                 cond_fmt: str, not_fmt: str,
> >                 all_operator: str, any_operator: str) -> str:
>
> Can we generalize gen_ifcond() to work for rsgen_ifcond(), too?
>
>
Not elegantly, I am afraid. The logic of gen_ifcond() is based around the
distinct prefix vs infix handling. In contrast, Rust cfg are all infix. As
you can see from the code above, it is quite straightforward. Reusing
gen_ifcond() would make it quite convoluted.


> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index 6455a8f425..c61f35e13f 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -26,6 +26,7 @@
> >      docgen_ifcond,
> >      gen_endif,
> >      gen_if,
> > +    rsgen_ifcond,
> >  )
> >  from .error import QAPIError, QAPISemError, QAPISourceError
> >  from .expr import check_exprs
> > @@ -48,6 +49,9 @@ def gen_endif(self):
> >      def docgen(self):
> >          return docgen_ifcond(self.ifcond)
> >
> > +    def rsgen(self):
> > +        return rsgen_ifcond(self.ifcond)
> > +
> >      def is_present(self):
> >          return bool(self.ifcond)
>
>

[-- Attachment #2: Type: text/html, Size: 3699 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 10/32] build-sys: add --with-rust{-target} & basic build infrastructure
  2021-09-08 14:00   ` Peter Maydell
@ 2021-09-08 14:21     ` Marc-André Lureau
  0 siblings, 0 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 14:21 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU Developers,
	Stefan Hajnoczi, Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 2356 bytes --]

Hi

On Wed, Sep 8, 2021 at 6:01 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Tue, 7 Sept 2021 at 13:41, <marcandre.lureau@redhat.com> wrote:
> >
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Add the build-sys infrastructure to optionally build Rust code.
> > Introduce a top-level workspace, so various sub-projects (libraries,
> > executables etc) can be developed together, sharing the dependencies and
> > output directory.
> >
> > If not Tier 1 (aarch64 and x86), all of the host architecture QEMU
> > supports should be Tier 2:
> > https://doc.rust-lang.org/nightly/rustc/platform-support.html
>
> I don't think this is quite true -- for instance the riscv
> TCG backend supports 32-bit, but the riscv 32 bit linux rust
> port is down in Tier 3. But the major stuff seems to be in Tier
> 1 or 2.
>

That's why the first patch would clarify the situation a bit :)


>
> > +if with_rust
> > +  rust_target_triple = get_option('with_rust_target')
> > +  if meson.is_cross_build()
> > +    # more default target mappings may be added over time
> > +    if rust_target_triple == '' and targetos == 'windows'
> > +      rust_target_triple = host_machine.cpu() + '-pc-windows-gnu'
> > +    endif
> > +    if rust_target_triple == ''
> > +      error('cross-compiling, but no Rust target-triple defined.')
> > +    endif
> > +  endif
> > +endif
>
> Not a huge fan of this... I think we would be better to always
> require the user to specify the rust target triple explicitly.
>
>
In this case, no need to handle the i686 case. Fine with me, I was trying
to guess from the meson config and existing --cross-prefix to make things
simpler.


>
> > diff --git a/.gitignore b/.gitignore
> > index eb2553026c..78715bc7c4 100644
> > --- a/.gitignore
> > +++ b/.gitignore
> > @@ -1,5 +1,6 @@
> >  /GNUmakefile
> >  /build/
> > +/target/
>
> What's this for ?
>

Oops, that's wrong and unfortunate. It's the default target dir of Cargo.
But it's set to $builddir/rs-target in the scripts/cargo_wrapper.py.

It is convenient to run cargo commands manually, and dirty stuff end up in
target/.. I will configure .cargo/config.toml and try to prevent cargo
usage from top-level source directory.



> >  *.pyc
> >  .sdk
> >  .stgit-*
>
> thanks
> -- PMM
>
>

[-- Attachment #2: Type: text/html, Size: 3762 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 03/32] scripts/qapi: teach c_param_type() to return const argument type
  2021-09-08 12:10   ` Markus Armbruster
@ 2021-09-08 14:33     ` Marc-André Lureau
  0 siblings, 0 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 14:33 UTC (permalink / raw)
  To: Markus Armbruster
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi

[-- Attachment #1: Type: text/plain, Size: 2611 bytes --]

Hi

On Wed, Sep 8, 2021 at 4:12 PM Markus Armbruster <armbru@redhat.com> wrote:

> marcandre.lureau@redhat.com writes:
>
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > The argument isn't owned by the callee, so it better be const.
> > But a lot of code in QEMU rely on non-const arguments to tweak it (steal
> > values etc).
> >
> > Since Rust types / bindings are derived from the C version, we have to
> > be more accurate there to do correct ownership in the bindings.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 15 +++++++++++----
> >  1 file changed, 11 insertions(+), 4 deletions(-)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index 3d72c7dfc9..1f6301c394 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -226,8 +226,15 @@ def c_type(self):
> >          pass
> >
> >      # Return the C type to be used in a parameter list.
> > -    def c_param_type(self):
> > -        return self.c_type()
> > +    #
> > +    # The argument should be considered const, since no ownership is
> given to
> > +    # the callee, but qemu C code frequently tweaks it. Set const=True
> for a
> > +    # stricter declaration.
>
> This comment makes sense only if you're familiar with Rust, where "may
> change" is actually tied to ownership.
>
>
Arguably, this semantic can also apply to C.


> However, I can't see a use of .c_param_type(True).  Sure you need this
> patch in this series?
>
>
Indeed it looks like a leftover now. Let's drop it.

> +    def c_param_type(self, const: bool = False):
> > +        c_type = self.c_type()
> > +        if const and c_type.endswith(POINTER_SUFFIX):
> > +            c_type = 'const ' + c_type
> > +        return c_type
> >
> >      # Return the C type to be used where we suppress boxing.
> >      def c_unboxed_type(self):
> > @@ -280,10 +287,10 @@ def c_name(self):
> >      def c_type(self):
> >          return self._c_type_name
> >
> > -    def c_param_type(self):
> > +    def c_param_type(self, const: bool = False):
> >          if self.name == 'str':
> >              return 'const ' + self._c_type_name
> > -        return self._c_type_name
> > +        return super().c_param_type(const)
>
> Would
>
>        def c_param_type(self, const: bool = False):
>            return super().c_param_type(const or self.name == 'str')
>
> do?
>
> >
> >      def json_type(self):
> >          return self._json_type_name
>
>
>

-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 4109 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-07 12:19 ` [RFC v3 13/32] rust: use vendored-sources marcandre.lureau
@ 2021-09-08 15:38   ` Ian Jackson
  2021-09-08 15:47     ` Marc-André Lureau
  2021-09-08 16:20     ` Marc-André Lureau
  2021-09-09 16:02   ` Peter Maydell
  1 sibling, 2 replies; 74+ messages in thread
From: Ian Jackson @ 2021-09-08 15:38 UTC (permalink / raw)
  To: marcandre.lureau; +Cc: pbonzini, berrange, qemu-devel, stefanha, armbru

marcandre.lureau@redhat.com writes ("[RFC v3 13/32] rust: use vendored-sources"):
> Most likely, QEMU will want tighter control over the sources, rather
> than relying on crates.io downloading, use a git submodule with all the
> dependencies. However, cargo --offline was added in 1.36.

Hi.

pm215 pointed me at this, as I have some background in Rust.
I definitely approve of having Rust in Qemu.  I don't have an opinion
about whether the sources should be vendored this way.

But, I tried to build this, and

    error: failed to select a version for the requirement `cc = "=1.0.70"`
    candidate versions found which didn't match: 1.0.69
    location searched: directory source `/volatile/rustcargo/Rustup/Qemu/qemu.pwt/rust/vendored` (which is replacing registry `crates-io`)
    required by package `nix v0.20.1`
        ... which is depended on by `qga v0.1.0 (/volatile/rustcargo/Rustup/Qemu/qemu.pwt/qga)`
    perhaps a crate was updated and forgotten to be re-vendored?
    As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.

I think the most important part here is to get the general APIs,
presented to general Rust code in Qemu, right.  So I wanted to review
those via the output from rustdoc.

I tried commenting out the `replace-with` in .cargo/config.toml
but evidently the systme isn't intended to be used that way.

Ian.


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 15:38   ` Ian Jackson
@ 2021-09-08 15:47     ` Marc-André Lureau
  2021-09-08 15:55       ` Ian Jackson
  2021-09-08 16:20     ` Marc-André Lureau
  1 sibling, 1 reply; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 15:47 UTC (permalink / raw)
  To: Ian Jackson
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi,
	Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 2425 bytes --]

Hi

On Wed, Sep 8, 2021 at 7:40 PM Ian Jackson <iwj@xenproject.org> wrote:

> marcandre.lureau@redhat.com writes ("[RFC v3 13/32] rust: use
> vendored-sources"):
> > Most likely, QEMU will want tighter control over the sources, rather
> > than relying on crates.io downloading, use a git submodule with all the
> > dependencies. However, cargo --offline was added in 1.36.
>
> Hi.
>
> pm215 pointed me at this, as I have some background in Rust.
> I definitely approve of having Rust in Qemu.  I don't have an opinion
> about whether the sources should be vendored this way.
>
> But, I tried to build this, and
>
>     error: failed to select a version for the requirement `cc = "=1.0.70"`
>     candidate versions found which didn't match: 1.0.69
>     location searched: directory source
> `/volatile/rustcargo/Rustup/Qemu/qemu.pwt/rust/vendored` (which is
> replacing registry `crates-io`)
>     required by package `nix v0.20.1`
>         ... which is depended on by `qga v0.1.0
> (/volatile/rustcargo/Rustup/Qemu/qemu.pwt/qga)`
>     perhaps a crate was updated and forgotten to be re-vendored?
>     As a reminder, you're using offline mode (--offline) which can
> sometimes cause surprising resolution failures, if this error is too
> confusing you may wish to retry without the offline flag.
>
> I think the most important part here is to get the general APIs,
> presented to general Rust code in Qemu, right.  So I wanted to review
> those via the output from rustdoc.
>
> I tried commenting out the `replace-with` in .cargo/config.toml
> but evidently the systme isn't intended to be used that way.
>
> Ian.
>
>
Hmm, I do "cargo vendor --versioned-dirs ../rust/vendored" to vendor crates.

It seems cc was updated, and I didn't update the submodule accordingly. For
reference, this is the dependency tree that WFM:

$ cargo tree -p qga
qga v0.1.0 (/home/elmarco/src/qemu/qga)
├── common v0.1.0 (/home/elmarco/src/qemu/rust/common)
│   ├── libc v0.2.101
│   └── nix v0.20.1
│       ├── bitflags v1.2.1
│       ├── cfg-if v1.0.0
│       ├── libc v0.2.101
│       └── memoffset v0.6.4
│           [build-dependencies]
│           └── autocfg v1.0.1
├── hostname v0.3.1
│   ├── libc v0.2.101
│   └── match_cfg v0.1.0
└── nix v0.20.1 (*)


-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 3247 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 15:47     ` Marc-André Lureau
@ 2021-09-08 15:55       ` Ian Jackson
  2021-09-08 16:15         ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Ian Jackson @ 2021-09-08 15:55 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi,
	Markus Armbruster

Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> Hmm, I do "cargo vendor --versioned-dirs ../rust/vendored" to vendor crates.
> 
> It seems cc was updated, and I didn't update the submodule accordingly. For
> reference, this is the dependency tree that WFM:

git submodules are just awful IMO.

> $ cargo tree -p qga                            
> qga v0.1.0 (/home/elmarco/src/qemu/qga)
> ├── common v0.1.0 (/home/elmarco/src/qemu/rust/common)
> │   ├── libc v0.2.101
> │   └── nix v0.20.1
> │       ├── bitflags v1.2.1
> │       ├── cfg-if v1.0.0
> │       ├── libc v0.2.101
> │       └── memoffset v0.6.4
> │           [build-dependencies]
> │           └── autocfg v1.0.1
> ├── hostname v0.3.1
> │   ├── libc v0.2.101
> │   └── match_cfg v0.1.0
> └── nix v0.20.1 (*)

With the .config/cargo.toml "replace-with" commented out, I see this:

rustcargo@zealot:~/Rustup/Qemu/qemu.pwt/build$ cargo tree -p qga
qga v0.1.0 (/volatile/rustcargo/Rustup/Qemu/qemu.pwt/qga)
├── common v0.1.0 (/volatile/rustcargo/Rustup/Qemu/qemu.pwt/rust/common)
│   ├── libc v0.2.101
│   └── nix v0.20.1
│       ├── bitflags v1.2.1
│       ├── cfg-if v1.0.0
│       ├── libc v0.2.101
│       └── memoffset v0.6.4
│           [build-dependencies]
│           └── autocfg v1.0.1
├── hostname v0.3.1
│   ├── libc v0.2.101
│   └── match_cfg v0.1.0
└── nix v0.20.1 (*)
rustcargo@zealot:~/Rustup/Qemu/qemu.pwt/build$ 

Which is the same as yours.  Although "cargo build" doesn't work
build, guessed from the messagese that perhaps this was the automatic
codegen hadn't run.  I'm now trying "make" and and it seems to be
running.

With the "replace-with" uncommented, cargo tree bombs out.  I'm afraid
I haven't used cargo vendor so I'm not sure if I am going in the right
direction with this workaround.  Hopefully it will finish the build.

Would it be possible to have a configure option to use unvendored
upstream Rust libraries from crates.io ?

Ian.


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 15:55       ` Ian Jackson
@ 2021-09-08 16:15         ` Marc-André Lureau
  2021-09-08 16:22           ` Peter Maydell
  2021-09-08 16:22           ` Ian Jackson
  0 siblings, 2 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 16:15 UTC (permalink / raw)
  To: Ian Jackson
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi,
	Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 2595 bytes --]

Hi

On Wed, Sep 8, 2021 at 7:55 PM Ian Jackson <iwj@xenproject.org> wrote:

> Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> > Hmm, I do "cargo vendor --versioned-dirs ../rust/vendored" to vendor
> crates.
> >
> > It seems cc was updated, and I didn't update the submodule accordingly.
> For
> > reference, this is the dependency tree that WFM:
>
> git submodules are just awful IMO.
>

Yes, but it's often (always?) the user fault. CI should help, when it will
check Rust code.


> > $ cargo tree -p qga
> > qga v0.1.0 (/home/elmarco/src/qemu/qga)
> > ├── common v0.1.0 (/home/elmarco/src/qemu/rust/common)
> > │   ├── libc v0.2.101
> > │   └── nix v0.20.1
> > │       ├── bitflags v1.2.1
> > │       ├── cfg-if v1.0.0
> > │       ├── libc v0.2.101
> > │       └── memoffset v0.6.4
> > │           [build-dependencies]
> > │           └── autocfg v1.0.1
> > ├── hostname v0.3.1
> > │   ├── libc v0.2.101
> > │   └── match_cfg v0.1.0
> > └── nix v0.20.1 (*)
>
> With the .config/cargo.toml "replace-with" commented out, I see this:
>
> rustcargo@zealot:~/Rustup/Qemu/qemu.pwt/build$ cargo tree -p qga
> qga v0.1.0 (/volatile/rustcargo/Rustup/Qemu/qemu.pwt/qga)
> ├── common v0.1.0 (/volatile/rustcargo/Rustup/Qemu/qemu.pwt/rust/common)
> │   ├── libc v0.2.101
> │   └── nix v0.20.1
> │       ├── bitflags v1.2.1
> │       ├── cfg-if v1.0.0
> │       ├── libc v0.2.101
> │       └── memoffset v0.6.4
> │           [build-dependencies]
> │           └── autocfg v1.0.1
> ├── hostname v0.3.1
> │   ├── libc v0.2.101
> │   └── match_cfg v0.1.0
> └── nix v0.20.1 (*)
> rustcargo@zealot:~/Rustup/Qemu/qemu.pwt/build$
>
> Which is the same as yours.  Although "cargo build" doesn't work
> build, guessed from the messagese that perhaps this was the automatic
> codegen hadn't run.  I'm now trying "make" and and it seems to be
> running.
>
> With the "replace-with" uncommented, cargo tree bombs out.  I'm afraid
> I haven't used cargo vendor so I'm not sure if I am going in the right
> direction with this workaround.  Hopefully it will finish the build.
>
> Would it be possible to have a configure option to use unvendored
> upstream Rust libraries from crates.io ?
>

Not easily, but we could have a --disable-rust-offline configure option.
Whether this is desirable, I am not sure.


-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 3582 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 15:38   ` Ian Jackson
  2021-09-08 15:47     ` Marc-André Lureau
@ 2021-09-08 16:20     ` Marc-André Lureau
  2021-09-08 16:29       ` Ian Jackson
  1 sibling, 1 reply; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 16:20 UTC (permalink / raw)
  To: Ian Jackson
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi,
	Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 2094 bytes --]

Hi

On Wed, Sep 8, 2021 at 7:40 PM Ian Jackson <iwj@xenproject.org> wrote:

> marcandre.lureau@redhat.com writes ("[RFC v3 13/32] rust: use
> vendored-sources"):
> > Most likely, QEMU will want tighter control over the sources, rather
> > than relying on crates.io downloading, use a git submodule with all the
> > dependencies. However, cargo --offline was added in 1.36.
>
> Hi.
>
> pm215 pointed me at this, as I have some background in Rust.
> I definitely approve of having Rust in Qemu.  I don't have an opinion
> about whether the sources should be vendored this way.
>
> But, I tried to build this, and
>
>     error: failed to select a version for the requirement `cc = "=1.0.70"`
>     candidate versions found which didn't match: 1.0.69
>     location searched: directory source
> `/volatile/rustcargo/Rustup/Qemu/qemu.pwt/rust/vendored` (which is
> replacing registry `crates-io`)
>     required by package `nix v0.20.1`
>         ... which is depended on by `qga v0.1.0
> (/volatile/rustcargo/Rustup/Qemu/qemu.pwt/qga)`
>     perhaps a crate was updated and forgotten to be re-vendored?
>     As a reminder, you're using offline mode (--offline) which can
> sometimes cause surprising resolution failures, if this error is too
> confusing you may wish to retry without the offline flag.
>
> I think the most important part here is to get the general APIs,
> presented to general Rust code in Qemu, right.  So I wanted to review
> those via the output from rustdoc.
>

You can start by reading `cargo doc -p common --open`. The generated code
needs some environment variables set, so `cargo doc -p qga` will fail
unless you set the environment variable

MESON_BUILD_ROOT=`pwd` cargo doc -p qga --open --document-private-items

works, but the QAPI types aren't documented, so this is a bit useless at
this point. I wonder if I could put the schema doc, hmm...


> I tried commenting out the `replace-with` in .cargo/config.toml
> but evidently the systme isn't intended to be used that way.
>
> Ian.
>
>

-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 2916 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 16:15         ` Marc-André Lureau
@ 2021-09-08 16:22           ` Peter Maydell
  2021-09-08 16:22           ` Ian Jackson
  1 sibling, 0 replies; 74+ messages in thread
From: Peter Maydell @ 2021-09-08 16:22 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Daniel P. Berrange, Markus Armbruster, QEMU, Ian Jackson,
	Stefan Hajnoczi, Paolo Bonzini

On Wed, 8 Sept 2021 at 17:17, Marc-André Lureau
<marcandre.lureau@gmail.com> wrote:
>
> Hi
>
> On Wed, Sep 8, 2021 at 7:55 PM Ian Jackson <iwj@xenproject.org> wrote:
>>
>> Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
>> > Hmm, I do "cargo vendor --versioned-dirs ../rust/vendored" to vendor crates.
>> >
>> > It seems cc was updated, and I didn't update the submodule accordingly. For
>> > reference, this is the dependency tree that WFM:
>>
>> git submodules are just awful IMO.
>
>
> Yes, but it's often (always?) the user fault.

I tend to agree with Ian -- submodules are badly designed, and
have lots of sharp edges that it's easy to cut yourself on.
Yes, you can say "well, the user should have held it by the other handle
because that one isn't fitted with the spring-loaded razorblades", but I
would argue that fault is better placed at the door of the designer in
that kind of situation...

-- PMM


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 16:15         ` Marc-André Lureau
  2021-09-08 16:22           ` Peter Maydell
@ 2021-09-08 16:22           ` Ian Jackson
  1 sibling, 0 replies; 74+ messages in thread
From: Ian Jackson @ 2021-09-08 16:22 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Daniel P. Berrange, Markus Armbruster, QEMU, Ian Jackson,
	Stefan Hajnoczi, Paolo Bonzini

Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> On Wed, Sep 8, 2021 at 7:55 PM Ian Jackson <iwj@xenproject.org> wrote:
>  >   git submodules are just awful IMO.
> 
> Yes, but it's often (always?) the user fault.

I must disagree in the strongest possible terms.  I don't think I can
express my feelings on this in a way that would be appropriate in this
context.

Anyway...

My trickery as described above (run configure, edit the "replace-with"
out of .cargo/config.toml, run make) did produce a build.

But to review the internal API I want the rustdoc output.

How can I run rustdoc in a way that will work ?  I tried "cargo doc"
and it complained about a lack of "MESON_BUILD_ROOT".  I guessed and
ran
  MESON_BUILD_ROOT=$PWD cargo doc
which seemed to produce some output and complete but I can't find the
results anywhere.

Can you please give me the set of runes to type view the rustdoc-built
API documentation for the qemu-internal Rust APIs ?

Thanks,
Ian.


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 16:20     ` Marc-André Lureau
@ 2021-09-08 16:29       ` Ian Jackson
  2021-09-08 16:34         ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Ian Jackson @ 2021-09-08 16:29 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi,
	Markus Armbruster

Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> You can start by reading `cargo doc -p common --open`. The generated
> code needs some environment variables set, so `cargo doc -p qga`
> will fail unless you set the environment variable
> 
> MESON_BUILD_ROOT=`pwd` cargo doc -p qga --open --document-private-items

Thanks.  I did this (and your rune from bofere) and I have the docs
open.

I wasn't quite sure where to start.  I didn't see where the
entrypoints were.  I did find

 .../target/doc/qga/qmp/fn.qmp_guest_set_vcpus.html

which err, doesn't look like the kind of safe api I was hoping to
find.

Ian.


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 16:29       ` Ian Jackson
@ 2021-09-08 16:34         ` Marc-André Lureau
  2021-09-08 16:50           ` Ian Jackson
  0 siblings, 1 reply; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 16:34 UTC (permalink / raw)
  To: Ian Jackson
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi,
	Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

Hi

On Wed, Sep 8, 2021 at 8:29 PM Ian Jackson <iwj@xenproject.org> wrote:

> Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> > You can start by reading `cargo doc -p common --open`. The generated
> > code needs some environment variables set, so `cargo doc -p qga`
> > will fail unless you set the environment variable
> >
> > MESON_BUILD_ROOT=`pwd` cargo doc -p qga --open --document-private-items
>
> Thanks.  I did this (and your rune from bofere) and I have the docs
> open.
>
> I wasn't quite sure where to start.  I didn't see where the
> entrypoints were.  I did find
>
>  .../target/doc/qga/qmp/fn.qmp_guest_set_vcpus.html
>
> which err, doesn't look like the kind of safe api I was hoping to
> find.
>

Yes, this is the shim to provide a C ABI QMP handler from Rust. This is
where all the FFI<->Rust conversion takes place.

The "safe" code is qga/qmp/vcpus.rs. However, there is no documentation
there, since it's not meant to be the public interface. It's documented
with the QAPI schema.


-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 1632 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 16:34         ` Marc-André Lureau
@ 2021-09-08 16:50           ` Ian Jackson
  2021-09-08 19:33             ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Ian Jackson @ 2021-09-08 16:50 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi,
	Markus Armbruster

Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> Yes, this is the shim to provide a C ABI QMP handler from Rust. This is where
> all the FFI<->Rust conversion takes place.
> 
> The "safe" code is qga/qmp/vcpus.rs. However, there is no
> documentation there, since it's not meant to be the public
> interface. It's documented with the QAPI schema.

Right, thanks.  That does look like a PoC of a Rust API.  I wanted the
rustdoc output because I find it provides a very uniform and readable
presentation even of an API with no doc comments.

I think maybe a thing I am missing is how you expect this to be used.
Which parts of the system are going to be in Rust.  etc.
And that would help explain what "public" means.

I think the answer is probably in this example:

https://patchew.org/QEMU/20210907121943.3498701-1-marcandre.lureau@redhat.com/20210907121943.3498701-30-marcandre.lureau@redhat.com/

but although my C and Rust are both fine, I don't understand qemu well
enough to make sense of it.

... wait, qga is "qemu guest agent" ?

I think I am sort of seeing this use case now.  But presuambly there
are other use cases for this QMP/QAPI type bridge stuff.

Sorry to be asking such stupid questions.

Thanks,
Ian.


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-08 16:50           ` Ian Jackson
@ 2021-09-08 19:33             ` Marc-André Lureau
  0 siblings, 0 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-08 19:33 UTC (permalink / raw)
  To: Ian Jackson
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi,
	Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 4321 bytes --]

Hi

On Wed, Sep 8, 2021 at 8:51 PM Ian Jackson <iwj@xenproject.org> wrote:

> Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> > Yes, this is the shim to provide a C ABI QMP handler from Rust. This is
> where
> > all the FFI<->Rust conversion takes place.
> >
> > The "safe" code is qga/qmp/vcpus.rs. However, there is no
> > documentation there, since it's not meant to be the public
> > interface. It's documented with the QAPI schema.
>
> Right, thanks.  That does look like a PoC of a Rust API.  I wanted the
> rustdoc output because I find it provides a very uniform and readable
> presentation even of an API with no doc comments.
>
> I think maybe a thing I am missing is how you expect this to be used.
> Which parts of the system are going to be in Rust.  etc.
> And that would help explain what "public" means.
>
> I think the answer is probably in this example:
>
>
> https://patchew.org/QEMU/20210907121943.3498701-1-marcandre.lureau@redhat.com/20210907121943.3498701-30-marcandre.lureau@redhat.com/
>
> but although my C and Rust are both fine, I don't understand qemu well
> enough to make sense of it.
>
> ... wait, qga is "qemu guest agent" ?
>
> I think I am sort of seeing this use case now.  But presuambly there
> are other use cases for this QMP/QAPI type bridge stuff.
>
> Sorry to be asking such stupid questions.
>

There is no magic wand to introduce Rust code in an existing C code base.
You need to glue some C ABI to/from Rust. It's a lot of manual work to
properly bind a C API to Rust (it's a project on its own I would say).
Typically, FFI bindings can be automated from headers, and high-level Rust
bindings are done by hand. Then you want high-level bindings to take
advantage of Rust, for idiomatic and safe code. Various internal QEMU API
will have to be bound by hand to start using them from Rust. An isolated
unit (say a parser, a function) could be rewritten in Rust and a C ABI be
provided without much hassle. But in general, code is quickly
interdependent, or the amount of stuff to rewrite in one go is large and
risky to do it that way.

In the glib/gobject world, the ABI are annotated, and you can automate much
of the high-level binding process (which is amazing, given the complexity
of the APIs, with objects, async methods, signals, properties, .. various
concepts that don't match easily in Rust). To help with this process, they
introduced conversion traits (the ToQemu/FromQemu adapted here), common
interfaces, to help automate and compose complex types (and their own
binding generator).

(Unfortunately) QEMU doesn't use gobject, but it relies heavily on two type
systems of its own: QAPI and QOM. QAPI is actually more of an IDL, which
translates C from/to JSON/QMP and has commands and signals (and is the
protocol used to communicate with qemu or qemu-ga etc). A large part of
QEMU are direct users of the QAPI generated types and functions. It is thus
a good target to generate bindings automatically. As demonstrated at the
end, it allows writing QMP handlers in idiomatic Rust. Since
qemu-guest-agent doesn't have a complex internal state (most commands are
really independent, they could be different programs..!), I started
rewriting some handlers there. It feels relatively straightforward to
rewrite in Rust, and we could imagine a complete rewrite of qemu-ga...
However, it is less of a waste to focus on critical parts or newly added
code instead, imho! Furthermore, the Rust doesn't cover all targets C can
currently target, so we must have some care when deciding a language. (you
can imagine I would encourage anyone to do it in Rust!)

QOM is the QEMU object system and is spread throughout the code. If there
is enough interest for this Rust effort, I plan to look at binding it next
(based on my experience working on GObject bindings). I am afraid we are
not going to have as friendly bindings as what the GNOME team achieved, but
since QOM is a bit simpler, we may find acceptable compromises.

With QOM & QAPI, and manual bindings of internal APIs, I think we could
start writing interesting code in Rust, simple devices, external interfaces
etc.


 Hope that helps clarify a bit the current goals



-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 5244 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples
  2021-09-08 13:55   ` Marc-André Lureau
@ 2021-09-09 10:31     ` Markus Armbruster
  2021-09-09 15:22       ` Marc-André Lureau
  0 siblings, 1 reply; 74+ messages in thread
From: Markus Armbruster @ 2021-09-09 10:31 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi

Marc-André Lureau <marcandre.lureau@gmail.com> writes:

> Hi
>
> On Wed, Sep 8, 2021 at 5:23 PM Markus Armbruster <armbru@redhat.com> wrote:
>
>> Build fails for me:
>>
>> make: Entering directory '/work/armbru/qemu/bld'
>> config-host.mak is out-of-date, running configure
>>   GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
>> tests/fp/berkeley-softfloat-3 dtc capstone slirp
>> fatal: remote error: upload-pack: not our ref
>> 7077bbbd11a67d60062a9483f996113a349a4ca1
>>
>
> Looks like I didn't update the repository on github. done
>
> I will update .gitmodule to use the repo on gitlab instead (gitlab.com:
> marcandre.lureau/qemu-rust-vendored.git)

Next error:

make -k -C ~/work/qemu/bld-x86 
make: Entering directory '/work/armbru/qemu/bld-x86'
  GIT     ui/keycodemapdb rust/vendored tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc capstone slirp
[1/71] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
[2/55] Generating cargo-qga with a custom command
FAILED: qga/libqga.args qga/libqga.a 
/usr/bin/python3 /work/armbru/qemu/scripts/cargo_wrapper.py --configh /work/armbru/qemu/bld-x86/config-host.h /work/armbru/qemu/bld-x86/qga /work/armbru/qemu/qga /work/armbru/qemu/bld-x86 debug '' build-lib
Environment: MESON_CURRENT_BUILD_DIR=/work/armbru/qemu/bld-x86/qga MESON_BUILD_ROOT=/work/armbru/qemu/bld-x86 WINAPI_NO_BUNDLED_LIBRARIES=1
Command: cargo rustc --target-dir /work/armbru/qemu/bld-x86/qga/rs-target --manifest-path /work/armbru/qemu/qga/Cargo.toml --offline --lib -- --cfg CONFIG_ACCEPT4 --cfg CONFIG_AF_VSOCK --cfg CONFIG_ATOMIC64 --cfg CONFIG_ATTR --cfg CONFIG_AUDIO_OSS --cfg CONFIG_AUDIO_PA --cfg CONFIG_AVX2_OPT --cfg CONFIG_BDRV_RO_WHITELIST --cfg CONFIG_BDRV_RW_WHITELIST --cfg CONFIG_BOCHS --cfg CONFIG_BRLAPI --cfg CONFIG_BYTESWAP_H --cfg CONFIG_CAPSTONE --cfg CONFIG_CLOCK_ADJTIME --cfg CONFIG_CLOOP --cfg CONFIG_CMPXCHG128 --cfg CONFIG_COROUTINE_POOL --cfg CONFIG_CPUID_H --cfg CONFIG_CURL --cfg CONFIG_CURSES --cfg CONFIG_DEBUG_MUTEX --cfg CONFIG_DEBUG_TCG --cfg CONFIG_DMG --cfg CONFIG_DUP3 --cfg CONFIG_EBPF --cfg CONFIG_EPOLL --cfg CONFIG_EPOLL_CREATE1 --cfg CONFIG_EVENTFD --cfg CONFIG_FALLOCATE --cfg CONFIG_FALLOCATE_PUNCH_HOLE --cfg CONFIG_FALLOCATE_ZERO_RANGE --cfg CONFIG_FDATASYNC --cfg CONFIG_FDT --cfg CONFIG_FIEMAP --cfg CONFIG_GBM --cfg CONFIG_GETAUXVAL --cfg CONFIG_GETRANDOM --cfg CONFIG_GETTID --cfg CONFIG_GIO --cfg CONFIG_GLUSTERFS --cfg CONFIG_GLUSTERFS_DISCARD --cfg CONFIG_GLUSTERFS_FALLOCATE --cfg CONFIG_GLUSTERFS_FTRUNCATE_HAS_STAT --cfg CONFIG_GLUSTERFS_IOCB_HAS_STAT --cfg CONFIG_GLUSTERFS_XLATOR_OPT --cfg CONFIG_GLUSTERFS_ZEROFILL --cfg CONFIG_GNUTLS --cfg CONFIG_GNUTLS_CRYPTO --cfg CONFIG_GTK --cfg CONFIG_GUEST_AGENT --cfg CONFIG_HAS_ENVIRON --cfg CONFIG_INOTIFY --cfg CONFIG_INOTIFY1 --cfg CONFIG_INT128 --cfg CONFIG_IOVEC --cfg CONFIG_L2TPV3 --cfg CONFIG_LIBCAP_NG --cfg CONFIG_LIBISCSI --cfg CONFIG_LIBNFS --cfg CONFIG_LIBPMEM --cfg CONFIG_LIBPULSE --cfg CONFIG_LIBSSH --cfg CONFIG_LIBUDEV --cfg CONFIG_LINUX --cfg CONFIG_LINUX_AIO --cfg CONFIG_LINUX_MAGIC_H --cfg CONFIG_LIVE_BLOCK_MIGRATION --cfg CONFIG_LZO --cfg CONFIG_MADVISE --cfg CONFIG_MALLOC_TRIM --cfg CONFIG_MEMFD --cfg CONFIG_MPATH --cfg CONFIG_MPATH_NEW_API --cfg CONFIG_NUMA --cfg CONFIG_OPENGL --cfg CONFIG_OPEN_BY_HANDLE --cfg CONFIG_PARALLELS --cfg CONFIG_PIPE2 --cfg CONFIG_PLUGIN --cfg CONFIG_POSIX --cfg CONFIG_POSIX_FALLOCATE --cfg CONFIG_POSIX_MADVISE --cfg CONFIG_POSIX_MEMALIGN --cfg CONFIG_PPOLL --cfg CONFIG_PRCTL_PR_SET_TIMERSLACK --cfg CONFIG_PREADV --cfg CONFIG_PTHREAD_SETNAME_NP_W_TID --cfg CONFIG_PVRDMA --cfg CONFIG_QCOW1 --cfg CONFIG_QED --cfg CONFIG_QOM_CAST_DEBUG --cfg CONFIG_RBD --cfg CONFIG_RDMA --cfg CONFIG_REPLICATION --cfg CONFIG_RTNETLINK --cfg CONFIG_SDL --cfg CONFIG_SDL_IMAGE --cfg CONFIG_SECCOMP --cfg CONFIG_SECRET_KEYRING --cfg CONFIG_SEM_TIMEDWAIT --cfg CONFIG_SENDFILE --cfg CONFIG_SETNS --cfg CONFIG_SIGNALFD --cfg CONFIG_SLIRP --cfg CONFIG_SLIRP_SMBD --cfg CONFIG_SNAPPY --cfg CONFIG_SPICE --cfg CONFIG_SPICE_PROTOCOL --cfg CONFIG_SPLICE --cfg CONFIG_STATX --cfg CONFIG_SYNCFS --cfg CONFIG_SYNC_FILE_RANGE --cfg CONFIG_SYSMACROS --cfg CONFIG_TCG --cfg CONFIG_THREAD_SETNAME_BYTHREAD --cfg CONFIG_TIMERFD --cfg CONFIG_TOOLS --cfg CONFIG_TPM --cfg CONFIG_TRACE_LOG --cfg CONFIG_USBFS --cfg CONFIG_USB_LIBUSB --cfg CONFIG_VALGRIND_H --cfg CONFIG_VDI --cfg CONFIG_VHOST_CRYPTO --cfg CONFIG_VHOST_KERNEL --cfg CONFIG_VHOST_NET --cfg CONFIG_VHOST_NET_USER --cfg CONFIG_VHOST_NET_VDPA --cfg CONFIG_VHOST_SCSI --cfg CONFIG_VHOST_USER --cfg CONFIG_VHOST_USER_BLK_SERVER --cfg CONFIG_VHOST_USER_FS --cfg CONFIG_VHOST_USER_VSOCK --cfg CONFIG_VHOST_VDPA --cfg CONFIG_VHOST_VSOCK --cfg CONFIG_VIRTFS --cfg CONFIG_VNC --cfg CONFIG_VNC_JPEG --cfg CONFIG_VNC_PNG --cfg CONFIG_VNC_SASL --cfg CONFIG_VVFAT --cfg CONFIG_WITH_RUST --cfg CONFIG_X11 --cfg CONFIG_XEN_BACKEND --cfg CONFIG_XFS --cfg CONFIG_XKBCOMMON --cfg CONFIG_ZSTD --cfg HAVE_BTRFS_H --cfg HAVE_COPY_FILE_RANGE --cfg HAVE_DRM_H --cfg HAVE_FSXATTR --cfg HAVE_HOST_BLOCK_DEVICE --cfg HAVE_IPPROTO_MPTCP --cfg HAVE_MLOCKALL --cfg HAVE_OPENPTY --cfg HAVE_PTY_H --cfg HAVE_STRCHRNUL --cfg HAVE_STRUCT_STAT_ST_ATIM --cfg HAVE_SYSTEM_FUNCTION --cfg HAVE_UTMPX --print native-static-libs
error: failed to select a version for the requirement `cc = "=1.0.70"`
candidate versions found which didn't match: 1.0.69
location searched: directory source `/work/armbru/qemu/rust/vendored` (which is replacing registry `https://github.com/rust-lang/crates.io-index`)
required by package `nix v0.20.1`
    ... which is depended on by `qga v0.1.0 (/work/armbru/qemu/qga)`
perhaps a crate was updated and forgotten to be re-vendored?
As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.

[3/55] Generating qga-qapi-cabi-rs with a custom command
FAILED: qga/qapi-cabi-rs 
/usr/bin/python3 /work/armbru/qemu/scripts/cargo_wrapper.py --configh /work/armbru/qemu/bld-x86/config-host.h /work/armbru/qemu/bld-x86/qga /work/armbru/qemu/qga /work/armbru/qemu/bld-x86 debug '' build-bin qapi-cabi-rs -- --cfg QAPI_CABI
error: the option `Z` is only accepted on the nightly compiler

Environment: MESON_CURRENT_BUILD_DIR=/work/armbru/qemu/bld-x86/qga MESON_BUILD_ROOT=/work/armbru/qemu/bld-x86 WINAPI_NO_BUNDLED_LIBRARIES=1
Command: cargo rustc --target-dir /work/armbru/qemu/bld-x86/qga/rs-target --manifest-path /work/armbru/qemu/qga/Cargo.toml --offline --bin qapi-cabi-rs -- --cfg CONFIG_ACCEPT4 --cfg CONFIG_AF_VSOCK --cfg CONFIG_ATOMIC64 --cfg CONFIG_ATTR --cfg CONFIG_AUDIO_OSS --cfg CONFIG_AUDIO_PA --cfg CONFIG_AVX2_OPT --cfg CONFIG_BDRV_RO_WHITELIST --cfg CONFIG_BDRV_RW_WHITELIST --cfg CONFIG_BOCHS --cfg CONFIG_BRLAPI --cfg CONFIG_BYTESWAP_H --cfg CONFIG_CAPSTONE --cfg CONFIG_CLOCK_ADJTIME --cfg CONFIG_CLOOP --cfg CONFIG_CMPXCHG128 --cfg CONFIG_COROUTINE_POOL --cfg CONFIG_CPUID_H --cfg CONFIG_CURL --cfg CONFIG_CURSES --cfg CONFIG_DEBUG_MUTEX --cfg CONFIG_DEBUG_TCG --cfg CONFIG_DMG --cfg CONFIG_DUP3 --cfg CONFIG_EBPF --cfg CONFIG_EPOLL --cfg CONFIG_EPOLL_CREATE1 --cfg CONFIG_EVENTFD --cfg CONFIG_FALLOCATE --cfg CONFIG_FALLOCATE_PUNCH_HOLE --cfg CONFIG_FALLOCATE_ZERO_RANGE --cfg CONFIG_FDATASYNC --cfg CONFIG_FDT --cfg CONFIG_FIEMAP --cfg CONFIG_GBM --cfg CONFIG_GETAUXVAL --cfg CONFIG_GETRANDOM --cfg CONFIG_GETTID --cfg CONFIG_GIO --cfg CONFIG_GLUSTERFS --cfg CONFIG_GLUSTERFS_DISCARD --cfg CONFIG_GLUSTERFS_FALLOCATE --cfg CONFIG_GLUSTERFS_FTRUNCATE_HAS_STAT --cfg CONFIG_GLUSTERFS_IOCB_HAS_STAT --cfg CONFIG_GLUSTERFS_XLATOR_OPT --cfg CONFIG_GLUSTERFS_ZEROFILL --cfg CONFIG_GNUTLS --cfg CONFIG_GNUTLS_CRYPTO --cfg CONFIG_GTK --cfg CONFIG_GUEST_AGENT --cfg CONFIG_HAS_ENVIRON --cfg CONFIG_INOTIFY --cfg CONFIG_INOTIFY1 --cfg CONFIG_INT128 --cfg CONFIG_IOVEC --cfg CONFIG_L2TPV3 --cfg CONFIG_LIBCAP_NG --cfg CONFIG_LIBISCSI --cfg CONFIG_LIBNFS --cfg CONFIG_LIBPMEM --cfg CONFIG_LIBPULSE --cfg CONFIG_LIBSSH --cfg CONFIG_LIBUDEV --cfg CONFIG_LINUX --cfg CONFIG_LINUX_AIO --cfg CONFIG_LINUX_MAGIC_H --cfg CONFIG_LIVE_BLOCK_MIGRATION --cfg CONFIG_LZO --cfg CONFIG_MADVISE --cfg CONFIG_MALLOC_TRIM --cfg CONFIG_MEMFD --cfg CONFIG_MPATH --cfg CONFIG_MPATH_NEW_API --cfg CONFIG_NUMA --cfg CONFIG_OPENGL --cfg CONFIG_OPEN_BY_HANDLE --cfg CONFIG_PARALLELS --cfg CONFIG_PIPE2 --cfg CONFIG_PLUGIN --cfg CONFIG_POSIX --cfg CONFIG_POSIX_FALLOCATE --cfg CONFIG_POSIX_MADVISE --cfg CONFIG_POSIX_MEMALIGN --cfg CONFIG_PPOLL --cfg CONFIG_PRCTL_PR_SET_TIMERSLACK --cfg CONFIG_PREADV --cfg CONFIG_PTHREAD_SETNAME_NP_W_TID --cfg CONFIG_PVRDMA --cfg CONFIG_QCOW1 --cfg CONFIG_QED --cfg CONFIG_QOM_CAST_DEBUG --cfg CONFIG_RBD --cfg CONFIG_RDMA --cfg CONFIG_REPLICATION --cfg CONFIG_RTNETLINK --cfg CONFIG_SDL --cfg CONFIG_SDL_IMAGE --cfg CONFIG_SECCOMP --cfg CONFIG_SECRET_KEYRING --cfg CONFIG_SEM_TIMEDWAIT --cfg CONFIG_SENDFILE --cfg CONFIG_SETNS --cfg CONFIG_SIGNALFD --cfg CONFIG_SLIRP --cfg CONFIG_SLIRP_SMBD --cfg CONFIG_SNAPPY --cfg CONFIG_SPICE --cfg CONFIG_SPICE_PROTOCOL --cfg CONFIG_SPLICE --cfg CONFIG_STATX --cfg CONFIG_SYNCFS --cfg CONFIG_SYNC_FILE_RANGE --cfg CONFIG_SYSMACROS --cfg CONFIG_TCG --cfg CONFIG_THREAD_SETNAME_BYTHREAD --cfg CONFIG_TIMERFD --cfg CONFIG_TOOLS --cfg CONFIG_TPM --cfg CONFIG_TRACE_LOG --cfg CONFIG_USBFS --cfg CONFIG_USB_LIBUSB --cfg CONFIG_VALGRIND_H --cfg CONFIG_VDI --cfg CONFIG_VHOST_CRYPTO --cfg CONFIG_VHOST_KERNEL --cfg CONFIG_VHOST_NET --cfg CONFIG_VHOST_NET_USER --cfg CONFIG_VHOST_NET_VDPA --cfg CONFIG_VHOST_SCSI --cfg CONFIG_VHOST_USER --cfg CONFIG_VHOST_USER_BLK_SERVER --cfg CONFIG_VHOST_USER_FS --cfg CONFIG_VHOST_USER_VSOCK --cfg CONFIG_VHOST_VDPA --cfg CONFIG_VHOST_VSOCK --cfg CONFIG_VIRTFS --cfg CONFIG_VNC --cfg CONFIG_VNC_JPEG --cfg CONFIG_VNC_PNG --cfg CONFIG_VNC_SASL --cfg CONFIG_VVFAT --cfg CONFIG_WITH_RUST --cfg CONFIG_X11 --cfg CONFIG_XEN_BACKEND --cfg CONFIG_XFS --cfg CONFIG_XKBCOMMON --cfg CONFIG_ZSTD --cfg HAVE_BTRFS_H --cfg HAVE_COPY_FILE_RANGE --cfg HAVE_DRM_H --cfg HAVE_FSXATTR --cfg HAVE_HOST_BLOCK_DEVICE --cfg HAVE_IPPROTO_MPTCP --cfg HAVE_MLOCKALL --cfg HAVE_OPENPTY --cfg HAVE_PTY_H --cfg HAVE_STRCHRNUL --cfg HAVE_STRUCT_STAT_ST_ATIM --cfg HAVE_SYSTEM_FUNCTION --cfg HAVE_UTMPX --cfg QAPI_CABI
error: failed to select a version for the requirement `cc = "=1.0.70"`
candidate versions found which didn't match: 1.0.69
location searched: directory source `/work/armbru/qemu/rust/vendored` (which is replacing registry `https://github.com/rust-lang/crates.io-index`)
required by package `nix v0.20.1`
    ... which is depended on by `qga v0.1.0 (/work/armbru/qemu/qga)`
perhaps a crate was updated and forgotten to be re-vendored?
As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.

[4/55] Generating Test Rust library with a custom command
FAILED: tests/libqemu_tests.args tests/libqemu_tests.a 
/usr/bin/python3 /work/armbru/qemu/scripts/cargo_wrapper.py --configh /work/armbru/qemu/bld-x86/config-host.h /work/armbru/qemu/bld-x86/tests /work/armbru/qemu/tests /work/armbru/qemu/bld-x86 debug '' build-lib
Environment: MESON_CURRENT_BUILD_DIR=/work/armbru/qemu/bld-x86/tests MESON_BUILD_ROOT=/work/armbru/qemu/bld-x86 WINAPI_NO_BUNDLED_LIBRARIES=1
Command: cargo rustc --target-dir /work/armbru/qemu/bld-x86/tests/rs-target --manifest-path /work/armbru/qemu/tests/Cargo.toml --offline --lib -- --cfg CONFIG_ACCEPT4 --cfg CONFIG_AF_VSOCK --cfg CONFIG_ATOMIC64 --cfg CONFIG_ATTR --cfg CONFIG_AUDIO_OSS --cfg CONFIG_AUDIO_PA --cfg CONFIG_AVX2_OPT --cfg CONFIG_BDRV_RO_WHITELIST --cfg CONFIG_BDRV_RW_WHITELIST --cfg CONFIG_BOCHS --cfg CONFIG_BRLAPI --cfg CONFIG_BYTESWAP_H --cfg CONFIG_CAPSTONE --cfg CONFIG_CLOCK_ADJTIME --cfg CONFIG_CLOOP --cfg CONFIG_CMPXCHG128 --cfg CONFIG_COROUTINE_POOL --cfg CONFIG_CPUID_H --cfg CONFIG_CURL --cfg CONFIG_CURSES --cfg CONFIG_DEBUG_MUTEX --cfg CONFIG_DEBUG_TCG --cfg CONFIG_DMG --cfg CONFIG_DUP3 --cfg CONFIG_EBPF --cfg CONFIG_EPOLL --cfg CONFIG_EPOLL_CREATE1 --cfg CONFIG_EVENTFD --cfg CONFIG_FALLOCATE --cfg CONFIG_FALLOCATE_PUNCH_HOLE --cfg CONFIG_FALLOCATE_ZERO_RANGE --cfg CONFIG_FDATASYNC --cfg CONFIG_FDT --cfg CONFIG_FIEMAP --cfg CONFIG_GBM --cfg CONFIG_GETAUXVAL --cfg CONFIG_GETRANDOM --cfg CONFIG_GETTID --cfg CONFIG_GIO --cfg CONFIG_GLUSTERFS --cfg CONFIG_GLUSTERFS_DISCARD --cfg CONFIG_GLUSTERFS_FALLOCATE --cfg CONFIG_GLUSTERFS_FTRUNCATE_HAS_STAT --cfg CONFIG_GLUSTERFS_IOCB_HAS_STAT --cfg CONFIG_GLUSTERFS_XLATOR_OPT --cfg CONFIG_GLUSTERFS_ZEROFILL --cfg CONFIG_GNUTLS --cfg CONFIG_GNUTLS_CRYPTO --cfg CONFIG_GTK --cfg CONFIG_GUEST_AGENT --cfg CONFIG_HAS_ENVIRON --cfg CONFIG_INOTIFY --cfg CONFIG_INOTIFY1 --cfg CONFIG_INT128 --cfg CONFIG_IOVEC --cfg CONFIG_L2TPV3 --cfg CONFIG_LIBCAP_NG --cfg CONFIG_LIBISCSI --cfg CONFIG_LIBNFS --cfg CONFIG_LIBPMEM --cfg CONFIG_LIBPULSE --cfg CONFIG_LIBSSH --cfg CONFIG_LIBUDEV --cfg CONFIG_LINUX --cfg CONFIG_LINUX_AIO --cfg CONFIG_LINUX_MAGIC_H --cfg CONFIG_LIVE_BLOCK_MIGRATION --cfg CONFIG_LZO --cfg CONFIG_MADVISE --cfg CONFIG_MALLOC_TRIM --cfg CONFIG_MEMFD --cfg CONFIG_MPATH --cfg CONFIG_MPATH_NEW_API --cfg CONFIG_NUMA --cfg CONFIG_OPENGL --cfg CONFIG_OPEN_BY_HANDLE --cfg CONFIG_PARALLELS --cfg CONFIG_PIPE2 --cfg CONFIG_PLUGIN --cfg CONFIG_POSIX --cfg CONFIG_POSIX_FALLOCATE --cfg CONFIG_POSIX_MADVISE --cfg CONFIG_POSIX_MEMALIGN --cfg CONFIG_PPOLL --cfg CONFIG_PRCTL_PR_SET_TIMERSLACK --cfg CONFIG_PREADV --cfg CONFIG_PTHREAD_SETNAME_NP_W_TID --cfg CONFIG_PVRDMA --cfg CONFIG_QCOW1 --cfg CONFIG_QED --cfg CONFIG_QOM_CAST_DEBUG --cfg CONFIG_RBD --cfg CONFIG_RDMA --cfg CONFIG_REPLICATION --cfg CONFIG_RTNETLINK --cfg CONFIG_SDL --cfg CONFIG_SDL_IMAGE --cfg CONFIG_SECCOMP --cfg CONFIG_SECRET_KEYRING --cfg CONFIG_SEM_TIMEDWAIT --cfg CONFIG_SENDFILE --cfg CONFIG_SETNS --cfg CONFIG_SIGNALFD --cfg CONFIG_SLIRP --cfg CONFIG_SLIRP_SMBD --cfg CONFIG_SNAPPY --cfg CONFIG_SPICE --cfg CONFIG_SPICE_PROTOCOL --cfg CONFIG_SPLICE --cfg CONFIG_STATX --cfg CONFIG_SYNCFS --cfg CONFIG_SYNC_FILE_RANGE --cfg CONFIG_SYSMACROS --cfg CONFIG_TCG --cfg CONFIG_THREAD_SETNAME_BYTHREAD --cfg CONFIG_TIMERFD --cfg CONFIG_TOOLS --cfg CONFIG_TPM --cfg CONFIG_TRACE_LOG --cfg CONFIG_USBFS --cfg CONFIG_USB_LIBUSB --cfg CONFIG_VALGRIND_H --cfg CONFIG_VDI --cfg CONFIG_VHOST_CRYPTO --cfg CONFIG_VHOST_KERNEL --cfg CONFIG_VHOST_NET --cfg CONFIG_VHOST_NET_USER --cfg CONFIG_VHOST_NET_VDPA --cfg CONFIG_VHOST_SCSI --cfg CONFIG_VHOST_USER --cfg CONFIG_VHOST_USER_BLK_SERVER --cfg CONFIG_VHOST_USER_FS --cfg CONFIG_VHOST_USER_VSOCK --cfg CONFIG_VHOST_VDPA --cfg CONFIG_VHOST_VSOCK --cfg CONFIG_VIRTFS --cfg CONFIG_VNC --cfg CONFIG_VNC_JPEG --cfg CONFIG_VNC_PNG --cfg CONFIG_VNC_SASL --cfg CONFIG_VVFAT --cfg CONFIG_WITH_RUST --cfg CONFIG_X11 --cfg CONFIG_XEN_BACKEND --cfg CONFIG_XFS --cfg CONFIG_XKBCOMMON --cfg CONFIG_ZSTD --cfg HAVE_BTRFS_H --cfg HAVE_COPY_FILE_RANGE --cfg HAVE_DRM_H --cfg HAVE_FSXATTR --cfg HAVE_HOST_BLOCK_DEVICE --cfg HAVE_IPPROTO_MPTCP --cfg HAVE_MLOCKALL --cfg HAVE_OPENPTY --cfg HAVE_PTY_H --cfg HAVE_STRCHRNUL --cfg HAVE_STRUCT_STAT_ST_ATIM --cfg HAVE_SYSTEM_FUNCTION --cfg HAVE_UTMPX --print native-static-libs
error: failed to select a version for the requirement `cc = "=1.0.70"`
candidate versions found which didn't match: 1.0.69
location searched: directory source `/work/armbru/qemu/rust/vendored` (which is replacing registry `https://github.com/rust-lang/crates.io-index`)
required by package `nix v0.20.1`
    ... which is depended on by `qga v0.1.0 (/work/armbru/qemu/qga)`
perhaps a crate was updated and forgotten to be re-vendored?
As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.

[5/55] Generating QAPI test (include) with a custom command
[6/20] Generating qapi-cabi-rs with a custom command
FAILED: tests/qapi-cabi-rs 
/usr/bin/python3 /work/armbru/qemu/scripts/cargo_wrapper.py --configh /work/armbru/qemu/bld-x86/config-host.h /work/armbru/qemu/bld-x86/tests /work/armbru/qemu/tests /work/armbru/qemu/bld-x86 debug '' build-bin qapi-cabi-rs -- --cfg QAPI_CABI
error: the option `Z` is only accepted on the nightly compiler

Environment: MESON_CURRENT_BUILD_DIR=/work/armbru/qemu/bld-x86/tests MESON_BUILD_ROOT=/work/armbru/qemu/bld-x86 WINAPI_NO_BUNDLED_LIBRARIES=1
Command: cargo rustc --target-dir /work/armbru/qemu/bld-x86/tests/rs-target --manifest-path /work/armbru/qemu/tests/Cargo.toml --offline --bin qapi-cabi-rs -- --cfg CONFIG_ACCEPT4 --cfg CONFIG_AF_VSOCK --cfg CONFIG_ATOMIC64 --cfg CONFIG_ATTR --cfg CONFIG_AUDIO_OSS --cfg CONFIG_AUDIO_PA --cfg CONFIG_AVX2_OPT --cfg CONFIG_BDRV_RO_WHITELIST --cfg CONFIG_BDRV_RW_WHITELIST --cfg CONFIG_BOCHS --cfg CONFIG_BRLAPI --cfg CONFIG_BYTESWAP_H --cfg CONFIG_CAPSTONE --cfg CONFIG_CLOCK_ADJTIME --cfg CONFIG_CLOOP --cfg CONFIG_CMPXCHG128 --cfg CONFIG_COROUTINE_POOL --cfg CONFIG_CPUID_H --cfg CONFIG_CURL --cfg CONFIG_CURSES --cfg CONFIG_DEBUG_MUTEX --cfg CONFIG_DEBUG_TCG --cfg CONFIG_DMG --cfg CONFIG_DUP3 --cfg CONFIG_EBPF --cfg CONFIG_EPOLL --cfg CONFIG_EPOLL_CREATE1 --cfg CONFIG_EVENTFD --cfg CONFIG_FALLOCATE --cfg CONFIG_FALLOCATE_PUNCH_HOLE --cfg CONFIG_FALLOCATE_ZERO_RANGE --cfg CONFIG_FDATASYNC --cfg CONFIG_FDT --cfg CONFIG_FIEMAP --cfg CONFIG_GBM --cfg CONFIG_GETAUXVAL --cfg CONFIG_GETRANDOM --cfg CONFIG_GETTID --cfg CONFIG_GIO --cfg CONFIG_GLUSTERFS --cfg CONFIG_GLUSTERFS_DISCARD --cfg CONFIG_GLUSTERFS_FALLOCATE --cfg CONFIG_GLUSTERFS_FTRUNCATE_HAS_STAT --cfg CONFIG_GLUSTERFS_IOCB_HAS_STAT --cfg CONFIG_GLUSTERFS_XLATOR_OPT --cfg CONFIG_GLUSTERFS_ZEROFILL --cfg CONFIG_GNUTLS --cfg CONFIG_GNUTLS_CRYPTO --cfg CONFIG_GTK --cfg CONFIG_GUEST_AGENT --cfg CONFIG_HAS_ENVIRON --cfg CONFIG_INOTIFY --cfg CONFIG_INOTIFY1 --cfg CONFIG_INT128 --cfg CONFIG_IOVEC --cfg CONFIG_L2TPV3 --cfg CONFIG_LIBCAP_NG --cfg CONFIG_LIBISCSI --cfg CONFIG_LIBNFS --cfg CONFIG_LIBPMEM --cfg CONFIG_LIBPULSE --cfg CONFIG_LIBSSH --cfg CONFIG_LIBUDEV --cfg CONFIG_LINUX --cfg CONFIG_LINUX_AIO --cfg CONFIG_LINUX_MAGIC_H --cfg CONFIG_LIVE_BLOCK_MIGRATION --cfg CONFIG_LZO --cfg CONFIG_MADVISE --cfg CONFIG_MALLOC_TRIM --cfg CONFIG_MEMFD --cfg CONFIG_MPATH --cfg CONFIG_MPATH_NEW_API --cfg CONFIG_NUMA --cfg CONFIG_OPENGL --cfg CONFIG_OPEN_BY_HANDLE --cfg CONFIG_PARALLELS --cfg CONFIG_PIPE2 --cfg CONFIG_PLUGIN --cfg CONFIG_POSIX --cfg CONFIG_POSIX_FALLOCATE --cfg CONFIG_POSIX_MADVISE --cfg CONFIG_POSIX_MEMALIGN --cfg CONFIG_PPOLL --cfg CONFIG_PRCTL_PR_SET_TIMERSLACK --cfg CONFIG_PREADV --cfg CONFIG_PTHREAD_SETNAME_NP_W_TID --cfg CONFIG_PVRDMA --cfg CONFIG_QCOW1 --cfg CONFIG_QED --cfg CONFIG_QOM_CAST_DEBUG --cfg CONFIG_RBD --cfg CONFIG_RDMA --cfg CONFIG_REPLICATION --cfg CONFIG_RTNETLINK --cfg CONFIG_SDL --cfg CONFIG_SDL_IMAGE --cfg CONFIG_SECCOMP --cfg CONFIG_SECRET_KEYRING --cfg CONFIG_SEM_TIMEDWAIT --cfg CONFIG_SENDFILE --cfg CONFIG_SETNS --cfg CONFIG_SIGNALFD --cfg CONFIG_SLIRP --cfg CONFIG_SLIRP_SMBD --cfg CONFIG_SNAPPY --cfg CONFIG_SPICE --cfg CONFIG_SPICE_PROTOCOL --cfg CONFIG_SPLICE --cfg CONFIG_STATX --cfg CONFIG_SYNCFS --cfg CONFIG_SYNC_FILE_RANGE --cfg CONFIG_SYSMACROS --cfg CONFIG_TCG --cfg CONFIG_THREAD_SETNAME_BYTHREAD --cfg CONFIG_TIMERFD --cfg CONFIG_TOOLS --cfg CONFIG_TPM --cfg CONFIG_TRACE_LOG --cfg CONFIG_USBFS --cfg CONFIG_USB_LIBUSB --cfg CONFIG_VALGRIND_H --cfg CONFIG_VDI --cfg CONFIG_VHOST_CRYPTO --cfg CONFIG_VHOST_KERNEL --cfg CONFIG_VHOST_NET --cfg CONFIG_VHOST_NET_USER --cfg CONFIG_VHOST_NET_VDPA --cfg CONFIG_VHOST_SCSI --cfg CONFIG_VHOST_USER --cfg CONFIG_VHOST_USER_BLK_SERVER --cfg CONFIG_VHOST_USER_FS --cfg CONFIG_VHOST_USER_VSOCK --cfg CONFIG_VHOST_VDPA --cfg CONFIG_VHOST_VSOCK --cfg CONFIG_VIRTFS --cfg CONFIG_VNC --cfg CONFIG_VNC_JPEG --cfg CONFIG_VNC_PNG --cfg CONFIG_VNC_SASL --cfg CONFIG_VVFAT --cfg CONFIG_WITH_RUST --cfg CONFIG_X11 --cfg CONFIG_XEN_BACKEND --cfg CONFIG_XFS --cfg CONFIG_XKBCOMMON --cfg CONFIG_ZSTD --cfg HAVE_BTRFS_H --cfg HAVE_COPY_FILE_RANGE --cfg HAVE_DRM_H --cfg HAVE_FSXATTR --cfg HAVE_HOST_BLOCK_DEVICE --cfg HAVE_IPPROTO_MPTCP --cfg HAVE_MLOCKALL --cfg HAVE_OPENPTY --cfg HAVE_PTY_H --cfg HAVE_STRCHRNUL --cfg HAVE_STRUCT_STAT_ST_ATIM --cfg HAVE_SYSTEM_FUNCTION --cfg HAVE_UTMPX --cfg QAPI_CABI
error: failed to select a version for the requirement `cc = "=1.0.70"`
candidate versions found which didn't match: 1.0.69
location searched: directory source `/work/armbru/qemu/rust/vendored` (which is replacing registry `https://github.com/rust-lang/crates.io-index`)
required by package `nix v0.20.1`
    ... which is depended on by `qga v0.1.0 (/work/armbru/qemu/qga)`
perhaps a crate was updated and forgotten to be re-vendored?
As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.

ninja: build stopped: cannot make progress due to previous errors.
make: *** [Makefile:156: run-ninja] Error 1
make: Target 'all' not remade because of errors.
make: Leaving directory '/work/armbru/qemu/bld-x86'



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples
  2021-09-09 10:31     ` Markus Armbruster
@ 2021-09-09 15:22       ` Marc-André Lureau
  0 siblings, 0 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-09 15:22 UTC (permalink / raw)
  To: Markus Armbruster
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU, Stefan Hajnoczi

[-- Attachment #1: Type: text/plain, Size: 22659 bytes --]

Hi

On Thu, Sep 9, 2021 at 2:31 PM Markus Armbruster <armbru@redhat.com> wrote:

> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
>
> > Hi
> >
> > On Wed, Sep 8, 2021 at 5:23 PM Markus Armbruster <armbru@redhat.com>
> wrote:
> >
> >> Build fails for me:
> >>
> >> make: Entering directory '/work/armbru/qemu/bld'
> >> config-host.mak is out-of-date, running configure
> >>   GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
> >> tests/fp/berkeley-softfloat-3 dtc capstone slirp
> >> fatal: remote error: upload-pack: not our ref
> >> 7077bbbd11a67d60062a9483f996113a349a4ca1
> >>
> >
> > Looks like I didn't update the repository on github. done
> >
> > I will update .gitmodule to use the repo on gitlab instead (gitlab.com:
> > marcandre.lureau/qemu-rust-vendored.git)
>
> Next error:
>
> make -k -C ~/work/qemu/bld-x86
> make: Entering directory '/work/armbru/qemu/bld-x86'
>   GIT     ui/keycodemapdb rust/vendored tests/fp/berkeley-testfloat-3
> tests/fp/berkeley-softfloat-3 dtc capstone slirp
> [1/71] Generating qemu-version.h with a custom command (wrapped by meson
> to capture output)
> [2/55] Generating cargo-qga with a custom command
> FAILED: qga/libqga.args qga/libqga.a
> /usr/bin/python3 /work/armbru/qemu/scripts/cargo_wrapper.py --configh
> /work/armbru/qemu/bld-x86/config-host.h /work/armbru/qemu/bld-x86/qga
> /work/armbru/qemu/qga /work/armbru/qemu/bld-x86 debug '' build-lib
> Environment: MESON_CURRENT_BUILD_DIR=/work/armbru/qemu/bld-x86/qga
> MESON_BUILD_ROOT=/work/armbru/qemu/bld-x86 WINAPI_NO_BUNDLED_LIBRARIES=1
> Command: cargo rustc --target-dir /work/armbru/qemu/bld-x86/qga/rs-target
> --manifest-path /work/armbru/qemu/qga/Cargo.toml --offline --lib -- --cfg
> CONFIG_ACCEPT4 --cfg CONFIG_AF_VSOCK --cfg CONFIG_ATOMIC64 --cfg
> CONFIG_ATTR --cfg CONFIG_AUDIO_OSS --cfg CONFIG_AUDIO_PA --cfg
> CONFIG_AVX2_OPT --cfg CONFIG_BDRV_RO_WHITELIST --cfg
> CONFIG_BDRV_RW_WHITELIST --cfg CONFIG_BOCHS --cfg CONFIG_BRLAPI --cfg
> CONFIG_BYTESWAP_H --cfg CONFIG_CAPSTONE --cfg CONFIG_CLOCK_ADJTIME --cfg
> CONFIG_CLOOP --cfg CONFIG_CMPXCHG128 --cfg CONFIG_COROUTINE_POOL --cfg
> CONFIG_CPUID_H --cfg CONFIG_CURL --cfg CONFIG_CURSES --cfg
> CONFIG_DEBUG_MUTEX --cfg CONFIG_DEBUG_TCG --cfg CONFIG_DMG --cfg
> CONFIG_DUP3 --cfg CONFIG_EBPF --cfg CONFIG_EPOLL --cfg CONFIG_EPOLL_CREATE1
> --cfg CONFIG_EVENTFD --cfg CONFIG_FALLOCATE --cfg
> CONFIG_FALLOCATE_PUNCH_HOLE --cfg CONFIG_FALLOCATE_ZERO_RANGE --cfg
> CONFIG_FDATASYNC --cfg CONFIG_FDT --cfg CONFIG_FIEMAP --cfg CONFIG_GBM
> --cfg CONFIG_GETAUXVAL --cfg CONFIG_GETRANDOM --cfg CONFIG_GETTID --cfg
> CONFIG_GIO --cfg CONFIG_GLUSTERFS --cfg CONFIG_GLUSTERFS_DISCARD --cfg
> CONFIG_GLUSTERFS_FALLOCATE --cfg CONFIG_GLUSTERFS_FTRUNCATE_HAS_STAT --cfg
> CONFIG_GLUSTERFS_IOCB_HAS_STAT --cfg CONFIG_GLUSTERFS_XLATOR_OPT --cfg
> CONFIG_GLUSTERFS_ZEROFILL --cfg CONFIG_GNUTLS --cfg CONFIG_GNUTLS_CRYPTO
> --cfg CONFIG_GTK --cfg CONFIG_GUEST_AGENT --cfg CONFIG_HAS_ENVIRON --cfg
> CONFIG_INOTIFY --cfg CONFIG_INOTIFY1 --cfg CONFIG_INT128 --cfg CONFIG_IOVEC
> --cfg CONFIG_L2TPV3 --cfg CONFIG_LIBCAP_NG --cfg CONFIG_LIBISCSI --cfg
> CONFIG_LIBNFS --cfg CONFIG_LIBPMEM --cfg CONFIG_LIBPULSE --cfg
> CONFIG_LIBSSH --cfg CONFIG_LIBUDEV --cfg CONFIG_LINUX --cfg
> CONFIG_LINUX_AIO --cfg CONFIG_LINUX_MAGIC_H --cfg
> CONFIG_LIVE_BLOCK_MIGRATION --cfg CONFIG_LZO --cfg CONFIG_MADVISE --cfg
> CONFIG_MALLOC_TRIM --cfg CONFIG_MEMFD --cfg CONFIG_MPATH --cfg
> CONFIG_MPATH_NEW_API --cfg CONFIG_NUMA --cfg CONFIG_OPENGL --cfg
> CONFIG_OPEN_BY_HANDLE --cfg CONFIG_PARALLELS --cfg CONFIG_PIPE2 --cfg
> CONFIG_PLUGIN --cfg CONFIG_POSIX --cfg CONFIG_POSIX_FALLOCATE --cfg
> CONFIG_POSIX_MADVISE --cfg CONFIG_POSIX_MEMALIGN --cfg CONFIG_PPOLL --cfg
> CONFIG_PRCTL_PR_SET_TIMERSLACK --cfg CONFIG_PREADV --cfg
> CONFIG_PTHREAD_SETNAME_NP_W_TID --cfg CONFIG_PVRDMA --cfg CONFIG_QCOW1
> --cfg CONFIG_QED --cfg CONFIG_QOM_CAST_DEBUG --cfg CONFIG_RBD --cfg
> CONFIG_RDMA --cfg CONFIG_REPLICATION --cfg CONFIG_RTNETLINK --cfg
> CONFIG_SDL --cfg CONFIG_SDL_IMAGE --cfg CONFIG_SECCOMP --cfg
> CONFIG_SECRET_KEYRING --cfg CONFIG_SEM_TIMEDWAIT --cfg CONFIG_SENDFILE
> --cfg CONFIG_SETNS --cfg CONFIG_SIGNALFD --cfg CONFIG_SLIRP --cfg
> CONFIG_SLIRP_SMBD --cfg CONFIG_SNAPPY --cfg CONFIG_SPICE --cfg
> CONFIG_SPICE_PROTOCOL --cfg CONFIG_SPLICE --cfg CONFIG_STATX --cfg
> CONFIG_SYNCFS --cfg CONFIG_SYNC_FILE_RANGE --cfg CONFIG_SYSMACROS --cfg
> CONFIG_TCG --cfg CONFIG_THREAD_SETNAME_BYTHREAD --cfg CONFIG_TIMERFD --cfg
> CONFIG_TOOLS --cfg CONFIG_TPM --cfg CONFIG_TRACE_LOG --cfg CONFIG_USBFS
> --cfg CONFIG_USB_LIBUSB --cfg CONFIG_VALGRIND_H --cfg CONFIG_VDI --cfg
> CONFIG_VHOST_CRYPTO --cfg CONFIG_VHOST_KERNEL --cfg CONFIG_VHOST_NET --cfg
> CONFIG_VHOST_NET_USER --cfg CONFIG_VHOST_NET_VDPA --cfg CONFIG_VHOST_SCSI
> --cfg CONFIG_VHOST_USER --cfg CONFIG_VHOST_USER_BLK_SERVER --cfg
> CONFIG_VHOST_USER_FS --cfg CONFIG_VHOST_USER_VSOCK --cfg CONFIG_VHOST_VDPA
> --cfg CONFIG_VHOST_VSOCK --cfg CONFIG_VIRTFS --cfg CONFIG_VNC --cfg
> CONFIG_VNC_JPEG --cfg CONFIG_VNC_PNG --cfg CONFIG_VNC_SASL --cfg
> CONFIG_VVFAT --cfg CONFIG_WITH_RUST --cfg CONFIG_X11 --cfg
> CONFIG_XEN_BACKEND --cfg CONFIG_XFS --cfg CONFIG_XKBCOMMON --cfg
> CONFIG_ZSTD --cfg HAVE_BTRFS_H --cfg HAVE_COPY_FILE_RANGE --cfg HAVE_DRM_H
> --cfg HAVE_FSXATTR --cfg HAVE_HOST_BLOCK_DEVICE --cfg HAVE_IPPROTO_MPTCP
> --cfg HAVE_MLOCKALL --cfg HAVE_OPENPTY --cfg HAVE_PTY_H --cfg
> HAVE_STRCHRNUL --cfg HAVE_STRUCT_STAT_ST_ATIM --cfg HAVE_SYSTEM_FUNCTION
> --cfg HAVE_UTMPX --print native-static-libs
> error: failed to select a version for the requirement `cc = "=1.0.70"`
> candidate versions found which didn't match: 1.0.69
> location searched: directory source `/work/armbru/qemu/rust/vendored`
> (which is replacing registry `
> https://github.com/rust-lang/crates.io-index`
> <https://github.com/rust-lang/crates.io-index>)
> required by package `nix v0.20.1`
>     ... which is depended on by `qga v0.1.0 (/work/armbru/qemu/qga)`
> perhaps a crate was updated and forgotten to be re-vendored?
> As a reminder, you're using offline mode (--offline) which can sometimes
> cause surprising resolution failures, if this error is too confusing you
> may wish to retry without the offline flag.
>
>
Yes, my bad. See discussion in "[RFC v3 13/32] rust: use vendored-sources".
If you reset rust/vendored to origin/master (and commit that to avoid sync
back), it should work.


> [3/55] Generating qga-qapi-cabi-rs with a custom command
> FAILED: qga/qapi-cabi-rs
> /usr/bin/python3 /work/armbru/qemu/scripts/cargo_wrapper.py --configh
> /work/armbru/qemu/bld-x86/config-host.h /work/armbru/qemu/bld-x86/qga
> /work/armbru/qemu/qga /work/armbru/qemu/bld-x86 debug '' build-bin
> qapi-cabi-rs -- --cfg QAPI_CABI
> error: the option `Z` is only accepted on the nightly compiler
>
> Environment: MESON_CURRENT_BUILD_DIR=/work/armbru/qemu/bld-x86/qga
> MESON_BUILD_ROOT=/work/armbru/qemu/bld-x86 WINAPI_NO_BUNDLED_LIBRARIES=1
> Command: cargo rustc --target-dir /work/armbru/qemu/bld-x86/qga/rs-target
> --manifest-path /work/armbru/qemu/qga/Cargo.toml --offline --bin
> qapi-cabi-rs -- --cfg CONFIG_ACCEPT4 --cfg CONFIG_AF_VSOCK --cfg
> CONFIG_ATOMIC64 --cfg CONFIG_ATTR --cfg CONFIG_AUDIO_OSS --cfg
> CONFIG_AUDIO_PA --cfg CONFIG_AVX2_OPT --cfg CONFIG_BDRV_RO_WHITELIST --cfg
> CONFIG_BDRV_RW_WHITELIST --cfg CONFIG_BOCHS --cfg CONFIG_BRLAPI --cfg
> CONFIG_BYTESWAP_H --cfg CONFIG_CAPSTONE --cfg CONFIG_CLOCK_ADJTIME --cfg
> CONFIG_CLOOP --cfg CONFIG_CMPXCHG128 --cfg CONFIG_COROUTINE_POOL --cfg
> CONFIG_CPUID_H --cfg CONFIG_CURL --cfg CONFIG_CURSES --cfg
> CONFIG_DEBUG_MUTEX --cfg CONFIG_DEBUG_TCG --cfg CONFIG_DMG --cfg
> CONFIG_DUP3 --cfg CONFIG_EBPF --cfg CONFIG_EPOLL --cfg CONFIG_EPOLL_CREATE1
> --cfg CONFIG_EVENTFD --cfg CONFIG_FALLOCATE --cfg
> CONFIG_FALLOCATE_PUNCH_HOLE --cfg CONFIG_FALLOCATE_ZERO_RANGE --cfg
> CONFIG_FDATASYNC --cfg CONFIG_FDT --cfg CONFIG_FIEMAP --cfg CONFIG_GBM
> --cfg CONFIG_GETAUXVAL --cfg CONFIG_GETRANDOM --cfg CONFIG_GETTID --cfg
> CONFIG_GIO --cfg CONFIG_GLUSTERFS --cfg CONFIG_GLUSTERFS_DISCARD --cfg
> CONFIG_GLUSTERFS_FALLOCATE --cfg CONFIG_GLUSTERFS_FTRUNCATE_HAS_STAT --cfg
> CONFIG_GLUSTERFS_IOCB_HAS_STAT --cfg CONFIG_GLUSTERFS_XLATOR_OPT --cfg
> CONFIG_GLUSTERFS_ZEROFILL --cfg CONFIG_GNUTLS --cfg CONFIG_GNUTLS_CRYPTO
> --cfg CONFIG_GTK --cfg CONFIG_GUEST_AGENT --cfg CONFIG_HAS_ENVIRON --cfg
> CONFIG_INOTIFY --cfg CONFIG_INOTIFY1 --cfg CONFIG_INT128 --cfg CONFIG_IOVEC
> --cfg CONFIG_L2TPV3 --cfg CONFIG_LIBCAP_NG --cfg CONFIG_LIBISCSI --cfg
> CONFIG_LIBNFS --cfg CONFIG_LIBPMEM --cfg CONFIG_LIBPULSE --cfg
> CONFIG_LIBSSH --cfg CONFIG_LIBUDEV --cfg CONFIG_LINUX --cfg
> CONFIG_LINUX_AIO --cfg CONFIG_LINUX_MAGIC_H --cfg
> CONFIG_LIVE_BLOCK_MIGRATION --cfg CONFIG_LZO --cfg CONFIG_MADVISE --cfg
> CONFIG_MALLOC_TRIM --cfg CONFIG_MEMFD --cfg CONFIG_MPATH --cfg
> CONFIG_MPATH_NEW_API --cfg CONFIG_NUMA --cfg CONFIG_OPENGL --cfg
> CONFIG_OPEN_BY_HANDLE --cfg CONFIG_PARALLELS --cfg CONFIG_PIPE2 --cfg
> CONFIG_PLUGIN --cfg CONFIG_POSIX --cfg CONFIG_POSIX_FALLOCATE --cfg
> CONFIG_POSIX_MADVISE --cfg CONFIG_POSIX_MEMALIGN --cfg CONFIG_PPOLL --cfg
> CONFIG_PRCTL_PR_SET_TIMERSLACK --cfg CONFIG_PREADV --cfg
> CONFIG_PTHREAD_SETNAME_NP_W_TID --cfg CONFIG_PVRDMA --cfg CONFIG_QCOW1
> --cfg CONFIG_QED --cfg CONFIG_QOM_CAST_DEBUG --cfg CONFIG_RBD --cfg
> CONFIG_RDMA --cfg CONFIG_REPLICATION --cfg CONFIG_RTNETLINK --cfg
> CONFIG_SDL --cfg CONFIG_SDL_IMAGE --cfg CONFIG_SECCOMP --cfg
> CONFIG_SECRET_KEYRING --cfg CONFIG_SEM_TIMEDWAIT --cfg CONFIG_SENDFILE
> --cfg CONFIG_SETNS --cfg CONFIG_SIGNALFD --cfg CONFIG_SLIRP --cfg
> CONFIG_SLIRP_SMBD --cfg CONFIG_SNAPPY --cfg CONFIG_SPICE --cfg
> CONFIG_SPICE_PROTOCOL --cfg CONFIG_SPLICE --cfg CONFIG_STATX --cfg
> CONFIG_SYNCFS --cfg CONFIG_SYNC_FILE_RANGE --cfg CONFIG_SYSMACROS --cfg
> CONFIG_TCG --cfg CONFIG_THREAD_SETNAME_BYTHREAD --cfg CONFIG_TIMERFD --cfg
> CONFIG_TOOLS --cfg CONFIG_TPM --cfg CONFIG_TRACE_LOG --cfg CONFIG_USBFS
> --cfg CONFIG_USB_LIBUSB --cfg CONFIG_VALGRIND_H --cfg CONFIG_VDI --cfg
> CONFIG_VHOST_CRYPTO --cfg CONFIG_VHOST_KERNEL --cfg CONFIG_VHOST_NET --cfg
> CONFIG_VHOST_NET_USER --cfg CONFIG_VHOST_NET_VDPA --cfg CONFIG_VHOST_SCSI
> --cfg CONFIG_VHOST_USER --cfg CONFIG_VHOST_USER_BLK_SERVER --cfg
> CONFIG_VHOST_USER_FS --cfg CONFIG_VHOST_USER_VSOCK --cfg CONFIG_VHOST_VDPA
> --cfg CONFIG_VHOST_VSOCK --cfg CONFIG_VIRTFS --cfg CONFIG_VNC --cfg
> CONFIG_VNC_JPEG --cfg CONFIG_VNC_PNG --cfg CONFIG_VNC_SASL --cfg
> CONFIG_VVFAT --cfg CONFIG_WITH_RUST --cfg CONFIG_X11 --cfg
> CONFIG_XEN_BACKEND --cfg CONFIG_XFS --cfg CONFIG_XKBCOMMON --cfg
> CONFIG_ZSTD --cfg HAVE_BTRFS_H --cfg HAVE_COPY_FILE_RANGE --cfg HAVE_DRM_H
> --cfg HAVE_FSXATTR --cfg HAVE_HOST_BLOCK_DEVICE --cfg HAVE_IPPROTO_MPTCP
> --cfg HAVE_MLOCKALL --cfg HAVE_OPENPTY --cfg HAVE_PTY_H --cfg
> HAVE_STRCHRNUL --cfg HAVE_STRUCT_STAT_ST_ATIM --cfg HAVE_SYSTEM_FUNCTION
> --cfg HAVE_UTMPX --cfg QAPI_CABI
> error: failed to select a version for the requirement `cc = "=1.0.70"`
> candidate versions found which didn't match: 1.0.69
> location searched: directory source `/work/armbru/qemu/rust/vendored`
> (which is replacing registry `
> https://github.com/rust-lang/crates.io-index`
> <https://github.com/rust-lang/crates.io-index>)
> required by package `nix v0.20.1`
>     ... which is depended on by `qga v0.1.0 (/work/armbru/qemu/qga)`
> perhaps a crate was updated and forgotten to be re-vendored?
> As a reminder, you're using offline mode (--offline) which can sometimes
> cause surprising resolution failures, if this error is too confusing you
> may wish to retry without the offline flag.
>
> [4/55] Generating Test Rust library with a custom command
> FAILED: tests/libqemu_tests.args tests/libqemu_tests.a
> /usr/bin/python3 /work/armbru/qemu/scripts/cargo_wrapper.py --configh
> /work/armbru/qemu/bld-x86/config-host.h /work/armbru/qemu/bld-x86/tests
> /work/armbru/qemu/tests /work/armbru/qemu/bld-x86 debug '' build-lib
> Environment: MESON_CURRENT_BUILD_DIR=/work/armbru/qemu/bld-x86/tests
> MESON_BUILD_ROOT=/work/armbru/qemu/bld-x86 WINAPI_NO_BUNDLED_LIBRARIES=1
> Command: cargo rustc --target-dir
> /work/armbru/qemu/bld-x86/tests/rs-target --manifest-path
> /work/armbru/qemu/tests/Cargo.toml --offline --lib -- --cfg CONFIG_ACCEPT4
> --cfg CONFIG_AF_VSOCK --cfg CONFIG_ATOMIC64 --cfg CONFIG_ATTR --cfg
> CONFIG_AUDIO_OSS --cfg CONFIG_AUDIO_PA --cfg CONFIG_AVX2_OPT --cfg
> CONFIG_BDRV_RO_WHITELIST --cfg CONFIG_BDRV_RW_WHITELIST --cfg CONFIG_BOCHS
> --cfg CONFIG_BRLAPI --cfg CONFIG_BYTESWAP_H --cfg CONFIG_CAPSTONE --cfg
> CONFIG_CLOCK_ADJTIME --cfg CONFIG_CLOOP --cfg CONFIG_CMPXCHG128 --cfg
> CONFIG_COROUTINE_POOL --cfg CONFIG_CPUID_H --cfg CONFIG_CURL --cfg
> CONFIG_CURSES --cfg CONFIG_DEBUG_MUTEX --cfg CONFIG_DEBUG_TCG --cfg
> CONFIG_DMG --cfg CONFIG_DUP3 --cfg CONFIG_EBPF --cfg CONFIG_EPOLL --cfg
> CONFIG_EPOLL_CREATE1 --cfg CONFIG_EVENTFD --cfg CONFIG_FALLOCATE --cfg
> CONFIG_FALLOCATE_PUNCH_HOLE --cfg CONFIG_FALLOCATE_ZERO_RANGE --cfg
> CONFIG_FDATASYNC --cfg CONFIG_FDT --cfg CONFIG_FIEMAP --cfg CONFIG_GBM
> --cfg CONFIG_GETAUXVAL --cfg CONFIG_GETRANDOM --cfg CONFIG_GETTID --cfg
> CONFIG_GIO --cfg CONFIG_GLUSTERFS --cfg CONFIG_GLUSTERFS_DISCARD --cfg
> CONFIG_GLUSTERFS_FALLOCATE --cfg CONFIG_GLUSTERFS_FTRUNCATE_HAS_STAT --cfg
> CONFIG_GLUSTERFS_IOCB_HAS_STAT --cfg CONFIG_GLUSTERFS_XLATOR_OPT --cfg
> CONFIG_GLUSTERFS_ZEROFILL --cfg CONFIG_GNUTLS --cfg CONFIG_GNUTLS_CRYPTO
> --cfg CONFIG_GTK --cfg CONFIG_GUEST_AGENT --cfg CONFIG_HAS_ENVIRON --cfg
> CONFIG_INOTIFY --cfg CONFIG_INOTIFY1 --cfg CONFIG_INT128 --cfg CONFIG_IOVEC
> --cfg CONFIG_L2TPV3 --cfg CONFIG_LIBCAP_NG --cfg CONFIG_LIBISCSI --cfg
> CONFIG_LIBNFS --cfg CONFIG_LIBPMEM --cfg CONFIG_LIBPULSE --cfg
> CONFIG_LIBSSH --cfg CONFIG_LIBUDEV --cfg CONFIG_LINUX --cfg
> CONFIG_LINUX_AIO --cfg CONFIG_LINUX_MAGIC_H --cfg
> CONFIG_LIVE_BLOCK_MIGRATION --cfg CONFIG_LZO --cfg CONFIG_MADVISE --cfg
> CONFIG_MALLOC_TRIM --cfg CONFIG_MEMFD --cfg CONFIG_MPATH --cfg
> CONFIG_MPATH_NEW_API --cfg CONFIG_NUMA --cfg CONFIG_OPENGL --cfg
> CONFIG_OPEN_BY_HANDLE --cfg CONFIG_PARALLELS --cfg CONFIG_PIPE2 --cfg
> CONFIG_PLUGIN --cfg CONFIG_POSIX --cfg CONFIG_POSIX_FALLOCATE --cfg
> CONFIG_POSIX_MADVISE --cfg CONFIG_POSIX_MEMALIGN --cfg CONFIG_PPOLL --cfg
> CONFIG_PRCTL_PR_SET_TIMERSLACK --cfg CONFIG_PREADV --cfg
> CONFIG_PTHREAD_SETNAME_NP_W_TID --cfg CONFIG_PVRDMA --cfg CONFIG_QCOW1
> --cfg CONFIG_QED --cfg CONFIG_QOM_CAST_DEBUG --cfg CONFIG_RBD --cfg
> CONFIG_RDMA --cfg CONFIG_REPLICATION --cfg CONFIG_RTNETLINK --cfg
> CONFIG_SDL --cfg CONFIG_SDL_IMAGE --cfg CONFIG_SECCOMP --cfg
> CONFIG_SECRET_KEYRING --cfg CONFIG_SEM_TIMEDWAIT --cfg CONFIG_SENDFILE
> --cfg CONFIG_SETNS --cfg CONFIG_SIGNALFD --cfg CONFIG_SLIRP --cfg
> CONFIG_SLIRP_SMBD --cfg CONFIG_SNAPPY --cfg CONFIG_SPICE --cfg
> CONFIG_SPICE_PROTOCOL --cfg CONFIG_SPLICE --cfg CONFIG_STATX --cfg
> CONFIG_SYNCFS --cfg CONFIG_SYNC_FILE_RANGE --cfg CONFIG_SYSMACROS --cfg
> CONFIG_TCG --cfg CONFIG_THREAD_SETNAME_BYTHREAD --cfg CONFIG_TIMERFD --cfg
> CONFIG_TOOLS --cfg CONFIG_TPM --cfg CONFIG_TRACE_LOG --cfg CONFIG_USBFS
> --cfg CONFIG_USB_LIBUSB --cfg CONFIG_VALGRIND_H --cfg CONFIG_VDI --cfg
> CONFIG_VHOST_CRYPTO --cfg CONFIG_VHOST_KERNEL --cfg CONFIG_VHOST_NET --cfg
> CONFIG_VHOST_NET_USER --cfg CONFIG_VHOST_NET_VDPA --cfg CONFIG_VHOST_SCSI
> --cfg CONFIG_VHOST_USER --cfg CONFIG_VHOST_USER_BLK_SERVER --cfg
> CONFIG_VHOST_USER_FS --cfg CONFIG_VHOST_USER_VSOCK --cfg CONFIG_VHOST_VDPA
> --cfg CONFIG_VHOST_VSOCK --cfg CONFIG_VIRTFS --cfg CONFIG_VNC --cfg
> CONFIG_VNC_JPEG --cfg CONFIG_VNC_PNG --cfg CONFIG_VNC_SASL --cfg
> CONFIG_VVFAT --cfg CONFIG_WITH_RUST --cfg CONFIG_X11 --cfg
> CONFIG_XEN_BACKEND --cfg CONFIG_XFS --cfg CONFIG_XKBCOMMON --cfg
> CONFIG_ZSTD --cfg HAVE_BTRFS_H --cfg HAVE_COPY_FILE_RANGE --cfg HAVE_DRM_H
> --cfg HAVE_FSXATTR --cfg HAVE_HOST_BLOCK_DEVICE --cfg HAVE_IPPROTO_MPTCP
> --cfg HAVE_MLOCKALL --cfg HAVE_OPENPTY --cfg HAVE_PTY_H --cfg
> HAVE_STRCHRNUL --cfg HAVE_STRUCT_STAT_ST_ATIM --cfg HAVE_SYSTEM_FUNCTION
> --cfg HAVE_UTMPX --print native-static-libs
> error: failed to select a version for the requirement `cc = "=1.0.70"`
> candidate versions found which didn't match: 1.0.69
> location searched: directory source `/work/armbru/qemu/rust/vendored`
> (which is replacing registry `
> https://github.com/rust-lang/crates.io-index`
> <https://github.com/rust-lang/crates.io-index>)
> required by package `nix v0.20.1`
>     ... which is depended on by `qga v0.1.0 (/work/armbru/qemu/qga)`
> perhaps a crate was updated and forgotten to be re-vendored?
> As a reminder, you're using offline mode (--offline) which can sometimes
> cause surprising resolution failures, if this error is too confusing you
> may wish to retry without the offline flag.
>
> [5/55] Generating QAPI test (include) with a custom command
> [6/20] Generating qapi-cabi-rs with a custom command
> FAILED: tests/qapi-cabi-rs
> /usr/bin/python3 /work/armbru/qemu/scripts/cargo_wrapper.py --configh
> /work/armbru/qemu/bld-x86/config-host.h /work/armbru/qemu/bld-x86/tests
> /work/armbru/qemu/tests /work/armbru/qemu/bld-x86 debug '' build-bin
> qapi-cabi-rs -- --cfg QAPI_CABI
> error: the option `Z` is only accepted on the nightly compiler
>
> Environment: MESON_CURRENT_BUILD_DIR=/work/armbru/qemu/bld-x86/tests
> MESON_BUILD_ROOT=/work/armbru/qemu/bld-x86 WINAPI_NO_BUNDLED_LIBRARIES=1
> Command: cargo rustc --target-dir
> /work/armbru/qemu/bld-x86/tests/rs-target --manifest-path
> /work/armbru/qemu/tests/Cargo.toml --offline --bin qapi-cabi-rs -- --cfg
> CONFIG_ACCEPT4 --cfg CONFIG_AF_VSOCK --cfg CONFIG_ATOMIC64 --cfg
> CONFIG_ATTR --cfg CONFIG_AUDIO_OSS --cfg CONFIG_AUDIO_PA --cfg
> CONFIG_AVX2_OPT --cfg CONFIG_BDRV_RO_WHITELIST --cfg
> CONFIG_BDRV_RW_WHITELIST --cfg CONFIG_BOCHS --cfg CONFIG_BRLAPI --cfg
> CONFIG_BYTESWAP_H --cfg CONFIG_CAPSTONE --cfg CONFIG_CLOCK_ADJTIME --cfg
> CONFIG_CLOOP --cfg CONFIG_CMPXCHG128 --cfg CONFIG_COROUTINE_POOL --cfg
> CONFIG_CPUID_H --cfg CONFIG_CURL --cfg CONFIG_CURSES --cfg
> CONFIG_DEBUG_MUTEX --cfg CONFIG_DEBUG_TCG --cfg CONFIG_DMG --cfg
> CONFIG_DUP3 --cfg CONFIG_EBPF --cfg CONFIG_EPOLL --cfg CONFIG_EPOLL_CREATE1
> --cfg CONFIG_EVENTFD --cfg CONFIG_FALLOCATE --cfg
> CONFIG_FALLOCATE_PUNCH_HOLE --cfg CONFIG_FALLOCATE_ZERO_RANGE --cfg
> CONFIG_FDATASYNC --cfg CONFIG_FDT --cfg CONFIG_FIEMAP --cfg CONFIG_GBM
> --cfg CONFIG_GETAUXVAL --cfg CONFIG_GETRANDOM --cfg CONFIG_GETTID --cfg
> CONFIG_GIO --cfg CONFIG_GLUSTERFS --cfg CONFIG_GLUSTERFS_DISCARD --cfg
> CONFIG_GLUSTERFS_FALLOCATE --cfg CONFIG_GLUSTERFS_FTRUNCATE_HAS_STAT --cfg
> CONFIG_GLUSTERFS_IOCB_HAS_STAT --cfg CONFIG_GLUSTERFS_XLATOR_OPT --cfg
> CONFIG_GLUSTERFS_ZEROFILL --cfg CONFIG_GNUTLS --cfg CONFIG_GNUTLS_CRYPTO
> --cfg CONFIG_GTK --cfg CONFIG_GUEST_AGENT --cfg CONFIG_HAS_ENVIRON --cfg
> CONFIG_INOTIFY --cfg CONFIG_INOTIFY1 --cfg CONFIG_INT128 --cfg CONFIG_IOVEC
> --cfg CONFIG_L2TPV3 --cfg CONFIG_LIBCAP_NG --cfg CONFIG_LIBISCSI --cfg
> CONFIG_LIBNFS --cfg CONFIG_LIBPMEM --cfg CONFIG_LIBPULSE --cfg
> CONFIG_LIBSSH --cfg CONFIG_LIBUDEV --cfg CONFIG_LINUX --cfg
> CONFIG_LINUX_AIO --cfg CONFIG_LINUX_MAGIC_H --cfg
> CONFIG_LIVE_BLOCK_MIGRATION --cfg CONFIG_LZO --cfg CONFIG_MADVISE --cfg
> CONFIG_MALLOC_TRIM --cfg CONFIG_MEMFD --cfg CONFIG_MPATH --cfg
> CONFIG_MPATH_NEW_API --cfg CONFIG_NUMA --cfg CONFIG_OPENGL --cfg
> CONFIG_OPEN_BY_HANDLE --cfg CONFIG_PARALLELS --cfg CONFIG_PIPE2 --cfg
> CONFIG_PLUGIN --cfg CONFIG_POSIX --cfg CONFIG_POSIX_FALLOCATE --cfg
> CONFIG_POSIX_MADVISE --cfg CONFIG_POSIX_MEMALIGN --cfg CONFIG_PPOLL --cfg
> CONFIG_PRCTL_PR_SET_TIMERSLACK --cfg CONFIG_PREADV --cfg
> CONFIG_PTHREAD_SETNAME_NP_W_TID --cfg CONFIG_PVRDMA --cfg CONFIG_QCOW1
> --cfg CONFIG_QED --cfg CONFIG_QOM_CAST_DEBUG --cfg CONFIG_RBD --cfg
> CONFIG_RDMA --cfg CONFIG_REPLICATION --cfg CONFIG_RTNETLINK --cfg
> CONFIG_SDL --cfg CONFIG_SDL_IMAGE --cfg CONFIG_SECCOMP --cfg
> CONFIG_SECRET_KEYRING --cfg CONFIG_SEM_TIMEDWAIT --cfg CONFIG_SENDFILE
> --cfg CONFIG_SETNS --cfg CONFIG_SIGNALFD --cfg CONFIG_SLIRP --cfg
> CONFIG_SLIRP_SMBD --cfg CONFIG_SNAPPY --cfg CONFIG_SPICE --cfg
> CONFIG_SPICE_PROTOCOL --cfg CONFIG_SPLICE --cfg CONFIG_STATX --cfg
> CONFIG_SYNCFS --cfg CONFIG_SYNC_FILE_RANGE --cfg CONFIG_SYSMACROS --cfg
> CONFIG_TCG --cfg CONFIG_THREAD_SETNAME_BYTHREAD --cfg CONFIG_TIMERFD --cfg
> CONFIG_TOOLS --cfg CONFIG_TPM --cfg CONFIG_TRACE_LOG --cfg CONFIG_USBFS
> --cfg CONFIG_USB_LIBUSB --cfg CONFIG_VALGRIND_H --cfg CONFIG_VDI --cfg
> CONFIG_VHOST_CRYPTO --cfg CONFIG_VHOST_KERNEL --cfg CONFIG_VHOST_NET --cfg
> CONFIG_VHOST_NET_USER --cfg CONFIG_VHOST_NET_VDPA --cfg CONFIG_VHOST_SCSI
> --cfg CONFIG_VHOST_USER --cfg CONFIG_VHOST_USER_BLK_SERVER --cfg
> CONFIG_VHOST_USER_FS --cfg CONFIG_VHOST_USER_VSOCK --cfg CONFIG_VHOST_VDPA
> --cfg CONFIG_VHOST_VSOCK --cfg CONFIG_VIRTFS --cfg CONFIG_VNC --cfg
> CONFIG_VNC_JPEG --cfg CONFIG_VNC_PNG --cfg CONFIG_VNC_SASL --cfg
> CONFIG_VVFAT --cfg CONFIG_WITH_RUST --cfg CONFIG_X11 --cfg
> CONFIG_XEN_BACKEND --cfg CONFIG_XFS --cfg CONFIG_XKBCOMMON --cfg
> CONFIG_ZSTD --cfg HAVE_BTRFS_H --cfg HAVE_COPY_FILE_RANGE --cfg HAVE_DRM_H
> --cfg HAVE_FSXATTR --cfg HAVE_HOST_BLOCK_DEVICE --cfg HAVE_IPPROTO_MPTCP
> --cfg HAVE_MLOCKALL --cfg HAVE_OPENPTY --cfg HAVE_PTY_H --cfg
> HAVE_STRCHRNUL --cfg HAVE_STRUCT_STAT_ST_ATIM --cfg HAVE_SYSTEM_FUNCTION
> --cfg HAVE_UTMPX --cfg QAPI_CABI
> error: failed to select a version for the requirement `cc = "=1.0.70"`
> candidate versions found which didn't match: 1.0.69
> location searched: directory source `/work/armbru/qemu/rust/vendored`
> (which is replacing registry `
> https://github.com/rust-lang/crates.io-index`
> <https://github.com/rust-lang/crates.io-index>)
> required by package `nix v0.20.1`
>     ... which is depended on by `qga v0.1.0 (/work/armbru/qemu/qga)`
> perhaps a crate was updated and forgotten to be re-vendored?
> As a reminder, you're using offline mode (--offline) which can sometimes
> cause surprising resolution failures, if this error is too confusing you
> may wish to retry without the offline flag.
>
> ninja: build stopped: cannot make progress due to previous errors.
> make: *** [Makefile:156: run-ninja] Error 1
> make: Target 'all' not remade because of errors.
> make: Leaving directory '/work/armbru/qemu/bld-x86'
>
>

-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 23204 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-07 12:19 ` [RFC v3 13/32] rust: use vendored-sources marcandre.lureau
  2021-09-08 15:38   ` Ian Jackson
@ 2021-09-09 16:02   ` Peter Maydell
  2021-09-09 16:29     ` Marc-André Lureau
  2021-09-09 16:49     ` Daniel P. Berrangé
  1 sibling, 2 replies; 74+ messages in thread
From: Peter Maydell @ 2021-09-09 16:02 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Daniel P. Berrange, QEMU Developers, Markus Armbruster,
	Ian Jackson, Stefan Hajnoczi, Paolo Bonzini

On Tue, 7 Sept 2021 at 13:32, <marcandre.lureau@redhat.com> wrote:
>
> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> Most likely, QEMU will want tighter control over the sources, rather
> than relying on crates.io downloading, use a git submodule with all the
> dependencies. However, cargo --offline was added in 1.36.
>
> "cargo vendor" helps gathering and updating the dependencies.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  configure                 | 8 ++++++++
>  meson.build               | 7 ++++++-
>  .cargo/config.toml.in     | 5 +++++
>  .cargo/meson.build        | 5 +++++
>  .gitmodules               | 4 ++++
>  rust/vendored             | 1 +
>  scripts/archive-source.sh | 2 +-
>  scripts/cargo_wrapper.py  | 1 +
>  8 files changed, 31 insertions(+), 2 deletions(-)
>  create mode 100644 .cargo/config.toml.in
>  create mode 100644 .cargo/meson.build
>  create mode 160000 rust/vendored

So, this is a lot of extra code in a submodule. Historically we've
found that submodules are a colossal pain, and so I think we should
think about whether we really want to have all our rust dependencies
in a submodule forever.

I am definitely only at the beginner stage with Rust, but I think
we should have a discussion about what the different alternative
options are here, and what we want to achieve, so that we know
why we're doing this and what we're gaining from the pain...

For instance, could we instead commit Cargo.lock in git and
use that to nail down specific versions of the dependencies ?

FWIW, the "why submodules" for the C dependencies we ship
like that is basically
 * C doesn't have a package manager, so if we need a dependency that
   distros don't ship then we need to wrap it up and provide it ourselves
 * where we ship binary blobs (guest BIOS etc) we want to also ship
   the source code for those blobs
I think for Rust dependencies those don't really apply.

Overall, I think that to the extent that we can look like a "normal"
user of Rust, that's a good plan. Distros may well want to be able
to do "build against our packaged rust stuff rather than downloading
from crates.io" but I imagine they have machinery for that already;
if we act like most other Rust programs we have better chances of
not breaking that machinery.

We do already effectively do "download code when QEMU is built" --
the makefile invokes scripts/git-submodule-update which pulls
down submodule code. (Thanks to Ian for pointing out this framing
of the question.)

(I'm not personally a fan of the "download everything from crates.io"
Rust ecosystem, but it is what it is, and wishing the Rust world
worked more like a trad Linux-distro-provides-all-your-dependencies
isn't, alas, going to make it so :-))

thanks
-- PMM


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-09 16:02   ` Peter Maydell
@ 2021-09-09 16:29     ` Marc-André Lureau
  2021-09-09 16:53       ` Daniel P. Berrangé
  2021-09-13 14:21       ` Paolo Bonzini
  2021-09-09 16:49     ` Daniel P. Berrangé
  1 sibling, 2 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-09 16:29 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Daniel P. Berrange, QEMU Developers, Markus Armbruster,
	Ian Jackson, Stefan Hajnoczi, Paolo Bonzini

[-- Attachment #1: Type: text/plain, Size: 3923 bytes --]

Hi

On Thu, Sep 9, 2021 at 8:04 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Tue, 7 Sept 2021 at 13:32, <marcandre.lureau@redhat.com> wrote:
> >
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Most likely, QEMU will want tighter control over the sources, rather
> > than relying on crates.io downloading, use a git submodule with all the
> > dependencies. However, cargo --offline was added in 1.36.
> >
> > "cargo vendor" helps gathering and updating the dependencies.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  configure                 | 8 ++++++++
> >  meson.build               | 7 ++++++-
> >  .cargo/config.toml.in     | 5 +++++
> >  .cargo/meson.build        | 5 +++++
> >  .gitmodules               | 4 ++++
> >  rust/vendored             | 1 +
> >  scripts/archive-source.sh | 2 +-
> >  scripts/cargo_wrapper.py  | 1 +
> >  8 files changed, 31 insertions(+), 2 deletions(-)
> >  create mode 100644 .cargo/config.toml.in
> >  create mode 100644 .cargo/meson.build
> >  create mode 160000 rust/vendored
>
> So, this is a lot of extra code in a submodule. Historically we've
> found that submodules are a colossal pain, and so I think we should
> think about whether we really want to have all our rust dependencies
> in a submodule forever.
>
> I am definitely only at the beginner stage with Rust, but I think
> we should have a discussion about what the different alternative
> options are here, and what we want to achieve, so that we know
> why we're doing this and what we're gaining from the pain...
>
> For instance, could we instead commit Cargo.lock in git and
> use that to nail down specific versions of the dependencies ?
>
>
Yes, that's the main reason this file exists I think.

FWIW, the "why submodules" for the C dependencies we ship
> like that is basically
>  * C doesn't have a package manager, so if we need a dependency that
>    distros don't ship then we need to wrap it up and provide it ourselves
>

Have we considered meson wrap? I never really looked at it in detail, not
sure if that would work for us.
https://mesonbuild.com/Wrap-dependency-system-manual.html

 * where we ship binary blobs (guest BIOS etc) we want to also ship
>    the source code for those blobs
> I think for Rust dependencies those don't really apply.
>

And we mirror all those dependencies at the same location.


> Overall, I think that to the extent that we can look like a "normal"
> user of Rust, that's a good plan. Distros may well want to be able
> to do "build against our packaged rust stuff rather than downloading
> from crates.io" but I imagine they have machinery for that already;
> if we act like most other Rust programs we have better chances of
> not breaking that machinery.
>

True, at least on Fedora, there is machinery to package "regular" Rust
programs/crates in an automated way.  Vendoring dependencies should work
equally, but may not conform with distro policies, so they have extra work
eventually (it seems vendoring is more and more common though, with go
projects for example)


> We do already effectively do "download code when QEMU is built" --
> the makefile invokes scripts/git-submodule-update which pulls
> down submodule code. (Thanks to Ian for pointing out this framing
> of the question.)
>
> (I'm not personally a fan of the "download everything from crates.io"
> Rust ecosystem, but it is what it is, and wishing the Rust world
> worked more like a trad Linux-distro-provides-all-your-dependencies
> isn't, alas, going to make it so :-))
>
>
A nice alternative to vendoring that could work well for QEMU is to mirror
the Rust crate we use, so we have similar control and guarantee over them
as submodules, and use `[patch.crates-io]` to point at qemu-project
locations.


-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 5979 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-09 16:02   ` Peter Maydell
  2021-09-09 16:29     ` Marc-André Lureau
@ 2021-09-09 16:49     ` Daniel P. Berrangé
  2021-09-09 17:02       ` Ian Jackson
  1 sibling, 1 reply; 74+ messages in thread
From: Daniel P. Berrangé @ 2021-09-09 16:49 UTC (permalink / raw)
  To: Peter Maydell
  Cc: QEMU Developers, Markus Armbruster, Ian Jackson, Stefan Hajnoczi,
	Paolo Bonzini, Marc-André Lureau

On Thu, Sep 09, 2021 at 05:02:01PM +0100, Peter Maydell wrote:
> On Tue, 7 Sept 2021 at 13:32, <marcandre.lureau@redhat.com> wrote:
> >
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Most likely, QEMU will want tighter control over the sources, rather
> > than relying on crates.io downloading, use a git submodule with all the
> > dependencies. However, cargo --offline was added in 1.36.
> >
> > "cargo vendor" helps gathering and updating the dependencies.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  configure                 | 8 ++++++++
> >  meson.build               | 7 ++++++-
> >  .cargo/config.toml.in     | 5 +++++
> >  .cargo/meson.build        | 5 +++++
> >  .gitmodules               | 4 ++++
> >  rust/vendored             | 1 +
> >  scripts/archive-source.sh | 2 +-
> >  scripts/cargo_wrapper.py  | 1 +
> >  8 files changed, 31 insertions(+), 2 deletions(-)
> >  create mode 100644 .cargo/config.toml.in
> >  create mode 100644 .cargo/meson.build
> >  create mode 160000 rust/vendored
> 
> So, this is a lot of extra code in a submodule. Historically we've
> found that submodules are a colossal pain, and so I think we should
> think about whether we really want to have all our rust dependencies
> in a submodule forever.
> 
> I am definitely only at the beginner stage with Rust, but I think
> we should have a discussion about what the different alternative
> options are here, and what we want to achieve, so that we know
> why we're doing this and what we're gaining from the pain...
> 
> For instance, could we instead commit Cargo.lock in git and
> use that to nail down specific versions of the dependencies ?
> 
> FWIW, the "why submodules" for the C dependencies we ship
> like that is basically
>  * C doesn't have a package manager, so if we need a dependency that
>    distros don't ship then we need to wrap it up and provide it ourselves
>  * where we ship binary blobs (guest BIOS etc) we want to also ship
>    the source code for those blobs
> I think for Rust dependencies those don't really apply.

Even for our existing non-rust usage of submodules, it is very
borderline questionable whether the benefit outweighs the pain
they are causing us. I frequently wish we would just go "cold
turkey" and drop all our submodules and spin off all the bundled
blobs into separate downloads, despite the disruption it would
cause in the short term.

> Overall, I think that to the extent that we can look like a "normal"
> user of Rust, that's a good plan. Distros may well want to be able
> to do "build against our packaged rust stuff rather than downloading
> from crates.io" but I imagine they have machinery for that already;
> if we act like most other Rust programs we have better chances of
> not breaking that machinery.

Yes, distros do have machinery for this, although it is often
hard to fit in with it when you have a mixed language project.
Their machinery typically assumes pure single language project,
so would work nicer if any QEMU rust pieces were separately
released from the rest of QEMU. Obviously this is easier said
than done since QEMU tends towards a monolothic repo approach
historically.

> We do already effectively do "download code when QEMU is built" --
> the makefile invokes scripts/git-submodule-update which pulls
> down submodule code. (Thanks to Ian for pointing out this framing
> of the question.)
> 
> (I'm not personally a fan of the "download everything from crates.io"
> Rust ecosystem, but it is what it is, and wishing the Rust world
> worked more like a trad Linux-distro-provides-all-your-dependencies
> isn't, alas, going to make it so :-))

Yes, I'm inclined to agree here. For better or worse the battle is
over and "download everything from <repo> on the fly" is the accepted
approach for pretty much all modern languages. The language specific
repo essentially is the OS distro from their POV.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-09 16:29     ` Marc-André Lureau
@ 2021-09-09 16:53       ` Daniel P. Berrangé
  2021-09-09 17:04         ` Ian Jackson
  2021-09-13 14:21       ` Paolo Bonzini
  1 sibling, 1 reply; 74+ messages in thread
From: Daniel P. Berrangé @ 2021-09-09 16:53 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Peter Maydell, QEMU Developers, Markus Armbruster, Ian Jackson,
	Stefan Hajnoczi, Paolo Bonzini

On Thu, Sep 09, 2021 at 08:29:58PM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Thu, Sep 9, 2021 at 8:04 PM Peter Maydell <peter.maydell@linaro.org>
> wrote:
> 
> > On Tue, 7 Sept 2021 at 13:32, <marcandre.lureau@redhat.com> wrote:
> > >
> > > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> > >
> > > Most likely, QEMU will want tighter control over the sources, rather
> > > than relying on crates.io downloading, use a git submodule with all the
> > > dependencies. However, cargo --offline was added in 1.36.
> > >
> > > "cargo vendor" helps gathering and updating the dependencies.
> > >
> > > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > ---
> > >  configure                 | 8 ++++++++
> > >  meson.build               | 7 ++++++-
> > >  .cargo/config.toml.in     | 5 +++++
> > >  .cargo/meson.build        | 5 +++++
> > >  .gitmodules               | 4 ++++
> > >  rust/vendored             | 1 +
> > >  scripts/archive-source.sh | 2 +-
> > >  scripts/cargo_wrapper.py  | 1 +
> > >  8 files changed, 31 insertions(+), 2 deletions(-)
> > >  create mode 100644 .cargo/config.toml.in
> > >  create mode 100644 .cargo/meson.build
> > >  create mode 160000 rust/vendored

> > Overall, I think that to the extent that we can look like a "normal"
> > user of Rust, that's a good plan. Distros may well want to be able
> > to do "build against our packaged rust stuff rather than downloading
> > from crates.io" but I imagine they have machinery for that already;
> > if we act like most other Rust programs we have better chances of
> > not breaking that machinery.
> >
> 
> True, at least on Fedora, there is machinery to package "regular" Rust
> programs/crates in an automated way.  Vendoring dependencies should work
> equally, but may not conform with distro policies, so they have extra work
> eventually (it seems vendoring is more and more common though, with go
> projects for example)

I wouldn't assume that we're going to be able to use that RPM support
for rust, if we bundle our rust code inside the QEMU tarball and hidden
behind meson. It generally only works well in single language projects
using the preferred build tool exclusively (Cargo in this case).

> > (I'm not personally a fan of the "download everything from crates.io"
> > Rust ecosystem, but it is what it is, and wishing the Rust world
> > worked more like a trad Linux-distro-provides-all-your-dependencies
> > isn't, alas, going to make it so :-))
> >
> >
> A nice alternative to vendoring that could work well for QEMU is to mirror
> the Rust crate we use, so we have similar control and guarantee over them
> as submodules, and use `[patch.crates-io]` to point at qemu-project
> locations.

The Cargo metadata specifies the versions we'll get so we have full
control over what deps are pulled in. All the mirroring our do is to
cope with occassions where the main crate download website is
inaccessible for some reason. I'm not convinced that's enough to
justify creating extra work for ourselves through mirroring. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-09 16:49     ` Daniel P. Berrangé
@ 2021-09-09 17:02       ` Ian Jackson
  0 siblings, 0 replies; 74+ messages in thread
From: Ian Jackson @ 2021-09-09 17:02 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Peter Maydell, QEMU Developers, Markus Armbruster,
	Stefan Hajnoczi, Marc-André Lureau, Paolo Bonzini

Daniel P. Berrangé writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> Yes, distros do have machinery for this, although it is often
> hard to fit in with it when you have a mixed language project.
> Their machinery typically assumes pure single language project,
> so would work nicer if any QEMU rust pieces were separately
> released from the rest of QEMU. Obviously this is easier said
> than done since QEMU tends towards a monolothic repo approach
> historically.

Right.

However, for a project that has Rust dependencies, the distros will
(or will soon need) machinery to divert the
langage-specific-package-manager downloads to their own repo.  For
example, the Debian Rust team provide a .cargo/config.toml to replace
crates.io with the local Debian Rust packages, which the Debian
package management system has provided via the (translated)
build-dependencies.

Debian certainly wouldn't want to use any vendored crates bundled with
Qemu.  Indeed Debian people hate vendoring more than they hate
language specific package managers.  At least with the LSPM you can
usually nobble it in one place - ie many of the problems can be solved
automatically.  Vendoring typically involves playing whack-a-mole with
compatibility problems, actually-modified versions, etc. - much human
work (and quite annoying work too!)  (Of course this is less of an
issue if you don't actually modify the vendored code, but anyone who
knows Rust and sees a vendored Rust crate will think it's been
modified.)

> > (I'm not personally a fan of the "download everything from crates.io"
> > Rust ecosystem, but it is what it is, and wishing the Rust world
> > worked more like a trad Linux-distro-provides-all-your-dependencies
> > isn't, alas, going to make it so :-))
> 
> Yes, I'm inclined to agree here. For better or worse the battle is
> over and "download everything from <repo> on the fly" is the accepted
> approach for pretty much all modern languages. The language specific
> repo essentially is the OS distro from their POV.

To be honest, I am of the same mind as you about cargo and crates.io
(and things like it) But I think the ship has sailed.

At least, committing the Cargo.lock file will ensure that the same
versions of the dependencies are used - at least by people who don't
know anything much about Rust.

Marc-André Lureau writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> A nice alternative to vendoring that could work well for QEMU is to
> mirror the Rust crate we use, so we have similar control and
> guarantee over them as submodules, and use `[patch.crates-io]` to
> point at qemu-project locations.

This is a very reasonable suggestion.  In my experience crates.io is
very reliable - but (as a test system onwer myself) I know how much
you want to reduce the number of different sites whose upness your CI
depends on, no matter how good their communities think they are :-).

I think there should be a documented config option to disable this.
People who know enough Rust to run `cargo update` etc. will need it,
and having them hand-hack the config files is not really desirable.

Ian.


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-09 16:53       ` Daniel P. Berrangé
@ 2021-09-09 17:04         ` Ian Jackson
  0 siblings, 0 replies; 74+ messages in thread
From: Ian Jackson @ 2021-09-09 17:04 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Peter Maydell, QEMU Developers, Markus Armbruster,
	Marc-André Lureau, Stefan Hajnoczi, Paolo Bonzini

Daniel P. Berrangé writes ("Re: [RFC v3 13/32] rust: use vendored-sources"):
> On Thu, Sep 09, 2021 at 08:29:58PM +0400, Marc-André Lureau wrote:
> > True, at least on Fedora, there is machinery to package "regular" Rust
> > programs/crates in an automated way.  Vendoring dependencies should work
> > equally, but may not conform with distro policies, so they have extra work
> > eventually (it seems vendoring is more and more common though, with go
> > projects for example)
> 
> I wouldn't assume that we're going to be able to use that RPM support
> for rust, if we bundle our rust code inside the QEMU tarball and hidden
> behind meson. It generally only works well in single language projects
> using the preferred build tool exclusively (Cargo in this case).

I can't speak to RPM, but Debian's arrangements for avoiding crates.io
downloads at packag-ebuild-time will work just fine even if it
the cargo calls are buried inside meson.  All that would be needed is
a way to disable any source replacement done by qemu (see previous
mail) so that Debian's source replacement takes effect.

Ian.


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 12/32] rust: provide a common crate for QEMU
  2021-09-07 12:19 ` [RFC v3 12/32] rust: provide a common crate for QEMU marcandre.lureau
@ 2021-09-10  1:18   ` Alistair Francis
  2021-09-10  7:43     ` Marc-André Lureau
  2021-09-13 17:11   ` Paolo Bonzini
  1 sibling, 1 reply; 74+ messages in thread
From: Alistair Francis @ 2021-09-10  1:18 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange,
	qemu-devel@nongnu.org Developers, Stefan Hajnoczi,
	Markus Armbruster

On Tue, Sep 7, 2021 at 10:41 PM <marcandre.lureau@redhat.com> wrote:
>
> From: Marc-André Lureau <marcandre.lureau@redhat.com>
>
> This crates provides common bindings and facilities for QEMU C API
> shared by various projects.
>
> Most importantly, it defines the conversion traits used to convert from
> C to Rust types. Those traits are largely adapted from glib-rs, since
> those have proved to be very flexible, and should guide us to bind
> further QEMU types such as QOM. If glib-rs becomes a dependency, we
> should consider adopting glib translate traits. For QAPI, we need a
> smaller subset.
>
> Cargo.lock is checked-in, as QEMU produces end-of-chain binaries, and it
> is desirable to track the exact set of packages that are involved in
> managed builds.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  Cargo.lock                   |  63 +++++
>  Cargo.toml                   |   4 +-
>  rust/common/Cargo.toml       |  11 +
>  rust/common/src/error.rs     | 113 ++++++++
>  rust/common/src/ffi.rs       |  93 +++++++
>  rust/common/src/lib.rs       |  21 ++
>  rust/common/src/qemu.rs      | 101 ++++++++
>  rust/common/src/qmp.rs       |   0
>  rust/common/src/translate.rs | 482 +++++++++++++++++++++++++++++++++++
>  9 files changed, 887 insertions(+), 1 deletion(-)
>  create mode 100644 Cargo.lock
>  create mode 100644 rust/common/Cargo.toml
>  create mode 100644 rust/common/src/error.rs
>  create mode 100644 rust/common/src/ffi.rs
>  create mode 100644 rust/common/src/lib.rs
>  create mode 100644 rust/common/src/qemu.rs
>  create mode 100644 rust/common/src/qmp.rs
>  create mode 100644 rust/common/src/translate.rs
>
> diff --git a/Cargo.lock b/Cargo.lock
> new file mode 100644
> index 0000000000..8dc2dd9da7
> --- /dev/null
> +++ b/Cargo.lock
> @@ -0,0 +1,63 @@
> +# This file is automatically @generated by Cargo.
> +# It is not intended for manual editing.
> +version = 3
> +
> +[[package]]
> +name = "autocfg"
> +version = "1.0.1"
> +source = "registry+https://github.com/rust-lang/crates.io-index"
> +checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a"
> +
> +[[package]]
> +name = "bitflags"
> +version = "1.2.1"
> +source = "registry+https://github.com/rust-lang/crates.io-index"
> +checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
> +
> +[[package]]
> +name = "cc"
> +version = "1.0.70"
> +source = "registry+https://github.com/rust-lang/crates.io-index"
> +checksum = "d26a6ce4b6a484fa3edb70f7efa6fc430fd2b87285fe8b84304fd0936faa0dc0"
> +
> +[[package]]
> +name = "cfg-if"
> +version = "1.0.0"
> +source = "registry+https://github.com/rust-lang/crates.io-index"
> +checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
> +
> +[[package]]
> +name = "common"
> +version = "0.1.0"
> +dependencies = [
> + "libc",
> + "nix",
> +]
> +
> +[[package]]
> +name = "libc"
> +version = "0.2.101"
> +source = "registry+https://github.com/rust-lang/crates.io-index"
> +checksum = "3cb00336871be5ed2c8ed44b60ae9959dc5b9f08539422ed43f09e34ecaeba21"
> +
> +[[package]]
> +name = "memoffset"
> +version = "0.6.4"
> +source = "registry+https://github.com/rust-lang/crates.io-index"
> +checksum = "59accc507f1338036a0477ef61afdae33cde60840f4dfe481319ce3ad116ddf9"
> +dependencies = [
> + "autocfg",
> +]
> +
> +[[package]]
> +name = "nix"
> +version = "0.20.1"
> +source = "registry+https://github.com/rust-lang/crates.io-index"
> +checksum = "df8e5e343312e7fbeb2a52139114e9e702991ef9c2aea6817ff2440b35647d56"
> +dependencies = [
> + "bitflags",
> + "cc",
> + "cfg-if",
> + "libc",
> + "memoffset",
> +]
> diff --git a/Cargo.toml b/Cargo.toml
> index c4b464ff15..14131eed3c 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -1,2 +1,4 @@
>  [workspace]
> -members = []
> +members = [
> +  "rust/common",
> +]
> diff --git a/rust/common/Cargo.toml b/rust/common/Cargo.toml
> new file mode 100644
> index 0000000000..6c240447f3
> --- /dev/null
> +++ b/rust/common/Cargo.toml
> @@ -0,0 +1,11 @@
> +[package]
> +name = "common"
> +version = "0.1.0"
> +edition = "2018"
> +publish = false
> +
> +[dependencies]
> +libc = "0.2.92"
> +
> +[target."cfg(unix)".dependencies]
> +nix = "0.20.0"
> diff --git a/rust/common/src/error.rs b/rust/common/src/error.rs
> new file mode 100644
> index 0000000000..f166ac42ea
> --- /dev/null
> +++ b/rust/common/src/error.rs
> @@ -0,0 +1,113 @@
> +use std::{self, ffi::CString, fmt, io, ptr};
> +
> +use crate::translate::*;

It's not uncommon to ban wildcard imports that aren't preludes as it
can make it confusing to read. Does QEMU have a stance on that type of
thing?

> +use crate::{ffi, qemu};
> +
> +/// Common error type for QEMU and related projects.
> +#[derive(Debug)]
> +pub enum Error {
> +    /// A generic error with file and line location.
> +    FailedAt(String, &'static str, u32),
> +    /// An IO error.
> +    Io(io::Error),
> +    #[cfg(unix)]
> +    /// A nix error.
> +    Nix(nix::Error),
> +}
> +
> +/// Alias for a `Result` with the error type for QEMU.
> +pub type Result<T> = std::result::Result<T, Error>;

I think this is very confusing. Rust developers expect `Result` to be
the one from `std::result`, it would be better to call this
`QEMUResult`

> +
> +impl Error {
> +    fn message(&self) -> String {
> +        use Error::*;

Do we need this here? Why not put it at the top of the file?

> +        match self {
> +            FailedAt(msg, _, _) => msg.into(),
> +            Io(io) => format!("IO error: {}", io),
> +            #[cfg(unix)]
> +            Nix(nix) => format!("Nix error: {}", nix),
> +        }
> +    }
> +
> +    fn location(&self) -> Option<(&'static str, u32)> {
> +        use Error::*;
> +        match self {
> +            FailedAt(_, file, line) => Some((file, *line)),
> +            Io(_) => None,
> +            #[cfg(unix)]
> +            Nix(_) => None,
> +        }
> +    }
> +}
> +
> +impl fmt::Display for Error {
> +    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
> +        use Error::*;
> +        match self {
> +            FailedAt(msg, file, line) => write!(f, "{} ({}:{})", msg, file, line),
> +            _ => write!(f, "{}", self.message()),
> +        }
> +    }
> +}
> +
> +impl From<io::Error> for Error {
> +    fn from(val: io::Error) -> Self {
> +        Error::Io(val)
> +    }
> +}
> +
> +#[cfg(unix)]
> +impl From<nix::Error> for Error {
> +    fn from(val: nix::Error) -> Self {
> +        Error::Nix(val)
> +    }
> +}
> +
> +impl QemuPtrDefault for Error {
> +    type QemuType = *mut ffi::Error;
> +}
> +
> +impl<'a> ToQemuPtr<'a, *mut ffi::Error> for Error {
> +    type Storage = qemu::CError;
> +
> +    fn to_qemu_none(&'a self) -> Stash<'a, *mut ffi::Error, Self> {
> +        let err = self.to_qemu_full();
> +
> +        Stash(err, unsafe { from_qemu_full(err) })
> +    }
> +
> +    fn to_qemu_full(&self) -> *mut ffi::Error {
> +        let cmsg =
> +            CString::new(self.message()).expect("ToQemuPtr<Error>: unexpected '\0' character");
> +        let mut csrc = CString::new("").unwrap();
> +        let (src, line) = self.location().map_or((ptr::null(), 0_i32), |loc| {
> +            csrc = CString::new(loc.0).expect("ToQemuPtr<Error>:: unexpected '\0' character");
> +            (csrc.as_ptr() as *const libc::c_char, loc.1 as i32)
> +        });
> +        let func = ptr::null();
> +
> +        let mut err: *mut ffi::Error = ptr::null_mut();
> +        unsafe {
> +            ffi::error_setg_internal(
> +                &mut err as *mut *mut _,
> +                src,
> +                line,
> +                func,
> +                cmsg.as_ptr() as *const libc::c_char,
> +            );
> +            err
> +        }
> +    }
> +}
> +
> +/// Convenience macro to build a [`Error::FailedAt`] error.
> +///
> +/// Returns a `Result::Err` with the file:line location.
> +/// (the error can then be converted to a QEMU `ffi::Error`)
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! err {
> +    ($msg:expr) => {
> +        Err(Error::FailedAt($msg.into(), file!(), line!()))
> +    };
> +}
> diff --git a/rust/common/src/ffi.rs b/rust/common/src/ffi.rs
> new file mode 100644
> index 0000000000..82818d503a
> --- /dev/null
> +++ b/rust/common/src/ffi.rs
> @@ -0,0 +1,93 @@
> +//! Bindings to the raw low-level C API commonly provided by QEMU projects.
> +//!
> +//! Manual bindings to C API availabe when linking QEMU projects.

s/availabe/available/g

> +//! It includes minimal glib allocation functions too, since it's the default
> +//! allocator used by QEMU, and we don't depend on glib-rs crate yet).
> +//!
> +//! Higher-level Rust-friendly bindings are provided by different modules.
> +
> +use libc::{c_char, c_void, size_t};
> +
> +extern "C" {
> +    pub fn g_malloc0(n_bytes: size_t) -> *mut c_void;
> +    pub fn g_free(ptr: *mut c_void);
> +    pub fn g_strndup(str: *const c_char, n: size_t) -> *mut c_char;
> +}

We can get there from the glib/glib_sys crate:
https://gtk-rs.org/gtk-rs-core/stable/latest/docs/glib_sys/fn.g_malloc0.html

If we only plan on using these 3 I think this approach is fine, but
something to keep in mind if we use more glib functions.

> +
> +#[repr(C)]
> +pub struct QObject(c_void);
> +
> +impl ::std::fmt::Debug for QObject {
> +    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
> +        f.debug_struct(&format!("QObject @ {:?}", self as *const _))
> +            .finish()
> +    }
> +}
> +
> +#[repr(C)]
> +pub struct QNull(c_void);
> +
> +impl ::std::fmt::Debug for QNull {
> +    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
> +        f.debug_struct(&format!("QNull @ {:?}", self as *const _))
> +            .finish()
> +    }
> +}
> +
> +#[repr(C)]
> +pub struct Error(c_void);
> +
> +impl ::std::fmt::Debug for Error {
> +    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
> +        f.debug_struct(&format!("Error @ {:?}", self as *const _))
> +            .finish()
> +    }
> +}
> +
> +extern "C" {
> +    pub fn error_setg_internal(
> +        errp: *mut *mut Error,
> +        src: *const c_char,
> +        line: i32,
> +        func: *const c_char,
> +        fmt: *const c_char,
> +        ...
> +    );
> +    pub fn error_get_pretty(err: *const Error) -> *const c_char;
> +    pub fn error_free(err: *mut Error);
> +}
> +
> +/// Wrap a QMP hanlder.

handler

> +#[macro_export]
> +macro_rules! qmp {
> +    // the basic return value variant
> +    ($e:expr, $errp:ident, $errval:expr) => {{
> +        assert!(!$errp.is_null());
> +        unsafe {
> +            *$errp = std::ptr::null_mut();
> +        }
> +
> +        match $e {
> +            Ok(val) => val,
> +            Err(err) => unsafe {
> +                *$errp = err.to_qemu_full();
> +                $errval
> +            },
> +        }
> +    }};
> +    // the ptr return value variant
> +    ($e:expr, $errp:ident) => {{
> +        assert!(!$errp.is_null());
> +        unsafe {
> +            *$errp = std::ptr::null_mut();
> +        }
> +
> +        match $e {
> +            Ok(val) => val.to_qemu_full().into(),
> +            Err(err) => unsafe {
> +                *$errp = err.to_qemu_full();
> +                std::ptr::null_mut()
> +            },
> +        }
> +    }};
> +}

It would be a good idea to document why this code is safe

Alistair

> diff --git a/rust/common/src/lib.rs b/rust/common/src/lib.rs
> new file mode 100644
> index 0000000000..4de826bc2e
> --- /dev/null
> +++ b/rust/common/src/lib.rs
> @@ -0,0 +1,21 @@
> +//! Common code for QEMU
> +//!
> +//! This crates provides common bindings and facilities for QEMU C API shared by
> +//! various projects. Most importantly, it defines the conversion traits used to
> +//! convert from C to Rust types. Those traits are largely adapted from glib-rs,
> +//! since those have prooven to be very flexible, and should guide us to bind
> +//! further QEMU types such as QOM. If glib-rs becomes a dependency, we should
> +//! consider adopting glib translate traits. For QAPI, we need a smaller subset.
> +
> +pub use libc;
> +
> +mod error;
> +pub use error::*;
> +
> +mod qemu;
> +pub use qemu::*;
> +
> +mod translate;
> +pub use translate::*;
> +
> +pub mod ffi;
> diff --git a/rust/common/src/qemu.rs b/rust/common/src/qemu.rs
> new file mode 100644
> index 0000000000..dd01c6d92d
> --- /dev/null
> +++ b/rust/common/src/qemu.rs
> @@ -0,0 +1,101 @@
> +use std::{ffi::CStr, ptr, str};
> +
> +use crate::{ffi, translate};
> +use translate::{FromQemuPtrFull, FromQemuPtrNone, QemuPtrDefault, Stash, ToQemuPtr};
> +
> +/// A type representing an owned C QEMU Error.
> +pub struct CError(ptr::NonNull<ffi::Error>);
> +
> +impl translate::FromQemuPtrFull<*mut ffi::Error> for CError {
> +    unsafe fn from_qemu_full(ptr: *mut ffi::Error) -> Self {
> +        assert!(!ptr.is_null());
> +        Self(ptr::NonNull::new_unchecked(ptr))
> +    }
> +}
> +
> +impl CError {
> +    pub fn pretty(&self) -> &str {
> +        unsafe {
> +            let pretty = ffi::error_get_pretty(self.0.as_ptr());
> +            let bytes = CStr::from_ptr(pretty).to_bytes();
> +            str::from_utf8(bytes)
> +                .unwrap_or_else(|err| str::from_utf8(&bytes[..err.valid_up_to()]).unwrap())
> +        }
> +    }
> +}
> +
> +impl Drop for CError {
> +    fn drop(&mut self) {
> +        unsafe { ffi::error_free(self.0.as_ptr()) }
> +    }
> +}
> +
> +/// QObject (JSON object)
> +#[derive(Clone, Debug)]
> +pub struct QObject;
> +
> +impl QemuPtrDefault for QObject {
> +    type QemuType = *mut ffi::QObject;
> +}
> +
> +impl FromQemuPtrFull<*mut ffi::QObject> for QObject {
> +    #[inline]
> +    unsafe fn from_qemu_full(_ffi: *mut ffi::QObject) -> Self {
> +        unimplemented!()
> +    }
> +}
> +
> +impl FromQemuPtrNone<*const ffi::QObject> for QObject {
> +    #[inline]
> +    unsafe fn from_qemu_none(_ffi: *const ffi::QObject) -> Self {
> +        unimplemented!()
> +    }
> +}
> +
> +impl<'a> ToQemuPtr<'a, *mut ffi::QObject> for QObject {
> +    type Storage = ();
> +
> +    #[inline]
> +    fn to_qemu_none(&self) -> Stash<'a, *mut ffi::QObject, QObject> {
> +        unimplemented!()
> +    }
> +    #[inline]
> +    fn to_qemu_full(&self) -> *mut ffi::QObject {
> +        unimplemented!()
> +    }
> +}
> +
> +/// QNull (JSON null)
> +#[derive(Clone, Debug)]
> +pub struct QNull;
> +
> +impl QemuPtrDefault for QNull {
> +    type QemuType = *mut ffi::QNull;
> +}
> +
> +impl FromQemuPtrFull<*mut ffi::QObject> for QNull {
> +    #[inline]
> +    unsafe fn from_qemu_full(_ffi: *mut ffi::QObject) -> Self {
> +        unimplemented!()
> +    }
> +}
> +
> +impl FromQemuPtrNone<*const ffi::QObject> for QNull {
> +    #[inline]
> +    unsafe fn from_qemu_none(_ffi: *const ffi::QObject) -> Self {
> +        unimplemented!()
> +    }
> +}
> +
> +impl<'a> ToQemuPtr<'a, *mut ffi::QNull> for QNull {
> +    type Storage = ();
> +
> +    #[inline]
> +    fn to_qemu_none(&self) -> Stash<'a, *mut ffi::QNull, QNull> {
> +        unimplemented!()
> +    }
> +    #[inline]
> +    fn to_qemu_full(&self) -> *mut ffi::QNull {
> +        unimplemented!()
> +    }
> +}
> diff --git a/rust/common/src/qmp.rs b/rust/common/src/qmp.rs
> new file mode 100644
> index 0000000000..e69de29bb2
> diff --git a/rust/common/src/translate.rs b/rust/common/src/translate.rs
> new file mode 100644
> index 0000000000..315e14fa25
> --- /dev/null
> +++ b/rust/common/src/translate.rs
> @@ -0,0 +1,482 @@
> +// largely adapted from glib-rs
> +// we don't depend on glib-rs as this brings a lot more code that we may not need
> +// and also because there are issues with the conversion traits for our ffi::*mut.
> +use libc::{c_char, size_t};
> +use std::ffi::{CStr, CString};
> +use std::ptr;
> +
> +use crate::ffi;
> +
> +/// A pointer.
> +pub trait Ptr: Copy + 'static {
> +    fn is_null(&self) -> bool;
> +    fn from<X>(ptr: *mut X) -> Self;
> +    fn to<X>(self) -> *mut X;
> +}
> +
> +impl<T: 'static> Ptr for *const T {
> +    #[inline]
> +    fn is_null(&self) -> bool {
> +        (*self).is_null()
> +    }
> +
> +    #[inline]
> +    fn from<X>(ptr: *mut X) -> *const T {
> +        ptr as *const T
> +    }
> +
> +    #[inline]
> +    fn to<X>(self) -> *mut X {
> +        self as *mut X
> +    }
> +}
> +
> +impl<T: 'static> Ptr for *mut T {
> +    #[inline]
> +    fn is_null(&self) -> bool {
> +        (*self).is_null()
> +    }
> +
> +    #[inline]
> +    fn from<X>(ptr: *mut X) -> *mut T {
> +        ptr as *mut T
> +    }
> +
> +    #[inline]
> +    fn to<X>(self) -> *mut X {
> +        self as *mut X
> +    }
> +}
> +
> +/// Macro to declare a `NewPtr` struct.
> +///
> +/// A macro to declare a newtype for pointers, to workaround that *T are not
> +/// defined in our binding crates, and allow foreign traits implementations.
> +/// (this is used by qapi-gen bindings)
> +#[allow(unused_macros)]
> +#[macro_export]
> +#[doc(hidden)]
> +macro_rules! new_ptr {
> +    () => {
> +        #[derive(Copy, Clone)]
> +        pub struct NewPtr<P: Ptr>(pub P);
> +
> +        impl<P: Ptr> Ptr for NewPtr<P> {
> +            #[inline]
> +            fn is_null(&self) -> bool {
> +                self.0.is_null()
> +            }
> +
> +            #[inline]
> +            fn from<X>(ptr: *mut X) -> Self {
> +                NewPtr(P::from(ptr))
> +            }
> +
> +            #[inline]
> +            fn to<X>(self) -> *mut X {
> +                self.0.to()
> +            }
> +        }
> +    };
> +}
> +
> +/// Provides the default pointer type to be used in some container conversions.
> +///
> +/// It's `*mut c_char` for `String`, `*mut ffi::GuestInfo` for `GuestInfo`...
> +pub trait QemuPtrDefault {
> +    type QemuType: Ptr;
> +}
> +
> +impl QemuPtrDefault for String {
> +    type QemuType = *mut c_char;
> +}
> +
> +/// A Stash contains the temporary storage and a pointer into it.
> +///
> +/// The pointer is valid for the lifetime of the `Stash`. As the lifetime of the
> +/// `Stash` returned from `to_qemu_none` is at least the enclosing statement,
> +/// you can avoid explicitly binding the stash in most cases and just take the
> +/// pointer out of it:
> +///
> +/// ```ignore
> +///     pub fn set_device_name(&self, name: &str) {
> +///         unsafe {
> +///             ffi::qemu_device_set_name(self.pointer, name.to_qemu_none().0)
> +///         }
> +///     }
> +/// ```
> +pub struct Stash<'a, P: Copy, T: ?Sized + ToQemuPtr<'a, P>>(
> +    pub P,
> +    pub <T as ToQemuPtr<'a, P>>::Storage,
> +);
> +
> +/// Translate to a pointer.
> +pub trait ToQemuPtr<'a, P: Copy> {
> +    type Storage;
> +
> +    /// The pointer in the `Stash` is only valid for the lifetime of the `Stash`.
> +    fn to_qemu_none(&'a self) -> Stash<'a, P, Self>;
> +
> +    /// Transfer the ownership to the ffi.
> +    fn to_qemu_full(&self) -> P {
> +        unimplemented!();
> +    }
> +}
> +
> +impl<'a, P: Ptr, T: ToQemuPtr<'a, P>> ToQemuPtr<'a, P> for Option<T> {
> +    type Storage = Option<<T as ToQemuPtr<'a, P>>::Storage>;
> +
> +    #[inline]
> +    fn to_qemu_none(&'a self) -> Stash<'a, P, Option<T>> {
> +        self.as_ref()
> +            .map_or(Stash(Ptr::from::<()>(ptr::null_mut()), None), |s| {
> +                let s = s.to_qemu_none();
> +                Stash(s.0, Some(s.1))
> +            })
> +    }
> +
> +    #[inline]
> +    fn to_qemu_full(&self) -> P {
> +        self.as_ref()
> +            .map_or(Ptr::from::<()>(ptr::null_mut()), ToQemuPtr::to_qemu_full)
> +    }
> +}
> +
> +impl<'a, P: Ptr, T: ToQemuPtr<'a, P>> ToQemuPtr<'a, P> for Box<T> {
> +    type Storage = <T as ToQemuPtr<'a, P>>::Storage;
> +
> +    #[inline]
> +    fn to_qemu_none(&'a self) -> Stash<'a, P, Box<T>> {
> +        let s = self.as_ref().to_qemu_none();
> +        Stash(s.0, s.1)
> +    }
> +
> +    #[inline]
> +    fn to_qemu_full(&self) -> P {
> +        ToQemuPtr::to_qemu_full(self.as_ref())
> +    }
> +}
> +
> +impl<'a> ToQemuPtr<'a, *mut c_char> for String {
> +    type Storage = CString;
> +
> +    #[inline]
> +    fn to_qemu_none(&self) -> Stash<'a, *mut c_char, String> {
> +        let tmp = CString::new(&self[..])
> +            .expect("String::ToQemuPtr<*mut c_char>: unexpected '\0' character");
> +        Stash(tmp.as_ptr() as *mut c_char, tmp)
> +    }
> +
> +    #[inline]
> +    fn to_qemu_full(&self) -> *mut c_char {
> +        unsafe { ffi::g_strndup(self.as_ptr() as *const c_char, self.len() as size_t) }
> +    }
> +}
> +
> +/// Translate from a pointer type, without taking ownership.
> +pub trait FromQemuPtrNone<P: Ptr>: Sized {
> +    /// # Safety
> +    ///
> +    /// `ptr` must be a valid pointer. It is not referenced after the call.
> +    unsafe fn from_qemu_none(ptr: P) -> Self;
> +}
> +
> +/// Translate from a pointer type, taking ownership.
> +pub trait FromQemuPtrFull<P: Ptr>: Sized {
> +    /// # Safety
> +    ///
> +    /// `ptr` must be a valid pointer. Ownership is transferred.
> +    unsafe fn from_qemu_full(ptr: P) -> Self;
> +}
> +
> +/// See [`FromQemuPtrNone`](trait.FromQemuPtrNone.html).
> +#[inline]
> +#[allow(clippy::missing_safety_doc)]
> +pub unsafe fn from_qemu_none<P: Ptr, T: FromQemuPtrNone<P>>(ptr: P) -> T {
> +    FromQemuPtrNone::from_qemu_none(ptr)
> +}
> +
> +/// See [`FromQemuPtrFull`](trait.FromQemuPtrFull.html).
> +#[inline]
> +#[allow(clippy::missing_safety_doc)]
> +pub unsafe fn from_qemu_full<P: Ptr, T: FromQemuPtrFull<P>>(ptr: P) -> T {
> +    FromQemuPtrFull::from_qemu_full(ptr)
> +}
> +
> +impl<P: Ptr, T: FromQemuPtrNone<P>> FromQemuPtrNone<P> for Option<T> {
> +    #[inline]
> +    unsafe fn from_qemu_none(ptr: P) -> Option<T> {
> +        if ptr.is_null() {
> +            None
> +        } else {
> +            Some(from_qemu_none(ptr))
> +        }
> +    }
> +}
> +
> +impl<P: Ptr, T: FromQemuPtrFull<P>> FromQemuPtrFull<P> for Option<T> {
> +    #[inline]
> +    unsafe fn from_qemu_full(ptr: P) -> Option<T> {
> +        if ptr.is_null() {
> +            None
> +        } else {
> +            Some(from_qemu_full(ptr))
> +        }
> +    }
> +}
> +
> +impl FromQemuPtrNone<*const c_char> for String {
> +    #[inline]
> +    unsafe fn from_qemu_none(ptr: *const c_char) -> Self {
> +        assert!(!ptr.is_null());
> +        String::from_utf8_lossy(CStr::from_ptr(ptr).to_bytes()).into_owned()
> +    }
> +}
> +
> +impl FromQemuPtrFull<*mut c_char> for String {
> +    #[inline]
> +    unsafe fn from_qemu_full(ptr: *mut c_char) -> Self {
> +        let res = from_qemu_none(ptr as *const _);
> +        ffi::g_free(ptr as *mut _);
> +        res
> +    }
> +}
> +
> +#[doc(hidden)]
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! vec_ffi_wrapper {
> +    ($ffi:ident) => {
> +        #[allow(non_camel_case_types)]
> +        pub struct $ffi(*mut qapi_ffi::$ffi);
> +
> +        impl Drop for $ffi {
> +            fn drop(&mut self) {
> +                let mut list = self.0;
> +                unsafe {
> +                    while !list.is_null() {
> +                        let next = (*list).next;
> +                        Box::from_raw(list);
> +                        list = next;
> +                    }
> +                }
> +            }
> +        }
> +
> +        impl From<NewPtr<*mut qapi_ffi::$ffi>> for *mut qapi_ffi::$ffi {
> +            fn from(p: NewPtr<*mut qapi_ffi::$ffi>) -> Self {
> +                p.0
> +            }
> +        }
> +    };
> +}
> +
> +#[doc(hidden)]
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! impl_vec_scalars_to_qemu {
> +    ($rs:ty, $ffi:ident) => {
> +        impl<'a> ToQemuPtr<'a, NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
> +            type Storage = $ffi;
> +
> +            #[inline]
> +            fn to_qemu_none(&self) -> Stash<NewPtr<*mut qapi_ffi::$ffi>, Self> {
> +                let mut list: *mut qapi_ffi::$ffi = std::ptr::null_mut();
> +                for value in self.iter().rev() {
> +                    let b = Box::new(qapi_ffi::$ffi {
> +                        next: list,
> +                        value: *value,
> +                    });
> +                    list = Box::into_raw(b);
> +                }
> +                Stash(NewPtr(list), $ffi(list))
> +            }
> +
> +            #[inline]
> +            fn to_qemu_full(&self) -> NewPtr<*mut qapi_ffi::$ffi> {
> +                let mut list: *mut qapi_ffi::$ffi = std::ptr::null_mut();
> +                unsafe {
> +                    for value in self.iter().rev() {
> +                        let l = ffi::g_malloc0(std::mem::size_of::<qapi_ffi::$ffi>())
> +                            as *mut qapi_ffi::$ffi;
> +                        (*l).next = list;
> +                        (*l).value = *value;
> +                        list = l;
> +                    }
> +                }
> +                NewPtr(list)
> +            }
> +        }
> +    };
> +}
> +
> +#[doc(hidden)]
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! impl_vec_scalars_from_qemu {
> +    ($rs:ty, $ffi:ident, $free_ffi:ident) => {
> +        impl FromQemuPtrFull<NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
> +            #[inline]
> +            unsafe fn from_qemu_full(ffi: NewPtr<*mut qapi_ffi::$ffi>) -> Self {
> +                let ret = from_qemu_none(NewPtr(ffi.0 as *const _));
> +                qapi_ffi::$free_ffi(ffi.0);
> +                ret
> +            }
> +        }
> +
> +        impl FromQemuPtrNone<NewPtr<*const qapi_ffi::$ffi>> for Vec<$rs> {
> +            #[inline]
> +            unsafe fn from_qemu_none(ffi: NewPtr<*const qapi_ffi::$ffi>) -> Self {
> +                let mut ret = vec![];
> +                let mut it = ffi.0;
> +                while !it.is_null() {
> +                    let e = &*it;
> +                    ret.push(e.value);
> +                    it = e.next;
> +                }
> +                ret
> +            }
> +        }
> +    };
> +}
> +
> +#[doc(hidden)]
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! impl_vec_to_qemu {
> +    ($rs:ty, $ffi:ident) => {
> +        // impl doesn't use only types from inside the current crate
> +        // impl QemuPtrDefault for Vec<$rs> {
> +        //     type QemuType = NewPtr<*mut qapi_ffi::$ffi>;
> +        // }
> +
> +        impl<'a> ToQemuPtr<'a, NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
> +            type Storage = ($ffi, Vec<Stash<'a, <$rs as QemuPtrDefault>::QemuType, $rs>>);
> +
> +            #[inline]
> +            fn to_qemu_none(&self) -> Stash<NewPtr<*mut qapi_ffi::$ffi>, Self> {
> +                let stash_vec: Vec<_> = self.iter().rev().map(ToQemuPtr::to_qemu_none).collect();
> +                let mut list: *mut qapi_ffi::$ffi = std::ptr::null_mut();
> +                for stash in &stash_vec {
> +                    let b = Box::new(qapi_ffi::$ffi {
> +                        next: list,
> +                        value: Ptr::to(stash.0),
> +                    });
> +                    list = Box::into_raw(b);
> +                }
> +                Stash(NewPtr(list), ($ffi(list), stash_vec))
> +            }
> +
> +            #[inline]
> +            fn to_qemu_full(&self) -> NewPtr<*mut qapi_ffi::$ffi> {
> +                let v: Vec<_> = self.iter().rev().map(ToQemuPtr::to_qemu_full).collect();
> +                let mut list: *mut qapi_ffi::$ffi = std::ptr::null_mut();
> +                unsafe {
> +                    for val in v {
> +                        let l = ffi::g_malloc0(std::mem::size_of::<qapi_ffi::$ffi>())
> +                            as *mut qapi_ffi::$ffi;
> +                        (*l).next = list;
> +                        (*l).value = val;
> +                        list = l;
> +                    }
> +                }
> +                NewPtr(list)
> +            }
> +        }
> +    };
> +}
> +
> +#[doc(hidden)]
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! impl_vec_from_qemu {
> +    ($rs:ty, $ffi:ident, $free_ffi:ident) => {
> +        impl FromQemuPtrFull<NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
> +            #[inline]
> +            unsafe fn from_qemu_full(ffi: NewPtr<*mut qapi_ffi::$ffi>) -> Self {
> +                let ret = from_qemu_none(NewPtr(ffi.0 as *const _));
> +                qapi_ffi::$free_ffi(ffi.0);
> +                ret
> +            }
> +        }
> +
> +        impl FromQemuPtrNone<NewPtr<*const qapi_ffi::$ffi>> for Vec<$rs> {
> +            #[inline]
> +            unsafe fn from_qemu_none(ffi: NewPtr<*const qapi_ffi::$ffi>) -> Self {
> +                let mut ret = vec![];
> +                let mut it = ffi.0;
> +                while !it.is_null() {
> +                    let e = &*it;
> +                    ret.push(from_qemu_none(e.value as *const _));
> +                    it = e.next;
> +                }
> +                ret
> +            }
> +        }
> +    };
> +}
> +
> +/// A macro to help the implementation of `Vec<T>` translations.
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! vec_type {
> +    (Vec<$rs:ty>, $ffi:ident, $free_ffi:ident, 0) => {
> +        vec_ffi_wrapper!($ffi);
> +        impl_vec_from_qemu!($rs, $ffi, $free_ffi);
> +        impl_vec_to_qemu!($rs, $ffi);
> +    };
> +    (Vec<$rs:ty>, $ffi:ident, $free_ffi:ident, 1) => {
> +        vec_ffi_wrapper!($ffi);
> +        impl_vec_scalars_from_qemu!($rs, $ffi, $free_ffi);
> +        impl_vec_scalars_to_qemu!($rs, $ffi);
> +    };
> +}
> +
> +/// A macro to implement [`ToQemuPtr`] as boxed scalars
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! impl_to_qemu_scalar_boxed {
> +    ($ty:ty) => {
> +        impl<'a> ToQemuPtr<'a, *mut $ty> for $ty {
> +            type Storage = Box<$ty>;
> +
> +            fn to_qemu_none(&'a self) -> Stash<'a, *mut $ty, Self> {
> +                let mut box_ = Box::new(*self);
> +                Stash(&mut *box_, box_)
> +            }
> +
> +            fn to_qemu_full(&self) -> *mut $ty {
> +                unsafe {
> +                    let ptr = ffi::g_malloc0(std::mem::size_of::<$ty>()) as *mut _;
> +                    *ptr = *self;
> +                    ptr
> +                }
> +            }
> +        }
> +    };
> +}
> +
> +/// A macro to implement [`FromQemuPtrNone`] for scalar pointers.
> +#[allow(unused_macros)]
> +#[macro_export]
> +macro_rules! impl_from_qemu_none_scalar {
> +    ($ty:ty) => {
> +        impl FromQemuPtrNone<*const $ty> for $ty {
> +            unsafe fn from_qemu_none(ptr: *const $ty) -> Self {
> +                *ptr
> +            }
> +        }
> +    };
> +}
> +
> +macro_rules! impl_scalar_boxed {
> +    ($($t:ident)*) => (
> +        $(
> +            impl_to_qemu_scalar_boxed!($t);
> +            impl_from_qemu_none_scalar!($t);
> +        )*
> +    )
> +}
> +
> +// the only built-in used so far, feel free to add more as needed
> +impl_scalar_boxed!(bool i64 f64);
> --
> 2.33.0.113.g6c40894d24
>
>


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 12/32] rust: provide a common crate for QEMU
  2021-09-10  1:18   ` Alistair Francis
@ 2021-09-10  7:43     ` Marc-André Lureau
  0 siblings, 0 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-10  7:43 UTC (permalink / raw)
  To: Alistair Francis
  Cc: Paolo Bonzini, Daniel P. Berrange,
	qemu-devel@nongnu.org Developers, Stefan Hajnoczi,
	Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 35658 bytes --]

Hi

On Fri, Sep 10, 2021 at 5:19 AM Alistair Francis <alistair23@gmail.com>
wrote:

> On Tue, Sep 7, 2021 at 10:41 PM <marcandre.lureau@redhat.com> wrote:
> >
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > This crates provides common bindings and facilities for QEMU C API
> > shared by various projects.
> >
> > Most importantly, it defines the conversion traits used to convert from
> > C to Rust types. Those traits are largely adapted from glib-rs, since
> > those have proved to be very flexible, and should guide us to bind
> > further QEMU types such as QOM. If glib-rs becomes a dependency, we
> > should consider adopting glib translate traits. For QAPI, we need a
> > smaller subset.
> >
> > Cargo.lock is checked-in, as QEMU produces end-of-chain binaries, and it
> > is desirable to track the exact set of packages that are involved in
> > managed builds.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  Cargo.lock                   |  63 +++++
> >  Cargo.toml                   |   4 +-
> >  rust/common/Cargo.toml       |  11 +
> >  rust/common/src/error.rs     | 113 ++++++++
> >  rust/common/src/ffi.rs       |  93 +++++++
> >  rust/common/src/lib.rs       |  21 ++
> >  rust/common/src/qemu.rs      | 101 ++++++++
> >  rust/common/src/qmp.rs       |   0
> >  rust/common/src/translate.rs | 482 +++++++++++++++++++++++++++++++++++
> >  9 files changed, 887 insertions(+), 1 deletion(-)
> >  create mode 100644 Cargo.lock
> >  create mode 100644 rust/common/Cargo.toml
> >  create mode 100644 rust/common/src/error.rs
> >  create mode 100644 rust/common/src/ffi.rs
> >  create mode 100644 rust/common/src/lib.rs
> >  create mode 100644 rust/common/src/qemu.rs
> >  create mode 100644 rust/common/src/qmp.rs
> >  create mode 100644 rust/common/src/translate.rs
> >
> > diff --git a/Cargo.lock b/Cargo.lock
> > new file mode 100644
> > index 0000000000..8dc2dd9da7
> > --- /dev/null
> > +++ b/Cargo.lock
> > @@ -0,0 +1,63 @@
> > +# This file is automatically @generated by Cargo.
> > +# It is not intended for manual editing.
> > +version = 3
> > +
> > +[[package]]
> > +name = "autocfg"
> > +version = "1.0.1"
> > +source = "registry+https://github.com/rust-lang/crates.io-index"
> > +checksum =
> "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a"
> > +
> > +[[package]]
> > +name = "bitflags"
> > +version = "1.2.1"
> > +source = "registry+https://github.com/rust-lang/crates.io-index"
> > +checksum =
> "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
> > +
> > +[[package]]
> > +name = "cc"
> > +version = "1.0.70"
> > +source = "registry+https://github.com/rust-lang/crates.io-index"
> > +checksum =
> "d26a6ce4b6a484fa3edb70f7efa6fc430fd2b87285fe8b84304fd0936faa0dc0"
> > +
> > +[[package]]
> > +name = "cfg-if"
> > +version = "1.0.0"
> > +source = "registry+https://github.com/rust-lang/crates.io-index"
> > +checksum =
> "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
> > +
> > +[[package]]
> > +name = "common"
> > +version = "0.1.0"
> > +dependencies = [
> > + "libc",
> > + "nix",
> > +]
> > +
> > +[[package]]
> > +name = "libc"
> > +version = "0.2.101"
> > +source = "registry+https://github.com/rust-lang/crates.io-index"
> > +checksum =
> "3cb00336871be5ed2c8ed44b60ae9959dc5b9f08539422ed43f09e34ecaeba21"
> > +
> > +[[package]]
> > +name = "memoffset"
> > +version = "0.6.4"
> > +source = "registry+https://github.com/rust-lang/crates.io-index"
> > +checksum =
> "59accc507f1338036a0477ef61afdae33cde60840f4dfe481319ce3ad116ddf9"
> > +dependencies = [
> > + "autocfg",
> > +]
> > +
> > +[[package]]
> > +name = "nix"
> > +version = "0.20.1"
> > +source = "registry+https://github.com/rust-lang/crates.io-index"
> > +checksum =
> "df8e5e343312e7fbeb2a52139114e9e702991ef9c2aea6817ff2440b35647d56"
> > +dependencies = [
> > + "bitflags",
> > + "cc",
> > + "cfg-if",
> > + "libc",
> > + "memoffset",
> > +]
> > diff --git a/Cargo.toml b/Cargo.toml
> > index c4b464ff15..14131eed3c 100644
> > --- a/Cargo.toml
> > +++ b/Cargo.toml
> > @@ -1,2 +1,4 @@
> >  [workspace]
> > -members = []
> > +members = [
> > +  "rust/common",
> > +]
> > diff --git a/rust/common/Cargo.toml b/rust/common/Cargo.toml
> > new file mode 100644
> > index 0000000000..6c240447f3
> > --- /dev/null
> > +++ b/rust/common/Cargo.toml
> > @@ -0,0 +1,11 @@
> > +[package]
> > +name = "common"
> > +version = "0.1.0"
> > +edition = "2018"
> > +publish = false
> > +
> > +[dependencies]
> > +libc = "0.2.92"
> > +
> > +[target."cfg(unix)".dependencies]
> > +nix = "0.20.0"
> > diff --git a/rust/common/src/error.rs b/rust/common/src/error.rs
> > new file mode 100644
> > index 0000000000..f166ac42ea
> > --- /dev/null
> > +++ b/rust/common/src/error.rs
> > @@ -0,0 +1,113 @@
> > +use std::{self, ffi::CString, fmt, io, ptr};
> > +
> > +use crate::translate::*;
>
> It's not uncommon to ban wildcard imports that aren't preludes as it
> can make it confusing to read. Does QEMU have a stance on that type of
> thing?
>

There is no such common rule in Rust afaik. It's based on judgement and
style. If the imported symbols pollute your namespace or not. But yes, in
general, it's better to selectively import what you need, mostly for
readability.


> > +use crate::{ffi, qemu};
> > +
> > +/// Common error type for QEMU and related projects.
> > +#[derive(Debug)]
> > +pub enum Error {
> > +    /// A generic error with file and line location.
> > +    FailedAt(String, &'static str, u32),
> > +    /// An IO error.
> > +    Io(io::Error),
> > +    #[cfg(unix)]
> > +    /// A nix error.
> > +    Nix(nix::Error),
> > +}
> > +
> > +/// Alias for a `Result` with the error type for QEMU.
> > +pub type Result<T> = std::result::Result<T, Error>;
>
> I think this is very confusing. Rust developers expect `Result` to be
> the one from `std::result`, it would be better to call this
> `QEMUResult`
>

It's very common in Rust to redefine Result for your crate error. Users
don't have to import it if they prefer the std::result::Result<T,E>. This
redefinition was probably popularized with the `std::io::Result<T>` type. (
https://doc.rust-lang.org/std/io/type.Result.html)


>
> > +
> > +impl Error {
> > +    fn message(&self) -> String {
> > +        use Error::*;
>
> Do we need this here? Why not put it at the top of the file?
>

It's limited here to avoid enum prefix repetition:
match self {
 Error::FailedAt ..
 Error::Io ..
 Error::Foo ..
}

(It wouldn't be a good idea to import it in the top namespace)


> > +        match self {
> > +            FailedAt(msg, _, _) => msg.into(),
> > +            Io(io) => format!("IO error: {}", io),
> > +            #[cfg(unix)]
> > +            Nix(nix) => format!("Nix error: {}", nix),
> > +        }
> > +    }
> > +
> > +    fn location(&self) -> Option<(&'static str, u32)> {
> > +        use Error::*;
> > +        match self {
> > +            FailedAt(_, file, line) => Some((file, *line)),
> > +            Io(_) => None,
> > +            #[cfg(unix)]
> > +            Nix(_) => None,
> > +        }
> > +    }
> > +}
> > +
> > +impl fmt::Display for Error {
> > +    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
> > +        use Error::*;
> > +        match self {
> > +            FailedAt(msg, file, line) => write!(f, "{} ({}:{})", msg,
> file, line),
> > +            _ => write!(f, "{}", self.message()),
> > +        }
> > +    }
> > +}
> > +
> > +impl From<io::Error> for Error {
> > +    fn from(val: io::Error) -> Self {
> > +        Error::Io(val)
> > +    }
> > +}
> > +
> > +#[cfg(unix)]
> > +impl From<nix::Error> for Error {
> > +    fn from(val: nix::Error) -> Self {
> > +        Error::Nix(val)
> > +    }
> > +}
> > +
> > +impl QemuPtrDefault for Error {
> > +    type QemuType = *mut ffi::Error;
> > +}
> > +
> > +impl<'a> ToQemuPtr<'a, *mut ffi::Error> for Error {
> > +    type Storage = qemu::CError;
> > +
> > +    fn to_qemu_none(&'a self) -> Stash<'a, *mut ffi::Error, Self> {
> > +        let err = self.to_qemu_full();
> > +
> > +        Stash(err, unsafe { from_qemu_full(err) })
> > +    }
> > +
> > +    fn to_qemu_full(&self) -> *mut ffi::Error {
> > +        let cmsg =
> > +            CString::new(self.message()).expect("ToQemuPtr<Error>:
> unexpected '\0' character");
> > +        let mut csrc = CString::new("").unwrap();
> > +        let (src, line) = self.location().map_or((ptr::null(), 0_i32),
> |loc| {
> > +            csrc = CString::new(loc.0).expect("ToQemuPtr<Error>::
> unexpected '\0' character");
> > +            (csrc.as_ptr() as *const libc::c_char, loc.1 as i32)
> > +        });
> > +        let func = ptr::null();
> > +
> > +        let mut err: *mut ffi::Error = ptr::null_mut();
> > +        unsafe {
> > +            ffi::error_setg_internal(
> > +                &mut err as *mut *mut _,
> > +                src,
> > +                line,
> > +                func,
> > +                cmsg.as_ptr() as *const libc::c_char,
> > +            );
> > +            err
> > +        }
> > +    }
> > +}
> > +
> > +/// Convenience macro to build a [`Error::FailedAt`] error.
> > +///
> > +/// Returns a `Result::Err` with the file:line location.
> > +/// (the error can then be converted to a QEMU `ffi::Error`)
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! err {
> > +    ($msg:expr) => {
> > +        Err(Error::FailedAt($msg.into(), file!(), line!()))
> > +    };
> > +}
> > diff --git a/rust/common/src/ffi.rs b/rust/common/src/ffi.rs
> > new file mode 100644
> > index 0000000000..82818d503a
> > --- /dev/null
> > +++ b/rust/common/src/ffi.rs
> > @@ -0,0 +1,93 @@
> > +//! Bindings to the raw low-level C API commonly provided by QEMU
> projects.
> > +//!
> > +//! Manual bindings to C API availabe when linking QEMU projects.
>
> s/availabe/available/g
>
>
yup thanks

> +//! It includes minimal glib allocation functions too, since it's the
> default
> > +//! allocator used by QEMU, and we don't depend on glib-rs crate yet).
> > +//!
> > +//! Higher-level Rust-friendly bindings are provided by different
> modules.
> > +
> > +use libc::{c_char, c_void, size_t};
> > +
> > +extern "C" {
> > +    pub fn g_malloc0(n_bytes: size_t) -> *mut c_void;
> > +    pub fn g_free(ptr: *mut c_void);
> > +    pub fn g_strndup(str: *const c_char, n: size_t) -> *mut c_char;
> > +}
>
> We can get there from the glib/glib_sys crate:
>
> https://gtk-rs.org/gtk-rs-core/stable/latest/docs/glib_sys/fn.g_malloc0.html
>
> If we only plan on using these 3 I think this approach is fine, but
> something to keep in mind if we use more glib functions.
>
>
Yes, I think I mentioned this somewhere. We might need to import glib-sys
or glib-rs depending on what we need to write in Rust. We may actually not
need more than a few FFI functions though, so importing external crates for
that isn't worth it.


> > +
> > +#[repr(C)]
> > +pub struct QObject(c_void);
> > +
> > +impl ::std::fmt::Debug for QObject {
> > +    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
> > +        f.debug_struct(&format!("QObject @ {:?}", self as *const _))
> > +            .finish()
> > +    }
> > +}
> > +
> > +#[repr(C)]
> > +pub struct QNull(c_void);
> > +
> > +impl ::std::fmt::Debug for QNull {
> > +    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
> > +        f.debug_struct(&format!("QNull @ {:?}", self as *const _))
> > +            .finish()
> > +    }
> > +}
> > +
> > +#[repr(C)]
> > +pub struct Error(c_void);
> > +
> > +impl ::std::fmt::Debug for Error {
> > +    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
> > +        f.debug_struct(&format!("Error @ {:?}", self as *const _))
> > +            .finish()
> > +    }
> > +}
> > +
> > +extern "C" {
> > +    pub fn error_setg_internal(
> > +        errp: *mut *mut Error,
> > +        src: *const c_char,
> > +        line: i32,
> > +        func: *const c_char,
> > +        fmt: *const c_char,
> > +        ...
> > +    );
> > +    pub fn error_get_pretty(err: *const Error) -> *const c_char;
> > +    pub fn error_free(err: *mut Error);
> > +}
> > +
> > +/// Wrap a QMP hanlder.
>
> handler
>

thanks!


> > +#[macro_export]
> > +macro_rules! qmp {
> > +    // the basic return value variant
> > +    ($e:expr, $errp:ident, $errval:expr) => {{
> > +        assert!(!$errp.is_null());
> > +        unsafe {
> > +            *$errp = std::ptr::null_mut();
> > +        }
> > +
> > +        match $e {
> > +            Ok(val) => val,
> > +            Err(err) => unsafe {
> > +                *$errp = err.to_qemu_full();
> > +                $errval
> > +            },
> > +        }
> > +    }};
> > +    // the ptr return value variant
> > +    ($e:expr, $errp:ident) => {{
> > +        assert!(!$errp.is_null());
> > +        unsafe {
> > +            *$errp = std::ptr::null_mut();
> > +        }
> > +
> > +        match $e {
> > +            Ok(val) => val.to_qemu_full().into(),
> > +            Err(err) => unsafe {
> > +                *$errp = err.to_qemu_full();
> > +                std::ptr::null_mut()
> > +            },
> > +        }
> > +    }};
> > +}
>
> It would be a good idea to document why this code is safe
>

Hmm, I am not sure that's the question. I assume Rust code is safe :)
However, the FFI borders are full of unsafe {}. It's basically a trust
relationship with the other side. But it's always great to document tricky
unsafe {}, or where panic is unexpected (unwrap()/expect()).. based on
review, judgement, style.. (read also
https://doc.rust-lang.org/book/ch09-03-to-panic-or-not-to-panic.html).

Thanks!

Alistair
>
> > diff --git a/rust/common/src/lib.rs b/rust/common/src/lib.rs
> > new file mode 100644
> > index 0000000000..4de826bc2e
> > --- /dev/null
> > +++ b/rust/common/src/lib.rs
> > @@ -0,0 +1,21 @@
> > +//! Common code for QEMU
> > +//!
> > +//! This crates provides common bindings and facilities for QEMU C API
> shared by
> > +//! various projects. Most importantly, it defines the conversion
> traits used to
> > +//! convert from C to Rust types. Those traits are largely adapted from
> glib-rs,
> > +//! since those have prooven to be very flexible, and should guide us
> to bind
> > +//! further QEMU types such as QOM. If glib-rs becomes a dependency, we
> should
> > +//! consider adopting glib translate traits. For QAPI, we need a
> smaller subset.
> > +
> > +pub use libc;
> > +
> > +mod error;
> > +pub use error::*;
> > +
> > +mod qemu;
> > +pub use qemu::*;
> > +
> > +mod translate;
> > +pub use translate::*;
> > +
> > +pub mod ffi;
> > diff --git a/rust/common/src/qemu.rs b/rust/common/src/qemu.rs
> > new file mode 100644
> > index 0000000000..dd01c6d92d
> > --- /dev/null
> > +++ b/rust/common/src/qemu.rs
> > @@ -0,0 +1,101 @@
> > +use std::{ffi::CStr, ptr, str};
> > +
> > +use crate::{ffi, translate};
> > +use translate::{FromQemuPtrFull, FromQemuPtrNone, QemuPtrDefault,
> Stash, ToQemuPtr};
> > +
> > +/// A type representing an owned C QEMU Error.
> > +pub struct CError(ptr::NonNull<ffi::Error>);
> > +
> > +impl translate::FromQemuPtrFull<*mut ffi::Error> for CError {
> > +    unsafe fn from_qemu_full(ptr: *mut ffi::Error) -> Self {
> > +        assert!(!ptr.is_null());
> > +        Self(ptr::NonNull::new_unchecked(ptr))
> > +    }
> > +}
> > +
> > +impl CError {
> > +    pub fn pretty(&self) -> &str {
> > +        unsafe {
> > +            let pretty = ffi::error_get_pretty(self.0.as_ptr());
> > +            let bytes = CStr::from_ptr(pretty).to_bytes();
> > +            str::from_utf8(bytes)
> > +                .unwrap_or_else(|err|
> str::from_utf8(&bytes[..err.valid_up_to()]).unwrap())
> > +        }
> > +    }
> > +}
> > +
> > +impl Drop for CError {
> > +    fn drop(&mut self) {
> > +        unsafe { ffi::error_free(self.0.as_ptr()) }
> > +    }
> > +}
> > +
> > +/// QObject (JSON object)
> > +#[derive(Clone, Debug)]
> > +pub struct QObject;
> > +
> > +impl QemuPtrDefault for QObject {
> > +    type QemuType = *mut ffi::QObject;
> > +}
> > +
> > +impl FromQemuPtrFull<*mut ffi::QObject> for QObject {
> > +    #[inline]
> > +    unsafe fn from_qemu_full(_ffi: *mut ffi::QObject) -> Self {
> > +        unimplemented!()
> > +    }
> > +}
> > +
> > +impl FromQemuPtrNone<*const ffi::QObject> for QObject {
> > +    #[inline]
> > +    unsafe fn from_qemu_none(_ffi: *const ffi::QObject) -> Self {
> > +        unimplemented!()
> > +    }
> > +}
> > +
> > +impl<'a> ToQemuPtr<'a, *mut ffi::QObject> for QObject {
> > +    type Storage = ();
> > +
> > +    #[inline]
> > +    fn to_qemu_none(&self) -> Stash<'a, *mut ffi::QObject, QObject> {
> > +        unimplemented!()
> > +    }
> > +    #[inline]
> > +    fn to_qemu_full(&self) -> *mut ffi::QObject {
> > +        unimplemented!()
> > +    }
> > +}
> > +
> > +/// QNull (JSON null)
> > +#[derive(Clone, Debug)]
> > +pub struct QNull;
> > +
> > +impl QemuPtrDefault for QNull {
> > +    type QemuType = *mut ffi::QNull;
> > +}
> > +
> > +impl FromQemuPtrFull<*mut ffi::QObject> for QNull {
> > +    #[inline]
> > +    unsafe fn from_qemu_full(_ffi: *mut ffi::QObject) -> Self {
> > +        unimplemented!()
> > +    }
> > +}
> > +
> > +impl FromQemuPtrNone<*const ffi::QObject> for QNull {
> > +    #[inline]
> > +    unsafe fn from_qemu_none(_ffi: *const ffi::QObject) -> Self {
> > +        unimplemented!()
> > +    }
> > +}
> > +
> > +impl<'a> ToQemuPtr<'a, *mut ffi::QNull> for QNull {
> > +    type Storage = ();
> > +
> > +    #[inline]
> > +    fn to_qemu_none(&self) -> Stash<'a, *mut ffi::QNull, QNull> {
> > +        unimplemented!()
> > +    }
> > +    #[inline]
> > +    fn to_qemu_full(&self) -> *mut ffi::QNull {
> > +        unimplemented!()
> > +    }
> > +}
> > diff --git a/rust/common/src/qmp.rs b/rust/common/src/qmp.rs
> > new file mode 100644
> > index 0000000000..e69de29bb2
> > diff --git a/rust/common/src/translate.rs b/rust/common/src/translate.rs
> > new file mode 100644
> > index 0000000000..315e14fa25
> > --- /dev/null
> > +++ b/rust/common/src/translate.rs
> > @@ -0,0 +1,482 @@
> > +// largely adapted from glib-rs
> > +// we don't depend on glib-rs as this brings a lot more code that we
> may not need
> > +// and also because there are issues with the conversion traits for our
> ffi::*mut.
> > +use libc::{c_char, size_t};
> > +use std::ffi::{CStr, CString};
> > +use std::ptr;
> > +
> > +use crate::ffi;
> > +
> > +/// A pointer.
> > +pub trait Ptr: Copy + 'static {
> > +    fn is_null(&self) -> bool;
> > +    fn from<X>(ptr: *mut X) -> Self;
> > +    fn to<X>(self) -> *mut X;
> > +}
> > +
> > +impl<T: 'static> Ptr for *const T {
> > +    #[inline]
> > +    fn is_null(&self) -> bool {
> > +        (*self).is_null()
> > +    }
> > +
> > +    #[inline]
> > +    fn from<X>(ptr: *mut X) -> *const T {
> > +        ptr as *const T
> > +    }
> > +
> > +    #[inline]
> > +    fn to<X>(self) -> *mut X {
> > +        self as *mut X
> > +    }
> > +}
> > +
> > +impl<T: 'static> Ptr for *mut T {
> > +    #[inline]
> > +    fn is_null(&self) -> bool {
> > +        (*self).is_null()
> > +    }
> > +
> > +    #[inline]
> > +    fn from<X>(ptr: *mut X) -> *mut T {
> > +        ptr as *mut T
> > +    }
> > +
> > +    #[inline]
> > +    fn to<X>(self) -> *mut X {
> > +        self as *mut X
> > +    }
> > +}
> > +
> > +/// Macro to declare a `NewPtr` struct.
> > +///
> > +/// A macro to declare a newtype for pointers, to workaround that *T
> are not
> > +/// defined in our binding crates, and allow foreign traits
> implementations.
> > +/// (this is used by qapi-gen bindings)
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +#[doc(hidden)]
> > +macro_rules! new_ptr {
> > +    () => {
> > +        #[derive(Copy, Clone)]
> > +        pub struct NewPtr<P: Ptr>(pub P);
> > +
> > +        impl<P: Ptr> Ptr for NewPtr<P> {
> > +            #[inline]
> > +            fn is_null(&self) -> bool {
> > +                self.0.is_null()
> > +            }
> > +
> > +            #[inline]
> > +            fn from<X>(ptr: *mut X) -> Self {
> > +                NewPtr(P::from(ptr))
> > +            }
> > +
> > +            #[inline]
> > +            fn to<X>(self) -> *mut X {
> > +                self.0.to()
> > +            }
> > +        }
> > +    };
> > +}
> > +
> > +/// Provides the default pointer type to be used in some container
> conversions.
> > +///
> > +/// It's `*mut c_char` for `String`, `*mut ffi::GuestInfo` for
> `GuestInfo`...
> > +pub trait QemuPtrDefault {
> > +    type QemuType: Ptr;
> > +}
> > +
> > +impl QemuPtrDefault for String {
> > +    type QemuType = *mut c_char;
> > +}
> > +
> > +/// A Stash contains the temporary storage and a pointer into it.
> > +///
> > +/// The pointer is valid for the lifetime of the `Stash`. As the
> lifetime of the
> > +/// `Stash` returned from `to_qemu_none` is at least the enclosing
> statement,
> > +/// you can avoid explicitly binding the stash in most cases and just
> take the
> > +/// pointer out of it:
> > +///
> > +/// ```ignore
> > +///     pub fn set_device_name(&self, name: &str) {
> > +///         unsafe {
> > +///             ffi::qemu_device_set_name(self.pointer,
> name.to_qemu_none().0)
> > +///         }
> > +///     }
> > +/// ```
> > +pub struct Stash<'a, P: Copy, T: ?Sized + ToQemuPtr<'a, P>>(
> > +    pub P,
> > +    pub <T as ToQemuPtr<'a, P>>::Storage,
> > +);
> > +
> > +/// Translate to a pointer.
> > +pub trait ToQemuPtr<'a, P: Copy> {
> > +    type Storage;
> > +
> > +    /// The pointer in the `Stash` is only valid for the lifetime of
> the `Stash`.
> > +    fn to_qemu_none(&'a self) -> Stash<'a, P, Self>;
> > +
> > +    /// Transfer the ownership to the ffi.
> > +    fn to_qemu_full(&self) -> P {
> > +        unimplemented!();
> > +    }
> > +}
> > +
> > +impl<'a, P: Ptr, T: ToQemuPtr<'a, P>> ToQemuPtr<'a, P> for Option<T> {
> > +    type Storage = Option<<T as ToQemuPtr<'a, P>>::Storage>;
> > +
> > +    #[inline]
> > +    fn to_qemu_none(&'a self) -> Stash<'a, P, Option<T>> {
> > +        self.as_ref()
> > +            .map_or(Stash(Ptr::from::<()>(ptr::null_mut()), None), |s| {
> > +                let s = s.to_qemu_none();
> > +                Stash(s.0, Some(s.1))
> > +            })
> > +    }
> > +
> > +    #[inline]
> > +    fn to_qemu_full(&self) -> P {
> > +        self.as_ref()
> > +            .map_or(Ptr::from::<()>(ptr::null_mut()),
> ToQemuPtr::to_qemu_full)
> > +    }
> > +}
> > +
> > +impl<'a, P: Ptr, T: ToQemuPtr<'a, P>> ToQemuPtr<'a, P> for Box<T> {
> > +    type Storage = <T as ToQemuPtr<'a, P>>::Storage;
> > +
> > +    #[inline]
> > +    fn to_qemu_none(&'a self) -> Stash<'a, P, Box<T>> {
> > +        let s = self.as_ref().to_qemu_none();
> > +        Stash(s.0, s.1)
> > +    }
> > +
> > +    #[inline]
> > +    fn to_qemu_full(&self) -> P {
> > +        ToQemuPtr::to_qemu_full(self.as_ref())
> > +    }
> > +}
> > +
> > +impl<'a> ToQemuPtr<'a, *mut c_char> for String {
> > +    type Storage = CString;
> > +
> > +    #[inline]
> > +    fn to_qemu_none(&self) -> Stash<'a, *mut c_char, String> {
> > +        let tmp = CString::new(&self[..])
> > +            .expect("String::ToQemuPtr<*mut c_char>: unexpected '\0'
> character");
> > +        Stash(tmp.as_ptr() as *mut c_char, tmp)
> > +    }
> > +
> > +    #[inline]
> > +    fn to_qemu_full(&self) -> *mut c_char {
> > +        unsafe { ffi::g_strndup(self.as_ptr() as *const c_char,
> self.len() as size_t) }
> > +    }
> > +}
> > +
> > +/// Translate from a pointer type, without taking ownership.
> > +pub trait FromQemuPtrNone<P: Ptr>: Sized {
> > +    /// # Safety
> > +    ///
> > +    /// `ptr` must be a valid pointer. It is not referenced after the
> call.
> > +    unsafe fn from_qemu_none(ptr: P) -> Self;
> > +}
> > +
> > +/// Translate from a pointer type, taking ownership.
> > +pub trait FromQemuPtrFull<P: Ptr>: Sized {
> > +    /// # Safety
> > +    ///
> > +    /// `ptr` must be a valid pointer. Ownership is transferred.
> > +    unsafe fn from_qemu_full(ptr: P) -> Self;
> > +}
> > +
> > +/// See [`FromQemuPtrNone`](trait.FromQemuPtrNone.html).
> > +#[inline]
> > +#[allow(clippy::missing_safety_doc)]
> > +pub unsafe fn from_qemu_none<P: Ptr, T: FromQemuPtrNone<P>>(ptr: P) ->
> T {
> > +    FromQemuPtrNone::from_qemu_none(ptr)
> > +}
> > +
> > +/// See [`FromQemuPtrFull`](trait.FromQemuPtrFull.html).
> > +#[inline]
> > +#[allow(clippy::missing_safety_doc)]
> > +pub unsafe fn from_qemu_full<P: Ptr, T: FromQemuPtrFull<P>>(ptr: P) ->
> T {
> > +    FromQemuPtrFull::from_qemu_full(ptr)
> > +}
> > +
> > +impl<P: Ptr, T: FromQemuPtrNone<P>> FromQemuPtrNone<P> for Option<T> {
> > +    #[inline]
> > +    unsafe fn from_qemu_none(ptr: P) -> Option<T> {
> > +        if ptr.is_null() {
> > +            None
> > +        } else {
> > +            Some(from_qemu_none(ptr))
> > +        }
> > +    }
> > +}
> > +
> > +impl<P: Ptr, T: FromQemuPtrFull<P>> FromQemuPtrFull<P> for Option<T> {
> > +    #[inline]
> > +    unsafe fn from_qemu_full(ptr: P) -> Option<T> {
> > +        if ptr.is_null() {
> > +            None
> > +        } else {
> > +            Some(from_qemu_full(ptr))
> > +        }
> > +    }
> > +}
> > +
> > +impl FromQemuPtrNone<*const c_char> for String {
> > +    #[inline]
> > +    unsafe fn from_qemu_none(ptr: *const c_char) -> Self {
> > +        assert!(!ptr.is_null());
> > +
> String::from_utf8_lossy(CStr::from_ptr(ptr).to_bytes()).into_owned()
> > +    }
> > +}
> > +
> > +impl FromQemuPtrFull<*mut c_char> for String {
> > +    #[inline]
> > +    unsafe fn from_qemu_full(ptr: *mut c_char) -> Self {
> > +        let res = from_qemu_none(ptr as *const _);
> > +        ffi::g_free(ptr as *mut _);
> > +        res
> > +    }
> > +}
> > +
> > +#[doc(hidden)]
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! vec_ffi_wrapper {
> > +    ($ffi:ident) => {
> > +        #[allow(non_camel_case_types)]
> > +        pub struct $ffi(*mut qapi_ffi::$ffi);
> > +
> > +        impl Drop for $ffi {
> > +            fn drop(&mut self) {
> > +                let mut list = self.0;
> > +                unsafe {
> > +                    while !list.is_null() {
> > +                        let next = (*list).next;
> > +                        Box::from_raw(list);
> > +                        list = next;
> > +                    }
> > +                }
> > +            }
> > +        }
> > +
> > +        impl From<NewPtr<*mut qapi_ffi::$ffi>> for *mut qapi_ffi::$ffi {
> > +            fn from(p: NewPtr<*mut qapi_ffi::$ffi>) -> Self {
> > +                p.0
> > +            }
> > +        }
> > +    };
> > +}
> > +
> > +#[doc(hidden)]
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! impl_vec_scalars_to_qemu {
> > +    ($rs:ty, $ffi:ident) => {
> > +        impl<'a> ToQemuPtr<'a, NewPtr<*mut qapi_ffi::$ffi>> for
> Vec<$rs> {
> > +            type Storage = $ffi;
> > +
> > +            #[inline]
> > +            fn to_qemu_none(&self) -> Stash<NewPtr<*mut
> qapi_ffi::$ffi>, Self> {
> > +                let mut list: *mut qapi_ffi::$ffi =
> std::ptr::null_mut();
> > +                for value in self.iter().rev() {
> > +                    let b = Box::new(qapi_ffi::$ffi {
> > +                        next: list,
> > +                        value: *value,
> > +                    });
> > +                    list = Box::into_raw(b);
> > +                }
> > +                Stash(NewPtr(list), $ffi(list))
> > +            }
> > +
> > +            #[inline]
> > +            fn to_qemu_full(&self) -> NewPtr<*mut qapi_ffi::$ffi> {
> > +                let mut list: *mut qapi_ffi::$ffi =
> std::ptr::null_mut();
> > +                unsafe {
> > +                    for value in self.iter().rev() {
> > +                        let l =
> ffi::g_malloc0(std::mem::size_of::<qapi_ffi::$ffi>())
> > +                            as *mut qapi_ffi::$ffi;
> > +                        (*l).next = list;
> > +                        (*l).value = *value;
> > +                        list = l;
> > +                    }
> > +                }
> > +                NewPtr(list)
> > +            }
> > +        }
> > +    };
> > +}
> > +
> > +#[doc(hidden)]
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! impl_vec_scalars_from_qemu {
> > +    ($rs:ty, $ffi:ident, $free_ffi:ident) => {
> > +        impl FromQemuPtrFull<NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
> > +            #[inline]
> > +            unsafe fn from_qemu_full(ffi: NewPtr<*mut qapi_ffi::$ffi>)
> -> Self {
> > +                let ret = from_qemu_none(NewPtr(ffi.0 as *const _));
> > +                qapi_ffi::$free_ffi(ffi.0);
> > +                ret
> > +            }
> > +        }
> > +
> > +        impl FromQemuPtrNone<NewPtr<*const qapi_ffi::$ffi>> for
> Vec<$rs> {
> > +            #[inline]
> > +            unsafe fn from_qemu_none(ffi: NewPtr<*const
> qapi_ffi::$ffi>) -> Self {
> > +                let mut ret = vec![];
> > +                let mut it = ffi.0;
> > +                while !it.is_null() {
> > +                    let e = &*it;
> > +                    ret.push(e.value);
> > +                    it = e.next;
> > +                }
> > +                ret
> > +            }
> > +        }
> > +    };
> > +}
> > +
> > +#[doc(hidden)]
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! impl_vec_to_qemu {
> > +    ($rs:ty, $ffi:ident) => {
> > +        // impl doesn't use only types from inside the current crate
> > +        // impl QemuPtrDefault for Vec<$rs> {
> > +        //     type QemuType = NewPtr<*mut qapi_ffi::$ffi>;
> > +        // }
> > +
> > +        impl<'a> ToQemuPtr<'a, NewPtr<*mut qapi_ffi::$ffi>> for
> Vec<$rs> {
> > +            type Storage = ($ffi, Vec<Stash<'a, <$rs as
> QemuPtrDefault>::QemuType, $rs>>);
> > +
> > +            #[inline]
> > +            fn to_qemu_none(&self) -> Stash<NewPtr<*mut
> qapi_ffi::$ffi>, Self> {
> > +                let stash_vec: Vec<_> =
> self.iter().rev().map(ToQemuPtr::to_qemu_none).collect();
> > +                let mut list: *mut qapi_ffi::$ffi =
> std::ptr::null_mut();
> > +                for stash in &stash_vec {
> > +                    let b = Box::new(qapi_ffi::$ffi {
> > +                        next: list,
> > +                        value: Ptr::to(stash.0),
> > +                    });
> > +                    list = Box::into_raw(b);
> > +                }
> > +                Stash(NewPtr(list), ($ffi(list), stash_vec))
> > +            }
> > +
> > +            #[inline]
> > +            fn to_qemu_full(&self) -> NewPtr<*mut qapi_ffi::$ffi> {
> > +                let v: Vec<_> =
> self.iter().rev().map(ToQemuPtr::to_qemu_full).collect();
> > +                let mut list: *mut qapi_ffi::$ffi =
> std::ptr::null_mut();
> > +                unsafe {
> > +                    for val in v {
> > +                        let l =
> ffi::g_malloc0(std::mem::size_of::<qapi_ffi::$ffi>())
> > +                            as *mut qapi_ffi::$ffi;
> > +                        (*l).next = list;
> > +                        (*l).value = val;
> > +                        list = l;
> > +                    }
> > +                }
> > +                NewPtr(list)
> > +            }
> > +        }
> > +    };
> > +}
> > +
> > +#[doc(hidden)]
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! impl_vec_from_qemu {
> > +    ($rs:ty, $ffi:ident, $free_ffi:ident) => {
> > +        impl FromQemuPtrFull<NewPtr<*mut qapi_ffi::$ffi>> for Vec<$rs> {
> > +            #[inline]
> > +            unsafe fn from_qemu_full(ffi: NewPtr<*mut qapi_ffi::$ffi>)
> -> Self {
> > +                let ret = from_qemu_none(NewPtr(ffi.0 as *const _));
> > +                qapi_ffi::$free_ffi(ffi.0);
> > +                ret
> > +            }
> > +        }
> > +
> > +        impl FromQemuPtrNone<NewPtr<*const qapi_ffi::$ffi>> for
> Vec<$rs> {
> > +            #[inline]
> > +            unsafe fn from_qemu_none(ffi: NewPtr<*const
> qapi_ffi::$ffi>) -> Self {
> > +                let mut ret = vec![];
> > +                let mut it = ffi.0;
> > +                while !it.is_null() {
> > +                    let e = &*it;
> > +                    ret.push(from_qemu_none(e.value as *const _));
> > +                    it = e.next;
> > +                }
> > +                ret
> > +            }
> > +        }
> > +    };
> > +}
> > +
> > +/// A macro to help the implementation of `Vec<T>` translations.
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! vec_type {
> > +    (Vec<$rs:ty>, $ffi:ident, $free_ffi:ident, 0) => {
> > +        vec_ffi_wrapper!($ffi);
> > +        impl_vec_from_qemu!($rs, $ffi, $free_ffi);
> > +        impl_vec_to_qemu!($rs, $ffi);
> > +    };
> > +    (Vec<$rs:ty>, $ffi:ident, $free_ffi:ident, 1) => {
> > +        vec_ffi_wrapper!($ffi);
> > +        impl_vec_scalars_from_qemu!($rs, $ffi, $free_ffi);
> > +        impl_vec_scalars_to_qemu!($rs, $ffi);
> > +    };
> > +}
> > +
> > +/// A macro to implement [`ToQemuPtr`] as boxed scalars
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! impl_to_qemu_scalar_boxed {
> > +    ($ty:ty) => {
> > +        impl<'a> ToQemuPtr<'a, *mut $ty> for $ty {
> > +            type Storage = Box<$ty>;
> > +
> > +            fn to_qemu_none(&'a self) -> Stash<'a, *mut $ty, Self> {
> > +                let mut box_ = Box::new(*self);
> > +                Stash(&mut *box_, box_)
> > +            }
> > +
> > +            fn to_qemu_full(&self) -> *mut $ty {
> > +                unsafe {
> > +                    let ptr =
> ffi::g_malloc0(std::mem::size_of::<$ty>()) as *mut _;
> > +                    *ptr = *self;
> > +                    ptr
> > +                }
> > +            }
> > +        }
> > +    };
> > +}
> > +
> > +/// A macro to implement [`FromQemuPtrNone`] for scalar pointers.
> > +#[allow(unused_macros)]
> > +#[macro_export]
> > +macro_rules! impl_from_qemu_none_scalar {
> > +    ($ty:ty) => {
> > +        impl FromQemuPtrNone<*const $ty> for $ty {
> > +            unsafe fn from_qemu_none(ptr: *const $ty) -> Self {
> > +                *ptr
> > +            }
> > +        }
> > +    };
> > +}
> > +
> > +macro_rules! impl_scalar_boxed {
> > +    ($($t:ident)*) => (
> > +        $(
> > +            impl_to_qemu_scalar_boxed!($t);
> > +            impl_from_qemu_none_scalar!($t);
> > +        )*
> > +    )
> > +}
> > +
> > +// the only built-in used so far, feel free to add more as needed
> > +impl_scalar_boxed!(bool i64 f64);
> > --
> > 2.33.0.113.g6c40894d24
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 49488 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 01/32] RFC: docs: add supported host CPUs section
  2021-09-07 12:33   ` Peter Maydell
@ 2021-09-13 11:32     ` Marc-André Lureau
  2021-09-13 11:46       ` Peter Maydell
  0 siblings, 1 reply; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-13 11:32 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU Developers,
	Stefan Hajnoczi, Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 3261 bytes --]

Hi

On Tue, Sep 7, 2021 at 4:34 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Tue, 7 Sept 2021 at 13:23, <marcandre.lureau@redhat.com> wrote:
> >
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > I was looking for such documentation, but couldn't find it.
>
> Yes; this is definitely something we should document, and in
> the build-platforms doc is as good a place as any.
>
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  docs/about/build-platforms.rst | 28 ++++++++++++++++++++++++++++
> >  meson.build                    |  2 +-
> >  2 files changed, 29 insertions(+), 1 deletion(-)
> >
> > diff --git a/docs/about/build-platforms.rst
> b/docs/about/build-platforms.rst
> > index 692323609e..bfe90e574e 100644
> > --- a/docs/about/build-platforms.rst
> > +++ b/docs/about/build-platforms.rst
> > @@ -29,6 +29,34 @@ The `Repology`_ site is a useful resource to identify
> >  currently shipped versions of software in various operating systems,
> >  though it does not cover all distros listed below.
> >
> > +Supported host CPUs
> > +-------------------
> > +
> > +Those host CPUs have a native TCG backend and are regularly tested:
>
> This is a list of host architectures, not CPUs.
>

Isn't it CPU architecture we are talking about? (CPU for short in the title)


> > +  .. list-table::
> > +   :header-rows: 1
> > +
> > +   * - CPU Family
>

 I'll change this to CPU Architecture

> +     - Accelerators
> > +   * - ARM
>
> The correct capitalization these days is "Arm", by the way :-)
>
>
ok

You also should split 64-bit and 32-bit Arm; we support
> KVM on 64-bit but not 32-bit.
>
>
When such a difference exists, I just added "(64 bit only)", see below for
x86.


> > +     - kvm, xen
> > +   * - MIPS
> > +     - kvm
> > +   * - PPC
> > +     - kvm
> > +   * - RISC-V
> > +     -
> > +   * - s390x
> > +     - kvm
> > +   * - SPARC
> > +     -
> > +   * - x86
> > +     - kvm, xen, hax, hvf (64 bit only), nvmm, whpx (64 bit only)
> > +
> > +Other architectures are not actively maintained. They use the slow and
> > +experimental TCG interpreter. They may be removed in future releases.
>
> This seems to be conflating TCG and the TCG interpreter.
> We should just list which architectures we support (proper)
> TCG for, and say that everything else is unsupported
> (not mentioning the TCG interpreter at all; using it is
> pretty much always a mistake IMHO).
>

ok


> The table also seems to me to be a bit confusing, because
> the introductory text suggests it's a list of the TCG
> support for each architecture, but the table itself lists
> only the non-TCG accelerators. I think we should just list
> all the accelerators supported for each host architecture.
>

All the architectures we support (in the list) have proper TCG, right?

>
> Perhaps we should also (eventually) have somewhere some text
> describing each accelerator in more detail, though probably
> not in this file. A docs/system/accels.rst that described all
> the accelerators with a paragraph or so for each, maybe ?
>

That could be really useful, but I am not up to the task at this point.

thanks

[-- Attachment #2: Type: text/html, Size: 5216 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 01/32] RFC: docs: add supported host CPUs section
  2021-09-13 11:32     ` Marc-André Lureau
@ 2021-09-13 11:46       ` Peter Maydell
  0 siblings, 0 replies; 74+ messages in thread
From: Peter Maydell @ 2021-09-13 11:46 UTC (permalink / raw)
  To: Marc-André Lureau
  Cc: Paolo Bonzini, Daniel P. Berrange, QEMU Developers,
	Stefan Hajnoczi, Markus Armbruster

On Mon, 13 Sept 2021 at 12:32, Marc-André Lureau
<marcandre.lureau@redhat.com> wrote:
>
> Hi
>
> On Tue, Sep 7, 2021 at 4:34 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Tue, 7 Sept 2021 at 13:23, <marcandre.lureau@redhat.com> wrote:
>> >
>> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
>> >
>> > I was looking for such documentation, but couldn't find it.
>>
>> Yes; this is definitely something we should document, and in
>> the build-platforms doc is as good a place as any.
>>
>> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
>> > ---
>> >  docs/about/build-platforms.rst | 28 ++++++++++++++++++++++++++++
>> >  meson.build                    |  2 +-
>> >  2 files changed, 29 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
>> > index 692323609e..bfe90e574e 100644
>> > --- a/docs/about/build-platforms.rst
>> > +++ b/docs/about/build-platforms.rst
>> > @@ -29,6 +29,34 @@ The `Repology`_ site is a useful resource to identify
>> >  currently shipped versions of software in various operating systems,
>> >  though it does not cover all distros listed below.
>> >
>> > +Supported host CPUs
>> > +-------------------
>> > +
>> > +Those host CPUs have a native TCG backend and are regularly tested:
>>
>> This is a list of host architectures, not CPUs.
>
>
> Isn't it CPU architecture we are talking about? (CPU for short in the title)

My point is that "CPU" != "CPU architecture". "CPU" is something
like "Skylake" or "Cortex-A15". "CPU architecture" is "x86-64",
"arm", etc.

>> The table also seems to me to be a bit confusing, because
>> the introductory text suggests it's a list of the TCG
>> support for each architecture, but the table itself lists
>> only the non-TCG accelerators. I think we should just list
>> all the accelerators supported for each host architecture.
>
>
> All the architectures we support (in the list) have proper TCG, right?

Yes.

thanks
-- PMM


^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 02/32] build-sys: add HAVE_IPPROTO_MPTCP
  2021-09-07 12:19 ` [RFC v3 02/32] build-sys: add HAVE_IPPROTO_MPTCP marcandre.lureau
  2021-09-08 12:01   ` Markus Armbruster
@ 2021-09-13 13:02   ` Paolo Bonzini
  1 sibling, 0 replies; 74+ messages in thread
From: Paolo Bonzini @ 2021-09-13 13:02 UTC (permalink / raw)
  To: marcandre.lureau, qemu-devel; +Cc: berrange, armbru, stefanha

On 07/09/21 14:19, marcandre.lureau@redhat.com wrote:
> From: Marc-André Lureau <marcandre.lureau@redhat.com>
> 
> The QAPI schema shouldn't rely on C system headers #define, but on
> configure-time project #define, so we can express the build condition in
> a C-independent way.
> 
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>   meson.build         | 2 ++
>   qapi/sockets.json   | 2 +-
>   io/dns-resolver.c   | 2 +-
>   util/qemu-sockets.c | 6 +++---
>   4 files changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/meson.build b/meson.build
> index 9e43c9b311..6e871af4d0 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -1367,6 +1367,8 @@ config_host_data.set('HAVE_OPTRESET',
>                        cc.has_header_symbol('getopt.h', 'optreset'))
>   config_host_data.set('HAVE_UTMPX',
>                        cc.has_header_symbol('utmpx.h', 'struct utmpx'))
> +config_host_data.set('HAVE_IPPROTO_MPTCP',
> +                     cc.has_header_symbol('netinet/in.h', 'IPPROTO_MPTCP'))
>   
>   # has_member
>   config_host_data.set('HAVE_SIGEV_NOTIFY_THREAD_ID',
> diff --git a/qapi/sockets.json b/qapi/sockets.json
> index 7866dc27d6..c9101f937f 100644
> --- a/qapi/sockets.json
> +++ b/qapi/sockets.json
> @@ -69,7 +69,7 @@
>       '*ipv4': 'bool',
>       '*ipv6': 'bool',
>       '*keep-alive': 'bool',
> -    '*mptcp': { 'type': 'bool', 'if': 'IPPROTO_MPTCP' } } }
> +    '*mptcp': { 'type': 'bool', 'if': 'HAVE_IPPROTO_MPTCP' } } }
>   
>   ##
>   # @UnixSocketAddress:
> diff --git a/io/dns-resolver.c b/io/dns-resolver.c
> index a5946a93bf..53b0e8407a 100644
> --- a/io/dns-resolver.c
> +++ b/io/dns-resolver.c
> @@ -122,7 +122,7 @@ static int qio_dns_resolver_lookup_sync_inet(QIODNSResolver *resolver,
>               .ipv4 = iaddr->ipv4,
>               .has_ipv6 = iaddr->has_ipv6,
>               .ipv6 = iaddr->ipv6,
> -#ifdef IPPROTO_MPTCP
> +#ifdef HAVE_IPPROTO_MPTCP
>               .has_mptcp = iaddr->has_mptcp,
>               .mptcp = iaddr->mptcp,
>   #endif
> diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c
> index f2f3676d1f..02eb2f3d34 100644
> --- a/util/qemu-sockets.c
> +++ b/util/qemu-sockets.c
> @@ -278,7 +278,7 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
>   
>       /* create socket + bind/listen */
>       for (e = res; e != NULL; e = e->ai_next) {
> -#ifdef IPPROTO_MPTCP
> +#ifdef HAVE_IPPROTO_MPTCP
>           if (saddr->has_mptcp && saddr->mptcp) {
>               e->ai_protocol = IPPROTO_MPTCP;
>           }
> @@ -462,7 +462,7 @@ int inet_connect_saddr(InetSocketAddress *saddr, Error **errp)
>           error_free(local_err);
>           local_err = NULL;
>   
> -#ifdef IPPROTO_MPTCP
> +#ifdef HAVE_IPPROTO_MPTCP
>           if (saddr->has_mptcp && saddr->mptcp) {
>               e->ai_protocol = IPPROTO_MPTCP;
>           }
> @@ -699,7 +699,7 @@ int inet_parse(InetSocketAddress *addr, const char *str, Error **errp)
>           }
>           addr->has_keep_alive = true;
>       }
> -#ifdef IPPROTO_MPTCP
> +#ifdef HAVE_IPPROTO_MPTCP
>       begin = strstr(optstr, ",mptcp");
>       if (begin) {
>           if (inet_parse_flag("mptcp", begin + strlen(",mptcp"),
> 

Queued, thanks.

Paolo



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 13/32] rust: use vendored-sources
  2021-09-09 16:29     ` Marc-André Lureau
  2021-09-09 16:53       ` Daniel P. Berrangé
@ 2021-09-13 14:21       ` Paolo Bonzini
  1 sibling, 0 replies; 74+ messages in thread
From: Paolo Bonzini @ 2021-09-13 14:21 UTC (permalink / raw)
  To: Marc-André Lureau, Peter Maydell
  Cc: Ian Jackson, Daniel P. Berrange, QEMU Developers,
	Stefan Hajnoczi, Markus Armbruster

On 09/09/21 18:29, Marc-André Lureau wrote:
> 
>       * C doesn't have a package manager, so if we need a dependency that
>         distros don't ship then we need to wrap it up and provide it
>     ourselves
> 
> Have we considered meson wrap? I never really looked at it in detail, 
> not sure if that would work for us. 
> https://mesonbuild.com/Wrap-dependency-system-manual.html 
> <https://mesonbuild.com/Wrap-dependency-system-manual.html>

Sure, it would be possible to use wrap with meson 0.56.x or newer (due 
to https://github.com/mesonbuild/meson/pull/7740).  It would all be 
hidden behind configure/Makefile, which would invoke "meson subprojects 
download" in addition to "git submodule update".

Paolo



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 12/32] rust: provide a common crate for QEMU
  2021-09-07 12:19 ` [RFC v3 12/32] rust: provide a common crate for QEMU marcandre.lureau
  2021-09-10  1:18   ` Alistair Francis
@ 2021-09-13 17:11   ` Paolo Bonzini
  2021-09-14 11:34     ` Marc-André Lureau
  1 sibling, 1 reply; 74+ messages in thread
From: Paolo Bonzini @ 2021-09-13 17:11 UTC (permalink / raw)
  To: marcandre.lureau, qemu-devel; +Cc: berrange, armbru, stefanha

On 07/09/21 14:19, marcandre.lureau@redhat.com wrote:
> From: Marc-André Lureau<marcandre.lureau@redhat.com>
> 
> This crates provides common bindings and facilities for QEMU C API
> shared by various projects.
> 
> Most importantly, it defines the conversion traits used to convert from
> C to Rust types. Those traits are largely adapted from glib-rs, since
> those have proved to be very flexible, and should guide us to bind
> further QEMU types such as QOM. If glib-rs becomes a dependency, we
> should consider adopting glib translate traits. For QAPI, we need a
> smaller subset.
> 
> Cargo.lock is checked-in, as QEMU produces end-of-chain binaries, and it
> is desirable to track the exact set of packages that are involved in
> managed builds.
> 
> Signed-off-by: Marc-André Lureau<marcandre.lureau@redhat.com>

As in my previous review, the main issue I have here is with the 
complexity of this code.

I understand that this has to be manually written, but right now I find 
it really hard to understand what is going on here.  The patch needs to 
be expanded in several parts:

1) generic traits (including implementations for Option/Box)

2) implementation of the generic traits

3) io/nix errors

and these parts should be moved around to the place where they become 
necessary.


Also regarding the code itself:

1) Stash should not be a tuple.  Accesses to it should use standard Rust 
methods, such as borrow()/borrow_mut(), and it should also support 
standard Rust idioms such as map():

pub struct BorrowedMutPointer<'a, P, T: 'a> {
     native: *mut P,
     storage: T,
     _marker: PhantomData<&'a P>,
}

#[allow(dead_code)]
impl<'a, P: Copy, T: 'a> BorrowedMutPointer<'a, P, T> {
     fn as_ptr(&self) -> *const P {
         self.native
     }

     fn as_mut_ptr(&mut self) -> *mut P {
         self.native
     }

     fn map<U: 'a, F: FnOnce(T) -> U>(self, f: F) -> 
BorrowedMutPointer<'a, P, U> {
         BorrowedMutPointer {
             native: self.native,
             storage: f(self.storage),
             _marker: PhantomData,
         }
     }
}

impl<'a, P, T> Borrow<T> for BorrowedMutPointer<'a, P, T> {
     fn borrow(&self) -> &T {
         &self.storage
     }
}

impl<'a, P, T> BorrowMut<T> for BorrowedMutPointer<'a, P, T> {
     fn borrow_mut(&mut self) -> &mut T {
         &mut self.storage
     }
}

2) Does ToQemuPtr need to allow multiple implementations?  Can the type 
parameter in ToQemuPtr<'a, P> be an associated type (for example 
"Native")?  Type parameters really add a lot of complexity.

3) I would rather not have "qemu" in the names.  The Rust parts *are* 
QEMU.  So "foreign" or "c" would be better.

4) full/none is still really confusing to me.  I have finally understood 
that it's because the pair that borrows is from_qemu_full/to_qemu_none, 
and the pair that copies is from_qemu_none/to_qemu_full.  I'd really 
like to use names following the Rust naming conventions.  A possible 
improvement of my proposal from the previous review:

- from_qemu_full -> from_foreign (or from_c, same below)
                     + possibly a dual method into_native or into_rust

- from_qemu_none -> cloned_from_foreign

- to_qemu_none -> as_foreign or as_foreign_mut

- to_qemu_full -> clone_to_foreign

I will see if I have some time to do some of this work.

Paolo



^ permalink raw reply	[flat|nested] 74+ messages in thread

* Re: [RFC v3 12/32] rust: provide a common crate for QEMU
  2021-09-13 17:11   ` Paolo Bonzini
@ 2021-09-14 11:34     ` Marc-André Lureau
  0 siblings, 0 replies; 74+ messages in thread
From: Marc-André Lureau @ 2021-09-14 11:34 UTC (permalink / raw)
  To: Paolo Bonzini
  Cc: Daniel P. Berrange, QEMU, Stefan Hajnoczi, Markus Armbruster

[-- Attachment #1: Type: text/plain, Size: 7936 bytes --]

Hi Paolo

On Mon, Sep 13, 2021 at 9:18 PM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 07/09/21 14:19, marcandre.lureau@redhat.com wrote:
> > From: Marc-André Lureau<marcandre.lureau@redhat.com>
> >
> > This crates provides common bindings and facilities for QEMU C API
> > shared by various projects.
> >
> > Most importantly, it defines the conversion traits used to convert from
> > C to Rust types. Those traits are largely adapted from glib-rs, since
> > those have proved to be very flexible, and should guide us to bind
> > further QEMU types such as QOM. If glib-rs becomes a dependency, we
> > should consider adopting glib translate traits. For QAPI, we need a
> > smaller subset.
> >
> > Cargo.lock is checked-in, as QEMU produces end-of-chain binaries, and it
> > is desirable to track the exact set of packages that are involved in
> > managed builds.
> >
> > Signed-off-by: Marc-André Lureau<marcandre.lureau@redhat.com>
>
> As in my previous review, the main issue I have here is with the
> complexity of this code.
>
> I understand that this has to be manually written, but right now I find
> it really hard to understand what is going on here.  The patch needs to
> be expanded in several parts:
>
> 1) generic traits (including implementations for Option/Box)
>
> 2) implementation of the generic traits
>
> 3) io/nix errors
>
> and these parts should be moved around to the place where they become
> necessary.
>
>
The common crate can be split in many pieces. It is easier to have it as a
single commit during PoC/RFC (the series is already large enough).

Is it really valuable to introduce a new crate/library piece by piece?
Instead, better documentation and tests are more valuable imho. But I will
certainly split it and clean it more as we iterate.


> Also regarding the code itself:
>
> 1) Stash should not be a tuple.  Accesses to it should use standard Rust
> methods, such as borrow()/borrow_mut(), and it should also support
> standard Rust idioms such as map():
>
>

I thought we already discussed that. The stash is an implementation detail
to handle FFI resource release. The only thing the user should care about
the stash is the pointer, which is shortly retrieved with ".0" (cannot be
more succinct).

Ideally, we would like to provide "foo.to_qemu_none()", but to keep the
associated resource allocated, the shortest we can offer is
"foo.to_qemu_none().0". Using "as_ptr()" instead would not only be longer
to type but possibly misleading: there is nothing else you should do with
the stash, no other method should exist.

I don't understand what "map()" would provide here either. The storage part
of the stash is an internal detail (the FFI data), not meant to be
manipulated at all (it would invalidate the stash pointer).



> pub struct BorrowedMutPointer<'a, P, T: 'a> {
>      native: *mut P,
>      storage: T,
>      _marker: PhantomData<&'a P>,
> }
>
> #[allow(dead_code)]
> impl<'a, P: Copy, T: 'a> BorrowedMutPointer<'a, P, T> {
>      fn as_ptr(&self) -> *const P {
>          self.native
>      }
>
>      fn as_mut_ptr(&mut self) -> *mut P {
>          self.native
>      }
>
>      fn map<U: 'a, F: FnOnce(T) -> U>(self, f: F) ->
> BorrowedMutPointer<'a, P, U> {
>          BorrowedMutPointer {
>              native: self.native,
>              storage: f(self.storage),
>              _marker: PhantomData,
>          }
>      }
> }
>
> impl<'a, P, T> Borrow<T> for BorrowedMutPointer<'a, P, T> {
>      fn borrow(&self) -> &T {
>          &self.storage
>      }
> }
>
> impl<'a, P, T> BorrowMut<T> for BorrowedMutPointer<'a, P, T> {
>      fn borrow_mut(&mut self) -> &mut T {
>          &mut self.storage
>      }
> }
>
> 2) Does ToQemuPtr need to allow multiple implementations?  Can the type
> parameter in ToQemuPtr<'a, P> be an associated type (for example
> "Native")?  Type parameters really add a lot of complexity.
>
>

ToQemuPtr is a trait, implemented for the various Rust types that we can
translate to FFI pointers.

For example:

impl<'a> ToQemuPtr<'a, *mut c_char> for String {
    type Storage = CString;
...
}

The pointer type parameter is to return the correct pointer. It is not
necessary to specify it as a user, since Rust infers it from the expected
type.


> 3) I would rather not have "qemu" in the names.  The Rust parts *are*
> QEMU.  So "foreign" or "c" would be better.
>


That's kind of cosmetic.

The main reason for "qemu" in the name is to avoid potential clashes with
other methods and traits.

Also "to_none" could associate in the reader's mind (wrongly) with the
ubiquitous Option::None.

But yes, we could drop "qemu" here.


>
> 4) full/none is still really confusing to me.  I have finally understood
> that it's because the pair that borrows is from_qemu_full/to_qemu_none,
> and the pair that copies is from_qemu_none/to_qemu_full.  I'd really
> like to use names following the Rust naming conventions.  A possible
> improvement of my proposal from the previous review:
>
>

"full" and "none" are inherited from the glib-introspection ownership
transfer annotations.

We discussed and weighed pros and cons last year. I haven't yet found or
agreed on better options. And you can be certain that the glib-rs folks
have already pondered a lot about the naming and design. And, I don't like
to reinvent what others have done (creating various limitations and
incompatibilities). Following the glib traits brings a lot of experience
and could help us reuse glib-rs crates more easily (because they would use
the same design), but also to port and interoperate. Imagine we need to
bind GHashTable<char *, QapiFoo> someday, we will be in a better position
if we have translation traits that follow glib-rs already.

The Rust conventions can be inferred from CString/CStr design and methods,
but they have to be adjusted, because they don't solve the same constraints
(no ownership transfer, and CString/CStr intermediary representations).

They are "CStr::from_ptr(*const)" and "CString::as_ptr(&self) -> *const".


- from_qemu_full -> from_foreign (or from_c, same below)
>                      + possibly a dual method into_native or into_rust
>
>
There is no standard naming for such FFI pointer ownership transfer. Rust
doesn't know how to release FFI resources, in general.

from_raw(*mut) is the closest, but "raw" is reserved for Rust data pointer
types, not FFI (except for slices, probably because the linear layout is
the same as FFI)

We can consider "from_mut_ptr(*mut)", but this doesn't convey the ownership
transfer as explicitly.


> - from_qemu_none -> cloned_from_foreign
>

You are being creative! This is like "CStr::from_ptr", except we go
directly to higher Rust types.


> - to_qemu_none -> as_foreign or as_foreign_mut
>
>
Similarity with "CString::as_ptr", except that it handles the temporary
stash (CString is the internal storage for String), so we use the
"to_qemu_none().0" described earler.

If we follow your considerations it could be "to_stash().as_ptr()", which
is less user friendly, exposes more internal details, etc.


> - to_qemu_full -> clone_to_foreign
>


There is no equivalent in Rust standard. An implementation may want to
store in an existing location, or allocated in different ways etc.

Closest would be to_ptr()

In summary:

from_qemu_full()  vs  from_mut_ptr()
from_qemu_none()      from_ptr()
to_qemu_full()        to_ptr()
to_qemu_none()        to_stash().as_ptr()

I argue there is more consistency and readability by following the glib-rs
traits and method names (among other advantages by staying close to glib-rs
design for the future).


>
> I will see if I have some time to do some of this work.
>
>
thanks


-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 11010 bytes --]

^ permalink raw reply	[flat|nested] 74+ messages in thread

end of thread, other threads:[~2021-09-14 11:35 UTC | newest]

Thread overview: 74+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-07 12:19 [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples marcandre.lureau
2021-09-07 12:19 ` [RFC v3 01/32] RFC: docs: add supported host CPUs section marcandre.lureau
2021-09-07 12:33   ` Peter Maydell
2021-09-13 11:32     ` Marc-André Lureau
2021-09-13 11:46       ` Peter Maydell
2021-09-07 12:19 ` [RFC v3 02/32] build-sys: add HAVE_IPPROTO_MPTCP marcandre.lureau
2021-09-08 12:01   ` Markus Armbruster
2021-09-13 13:02   ` Paolo Bonzini
2021-09-07 12:19 ` [RFC v3 03/32] scripts/qapi: teach c_param_type() to return const argument type marcandre.lureau
2021-09-08 12:10   ` Markus Armbruster
2021-09-08 14:33     ` Marc-André Lureau
2021-09-07 12:19 ` [RFC v3 04/32] glib-compat: add G_SIZEOF_MEMBER marcandre.lureau
2021-09-08 12:16   ` Markus Armbruster
2021-09-08 13:49     ` Marc-André Lureau
2021-09-07 12:19 ` [RFC v3 05/32] scripts/qapi: add QAPISchemaVisitor.visit_module_end marcandre.lureau
2021-09-08 12:26   ` Markus Armbruster
2021-09-07 12:19 ` [RFC v3 06/32] scripts/qapi: add a CABI module marcandre.lureau
2021-09-07 12:19 ` [RFC v3 07/32] scripts/qapi: generate CABI dump for C types marcandre.lureau
2021-09-07 12:19 ` [RFC v3 08/32] tests: build qapi-cabi (C ABI dump) marcandre.lureau
2021-09-07 12:19 ` [RFC v3 09/32] build-sys: add i686 cpu target marcandre.lureau
2021-09-08 13:45   ` Peter Maydell
2021-09-07 12:19 ` [RFC v3 10/32] build-sys: add --with-rust{-target} & basic build infrastructure marcandre.lureau
2021-09-08 14:00   ` Peter Maydell
2021-09-08 14:21     ` Marc-André Lureau
2021-09-07 12:19 ` [RFC v3 11/32] build-sys: add a cargo-wrapper script marcandre.lureau
2021-09-07 12:19 ` [RFC v3 12/32] rust: provide a common crate for QEMU marcandre.lureau
2021-09-10  1:18   ` Alistair Francis
2021-09-10  7:43     ` Marc-André Lureau
2021-09-13 17:11   ` Paolo Bonzini
2021-09-14 11:34     ` Marc-André Lureau
2021-09-07 12:19 ` [RFC v3 13/32] rust: use vendored-sources marcandre.lureau
2021-09-08 15:38   ` Ian Jackson
2021-09-08 15:47     ` Marc-André Lureau
2021-09-08 15:55       ` Ian Jackson
2021-09-08 16:15         ` Marc-André Lureau
2021-09-08 16:22           ` Peter Maydell
2021-09-08 16:22           ` Ian Jackson
2021-09-08 16:20     ` Marc-André Lureau
2021-09-08 16:29       ` Ian Jackson
2021-09-08 16:34         ` Marc-André Lureau
2021-09-08 16:50           ` Ian Jackson
2021-09-08 19:33             ` Marc-André Lureau
2021-09-09 16:02   ` Peter Maydell
2021-09-09 16:29     ` Marc-André Lureau
2021-09-09 16:53       ` Daniel P. Berrangé
2021-09-09 17:04         ` Ian Jackson
2021-09-13 14:21       ` Paolo Bonzini
2021-09-09 16:49     ` Daniel P. Berrangé
2021-09-09 17:02       ` Ian Jackson
2021-09-07 12:19 ` [RFC v3 14/32] scripts/qapi: add QAPISchemaIfCond.rsgen() marcandre.lureau
2021-09-08 12:33   ` Markus Armbruster
2021-09-08 14:06     ` Marc-André Lureau
2021-09-07 12:19 ` [RFC v3 15/32] scripts/qapi: strip trailing whitespaces marcandre.lureau
2021-09-07 12:19 ` [RFC v3 16/32] scripts/qapi: add Rust FFI bindings generation marcandre.lureau
2021-09-07 12:19 ` [RFC v3 17/32] scripts/qapi: learn to generate ABI dump for Rust FFI marcandre.lureau
2021-09-07 12:19 ` [RFC v3 18/32] tests: generate Rust bindings marcandre.lureau
2021-09-07 12:19 ` [RFC v3 19/32] tests: check Rust and C CABI diffs marcandre.lureau
2021-09-07 12:19 ` [RFC v3 20/32] scripts/qapi: generate high-level Rust bindings marcandre.lureau
2021-09-07 12:19 ` [RFC v3 21/32] tests/rust: build a common library, checking bindings compile marcandre.lureau
2021-09-07 12:19 ` [RFC v3 22/32] qga: build qapi-cabi binary (ABI from C) marcandre.lureau
2021-09-07 12:19 ` [RFC v3 23/32] qga/rust: build and link an empty static library marcandre.lureau
2021-09-07 12:19 ` [RFC v3 24/32] qga/rust: generate QGA QAPI types FFI bindings marcandre.lureau
2021-09-07 12:19 ` [RFC v3 25/32] qga/rust: build a qga-cabi-rs executable (ABI from Rust) marcandre.lureau
2021-09-07 12:19 ` [RFC v3 26/32] qga/rust: check the Rust C binding marcandre.lureau
2021-09-07 12:19 ` [RFC v3 27/32] qga/rust: build high-level Rust QAPI types marcandre.lureau
2021-09-07 12:19 ` [RFC v3 28/32] qga/rust: implement get-host-name in Rust (example) marcandre.lureau
2021-09-07 12:19 ` [RFC v3 29/32] qga/rust: implement {get,set}-vcpus " marcandre.lureau
2021-09-07 12:19 ` [RFC v3 30/32] tests/vm: add Rust to FreeBSD VM marcandre.lureau
2021-09-07 12:19 ` [RFC v3 31/32] tests/vm: bump fedora VM to f32 marcandre.lureau
2021-09-07 12:19 ` [RFC v3 32/32] tests/vm: add Rust to Fedora marcandre.lureau
2021-09-08 13:22 ` [RFC v3 00/32] Rust binding for QAPI and qemu-ga QMP handler examples Markus Armbruster
2021-09-08 13:55   ` Marc-André Lureau
2021-09-09 10:31     ` Markus Armbruster
2021-09-09 15:22       ` Marc-André Lureau

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).