From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAFD8C4727C for ; Tue, 29 Sep 2020 11:36:10 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3EE6D22207 for ; Tue, 29 Sep 2020 11:36:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZodrbDS/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3EE6D22207 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:52248 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kNDvV-0000QJ-DP for qemu-devel@archiver.kernel.org; Tue, 29 Sep 2020 07:36:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35860) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kNDuD-0007jY-Am for qemu-devel@nongnu.org; Tue, 29 Sep 2020 07:34:49 -0400 Received: from mail-ej1-x643.google.com ([2a00:1450:4864:20::643]:33042) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kNDuA-0007De-RZ for qemu-devel@nongnu.org; Tue, 29 Sep 2020 07:34:49 -0400 Received: by mail-ej1-x643.google.com with SMTP id j11so14552157ejk.0 for ; Tue, 29 Sep 2020 04:34:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=DEUy71vj+FhIBOXVwtpgyizlZ5mTKxHugER6o5FDIL4=; b=ZodrbDS/2RR9xp2nEiMKd+KJy8RR10whfTtwC+dPhz9gSCHr1OvEDYqCkD8s3n3oZN TGDvJ6XVz5enwboxPELFFF4v4rllCznC2Vd+sA1ZPm/3aPZ5F9KojJQJuqHFxr2FzYQ2 1NzubVe92/XQ2MlUWz2OVr8XViucMt6pCtIuIWehpZ92QQ8ZgCkG/TwWyWKSTZtQXv6Y IifvGAuWxe4cg8Yl8xKTjA6EL8dxs5AHaxcQB2Or2MiDVvC++7Ftsc3yOxPcVQ+MbVY4 +jTcm/YRGs302dmCoOUbtqqaE+8NLDER9bveCIZHXEiZFDeRgTXDzBHVegq0xv/i1sfO 6r3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=DEUy71vj+FhIBOXVwtpgyizlZ5mTKxHugER6o5FDIL4=; b=Ygp6JM1TuDl6c8yRWoRN19Hj9Z1JeYoB7I+BT633lbfc5vPeXru7Z95Q+2Bqk7g+/A Z/qxumaUhuYjGPoHb2gSw2t5eRbwsa7uGaAH5OLUyqWfspuIsdPUzaGW2XBybfK0QA6S 0eUr1Ne9yvNW2Yh2KxbTyubxtCp2m0nnnTZkhcqdtPpRtIqqTf6CvbI+xeacgX89piui RGMy5mzJnEmx+cjGAKyobgQmnIPONlSia1z23gA7URO+deMxrDFInL+LSIcP8HFxCFDT qWspvzThW/WZGoaFED5h92pWsqzBGBjd7sPFaWKmgRCfmkT7RAFJ2RjjD0PopAU+yQss gZ7w== X-Gm-Message-State: AOAM533g1bhXvJuEc46whpAjvvZTh8XI0WWtCICj1uLeLL7DcF+ZajLA NgN7KFZq3JMMhHhjxFXMunhjK71sO+0KQ714GII= X-Google-Smtp-Source: ABdhPJwMfpOJAdk/+Px2bCWhT7FsVbWHTh2R9bwGnAsHkFEP6q4eWrkLBdm7ZMzkILnDSI4TUpaB5/kJe2X19BZXBEE= X-Received: by 2002:a17:906:4a51:: with SMTP id a17mr3282464ejv.381.1601379284948; Tue, 29 Sep 2020 04:34:44 -0700 (PDT) MIME-Version: 1.0 References: <20200910174850.716104-1-marcandre.lureau@redhat.com> <87mu1j8p7p.fsf@dusky.pond.sub.org> <874knpluez.fsf@dusky.pond.sub.org> <9d6215d4-d0cd-67e4-3048-77127194f7e8@redhat.com> <8c1783d1-70f4-d751-3d5d-83459cb1db45@redhat.com> In-Reply-To: From: =?UTF-8?B?TWFyYy1BbmRyw6kgTHVyZWF1?= Date: Tue, 29 Sep 2020 15:34:32 +0400 Message-ID: Subject: Re: [PATCH] PoC: Rust binding for QAPI (qemu-ga only, for now) To: Paolo Bonzini Content-Type: multipart/alternative; boundary="0000000000006e1bba05b0722b36" Received-SPF: pass client-ip=2a00:1450:4864:20::643; envelope-from=marcandre.lureau@gmail.com; helo=mail-ej1-x643.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "P. Berrange, Daniel" , Sergio Lopez Pascual , "Hajnoczi, Stefan" , Markus Armbruster , qemu-devel , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --0000000000006e1bba05b0722b36 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi On Tue, Sep 29, 2020 at 3:01 PM Paolo Bonzini wrote: > On 29/09/20 12:34, Marc-Andr=C3=A9 Lureau wrote: > > That would not be backwards compatible as you would have to set all > > optional fields. Every time the command grows a new optional > argument, > > all clients would have to specify it; if a command becomes optional= , > > you'd have to add Some() around it. So I would say that... > > > > > > Not necessarily, with ..default::Default() > > That's true, I always forget about .. (though you'd still have to add > Some() for now-optional fields). > > > > Less idiomatic, but it also works around the optional arguments a= nd > > > ordering issue. > > > > ... the builder pattern is not a workaround: it's the best and mos= t > > common Rust idiom to achieve what QAPI expresses as optional fields= . > > Likewise for keyword-only arguments in Python. > > > > Except QAPI makes all fields potentially optional (and unordered), > > that's not idiomatic. > > Yes, for some APIs you can always add hand-written, more idiomatic > versions. Or you could mark them as fixed-arguments in the schema and > let the code generator do that (but then you need to add a compatibility > check). But that would be an explicit choice, not something required by > the transport. > That's my point, if we agree on keeping arguments & members non-optional and ordered, we are doing 50% of the job for nicer automated bindings. Second half would probably be involving some version information in the schema. > > D-Bus is machine-level oriented, it's easy to bind to various languages= , > > it can be pretty efficient too. It's not designed to be a good network > > RPC. QMP tries to be a bit of both, but is perhaps not good enough in > > either. > > No, only tries to be a good network RPC; not a particularly efficient > one, but future-proof. And it mostly succeeds at that---with one > notable exception: JSON parsers that mess up with numbers bigger than 2^5= 3. > > > If you want to "reinvent" QMP, instead of focusing on D-Bus you > should > > take a look at alternative IDLs and protocols (D-Bus is one but > there's > > also Protobuf and Flexbuffers), see how QAPI declarations would map > to > > those protocols, see how you would deal with extensibility, and ran= k > > them according to various criteria. For example: > > > > * JSON "just works" but needs a custom code generator and imposes > some > > extra complexity on the clients for the simplest commands > > > > * D-Bus has a good ecosystem and would keep simple commands simpler > but > > has issues with upgrade paths and is uglier for complex commands > > > > * Protobufs probably would also just work and would have better cod= e > > generators, but would require some kind of lint to ensure > > backwards-compatibility > > > > Again, the issues we are discussing are not specific to binding QMP ove= r > > D-Bus. Binding QMP to various languages has similar problems. > > Marc-Andr=C3=A9, we are totally in agreement about that! The problem is = that > you have already decided what the solution looks like, and that's what > I'm not sure about because your solution also implies completely > revisiting the schema. > Did I? Which schema change are you (or I) implying? Versioning the interface? It's necessary at the client level, unless everything is dynamic, after introspection, which makes automated static bindings impractical. > I say there are many candidates (the ones I know are Protobuf and > Flexbuffers) for serialization and many candidates for transport (REST > and gRPC to begin with) in addition to the two {QMP,JSON} and > {DBus,DBus} tuples. We should at least look at how they do code > generation before deciding that JSON is bad and DBus is good. > Contrary to what you believe I am not focusing so much on DBus here :) It took about 200 loc to bind it, effortlessly (compared to sys<->rs conversion). All it does is to expose the same API we have in the generated C somehow (similar static types & functions - not all as a{sv} opaque dictionaries). It's easy for QEMU to generate a good static binding for C, because the version always matches. For a client, you wouldn't be able to write a similar idiomatic API in C, Rust, Python or Go, unfortunately. Iow, I am not trying to sell DBus, I would like to make it easier to bind QMP in general. (although I do believe that DBus is a better protocol than QMP for local IPC, yes. And gRPC is probably better for remoting) > I would rather make those problems solved at the server level, that > > doesn't require any change to QMP today, just a more careful > > consideration when making changes (and probably some tools to help > > enforce some stability). > > Problem is, "more careful consideration when making changes" is not a > small thing. And other RPCs have evolved in a completely different > space (REST APIs for web services) that have chosen the same tradeoffs > as QMP, so why should we not learn from them? > > I don't buy that generalization. A very recent protocol in this space, that aims to be a good low-level RPC on Linux (for containers, cloud etc) is varlink. (In many ways, we could compare it to QMP, but it lacks some important features, like events) varlink does non-optional members and versioning the same way I propose here, for what I could tell. Although they use JSON, and have similar transport "benefits", this basic rule allow them to have very decent automated binding in various languages, without resorting to unorthodox solutions, ex: https://github.com/varlink/rust/blob/master/examples/example/src/main.rs --=20 Marc-Andr=C3=A9 Lureau --0000000000006e1bba05b0722b36 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi

On Tue, Sep 29, 2020 at 3:01 PM Pao= lo Bonzini <pbonzini@redhat.com> wrote:
On= 29/09/20 12:34, Marc-Andr=C3=A9 Lureau wrote:
>=C2=A0 =C2=A0 =C2=A0That would not be backwards compatible as you would= have to set all
>=C2=A0 =C2=A0 =C2=A0optional fields.=C2=A0 Every time the command grows= a new optional argument,
>=C2=A0 =C2=A0 =C2=A0all clients would have to specify it; if a command = becomes optional,
>=C2=A0 =C2=A0 =C2=A0you'd have to add Some() around it.=C2=A0 So I = would say that...
>
>
> Not necessarily, with ..default::Default()

That's true, I always forget about .. (though you'd still have to a= dd
Some() for now-optional fields).

>=C2=A0 =C2=A0 =C2=A0> Less idiomatic, but it also works around the o= ptional arguments and
>=C2=A0 =C2=A0 =C2=A0> ordering issue.
>
>=C2=A0 =C2=A0 =C2=A0...=C2=A0 the builder pattern is not a workaround: = it's the best and most
>=C2=A0 =C2=A0 =C2=A0common Rust idiom to achieve what QAPI expresses as= optional fields.
>=C2=A0 =C2=A0 =C2=A0Likewise for keyword-only arguments in Python.
>
> Except QAPI makes all fields potentially optional (and unordered),
> that's not idiomatic.

Yes, for some APIs you can always add hand-written, more idiomatic
versions.=C2=A0 Or you could mark them as fixed-arguments in the schema and=
let the code generator do that (but then you need to add a compatibility check).=C2=A0 But that would be an explicit choice, not something required = by
the transport.


Second half would pro= bably be involving some version information in the schema.


> D-Bus is machine-level oriented, it's easy to bind to various lang= uages,
> it can be pretty efficient too. It's not designed to be a good net= work
> RPC. QMP tries to be a bit of both, but is perhaps not good enough in<= br> > either.

No, only tries to be a good network RPC; not a particularly efficient
one, but future-proof.=C2=A0 And it mostly succeeds at that---with one
notable exception: JSON parsers that mess up with numbers bigger than 2^53.=

>=C2=A0 =C2=A0 =C2=A0If you want to "reinvent" QMP, instead of= focusing on D-Bus you should
>=C2=A0 =C2=A0 =C2=A0take a look at alternative IDLs and protocols (D-Bu= s is one but there's
>=C2=A0 =C2=A0 =C2=A0also Protobuf and Flexbuffers), see how QAPI declar= ations would map to
>=C2=A0 =C2=A0 =C2=A0those protocols, see how you would deal with extens= ibility, and rank
>=C2=A0 =C2=A0 =C2=A0them according to various criteria.=C2=A0 For examp= le:
>
>=C2=A0 =C2=A0 =C2=A0* JSON "just works" but needs a custom co= de generator and imposes some
>=C2=A0 =C2=A0 =C2=A0extra complexity on the clients for the simplest co= mmands
>
>=C2=A0 =C2=A0 =C2=A0* D-Bus has a good ecosystem and would keep simple = commands simpler but
>=C2=A0 =C2=A0 =C2=A0has issues with upgrade paths and is uglier for com= plex commands
>
>=C2=A0 =C2=A0 =C2=A0* Protobufs probably would also just work and would= have better code
>=C2=A0 =C2=A0 =C2=A0generators, but would require some kind of lint to = ensure
>=C2=A0 =C2=A0 =C2=A0backwards-compatibility
>
> Again, the issues we are discussing are not specific to binding QMP ov= er
> D-Bus. Binding QMP to various languages has similar problems.

Marc-Andr=C3=A9, we are totally in agreement about that!=C2=A0 The problem = is that
you have already decided what the solution looks like, and that's what<= br> I'm not sure about because your solution also implies completely
revisiting the schema.



I say there are many candidates (the ones I know are Protobuf and
Flexbuffers) for serialization and many candidates for transport (REST
and gRPC to begin with) in addition to the two {QMP,JSON} and
{DBus,DBus} tuples.=C2=A0 We should at least look at how they do code
generation before deciding that JSON is bad and DBus is good.
=C2=A0
Contrary to what you believe I am not focusing so = much on DBus here :) It took about 200 loc to bind it, effortlessly (compar= ed to sys<->rs conversion). All it does is to expose the same API we = have in the generated C somehow (similar static types & functions - not= all as a{sv} opaque dictionaries).

It's = easy for QEMU to generate a good static binding for C, because the version = always matches. For a client, you wouldn't be able to write a similar i= diomatic API in C, Rust, Python or Go, unfortunately.

Iow, I am not trying to sell DBus, I would like to make it easier = to bind QMP in general. (although I do believe that DBus is a better protoc= ol than QMP for local IPC, yes. And gRPC is probably better for remoting)

> I would rather make those problems solved at the server level, that > doesn't require any change to QMP today, just a more careful
> consideration when making changes (and probably some tools to help
> enforce some stability).

Problem is, "more careful consideration when making changes" is n= ot a
small thing.=C2=A0 And other RPCs have evolved in a completely different space (REST APIs for web services) that have chosen the same tradeoffs
as QMP, so why should we not learn from them?


I don't buy that generalization. A very= recent protocol in this space, that aims to be a good low-level RPC on Lin= ux (for containers, cloud etc) is varlink. (In many ways, we could compare = it to QMP, but it lacks some important features, like events)

varlink does non-op= tional members and versioning the same way I propose here, for what I could= tell. Although they use JSON, and have similar transport "benefits&qu= ot;, this basic rule allow them to have very decent automated binding in va= rious languages, without resorting to unorthodox solutions, ex: h= ttps://github.com/varlink/rust/blob/master/examples/example/src/main.rs=

--
Marc-Andr=C3=A9= Lureau
--0000000000006e1bba05b0722b36--