QAPI sync meeting

QAPI sync meeting

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

Hiya,

I'd like to propose that at least the three of us arrange a time to have a
meeting where we discuss our plans and ideas for QAPI going forward,
including rust, python, and golang extensions to the QAPI generator, what
we hope to accomplish with those projects, and so on.

What I am hoping to get out of this for myself is a high-level overview of
people's plans for QAPI and to produce some notes on those plans so that I
can have a reference that we've all acknowledged as roughly accurate to be
able to keep the community's design goals for QAPI in mind as I continue my
own development. Ultimately, I'd like some kind of rough draft of a "QAPI
roadmap".

I know there was a rust meetup during KVM Forum, but I was unable to attend
due to the timing. I'd like to expand the focus a little more broadly to
QAPI in general and discuss our "personal" roadmaps, goals, queued work,
etc so that we can collaboratively formulate a broader vision of our work.

I'm posting to qemu-devel in case anyone else has an interest in this area
and would like to eavesdrop or share opinions, but we should probably come
up with an agenda first. So:

Proposed agenda:

Current projects, wishlists, and goals for QAPI:
- Markus (~10 min)
- Marc-Andre (~10 min) (Rust, dbus, etc?)
- jsnow (~10 min) (Python, golang, etc)

Formulating short-term and long-term roadmaps:
- Open discussion, ~30 min
- Collaboratively produce a summary doc (etherpad?) outlining major work to
be done, separated into near and long terms
- Upload this summary to the QEMU wiki and mail it back out to qemu-devel
- We probably won't exactly finish this bit, but we can resume on the
mailing list afterwards perfectly well.

(Feel free to propose anything different for the meeting, this is just a
jumping off point for discussion.)

Proposed time:

- Any weekday after 13:00 UTC. Wednesdays, Thursdays and Fridays work
particularly well for me at the moment.
- bluejeans and google meeting both work well for me. Open to alternatives.


Thanks,
--js

[-- Attachment #2: Type: text/html, Size: 2473 bytes --]

Re: QAPI sync meeting

[Marc-André Lureau]

[-- Attachment #1: Type: text/plain, Size: 3985 bytes --]

Hi!

On Mon, Sep 27, 2021 at 8:55 PM John Snow <jsnow@redhat.com> wrote:

> Hiya,
>
> I'd like to propose that at least the three of us arrange a time to have a
> meeting where we discuss our plans and ideas for QAPI going forward,
> including rust, python, and golang extensions to the QAPI generator, what
> we hope to accomplish with those projects, and so on.
>
> What I am hoping to get out of this for myself is a high-level overview of
> people's plans for QAPI and to produce some notes on those plans so that I
> can have a reference that we've all acknowledged as roughly accurate to be
> able to keep the community's design goals for QAPI in mind as I continue my
> own development. Ultimately, I'd like some kind of rough draft of a "QAPI
> roadmap".
>
> I know there was a rust meetup during KVM Forum, but I was unable to
> attend due to the timing. I'd like to expand the focus a little more
> broadly to QAPI in general and discuss our "personal" roadmaps, goals,
> queued work, etc so that we can collaboratively formulate a broader vision
> of our work.
>
> I'm posting to qemu-devel in case anyone else has an interest in this area
> and would like to eavesdrop or share opinions, but we should probably come
> up with an agenda first. So:
>
> Proposed agenda:
>
> Current projects, wishlists, and goals for QAPI:
> - Markus (~10 min)
> - Marc-Andre (~10 min) (Rust, dbus, etc?)
>

The QAPI Rust binding RFC series aims to provide the QAPI types to Rust
with to/from C translations. This is just one brick allowing QEMU to have
some parts written in Rust: all other internal/subsystem binding pieces
remain to be done. I don't have other plans for QAPI at this point.

D-Bus (from the early qapi/rust series) was an experiment, showing that
QAPI/QMP can be exposed via "serde" with almost no effort. (it could most
likely be over other protocols, such as JSON, in full-Rust implementation).
We can imagine sharing canonical QAPI IDLs for daemons/helpers written in
different languages.

However, the biggest hurdle when binding QAPI to D-Bus or many programming
languages (c, python, go and rust foremost) is that it is not a
machine/ABI-friendly IDL. QAPI doesn't impose orderdering of fields or
arguments, and it is not versioned. I believe this is detrimental, because
bindings have to be written and maintained by hand in various languages,
with various flavours (some may add abstractions, some may version the
schema themself, some may use plain dictionaries everywhere etc). The small
flexibility advantage doesn't outweigh the cost. Imho, some of the pains of
QAPI/QMP is that it's not so easy to interact with, either from a human
point of view (who likes speaking JSON?) or a machine point of view (I
don't have good bindings to use from my language of choice). If we provided
(and generated) idiomatic client bindings, we would most likely have a few
rules to not break them, and end up versionizing the schema (version the
commands, version arguments etc, various options are possible). The wire
format becomes a detail, and JSON can keep its own flexibility wrt fields
ordering etc.

- jsnow (~10 min) (Python, golang, etc)
>
>
I am certainly interested to learn your updated plans.


> Formulating short-term and long-term roadmaps:
> - Open discussion, ~30 min
> - Collaboratively produce a summary doc (etherpad?) outlining major work
> to be done, separated into near and long terms
> - Upload this summary to the QEMU wiki and mail it back out to qemu-devel
> - We probably won't exactly finish this bit, but we can resume on the
> mailing list afterwards perfectly well.
>
> (Feel free to propose anything different for the meeting, this is just a
> jumping off point for discussion.)
>
> Proposed time:
>
> - Any weekday after 13:00 UTC. Wednesdays, Thursdays and Fridays work
> particularly well for me at the moment.
>

That could work for me.

- bluejeans and google meeting both work well for me. Open to alternatives.
>
>
> Thanks,
> --js
>

[-- Attachment #2: Type: text/html, Size: 5305 bytes --]

Re: QAPI sync meeting

[Kevin Wolf]

Am 27.09.2021 um 18:55 hat John Snow geschrieben:
> Hiya,
> 
> I'd like to propose that at least the three of us arrange a time to have a
> meeting where we discuss our plans and ideas for QAPI going forward,
> including rust, python, and golang extensions to the QAPI generator, what
> we hope to accomplish with those projects, and so on.
> 
> What I am hoping to get out of this for myself is a high-level overview of
> people's plans for QAPI and to produce some notes on those plans so that I
> can have a reference that we've all acknowledged as roughly accurate to be
> able to keep the community's design goals for QAPI in mind as I continue my
> own development. Ultimately, I'd like some kind of rough draft of a "QAPI
> roadmap".
> 
> I know there was a rust meetup during KVM Forum, but I was unable to attend
> due to the timing. I'd like to expand the focus a little more broadly to
> QAPI in general and discuss our "personal" roadmaps, goals, queued work,
> etc so that we can collaboratively formulate a broader vision of our work.
> 
> I'm posting to qemu-devel in case anyone else has an interest in this area
> and would like to eavesdrop or share opinions

Yes, I would definitely be interested to join such a meeting.

My own focus is on CLI QAPIfication (and I guess eventually a QAPI-based
config file?) and integration with QOM, which is a bit different from
the other topics you mentioned, but it still touches the QAPI schema and
generator.

Kevin

Re: QAPI sync meeting

[Daniel P. Berrangé]

On Mon, Sep 27, 2021 at 12:55:34PM -0400, John Snow wrote:
> Hiya,
> 
> I'd like to propose that at least the three of us arrange a time to have a
> meeting where we discuss our plans and ideas for QAPI going forward,
> including rust, python, and golang extensions to the QAPI generator, what
> we hope to accomplish with those projects, and so on.
> 
> What I am hoping to get out of this for myself is a high-level overview of
> people's plans for QAPI and to produce some notes on those plans so that I
> can have a reference that we've all acknowledged as roughly accurate to be
> able to keep the community's design goals for QAPI in mind as I continue my
> own development. Ultimately, I'd like some kind of rough draft of a "QAPI
> roadmap".
> 
> I know there was a rust meetup during KVM Forum, but I was unable to attend
> due to the timing. I'd like to expand the focus a little more broadly to
> QAPI in general and discuss our "personal" roadmaps, goals, queued work,
> etc so that we can collaboratively formulate a broader vision of our work.
> 
> I'm posting to qemu-devel in case anyone else has an interest in this area
> and would like to eavesdrop or share opinions, but we should probably come
> up with an agenda first. So:

I'd be interested in discussions

> 
> Proposed agenda:
> 
> Current projects, wishlists, and goals for QAPI:
> - Markus (~10 min)
> - Marc-Andre (~10 min) (Rust, dbus, etc?)
> - jsnow (~10 min) (Python, golang, etc)
> 
> Formulating short-term and long-term roadmaps:
> - Open discussion, ~30 min
> - Collaboratively produce a summary doc (etherpad?) outlining major work to
> be done, separated into near and long terms
> - Upload this summary to the QEMU wiki and mail it back out to qemu-devel
> - We probably won't exactly finish this bit, but we can resume on the
> mailing list afterwards perfectly well.
> 
> (Feel free to propose anything different for the meeting, this is just a
> jumping off point for discussion.)

My interest is in a roadmap for getting to a point where we have
ability to configure QEMU  VMs in a "legacy free" way. By this I
mean

   - 100% of configuration is driven by QAPI defined schemas
   - HMP and QemuOpts are banished/untangled from internals of
     QEMU, such that they are just a shim around the canonical
     QAPI and could even be done as a separate wrapper process
   - May or may not involve qemu-system-XXX vs a new "qemu-vm" binary
   - May or may not involve QMP vs a REST API taking QAPI JSON vs
     something else.

> - Any weekday after 13:00 UTC. Wednesdays, Thursdays and Fridays work
> particularly well for me at the moment.
> - bluejeans and google meeting both work well for me. Open to alternatives.

Are you suggesting a 1-off meeting or a regular meeting, or an adhoc
set of meetings ?


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: QAPI sync meeting

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 3882 bytes --]

On Tue, Sep 28, 2021 at 9:53 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Mon, Sep 27, 2021 at 12:55:34PM -0400, John Snow wrote:
> > Hiya,
> >
> > I'd like to propose that at least the three of us arrange a time to have
> a
> > meeting where we discuss our plans and ideas for QAPI going forward,
> > including rust, python, and golang extensions to the QAPI generator, what
> > we hope to accomplish with those projects, and so on.
> >
> > What I am hoping to get out of this for myself is a high-level overview
> of
> > people's plans for QAPI and to produce some notes on those plans so that
> I
> > can have a reference that we've all acknowledged as roughly accurate to
> be
> > able to keep the community's design goals for QAPI in mind as I continue
> my
> > own development. Ultimately, I'd like some kind of rough draft of a "QAPI
> > roadmap".
> >
> > I know there was a rust meetup during KVM Forum, but I was unable to
> attend
> > due to the timing. I'd like to expand the focus a little more broadly to
> > QAPI in general and discuss our "personal" roadmaps, goals, queued work,
> > etc so that we can collaboratively formulate a broader vision of our
> work.
> >
> > I'm posting to qemu-devel in case anyone else has an interest in this
> area
> > and would like to eavesdrop or share opinions, but we should probably
> come
> > up with an agenda first. So:
>
> I'd be interested in discussions
>
> >
> > Proposed agenda:
> >
> > Current projects, wishlists, and goals for QAPI:
> > - Markus (~10 min)
> > - Marc-Andre (~10 min) (Rust, dbus, etc?)
> > - jsnow (~10 min) (Python, golang, etc)
> >
> > Formulating short-term and long-term roadmaps:
> > - Open discussion, ~30 min
> > - Collaboratively produce a summary doc (etherpad?) outlining major work
> to
> > be done, separated into near and long terms
> > - Upload this summary to the QEMU wiki and mail it back out to qemu-devel
> > - We probably won't exactly finish this bit, but we can resume on the
> > mailing list afterwards perfectly well.
> >
> > (Feel free to propose anything different for the meeting, this is just a
> > jumping off point for discussion.)
>
> My interest is in a roadmap for getting to a point where we have
> ability to configure QEMU  VMs in a "legacy free" way. By this I
> mean
>
>    - 100% of configuration is driven by QAPI defined schemas
>    - HMP and QemuOpts are banished/untangled from internals of
>      QEMU, such that they are just a shim around the canonical
>      QAPI and could even be done as a separate wrapper process
>    - May or may not involve qemu-system-XXX vs a new "qemu-vm" binary
>    - May or may not involve QMP vs a REST API taking QAPI JSON vs
>      something else.
>
>
More or less in agreement. Devil's in the details as always!


> > - Any weekday after 13:00 UTC. Wednesdays, Thursdays and Fridays work
> > particularly well for me at the moment.
> > - bluejeans and google meeting both work well for me. Open to
> alternatives.
>
> Are you suggesting a 1-off meeting or a regular meeting, or an adhoc
> set of meetings ?
>
>
I'm proposing a one-off here just for some initial discussion and planning.
(We didn't have a QAPI BoF at KVM Forum and I'd like something similar
now.) I don't expect we'll really "finish" plotting out a roadmap in a
single 60 minute meeting, but it will probably give us good, targeted
discussions on the ML to follow. If more meetings are desired to discuss
specific topics, I'm not against scheduling them as-needed.

(I personally don't really mind semi-regular sync-up meetings, like once
every other month or something to that effect -- it keeps me motivated and
on track. I suspect that Markus does not quite feel the same way about 'em,
so I'm just trying to not sign him up for stuff.)

--js

[-- Attachment #2: Type: text/html, Size: 4808 bytes --]

Re: QAPI sync meeting

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Sep 28, 2021 at 9:53 AM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
>
>> On Mon, Sep 27, 2021 at 12:55:34PM -0400, John Snow wrote:

[...]

>> > - Any weekday after 13:00 UTC. Wednesdays, Thursdays and Fridays work
>> > particularly well for me at the moment.
>> > - bluejeans and google meeting both work well for me. Open to alternatives.

Congratulations, you've just crowned yourself meeting organizer!  Pick a
date, time, and place, then herd the cats there.

I can't do next Friday.  I think we should aim for next week, to give us
time to coordinate and to prepare.

>> Are you suggesting a 1-off meeting or a regular meeting, or an adhoc
>> set of meetings ?
>>
>>
> I'm proposing a one-off here just for some initial discussion and planning.
> (We didn't have a QAPI BoF at KVM Forum and I'd like something similar
> now.) I don't expect we'll really "finish" plotting out a roadmap in a
> single 60 minute meeting, but it will probably give us good, targeted
> discussions on the ML to follow. If more meetings are desired to discuss
> specific topics, I'm not against scheduling them as-needed.

Let's have one now, and then more as needed.

Jumping into the meeting unprepared, then spending half of the alloted
time listening to people explaining where they want to take QAPI seems
like a bad use of our time.  Better: write it up and post it, so we can
all read before we meet.  Do not reply, except perhaps to ask for
clarification.  Discuss in the meeting.

I can see such posts from Marc-Andre, Kevin, and Daniel.  Some of them
could use a bit more detail, perhaps.

Thoughts?

> (I personally don't really mind semi-regular sync-up meetings, like once
> every other month or something to that effect -- it keeps me motivated and
> on track. I suspect that Markus does not quite feel the same way about 'em,
> so I'm just trying to not sign him up for stuff.)

More regular meetings sit fairly low on my personal wish list.

Re: QAPI sync meeting

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2626 bytes --]

On Wed, Sep 29, 2021 at 8:18 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > On Tue, Sep 28, 2021 at 9:53 AM Daniel P. Berrangé <berrange@redhat.com>
> > wrote:
> >
> >> On Mon, Sep 27, 2021 at 12:55:34PM -0400, John Snow wrote:
>
> [...]
>
> >> > - Any weekday after 13:00 UTC. Wednesdays, Thursdays and Fridays work
> >> > particularly well for me at the moment.
> >> > - bluejeans and google meeting both work well for me. Open to
> alternatives.
>
> Congratulations, you've just crowned yourself meeting organizer!  Pick a
> date, time, and place, then herd the cats there.
>
>
ACK. Will send out a proper invite.


> I can't do next Friday.  I think we should aim for next week, to give us
> time to coordinate and to prepare.
>
>
ACK


> >> Are you suggesting a 1-off meeting or a regular meeting, or an adhoc
> >> set of meetings ?
> >>
> >>
> > I'm proposing a one-off here just for some initial discussion and
> planning.
> > (We didn't have a QAPI BoF at KVM Forum and I'd like something similar
> > now.) I don't expect we'll really "finish" plotting out a roadmap in a
> > single 60 minute meeting, but it will probably give us good, targeted
> > discussions on the ML to follow. If more meetings are desired to discuss
> > specific topics, I'm not against scheduling them as-needed.
>
> Let's have one now, and then more as needed.
>
> Jumping into the meeting unprepared, then spending half of the alloted
> time listening to people explaining where they want to take QAPI seems
> like a bad use of our time.  Better: write it up and post it, so we can
> all read before we meet.  Do not reply, except perhaps to ask for
> clarification.  Discuss in the meeting.
>
> I can see such posts from Marc-Andre, Kevin, and Daniel.  Some of them
> could use a bit more detail, perhaps.
>
> Thoughts?
>
>
Works for me. I can send out a short summary beforehand, too, but I will
want to aggregate them in bullet-list digestible form for the meeting
minutes. I am suggesting a call to begin with so that I can set aside a
time specifically to discuss items with all interested parties ... at a
time when I'm awake. O:-)


> > (I personally don't really mind semi-regular sync-up meetings, like once
> > every other month or something to that effect -- it keeps me motivated
> and
> > on track. I suspect that Markus does not quite feel the same way about
> 'em,
> > so I'm just trying to not sign him up for stuff.)
>
> More regular meetings sit fairly low on my personal wish list.
>
>
O:-) O:-) O:-)

[-- Attachment #2: Type: text/html, Size: 3811 bytes --]

Re: QAPI sync meeting

[Kevin Wolf]

Am 29.09.2021 um 14:18 hat Markus Armbruster geschrieben:
> > I'm proposing a one-off here just for some initial discussion and planning.
> > (We didn't have a QAPI BoF at KVM Forum and I'd like something similar
> > now.) I don't expect we'll really "finish" plotting out a roadmap in a
> > single 60 minute meeting, but it will probably give us good, targeted
> > discussions on the ML to follow. If more meetings are desired to discuss
> > specific topics, I'm not against scheduling them as-needed.
> 
> Let's have one now, and then more as needed.
> 
> Jumping into the meeting unprepared, then spending half of the alloted
> time listening to people explaining where they want to take QAPI seems
> like a bad use of our time.  Better: write it up and post it, so we can
> all read before we meet.  Do not reply, except perhaps to ask for
> clarification.  Discuss in the meeting.
> 
> I can see such posts from Marc-Andre, Kevin, and Daniel.  Some of them
> could use a bit more detail, perhaps.

To be a bit more specific and give more details about what I'm currently
working on:

* The larger theme is command line QAPIfication, with the eventual goals
  of having introspection and a config file.

* Support for member aliases: This provides a way to bridge much of the
  gap between command line and QMP syntax. Markus tells me that it's
  unclear yet whether we want this in both QMP and CLI (unlikely for
  now), just CLI or not at all.

  First posted in November 2020, latest version is v4:
  https://patchew.org/QEMU/20210917161320.201086-1-kwolf@redhat.com/

  * Does anyone apart from myself have additional uses for aliases that
    should be considered in the evaluation of the feature?

  * -chardev: I have patches that QAPIfy the option based on aliases,
    getting rid of the old handwritten parser that is inconsistent with
    QMP in non-obvious ways and replacing it with translation to QMP
    (both using aliases and a little C code) that makes the differences
    obvious.

    First posted in November 2020, more details in the cover letter:
    https://patchew.org/QEMU/20201112175905.404472-1-kwolf@redhat.com/

    Later versions (not yet posted as a series because I'm waiting for
    aliases) also make -chardev accept JSON syntax, which is what
    libvirt really wants to use.

  * Came up during the discussion of the patches:
    Maybe we should actually start defining some command line options in
    the schema now, even if many options will still have to defer to the
    old parsing code. But it would be a start, could set an example for
    any new options and provides a clear path to incrementally convert
    everything. If we don't even start, we're sure to never finish it.

* -object: First part is done, we have a full QAPI schema of all
  user-creatable objects. object-add and --object in tools go through
  pure QAPI paths. System emulator -object is still based on QemuOpts
  because of compatibility concerns with list hacks (affects one single
  property of one object), but JSON syntax has been added that goes
  straight to QAPI.

  Next part is actual integration of QOM with QAPI so that the
  configuration isn't duplicated between QOM code and the schema, but
  user-creatable QOM classes are defined and QOM boilerplate is
  generated from the schema. I've started looking into this, but still
  in an early stage.

* -device/device-add: This is in a terrible state all around. Even QMP
  device-add is based on QemuOpts, and libvirt makes use of this by
  passes everything as strings instead of the proper types.

  Peter Krempa is working on using the proper types in libvirt.

  I sent patches to at least add a JSON variant for the command line
  option -device which is a non-QemuOpts code path. It also tried to fix
  device-add to be type safe, but until libvirt is fixed, we must stay
  bug compatible. So for this and other reasons this is not ready yet
  and needs a v2.
  https://patchew.org/QEMU/20210924090427.9218-1-kwolf@redhat.com/

  Full QAPIfication with a schema and introspection will still take a
  while. The QOM/QAPI integration work mentioned above is a dependency
  there. A schema like for object-add, with a big union of all possible
  devices, just doesn't scale.

  Having JSON CLI and a non-QemuOpts QMP command allow us to make the
  traditional syntax unstable (HMP-like support) and should be very
  helpful when we eventually do get to do proper QAPIfication.

  * Should we have a general rule that for command line options that
    accept both JSON and a human format, only JSON is considered a
    stable API and the human format works more like HMP?

Also one non-technical topic that may or may not be worth discussing:

* Patches to QAPI tend to take a long time to be merged (or rejected).
  This is understandable as a lot of things are going on and QAPI is an
  external interface with compatibility promises, so we need to be
  careful. Can we still still do anything to improve the situation? For
  example, can the load be spread across more people?

Kevin

Re: QAPI sync meeting

[Paolo Bonzini]

On 07/10/21 12:01, Kevin Wolf wrote:
> 
>    * -chardev: I have patches that QAPIfy the option based on aliases,
>      getting rid of the old handwritten parser that is inconsistent with
>      QMP in non-obvious ways and replacing it with translation to QMP
>      (both using aliases and a little C code) that makes the differences
>      obvious.
> 
>      First posted in November 2020, more details in the cover letter:
>      https://patchew.org/QEMU/20201112175905.404472-1-kwolf@redhat.com/
> 
>      Later versions (not yet posted as a series because I'm waiting for
>      aliases) also make -chardev accept JSON syntax, which is what
>      libvirt really wants to use.

I'm still not sure about this...  It's an awful lot of code if the 
aliases are only used by -chardev, and I'd rather use -object/object-add 
for chardevs if that's at all possible.

Paolo

Re: QAPI sync meeting

[Kevin Wolf]

Am 07.10.2021 um 12:23 hat Paolo Bonzini geschrieben:
> On 07/10/21 12:01, Kevin Wolf wrote:
> > 
> >    * -chardev: I have patches that QAPIfy the option based on aliases,
> >      getting rid of the old handwritten parser that is inconsistent with
> >      QMP in non-obvious ways and replacing it with translation to QMP
> >      (both using aliases and a little C code) that makes the differences
> >      obvious.
> > 
> >      First posted in November 2020, more details in the cover letter:
> >      https://patchew.org/QEMU/20201112175905.404472-1-kwolf@redhat.com/
> > 
> >      Later versions (not yet posted as a series because I'm waiting for
> >      aliases) also make -chardev accept JSON syntax, which is what
> >      libvirt really wants to use.
> 
> I'm still not sure about this...  It's an awful lot of code if the aliases
> are only used by -chardev, and I'd rather use -object/object-add for
> chardevs if that's at all possible.

The important part for me there is getting rid of the second parser that
is inconsistent with QAPI - and people add to it without fully realising
that it's a separate implementation, so they test -chardev and leave
chardev-add behind broken.

My approach keeps the existing command line syntax and still makes sure
that inputs from both the CLI and QMP go through a single code path,
making sure that they are consistent.

Aliases are a helpful tool to achieve this, but the series can be
rewritten a bit if people are fundamentally against having aliases.
Aliases do nothing that C code can't do.

I don't think that aliases are a lot of code, or even complicated code.
Current v4 looks like a lot of lines of code because Markus made me add
big comments everywhere and tons of tests. The actual code additions are
rather small. But I also notice that there is resistance against having
multiple ways to specify the same thing (which is the essence of
aliases), so if people hate them, let's throw them away. The only part I
really dislike with this scenario is that I could have been told almost
a year ago...

Anyway, your approach provides a different solution to the goal of
getting rid of the second parser if you extend it: Add -object support
to all chardev backends, then deprecate -chardev wholesale and drop it
two releases later. This feels contentious, but I'm not opposed.

Timeline: My series could be done for 6.2. Yours could have the
replacement in 6.2 the earliest if we start working on it right now,
then libvirt starts using it, deprecation in 7.0 or 7.1, then drop the
old interface two releases later, i.e.  December next year or March
2023.

Kevin

Re: QAPI sync meeting

[Daniel P. Berrangé]

On Thu, Oct 07, 2021 at 02:53:19PM +0200, Kevin Wolf wrote:
> Am 07.10.2021 um 12:23 hat Paolo Bonzini geschrieben:
> > On 07/10/21 12:01, Kevin Wolf wrote:
> > > 
> > >    * -chardev: I have patches that QAPIfy the option based on aliases,
> > >      getting rid of the old handwritten parser that is inconsistent with
> > >      QMP in non-obvious ways and replacing it with translation to QMP
> > >      (both using aliases and a little C code) that makes the differences
> > >      obvious.
> > > 
> > >      First posted in November 2020, more details in the cover letter:
> > >      https://patchew.org/QEMU/20201112175905.404472-1-kwolf@redhat.com/
> > > 
> > >      Later versions (not yet posted as a series because I'm waiting for
> > >      aliases) also make -chardev accept JSON syntax, which is what
> > >      libvirt really wants to use.
> > 
> > I'm still not sure about this...  It's an awful lot of code if the aliases
> > are only used by -chardev, and I'd rather use -object/object-add for
> > chardevs if that's at all possible.
> 
> The important part for me there is getting rid of the second parser that
> is inconsistent with QAPI - and people add to it without fully realising
> that it's a separate implementation, so they test -chardev and leave
> chardev-add behind broken.
> 
> My approach keeps the existing command line syntax and still makes sure
> that inputs from both the CLI and QMP go through a single code path,
> making sure that they are consistent.
> 
> Aliases are a helpful tool to achieve this, but the series can be
> rewritten a bit if people are fundamentally against having aliases.
> Aliases do nothing that C code can't do.
> 
> I don't think that aliases are a lot of code, or even complicated code.
> Current v4 looks like a lot of lines of code because Markus made me add
> big comments everywhere and tons of tests. The actual code additions are
> rather small. But I also notice that there is resistance against having
> multiple ways to specify the same thing (which is the essence of
> aliases), so if people hate them, let's throw them away. The only part I
> really dislike with this scenario is that I could have been told almost
> a year ago...
> 
> Anyway, your approach provides a different solution to the goal of
> getting rid of the second parser if you extend it: Add -object support
> to all chardev backends, then deprecate -chardev wholesale and drop it
> two releases later. This feels contentious, but I'm not opposed.

If we were thinking about QEMU from new ignoring existing design,
I could even imagine that all of -chardev, -netdev, -device, etc
would actually be -object. So from my POV I don't think it is
unreasonable to take this direction.

> Timeline: My series could be done for 6.2. Yours could have the
> replacement in 6.2 the earliest if we start working on it right now,
> then libvirt starts using it, deprecation in 7.0 or 7.1, then drop the
> old interface two releases later, i.e.  December next year or March
> 2023.

Are the two approaches mutually exclusive rather than complementary ?
eg is Kevin's work a worthwhile incremental step forward, even if we
eventually get to replacing -chardev with -object ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: QAPI sync meeting

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 07/10/21 12:01, Kevin Wolf wrote:
>>    * -chardev: I have patches that QAPIfy the option based on
>> aliases,
>>      getting rid of the old handwritten parser that is inconsistent with
>>      QMP in non-obvious ways and replacing it with translation to QMP
>>      (both using aliases and a little C code) that makes the differences
>>      obvious.
>>      First posted in November 2020, more details in the cover
>> letter:
>>      https://patchew.org/QEMU/20201112175905.404472-1-kwolf@redhat.com/
>>      Later versions (not yet posted as a series because I'm waiting
>> for
>>      aliases) also make -chardev accept JSON syntax, which is what
>>      libvirt really wants to use.
>
> I'm still not sure about this...  It's an awful lot of code if the
> aliases are only used by -chardev,

We might use them for replacing other ad hoc parsers.  We have a bunch,
but -chardev's one is perhaps the worst one.  Whether aliases could be
useful for replacing others is not yet clear.

I initially hoped that they could help us clean up QMP some, but further
(and sadly much later) thought led me to obstacles.

>                                    and I'd rather use
> -object/object-add for chardevs if that's at all possible.

How far are we from making -object the preferred machine-friendly
interface for creating character devices?

Re: QAPI sync meeting

[Paolo Bonzini]

On 08/10/21 12:06, Markus Armbruster wrote:
> 
>>                                     and I'd rather use
>> -object/object-add for chardevs if that's at all possible.
> How far are we from making -object the preferred machine-friendly
> interface for creating character devices?

I'd guess a week of work if somebody could just dive in.  Plus then of 
course getting the patch in and the Libvirt enablement.

That said, I'm not going to be able to "dive in" for quite some time. 
My main goal was compound machine properties which were needed for SGX, 
and that has been achieved.

Paolo

Re: QAPI sync meeting

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 8814 bytes --]

On Thu, Oct 7, 2021 at 6:02 AM Kevin Wolf <kwolf@redhat.com> wrote:

> Am 29.09.2021 um 14:18 hat Markus Armbruster geschrieben:
> > > I'm proposing a one-off here just for some initial discussion and
> planning.
> > > (We didn't have a QAPI BoF at KVM Forum and I'd like something similar
> > > now.) I don't expect we'll really "finish" plotting out a roadmap in a
> > > single 60 minute meeting, but it will probably give us good, targeted
> > > discussions on the ML to follow. If more meetings are desired to
> discuss
> > > specific topics, I'm not against scheduling them as-needed.
> >
> > Let's have one now, and then more as needed.
> >
> > Jumping into the meeting unprepared, then spending half of the alloted
> > time listening to people explaining where they want to take QAPI seems
> > like a bad use of our time.  Better: write it up and post it, so we can
> > all read before we meet.  Do not reply, except perhaps to ask for
> > clarification.  Discuss in the meeting.
> >
> > I can see such posts from Marc-Andre, Kevin, and Daniel.  Some of them
> > could use a bit more detail, perhaps.
>
> To be a bit more specific and give more details about what I'm currently
> working on:
>
> * The larger theme is command line QAPIfication, with the eventual goals
>   of having introspection and a config file.
>
> * Support for member aliases: This provides a way to bridge much of the
>   gap between command line and QMP syntax. Markus tells me that it's
>   unclear yet whether we want this in both QMP and CLI (unlikely for
>   now), just CLI or not at all.
>
>   First posted in November 2020, latest version is v4:
>   https://patchew.org/QEMU/20210917161320.201086-1-kwolf@redhat.com/
>
>   * Does anyone apart from myself have additional uses for aliases that
>     should be considered in the evaluation of the feature?
>
>   * -chardev: I have patches that QAPIfy the option based on aliases,
>     getting rid of the old handwritten parser that is inconsistent with
>     QMP in non-obvious ways and replacing it with translation to QMP
>     (both using aliases and a little C code) that makes the differences
>     obvious.
>
>     First posted in November 2020, more details in the cover letter:
>     https://patchew.org/QEMU/20201112175905.404472-1-kwolf@redhat.com/
>
>     Later versions (not yet posted as a series because I'm waiting for
>     aliases) also make -chardev accept JSON syntax, which is what
>     libvirt really wants to use.
>
>   * Came up during the discussion of the patches:
>     Maybe we should actually start defining some command line options in
>     the schema now, even if many options will still have to defer to the
>     old parsing code. But it would be a start, could set an example for
>     any new options and provides a clear path to incrementally convert
>     everything. If we don't even start, we're sure to never finish it.
>
> * -object: First part is done, we have a full QAPI schema of all
>   user-creatable objects. object-add and --object in tools go through
>   pure QAPI paths. System emulator -object is still based on QemuOpts
>   because of compatibility concerns with list hacks (affects one single
>   property of one object), but JSON syntax has been added that goes
>   straight to QAPI.
>
>   Next part is actual integration of QOM with QAPI so that the
>   configuration isn't duplicated between QOM code and the schema, but
>   user-creatable QOM classes are defined and QOM boilerplate is
>   generated from the schema. I've started looking into this, but still
>   in an early stage.
>
> * -device/device-add: This is in a terrible state all around. Even QMP
>   device-add is based on QemuOpts, and libvirt makes use of this by
>   passes everything as strings instead of the proper types.
>
>   Peter Krempa is working on using the proper types in libvirt.
>
>   I sent patches to at least add a JSON variant for the command line
>   option -device which is a non-QemuOpts code path. It also tried to fix
>   device-add to be type safe, but until libvirt is fixed, we must stay
>   bug compatible. So for this and other reasons this is not ready yet
>   and needs a v2.
>   https://patchew.org/QEMU/20210924090427.9218-1-kwolf@redhat.com/
>
>   Full QAPIfication with a schema and introspection will still take a
>   while. The QOM/QAPI integration work mentioned above is a dependency
>   there. A schema like for object-add, with a big union of all possible
>   devices, just doesn't scale.
>
>   Having JSON CLI and a non-QemuOpts QMP command allow us to make the
>   traditional syntax unstable (HMP-like support) and should be very
>   helpful when we eventually do get to do proper QAPIfication.
>
>   * Should we have a general rule that for command line options that
>     accept both JSON and a human format, only JSON is considered a
>     stable API and the human format works more like HMP?
>
> Also one non-technical topic that may or may not be worth discussing:
>
> * Patches to QAPI tend to take a long time to be merged (or rejected).
>   This is understandable as a lot of things are going on and QAPI is an
>   external interface with compatibility promises, so we need to be
>   careful. Can we still still do anything to improve the situation? For
>   example, can the load be spread across more people?
>
> Kevin
>
>
I'll drop by and leave an extremely crude summary here; I had a monster
headache all day yesterday that has still not left me, my apologies for
this being a bit close to the wire:

My main motivators for QAPI are:

(1) A QEMU config file that is ultimately backed by the same schema as QMP.

Ideally, this config file would have intellisense syntax support in e.g.
pycharm/vscode. There is a large repository of "config file schemas" online
used for auto-complete and intellisense for various YAML/JSON configuration
files. I started researching this option around last year, for KVM Forum. I
began by giving a presentation on the state of the CLI as it existed at the
time, and submitted a demo that involved a very crudely hacked up
"x-configure" QMP command that re-routed to the CLI parser. It worked only
because I was fairly aggressive about removing a good number of CLI
arguments.

I consider this important for ease-of-use and discoverability reasons.

(2) The unification of QAPI and QDEV

There are numerous spots in the API where we stub out to an 'Any' type,
because the actual type information is in qdev or open-coded in various
device functions. In order to support the above goal, I see the unification
of the two dynamic systems as necessary for producing significantly
improved static schema that will facilitate users in discovering and using
QEMU's features.

This goal serves primarily as reinforcement for the first.

(3) The generation of an improved QAPI index in the Sphinx ReST docs.

At present, we do not have cross-references for QMP commands, events,
enums, structs, arguments, etc. I wrote a prototype "QAPI Domain" plugin
for Sphinx earlier this year, but there is work remaining to integrate it
to the current QAPIDoc class in the QAPI generator. In essence, I have a
"new" syntax that works to accomplish this goal, but I don't have a
translator from the old to new syntax yet. I also do not have much of the
error checking that we'd want to enforce layout, style, etc.

This will help further bolster the discoverability and ease of use for our
API. It will allow me to write ReST documents that natively cross-reference
the exact commands and structures I am talking about elsewhere in the docs.
It will allow me to reference QMP command documentation from Python
docstrings, for instance, which would be very nice for a Python SDK ...

(4) Libraries and SDKs for Python, Golang, and rust.

I've been working on the Python angle here, because we already have all of
the pieces for a python library. I want to package them and release them on
PyPI as developer tooling and "advanced user" tooling. Not something meant
to replace libvirt, but instead just well-supported, minimalist FOSS
tooling that allows you to drive and experiment with cutting-edge features
of QEMU. The intended audience is power users and developers of other
software that consumes QEMU. I have noticed many projects that wind up
relying on QEMU tend to reimplement the same code (sometimes poorly) over
and over again -- I want to provide a bit more first class support to our
API.

To this end, I've spent the last year cleaning up our Python code quite a
bit, including adding documentation and static typing to the QAPI generator
itself, in the hopes that this will facilitate adding new languages and
generators. I've also recently checked in a new version of the python QMP
library; if it tests well this release cycle I am hoping to release it on
PyPI this year.

[-- Attachment #2: Type: text/html, Size: 10474 bytes --]

Re: QAPI sync meeting

[Paolo Bonzini]

On 27/09/21 18:55, John Snow wrote:
> Hiya,
> 
> I'd like to propose that at least the three of us arrange a time to have 
> a meeting where we discuss our plans and ideas for QAPI going forward, 
> including rust, python, and golang extensions to the QAPI generator, 
> what we hope to accomplish with those projects, and so on.
> 
> What I am hoping to get out of this for myself is a high-level overview 
> of people's plans for QAPI and to produce some notes on those plans so 
> that I can have a reference that we've all acknowledged as roughly 
> accurate to be able to keep the community's design goals for QAPI in 
> mind as I continue my own development. Ultimately, I'd like some kind of 
> rough draft of a "QAPI roadmap".
> 
> I know there was a rust meetup during KVM Forum, but I was unable to 
> attend due to the timing. I'd like to expand the focus a little more 
> broadly to QAPI in general and discuss our "personal" roadmaps, goals, 
> queued work, etc so that we can collaboratively formulate a broader 
> vision of our work.
> 
> I'm posting to qemu-devel in case anyone else has an interest in this 
> area and would like to eavesdrop or share opinions, but we should 
> probably come up with an agenda first. So:
> 
> Proposed agenda:
> 
> Current projects, wishlists, and goals for QAPI:
> - Markus (~10 min)
> - Marc-Andre (~10 min) (Rust, dbus, etc?)
> - jsnow (~10 min) (Python, golang, etc)

I would be interested in attending too.

Paolo

Re: QAPI sync meeting

[Damien Hedde]

On 9/27/21 18:55, John Snow wrote:
> Hiya,
> 
> I'd like to propose that at least the three of us arrange a time to have 
> a meeting where we discuss our plans and ideas for QAPI going forward, 
> including rust, python, and golang extensions to the QAPI generator, 
> what we hope to accomplish with those projects, and so on.
> 
> What I am hoping to get out of this for myself is a high-level overview 
> of people's plans for QAPI and to produce some notes on those plans so 
> that I can have a reference that we've all acknowledged as roughly 
> accurate to be able to keep the community's design goals for QAPI in 
> mind as I continue my own development. Ultimately, I'd like some kind of 
> rough draft of a "QAPI roadmap".
> 
> I know there was a rust meetup during KVM Forum, but I was unable to 
> attend due to the timing. I'd like to expand the focus a little more 
> broadly to QAPI in general and discuss our "personal" roadmaps, goals, 
> queued work, etc so that we can collaboratively formulate a broader 
> vision of our work.
> 
> I'm posting to qemu-devel in case anyone else has an interest in this 
> area and would like to eavesdrop or share opinions, but we should 
> probably come up with an agenda first. So:
> 

I would be interested too.

My current topic of interest is somewhat distant (roughly: "remote" 
machine assembly) but I would like to better understand the QAPI roadmap.

--
Damien

Re: QAPI sync meeting

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On Wed, Sep 29, 2021 at 9:52 AM Damien Hedde <damien.hedde@greensocs.com>
wrote:

>
>
> On 9/27/21 18:55, John Snow wrote:
> > Hiya,
> >
> > I'd like to propose that at least the three of us arrange a time to have
> > a meeting where we discuss our plans and ideas for QAPI going forward,
> > including rust, python, and golang extensions to the QAPI generator,
> > what we hope to accomplish with those projects, and so on.
> >
> > What I am hoping to get out of this for myself is a high-level overview
> > of people's plans for QAPI and to produce some notes on those plans so
> > that I can have a reference that we've all acknowledged as roughly
> > accurate to be able to keep the community's design goals for QAPI in
> > mind as I continue my own development. Ultimately, I'd like some kind of
> > rough draft of a "QAPI roadmap".
> >
> > I know there was a rust meetup during KVM Forum, but I was unable to
> > attend due to the timing. I'd like to expand the focus a little more
> > broadly to QAPI in general and discuss our "personal" roadmaps, goals,
> > queued work, etc so that we can collaboratively formulate a broader
> > vision of our work.
> >
> > I'm posting to qemu-devel in case anyone else has an interest in this
> > area and would like to eavesdrop or share opinions, but we should
> > probably come up with an agenda first. So:
> >
>
> I would be interested too.
>
> My current topic of interest is somewhat distant (roughly: "remote"
> machine assembly) but I would like to better understand the QAPI roadmap.
>
>
The area that greensocs is working on is of direct interest to me as well:
I want 100% QEMU configuration via QMP.

(And then either I drop the command line, or re-architect the CLI such that
it is 100% a mapping onto equivalent QMP, or ... something. It's been a
somewhat hot topic at times, but 100% QMP configuration seems like where
the meat of the work is, anyway. Everything after that is kinda just
'details'.)


> --
> Damien
>
>

[-- Attachment #2: Type: text/html, Size: 2762 bytes --]