Re: QAPI sync meeting

[John Snow <jsnow@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 8814 bytes --]

On Thu, Oct 7, 2021 at 6:02 AM Kevin Wolf <kwolf@redhat.com> wrote:

> Am 29.09.2021 um 14:18 hat Markus Armbruster geschrieben:
> > > I'm proposing a one-off here just for some initial discussion and
> planning.
> > > (We didn't have a QAPI BoF at KVM Forum and I'd like something similar
> > > now.) I don't expect we'll really "finish" plotting out a roadmap in a
> > > single 60 minute meeting, but it will probably give us good, targeted
> > > discussions on the ML to follow. If more meetings are desired to
> discuss
> > > specific topics, I'm not against scheduling them as-needed.
> >
> > Let's have one now, and then more as needed.
> >
> > Jumping into the meeting unprepared, then spending half of the alloted
> > time listening to people explaining where they want to take QAPI seems
> > like a bad use of our time.  Better: write it up and post it, so we can
> > all read before we meet.  Do not reply, except perhaps to ask for
> > clarification.  Discuss in the meeting.
> >
> > I can see such posts from Marc-Andre, Kevin, and Daniel.  Some of them
> > could use a bit more detail, perhaps.
>
> To be a bit more specific and give more details about what I'm currently
> working on:
>
> * The larger theme is command line QAPIfication, with the eventual goals
>   of having introspection and a config file.
>
> * Support for member aliases: This provides a way to bridge much of the
>   gap between command line and QMP syntax. Markus tells me that it's
>   unclear yet whether we want this in both QMP and CLI (unlikely for
>   now), just CLI or not at all.
>
>   First posted in November 2020, latest version is v4:
>   https://patchew.org/QEMU/20210917161320.201086-1-kwolf@redhat.com/
>
>   * Does anyone apart from myself have additional uses for aliases that
>     should be considered in the evaluation of the feature?
>
>   * -chardev: I have patches that QAPIfy the option based on aliases,
>     getting rid of the old handwritten parser that is inconsistent with
>     QMP in non-obvious ways and replacing it with translation to QMP
>     (both using aliases and a little C code) that makes the differences
>     obvious.
>
>     First posted in November 2020, more details in the cover letter:
>     https://patchew.org/QEMU/20201112175905.404472-1-kwolf@redhat.com/
>
>     Later versions (not yet posted as a series because I'm waiting for
>     aliases) also make -chardev accept JSON syntax, which is what
>     libvirt really wants to use.
>
>   * Came up during the discussion of the patches:
>     Maybe we should actually start defining some command line options in
>     the schema now, even if many options will still have to defer to the
>     old parsing code. But it would be a start, could set an example for
>     any new options and provides a clear path to incrementally convert
>     everything. If we don't even start, we're sure to never finish it.
>
> * -object: First part is done, we have a full QAPI schema of all
>   user-creatable objects. object-add and --object in tools go through
>   pure QAPI paths. System emulator -object is still based on QemuOpts
>   because of compatibility concerns with list hacks (affects one single
>   property of one object), but JSON syntax has been added that goes
>   straight to QAPI.
>
>   Next part is actual integration of QOM with QAPI so that the
>   configuration isn't duplicated between QOM code and the schema, but
>   user-creatable QOM classes are defined and QOM boilerplate is
>   generated from the schema. I've started looking into this, but still
>   in an early stage.
>
> * -device/device-add: This is in a terrible state all around. Even QMP
>   device-add is based on QemuOpts, and libvirt makes use of this by
>   passes everything as strings instead of the proper types.
>
>   Peter Krempa is working on using the proper types in libvirt.
>
>   I sent patches to at least add a JSON variant for the command line
>   option -device which is a non-QemuOpts code path. It also tried to fix
>   device-add to be type safe, but until libvirt is fixed, we must stay
>   bug compatible. So for this and other reasons this is not ready yet
>   and needs a v2.
>   https://patchew.org/QEMU/20210924090427.9218-1-kwolf@redhat.com/
>
>   Full QAPIfication with a schema and introspection will still take a
>   while. The QOM/QAPI integration work mentioned above is a dependency
>   there. A schema like for object-add, with a big union of all possible
>   devices, just doesn't scale.
>
>   Having JSON CLI and a non-QemuOpts QMP command allow us to make the
>   traditional syntax unstable (HMP-like support) and should be very
>   helpful when we eventually do get to do proper QAPIfication.
>
>   * Should we have a general rule that for command line options that
>     accept both JSON and a human format, only JSON is considered a
>     stable API and the human format works more like HMP?
>
> Also one non-technical topic that may or may not be worth discussing:
>
> * Patches to QAPI tend to take a long time to be merged (or rejected).
>   This is understandable as a lot of things are going on and QAPI is an
>   external interface with compatibility promises, so we need to be
>   careful. Can we still still do anything to improve the situation? For
>   example, can the load be spread across more people?
>
> Kevin
>
>
I'll drop by and leave an extremely crude summary here; I had a monster
headache all day yesterday that has still not left me, my apologies for
this being a bit close to the wire:

My main motivators for QAPI are:

(1) A QEMU config file that is ultimately backed by the same schema as QMP.

Ideally, this config file would have intellisense syntax support in e.g.
pycharm/vscode. There is a large repository of "config file schemas" online
used for auto-complete and intellisense for various YAML/JSON configuration
files. I started researching this option around last year, for KVM Forum. I
began by giving a presentation on the state of the CLI as it existed at the
time, and submitted a demo that involved a very crudely hacked up
"x-configure" QMP command that re-routed to the CLI parser. It worked only
because I was fairly aggressive about removing a good number of CLI
arguments.

I consider this important for ease-of-use and discoverability reasons.

(2) The unification of QAPI and QDEV

There are numerous spots in the API where we stub out to an 'Any' type,
because the actual type information is in qdev or open-coded in various
device functions. In order to support the above goal, I see the unification
of the two dynamic systems as necessary for producing significantly
improved static schema that will facilitate users in discovering and using
QEMU's features.

This goal serves primarily as reinforcement for the first.

(3) The generation of an improved QAPI index in the Sphinx ReST docs.

At present, we do not have cross-references for QMP commands, events,
enums, structs, arguments, etc. I wrote a prototype "QAPI Domain" plugin
for Sphinx earlier this year, but there is work remaining to integrate it
to the current QAPIDoc class in the QAPI generator. In essence, I have a
"new" syntax that works to accomplish this goal, but I don't have a
translator from the old to new syntax yet. I also do not have much of the
error checking that we'd want to enforce layout, style, etc.

This will help further bolster the discoverability and ease of use for our
API. It will allow me to write ReST documents that natively cross-reference
the exact commands and structures I am talking about elsewhere in the docs.
It will allow me to reference QMP command documentation from Python
docstrings, for instance, which would be very nice for a Python SDK ...

(4) Libraries and SDKs for Python, Golang, and rust.

I've been working on the Python angle here, because we already have all of
the pieces for a python library. I want to package them and release them on
PyPI as developer tooling and "advanced user" tooling. Not something meant
to replace libvirt, but instead just well-supported, minimalist FOSS
tooling that allows you to drive and experiment with cutting-edge features
of QEMU. The intended audience is power users and developers of other
software that consumes QEMU. I have noticed many projects that wind up
relying on QEMU tend to reimplement the same code (sometimes poorly) over
and over again -- I want to provide a bit more first class support to our
API.

To this end, I've spent the last year cleaning up our Python code quite a
bit, including adding documentation and static typing to the QAPI generator
itself, in the hopes that this will facilitate adding new languages and
generators. I've also recently checked in a new version of the python QMP
library; if it tests well this release cycle I am hoping to release it on
PyPI this year.

[-- Attachment #2: Type: text/html, Size: 10474 bytes --]