Making QEMU easier for management tools and applications

Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

Hi,
QEMU presents a command-line interface and QMP monitor for
applications to interact with.  Applications actually need API
bindings in their programming language.  Bindings avoid reimplementing
code to spawn a QEMU process and interact with QMP.  QEMU is kind of
lazy and de facto relies on libvirt for API bindings.

Is it time for better QEMU APIs?

1. We have qapi-schema.json.  Let's render to HTML and publish
versioned documentation online.

2. scripts/qmp/ contains command-line tools for QMP communication.
They could use some polish and then be shipped.

3. python/qemu/ contains Python modules for managing a QEMU process
and QMP communication.  This should be packaged in distros and
available on PyPI.

4. Go and Rust bindings would also be useful.  There is
https://github.com/intel/govmm but I think it makes sense to keep it
in qemu.git and provide an interface similar to our Python modules.

5. A jailer is needed to isolate the QEMU process and vhost-user
device backends using seccomp, Linux namespaces, and maybe
SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
security, but it's becoming a common task for any device backend and
IMO should be its own launcher tool.

6. A configuration file format is sorely needed so that guest
configuration can be persisted and easily launched.  Today you have to
create a shell script that executes a QEMU command-line, but this is
suboptimal because sharing executable scripts is dangerous from a
security perspective and is hard to parse or modify programmatically.

In many of these areas we already have a partial solution.  It just
needs more work.  I think it would be worth the effort and the mental
shift to really providing APIs that are easy to use by applications.

What do you think?

Have I missed things that are needed?

Have I included things that are unnecessary?

Stefan

Re: Making QEMU easier for management tools and applications

[Richard W.M. Jones]

On Fri, Dec 20, 2019 at 04:13:59PM +0000, Stefan Hajnoczi wrote:
> Hi,
> QEMU presents a command-line interface and QMP monitor for
> applications to interact with.  Applications actually need API
> bindings in their programming language.  Bindings avoid reimplementing
> code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> lazy and de facto relies on libvirt for API bindings.

Just wanted to add that for a concrete example you might want to look
at how libguestfs launches qemu, either directly:

  https://github.com/libguestfs/libguestfs/blob/master/lib/launch-direct.c

or via libvirt:

  https://github.com/libguestfs/libguestfs/blob/master/lib/launch-libvirt.c

Obviously I support anything that makes that easier, especially
the direct case.

Some points that you didn't cover:

- Any APIs we use need to be callable from a C library.  This has a
  few weird and wonderful beartraps to be aware of, such as being
  unable to predict the state of signal masks, not knowing what file
  descriptors will be open, and being religious about cleaning up any
  resources or subprocesses that you allocate / fork.

- To be usable, the new APIs need to cover everything that libguestfs
  needs, otherwise we literally cannot switch over to them.  (I guess
  the same applies to libvirt to an even greater extent.)

> 6. A configuration file format is sorely needed so that guest
> configuration can be persisted and easily launched. 

Actually qemu already has that, but it's really half-assed, barely
documented, and doesn't cover major features that qemu provides :-)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

On Fri, Dec 20, 2019 at 09:07:50PM +0000, Richard W.M. Jones wrote:
> On Fri, Dec 20, 2019 at 04:13:59PM +0000, Stefan Hajnoczi wrote:
> > 6. A configuration file format is sorely needed so that guest
> > configuration can be persisted and easily launched. 
> 
> Actually qemu already has that, but it's really half-assed, barely
> documented, and doesn't cover major features that qemu provides :-)

Yes, I mean a fully-working configuration file format :).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> Hi,
> QEMU presents a command-line interface and QMP monitor for
> applications to interact with.  Applications actually need API
> bindings in their programming language.  Bindings avoid reimplementing
> code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> lazy and de facto relies on libvirt for API bindings.
>
> Is it time for better QEMU APIs?
>
> 1. We have qapi-schema.json.  Let's render to HTML and publish
> versioned documentation online.

Make can build docs/interop/qemu-qmp-ref.{7,html,info,pdf,txt}.  Grab
the .html and go for it.  There's also qmp-ga-ref.

Sadly, this documentation is not nearly as good as it could be.  The doc
comment format is an ad hoc solution to the problem of enabling doc
generation with minimal churn to existing comments.  The doc generator
makes an effort to play that hand (but the hand sucks).  Finally and
most seriously, the doc comments are simply not good enough.

> 2. scripts/qmp/ contains command-line tools for QMP communication.
> They could use some polish and then be shipped.

MAINTAINERS blames them on me, but they're effectively unmaintained.
Prerequisite for shipping: having a maintainer who actually gives a
damn.

Let's review them:

* scripts/qmp/qemu-ga-client

  Is anybody using this?

  I think it should be unter "QEMU Guest Agent", not under "QMP".

* scripts/qmp/qmp

  Half-hearted attempt at a CLI tool for sending a QMP command.  Is
  anybody using this?
 
* scripts/qmp/qmp-shell

  Half-hearted attempt at a human-friendly wrapper around the JSON
  syntax.  I have no use for this myself.

* scripts/qmp/qom-fuse

  I find this occasionally useful for QOM-spelunking.  The need for this
  indicates our QMP interface to QOM is too primitive for such uses.

* scripts/qmp/qom-get
  scripts/qmp/qom-list
  scripts/qmp/qom-set
  scripts/qmp/qom-tree

  Rather specialized.  Is anybody using them?

> 3. python/qemu/ contains Python modules for managing a QEMU process
> and QMP communication.  This should be packaged in distros and
> available on PyPI.

Currently maintained by Eduardo and Cleber (cc'ed) under "Python
scripts".

> 4. Go and Rust bindings would also be useful.  There is
> https://github.com/intel/govmm but I think it makes sense to keep it
> in qemu.git and provide an interface similar to our Python modules.

Mapping QAPI/QMP commands and events to function signatures isn't hard
(the QAPI code generator does).  Two problems (at least):

1. Leads to some pretty ridiculous functions.  Here's one:

    void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
                             const char *device,
                             const char *target,
                             bool has_replaces, const char *replaces,
                             MirrorSyncMode sync,
                             bool has_speed, int64_t speed,
                             bool has_granularity, uint32_t granularity,
                             bool has_buf_size, int64_t buf_size,
                             bool has_on_source_error,
                             BlockdevOnError on_source_error,
                             bool has_on_target_error, BlockdevOnError on_target_error,
                             bool has_filter_node_name, const char *filter_node_name,
                             bool has_copy_mode, MirrorCopyMode copy_mode, 
                             bool has_auto_finalize, bool auto_finalize,
                             bool has_auto_dismiss, bool auto_dismiss,
                             Error **errp);

  We commonly use 'boxed': true for such beasts, which results in
  functions like this one:

    void qmp_blockdev_add(BlockdevOptions *arg, Error **errp);

2. Many schema changes that are nicely backward compatible in QMP are
   anything but in such an "obvious" C API.  Adding optional arguments,
   for instance, or changing integer type width.  The former is less of
   an issue with 'boxed': true.

Perhaps less of an issue with dynamic languages.

I figure a static language would need much more expressive oomph than C
to be a good target.  No idea how well Go or Rust bindings can work.

> 5. A jailer is needed to isolate the QEMU process and vhost-user
> device backends using seccomp, Linux namespaces, and maybe
> SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> security, but it's becoming a common task for any device backend and
> IMO should be its own launcher tool.

Perhaps the libvirt refactoring effort can give us one.

> 6. A configuration file format is sorely needed so that guest
> configuration can be persisted and easily launched.  Today you have to
> create a shell script that executes a QEMU command-line, but this is
> suboptimal because sharing executable scripts is dangerous from a
> security perspective and is hard to parse or modify programmatically.

No argument.  There is -readconfig, but it falls way short.

With command line QAPIfication, a real configuration file will be quite
feasible.

The main reason for the lack of progress there is our dedication to
backward compatibility.  A functional replacement of our CLI is a huge
task already.  Throwing in backward compatibility makes it a daunting
one.  Not least because nobody fully understands all the quirks.

> In many of these areas we already have a partial solution.  It just
> needs more work.  I think it would be worth the effort and the mental
> shift to really providing APIs that are easy to use by applications.
>
> What do you think?
>
> Have I missed things that are needed?
>
> Have I included things that are unnecessary?

I feel we need to make up our minds what kind of interface(s) we want to
provide.

We provide a low-level interface for management applications.  I feel we
need to nail this one.  Its QMP part is okay, I think, the CLI part is
not.

We also try to provide friendlier interfaces for human users.  Not only
do we suck at it, we're also prone to create liabilities for later: much
of the compatibility woes that have been holding us back are rooted in
"convenience" features.  And don't get me started on the folly of us
dabbling in GUI.

Re: Making QEMU easier for management tools and applications

[Michal Prívozník]

On 12/21/19 10:02 AM, Markus Armbruster wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> 


>> 4. Go and Rust bindings would also be useful.  There is
>> https://github.com/intel/govmm but I think it makes sense to keep it
>> in qemu.git and provide an interface similar to our Python modules.
> 
> Mapping QAPI/QMP commands and events to function signatures isn't hard
> (the QAPI code generator does).  Two problems (at least):
> 
> 1. Leads to some pretty ridiculous functions.  Here's one:
> 
>     void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
>                              const char *device,
>                              const char *target,
>                              bool has_replaces, const char *replaces,
>                              MirrorSyncMode sync,
>                              bool has_speed, int64_t speed,
>                              bool has_granularity, uint32_t granularity,
>                              bool has_buf_size, int64_t buf_size,
>                              bool has_on_source_error,
>                              BlockdevOnError on_source_error,
>                              bool has_on_target_error, BlockdevOnError on_target_error,
>                              bool has_filter_node_name, const char *filter_node_name,
>                              bool has_copy_mode, MirrorCopyMode copy_mode, 
>                              bool has_auto_finalize, bool auto_finalize,
>                              bool has_auto_dismiss, bool auto_dismiss,
>                              Error **errp);
> 
>   We commonly use 'boxed': true for such beasts, which results in
>   functions like this one:
> 
>     void qmp_blockdev_add(BlockdevOptions *arg, Error **errp);
> 
> 2. Many schema changes that are nicely backward compatible in QMP are
>    anything but in such an "obvious" C API.  Adding optional arguments,
>    for instance, or changing integer type width.  The former is less of
>    an issue with 'boxed': true.
> 
> Perhaps less of an issue with dynamic languages.
> 
> I figure a static language would need much more expressive oomph than C
> to be a good target.  No idea how well Go or Rust bindings can work.

This is something that bothered me for a while now. Even though it's not
as bad as it used to be because we are not adding so much wrappers for
monitor commands as we used to. I mean, in libvirt the wrapper for a
monitor command has to be written by hand. Worse, whenever I'm adding a
wrapper I look at the QMP schema of it and let my muscle memory write
the wrapper.

However, it's not only what Markus already mentioned. Even if we
generated wrappers by a script, we need to be able to generate wrappers
for every single supported version of qemu.

For instance, if qemu version X has a command that accepts some set of
arguments and this set changes in version X+1 then libvirt needs both
wrappers and decides at runtime (depending on what version it is talking
to) what wrapper to use.

Unfortunately, I don't see any easy way out.

Michal

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 23.12.2019 um 16:04 hat Michal Prívozník geschrieben:
> On 12/21/19 10:02 AM, Markus Armbruster wrote:
> > Stefan Hajnoczi <stefanha@gmail.com> writes:
> > 
> 
> 
> >> 4. Go and Rust bindings would also be useful.  There is
> >> https://github.com/intel/govmm but I think it makes sense to keep it
> >> in qemu.git and provide an interface similar to our Python modules.
> > 
> > Mapping QAPI/QMP commands and events to function signatures isn't hard
> > (the QAPI code generator does).  Two problems (at least):
> > 
> > 1. Leads to some pretty ridiculous functions.  Here's one:
> > 
> >     void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
> >                              const char *device,
> >                              const char *target,
> >                              bool has_replaces, const char *replaces,
> >                              MirrorSyncMode sync,
> >                              bool has_speed, int64_t speed,
> >                              bool has_granularity, uint32_t granularity,
> >                              bool has_buf_size, int64_t buf_size,
> >                              bool has_on_source_error,
> >                              BlockdevOnError on_source_error,
> >                              bool has_on_target_error, BlockdevOnError on_target_error,
> >                              bool has_filter_node_name, const char *filter_node_name,
> >                              bool has_copy_mode, MirrorCopyMode copy_mode, 
> >                              bool has_auto_finalize, bool auto_finalize,
> >                              bool has_auto_dismiss, bool auto_dismiss,
> >                              Error **errp);
> > 
> >   We commonly use 'boxed': true for such beasts, which results in
> >   functions like this one:
> > 
> >     void qmp_blockdev_add(BlockdevOptions *arg, Error **errp);
> > 
> > 2. Many schema changes that are nicely backward compatible in QMP are
> >    anything but in such an "obvious" C API.  Adding optional arguments,
> >    for instance, or changing integer type width.  The former is less of
> >    an issue with 'boxed': true.
> > 
> > Perhaps less of an issue with dynamic languages.
> > 
> > I figure a static language would need much more expressive oomph than C
> > to be a good target.  No idea how well Go or Rust bindings can work.
> 
> This is something that bothered me for a while now. Even though it's not
> as bad as it used to be because we are not adding so much wrappers for
> monitor commands as we used to. I mean, in libvirt the wrapper for a
> monitor command has to be written by hand. Worse, whenever I'm adding a
> wrapper I look at the QMP schema of it and let my muscle memory write
> the wrapper.
> 
> However, it's not only what Markus already mentioned. Even if we
> generated wrappers by a script, we need to be able to generate wrappers
> for every single supported version of qemu.
> 
> For instance, if qemu version X has a command that accepts some set of
> arguments and this set changes in version X+1 then libvirt needs both
> wrappers and decides at runtime (depending on what version it is talking
> to) what wrapper to use.
> 
> Unfortunately, I don't see any easy way out.

The easy way out would be tying libvirt to a specific QEMU version. And
I'm only half joking.

If libvirt didn't exist yet and we needed a management library for QEMU,
what we would build now would probably not look much like libvirt looks
today. We wouldn't try to have basic support for every hypervisor out
there, but integrate it much closer with QEMU and avoid much of the
backwards compatibility requirements that the interface between QEMU and
libvirt has (which requires us to deal with compatibility twice for
everything).

Maybe it would even be part of the QEMU repository, allowing a single
patch series to implement a new feature in the system emulator and
expose it in the API immediately instead of waiting for the next QEMU
release before libvirt can even think about implementing support for it.

So should libvirt move in that direction? Do people actually still make
much use of its multi-hypervisor nature, or would it make sense to split
it into separate libraries for each hypervisor that can be much tighter
integrated with (a specific version of) the respective hypervisor?

Kevin

Re: Making QEMU easier for management tools and applications

[Michal Privoznik]

On 1/7/20 10:36 AM, Kevin Wolf wrote:
> The easy way out would be tying libvirt to a specific QEMU version. And
> I'm only half joking.
> 
> If libvirt didn't exist yet and we needed a management library for QEMU,
> what we would build now would probably not look much like libvirt looks
> today. We wouldn't try to have basic support for every hypervisor out
> there, but integrate it much closer with QEMU and avoid much of the
> backwards compatibility requirements that the interface between QEMU and
> libvirt has (which requires us to deal with compatibility twice for
> everything).
> 

By doing this, you would force your consumers to implement compatibility 
layer each on their own. Freshly finished blockdev is a beautiful 
example - OpenStack, oVirt and whatnot - they all are/can use blockdev 
without even noticing, because the API provided by libvirt is stable and 
provides abstraction, i.e. you don't need to change anything in your 
domain XML to use blockdev.

Of course, you can apply the argument one more time and have mgmt 
application tied to a specific version of qemu. But even that is not 
good enough, because with backports version is just meaningless number.

> Maybe it would even be part of the QEMU repository, allowing a single
> patch series to implement a new feature in the system emulator and
> expose it in the API immediately instead of waiting for the next QEMU
> release before libvirt can even think about implementing support for it.

Thing is, it's not just qmp that a mgmt application has to master, it's 
also process managing (and with growing number of helper binaries this 
is not as trivial as fork() + exec()). This would need to be the bare 
minimum your API layer has to provide to be consumable by anybody.
But then you have some advanced subsystems to take care of (CGroups, 
SELinux, etc.) which are used heavily by OpenStack. oVirt and friends.

> 
> So should libvirt move in that direction? Do people actually still make
> much use of its multi-hypervisor nature, or would it make sense to split
> it into separate libraries for each hypervisor that can be much tighter
> integrated with (a specific version of) the respective hypervisor?

Truth to be told, I don't think libvirt is held back by its attempts to 
provide hypervisor agnostic APIs. Sure, it leads to some weirdness (e.g. 
different naming in libvirt and qemu), but as a libvirt developer I 
don't remember feeling blocked by this multi-hypervisor nature (not to 
mention that this has saved us couple of times).

Also, it would be not correct to think that a feature is implemented for 
all hypervisors in libvirt. I mean, when implementing a feature I 
usually implement it only for qemu driver and don't even look at other 
drivers (unless I'm doing a change in a core that causes build 
failures). On the other hand, I do sometimes review patches posted by 
developers from other companies which have interest in different 
hypervisors (e.g. there is a SUSE guy working on LXC driver, and another 
SUSE guy working on libxenlight (basically Xen)), so I do spend some 
time not working on qemu driver, but I'd say it's negligible.

Michal

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 07.01.2020 um 11:55 hat Michal Privoznik geschrieben:
> On 1/7/20 10:36 AM, Kevin Wolf wrote:
> > The easy way out would be tying libvirt to a specific QEMU version. And
> > I'm only half joking.
> > 
> > If libvirt didn't exist yet and we needed a management library for QEMU,
> > what we would build now would probably not look much like libvirt looks
> > today. We wouldn't try to have basic support for every hypervisor out
> > there, but integrate it much closer with QEMU and avoid much of the
> > backwards compatibility requirements that the interface between QEMU and
> > libvirt has (which requires us to deal with compatibility twice for
> > everything).
> 
> By doing this, you would force your consumers to implement compatibility
> layer each on their own. Freshly finished blockdev is a beautiful example -
> OpenStack, oVirt and whatnot - they all are/can use blockdev without even
> noticing, because the API provided by libvirt is stable and provides
> abstraction, i.e. you don't need to change anything in your domain XML to
> use blockdev.

Yes and no.

You could still keep using the same abstraction that libvirt has always
used while doing this. What my imaginary newly written management
library would do differently isn't necessarily the interface between
libvirt and applications, but getting rid of backwards compatibility
requirements in the interface between QEMU and libvirt.

But of course, blockdev isn't even a feature per se. It's getting the
abstraction right so that it's actually abstract enough to represent
everything. As long as libvirt keeps using an abstraction that is based
on simplistic setups, it won't be able to expose the full feature set of
QEMU. This is less than satisfying. In the long run, libvirt will have
to extend its abstraction to make full use of new features either way.

> Of course, you can apply the argument one more time and have mgmt
> application tied to a specific version of qemu. But even that is not
> good enough, because with backports version is just meaningless
> number.

I think this would be too much indeed.

> > Maybe it would even be part of the QEMU repository, allowing a single
> > patch series to implement a new feature in the system emulator and
> > expose it in the API immediately instead of waiting for the next QEMU
> > release before libvirt can even think about implementing support for it.
> 
> Thing is, it's not just qmp that a mgmt application has to master, it's also
> process managing (and with growing number of helper binaries this is not as
> trivial as fork() + exec()). This would need to be the bare minimum your API
> layer has to provide to be consumable by anybody.
> But then you have some advanced subsystems to take care of (CGroups,
> SELinux, etc.) which are used heavily by OpenStack. oVirt and friends.

Someone has to do this anyway. Note that here I'm still talking about
the hypothetical case where no libvirt existed yet.

If we cared only about OpenStack, oVirt and friends, this would still
all be QEMU-based, so not a big problem to have it tied to QEMU.

I'm not sure what this looks like in practice in libvirt: Are these
components shared between multiple hypervisor interfaces or is it only
for QEMU anyway?

If multiple hypervisors make use of it, how crazy would it be to imagine
reversing which project consumes which? Instead of having the libvirt
core consume the hypervisor-specific sublibraries, could a QEMU-specific
part live closer to QEMU and consume the libvirt core as an external
library?

I guess much of what I write in this thread is pure heresy. :-)
Maybe most of it isn't even useful. But maybe there is an idea or two in
it that are worth having a closer look at.

> > So should libvirt move in that direction? Do people actually still make
> > much use of its multi-hypervisor nature, or would it make sense to split
> > it into separate libraries for each hypervisor that can be much tighter
> > integrated with (a specific version of) the respective hypervisor?
> 
> Truth to be told, I don't think libvirt is held back by its attempts to
> provide hypervisor agnostic APIs. Sure, it leads to some weirdness (e.g.
> different naming in libvirt and qemu), but as a libvirt developer I don't
> remember feeling blocked by this multi-hypervisor nature (not to mention
> that this has saved us couple of times).

I would imagine so, because the problem doesn't become visible in the
daily work, but only in the bigger picture: The other hypervisors are
what prevent libvirt from being more tightly intergrated with QEMU.

This means that there is a boundary between QEMU and libvirt that makes
it really slow to get new features to the user. And both QEMU and
libvirt waste a lot of time for maintaining backwards compatibility in
things that wouldn't necessarily have to be stable interfaces if the
management library were developed in lockstep with QEMU.

> Also, it would be not correct to think that a feature is implemented for all
> hypervisors in libvirt. I mean, when implementing a feature I usually
> implement it only for qemu driver and don't even look at other drivers
> (unless I'm doing a change in a core that causes build failures). On the
> other hand, I do sometimes review patches posted by developers from other
> companies which have interest in different hypervisors (e.g. there is a SUSE
> guy working on LXC driver, and another SUSE guy working on libxenlight
> (basically Xen)), so I do spend some time not working on qemu driver, but
> I'd say it's negligible.

Time spent on non-QEMU isn't really my concern. Time spent maintaining
stable interface between QEMU and libvirt, and time spent waiting for
QEMU releases before libvirt development starts are my concern.

Kevin

Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin]

> On 7 Jan 2020, at 13:57, Kevin Wolf <kwolf@redhat.com> wrote:
> 
> Am 07.01.2020 um 11:55 hat Michal Privoznik geschrieben:
>> On 1/7/20 10:36 AM, Kevin Wolf wrote:
>>> The easy way out would be tying libvirt to a specific QEMU version. And
>>> I'm only half joking.
>>> 
>>> If libvirt didn't exist yet and we needed a management library for QEMU,
>>> what we would build now would probably not look much like libvirt looks
>>> today. We wouldn't try to have basic support for every hypervisor out
>>> there, but integrate it much closer with QEMU and avoid much of the
>>> backwards compatibility requirements that the interface between QEMU and
>>> libvirt has (which requires us to deal with compatibility twice for
>>> everything).
>> 
>> By doing this, you would force your consumers to implement compatibility
>> layer each on their own. Freshly finished blockdev is a beautiful example -
>> OpenStack, oVirt and whatnot - they all are/can use blockdev without even
>> noticing, because the API provided by libvirt is stable and provides
>> abstraction, i.e. you don't need to change anything in your domain XML to
>> use blockdev.
> 
> Yes and no.
> 
> You could still keep using the same abstraction that libvirt has always
> used while doing this. What my imaginary newly written management
> library would do differently isn't necessarily the interface between
> libvirt and applications, but getting rid of backwards compatibility
> requirements in the interface between QEMU and libvirt.
> 
> But of course, blockdev isn't even a feature per se. It's getting the
> abstraction right so that it's actually abstract enough to represent
> everything. As long as libvirt keeps using an abstraction that is based
> on simplistic setups, it won't be able to expose the full feature set of
> QEMU. This is less than satisfying. In the long run, libvirt will have
> to extend its abstraction to make full use of new features either way.
> 
>> Of course, you can apply the argument one more time and have mgmt
>> application tied to a specific version of qemu. But even that is not
>> good enough, because with backports version is just meaningless
>> number.
> 
> I think this would be too much indeed.
> 
>>> Maybe it would even be part of the QEMU repository, allowing a single
>>> patch series to implement a new feature in the system emulator and
>>> expose it in the API immediately instead of waiting for the next QEMU
>>> release before libvirt can even think about implementing support for it.
>> 
>> Thing is, it's not just qmp that a mgmt application has to master, it's also
>> process managing (and with growing number of helper binaries this is not as
>> trivial as fork() + exec()). This would need to be the bare minimum your API
>> layer has to provide to be consumable by anybody.
>> But then you have some advanced subsystems to take care of (CGroups,
>> SELinux, etc.) which are used heavily by OpenStack. oVirt and friends.
> 
> Someone has to do this anyway. Note that here I'm still talking about
> the hypothetical case where no libvirt existed yet.
> 
> If we cared only about OpenStack, oVirt and friends, this would still
> all be QEMU-based, so not a big problem to have it tied to QEMU.
> 
> I'm not sure what this looks like in practice in libvirt: Are these
> components shared between multiple hypervisor interfaces or is it only
> for QEMU anyway?
> 
> If multiple hypervisors make use of it, how crazy would it be to imagine
> reversing which project consumes which? Instead of having the libvirt
> core consume the hypervisor-specific sublibraries, could a QEMU-specific
> part live closer to QEMU and consume the libvirt core as an external
> library?
> 
> I guess much of what I write in this thread is pure heresy. :-)
> Maybe most of it isn't even useful. But maybe there is an idea or two in
> it that are worth having a closer look at.

Well, I don’t know if it is heresy, but at least as far as sandboxing / jailing
is concerned, I suggested something similar in earlier iterations of
the discussion. My way to say that I am part of your heresy, I guess.

> 
>>> So should libvirt move in that direction? Do people actually still make
>>> much use of its multi-hypervisor nature, or would it make sense to split
>>> it into separate libraries for each hypervisor that can be much tighter
>>> integrated with (a specific version of) the respective hypervisor?
>> 
>> Truth to be told, I don't think libvirt is held back by its attempts to
>> provide hypervisor agnostic APIs. Sure, it leads to some weirdness (e.g.
>> different naming in libvirt and qemu), but as a libvirt developer I don't
>> remember feeling blocked by this multi-hypervisor nature (not to mention
>> that this has saved us couple of times).
> 
> I would imagine so, because the problem doesn't become visible in the
> daily work, but only in the bigger picture: The other hypervisors are
> what prevent libvirt from being more tightly intergrated with QEMU.
> 
> This means that there is a boundary between QEMU and libvirt that makes
> it really slow to get new features to the user. And both QEMU and
> libvirt waste a lot of time for maintaining backwards compatibility in
> things that wouldn't necessarily have to be stable interfaces if the
> management library were developed in lockstep with QEMU.
> 
>> Also, it would be not correct to think that a feature is implemented for all
>> hypervisors in libvirt. I mean, when implementing a feature I usually
>> implement it only for qemu driver and don't even look at other drivers
>> (unless I'm doing a change in a core that causes build failures). On the
>> other hand, I do sometimes review patches posted by developers from other
>> companies which have interest in different hypervisors (e.g. there is a SUSE
>> guy working on LXC driver, and another SUSE guy working on libxenlight
>> (basically Xen)), so I do spend some time not working on qemu driver, but
>> I'd say it's negligible.
> 
> Time spent on non-QEMU isn't really my concern. Time spent maintaining
> stable interface between QEMU and libvirt, and time spent waiting for
> QEMU releases before libvirt development starts are my concern.
> 
> Kevin

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Sat, Dec 21, 2019 at 10:02:23AM +0100, Markus Armbruster wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> 
> > Hi,
> > QEMU presents a command-line interface and QMP monitor for
> > applications to interact with.  Applications actually need API
> > bindings in their programming language.  Bindings avoid reimplementing
> > code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> > lazy and de facto relies on libvirt for API bindings.
> >
> > Is it time for better QEMU APIs?

> * scripts/qmp/qmp-shell
> 
>   Half-hearted attempt at a human-friendly wrapper around the JSON
>   syntax.  I have no use for this myself.

I use this fairly often as its a useful debugging / experimentation
/ trouble shooting tool. There's similar ish functionality in
virsh qemu-monitor-command. I think there's scope of a supported
tool here that can talk to libvirt or a UNIX socket for doing
QMP commands, with a friendlier syntax & pretty printing. 

> > 4. Go and Rust bindings would also be useful.  There is
> > https://github.com/intel/govmm but I think it makes sense to keep it
> > in qemu.git and provide an interface similar to our Python modules.
> 
> Mapping QAPI/QMP commands and events to function signatures isn't hard
> (the QAPI code generator does).  Two problems (at least):
> 
> 1. Leads to some pretty ridiculous functions.  Here's one:
> 
>     void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
>                              const char *device,
>                              const char *target,
>                              bool has_replaces, const char *replaces,
>                              MirrorSyncMode sync,
>                              bool has_speed, int64_t speed,
>                              bool has_granularity, uint32_t granularity,
>                              bool has_buf_size, int64_t buf_size,
>                              bool has_on_source_error,
>                              BlockdevOnError on_source_error,
>                              bool has_on_target_error, BlockdevOnError on_target_error,
>                              bool has_filter_node_name, const char *filter_node_name,
>                              bool has_copy_mode, MirrorCopyMode copy_mode, 
>                              bool has_auto_finalize, bool auto_finalize,
>                              bool has_auto_dismiss, bool auto_dismiss,
>                              Error **errp);
> 
>   We commonly use 'boxed': true for such beasts, which results in
>   functions like this one:
> 
>     void qmp_blockdev_add(BlockdevOptions *arg, Error **errp);
> 
> 2. Many schema changes that are nicely backward compatible in QMP are
>    anything but in such an "obvious" C API.  Adding optional arguments,
>    for instance, or changing integer type width.  The former is less of
>    an issue with 'boxed': true.
> 
> Perhaps less of an issue with dynamic languages.
> 
> I figure a static language would need much more expressive oomph than C
> to be a good target.  No idea how well Go or Rust bindings can work.

In libvirt we've got this kind of problem already and have two different
approaches to tackling it.

For the really complex cases with lots of conceptual nesting, we would
end up using XML.

For the simpler case where its just a set of flat parameters, we have
something we call "virTypedParameters" which is effectively a poor man's
hash table where the keys are parameter names, and the values are a
union accepting different types.

In the Go APIs for libvirt though we end up mapping both those approaches
into Go structs. The key reason why we are able todo this in Go and not
C, is that we don't promise ABI compat in Go bindings. We'll extend the
structs at will, and some times a struct field gets turned into a nested
struct which is an API break too. With Go apps typically bundle all their
3rd party deps & often lock themselves to a specific release tag, so they
don't need ABI compat in the same way apps want at C level.

> > 6. A configuration file format is sorely needed so that guest
> > configuration can be persisted and easily launched.  Today you have to
> > create a shell script that executes a QEMU command-line, but this is
> > suboptimal because sharing executable scripts is dangerous from a
> > security perspective and is hard to parse or modify programmatically.
> 
> No argument.  There is -readconfig, but it falls way short.
> 
> With command line QAPIfication, a real configuration file will be quite
> feasible.
> 
> The main reason for the lack of progress there is our dedication to
> backward compatibility.  A functional replacement of our CLI is a huge
> task already.  Throwing in backward compatibility makes it a daunting
> one.  Not least because nobody fully understands all the quirks.

I think we need to re-think how we're attacking this problem, because
despite heroic efforts in developing QAPI & converting code it use it,
we've still not got the finish line in sight after 10 years.

As I mentioned in my other reply I think we should target the CLI as
something exclusively for humans, with convenient syntax and more
relaxed back compat requirements. ie don't guarantee perfect back
compat for the CLI - just modest effor to not gratuitously break it
without cause. Humans using the CLI can adapt if they hit obscure
quirks.

Our back compat pain with the quirks is because we have mgmt apps using
the CLI which can't tolerate semantic changes without warning. We can't
be more relaxed with CLI back compat without getting mgmt apps away
from using this CLI.  We can get mgmt apps off using the CLI until they
have an alternative they can use.

The hard part is that developing the QAPI alternative is that its impl
might accidentaly cause breakage in existing CLI before mgmt apps can
switch to the QAPI based approach.



I'm wondering if we need to think about a temporary *throwaway* fork of
QEMU. I wasn't involved in the work, but as an outside observer I think
it is interesting to see how the NEMU project fork ultimately led to QEMU
moving forward with a new KConfig approach, and the integration of microvm.
They had the freedom to develop these new approaches with zero regard
to back compat or broader QEMU needs that might otherwise hold back dev.
Once the forked impl was mostly done, "unknown problems" had largely become
"known problems", and the work could be integrated back into QEMU with
much less risk of disruption.


Could such a temporary fork approach help us solve the QAPI-ification
end goal ?  It doesn't have to be a separate project - it could be a
temporary qemu-qapi.git on qemu-project.org or github for experimentation,
but still under umbrealla of QEMU.

It would free us to implement an idealized 100% QAPI-ified config
approach in the shortest possible time, with zero regard to back
compat of the current CLI.

We could develop support for this in libvirt in parallel, letting
us test the combination to gain confidence in the result.

Essentially we'll be aiming to flush out all the "unknown problems"
that hold us back today. As we get confidence we'll be able to merge
it back into QEMU proper with much lower risk of introducing suprising
compat problems.

At some level you can consider each subsys maintainer's branch a fork,
but those branches are really focused on stable code that's ready to
merge. An NEMU like fork is more aggressive than that, as it provided
a focused place for *experimentation* with no regard for immediate
merge to GIT master. The cost of such a fork approach of course is
that of trying to keep up2date with development in QEMU git master.

The benefit of being able to cut the back-compat knot might still
make this a win, allowing completion of QAPI-ification in a faster
timeframe.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[John Snow]

On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>> * scripts/qmp/qmp-shell
>>
>>   Half-hearted attempt at a human-friendly wrapper around the JSON
>>   syntax.  I have no use for this myself.
> I use this fairly often as its a useful debugging / experimentation
> / trouble shooting tool. There's similar ish functionality in
> virsh qemu-monitor-command. I think there's scope of a supported
> tool here that can talk to libvirt or a UNIX socket for doing
> QMP commands, with a friendlier syntax & pretty printing. 
> 

qmp-shell is one of my go-to tools for working through bitmap workflows
where we don't have convenience commands yet, as some of the setups
required for fleecing et al involve quite a number of steps.

I can copy-paste raw JSON into a socket, but personally I like seeing my
commands neatly organized in a format where I can visually reduce them
to their components at a glance.

(What I mean is: It's hard to remember which QMP commands you've barfed
into a terminal because JSON is hard to read and looks very visually
repetitive.)

I tried to rewrite qmp-shell late last year, actually. I wanted to write
a new REPL that was json-aware in some manner such that you could write
multi-line commands like this:

> example-command arg={
  "hello": "world"
}

This requires, sadly, a streamable JSON parser. Most JSON parsers built
into Python as-is simply take a file pointer and consume the entirety of
the rest of the stream -- they don't play very nice with incomplete
input or input that may have trailing data, e.g.:

> example-command arg={
  "hello": "world"
} arg2={
  "oops!": "more json!"
}

Also, due to the nature of JSON as being a single discrete object and
never a stream of objects, no existing JSON parser really supports the
idea of ever seeing more than one object per buffer.

...So I investigated writing a proper grammar for qmp-shell.
Unfortunately, this basically means including the JSON grammar as a
subset of the shell grammar and writing your own parser for it entirely.

I looked into using Python's own lexer; but it's designed to lex
*python*, not *json*. I got a prototype lexer working for this purpose
under a grammar that I think reflects JSON, but I got that sinking
feeling that it was all more trouble than it was worth, and scrapped
working on it any further.

I did not find any other flex/yacc-like tools that seemed properly
idiomatic or otherwise heavily specialized. I gave up on the idea of
writing a new parser.

I'd love to offer a nice robust QMP shell that is available for use by
end users, but the syntax of the shell will need some major considerations.


(Coffee break)


The other thing I wanted to do was offer a TUI-esque backlog of
commands; something reminiscent of mutt or TUI mail clients. Arrow keys
could be used to browse the list of previous TX/RX objects,
spacebar/enter to view them in detail; '?' to pull up the relevant QAPI
documentation in a side-pane. That sort of thing.

I thought it would be nice to offer an "IRC-esque" view of the QMP
protocol where new events stream in from the bottom, and new commands
can be typed into the buffer bar. (I found no convincing way to combine
e.g. readline with a TUI interface. It seemed like you get one or the
other.)

Allowing qmp-shell to show events received asynchronously would also be
a big leap up from the current shell.

It's a lot of fantasy stuff I wasn't *quite* able to tie together.

--js

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>>> * scripts/qmp/qmp-shell
>>>
>>>   Half-hearted attempt at a human-friendly wrapper around the JSON
>>>   syntax.  I have no use for this myself.
>> I use this fairly often as its a useful debugging / experimentation
>> / trouble shooting tool. There's similar ish functionality in
>> virsh qemu-monitor-command. I think there's scope of a supported
>> tool here that can talk to libvirt or a UNIX socket for doing
>> QMP commands, with a friendlier syntax & pretty printing. 
>> 
>
> qmp-shell is one of my go-to tools for working through bitmap workflows
> where we don't have convenience commands yet, as some of the setups
> required for fleecing et al involve quite a number of steps.
>
> I can copy-paste raw JSON into a socket, but personally I like seeing my
> commands neatly organized in a format where I can visually reduce them
> to their components at a glance.
>
> (What I mean is: It's hard to remember which QMP commands you've barfed
> into a terminal because JSON is hard to read and looks very visually
> repetitive.)
>
> I tried to rewrite qmp-shell late last year, actually. I wanted to write
> a new REPL that was json-aware in some manner such that you could write
> multi-line commands like this:
>
>> example-command arg={
>   "hello": "world"
> }
>
> This requires, sadly, a streamable JSON parser. Most JSON parsers built
> into Python as-is simply take a file pointer and consume the entirety of
> the rest of the stream -- they don't play very nice with incomplete
> input or input that may have trailing data, e.g.:
>
>> example-command arg={
>   "hello": "world"
> } arg2={
>   "oops!": "more json!"
> }

QMP is in the same boat: it needs to process input that isn't
necessarily full expressions (JSON-text in the RFC's grammar).

Any conventional parser can be made streaming by turning it into a
coroutine.  This is probably the simplest solution for handwritten
streaming LL parsers, because it permits recursive descent.  In Python,
I'd try a generator.

Our actual solution for QMP predates coroutine support in QEMU, and is
rather hamfisted:

* Streaming lexer: it gets fed characters one at a time, and when its
  state machine says "token complete", it feeds the token to the
  "streamer".

* "Streamer": gets fed tokens one at a time, buffers them up counting
  curly and square bracket nesting until the nesting is zero, then
  passes the buffered tokens to the parser.

* Non-streaming parser: it gets fed a sequence of tokens that constitute
  a full expression.

The best I can say about this is that it works.  The streamer's token
buffer eats a lot of memory compared to a real streaming parser, but in
practice, it's a drop in the bucket.

> Also, due to the nature of JSON as being a single discrete object and
> never a stream of objects, no existing JSON parser really supports the
> idea of ever seeing more than one object per buffer.

That plainly sucks.

> ...So I investigated writing a proper grammar for qmp-shell.

Any parser must start with a proper grammar.  If it doesn't, it's a toy,
or a highway to madness.

> Unfortunately, this basically means including the JSON grammar as a
> subset of the shell grammar and writing your own parser for it entirely.

Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
paper, so JSON sticks out left and right.

Scrap and start over.

> I looked into using Python's own lexer; but it's designed to lex
> *python*, not *json*. I got a prototype lexer working for this purpose
> under a grammar that I think reflects JSON, but I got that sinking
> feeling that it was all more trouble than it was worth, and scrapped
> working on it any further.

Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
weird derivative of JSON in 239 SLOC.

> I did not find any other flex/yacc-like tools that seemed properly
> idiomatic or otherwise heavily specialized. I gave up on the idea of
> writing a new parser.

While I recommend use of tools for parsing non-trivial grammars (you'll
screw up, they won't), they're massive overkill for JSON.

> I'd love to offer a nice robust QMP shell that is available for use by
> end users, but the syntax of the shell will need some major considerations.

Scrap and start over.

[...]

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/23/20 2:19 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>>>> * scripts/qmp/qmp-shell
>>>>
>>>>   Half-hearted attempt at a human-friendly wrapper around the JSON
>>>>   syntax.  I have no use for this myself.
>>> I use this fairly often as its a useful debugging / experimentation
>>> / trouble shooting tool. There's similar ish functionality in
>>> virsh qemu-monitor-command. I think there's scope of a supported
>>> tool here that can talk to libvirt or a UNIX socket for doing
>>> QMP commands, with a friendlier syntax & pretty printing. 
>>>
>>
>> qmp-shell is one of my go-to tools for working through bitmap workflows
>> where we don't have convenience commands yet, as some of the setups
>> required for fleecing et al involve quite a number of steps.
>>
>> I can copy-paste raw JSON into a socket, but personally I like seeing my
>> commands neatly organized in a format where I can visually reduce them
>> to their components at a glance.
>>
>> (What I mean is: It's hard to remember which QMP commands you've barfed
>> into a terminal because JSON is hard to read and looks very visually
>> repetitive.)
>>
>> I tried to rewrite qmp-shell late last year, actually. I wanted to write
>> a new REPL that was json-aware in some manner such that you could write
>> multi-line commands like this:
>>
>>> example-command arg={
>>   "hello": "world"
>> }
>>
>> This requires, sadly, a streamable JSON parser. Most JSON parsers built
>> into Python as-is simply take a file pointer and consume the entirety of
>> the rest of the stream -- they don't play very nice with incomplete
>> input or input that may have trailing data, e.g.:
>>
>>> example-command arg={
>>   "hello": "world"
>> } arg2={
>>   "oops!": "more json!"
>> }
> 
> QMP is in the same boat: it needs to process input that isn't
> necessarily full expressions (JSON-text in the RFC's grammar).
> 
> Any conventional parser can be made streaming by turning it into a
> coroutine.  This is probably the simplest solution for handwritten
> streaming LL parsers, because it permits recursive descent.  In Python,
> I'd try a generator.
> 
> Our actual solution for QMP predates coroutine support in QEMU, and is
> rather hamfisted:
> 
> * Streaming lexer: it gets fed characters one at a time, and when its
>   state machine says "token complete", it feeds the token to the
>   "streamer".
> 
> * "Streamer": gets fed tokens one at a time, buffers them up counting
>   curly and square bracket nesting until the nesting is zero, then
>   passes the buffered tokens to the parser.
> 
> * Non-streaming parser: it gets fed a sequence of tokens that constitute
>   a full expression.
> 
> The best I can say about this is that it works.  The streamer's token
> buffer eats a lot of memory compared to a real streaming parser, but in
> practice, it's a drop in the bucket.
> 

I looked into this at one point. I forget why I didn't like it. I had
some notion that I should replace this one too, but forget exactly why.
Maybe it wasn't that bad, if I've forgotten.

>> Also, due to the nature of JSON as being a single discrete object and
>> never a stream of objects, no existing JSON parser really supports the
>> idea of ever seeing more than one object per buffer.
> 
> That plainly sucks.
> 
>> ...So I investigated writing a proper grammar for qmp-shell.
> 
> Any parser must start with a proper grammar.  If it doesn't, it's a toy,
> or a highway to madness.
> 
>> Unfortunately, this basically means including the JSON grammar as a
>> subset of the shell grammar and writing your own parser for it entirely.
> 
> Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
> paper, so JSON sticks out left and right.
> 
> Scrap and start over.
> 
>> I looked into using Python's own lexer; but it's designed to lex
>> *python*, not *json*. I got a prototype lexer working for this purpose
>> under a grammar that I think reflects JSON, but I got that sinking
>> feeling that it was all more trouble than it was worth, and scrapped
>> working on it any further.
> 
> Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
> weird derivative of JSON in 239 SLOC.
> 
>> I did not find any other flex/yacc-like tools that seemed properly
>> idiomatic or otherwise heavily specialized. I gave up on the idea of
>> writing a new parser.
> 
> While I recommend use of tools for parsing non-trivial grammars (you'll
> screw up, they won't), they're massive overkill for JSON.
> 
>> I'd love to offer a nice robust QMP shell that is available for use by
>> end users, but the syntax of the shell will need some major considerations.
> 
> Scrap and start over.
> 
> [...]
> 

Yes, I agree: Scrap and start over.

What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
is that it offers a convenient way to enter the top-level keys of the
arguments dict. This works absolutely fine right up until you need to
start providing nested definitions.

For the nesting, we say: "Go ahead and use JSON, but you have to take
all the spaces out."

This... works, charitably, but is hardly what I would call usable.

For the CLI, we offer a dot syntax notation that resembles nothing in
particular. It often seems the case that it isn't expressive enough to
map losslessly to JSON. I suspect it doesn't handle siblings very well.

A proper HMP-esque TUI would likely have need of coming up with its own
pet syntax for commands that avoid complicated nested JSON definitions,
but for effort:value ratio, having a QMP shorthand shell that works
arbitrarily with any command might be a better win.

Do we still have a general-case problem of how to represent QAPI
structures in plaintext? Will this need to be solved for the CLI, too?

--js

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Thu, Jan 23, 2020 at 12:58:45PM -0500, John Snow wrote:
> Yes, I agree: Scrap and start over.
> 
> What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
> is that it offers a convenient way to enter the top-level keys of the
> arguments dict. This works absolutely fine right up until you need to
> start providing nested definitions.
> 
> For the nesting, we say: "Go ahead and use JSON, but you have to take
> all the spaces out."
> 
> This... works, charitably, but is hardly what I would call usable.
> 
> For the CLI, we offer a dot syntax notation that resembles nothing in
> particular. It often seems the case that it isn't expressive enough to
> map losslessly to JSON. I suspect it doesn't handle siblings very well.
> 
> A proper HMP-esque TUI would likely have need of coming up with its own
> pet syntax for commands that avoid complicated nested JSON definitions,
> but for effort:value ratio, having a QMP shorthand shell that works
> arbitrarily with any command might be a better win.
> 
> Do we still have a general-case problem of how to represent QAPI
> structures in plaintext? Will this need to be solved for the CLI, too?

I don't know if you've ever looked at how Kubernetes/OpenShift exposes
its functionality on the command line ? I think it is interesting to
note that they largely don't try to solve this problem of flattening
JSON for humans on the CLI using their client.

Everything in their world is an object described in JSON/YAML, and
there are a small set of generic commands that can act on any type
of object. These commands primarily input and output JSON or YAML
documents directly. As a user you can pick either format since it
can do a lossless conversion in both directions server side.

So when configuring objects you'll always provide a JSON/YAML doc.
They've got some clever stuff for updating objects where you can
provide a JSON patch for only the bits which need changing.

When querying/listing objects by default it displays only a small
subset of their config information in a human friendly-ish format.
If you want to see everything then you ask for it in JSON/YAML
format. There's also an expression language that lets you extract
particular pieces of information based on requested properties,
and you can filter the list of objects based on attributes and so
on.

I think it is fair to say the structure of kubernetes object config
is on a par with hierarchical complexity of QEMU. The lack of a simple
human targetted data input format does not appear to have negatively
impacted the adoption of Kubernetes. It is worth questioning why this
is the case, while we feel the human CLI syntax for QEMU is so
critically important to QEMU's future ?



Part of it is that the machine oriented data input format via QMP
suffers from the fact that it came second in QEMU after HMP. As a
result, 90% of the documentation that illustrates QEMU will use the
human CLI syntax, varying vintages of that. Since all the docs are
focused on the HMP/CLI syntax, whenever there's a new feature we
feel pressured to expose it & document it in the human syntax too.

All this results in a situation where JSON is functionally the best
way to configure QEMU, but practically the worse, since very few
people understand how to actually use it. This is a vicious circle
holding back QMP/JSON and making the human syntax an ever greater
burden for users & maintainers


IOW, the difficulty with configuring QEMU via JSON is not the fault
of JSON itself, it is the lack of knowledge amongst users and docs,
compounded by our never ending "improvements" to the human syntax.
There are other factors too, such as our only partial coverage of
config using JSON - some is only possible via the CLI still.


I guess my point is that with a scrap & startover view point, we
should arguably completely ignore the design question of how to
flatten JSON for humans/command line, as it is the wrong problem.
Instead focus on the problem of making use of JSON the best way
to deal with QEMU both functionally and practically, for humans
and machines alike.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
> On Thu, Jan 23, 2020 at 12:58:45PM -0500, John Snow wrote:
>> Yes, I agree: Scrap and start over.
>>
>> What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
>> is that it offers a convenient way to enter the top-level keys of the
>> arguments dict. This works absolutely fine right up until you need to
>> start providing nested definitions.
>>
>> For the nesting, we say: "Go ahead and use JSON, but you have to take
>> all the spaces out."
>>
>> This... works, charitably, but is hardly what I would call usable.
>>
>> For the CLI, we offer a dot syntax notation that resembles nothing in
>> particular. It often seems the case that it isn't expressive enough to
>> map losslessly to JSON. I suspect it doesn't handle siblings very well.
>>
>> A proper HMP-esque TUI would likely have need of coming up with its own
>> pet syntax for commands that avoid complicated nested JSON definitions,
>> but for effort:value ratio, having a QMP shorthand shell that works
>> arbitrarily with any command might be a better win.
>>
>> Do we still have a general-case problem of how to represent QAPI
>> structures in plaintext? Will this need to be solved for the CLI, too?
> 
> I don't know if you've ever looked at how Kubernetes/OpenShift exposes
> its functionality on the command line ? I think it is interesting to
> note that they largely don't try to solve this problem of flattening
> JSON for humans on the CLI using their client.
> 
> Everything in their world is an object described in JSON/YAML, and
> there are a small set of generic commands that can act on any type
> of object. These commands primarily input and output JSON or YAML
> documents directly. As a user you can pick either format since it
> can do a lossless conversion in both directions server side.
> 
> So when configuring objects you'll always provide a JSON/YAML doc.
> They've got some clever stuff for updating objects where you can
> provide a JSON patch for only the bits which need changing.
> 
> When querying/listing objects by default it displays only a small
> subset of their config information in a human friendly-ish format.
> If you want to see everything then you ask for it in JSON/YAML
> format. There's also an expression language that lets you extract
> particular pieces of information based on requested properties,
> and you can filter the list of objects based on attributes and so
> on.
> 
> I think it is fair to say the structure of kubernetes object config
> is on a par with hierarchical complexity of QEMU. The lack of a simple
> human targetted data input format does not appear to have negatively
> impacted the adoption of Kubernetes. It is worth questioning why this
> is the case, while we feel the human CLI syntax for QEMU is so
> critically important to QEMU's future ?
> 

Well, if the "human CLI syntax" is "Feed it YAML documents", that's
perfectly peachy for me, too! We need a good, consistent interface.
Exactly what that interface is isn't really a blocking concern.

Configuring a VM is a complicated process and has a lot of moving
widgets. Feeding it a YAML file is a reasonable thought.

Having JSON and requiring people to type bastardized and differing
versions of it in 8 places _is_ a concern. We can't document reasonably
all of the different flavors and why they differ from one place to the
next. We can unify it. If unifying it means using
JSON/YAML/TOML/MAML[1]* everywhere and abandoning a CLI altogether,
that's just as well.

> 
> 
> Part of it is that the machine oriented data input format via QMP
> suffers from the fact that it came second in QEMU after HMP. As a
> result, 90% of the documentation that illustrates QEMU will use the
> human CLI syntax, varying vintages of that. Since all the docs are
> focused on the HMP/CLI syntax, whenever there's a new feature we
> feel pressured to expose it & document it in the human syntax too.
> 

Decent observation; but there's often no reasonable way to NOT use the
CLI, so I think it's not unreasonable that we try to expose features via
the CLI still.

However, the docs being badly out of date are a problem. We actively
lead people towards harmful / difficult to support paradigms.

> All this results in a situation where JSON is functionally the best
> way to configure QEMU, but practically the worse, since very few
> people understand how to actually use it. This is a vicious circle
> holding back QMP/JSON and making the human syntax an ever greater
> burden for users & maintainers
> 

I do basically agree.

> 
> IOW, the difficulty with configuring QEMU via JSON is not the fault
> of JSON itself, it is the lack of knowledge amongst users and docs,
> compounded by our never ending "improvements" to the human syntax.
> There are other factors too, such as our only partial coverage of
> config using JSON - some is only possible via the CLI still.
> 

I'm fine with getting rid of HMP entirely, I think. It's a weird
interface with bizarre behavior that's hard to support.

There's a few commands in there we just don't support at all, but maybe
it's time to start deprecating one-by-one any of the individual commands
that are better served by QMP these days, to send the message that HMP's
days are numbered.

Bye-bye!

As for the CLI, well, that's part of the discussion at hand...

> 
> I guess my point is that with a scrap & startover view point, we
> should arguably completely ignore the design question of how to
> flatten JSON for humans/command line, as it is the wrong problem.
> Instead focus on the problem of making use of JSON the best way
> to deal with QEMU both functionally and practically, for humans
> and machines alike.
> 

Well, sure. The context of this email was qmp-shell though, which is
meant to help facilitate the entry of JSON commands so that you *can*
indeed just forego the CLI/HMP entirely.

If you are of the opinion that every user of QEMU should be copy/pasting
JSON straight into a socket and we should delete qmp-shell, that's
certainly a fine opinion.

I'm coming from the side that I love qmp-shell; I find it useful, but it
has some syntax problems. How do I solve them? Is there a way to solve
them? QAPI is here to stay, and QAPI involves hierarchical data. That
data is usually best represented by something like json or yaml, but
those are hard to type in a shell.

What do we do about that?

> 
> Regards,
> Daniel
> 

-- 
—js

[1]*: Markus Armbruster's Markup Language; I have written this joke with
the explicit goal of tormenting Markus.

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
>> On Thu, Jan 23, 2020 at 12:58:45PM -0500, John Snow wrote:
>>> Yes, I agree: Scrap and start over.
>>>
>>> What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
>>> is that it offers a convenient way to enter the top-level keys of the
>>> arguments dict. This works absolutely fine right up until you need to
>>> start providing nested definitions.
>>>
>>> For the nesting, we say: "Go ahead and use JSON, but you have to take
>>> all the spaces out."
>>>
>>> This... works, charitably, but is hardly what I would call usable.
>>>
>>> For the CLI, we offer a dot syntax notation that resembles nothing in
>>> particular. It often seems the case that it isn't expressive enough to
>>> map losslessly to JSON. I suspect it doesn't handle siblings very well.
>>>
>>> A proper HMP-esque TUI would likely have need of coming up with its own
>>> pet syntax for commands that avoid complicated nested JSON definitions,
>>> but for effort:value ratio, having a QMP shorthand shell that works
>>> arbitrarily with any command might be a better win.
>>>
>>> Do we still have a general-case problem of how to represent QAPI
>>> structures in plaintext? Will this need to be solved for the CLI, too?
>> 
>> I don't know if you've ever looked at how Kubernetes/OpenShift exposes
>> its functionality on the command line ? I think it is interesting to

Yes, superficially.

>> note that they largely don't try to solve this problem of flattening
>> JSON for humans on the CLI using their client.

The fact that their users are happy with this proves it reasonable.

>> Everything in their world is an object described in JSON/YAML, and
>> there are a small set of generic commands that can act on any type
>> of object. These commands primarily input and output JSON or YAML
>> documents directly. As a user you can pick either format since it
>> can do a lossless conversion in both directions server side.

Like Kubernetes/OpenShift, our configuration / control language has an
abstract syntax that permits use of JSON/YAML as concrete syntax.  They
support both, we support just JSON.  We could support YAML, too.

Digression: JSON is a poor choice for configuration files.  YAML is a
complex and confusing beast (it's spec is printed 116 pages, and
yaml-0.2.2/src is ~7kSLOC).  XML is XML, 'nuff said.  TOML is much
simpler than either of the two.

>> So when configuring objects you'll always provide a JSON/YAML doc.
>> They've got some clever stuff for updating objects where you can
>> provide a JSON patch for only the bits which need changing.
>> 
>> When querying/listing objects by default it displays only a small
>> subset of their config information in a human friendly-ish format.
>> If you want to see everything then you ask for it in JSON/YAML
>> format. There's also an expression language that lets you extract
>> particular pieces of information based on requested properties,
>> and you can filter the list of objects based on attributes and so
>> on.
>> 
>> I think it is fair to say the structure of kubernetes object config
>> is on a par with hierarchical complexity of QEMU. The lack of a simple
>> human targetted data input format does not appear to have negatively
>> impacted the adoption of Kubernetes. It is worth questioning why this
>> is the case, while we feel the human CLI syntax for QEMU is so
>> critically important to QEMU's future ?

I consider human CLI syntax for QEMU a mostly solved *design* problem:
dotted keys.  It's an unsolved *implementation* problem: the CLI is a
tangled mess of almost two decades' worth of ideas, and only (some of)
the latest strands actually use dotted keys infrastructure.  The
proposed solution is CLI QAPIfication.  Gives us configuration file(s)
and introspection.

Dotted keys are merely yet another concrete syntax.  They're designed to
satisfy the CLI requirements we have, which include a measure of
compatibility to what's in the tangled mess.  They're reasonably usable
for simple stuff, but complex stuff can be too verbose to be readable.
They can't express all of the abstract syntax.  Tolerable, since they
provide an escape to JSON.  I recommend programs use the JSON escape
always.  Awkward for humans due to shell quoting.

> Well, if the "human CLI syntax" is "Feed it YAML documents", that's
> perfectly peachy for me, too! We need a good, consistent interface.
> Exactly what that interface is isn't really a blocking concern.

Right.

> Configuring a VM is a complicated process and has a lot of moving
> widgets. Feeding it a YAML file is a reasonable thought.

We've grown used to configuring QEMU with gargantuan command lines.
Déformation profesionelle.

> Having JSON and requiring people to type bastardized and differing
> versions of it in 8 places _is_ a concern. We can't document reasonably
> all of the different flavors and why they differ from one place to the
> next.

Yes.

>       We can unify it. If unifying it means using
> JSON/YAML/TOML/MAML[1]* everywhere and abandoning a CLI altogether,
> that's just as well.

I see no need to argue about whether to keep dotted keys human CLI
syntax or throw it away.

>> Part of it is that the machine oriented data input format via QMP
>> suffers from the fact that it came second in QEMU after HMP. As a
>> result, 90% of the documentation that illustrates QEMU will use the
>> human CLI syntax, varying vintages of that. Since all the docs are
>> focused on the HMP/CLI syntax, whenever there's a new feature we
>> feel pressured to expose it & document it in the human syntax too.
>> 
>
> Decent observation; but there's often no reasonable way to NOT use the
> CLI, so I think it's not unreasonable that we try to expose features via
> the CLI still.
>
> However, the docs being badly out of date are a problem. We actively
> lead people towards harmful / difficult to support paradigms.
>
>> All this results in a situation where JSON is functionally the best
>> way to configure QEMU, but practically the worse, since very few
>> people understand how to actually use it. This is a vicious circle
>> holding back QMP/JSON and making the human syntax an ever greater
>> burden for users & maintainers
>> 
>
> I do basically agree.

What to do about it?

Here's an idea that hasn't been mentioned in this thread: a -writeconfig
that actually works.  After you configured QEMU in whatever ways you
rustled up on the 'net, you can -writeconfig the resulting command line
into a *modern* configuration file.

>> IOW, the difficulty with configuring QEMU via JSON is not the fault
>> of JSON itself, it is the lack of knowledge amongst users and docs,
>> compounded by our never ending "improvements" to the human syntax.
>> There are other factors too, such as our only partial coverage of
>> config using JSON - some is only possible via the CLI still.
>
> I'm fine with getting rid of HMP entirely, I think. It's a weird
> interface with bizarre behavior that's hard to support.
>
> There's a few commands in there we just don't support at all, but maybe
> it's time to start deprecating one-by-one any of the individual commands
> that are better served by QMP these days, to send the message that HMP's
> days are numbered.
>
> Bye-bye!

Experience tells that no matter how weird and bizarre a feature is, once
you attempt to remove it, it *will* find champions willing to fight for
it to the death.  I'm not trying to tell you "don't go there", only "if
you go there, go armed and prepared".

> As for the CLI, well, that's part of the discussion at hand...
>
>> 
>> I guess my point is that with a scrap & startover view point, we
>> should arguably completely ignore the design question of how to
>> flatten JSON for humans/command line, as it is the wrong problem.
>> Instead focus on the problem of making use of JSON the best way
>> to deal with QEMU both functionally and practically, for humans
>> and machines alike.
>> 
>
> Well, sure. The context of this email was qmp-shell though, which is
> meant to help facilitate the entry of JSON commands so that you *can*
> indeed just forego the CLI/HMP entirely.
>
> If you are of the opinion that every user of QEMU should be copy/pasting
> JSON straight into a socket and we should delete qmp-shell, that's
> certainly a fine opinion.
>
> I'm coming from the side that I love qmp-shell; I find it useful, but it
> has some syntax problems. How do I solve them? Is there a way to solve
> them? QAPI is here to stay, and QAPI involves hierarchical data. That
> data is usually best represented by something like json or yaml, but
> those are hard to type in a shell.
>
> What do we do about that?

I acknowledge that pain in dealing with JSON without suitable tooling.
Worse when the wrong tooling gets in the way, e.g. shell quoting.

Dotted keys are differently awkward.

Daniel's point is (1) we've got bigger fish to fry, and (2) Kubernetes
has proven punting the complex part to JSON/YAML text files (where you
do have suitable tooling) is good enough.

He's right, and that's why I've aggressively ignored qmp-shell since
forever.

If you want to invest some cycles into reducing the pain, I recommend
investing in suitable tooling, not in creating yet another language,
which then needs suitable tooling.

If I wanted to invest, I'd consider creating an Emacs mode to talk to
QMP.  But I don't; see "bigger fish".

> [1]*: Markus Armbruster's Markup Language; I have written this joke with
> the explicit goal of tormenting Markus.

Ha!  Let's call it MarML for "Markus's Markup Language", spoken like
"Marmel", which means "marble" in some dialects of German.  Then we can
joke about me having lost my marbles in specifying my own markup
language.

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Fri, Jan 24, 2020 at 08:59:41AM +0100, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
> > On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
> >> So when configuring objects you'll always provide a JSON/YAML doc.
> >> They've got some clever stuff for updating objects where you can
> >> provide a JSON patch for only the bits which need changing.
> >> 
> >> When querying/listing objects by default it displays only a small
> >> subset of their config information in a human friendly-ish format.
> >> If you want to see everything then you ask for it in JSON/YAML
> >> format. There's also an expression language that lets you extract
> >> particular pieces of information based on requested properties,
> >> and you can filter the list of objects based on attributes and so
> >> on.
> >> 
> >> I think it is fair to say the structure of kubernetes object config
> >> is on a par with hierarchical complexity of QEMU. The lack of a simple
> >> human targetted data input format does not appear to have negatively
> >> impacted the adoption of Kubernetes. It is worth questioning why this
> >> is the case, while we feel the human CLI syntax for QEMU is so
> >> critically important to QEMU's future ?
> 
> I consider human CLI syntax for QEMU a mostly solved *design* problem:
> dotted keys.  It's an unsolved *implementation* problem: the CLI is a
> tangled mess of almost two decades' worth of ideas, and only (some of)
> the latest strands actually use dotted keys infrastructure.  The
> proposed solution is CLI QAPIfication.  Gives us configuration file(s)
> and introspection.
> 
> Dotted keys are merely yet another concrete syntax.  They're designed to
> satisfy the CLI requirements we have, which include a measure of
> compatibility to what's in the tangled mess.  They're reasonably usable
> for simple stuff, but complex stuff can be too verbose to be readable.
> They can't express all of the abstract syntax.  Tolerable, since they
> provide an escape to JSON.  I recommend programs use the JSON escape
> always.  Awkward for humans due to shell quoting.

I agree that the dotted key syntax is our chosen / solved design
for expressing JSON on the CLI. I would also say that, in retrospect,
this was a incorrect design decision that is one of the key things
responsible for QEMU having a bad reputation for complexity.

We should simply never have tried to invent a way to map the full
hiearchy of JSON onto the CLI as the result will always be unpleasant.
The dotted notation is the most verbose way to do this type of
configuration, because of the string repetition it requires for
nested structures.

Lets consider how libvirt uses blockdev for a LUKS volume stored
in iSCSI

  $ qemu-system-x86_64 \
  -object secret,id=libvirt-5-storage-secret0,\
    data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
    keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
  -object secret,id=libvirt-5-format-luks-secret0,\
    data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
    keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
  -blockdev '{"driver":"iscsi","portal":"example.org:6000",\
    "target":"iqn.1992-01.com.example:storage","lun":1,"transport":"tcp",\
    "user":"myname","password-secret":"libvirt-5-storage-secret0",\
    "node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}' \
  -blockdev '{"node-name":"libvirt-5-format","read-only":false,"driver":"qcow2",\
    "encrypt":{"format":"luks","key-secret":"libvirt-5-format-luks-secret0"},\
    "file":"libvirt-5-storage"}' \

We all know JSON is horrible on the CLI, no surprise. So

Lets use human "friendly" dotted syntax instead:

  $ qemu-system-x86_64 \
  -object secret,id=libvirt-5-storage-secret0,\
    data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
    keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
  -object secret,id=libvirt-5-format-luks-secret0,\
    data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
    keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
  -blockdev driver=qcow2,node-name=libvirt-5-format,read-only=false,\
    encrypt.format=luks,encrypt.key-secret=libvirt-5-format-luks-secret0,\
    file.driver=iscsi,file.portal=example.org:6000,\
    file.target=iqn.1992-01.com.example:storage,file.lun=1,file.transport=tcp,\
    file.user=myname,file.password-secret=libvirt-6-storage-secret0,\
    file.node-name=libvirt-5-storage,file.auto-read-only=true,file.dicard=unmap

I don't think that's much of an improvement, aside from not having
to worry about matching "}".

If we move to JSON in a config file

  $ cat qemu.json
  {
    "arguments": [
      {
        "arg": "object",
        "data": {
          "type": "secret",
          "id":"libvirt-5-storage-secret0",
          "data": "9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
          "keyid": "masterKey0",
          "iv": "AAECAwQFBgcICQoLDA0ODw==",
          "format": "base64"
        }
      },
      {
        "arg": "object",
        "data": {
          "type": "secret",
          "id":"libvirt-5-format-luks-secret0",
          "data": "9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
          "keyid": "masterKey0",
          "iv": "AAECAwQFBgcICQoLDA0ODw==",
          "format": "base64"
        }
      },
      {
        "arg": "blockdev",
        "data": {
          "node-name":"libvirt-5-format",
          "read-only":false,"driver":"qcow2",
          "encrypt":{
            "format":"luks","key-secret":
            "libvirt-5-format-luks-secret0"
          },
          "file":{
            "driver": "iscsi",
            "portal": "example.org:6000",
            "target":"iqn.1992-01.com.example:storage",
            "lun": 1,
            "transport": "tcp",
            "user": "myname",
            "password-secret": "libvirt-5-storage-secret0",
            "node-name":"libvirt-5-storage",
            "auto-read-only":"true",
            "discard":"unmap"
          }
        }
      }
    ]
  }
  $ qemu-system-x86_64 -f qemu.json

The config file is more volumous than the CLI, but it is also
massively more intelligible to humans because you can see the
structure of the data.

I still screwed up many times with missing quotes, incorrect
commas, etc. All the fun of JSON

So if we allowed YAML instead of JSON, now we get...

  $ cat qemu.yaml
  ---
  arguments:
  - arg: object
    data:
      type: secret
      id: libvirt-5-storage-secret0
      data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
      keyid: masterKey0
      iv: AAECAwQFBgcICQoLDA0ODw==
      format: base64
  - arg: object
    data:
      type: secret
      id: libvirt-5-format-luks-secret0
      data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
      keyid: masterKey0
      iv: AAECAwQFBgcICQoLDA0ODw==
      format: base64
  - arg: blockdev
    data:
      node-name: libvirt-5-format
      read-only: false
      driver: qcow2
      encrypt:
        format: luks
        key-secret: libvirt-5-format-luks-secret0
      file:
        driver: iscsi
        portal: example.org:6000
        target: iqn.1992-01.com.example:storage
        lun: 1
        transport: tcp
        user: myname
        password-secret: libvirt-5-storage-secret0
        node-name: libvirt-5-storage
        auto-read-only: true
        discard: unmap
  $ qemu-system-x86_64 -f qemu.yaml

This is finally something I'd consider to be on a par with the
original QEMU syntax, before we added hierarchical data. You
have the minimal possible amount of syntax here. No commas,
no quotes, no curly brackets, etc.

I'm sure there are things about YAML that are not nice, but
at some point there's a tradeoff where we can declare it
"good enough". You mention that it has a huge design spec,
but does that really matter ? We shouldn't be implemneting
a parser or formatter ourselves, just re-use one that
already exists. What matters more is whether it is nice
for humans & machines, and I think it is a good syntax
in both those cases.


> > However, the docs being badly out of date are a problem. We actively
> > lead people towards harmful / difficult to support paradigms.
> >
> >> All this results in a situation where JSON is functionally the best
> >> way to configure QEMU, but practically the worse, since very few
> >> people understand how to actually use it. This is a vicious circle
> >> holding back QMP/JSON and making the human syntax an ever greater
> >> burden for users & maintainers
> >> 
> >
> > I do basically agree.
> 
> What to do about it?
> 
> Here's an idea that hasn't been mentioned in this thread: a -writeconfig
> that actually works.  After you configured QEMU in whatever ways you
> rustled up on the 'net, you can -writeconfig the resulting command line
> into a *modern* configuration file.

No matter what we do implementation / designwise, the pre-requisite for
all of them is that we're able to fully express the configuration of the
VM using QAPI. As a side effect this means that -loadconfig and -writeconfig
become usable in the real world, which is nice for the current users of
qemu-system-XXX.  Per my reply to John though, I think we could consider
being more ambituous & seek to obslete qemu-system-XXX entirely at some
point in the future.

> >> IOW, the difficulty with configuring QEMU via JSON is not the fault
> >> of JSON itself, it is the lack of knowledge amongst users and docs,
> >> compounded by our never ending "improvements" to the human syntax.
> >> There are other factors too, such as our only partial coverage of
> >> config using JSON - some is only possible via the CLI still.
> >
> > I'm fine with getting rid of HMP entirely, I think. It's a weird
> > interface with bizarre behavior that's hard to support.
> >
> > There's a few commands in there we just don't support at all, but maybe
> > it's time to start deprecating one-by-one any of the individual commands
> > that are better served by QMP these days, to send the message that HMP's
> > days are numbered.
> >
> > Bye-bye!
> 
> Experience tells that no matter how weird and bizarre a feature is, once
> you attempt to remove it, it *will* find champions willing to fight for
> it to the death.  I'm not trying to tell you "don't go there", only "if
> you go there, go armed and prepared".

Yeah and it gets worse the older / more mature a project is. This
is what pushes projects into doing a clean break "2.0" design,
which is something we've never done for QEMU.  I don't mean just
breaking backcompat in a handful of CLI args here. I mean a proper
scorched earth, everything is up for re-evaluation, "2.0".


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
> On Fri, Jan 24, 2020 at 08:59:41AM +0100, Markus Armbruster wrote:
> > John Snow <jsnow@redhat.com> writes:
> > 
> > > On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
> > >> So when configuring objects you'll always provide a JSON/YAML doc.
> > >> They've got some clever stuff for updating objects where you can
> > >> provide a JSON patch for only the bits which need changing.
> > >> 
> > >> When querying/listing objects by default it displays only a small
> > >> subset of their config information in a human friendly-ish format.
> > >> If you want to see everything then you ask for it in JSON/YAML
> > >> format. There's also an expression language that lets you extract
> > >> particular pieces of information based on requested properties,
> > >> and you can filter the list of objects based on attributes and so
> > >> on.
> > >> 
> > >> I think it is fair to say the structure of kubernetes object config
> > >> is on a par with hierarchical complexity of QEMU. The lack of a simple
> > >> human targetted data input format does not appear to have negatively
> > >> impacted the adoption of Kubernetes. It is worth questioning why this
> > >> is the case, while we feel the human CLI syntax for QEMU is so
> > >> critically important to QEMU's future ?
> > 
> > I consider human CLI syntax for QEMU a mostly solved *design* problem:
> > dotted keys.  It's an unsolved *implementation* problem: the CLI is a
> > tangled mess of almost two decades' worth of ideas, and only (some of)
> > the latest strands actually use dotted keys infrastructure.  The
> > proposed solution is CLI QAPIfication.  Gives us configuration file(s)
> > and introspection.
> > 
> > Dotted keys are merely yet another concrete syntax.  They're designed to
> > satisfy the CLI requirements we have, which include a measure of
> > compatibility to what's in the tangled mess.  They're reasonably usable
> > for simple stuff, but complex stuff can be too verbose to be readable.
> > They can't express all of the abstract syntax.  Tolerable, since they
> > provide an escape to JSON.  I recommend programs use the JSON escape
> > always.  Awkward for humans due to shell quoting.
> 
> I agree that the dotted key syntax is our chosen / solved design
> for expressing JSON on the CLI. I would also say that, in retrospect,
> this was a incorrect design decision that is one of the key things
> responsible for QEMU having a bad reputation for complexity.

I doubt this. Whenever I get a bug report with a command line created by
libvirt, the command line is huge, but basically nothing in it uses
dotted syntax. Yes, you may have cache.direct=on in it somewhere, but
that's not actual nesting.

The problem is the amount of options that is specified by management
tools, and then humans are looking at it and feel it's way too complex.

Command lines written by human users are usually much simpler because
they just use QEMU's defaults instead of explicitly specifying
everything.

> We should simply never have tried to invent a way to map the full
> hiearchy of JSON onto the CLI as the result will always be unpleasant.
> The dotted notation is the most verbose way to do this type of
> configuration, because of the string repetition it requires for
> nested structures.

True. I would have liked a different syntax that used some kind of
brackets (at least optionally), but Markus didn't like adding another
character that must be escaped.

I think if we want, we can still evolve our human syntax to be more user
friendly. My impression was that we don't want to.

> Lets consider how libvirt uses blockdev for a LUKS volume stored
> in iSCSI
> 
>   $ qemu-system-x86_64 \
>   -object secret,id=libvirt-5-storage-secret0,\
>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>   -object secret,id=libvirt-5-format-luks-secret0,\
>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>   -blockdev '{"driver":"iscsi","portal":"example.org:6000",\
>     "target":"iqn.1992-01.com.example:storage","lun":1,"transport":"tcp",\
>     "user":"myname","password-secret":"libvirt-5-storage-secret0",\
>     "node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}' \
>   -blockdev '{"node-name":"libvirt-5-format","read-only":false,"driver":"qcow2",\
>     "encrypt":{"format":"luks","key-secret":"libvirt-5-format-luks-secret0"},\
>     "file":"libvirt-5-storage"}' \
> 
> We all know JSON is horrible on the CLI, no surprise. So
> 
> Lets use human "friendly" dotted syntax instead:
> 
>   $ qemu-system-x86_64 \
>   -object secret,id=libvirt-5-storage-secret0,\
>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>   -object secret,id=libvirt-5-format-luks-secret0,\
>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>   -blockdev driver=qcow2,node-name=libvirt-5-format,read-only=false,\
>     encrypt.format=luks,encrypt.key-secret=libvirt-5-format-luks-secret0,\
>     file.driver=iscsi,file.portal=example.org:6000,\
>     file.target=iqn.1992-01.com.example:storage,file.lun=1,file.transport=tcp,\
>     file.user=myname,file.password-secret=libvirt-6-storage-secret0,\
>     file.node-name=libvirt-5-storage,file.auto-read-only=true,file.dicard=unmap
> 
> I don't think that's much of an improvement, aside from not having
> to worry about matching "}".

I see you merged the two -blockdev arguments into a single one in order
to get at least some repetition with the file.* prefixes. ;-)

> If we move to JSON in a config file
> 
>   $ cat qemu.json
>   {
>     "arguments": [
>       {
>         "arg": "object",
>         "data": {
>           "type": "secret",
>           "id":"libvirt-5-storage-secret0",
>           "data": "9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
>           "keyid": "masterKey0",
>           "iv": "AAECAwQFBgcICQoLDA0ODw==",
>           "format": "base64"
>         }
>       },
>       {
>         "arg": "object",
>         "data": {
>           "type": "secret",
>           "id":"libvirt-5-format-luks-secret0",
>           "data": "9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
>           "keyid": "masterKey0",
>           "iv": "AAECAwQFBgcICQoLDA0ODw==",
>           "format": "base64"
>         }
>       },
>       {
>         "arg": "blockdev",
>         "data": {
>           "node-name":"libvirt-5-format",
>           "read-only":false,"driver":"qcow2",
>           "encrypt":{
>             "format":"luks","key-secret":
>             "libvirt-5-format-luks-secret0"
>           },
>           "file":{
>             "driver": "iscsi",
>             "portal": "example.org:6000",
>             "target":"iqn.1992-01.com.example:storage",
>             "lun": 1,
>             "transport": "tcp",
>             "user": "myname",
>             "password-secret": "libvirt-5-storage-secret0",
>             "node-name":"libvirt-5-storage",
>             "auto-read-only":"true",
>             "discard":"unmap"
>           }
>         }
>       }
>     ]
>   }
>   $ qemu-system-x86_64 -f qemu.json
> 
> The config file is more volumous than the CLI, but it is also
> massively more intelligible to humans because you can see the
> structure of the data.
> 
> I still screwed up many times with missing quotes, incorrect
> commas, etc. All the fun of JSON

JSON is usually easy enough to read, quite reasonable for making minor
modifications, and a PITA for writing something (like a QMP request)
from scratch.

> So if we allowed YAML instead of JSON, now we get...
> 
>   $ cat qemu.yaml
>   ---
>   arguments:
>   - arg: object
>     data:
>       type: secret
>       id: libvirt-5-storage-secret0
>       data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
>       keyid: masterKey0
>       iv: AAECAwQFBgcICQoLDA0ODw==
>       format: base64
>   - arg: object
>     data:
>       type: secret
>       id: libvirt-5-format-luks-secret0
>       data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
>       keyid: masterKey0
>       iv: AAECAwQFBgcICQoLDA0ODw==
>       format: base64
>   - arg: blockdev
>     data:
>       node-name: libvirt-5-format
>       read-only: false
>       driver: qcow2
>       encrypt:
>         format: luks
>         key-secret: libvirt-5-format-luks-secret0
>       file:
>         driver: iscsi
>         portal: example.org:6000
>         target: iqn.1992-01.com.example:storage
>         lun: 1
>         transport: tcp
>         user: myname
>         password-secret: libvirt-5-storage-secret0
>         node-name: libvirt-5-storage
>         auto-read-only: true
>         discard: unmap
>   $ qemu-system-x86_64 -f qemu.yaml
> 
> This is finally something I'd consider to be on a par with the
> original QEMU syntax, before we added hierarchical data. You
> have the minimal possible amount of syntax here. No commas,
> no quotes, no curly brackets, etc.

This seems to have the same problems as the QEMU command line (how do
you distinguish strings from ints, from bools, from null?). It's
basically just a pretty-printed version of it with the consequence that
it needs to be stored in an external file and there is no reasonable way
to keep it in my shell history.

> > >> IOW, the difficulty with configuring QEMU via JSON is not the fault
> > >> of JSON itself, it is the lack of knowledge amongst users and docs,
> > >> compounded by our never ending "improvements" to the human syntax.
> > >> There are other factors too, such as our only partial coverage of
> > >> config using JSON - some is only possible via the CLI still.
> > >
> > > I'm fine with getting rid of HMP entirely, I think. It's a weird
> > > interface with bizarre behavior that's hard to support.
> > >
> > > There's a few commands in there we just don't support at all, but maybe
> > > it's time to start deprecating one-by-one any of the individual commands
> > > that are better served by QMP these days, to send the message that HMP's
> > > days are numbered.
> > >
> > > Bye-bye!
> > 
> > Experience tells that no matter how weird and bizarre a feature is, once
> > you attempt to remove it, it *will* find champions willing to fight for
> > it to the death.

Hi! :-)

Well, not fighting for it to the death and I'm certainly not married to
its exact syntax, but I use HMP all the time because it's convenient to
use manually and QMP isn't.

If you want to remove HMP, get us a decent QMP shell first. And ideally
a way how to use it without manually configuring a QMP socket and
starting up and connecting the shell to it with ten kilometers of
command line options on both sides first. HMP is available by default,
and so should its replacement be.

> > I'm not trying to tell you "don't go there", only "if
> > you go there, go armed and prepared".
> 
> Yeah and it gets worse the older / more mature a project is. This
> is what pushes projects into doing a clean break "2.0" design,
> which is something we've never done for QEMU.  I don't mean just
> breaking backcompat in a handful of CLI args here. I mean a proper
> scorched earth, everything is up for re-evaluation, "2.0".

As a developer I would love this because so many things in the current
design are wrong. As a user I would hate it because this kind of "2.0"
release tends to break lots of features I like. (Anyone remember KDE 4?)

Kevin

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 9:38 AM, Kevin Wolf wrote:
> Hi! :-)
> 
> Well, not fighting for it to the death and I'm certainly not married to
> its exact syntax, but I use HMP all the time because it's convenient to
> use manually and QMP isn't.
> 
> If you want to remove HMP, get us a decent QMP shell first. And ideally
> a way how to use it without manually configuring a QMP socket and
> starting up and connecting the shell to it with ten kilometers of
> command line options on both sides first. HMP is available by default,
> and so should its replacement be.

This is partly why I am asking about what a "qmp-shell" should look like
to be usable, so we can sunset HMP once and for all.

One idea for a qmp-shell is to implement some of the convenience syntax
in qmp-shell directly to give us some of the same shorthands, and that
logic lives in the shell now.

--js

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* John Snow (jsnow@redhat.com) wrote:
> 
> 
> On 1/24/20 9:38 AM, Kevin Wolf wrote:
> > Hi! :-)
> > 
> > Well, not fighting for it to the death and I'm certainly not married to
> > its exact syntax, but I use HMP all the time because it's convenient to
> > use manually and QMP isn't.
> > 
> > If you want to remove HMP, get us a decent QMP shell first. And ideally
> > a way how to use it without manually configuring a QMP socket and
> > starting up and connecting the shell to it with ten kilometers of
> > command line options on both sides first. HMP is available by default,
> > and so should its replacement be.
> 
> This is partly why I am asking about what a "qmp-shell" should look like
> to be usable, so we can sunset HMP once and for all.
> 
> One idea for a qmp-shell is to implement some of the convenience syntax
> in qmp-shell directly to give us some of the same shorthands, and that
> logic lives in the shell now.

I wonder about generating a set of python bindings for the qmp commands,
and generating wrappers for those, and having the qmp-shell be a python
interpreter with all that loaded.
That way it would be very easy to add extra sugar.

However, I caution that too many people think that QMP has all the type
of diagnostic stuff people want - it's just way too hard and beuracratic
to add a simple diagnostic command to QMP, when you just want to add
something to print some diagnostics out for your corner of qemu.

Dave

> --js
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 1:30 PM, Dr. David Alan Gilbert wrote:
> However, I caution that too many people think that QMP has all the type
> of diagnostic stuff people want - it's just way too hard and beuracratic
> to add a simple diagnostic command to QMP, when you just want to add
> something to print some diagnostics out for your corner of qemu.

I'm not sure I follow, do you mean bureaucratic in contrast to... HMP?

--js

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* John Snow (jsnow@redhat.com) wrote:
> 
> 
> On 1/24/20 1:30 PM, Dr. David Alan Gilbert wrote:
> > However, I caution that too many people think that QMP has all the type
> > of diagnostic stuff people want - it's just way too hard and beuracratic
> > to add a simple diagnostic command to QMP, when you just want to add
> > something to print some diagnostics out for your corner of qemu.
> 
> I'm not sure I follow, do you mean bureaucratic in contrast to... HMP?

I mean, someone who adds a QMP command is expected to be a lot more
careful about the long term stability and proper structures etc etc.
Where for HMP, if someone wants to add a command that's useful to the
diagnostics of their part of QEMU and doesn't break anything else, then
fine.

Dave

> --js
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 1:52 PM, Dr. David Alan Gilbert wrote:
> * John Snow (jsnow@redhat.com) wrote:
>>
>>
>> On 1/24/20 1:30 PM, Dr. David Alan Gilbert wrote:
>>> However, I caution that too many people think that QMP has all the type
>>> of diagnostic stuff people want - it's just way too hard and beuracratic
>>> to add a simple diagnostic command to QMP, when you just want to add
>>> something to print some diagnostics out for your corner of qemu.
>>
>> I'm not sure I follow, do you mean bureaucratic in contrast to... HMP?
> 
> I mean, someone who adds a QMP command is expected to be a lot more
> careful about the long term stability and proper structures etc etc.
> Where for HMP, if someone wants to add a command that's useful to the
> diagnostics of their part of QEMU and doesn't break anything else, then
> fine.
> 
> Dave

It sounds like we want a formalized notion of "x-debug-" for QMP
commands where we use feature flags to explicitly mark a command
interface as a debugging interface.

(We use such a command for bitmap iotests, to compute checksums of
bitmaps to ensure they migrate correctly. We make no guarantee that this
is stable or useful.)

I'm not sure if it's wise or not, but we could disable these commands
entirely unless you configure with --enable-qmp-debugging.

Might ease that pain of migrating everything onto QMP.

--js

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Kevin Wolf <kwolf@redhat.com> writes:

> Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
>> On Fri, Jan 24, 2020 at 08:59:41AM +0100, Markus Armbruster wrote:
>> > John Snow <jsnow@redhat.com> writes:
>> > 
>> > > On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
>> > >> So when configuring objects you'll always provide a JSON/YAML doc.
>> > >> They've got some clever stuff for updating objects where you can
>> > >> provide a JSON patch for only the bits which need changing.
>> > >> 
>> > >> When querying/listing objects by default it displays only a small
>> > >> subset of their config information in a human friendly-ish format.
>> > >> If you want to see everything then you ask for it in JSON/YAML
>> > >> format. There's also an expression language that lets you extract
>> > >> particular pieces of information based on requested properties,
>> > >> and you can filter the list of objects based on attributes and so
>> > >> on.
>> > >> 
>> > >> I think it is fair to say the structure of kubernetes object config
>> > >> is on a par with hierarchical complexity of QEMU. The lack of a simple
>> > >> human targetted data input format does not appear to have negatively
>> > >> impacted the adoption of Kubernetes. It is worth questioning why this
>> > >> is the case, while we feel the human CLI syntax for QEMU is so
>> > >> critically important to QEMU's future ?
>> > 
>> > I consider human CLI syntax for QEMU a mostly solved *design* problem:
>> > dotted keys.  It's an unsolved *implementation* problem: the CLI is a
>> > tangled mess of almost two decades' worth of ideas, and only (some of)
>> > the latest strands actually use dotted keys infrastructure.  The
>> > proposed solution is CLI QAPIfication.  Gives us configuration file(s)
>> > and introspection.
>> > 
>> > Dotted keys are merely yet another concrete syntax.  They're designed to
>> > satisfy the CLI requirements we have, which include a measure of
>> > compatibility to what's in the tangled mess.  They're reasonably usable
>> > for simple stuff, but complex stuff can be too verbose to be readable.
>> > They can't express all of the abstract syntax.  Tolerable, since they
>> > provide an escape to JSON.  I recommend programs use the JSON escape
>> > always.  Awkward for humans due to shell quoting.
>> 
>> I agree that the dotted key syntax is our chosen / solved design
>> for expressing JSON on the CLI. I would also say that, in retrospect,
>> this was a incorrect design decision that is one of the key things
>> responsible for QEMU having a bad reputation for complexity.
>
> I doubt this. Whenever I get a bug report with a command line created by
> libvirt, the command line is huge, but basically nothing in it uses
> dotted syntax. Yes, you may have cache.direct=on in it somewhere, but
> that's not actual nesting.
>
> The problem is the amount of options that is specified by management
> tools, and then humans are looking at it and feel it's way too complex.
>
> Command lines written by human users are usually much simpler because
> they just use QEMU's defaults instead of explicitly specifying
> everything.

Yes, machine-generated configuration is more verbose than what humans
produce.  Machines like it explicit.  It's simpler for them than relying
on defaults.  Bonus: immunity to changing defaults.

But no, verbosity is not the core problem, it merely aggravates the core
problem.  Complex configuration is much harder to read in a CLI syntax
than in a half-decent config file.  Yes, it's prohibitively harder just
for lengthy configurations.  That doesn't make it not harder for short
configurations.

>> We should simply never have tried to invent a way to map the full
>> hiearchy of JSON onto the CLI as the result will always be unpleasant.

It's what I had to do to secure a beach head for QAPI on the command
line coast.

>> The dotted notation is the most verbose way to do this type of
>> configuration, because of the string repetition it requires for
>> nested structures.
>
> True. I would have liked a different syntax that used some kind of
> brackets (at least optionally), but Markus didn't like adding another
> character that must be escaped.

Design thread:

    Subject: Non-flat command line option argument syntax
    Date: Thu, 02 Feb 2017 20:42:33 +0100
    Message-ID: <87bmukmlau.fsf@dusky.pond.sub.org>
    https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg00555.html

A few quick quotes:

* On JSON vs. dotted keys in CLI: "both variants are basically
  illegible.  This is simply something that belongs into a config file
  rather than the command line.  In a config file, JSON would be a
  better choice."

* On dotted keys vs. structured values: "dotted keys are weird and ugly,
  but at least they don't add to the quoting mess.  Structured values
  look better, except when they do add to the quoting mess.  I'm having
  a hard time deciding which one I like less :)"

* Final verdict: "the whole non-flat command line argument design has
  been an exercise in picking what I dislike least."

> I think if we want, we can still evolve our human syntax to be more user
> friendly. My impression was that we don't want to.

Correct.

Quote from the design thread on disambiguating scalars, but I feel the
gist of it applies more widely:

    * When it breaks, fall back to JSON

      Tolerable if it breaks pretty much only in obscure corner cases.

    * More syntax

      Add syntax to disambiguate the type (key sigils?).  Has to be optional
      for backward compatibility reasons.  When omitting type information
      breaks, you have to supply it, or fall back to JSON.

      More syntax for use in obscure cases is about the last thing the QEMU
      command line needs.

    * More magic

      Make the keyval variant of the QObject input visitor shift the
      breakage to hopefully less common cases.

      - Empty list magic

        When the visitor is asked for a list, and the list's key wasn't
        specified, return an empty list instead of failure.  Unbreaks empty
        list, breaks absent optional list.

      - Alternate magic

        When the visitor is asked for an alternate, pick the alternate's
        variant based on the value rather than the type for scalar values
        (the type is always 'str' then).  Unbreaks alternates when this
        picks the variant you want, breaks them when you really want 'str'.

      - Possibly more

      When the magic breaks, fall back to JSON.

      More magic in the QEMU command line feels even worse to me than more
      syntax.

>> Lets consider how libvirt uses blockdev for a LUKS volume stored
>> in iSCSI
>> 
>>   $ qemu-system-x86_64 \
>>   -object secret,id=libvirt-5-storage-secret0,\
>>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>>   -object secret,id=libvirt-5-format-luks-secret0,\
>>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>>   -blockdev '{"driver":"iscsi","portal":"example.org:6000",\
>>     "target":"iqn.1992-01.com.example:storage","lun":1,"transport":"tcp",\
>>     "user":"myname","password-secret":"libvirt-5-storage-secret0",\
>>     "node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}' \
>>   -blockdev '{"node-name":"libvirt-5-format","read-only":false,"driver":"qcow2",\
>>     "encrypt":{"format":"luks","key-secret":"libvirt-5-format-luks-secret0"},\
>>     "file":"libvirt-5-storage"}' \
>> 
>> We all know JSON is horrible on the CLI, no surprise. So
>> 
>> Lets use human "friendly" dotted syntax instead:
>> 
>>   $ qemu-system-x86_64 \
>>   -object secret,id=libvirt-5-storage-secret0,\
>>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>>   -object secret,id=libvirt-5-format-luks-secret0,\
>>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>>   -blockdev driver=qcow2,node-name=libvirt-5-format,read-only=false,\
>>     encrypt.format=luks,encrypt.key-secret=libvirt-5-format-luks-secret0,\
>>     file.driver=iscsi,file.portal=example.org:6000,\
>>     file.target=iqn.1992-01.com.example:storage,file.lun=1,file.transport=tcp,\
>>     file.user=myname,file.password-secret=libvirt-6-storage-secret0,\
>>     file.node-name=libvirt-5-storage,file.auto-read-only=true,file.dicard=unmap
>> 
>> I don't think that's much of an improvement, aside from not having
>> to worry about matching "}".
>
> I see you merged the two -blockdev arguments into a single one in order
> to get at least some repetition with the file.* prefixes. ;-)
>
>> If we move to JSON in a config file
>> 
>>   $ cat qemu.json
>>   {
>>     "arguments": [
>>       {
>>         "arg": "object",
>>         "data": {
>>           "type": "secret",
>>           "id":"libvirt-5-storage-secret0",
>>           "data": "9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
>>           "keyid": "masterKey0",
>>           "iv": "AAECAwQFBgcICQoLDA0ODw==",
>>           "format": "base64"
>>         }
>>       },
>>       {
>>         "arg": "object",
>>         "data": {
>>           "type": "secret",
>>           "id":"libvirt-5-format-luks-secret0",
>>           "data": "9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
>>           "keyid": "masterKey0",
>>           "iv": "AAECAwQFBgcICQoLDA0ODw==",
>>           "format": "base64"
>>         }
>>       },
>>       {
>>         "arg": "blockdev",
>>         "data": {
>>           "node-name":"libvirt-5-format",
>>           "read-only":false,"driver":"qcow2",
>>           "encrypt":{
>>             "format":"luks","key-secret":
>>             "libvirt-5-format-luks-secret0"
>>           },
>>           "file":{
>>             "driver": "iscsi",
>>             "portal": "example.org:6000",
>>             "target":"iqn.1992-01.com.example:storage",
>>             "lun": 1,
>>             "transport": "tcp",
>>             "user": "myname",
>>             "password-secret": "libvirt-5-storage-secret0",
>>             "node-name":"libvirt-5-storage",
>>             "auto-read-only":"true",
>>             "discard":"unmap"
>>           }
>>         }
>>       }
>>     ]
>>   }
>>   $ qemu-system-x86_64 -f qemu.json
>> 
>> The config file is more volumous than the CLI, but it is also
>> massively more intelligible to humans because you can see the
>> structure of the data.
>> 
>> I still screwed up many times with missing quotes, incorrect
>> commas, etc. All the fun of JSON
>
> JSON is usually easy enough to read, quite reasonable for making minor
> modifications, and a PITA for writing something (like a QMP request)
> from scratch.

JSON is by design a data interchange format.  We (and others) pressed it
into a configuration file role.  PITA ensues.  Crockford shrugs.

Suitable tooling can reduce it.

>> So if we allowed YAML instead of JSON, now we get...
>> 
>>   $ cat qemu.yaml
>>   ---
>>   arguments:
>>   - arg: object
>>     data:
>>       type: secret
>>       id: libvirt-5-storage-secret0
>>       data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
>>       keyid: masterKey0
>>       iv: AAECAwQFBgcICQoLDA0ODw==
>>       format: base64
>>   - arg: object
>>     data:
>>       type: secret
>>       id: libvirt-5-format-luks-secret0
>>       data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
>>       keyid: masterKey0
>>       iv: AAECAwQFBgcICQoLDA0ODw==
>>       format: base64
>>   - arg: blockdev
>>     data:
>>       node-name: libvirt-5-format
>>       read-only: false
>>       driver: qcow2
>>       encrypt:
>>         format: luks
>>         key-secret: libvirt-5-format-luks-secret0
>>       file:
>>         driver: iscsi
>>         portal: example.org:6000
>>         target: iqn.1992-01.com.example:storage
>>         lun: 1
>>         transport: tcp
>>         user: myname
>>         password-secret: libvirt-5-storage-secret0
>>         node-name: libvirt-5-storage
>>         auto-read-only: true
>>         discard: unmap
>>   $ qemu-system-x86_64 -f qemu.yaml
>> 
>> This is finally something I'd consider to be on a par with the
>> original QEMU syntax, before we added hierarchical data. You
>> have the minimal possible amount of syntax here. No commas,
>> no quotes, no curly brackets, etc.
>
> This seems to have the same problems as the QEMU command line (how do
> you distinguish strings from ints, from bools, from null?).

True: YAML provides only string scalars.

TOML provides strings, integers, floats, booleans, and several flavors
of time.  It lacks null.

>                                                             It's
> basically just a pretty-printed version of it with the consequence that
> it needs to be stored in an external file and there is no reasonable way
> to keep it in my shell history.

There is a reasonable way to keep it in my file system, though.  I find
that decidedly superior.

[...]

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Sat, Jan 25, 2020 at 11:18:41AM +0100, Markus Armbruster wrote:
> Kevin Wolf <kwolf@redhat.com> writes:
> 
> > Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
> >> So if we allowed YAML instead of JSON, now we get...
> >> 
> >>   $ cat qemu.yaml
> >>   ---
> >>   arguments:
> >>   - arg: object
> >>     data:
> >>       type: secret
> >>       id: libvirt-5-storage-secret0
> >>       data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
> >>       keyid: masterKey0
> >>       iv: AAECAwQFBgcICQoLDA0ODw==
> >>       format: base64
> >>   - arg: object
> >>     data:
> >>       type: secret
> >>       id: libvirt-5-format-luks-secret0
> >>       data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
> >>       keyid: masterKey0
> >>       iv: AAECAwQFBgcICQoLDA0ODw==
> >>       format: base64
> >>   - arg: blockdev
> >>     data:
> >>       node-name: libvirt-5-format
> >>       read-only: false
> >>       driver: qcow2
> >>       encrypt:
> >>         format: luks
> >>         key-secret: libvirt-5-format-luks-secret0
> >>       file:
> >>         driver: iscsi
> >>         portal: example.org:6000
> >>         target: iqn.1992-01.com.example:storage
> >>         lun: 1
> >>         transport: tcp
> >>         user: myname
> >>         password-secret: libvirt-5-storage-secret0
> >>         node-name: libvirt-5-storage
> >>         auto-read-only: true
> >>         discard: unmap
> >>   $ qemu-system-x86_64 -f qemu.yaml
> >> 
> >> This is finally something I'd consider to be on a par with the
> >> original QEMU syntax, before we added hierarchical data. You
> >> have the minimal possible amount of syntax here. No commas,
> >> no quotes, no curly brackets, etc.
> >
> > This seems to have the same problems as the QEMU command line (how do
> > you distinguish strings from ints, from bools, from null?).
> 
> True: YAML provides only string scalars.

Actually, YAML provides  strings, integers, booleans & nulls, the same
level of typing at JSON, but with added benefit of supporting comments
officially:

  https://rollout.io/blog/yaml-tutorial-everything-you-need-get-started/

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Sat, Jan 25, 2020 at 11:18:41AM +0100, Markus Armbruster wrote:
>> Kevin Wolf <kwolf@redhat.com> writes:
>> 
>> > Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
>> >> So if we allowed YAML instead of JSON, now we get...
>> >> 
>> >>   $ cat qemu.yaml
[...]
>> >> This is finally something I'd consider to be on a par with the
>> >> original QEMU syntax, before we added hierarchical data. You
>> >> have the minimal possible amount of syntax here. No commas,
>> >> no quotes, no curly brackets, etc.
>> >
>> > This seems to have the same problems as the QEMU command line (how do
>> > you distinguish strings from ints, from bools, from null?).
>> 
>> True: YAML provides only string scalars.
>
> Actually, YAML provides  strings, integers, booleans & nulls, the same
> level of typing at JSON, but with added benefit of supporting comments
> officially:
>
>   https://rollout.io/blog/yaml-tutorial-everything-you-need-get-started/

You're right.  I read section 2.3 Scalars of the YAML spec[*], and
neglected to study sections 2.4. Tags and 10.3. Core Schema.


[*] https://yaml.org/spec/1.2/spec.html

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 25.01.2020 um 11:18 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
> 
> > Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
> >> This is finally something I'd consider to be on a par with the
> >> original QEMU syntax, before we added hierarchical data. You
> >> have the minimal possible amount of syntax here. No commas,
> >> no quotes, no curly brackets, etc.
> >
> > This seems to have the same problems as the QEMU command line (how do
> > you distinguish strings from ints, from bools, from null?).
> 
> True: YAML provides only string scalars.
> 
> TOML provides strings, integers, floats, booleans, and several flavors
> of time.  It lacks null.
> 
> >                                                             It's
> > basically just a pretty-printed version of it with the consequence that
> > it needs to be stored in an external file and there is no reasonable way
> > to keep it in my shell history.
> 
> There is a reasonable way to keep it in my file system, though.  I find
> that decidedly superior.

That depends a lot on your use case.

If you have a long-lived production VM that you always run with the same
configuration, then yes, having a config file for it in the file system
is what you probably want. Currently, for this case, people directly
using QEMU tend to write a script that contains the command line. I
think I do have such scripts somewhere, but their number is very small.

My common case is short-lived VMs with configurations that change very
often between QEMU invocations. Here the command line is decidedly
superior.

Requiring me to create a file in the file system each time and to
remember deleting it after I'm done feels about as convenient as a *nix
shell that doesn't accept parameters for commands on the command line,
but instead requires you to write a one-off script first and then run
that.

Kevin

Re: Making QEMU easier for management tools and applications

[Peter Maydell]

On Mon, 27 Jan 2020 at 11:56, Kevin Wolf <kwolf@redhat.com> wrote:
> If you have a long-lived production VM that you always run with the same
> configuration, then yes, having a config file for it in the file system
> is what you probably want. Currently, for this case, people directly
> using QEMU tend to write a script that contains the command line. I
> think I do have such scripts somewhere, but their number is very small.

I have some similar scripts, which I use for launching one-off
"run and then kill soon" VMs, mostly as test setups. The
advantage of a script is that you get an actual programming
language and can do things like "substitute in the name of
the directory the script lives in" when setting up parameters
that are filenames, or easily support "this is a default, and
you can override it with an environment variable". So I'd
still need to have a script even if the script changed from
"run QEMU with these command line options" to "create a temp
file with this config and run QEMU on it".

thanks
-- PMM

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/27/20 6:56 AM, Kevin Wolf wrote:
> Am 25.01.2020 um 11:18 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>>
>>> Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
>>>> This is finally something I'd consider to be on a par with the
>>>> original QEMU syntax, before we added hierarchical data. You
>>>> have the minimal possible amount of syntax here. No commas,
>>>> no quotes, no curly brackets, etc.
>>>
>>> This seems to have the same problems as the QEMU command line (how do
>>> you distinguish strings from ints, from bools, from null?).
>>
>> True: YAML provides only string scalars.
>>
>> TOML provides strings, integers, floats, booleans, and several flavors
>> of time.  It lacks null.
>>
>>>                                                             It's
>>> basically just a pretty-printed version of it with the consequence that
>>> it needs to be stored in an external file and there is no reasonable way
>>> to keep it in my shell history.
>>
>> There is a reasonable way to keep it in my file system, though.  I find
>> that decidedly superior.
> 
> That depends a lot on your use case.
> 
> If you have a long-lived production VM that you always run with the same
> configuration, then yes, having a config file for it in the file system
> is what you probably want. Currently, for this case, people directly
> using QEMU tend to write a script that contains the command line. I
> think I do have such scripts somewhere, but their number is very small.
> 
> My common case is short-lived VMs with configurations that change very
> often between QEMU invocations. Here the command line is decidedly
> superior.
> 
> Requiring me to create a file in the file system each time and to
> remember deleting it after I'm done feels about as convenient as a *nix
> shell that doesn't accept parameters for commands on the command line,
> but instead requires you to write a one-off script first and then run
> that.
> 
> Kevin
> 

> ./qemu-core <<EOF
{
    "machine": "Q35",
    "memory": "2GiB",
    "accel": "kvm"
}
EOF

No file required, cooperates with readline, avoids crunchy,
hard-to-maintain CLI syntax. Directly and easily translatable to a
stored-file configuration. All configuration and documentation is
centralized via QAPI.

A little worse to type manually, yes. Maybe not bad /enough/ for me to
want to rescue the CLI which prevents full QAPI-fication and a working
configuration file.

Arguably, a well documented configuration schema will be much easier to
browse, discover, and use than a labyrinthine CLI with many stub
definitions whose options are not exposed in the documentation.

(The argument here is: It's a little harder and a little longer to type,
but the benefits from the schema organization may improve productivity
of using QEMU directly instead of harming it.)

--js

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1695 bytes --]

Il lun 27 gen 2020, 21:11 John Snow <jsnow@redhat.com> ha scritto:

>
> > ./qemu-core <<EOF
> {
>     "machine": "Q35",
>     "memory": "2GiB",
>     "accel": "kvm"
> }
> EOF
>

And now you have to keep all the syntactic sugar that is in vl.c. I don't
think a redesign of -readconfig should accept anything less verbose than

- machine:
    type: q35
    ram:
       type: memory-backend-hostmem
       size: 2G
- accel:
  - type: kvm

And this is not even taking into account disks.

I like the idea of refactoring all the syntactic sugar into a pre-pass on
command line options. This is not an entirely new idea, see
https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.

I am afraid that this thread is derailing a bit, with lots of pipe dreams
and no actionable items. How do we get it back in track?

Paolo


> No file required, cooperates with readline, avoids crunchy,
> hard-to-maintain CLI syntax. Directly and easily translatable to a
> stored-file configuration. All configuration and documentation is
> centralized via QAPI.
>
> A little worse to type manually, yes. Maybe not bad /enough/ for me to
> want to rescue the CLI which prevents full QAPI-fication and a working
> configuration file.
>
> Arguably, a well documented configuration schema will be much easier to
> browse, discover, and use than a labyrinthine CLI with many stub
> definitions whose options are not exposed in the documentation.
>
> (The argument here is: It's a little harder and a little longer to type,
> but the benefits from the schema organization may improve productivity
> of using QEMU directly instead of harming it.)
>
> --js
>
>

[-- Attachment #2: Type: text/html, Size: 2796 bytes --]

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/27/20 5:38 PM, Paolo Bonzini wrote:
> 
> 
> Il lun 27 gen 2020, 21:11 John Snow <jsnow@redhat.com
> <mailto:jsnow@redhat.com>> ha scritto:
> 
> 
>     > ./qemu-core <<EOF
>     {
>         "machine": "Q35",
>         "memory": "2GiB",
>         "accel": "kvm"
>     }
>     EOF
> 
> 
> And now you have to keep all the syntactic sugar that is in vl.c. I

Didn't mean to suggest literal values, I just grabbed some from the top
of my head. Let's pretend I didn't use sugared ones. This is a
distraction from the point.

The point is: you can specify JSON on the command line, as stdin to QEMU
as a here document. No config file needed.

> don't think a redesign of -readconfig should accept anything less
> verbose than
> 
> - machine:
>     type: q35
>     ram:
>        type: memory-backend-hostmem
>        size: 2G
> - accel:
>   - type: kvm
> 
> And this is not even taking into account disks.
> 

Though, point taken, even minimal configurations might get too cumbersome.

> I like the idea of refactoring all the syntactic sugar into a pre-pass
> on command line options. This is not an entirely new idea,
> see https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.

This could be very good, as long as the transformation was knowable in
some way. (Documented, modeled in the schema somehow, observable
post-transformation, whichever.)

(wow, 2010.)

> 
> I am afraid that this thread is derailing a bit, with lots of pipe
> dreams and no actionable items. How do we get it back in track?
> 

That depends on what your opinion of "on track" is. If there are
specific angles you find interesting, it would be easiest to start a
thread around the framing you find productive.

--js

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Mon, Jan 27, 2020 at 11:38:49PM +0100, Paolo Bonzini wrote:
> Il lun 27 gen 2020, 21:11 John Snow <jsnow@redhat.com> ha scritto:
> 
> >
> > > ./qemu-core <<EOF
> > {
> >     "machine": "Q35",
> >     "memory": "2GiB",
> >     "accel": "kvm"
> > }
> > EOF
> >
> 
> And now you have to keep all the syntactic sugar that is in vl.c. I don't
> think a redesign of -readconfig should accept anything less verbose than
> 
> - machine:
>     type: q35
>     ram:
>        type: memory-backend-hostmem
>        size: 2G
> - accel:
>   - type: kvm
> 
> And this is not even taking into account disks.
> 
> I like the idea of refactoring all the syntactic sugar into a pre-pass on
> command line options. This is not an entirely new idea, see
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.
> 
> I am afraid that this thread is derailing a bit, with lots of pipe dreams
> and no actionable items. How do we get it back in track?

To me the one thing that is clear. No matter what approach we want to
take to QEMU configuration/interaction/CLI/etc, one critical bit of
work is a pre-requisite...

...we must finish[1] the QAPI modelling of QEMU's features in some
short, finite timeframe. We can't let it drag on for another 5 years
or more. I'd say we need a timeframe that is 2 years max, preferrably
1 year.

I don't think we can achieve this by leaving the task up to to the
QAPI maintainers alone. It is unreasonable to put such a burden to
on a small number of people to both implement & review it all. We
need to consider it a project wide priority item so that we can get
broader involvement across all maintainers, in closing the gaps.

I'm not sure if we have any clear list of where our known gaps exist ?

If not a good first action point is do make such a list, so we know
what we're trying to attack.

Regards,
Daniel

[1] For a definition of "finish" that may or may not be equal to
    100% of all possible features. It might be valid to declare
    we only care about "finishing" QAPI for features used by
    libvirt, since those represent the "common case" usage.
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 28.01.2020 um 11:16 hat Daniel P. Berrangé geschrieben:
> On Mon, Jan 27, 2020 at 11:38:49PM +0100, Paolo Bonzini wrote:
> > Il lun 27 gen 2020, 21:11 John Snow <jsnow@redhat.com> ha scritto:
> > 
> > >
> > > > ./qemu-core <<EOF
> > > {
> > >     "machine": "Q35",
> > >     "memory": "2GiB",
> > >     "accel": "kvm"
> > > }
> > > EOF
> > >
> > 
> > And now you have to keep all the syntactic sugar that is in vl.c. I don't
> > think a redesign of -readconfig should accept anything less verbose than
> > 
> > - machine:
> >     type: q35
> >     ram:
> >        type: memory-backend-hostmem
> >        size: 2G
> > - accel:
> >   - type: kvm
> > 
> > And this is not even taking into account disks.
> > 
> > I like the idea of refactoring all the syntactic sugar into a pre-pass on
> > command line options. This is not an entirely new idea, see
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.
> > 
> > I am afraid that this thread is derailing a bit, with lots of pipe dreams
> > and no actionable items. How do we get it back in track?
> 
> To me the one thing that is clear. No matter what approach we want to
> take to QEMU configuration/interaction/CLI/etc, one critical bit of
> work is a pre-requisite...
> 
> ...we must finish[1] the QAPI modelling of QEMU's features in some
> short, finite timeframe. We can't let it drag on for another 5 years
> or more. I'd say we need a timeframe that is 2 years max, preferrably
> 1 year.
> 
> I don't think we can achieve this by leaving the task up to to the
> QAPI maintainers alone. It is unreasonable to put such a burden to
> on a small number of people to both implement & review it all. We
> need to consider it a project wide priority item so that we can get
> broader involvement across all maintainers, in closing the gaps.
> 
> I'm not sure if we have any clear list of where our known gaps exist ?

I don't know about a full list, but I've been discussing command line
QAPIfication a bit with Markus recently because we had the idea of using
qemu-storage-daemon as a guinea pig for it.

The big one seems to be QOM (and qdev). object-add and device-add are
both not defined in terms of QAPI. One of them uses an "any" type (which
results in QObjects with arbitrary content being passed), the other one
"gen": false (which avoids generating anything from the schema).

I know that some more options exist that have unusual syntax and are
hard to convert to QAPI while maintaining command line compatibility.
Maybe that should be solved by just designing new options and
deprecating the old ones.

Kevin

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Cc: Eric for netdev_add QAPIfication.

Kevin Wolf <kwolf@redhat.com> writes:

> Am 28.01.2020 um 11:16 hat Daniel P. Berrangé geschrieben:
>> On Mon, Jan 27, 2020 at 11:38:49PM +0100, Paolo Bonzini wrote:
>> > Il lun 27 gen 2020, 21:11 John Snow <jsnow@redhat.com> ha scritto:
>> > 
>> > >
>> > > > ./qemu-core <<EOF
>> > > {
>> > >     "machine": "Q35",
>> > >     "memory": "2GiB",
>> > >     "accel": "kvm"
>> > > }
>> > > EOF
>> > >
>> > 
>> > And now you have to keep all the syntactic sugar that is in vl.c. I don't
>> > think a redesign of -readconfig should accept anything less verbose than
>> > 
>> > - machine:
>> >     type: q35
>> >     ram:
>> >        type: memory-backend-hostmem
>> >        size: 2G
>> > - accel:
>> >   - type: kvm
>> > 
>> > And this is not even taking into account disks.
>> > 
>> > I like the idea of refactoring all the syntactic sugar into a pre-pass on
>> > command line options. This is not an entirely new idea, see
>> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.
>> > 
>> > I am afraid that this thread is derailing a bit, with lots of pipe dreams
>> > and no actionable items. How do we get it back in track?
>> 
>> To me the one thing that is clear. No matter what approach we want to
>> take to QEMU configuration/interaction/CLI/etc, one critical bit of
>> work is a pre-requisite...
>> 
>> ...we must finish[1] the QAPI modelling of QEMU's features in some
>> short, finite timeframe. We can't let it drag on for another 5 years
>> or more. I'd say we need a timeframe that is 2 years max, preferrably
>> 1 year.
>> 
>> I don't think we can achieve this by leaving the task up to to the
>> QAPI maintainers alone. It is unreasonable to put such a burden to
>> on a small number of people to both implement & review it all. We
>> need to consider it a project wide priority item so that we can get
>> broader involvement across all maintainers, in closing the gaps.

Thank you, Daniel.  More on it below.

>> I'm not sure if we have any clear list of where our known gaps exist ?
>
> I don't know about a full list, but I've been discussing command line
> QAPIfication a bit with Markus recently because we had the idea of using
> qemu-storage-daemon as a guinea pig for it.

I still like that idea.  We can explore a 100% QAPIfied CLI there with
minimal disruption elsewhere, and without CLI compatibility concerns.
Constraints due to the shared QAPI schema remain, unless we freely
duplicate stuff, which would probably be a bad idea.

> The big one seems to be QOM (and qdev). object-add and device-add are
> both not defined in terms of QAPI. One of them uses an "any" type (which
> results in QObjects with arbitrary content being passed), the other one
> "gen": false (which avoids generating anything from the schema).

These are the known "cheats" in QMP.  There's also netdev_add, but Eric
has patches to QAPIfy it properly.  Eric, I hope you can dust them off.

For CLI, we have numerous options to QAPIfy.  Some of them are trivial.
Others involve QAPIfying substantial code behind them: I don't want a
QAPIfied option to immediately stuff everything into QemuOpts for
outmoded internal interfaces.  Such shortcuts is what got us into the
netdev_add mess.

QAPIfying internal interfaces is one of the areas where we QAPI guys
will need help.

> I know that some more options exist that have unusual syntax and are
> hard to convert to QAPI while maintaining command line compatibility.

Weird semantics can also get in the way.  For instance, when we replaced
-drive with -blockdev, we used the opportunity to ditch block backend
auto-deletion.

> Maybe that should be solved by just designing new options and
> deprecating the old ones.

Re: Making QEMU easier for management tools and applications

[Eric Blake]

On 1/28/20 9:36 AM, Markus Armbruster wrote:
> Cc: Eric for netdev_add QAPIfication.
> 

> These are the known "cheats" in QMP.  There's also netdev_add, but Eric
> has patches to QAPIfy it properly.  Eric, I hope you can dust them off.

Yes, I'll move that up on my todo list.

When I originally did the work, we hesitated because we did not have a 
deprecation policy, and the QAPIfied version did not accept both string 
and int the way the old code did. I don't know if we want to do a hard 
break or start a deprecation clock now, but that can be decided as part 
of dusting off the series.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 27.01.2020 um 21:11 hat John Snow geschrieben:
> On 1/27/20 6:56 AM, Kevin Wolf wrote:
> > Am 25.01.2020 um 11:18 hat Markus Armbruster geschrieben:
> >> Kevin Wolf <kwolf@redhat.com> writes:
> >>> basically just a pretty-printed version of it with the consequence that
> >>> it needs to be stored in an external file and there is no reasonable way
> >>> to keep it in my shell history.
> >>
> >> There is a reasonable way to keep it in my file system, though.  I find
> >> that decidedly superior.
> > 
> > That depends a lot on your use case.
> > 
> > If you have a long-lived production VM that you always run with the same
> > configuration, then yes, having a config file for it in the file system
> > is what you probably want. Currently, for this case, people directly
> > using QEMU tend to write a script that contains the command line. I
> > think I do have such scripts somewhere, but their number is very small.
> > 
> > My common case is short-lived VMs with configurations that change very
> > often between QEMU invocations. Here the command line is decidedly
> > superior.
> > 
> > Requiring me to create a file in the file system each time and to
> > remember deleting it after I'm done feels about as convenient as a *nix
> > shell that doesn't accept parameters for commands on the command line,
> > but instead requires you to write a one-off script first and then run
> > that.
> > 
> > Kevin
> > 
> 
> > ./qemu-core <<EOF
> {
>     "machine": "Q35",
>     "memory": "2GiB",
>     "accel": "kvm"
> }
> EOF

I'm not sure why everybody is trying to come up with crude workarounds
when all I said is that keeping things in a file is not always
"decidedly superior".

Of course, I would have points to make about that workaround (like that
I often want to use stdin for an interactive monitor, or that I don't
see how this is superior to --machine Q35 --memory 2GiB --accel kvm),
but that's all beside the point.

> No file required, cooperates with readline, avoids crunchy,
> hard-to-maintain CLI syntax. Directly and easily translatable to a
> stored-file configuration. All configuration and documentation is
> centralized via QAPI.
> 
> A little worse to type manually, yes. Maybe not bad /enough/ for me to
> want to rescue the CLI which prevents full QAPI-fication and a working
> configuration file.
> 
> Arguably, a well documented configuration schema will be much easier to
> browse, discover, and use than a labyrinthine CLI with many stub
> definitions whose options are not exposed in the documentation.
> 
> (The argument here is: It's a little harder and a little longer to type,
> but the benefits from the schema organization may improve productivity
> of using QEMU directly instead of harming it.)

I think this is a false dichotomy.

You can have everything defined by the schema and properly documented
and still have a non-JSON command line. Translating the QAPI schema to
a command line option is a solved problem, this is exactly how
-blockdev works.

The unsolved part is how to compatibly convert the existing options. If
you're willing to sacrifice compatibility, great. Then we can just
define stuff in the QAPI schema and still keep a command line syntax
that is usable for humans. The code for mapping a QAPI type to the
argument of an option is basically already there.

The only question is "is compatibility important"? If the answer is no,
then we'll be there in no time.

Kevin

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Kevin Wolf <kwolf@redhat.com> writes:

> Am 27.01.2020 um 21:11 hat John Snow geschrieben:
[...]
>> (The argument here is: It's a little harder and a little longer to type,
>> but the benefits from the schema organization may improve productivity
>> of using QEMU directly instead of harming it.)
>
> I think this is a false dichotomy.
>
> You can have everything defined by the schema and properly documented
> and still have a non-JSON command line. Translating the QAPI schema to
> a command line option is a solved problem, this is exactly how
> -blockdev works.
>
> The unsolved part is how to compatibly convert the existing options. If
> you're willing to sacrifice compatibility, great. Then we can just
> define stuff in the QAPI schema and still keep a command line syntax
> that is usable for humans. The code for mapping a QAPI type to the
> argument of an option is basically already there.

Correct.

Solving that problem took time, but that's sunk cost now.

> The only question is "is compatibility important"? If the answer is no,
> then we'll be there in no time.

I doubt we'll be there in no time, but certainly much sooner than if we
have to grapple with compatibility to a byzantine CLI nobody truly
understands.

There's one known issue caused by having "a non-JSON command line"
(actually: dotted keys as sugar for JSON): pressure to reduce nesting.

Consider chardev-add.  Example:

    {"execute": "chardev-add",
     "arguments": {"id": "bar",
                   "backend": {"type": "file",
                               "data": {"out": "/tmp/bar.log"}}}}

The arguments as dotted keys:

    id=bar,backend.type=file,backend.data.out=/tmp/bar.log

Observe there's quite some of nesting.  While that's somewhat cumbersome
in JSON, it's a lot worse with dotted keys, because there nesting means
repeated key prefixes.  I could give much worse examples, actually.

We'd rather have something like

    id=bar,type=file,out=/tmp/bar.log

Back to JSON:

    "arguments": {"id": "bar", "type": "file", "out": "/tmp/bar.log"}

QAPI can do this, but it uses feature that predate chardev-add.

We don't want to duplicate the chardev-add schema in modern, flattened
form for the CLI.

So the compatibility problem actually shifts to QMP: can we evolve the
existing QMP command compatibly at a reasonable cost in design, coding
and complexity to support flat arguments?

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 28.01.2020 um 13:36 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
> 
> > Am 27.01.2020 um 21:11 hat John Snow geschrieben:
> [...]
> >> (The argument here is: It's a little harder and a little longer to type,
> >> but the benefits from the schema organization may improve productivity
> >> of using QEMU directly instead of harming it.)
> >
> > I think this is a false dichotomy.
> >
> > You can have everything defined by the schema and properly documented
> > and still have a non-JSON command line. Translating the QAPI schema to
> > a command line option is a solved problem, this is exactly how
> > -blockdev works.
> >
> > The unsolved part is how to compatibly convert the existing options. If
> > you're willing to sacrifice compatibility, great. Then we can just
> > define stuff in the QAPI schema and still keep a command line syntax
> > that is usable for humans. The code for mapping a QAPI type to the
> > argument of an option is basically already there.
> 
> Correct.
> 
> Solving that problem took time, but that's sunk cost now.
> 
> > The only question is "is compatibility important"? If the answer is no,
> > then we'll be there in no time.
> 
> I doubt we'll be there in no time, but certainly much sooner than if we
> have to grapple with compatibility to a byzantine CLI nobody truly
> understands.
> 
> There's one known issue caused by having "a non-JSON command line"
> (actually: dotted keys as sugar for JSON): pressure to reduce nesting.
> 
> Consider chardev-add.  Example:
> 
>     {"execute": "chardev-add",
>      "arguments": {"id": "bar",
>                    "backend": {"type": "file",
>                                "data": {"out": "/tmp/bar.log"}}}}
> 
> The arguments as dotted keys:
> 
>     id=bar,backend.type=file,backend.data.out=/tmp/bar.log
> 
> Observe there's quite some of nesting.  While that's somewhat cumbersome
> in JSON, it's a lot worse with dotted keys, because there nesting means
> repeated key prefixes.  I could give much worse examples, actually.

This is true, but even without the repeated keys (e.g. in a syntax that
would use brackets), it would still be unnecessarily verbose and
probably hard to remember:

    id=bar,backend={type=file,data={out=/tmp/bar.log}}

> We'd rather have something like
> 
>     id=bar,type=file,out=/tmp/bar.log
> 
> Back to JSON:
> 
>     "arguments": {"id": "bar", "type": "file", "out": "/tmp/bar.log"}
> 
> QAPI can do this, but it uses feature that predate chardev-add.
> 
> We don't want to duplicate the chardev-add schema in modern, flattened
> form for the CLI.
> 
> So the compatibility problem actually shifts to QMP: can we evolve the
> existing QMP command compatibly at a reasonable cost in design, coding
> and complexity to support flat arguments?

Well, first of all: Do we need compatibility? If we don't, then we can
just make the change.

Much of this threads plays with the though that maybe we don't need any
compatibility and make the radical conclusion that we don't need any
human-friendly interface at all. Keeping full compatibility is the other
extreme.

There might be some middle ground where we break compatibility where the
old way can't easily be maintained with the new infrastructure, but
don't give up on the idea of being used by humans.

Kevin

Re: Making QEMU easier for management tools and applications

[Gerd Hoffmann]

> > Consider chardev-add.  Example:
> > 
> >     {"execute": "chardev-add",
> >      "arguments": {"id": "bar",
> >                    "backend": {"type": "file",
> >                                "data": {"out": "/tmp/bar.log"}}}}

> Well, first of all: Do we need compatibility?

Well, libvirt uses that, so yes.  Or at least support both old and new
(chardev-add-v2?) in parallel for while to give libvirt time to switch
over.

cheers,
  Gerd

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Kevin Wolf <kwolf@redhat.com> writes:

> Am 28.01.2020 um 13:36 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>> 
>> > Am 27.01.2020 um 21:11 hat John Snow geschrieben:
>> [...]
>> >> (The argument here is: It's a little harder and a little longer to type,
>> >> but the benefits from the schema organization may improve productivity
>> >> of using QEMU directly instead of harming it.)
>> >
>> > I think this is a false dichotomy.
>> >
>> > You can have everything defined by the schema and properly documented
>> > and still have a non-JSON command line. Translating the QAPI schema to
>> > a command line option is a solved problem, this is exactly how
>> > -blockdev works.
>> >
>> > The unsolved part is how to compatibly convert the existing options. If
>> > you're willing to sacrifice compatibility, great. Then we can just
>> > define stuff in the QAPI schema and still keep a command line syntax
>> > that is usable for humans. The code for mapping a QAPI type to the
>> > argument of an option is basically already there.
>> 
>> Correct.
>> 
>> Solving that problem took time, but that's sunk cost now.
>> 
>> > The only question is "is compatibility important"? If the answer is no,
>> > then we'll be there in no time.
>> 
>> I doubt we'll be there in no time, but certainly much sooner than if we
>> have to grapple with compatibility to a byzantine CLI nobody truly
>> understands.
>> 
>> There's one known issue caused by having "a non-JSON command line"
>> (actually: dotted keys as sugar for JSON): pressure to reduce nesting.
>> 
>> Consider chardev-add.  Example:
>> 
>>     {"execute": "chardev-add",
>>      "arguments": {"id": "bar",
>>                    "backend": {"type": "file",
>>                                "data": {"out": "/tmp/bar.log"}}}}
>> 
>> The arguments as dotted keys:
>> 
>>     id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>> 
>> Observe there's quite some of nesting.  While that's somewhat cumbersome
>> in JSON, it's a lot worse with dotted keys, because there nesting means
>> repeated key prefixes.  I could give much worse examples, actually.
>
> This is true, but even without the repeated keys (e.g. in a syntax that
> would use brackets), it would still be unnecessarily verbose and
> probably hard to remember:
>
>     id=bar,backend={type=file,data={out=/tmp/bar.log}}

No argument.  It's unnecessarily verbose in JSON, too.

>> We'd rather have something like
>> 
>>     id=bar,type=file,out=/tmp/bar.log
>> 
>> Back to JSON:
>> 
>>     "arguments": {"id": "bar", "type": "file", "out": "/tmp/bar.log"}
>> 
>> QAPI can do this, but it uses feature that predate chardev-add.
>> 
>> We don't want to duplicate the chardev-add schema in modern, flattened
>> form for the CLI.
>> 
>> So the compatibility problem actually shifts to QMP: can we evolve the
>> existing QMP command compatibly at a reasonable cost in design, coding
>> and complexity to support flat arguments?
>
> Well, first of all: Do we need compatibility? If we don't, then we can
> just make the change.

The trouble with flattening this one is QMP, where we promise stability.

> Much of this threads plays with the though that maybe we don't need any
> compatibility and make the radical conclusion that we don't need any
> human-friendly interface at all. Keeping full compatibility is the other
> extreme.
>
> There might be some middle ground where we break compatibility where the
> old way can't easily be maintained with the new infrastructure, but
> don't give up on the idea of being used by humans.

I'm not sure the connection between maintaining compatibility and
supporting human use is as strong as you seem to imply.

As far as I can tell, the "maybe we don't need any compatibility"
discussion is about the CLI.  I'd rephrase it as "maybe we need a
machine-friendly CLI on par with QMP more than we need compatibility to
the current CLI".

"We don't need any human-friendly interface at all" comes in not because
machine-friendly necessarily precludes human-friendly, but only if we're
unwilling (unable?) to do the extra work for it.

Compare the monitor:

* QMP is primarily for machines.  We promise stability: no incompatible
  changes without clear communicaton of intent and a grace period.  We
  provide machine clients tools to deal with the interface evolution,
  e.g. query-qmp-schema.

* HMP is exclusively for humans.  It may change at any time.

For the CLI, we don't have such a separation, and our offerings for
dealing with interface evolution are wholly inadequate.  We *need* to do
better for machines.

Now, the monitor also informs us about the cost of providing a
completely separate interface for humans.

Elsewhere in this thread, we discussed layering (a replacement for) HMP
on top of QMP cleanly, possibly in a separate process, possibly written
in a high-level language like Python.

HMP predates QMP.  We reworked it so the HMP commands are implemented on
top of the QMP commands, or at least on top of common helpers.  But this
is not quite the same as layering HMP on top of QMP.

If we decide to radically break the CLI so we can start over, we get to
decide whether and how to do a human-friendly CLI, in particular how it
relates to the machine-friendly CLI.

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 31/01/20 07:50, Markus Armbruster wrote:
>>> Consider chardev-add.  Example:
>>>
>>>     {"execute": "chardev-add",
>>>      "arguments": {"id": "bar",
>>>                    "backend": {"type": "file",
>>>                                "data": {"out": "/tmp/bar.log"}}}}
>>>
>>> The arguments as dotted keys:
>>>
>>>     id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>>>
>>> Observe there's quite some of nesting.  While that's somewhat cumbersome
>>> in JSON, it's a lot worse with dotted keys, because there nesting means
>>> repeated key prefixes.  I could give much worse examples, actually.
>> This is true, but even without the repeated keys (e.g. in a syntax that
>> would use brackets), it would still be unnecessarily verbose and
>> probably hard to remember:
>>
>>     id=bar,backend={type=file,data={out=/tmp/bar.log}}
> No argument.  It's unnecessarily verbose in JSON, too.
> 

I think we should be able to switch chardevs to -object/object_add these
days.  Not right now, but it may be possible.  Introducing a warning
when chardev and object ids conflict would be a start.

Paolo

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 31/01/20 07:50, Markus Armbruster wrote:
>>>> Consider chardev-add.  Example:
>>>>
>>>>     {"execute": "chardev-add",
>>>>      "arguments": {"id": "bar",
>>>>                    "backend": {"type": "file",
>>>>                                "data": {"out": "/tmp/bar.log"}}}}
>>>>
>>>> The arguments as dotted keys:
>>>>
>>>>     id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>>>>
>>>> Observe there's quite some of nesting.  While that's somewhat cumbersome
>>>> in JSON, it's a lot worse with dotted keys, because there nesting means
>>>> repeated key prefixes.  I could give much worse examples, actually.
>>> This is true, but even without the repeated keys (e.g. in a syntax that
>>> would use brackets), it would still be unnecessarily verbose and
>>> probably hard to remember:
>>>
>>>     id=bar,backend={type=file,data={out=/tmp/bar.log}}
>> No argument.  It's unnecessarily verbose in JSON, too.
>> 
>
> I think we should be able to switch chardevs to -object/object_add these
> days.  Not right now, but it may be possible.

Intriguing idea.  Would avoid the ugliness of chardev-add-2.

>                                                Introducing a warning
> when chardev and object ids conflict would be a start.

Yes.

Perhaps even any kind instead of just chardev and object IDs.

Re: Making QEMU easier for management tools and applications

[Andrea Bolognani]

[-- Attachment #1: Type: text/plain, Size: 5321 bytes --]

On Fri, 2020-01-31 at 07:50 +0100, Markus Armbruster wrote:
> Kevin Wolf <kwolf@redhat.com> writes:
> > Much of this threads plays with the though that maybe we don't need any
> > compatibility and make the radical conclusion that we don't need any
> > human-friendly interface at all. Keeping full compatibility is the other
> > extreme.
> > 
> > There might be some middle ground where we break compatibility where the
> > old way can't easily be maintained with the new infrastructure, but
> > don't give up on the idea of being used by humans.
> 
> I'm not sure the connection between maintaining compatibility and
> supporting human use is as strong as you seem to imply.
> 
> As far as I can tell, the "maybe we don't need any compatibility"
> discussion is about the CLI.  I'd rephrase it as "maybe we need a
> machine-friendly CLI on par with QMP more than we need compatibility to
> the current CLI".
> 
> "We don't need any human-friendly interface at all" comes in not because
> machine-friendly necessarily precludes human-friendly, but only if we're
> unwilling (unable?) to do the extra work for it.
> 
> Compare the monitor:
> 
> * QMP is primarily for machines.  We promise stability: no incompatible
>   changes without clear communicaton of intent and a grace period.  We
>   provide machine clients tools to deal with the interface evolution,
>   e.g. query-qmp-schema.
> 
> * HMP is exclusively for humans.  It may change at any time.
> 
> For the CLI, we don't have such a separation, and our offerings for
> dealing with interface evolution are wholly inadequate.  We *need* to do
> better for machines.
> 
> Now, the monitor also informs us about the cost of providing a
> completely separate interface for humans.
> 
> Elsewhere in this thread, we discussed layering (a replacement for) HMP
> on top of QMP cleanly, possibly in a separate process, possibly written
> in a high-level language like Python.
> 
> HMP predates QMP.  We reworked it so the HMP commands are implemented on
> top of the QMP commands, or at least on top of common helpers.  But this
> is not quite the same as layering HMP on top of QMP.
> 
> If we decide to radically break the CLI so we can start over, we get to
> decide whether and how to do a human-friendly CLI, in particular how it
> relates to the machine-friendly CLI.

Does a machine-friendly CLI need to exist at all? Once you decide
that throwing away the current one is acceptable, you might as well
reduce the maintainance burden by requiring that software only
communicates with QEMU via QMP.

Does a human-friendly CLI need to be part of QEMU? We have built so
much amazing infrastructure on top of QEMU, and as of today none of
that work is benefiting people who run it directly.


As a proof of concept, I have spent a couple of hours writing the
attached shell script, which I hope will illustrate my point.

Usage is extremely simple: just do something like

  $ ./virt-run debian-10-openstack-amd64.qcow2

and after a few seconds the guest display will appear on your screen.

Behind the scenes, it uses a number of existing high-level tools:

  * virt-inspector, to figure out what guest OS is installed in the
    image;

  * virt-install, to produce a domain XML tailored to that specific
    guest OS and to create the corresponding libvirt domain;

  * virt-viewer, to provide the UI.

All these tools use libvirt under the hood, and additionally
virt-install uses libosinfo to obtain information about the guest
OS, such as whether or not it supports Virtio devices and how much
memory it needs to run smoothly.


The result is that, if you run

  $ qemu-system-x86_64 -hda debian-10-openstack-amd64.qcow2

you will get

  * a single CPU emulated with TCG;

  * 128 MiB of memory;

  * emulated I/O devices;

whereas the script will give you

  * 2 CPUs accelerated with KVM;

  * 1 GiB of memory;

  * Virtio devices for pretty much everything, including a
    virtio-rng device that will for example speed up the first boot
    significantly if SSH keys need to be (re)created.

Unsurprisingly, performance is different: when QEMU is invoked
directly, the login prompt for this specific image shows up after
~40 seconds, whereas when we use the script it only takes ~13 seconds
to get there. And the command line is just as simple, if not more so!

All of the above was obtained by hastily cobbling together existing
tools with <100 lines of shell scripting. Imagine how much better it
could be if we actually put some serious work in!


With my argument hopefully demonstrated: I think an architecture akin
to the one Dan has outlined earlier[1] would be a great direction to
take. QEMU can continue to focus on its core competency, that is,
virtual hardware, and leave most of the user interaction up to the
software interacting with its JSON-based API.


Obviously QEMU developers, for their own use, could still benefit
from having access to a user interface that doesn't require either
rigging up libvirt support or messing with JSON directly, and such
an interface could even look very similarly to the current CLI, but
it could simply not be made user-facing and thus not subject to any
compatibility concerns.


[1] https://lists.nongnu.org/archive/html/qemu-devel/2020-01/msg06034.html
-- 
Andrea Bolognani / Red Hat / Virtualization

[-- Attachment #2: virt-run --]
[-- Type: application/x-shellscript, Size: 1774 bytes --]

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Andrea Bolognani <abologna@redhat.com> writes:

> On Fri, 2020-01-31 at 07:50 +0100, Markus Armbruster wrote:
>> Kevin Wolf <kwolf@redhat.com> writes:
>> > Much of this threads plays with the though that maybe we don't need any
>> > compatibility and make the radical conclusion that we don't need any
>> > human-friendly interface at all. Keeping full compatibility is the other
>> > extreme.
>> > 
>> > There might be some middle ground where we break compatibility where the
>> > old way can't easily be maintained with the new infrastructure, but
>> > don't give up on the idea of being used by humans.
>> 
>> I'm not sure the connection between maintaining compatibility and
>> supporting human use is as strong as you seem to imply.
>> 
>> As far as I can tell, the "maybe we don't need any compatibility"
>> discussion is about the CLI.  I'd rephrase it as "maybe we need a
>> machine-friendly CLI on par with QMP more than we need compatibility to
>> the current CLI".
>> 
>> "We don't need any human-friendly interface at all" comes in not because
>> machine-friendly necessarily precludes human-friendly, but only if we're
>> unwilling (unable?) to do the extra work for it.
>> 
>> Compare the monitor:
>> 
>> * QMP is primarily for machines.  We promise stability: no incompatible
>>   changes without clear communicaton of intent and a grace period.  We
>>   provide machine clients tools to deal with the interface evolution,
>>   e.g. query-qmp-schema.
>> 
>> * HMP is exclusively for humans.  It may change at any time.
>> 
>> For the CLI, we don't have such a separation, and our offerings for
>> dealing with interface evolution are wholly inadequate.  We *need* to do
>> better for machines.
>> 
>> Now, the monitor also informs us about the cost of providing a
>> completely separate interface for humans.
>> 
>> Elsewhere in this thread, we discussed layering (a replacement for) HMP
>> on top of QMP cleanly, possibly in a separate process, possibly written
>> in a high-level language like Python.
>> 
>> HMP predates QMP.  We reworked it so the HMP commands are implemented on
>> top of the QMP commands, or at least on top of common helpers.  But this
>> is not quite the same as layering HMP on top of QMP.
>> 
>> If we decide to radically break the CLI so we can start over, we get to
>> decide whether and how to do a human-friendly CLI, in particular how it
>> relates to the machine-friendly CLI.
>
> Does a machine-friendly CLI need to exist at all? Once you decide
> that throwing away the current one is acceptable, you might as well
> reduce the maintainance burden by requiring that software only
> communicates with QEMU via QMP.

We need to provide a machine-friendly interface for initial
configuration.

To provide it, we can extend machine-friendly QMP, or make CLI
machine-friendly like QMP.

In both cases, we need to QAPIfy initial configuration.  I believe
that's going to be a big chunk of the work.

To extend QMP, we wrap the QAPIfied initial configuration in QMP
commands.  Many of them will only make sense during initial
configuration.  We'll want to express that in the schema, and enforce it
in the QMP core.

To improve the CLI, we wrap them in CLI options.  We'll want some
infrastructure to generate the boilerplate, just like we generate QMP
command boilerplate.

With an improved CLI, configuration files are just CLI options read from
a file instead of argv[].  We'll want a more suitable concrete syntax
there, of course.

With extended QMP, configuration files are just QMP commands from the
initial configuration subset read from a file instead of a chardev.
Again, we'll likely want more suitable concrete syntax there.

With an improved CLI, I'd expect machines to use configuration files so
they don't have to mess with shell quoting.  With extended QMP, they'd
perhaps rather reuse their existing QMP code to send the configuration
down a socket.  Less efficient, but I doubt it'll matter.

> Does a human-friendly CLI need to be part of QEMU? We have built so
> much amazing infrastructure on top of QEMU, and as of today none of
> that work is benefiting people who run it directly.

I think we do, for developers and for users who don't need the amazing
infrastructure we built on top of QEMU.  When you use QEMU for
virtualization in production, you generally need it, but there are
other, simpler uses, e.g. certain hardware emulation uses.

A human-friendly CLI can be built both as a wrapper around the
machine-friendly CLI and as a wrapper around the initial configuration
subset of QMP.

Once a machine-friendly replacement for the current CLI is in place, we
can relax the current CLI's compatibility requirements.  I figure we
need this to be able to turn it into a wrapper with reasonable effort.
Hopefully, we can generate much of the wrapping boilerplate.

> As a proof of concept, I have spent a couple of hours writing the
> attached shell script, which I hope will illustrate my point.
>
> Usage is extremely simple: just do something like
>
>   $ ./virt-run debian-10-openstack-amd64.qcow2
>
> and after a few seconds the guest display will appear on your screen.
>
> Behind the scenes, it uses a number of existing high-level tools:
>
>   * virt-inspector, to figure out what guest OS is installed in the
>     image;
>
>   * virt-install, to produce a domain XML tailored to that specific
>     guest OS and to create the corresponding libvirt domain;
>
>   * virt-viewer, to provide the UI.
>
> All these tools use libvirt under the hood, and additionally
> virt-install uses libosinfo to obtain information about the guest
> OS, such as whether or not it supports Virtio devices and how much
> memory it needs to run smoothly.
>
>
> The result is that, if you run
>
>   $ qemu-system-x86_64 -hda debian-10-openstack-amd64.qcow2
>
> you will get
>
>   * a single CPU emulated with TCG;
>
>   * 128 MiB of memory;
>
>   * emulated I/O devices;
>
> whereas the script will give you
>
>   * 2 CPUs accelerated with KVM;
>
>   * 1 GiB of memory;
>
>   * Virtio devices for pretty much everything, including a
>     virtio-rng device that will for example speed up the first boot
>     significantly if SSH keys need to be (re)created.
>
> Unsurprisingly, performance is different: when QEMU is invoked
> directly, the login prompt for this specific image shows up after
> ~40 seconds, whereas when we use the script it only takes ~13 seconds
> to get there. And the command line is just as simple, if not more so!
>
> All of the above was obtained by hastily cobbling together existing
> tools with <100 lines of shell scripting. Imagine how much better it
> could be if we actually put some serious work in!

I don't doubt that many, many users of QEMU are better off consuming it
via libvirt.  I've told a few of them over the years :)

> With my argument hopefully demonstrated: I think an architecture akin
> to the one Dan has outlined earlier[1] would be a great direction to
> take. QEMU can continue to focus on its core competency, that is,
> virtual hardware, and leave most of the user interaction up to the
> software interacting with its JSON-based API.
>
>
> Obviously QEMU developers, for their own use, could still benefit
> from having access to a user interface that doesn't require either
> rigging up libvirt support or messing with JSON directly, and such
> an interface could even look very similarly to the current CLI, but
> it could simply not be made user-facing and thus not subject to any
> compatibility concerns.

"Not user-facing" only for a particular value of "user".  But your point
is well taken: only the machine-friendly initial configuration interface
needs to be subject to a compatibility promise.

I believe the difference between providing it in QMP and providing it
separately isn't all that great.

> [1] https://lists.nongnu.org/archive/html/qemu-devel/2020-01/msg06034.html

Re: Making QEMU easier for management tools and applications

[Eric Blake]

On 1/28/20 6:54 AM, Kevin Wolf wrote:

>>
>> The arguments as dotted keys:
>>
>>      id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>>
>> Observe there's quite some of nesting.  While that's somewhat cumbersome
>> in JSON, it's a lot worse with dotted keys, because there nesting means
>> repeated key prefixes.  I could give much worse examples, actually.
> 
> This is true, but even without the repeated keys (e.g. in a syntax that
> would use brackets), it would still be unnecessarily verbose and
> probably hard to remember:
> 
>      id=bar,backend={type=file,data={out=/tmp/bar.log}}

With shells like bash, that would need quoting to avoid unintended brace 
expansions.  It is not the end of the world to require shell quoting 
(and passing JSON on the command line definitely needs it), but a syntax 
that avoids shell quoting is marginally easier to type and reason about.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 31.01.2020 um 13:27 hat Eric Blake geschrieben:
> On 1/28/20 6:54 AM, Kevin Wolf wrote:
> 
> > > 
> > > The arguments as dotted keys:
> > > 
> > >      id=bar,backend.type=file,backend.data.out=/tmp/bar.log
> > > 
> > > Observe there's quite some of nesting.  While that's somewhat cumbersome
> > > in JSON, it's a lot worse with dotted keys, because there nesting means
> > > repeated key prefixes.  I could give much worse examples, actually.
> > 
> > This is true, but even without the repeated keys (e.g. in a syntax that
> > would use brackets), it would still be unnecessarily verbose and
> > probably hard to remember:
> > 
> >      id=bar,backend={type=file,data={out=/tmp/bar.log}}
> 
> With shells like bash, that would need quoting to avoid unintended brace
> expansions.  It is not the end of the world to require shell quoting (and
> passing JSON on the command line definitely needs it), but a syntax that
> avoids shell quoting is marginally easier to type and reason about.

My point was that even with such a simplified syntax (ignoring all the
implication of using it on the command line), the additional nesting
that simple unions give you would still be bad.

That said, I actually think that a syntax like this might make sense for
something like qmp-shell. It might even be more convenient on the
command line than dotted keys if you get a lot of repetition (despite
the required quoting), but it's strictly speaking incompatible because
you could use {} in strings today.

Kevin

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

Il dom 2 feb 2020, 10:22 Kevin Wolf <kwolf@redhat.com> ha scritto:

> Am 31.01.2020 um 13:27 hat Eric Blake geschrieben:
> > On 1/28/20 6:54 AM, Kevin Wolf wrote:
> >
> > > >
> > > > The arguments as dotted keys:
> > > >
> > > >      id=bar,backend.type=file,backend.data.out=/tmp/bar.log
> > > >
> > > > Observe there's quite some of nesting.  While that's somewhat
> cumbersome
> > > > in JSON, it's a lot worse with dotted keys, because there nesting
> means
> > > > repeated key prefixes.  I could give much worse examples, actually.
> > >
> > > This is true, but even without the repeated keys (e.g. in a syntax that
> > > would use brackets), it would still be unnecessarily verbose and
> > > probably hard to remember:
> > >
> > >      id=bar,backend={type=file,data={out=/tmp/bar.log}}
>
> [...] I actually think that a syntax like this might make sense for
> something like qmp-shell. It might even be more convenient on the
> command line than dotted keys if you get a lot of repetition (despite
> the required quoting), but it's strictly speaking incompatible because
> you could use {} in strings today.
>

If you are willing to feed schema info to the parser, in principle you
could keep backwards compatibility. There would be limitations such as
putting the discriminator before the fields, so I am not sure it's a good
idea.

Better QOM introspection would be a requirement, too.

Paolo


> Kevin
>
>

[-- Attachment #2: Type: text/html, Size: 2243 bytes --]

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Il dom 2 feb 2020, 10:22 Kevin Wolf <kwolf@redhat.com> ha scritto:
>
>> Am 31.01.2020 um 13:27 hat Eric Blake geschrieben:
>> > On 1/28/20 6:54 AM, Kevin Wolf wrote:
>> >
>> > > >
>> > > > The arguments as dotted keys:
>> > > >
>> > > >      id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>> > > >
>> > > > Observe there's quite some of nesting.  While that's somewhat
>> cumbersome
>> > > > in JSON, it's a lot worse with dotted keys, because there nesting
>> means
>> > > > repeated key prefixes.  I could give much worse examples, actually.
>> > >
>> > > This is true, but even without the repeated keys (e.g. in a syntax that
>> > > would use brackets), it would still be unnecessarily verbose and
>> > > probably hard to remember:
>> > >
>> > >      id=bar,backend={type=file,data={out=/tmp/bar.log}}
>>
>> [...] I actually think that a syntax like this might make sense for
>> something like qmp-shell. It might even be more convenient on the
>> command line than dotted keys if you get a lot of repetition (despite
>> the required quoting), but it's strictly speaking incompatible because
>> you could use {} in strings today.
>>
>
> If you are willing to feed schema info to the parser, in principle you
> could keep backwards compatibility. There would be limitations such as
> putting the discriminator before the fields, so I am not sure it's a good
> idea.

Problem: the 'any' type, where the schema doesn't provide the necessary
information.

Problem: 'gen': false, where we pass the arguments raw, ignoring the
schema.

If we didn't restrict alternate types so severly, it would also be a
problem.  For instance, with

    { 'alternate': 'Alt',
      'data': { 'one': 'number',
                'two': 'str' } }

we don't know what to do for value "on" branch to take for value 42.
Not a problem because we reject this alternate.  See
tests/qapi-schema/alternate-conflict-*json for more examples.

> Better QOM introspection would be a requirement, too.

I guess this is what you believe is needed to solve these problems.

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Markus Armbruster <armbru@redhat.com> writes:

> Paolo Bonzini <pbonzini@redhat.com> writes:
>
>> Il dom 2 feb 2020, 10:22 Kevin Wolf <kwolf@redhat.com> ha scritto:
>>
>>> Am 31.01.2020 um 13:27 hat Eric Blake geschrieben:
>>> > On 1/28/20 6:54 AM, Kevin Wolf wrote:
>>> >
>>> > > >
>>> > > > The arguments as dotted keys:
>>> > > >
>>> > > >      id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>>> > > >
>>> > > > Observe there's quite some of nesting.  While that's somewhat
>>> cumbersome
>>> > > > in JSON, it's a lot worse with dotted keys, because there nesting
>>> means
>>> > > > repeated key prefixes.  I could give much worse examples, actually.
>>> > >
>>> > > This is true, but even without the repeated keys (e.g. in a syntax that
>>> > > would use brackets), it would still be unnecessarily verbose and
>>> > > probably hard to remember:
>>> > >
>>> > >      id=bar,backend={type=file,data={out=/tmp/bar.log}}
>>>
>>> [...] I actually think that a syntax like this might make sense for
>>> something like qmp-shell. It might even be more convenient on the
>>> command line than dotted keys if you get a lot of repetition (despite
>>> the required quoting), but it's strictly speaking incompatible because
>>> you could use {} in strings today.
>>>
>>
>> If you are willing to feed schema info to the parser, in principle you
>> could keep backwards compatibility. There would be limitations such as
>> putting the discriminator before the fields, so I am not sure it's a good
>> idea.
>
> Problem: the 'any' type, where the schema doesn't provide the necessary
> information.
>
> Problem: 'gen': false, where we pass the arguments raw, ignoring the
> schema.
>
> If we didn't restrict alternate types so severly, it would also be a
> problem.  For instance, with
>
>     { 'alternate': 'Alt',
>       'data': { 'one': 'number',
>                 'two': 'str' } }
>
> we don't know what to do for value "on" branch to take for value 42.
> Not a problem because we reject this alternate.  See
> tests/qapi-schema/alternate-conflict-*json for more examples.
>
>> Better QOM introspection would be a requirement, too.
>
> I guess this is what you believe is needed to solve these problems.

Here's how we currently solve them.

We have four pipelines from text to QAPI objects:

1. JSON:
            JSON                   QObject wrapped                 QAPI
   text --> parser --> QObject --> in plain qobject --> visit --> object
                                    input visitor

2. keyval:
            keyval                 QObject wrapped                 QAPI
   text --> parser --> QObject --> in keyval qobj.  --> visit --> object
                                    input visitor

3. "string":
                                    text wrapped                   QAPI
   text ------------------------->    in string     --> visit --> object
                                    input visitor

4. QemuOpts:
             opts                   QemOpts wrapped                QAPI 
   text --> parser --> QemuOpts -->   in options    --> visit --> object
                                        visitor    


In the JSON pipeline, the parser produces an appropriately typed
QObject, and the qobject input visitor checks it against the schema.

Dotted keys syntax doesn't tell us which scalar type to use, so the
parser uses QString for all scalars.  We then have to use a special
keyval qobject input visitor that expects strings instead of
appropriately typed scalars.  If the resulting QAPI object contains
'any' types, then those remain unconverted.  To visit them, you must use
the keyval qobject input visitor, not the plan one.

These pipelines share a tail:

                                   Input wrapped                   QAPI
                                   in appropriate   --> visit --> object
                                   input visitor

The (still few) QAPIfied CLI options exploit this: if the option
argument looks like JSON, we feed it to the JSON half of the joint
pipeline, else to the keyval half.  The entire part before "visit" is
qobject_input_visitor_new_str(): it returns the QObject wrapped in the
appropriate visitor.

To go from QObject wrapped in visitor to a QAPI object of type T, we
generally call generated function visit_type_T().

The string pipeline is used chiefly for QOM.  The visit part is
generally manual, in the property set() method.  The string visitor is
seriously limited: it supports only scalars and lists of certain
integers.

The QemuOpts pipeline is used by a few CLI options and HMP commands,
such as -netdev, -numa, -object.  It is also almost as seriously
limited.

The code for "lists of certain integers" will make your eyes bleed.

I'd love to replace the string and the options pipeline by the keyval
pipeline.

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* Kevin Wolf (kwolf@redhat.com) wrote:
> Am 25.01.2020 um 11:18 hat Markus Armbruster geschrieben:
> > Kevin Wolf <kwolf@redhat.com> writes:
> > 
> > > Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
> > >> This is finally something I'd consider to be on a par with the
> > >> original QEMU syntax, before we added hierarchical data. You
> > >> have the minimal possible amount of syntax here. No commas,
> > >> no quotes, no curly brackets, etc.
> > >
> > > This seems to have the same problems as the QEMU command line (how do
> > > you distinguish strings from ints, from bools, from null?).
> > 
> > True: YAML provides only string scalars.
> > 
> > TOML provides strings, integers, floats, booleans, and several flavors
> > of time.  It lacks null.
> > 
> > >                                                             It's
> > > basically just a pretty-printed version of it with the consequence that
> > > it needs to be stored in an external file and there is no reasonable way
> > > to keep it in my shell history.
> > 
> > There is a reasonable way to keep it in my file system, though.  I find
> > that decidedly superior.
> 
> That depends a lot on your use case.
> 
> If you have a long-lived production VM that you always run with the same
> configuration, then yes, having a config file for it in the file system
> is what you probably want. Currently, for this case, people directly
> using QEMU tend to write a script that contains the command line. I
> think I do have such scripts somewhere, but their number is very small.
> 
> My common case is short-lived VMs with configurations that change very
> often between QEMU invocations. Here the command line is decidedly
> superior.

I can imagine you could do something similar to gdb's --eval-command
option which lets you pass a command to be executed at startup; gdb's
syntax is a bit painful but it feels like that could be fixed, e.g. I
have:

gdb --eval-command='set pagination off' --eval-command='set confirm no' --eval-command='handle SIGPIPE print pas
s nostop' --eval-command='handle SIGBUS print pass nostop' --eval-command=r --eval-command='thread apply all bt full'
 --eval-command='info proc mappings' --eval-command='info threads' --eval-command=q --args $QEMUTODEBUG "$@"

in a script.

Dave

> Requiring me to create a file in the file system each time and to
> remember deleting it after I'm done feels about as convenient as a *nix
> shell that doesn't accept parameters for commands on the command line,
> but instead requires you to write a one-off script first and then run
> that.
> 
> Kevin
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 2:59 AM, Markus Armbruster wrote:
> Digression: JSON is a poor choice for configuration files.  YAML is a
> complex and confusing beast (it's spec is printed 116 pages, and
> yaml-0.2.2/src is ~7kSLOC).  XML is XML, 'nuff said.  TOML is much
> simpler than either of the two.
JSON has a built-in Python parser. YAML has a widely availably PyYAML
parser module. TOML is not yet in widespread use, but does offer a
python parser.

No XML parser I have ever used appears to be feature-complete to the XML
spec. Most modern toolings support JSON and/or YAML.

I personally like the idea of YAML as it was intended for human-written
configuration files without brackets and other such visual confetti. It
even allows for comments as a first-class citizen of the specification.
This would be useful to documentation efforts.

Hearsay: most of the existing high-level users of the virt stack are in
Go or Python. Python users are quite comfortable with whitespace-markup
formats. Go developers are likely working in the cloud space, which
overwhelmingly uses YAML.


Suggestion:

(1) We use YAML and use Python's built-in YAML parser to build a machine
specification format defined by QAPI.

We build a "configuration" struct in QAPI, and extend from there.

(2) We offer "--config myconfig.yaml" as an option for specifying
options. This precludes the use of *any* traditional command line flags.
Security wonkery permitting, we may also allow QEMU to take a YAML file
on STDIN at boot, if this is easier for cloud management programs to script.

(3) We offer "--writeconfig out.yaml" to convert to the new format... if
possible. It very well may NOT be, given the nature of our wild west
parsing system. I consider this an optional olive branch, not a necessity.

(4) Just scrap the existing config file system entirely. It was never
finished and does not help solve the existing problem that we do not
have a schema for our configuration syntax.

(5) Deprecate the traditional CLI configuration syntax. Remove it only
on a major version transition. Advertise this as our "2.0" break. Use
this opportunity to do heavier refactoring of our worst design
transgressions as desired. Make no effort to add additional features to
the CLI if it doesn't come for free.

(Bonus: If we work with libvirt to adapt to the new format, libvirt
users will enjoy a seamless transition. This is an insanely good
marketing point for libvirt's usefulness, allowing QEMU to adapt as
necessary to remove baggage with minimal interruption to end-users.)

(Downside: Any libvirt XML passing CLI flags directly will lose that
ability. They will need to specify YAML object overrides that plug into
specific parts of the hierarchy instead.)

(Upside to the downside: These overrides are often not supported by
downstream vendors anyway.)


Benefits:

- The entirety of the ugly parsing code in vl.c and strewn about
submodules goes away. Many of the .hx files go away.

- QemuOpts goes away. Everything is handled at the QAPI level via YAML
and QMP parsers. Input errors are handled at the schema level.

- Extending QMP and the CLI occurs with one schema definition change.

- We don't have to write a new python parser; The complexity of YAML
itself poses no particular concern here.

- YAML is in wide use. Support in emacs/vim is widespread. Third party
tools for manipulating YAML documents are widely available and in-use
due to usage in the cloud infrastructure world.

- Once deserialized, JSON and YAML configuration statements are
identical in-memory in Python. Many of the same tools and infrastructure
can be recycled.

- PyYaml uses libYAML, a C library. That same library can be used to
build the same structures that we already use for QMP.

- YAML examples can be trivially generated from the QAPI spec.

- YAML allows for in-line comments, which would be a tremendous feature
for writing documentation and annotating machine configuration files.


Downsides:

- We WOULD need a new YAML parsing layer in QEMU, generated by QAPI.
Supporting two deserialization layers could lead to strange
discrepancies between the two formats at runtime. So, admittedly, using
JSON would be *even easier*, as we could re-use the same parsers already
battle-tested in QEMU. We could get *MAXIMUM* code re-use this way.

- YAML does not allow duplicate keys. On its face, this is not an issue,
we can't support parallel universe configurations anyway. If it's
important to allow "overrides" of base configurations, this might intuit
the need for cascading YAML files. (--config base.yaml --config
overlay.yaml)

- We will break compatibility with our existing CLI. People will not be
happy about this, especially, perhaps, embedded board and TCG developers
who use fairly minimal command lines regularly.

- A lot of iotests and python scripts will need serious love to bring
forward into the new world.



I guess at the end of writing this out, I wind up making a slightly
better case for JSON in terms of implementation difficulty, but still
have a personal bias and fondness for YAML, in part for the in-line
comments and in part for bringing us closer to cloud projects which use
YAML heavily. Not sure if it's worth the pain in the end, but I haven't
given up hope yet.


Some things I think are a given:

- We want a config file. We don't know what format it should be in, but
it should be defined by the QAPI schema in some way. Declarative and
data driven is the most beautiful of all programming paradigms.

- Our existing CLI syntax is large, unknowable, and poorly documented.
The work it would take to turn it into a schema-defined language is
likely as-complex as it would be to develop a new format from first
principles anyway.

- Due to the ad-hoc nature of how each top-level CLI flag/option parses
its own argument data, making the CLI regular is going to involve a lot
of broken backwards compatibility and unhappy users anyway.

- So we may as well go for broke and make one hell of a config file
format, whatever language it ends up being in.


--js

Re: Making QEMU easier for management tools and applications

[Gerd Hoffmann]

  Hi,

> We build a "configuration" struct in QAPI, and extend from there.
> 
> (2) We offer "--config myconfig.yaml" as an option for specifying
> options.

Yes.

> This precludes the use of *any* traditional command line flags.

Hmm.  Given that the transition effort will probably take a while
I think it makes sense to allow mixing config file and cmd line
switches, if it is only for testing the config file parsing and
processing.  Maybe have a (temporary) -x-config for that?

We already have qapi schema for -blockdev + -display + -audiodev +
-chardev, creating a configuration struct which supports these
shouldn't be that much effort.  Then move over QemuOpts one by one.

The most tricky part here probably is -device support.

> (4) Just scrap the existing config file system entirely. It was never
> finished and does not help solve the existing problem that we do not
> have a schema for our configuration syntax.

Yep.  Zero reason to keep that once everything transitioned to qapi.

> - We WOULD need a new YAML parsing layer in QEMU, generated by QAPI.
> Supporting two deserialization layers could lead to strange
> discrepancies between the two formats at runtime. So, admittedly, using
> JSON would be *even easier*, as we could re-use the same parsers already
> battle-tested in QEMU. We could get *MAXIMUM* code re-use this way.

Well, we can support multiple formats, much like openshift accepts both
json and yaml.  json is pretty much there already, so we could start
with that and add yaml later on.  Possibly even the ini-style syntax
accepted by -readconfig today, although I'm not sure there is much
benefit in that.

> - We will break compatibility with our existing CLI. People will not be
> happy about this, especially, perhaps, embedded board and TCG developers
> who use fairly minimal command lines regularly.

Once config file support is complete enough we should be able to offload
backward compatibility command line parsing to some script which
transforms the cli into a config file.  Which is probably better than
trying to add -writeconfig to qemu as we don't have to do the string
processing in C then.

cheers,
  Gerd

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 27.01.2020 um 09:35 hat Gerd Hoffmann geschrieben:
>   Hi,
> 
> > We build a "configuration" struct in QAPI, and extend from there.
> > 
> > (2) We offer "--config myconfig.yaml" as an option for specifying
> > options.
> 
> Yes.
> 
> > This precludes the use of *any* traditional command line flags.
> 
> Hmm.  Given that the transition effort will probably take a while
> I think it makes sense to allow mixing config file and cmd line
> switches, if it is only for testing the config file parsing and
> processing.  Maybe have a (temporary) -x-config for that?
> 
> We already have qapi schema for -blockdev + -display + -audiodev +
> -chardev, creating a configuration struct which supports these
> shouldn't be that much effort.  Then move over QemuOpts one by one.

chardev-add uses a QAPI type that has too deep nesting to be user
friendly. Markus and I are trying to use mostly QAPIfied command line
options for qemu-storage-daemon, and --chardev is one that we agree must
be simplified to be bearable.

> The most tricky part here probably is -device support.

Possibly, though -object might not be much easier.

> > - We WOULD need a new YAML parsing layer in QEMU, generated by QAPI.
> > Supporting two deserialization layers could lead to strange
> > discrepancies between the two formats at runtime. So, admittedly, using
> > JSON would be *even easier*, as we could re-use the same parsers already
> > battle-tested in QEMU. We could get *MAXIMUM* code re-use this way.
> 
> Well, we can support multiple formats, much like openshift accepts both
> json and yaml.  json is pretty much there already, so we could start
> with that and add yaml later on.  Possibly even the ini-style syntax
> accepted by -readconfig today, although I'm not sure there is much
> benefit in that.
> 
> > - We will break compatibility with our existing CLI. People will not be
> > happy about this, especially, perhaps, embedded board and TCG developers
> > who use fairly minimal command lines regularly.
> 
> Once config file support is complete enough we should be able to offload
> backward compatibility command line parsing to some script which
> transforms the cli into a config file.  Which is probably better than
> trying to add -writeconfig to qemu as we don't have to do the string
> processing in C then.

If we get a launcher script anyway, I would argue that the system
emulator binary written in C should only support JSON (like it already
does) and the script can easily translate from YAML to JSON.

Kevin

Re: Making QEMU easier for management tools and applications

[Gerd Hoffmann]

 Hi,

> > We already have qapi schema for -blockdev + -display + -audiodev +
> > -chardev, creating a configuration struct which supports these
> > shouldn't be that much effort.  Then move over QemuOpts one by one.
> 
> chardev-add uses a QAPI type that has too deep nesting to be user
> friendly.

Yea, right, it is relatively old and doesn't use stuff like base types
which didn't exist back then ...

> > Once config file support is complete enough we should be able to offload
> > backward compatibility command line parsing to some script which
> > transforms the cli into a config file.  Which is probably better than
> > trying to add -writeconfig to qemu as we don't have to do the string
> > processing in C then.
> 
> If we get a launcher script anyway, I would argue that the system
> emulator binary written in C should only support JSON (like it already
> does) and the script can easily translate from YAML to JSON.

That makes sense indeed.

cheers,
  Gerd

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
> 
> 
> On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
> > IOW, the difficulty with configuring QEMU via JSON is not the fault
> > of JSON itself, it is the lack of knowledge amongst users and docs,
> > compounded by our never ending "improvements" to the human syntax.
> > There are other factors too, such as our only partial coverage of
> > config using JSON - some is only possible via the CLI still.
> > 
> 
> I'm fine with getting rid of HMP entirely, I think. It's a weird
> interface with bizarre behavior that's hard to support.
> 
> There's a few commands in there we just don't support at all, but maybe
> it's time to start deprecating one-by-one any of the individual commands
> that are better served by QMP these days, to send the message that HMP's
> days are numbered.
> 
> Bye-bye!
> 
> As for the CLI, well, that's part of the discussion at hand...
> 
> > 
> > I guess my point is that with a scrap & startover view point, we
> > should arguably completely ignore the design question of how to
> > flatten JSON for humans/command line, as it is the wrong problem.
> > Instead focus on the problem of making use of JSON the best way
> > to deal with QEMU both functionally and practically, for humans
> > and machines alike.
> > 
> 
> Well, sure. The context of this email was qmp-shell though, which is
> meant to help facilitate the entry of JSON commands so that you *can*
> indeed just forego the CLI/HMP entirely.
> 
> If you are of the opinion that every user of QEMU should be copy/pasting
> JSON straight into a socket and we should delete qmp-shell, that's
> certainly a fine opinion.

I think part of the pain of qmp-shell comes from the very fact that
it is trying to be an interactive shell. This points people towards
interactively typing in the commands, which is horrific when you get
anywhere near the JSON, or even dot-notation traditional commands.

If it was just a qmp-client that was single shot, we'd encourage
people to create the JSON in a sensible way - vim/emacs/whatever.

Bash/dash/zsh/$whatever is their interactive shell, with massively
more features than qmp-shell. You have command history, autocomplete,
conditional and looping constructs, and everything a normal shell
offers.

The only strong reason for qmp-shell to be interactive would be if
the initial protoocl handshake was too slow. I can't see that being
a problem with QMP. 

> I'm coming from the side that I love qmp-shell; I find it useful, but it
> has some syntax problems. How do I solve them? Is there a way to solve
> them? QAPI is here to stay, and QAPI involves hierarchical data. That
> data is usually best represented by something like json or yaml, but
> those are hard to type in a shell.
> 
> What do we do about that?

Here's one conceptual vision of how a better QEMU might look:

  * qemu-runtime-$TARGET

    A binary that contains the implementation for the machine
    emulator for $TARGET.

    This has no command line arguments except for a UNIX
    socket path which is a QMP server


  * qemu-launcher-$TARGET

    A binary that is able to launch qemu-runtime-$TARGET
    with jailers active.

    This has no command line arguments except for a pair
    of UNIX socket paths. One is a QMP server, the other
    is the path for the QMP of qemu-runtime-$TARGET.

    Commands it processes will be in automatically proxied
    through to the qemu-runtime-$TARGET QMP, with appropriate
    jailer updates being done in between.


  * qemu-client

    A binary that speaks QMP, connects, runs a single command,
    disconnects.

    It is used to talk to either qemu-runtime-$TARGET or
    qemu-launcher-$TARGET, depending on whether the mgmt app
    or user wants to be making use of the jailer facilities
    or not.  


  * qemu-system-$TARGET

    The current binaries that exist today.

    qemu-system-$TARGET should not be part of our formal
    stability promise. We won't gratuitously / knowingly
    break without good reason, but we will accept that
    breakage can happen. Stability is only offered by
    the qemu-{runtime,launcher}-$TARGET.

    Several choices for their future in long term:

      - Leave them as-is and basically ignore them
        whereever practical going forward, so we
	minimally worry about backcompat breakage

      - Plan to re-write them so that they are simply
        a shim the forks+execs qemu-runtime-$TARGET
	and does syntax translation from CLI/HMP/QMP.

      - Deprecate them with a view to deletion entirely
        in $NNN years. For some large-ish value of NNN,
	given how well known they are


Example usage:

1. Launch the QEMU runtime for the desired target

     $ qemu-runtime-x86_64 myvm.sock

2. Load the configuration to define the VM

   $ cat myvm.yaml
   commands:
     - machine_declare:
         name: pc-q35-5.0
	 ...
     - blockdev_add:
         ...
     - device_add:
         ...
     - blockdev_add:
         ...
     - device_add:
         ...
   $ qemu-client myvm.sock myvm.yaml


3. Hotplug a disk

   $ cat mynewdisk.yaml
   commands:
     - blockdev_add:
         ...
     - device_add:
         ...
   $ qemu-client myvm.sock mynewdisk.yaml


3. Hotunplug a disk

   $ cat myolddisk.yaml
   commands:
     - device_del:
         ...
     - blockdev_del:
         ...
   $ qemu-client myvm.sock myolddisk.yaml

Using jailers, just means adding in a use of qemu-launcher-$TARGET
at the start.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 24/01/20 10:50, Daniel P. Berrangé wrote:
>   * qemu-launcher-$TARGET
> 
>     A binary that is able to launch qemu-runtime-$TARGET
>     with jailers active.
> 
>     This has no command line arguments except for a pair
>     of UNIX socket paths. One is a QMP server, the other
>     is the path for the QMP of qemu-runtime-$TARGET.
> 
>     Commands it processes will be in automatically proxied
>     through to the qemu-runtime-$TARGET QMP, with appropriate
>     jailer updates being done in between.
> 

What would be the advantage of this over the Libvirt embedded driver?
Especially if you include in the picture something like libvirt-go-xml
(or libvirt-GObject, does it still exist?) that hides the XML from the
code that uses it.

The main complication in the launcher is hotplug, which means that a
simple "do a couple bind mounts, unshare, drop privileges and forget
about it" approach doesn't work.  Proxying QMP commands doesn't seem
that easy, and I don't see much code being shared between the launcher
and QEMU; if the existing QEMU code is not suitable for Libvirt, it
wouldn't be suitable for a qemu.git launcher either.

Also, as you mentioned earlier, QEMU wants to keep its vocabulary
lower-level, and therefore the launcher's vocabulary would end up
diverging from QEMU.  Some example:

- QEMU wants a qemu-pr-helper socket path, the launcher would take care
of launching qemu-pr-helper itself

- QEMU wants the complete configuration on the migration destination,
the launcher might take care of sending it from the source?

At this point, you get something that looks very much like Libvirt and,
especially if you include live migration, it has to take into account
the same compatibility considerations as Libvirt.

Thanks,

Paolo

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Sat, Jan 25, 2020 at 12:52:30PM +0100, Paolo Bonzini wrote:
> On 24/01/20 10:50, Daniel P. Berrangé wrote:
> >   * qemu-launcher-$TARGET
> > 
> >     A binary that is able to launch qemu-runtime-$TARGET
> >     with jailers active.
> > 
> >     This has no command line arguments except for a pair
> >     of UNIX socket paths. One is a QMP server, the other
> >     is the path for the QMP of qemu-runtime-$TARGET.
> > 
> >     Commands it processes will be in automatically proxied
> >     through to the qemu-runtime-$TARGET QMP, with appropriate
> >     jailer updates being done in between.
> > 
> 
> What would be the advantage of this over the Libvirt embedded driver?
> Especially if you include in the picture something like libvirt-go-xml
> (or libvirt-GObject, does it still exist?) that hides the XML from the
> code that uses it.

Yes, libvirt-gobject still exists & is used by GNOME Boxes.

> The main complication in the launcher is hotplug, which means that a
> simple "do a couple bind mounts, unshare, drop privileges and forget
> about it" approach doesn't work.  Proxying QMP commands doesn't seem
> that easy, and I don't see much code being shared between the launcher
> and QEMU; if the existing QEMU code is not suitable for Libvirt, it
> wouldn't be suitable for a qemu.git launcher either.

Mostly this suggestion is a response to discussions elsewhere in this
thread expressing a desire for a way to do QEMU jailing natively. In
this proposal I'm describing an approach that might satisfy that desire,
while ignoring whatever exists in the current ecosystem including
libvirt.

So yes, there is clearly overlap here with libvirt, and this tool is
not an easy thing to build to a level which is feature comparable
with what libvirt offers. For this launcher to be credible I think
it would have to be something libvirt were capable of using itself
and that's a challenge. So quite possibly the libvirt embedded driver
is indeed a better bet.  


> Also, as you mentioned earlier, QEMU wants to keep its vocabulary
> lower-level, and therefore the launcher's vocabulary would end up
> diverging from QEMU.  Some example:
> 
> - QEMU wants a qemu-pr-helper socket path, the launcher would take care
> of launching qemu-pr-helper itself
> 
> - QEMU wants the complete configuration on the migration destination,
> the launcher might take care of sending it from the source?
> 
> At this point, you get something that looks very much like Libvirt and,
> especially if you include live migration, it has to take into account
> the same compatibility considerations as Libvirt.

Yes indeed.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Tooling to help humans use JSON (was: Making QEMU easier for management tools and applications)

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
>> 
>> 
>> On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
[...]
>> Well, sure. The context of this email was qmp-shell though, which is
>> meant to help facilitate the entry of JSON commands so that you *can*
>> indeed just forego the CLI/HMP entirely.
>> 
>> If you are of the opinion that every user of QEMU should be copy/pasting
>> JSON straight into a socket and we should delete qmp-shell, that's
>> certainly a fine opinion.
>
> I think part of the pain of qmp-shell comes from the very fact that
> it is trying to be an interactive shell. This points people towards
> interactively typing in the commands, which is horrific when you get
> anywhere near the JSON, or even dot-notation traditional commands.
>
> If it was just a qmp-client that was single shot, we'd encourage
> people to create the JSON in a sensible way - vim/emacs/whatever.

Tooling can help with typing JSON.  Mature tooling for editing JSON
files exists.  I'm not aware of such tooling for interactive use of
JSON.  It certainly could be made.  qmp-shell tries to help by
"improving" the JSON syntax, half-heartedly.  I doubt that's a good
idea.

Kubernetes went the "put complex configuration into files, use the
mature tooling for editing files" route.  I think we can call that a
success.

The fact that JSON is not meant to serve as configuration file format /
command language for human users bears repeating.  It's a data
interchange format.

Kubernetes supplemented JSON (adequate for machines) with YAML (more
pleasant for humans, I guess).  I'm no friend of YAML myself (too much
syntax, not enough types), but once again, I think we can call that a
success, at least for Kubernetes.

> Bash/dash/zsh/$whatever is their interactive shell, with massively
> more features than qmp-shell. You have command history, autocomplete,
> conditional and looping constructs, and everything a normal shell
> offers.
>
> The only strong reason for qmp-shell to be interactive would be if
> the initial protoocl handshake was too slow. I can't see that being
> a problem with QMP. 

Me neither.

The connection overhead may matter for non-interactive uses.  These are
better off without a middleman.

[...]

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
>> 
>> 
>> On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
[...]
>> > I guess my point is that with a scrap & startover view point, we
>> > should arguably completely ignore the design question of how to
>> > flatten JSON for humans/command line, as it is the wrong problem.
>> > Instead focus on the problem of making use of JSON the best way
>> > to deal with QEMU both functionally and practically, for humans
>> > and machines alike.

Note: I understand "JSON" to stand for "JSON and possibly another
concrete syntax better suited for humans".  Your examples below show
YAML in that role.

[...]
> Here's one conceptual vision of how a better QEMU might look:
>
>   * qemu-runtime-$TARGET
>
>     A binary that contains the implementation for the machine
>     emulator for $TARGET.
>
>     This has no command line arguments except for a UNIX
>     socket path which is a QMP server
>
>
>   * qemu-launcher-$TARGET
>
>     A binary that is able to launch qemu-runtime-$TARGET
>     with jailers active.
>
>     This has no command line arguments except for a pair
>     of UNIX socket paths. One is a QMP server, the other
>     is the path for the QMP of qemu-runtime-$TARGET.
>
>     Commands it processes will be in automatically proxied
>     through to the qemu-runtime-$TARGET QMP, with appropriate
>     jailer updates being done in between.

See Paolo's reply.

>   * qemu-client
>
>     A binary that speaks QMP, connects, runs a single command,
>     disconnects.

The single command is specified how?

Ah, "Example usage" below shows it's taken from a file argument.

>     It is used to talk to either qemu-runtime-$TARGET or
>     qemu-launcher-$TARGET, depending on whether the mgmt app
>     or user wants to be making use of the jailer facilities
>     or not.  

Effectively ditches the CLI in favor of QMP.  One major external host
interface instead of two.

The connection to "making use of JSON the best way to deal with QEMU
both functionally and practically, for humans and machines alike" is
"make it possible to use just QMP for *everything*".  Correct?

We've toyed with the "Just QMP" idea almost since QMP exists.  We never
went beyond "wouldn't it be nice if".

The main chunk of work is providing the CLI functionality we need in
QMP.  Requires QAPIfication of the parts that aren't QAPIfied, yet.

Since parts of CLI overlap with QMP, providing CLI functionality need
not require new QMP commands.  For instance, CLI does cold plug, QMP
does hot plug.  The existing QMP interface could do both: hot while the
machine runs, else cold.

Some functionality makes sense only during initial startup.  Implied in
CLI.  In QMP, it has to be tied to the run state.  No big deal.

"Just QMP" is of course not the only way to do "JSON first".  Another
one is bringing JSON to the CLI.  I explored that: CLI QAPIfication.

Requires the same QAPIfication as "Just QMP" does.

Where "Just QMP" is a hard compatibility break by design, "JSON in CLI"
gives us a choice: we can choose to break compatibility, e.g. by
creating the exact same set of new executables you propose.  Or we can
try to evolve the existing CLI compatibly.  The latter is a tar pit,
because the existing CLI is.  But it's a tar pit *we* get to choose.  We
can't blame "JSON in CLI" for it, only ourselves.

Even though I've worked on "JSON in CLI", I'm not overly attached to it.
If we decide "Just QMP" is the better path forward, I'll happily go
along.  I believe the bulk of the work will be the same: QAPIfying
stuff.

>   * qemu-system-$TARGET
>
>     The current binaries that exist today.
>
>     qemu-system-$TARGET should not be part of our formal
>     stability promise. We won't gratuitously / knowingly
>     break without good reason, but we will accept that
>     breakage can happen. Stability is only offered by
>     the qemu-{runtime,launcher}-$TARGET.
>
>     Several choices for their future in long term:
>
>       - Leave them as-is and basically ignore them
>         whereever practical going forward, so we
> 	minimally worry about backcompat breakage
>
>       - Plan to re-write them so that they are simply
>         a shim the forks+execs qemu-runtime-$TARGET
> 	and does syntax translation from CLI/HMP/QMP.
>
>       - Deprecate them with a view to deletion entirely
>         in $NNN years. For some large-ish value of NNN,
> 	given how well known they are

How do the other complex executables like qemu-img, qemu-nbd fit into
this picture?

Do they become redundant somehow for non-human users?

If not, will they get a QMP-only sibling, like qemu-system-$TARGET gets
qemu-runtime-$TARGET?

> Example usage:
>
> 1. Launch the QEMU runtime for the desired target
>
>      $ qemu-runtime-x86_64 myvm.sock
>
> 2. Load the configuration to define the VM
>
>    $ cat myvm.yaml
>    commands:
>      - machine_declare:
>          name: pc-q35-5.0
> 	 ...
>      - blockdev_add:
>          ...
>      - device_add:
>          ...
>      - blockdev_add:
>          ...
>      - device_add:
>          ...
>    $ qemu-client myvm.sock myvm.yaml
>
>
> 3. Hotplug a disk
>
>    $ cat mynewdisk.yaml
>    commands:
>      - blockdev_add:
>          ...
>      - device_add:
>          ...
>    $ qemu-client myvm.sock mynewdisk.yaml
>
>
> 3. Hotunplug a disk
>
>    $ cat myolddisk.yaml
>    commands:
>      - device_del:
>          ...
>      - blockdev_del:
>          ...
>    $ qemu-client myvm.sock myolddisk.yaml
>
> Using jailers, just means adding in a use of qemu-launcher-$TARGET
> at the start.
>
>
> Regards,
> Daniel

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Mon, Jan 27, 2020 at 10:06:45AM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> >   * qemu-system-$TARGET
> >
> >     The current binaries that exist today.
> >
> >     qemu-system-$TARGET should not be part of our formal
> >     stability promise. We won't gratuitously / knowingly
> >     break without good reason, but we will accept that
> >     breakage can happen. Stability is only offered by
> >     the qemu-{runtime,launcher}-$TARGET.
> >
> >     Several choices for their future in long term:
> >
> >       - Leave them as-is and basically ignore them
> >         whereever practical going forward, so we
> > 	minimally worry about backcompat breakage
> >
> >       - Plan to re-write them so that they are simply
> >         a shim the forks+execs qemu-runtime-$TARGET
> > 	and does syntax translation from CLI/HMP/QMP.
> >
> >       - Deprecate them with a view to deletion entirely
> >         in $NNN years. For some large-ish value of NNN,
> > 	given how well known they are
> 
> How do the other complex executables like qemu-img, qemu-nbd fit into
> this picture?
> 
> Do they become redundant somehow for non-human users?
> 
> If not, will they get a QMP-only sibling, like qemu-system-$TARGET gets
> qemu-runtime-$TARGET?

I've not really thought about the other tools when writing this.

In libvirt we do see much of the same pain points in dealing with
these tools, as we do with QEMU emulator targets though. eg need
to probe features, the complexity of specifying block devices.
So there's clearly some conceptual overlap here to consider.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
> > Well, sure. The context of this email was qmp-shell though, which is
> > meant to help facilitate the entry of JSON commands so that you *can*
> > indeed just forego the CLI/HMP entirely.
> > 
> > If you are of the opinion that every user of QEMU should be copy/pasting
> > JSON straight into a socket and we should delete qmp-shell, that's
> > certainly a fine opinion.
> 
> I think part of the pain of qmp-shell comes from the very fact that
> it is trying to be an interactive shell. This points people towards
> interactively typing in the commands, which is horrific when you get
> anywhere near the JSON, or even dot-notation traditional commands.
> 
> If it was just a qmp-client that was single shot, we'd encourage
> people to create the JSON in a sensible way - vim/emacs/whatever.

I don't see how this is sensible. QMP commands are something that I
reuse even less than VM configurations, so creating a one-off file for
each would take me a lot more time and I would still have to type the
same JSON code that I have to type with -qmp stdio.

The reason it is and should be an interactive shell is that I'm
interacting with it. Switching back and forth between a text editor and
a shell to actually send the command to QEMU would make things only even
more cumbersome than they already are.

> Bash/dash/zsh/$whatever is their interactive shell, with massively
> more features than qmp-shell. You have command history, autocomplete,
> conditional and looping constructs, and everything a normal shell
> offers.

If I wanted to program a QMP client, I would use Python. For me,
conditionals and loops are completely out of scope for a QMP shell. I
just want an easy way to tell QEMU to do something specific.

A command history already exists for qmp-shell. It's better than bash
because it doesn't mix QMP history with whatever else I do on my
computer.

Autocomplete in qmp-shell doesn't exist, as far as I know, but if
implemented, it could be a lot more useful than bash completion because
it could offer key completion based on the QMP schema.

This is in fact a big part of the problem that qmp-shell really needs to
solve before it can replace HMP: How to make writing commands at least
almost as simple as with HMP. If I can just press tab a few times to
cycle through the available options for the command, that would already
be a massive improvement over writing JSON manually (which you would
still have to do with your text-file based approach, without any
QMP-specific support).

The other part that it needs to solve is how to be available by default
without specifying anything on the command line. Basically, if I press
Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
implemented internally or by an external Python process, I don't mind.

> The only strong reason for qmp-shell to be interactive would be if
> the initial protoocl handshake was too slow. I can't see that being
> a problem with QMP.

Speed would be the least of my concerns. This is about manual use, and
it already takes me a while to type in my commands.

> Example usage:
> 
> 1. Launch the QEMU runtime for the desired target
> 
>      $ qemu-runtime-x86_64 myvm.sock
> 
> 2. Load the configuration to define the VM
> 
>    $ cat myvm.yaml
>    commands:
>      - machine_declare:
>          name: pc-q35-5.0
> 	 ...
>      - blockdev_add:
>          ...
>      - device_add:
>          ...
>      - blockdev_add:
>          ...
>      - device_add:
>          ...
>    $ qemu-client myvm.sock myvm.yaml
> 
> 
> 3. Hotplug a disk
> 
>    $ cat mynewdisk.yaml
>    commands:
>      - blockdev_add:
>          ...
>      - device_add:
>          ...
>    $ qemu-client myvm.sock mynewdisk.yaml
> 
> 
> 3. Hotunplug a disk
> 
>    $ cat myolddisk.yaml
>    commands:
>      - device_del:
>          ...
>      - blockdev_del:
>          ...
>    $ qemu-client myvm.sock myolddisk.yaml

Just to compare, this is what the human user oriented flow looks like
today:

1. qemu-system-x86_64 -M pc-q35-5.0 -drive if=virtio,... -cdrom ...

2. <Press Ctrl-Alt-2 to get to the HMP shell>
   (qemu) drive_add ...
   <Press Ctrl-Alt-1 to get back to the guest>

3. <Press Ctrl-Alt-2 to get to the HMP shell>
   (qemu) device_del ...
   <Press Ctrl-Alt-1 to get back to the guest>

This is what we're competing with, and honestly I don't see how your
qemu-runtime-*/qemu-client based flow comes even close to it in terms of
usability.

QMP, JSON and YAML may be nice machine interfaces, but having nice
machine interfaces doesn't mean that you shouldn't also have something
that is suitable for humans. qmp-shell is trying to be that, and while
it leaves much to be desired in its current state, replacing it with
even more machine-friendly stuff that is cumbersome for humans isn't the
right answer.

Kevin

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* Kevin Wolf (kwolf@redhat.com) wrote:
> Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
> > On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
> > > Well, sure. The context of this email was qmp-shell though, which is
> > > meant to help facilitate the entry of JSON commands so that you *can*
> > > indeed just forego the CLI/HMP entirely.
> > > 
> > > If you are of the opinion that every user of QEMU should be copy/pasting
> > > JSON straight into a socket and we should delete qmp-shell, that's
> > > certainly a fine opinion.
> > 
> > I think part of the pain of qmp-shell comes from the very fact that
> > it is trying to be an interactive shell. This points people towards
> > interactively typing in the commands, which is horrific when you get
> > anywhere near the JSON, or even dot-notation traditional commands.
> > 
> > If it was just a qmp-client that was single shot, we'd encourage
> > people to create the JSON in a sensible way - vim/emacs/whatever.
> 
> I don't see how this is sensible. QMP commands are something that I
> reuse even less than VM configurations, so creating a one-off file for
> each would take me a lot more time and I would still have to type the
> same JSON code that I have to type with -qmp stdio.
> 
> The reason it is and should be an interactive shell is that I'm
> interacting with it. Switching back and forth between a text editor and
> a shell to actually send the command to QEMU would make things only even
> more cumbersome than they already are.
> 
> > Bash/dash/zsh/$whatever is their interactive shell, with massively
> > more features than qmp-shell. You have command history, autocomplete,
> > conditional and looping constructs, and everything a normal shell
> > offers.
> 
> If I wanted to program a QMP client, I would use Python. For me,
> conditionals and loops are completely out of scope for a QMP shell. I
> just want an easy way to tell QEMU to do something specific.
> 
> A command history already exists for qmp-shell. It's better than bash
> because it doesn't mix QMP history with whatever else I do on my
> computer.
> 
> Autocomplete in qmp-shell doesn't exist, as far as I know, but if
> implemented, it could be a lot more useful than bash completion because
> it could offer key completion based on the QMP schema.
> 
> This is in fact a big part of the problem that qmp-shell really needs to
> solve before it can replace HMP: How to make writing commands at least
> almost as simple as with HMP. If I can just press tab a few times to
> cycle through the available options for the command, that would already
> be a massive improvement over writing JSON manually (which you would
> still have to do with your text-file based approach, without any
> QMP-specific support).

Doing all that in a python process (i.e. an actual python shell with a
bunch of qemu commands added) seems easyish.

> The other part that it needs to solve is how to be available by default
> without specifying anything on the command line. Basically, if I press
> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> implemented internally or by an external Python process, I don't mind.

That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
use HMP on stdin).

Dave

> > The only strong reason for qmp-shell to be interactive would be if
> > the initial protoocl handshake was too slow. I can't see that being
> > a problem with QMP.
> 
> Speed would be the least of my concerns. This is about manual use, and
> it already takes me a while to type in my commands.
> 
> > Example usage:
> > 
> > 1. Launch the QEMU runtime for the desired target
> > 
> >      $ qemu-runtime-x86_64 myvm.sock
> > 
> > 2. Load the configuration to define the VM
> > 
> >    $ cat myvm.yaml
> >    commands:
> >      - machine_declare:
> >          name: pc-q35-5.0
> > 	 ...
> >      - blockdev_add:
> >          ...
> >      - device_add:
> >          ...
> >      - blockdev_add:
> >          ...
> >      - device_add:
> >          ...
> >    $ qemu-client myvm.sock myvm.yaml
> > 
> > 
> > 3. Hotplug a disk
> > 
> >    $ cat mynewdisk.yaml
> >    commands:
> >      - blockdev_add:
> >          ...
> >      - device_add:
> >          ...
> >    $ qemu-client myvm.sock mynewdisk.yaml
> > 
> > 
> > 3. Hotunplug a disk
> > 
> >    $ cat myolddisk.yaml
> >    commands:
> >      - device_del:
> >          ...
> >      - blockdev_del:
> >          ...
> >    $ qemu-client myvm.sock myolddisk.yaml
> 
> Just to compare, this is what the human user oriented flow looks like
> today:
> 
> 1. qemu-system-x86_64 -M pc-q35-5.0 -drive if=virtio,... -cdrom ...
> 
> 2. <Press Ctrl-Alt-2 to get to the HMP shell>
>    (qemu) drive_add ...
>    <Press Ctrl-Alt-1 to get back to the guest>
> 
> 3. <Press Ctrl-Alt-2 to get to the HMP shell>
>    (qemu) device_del ...
>    <Press Ctrl-Alt-1 to get back to the guest>
> 
> This is what we're competing with, and honestly I don't see how your
> qemu-runtime-*/qemu-client based flow comes even close to it in terms of
> usability.
> 
> QMP, JSON and YAML may be nice machine interfaces, but having nice
> machine interfaces doesn't mean that you shouldn't also have something
> that is suitable for humans. qmp-shell is trying to be that, and while
> it leaves much to be desired in its current state, replacing it with
> even more machine-friendly stuff that is cumbersome for humans isn't the
> right answer.
> 
> Kevin
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 27.01.2020 um 21:29 hat Dr. David Alan Gilbert geschrieben:
> * Kevin Wolf (kwolf@redhat.com) wrote:
> > A command history already exists for qmp-shell. It's better than bash
> > because it doesn't mix QMP history with whatever else I do on my
> > computer.
> > 
> > Autocomplete in qmp-shell doesn't exist, as far as I know, but if
> > implemented, it could be a lot more useful than bash completion because
> > it could offer key completion based on the QMP schema.
> > 
> > This is in fact a big part of the problem that qmp-shell really needs to
> > solve before it can replace HMP: How to make writing commands at least
> > almost as simple as with HMP. If I can just press tab a few times to
> > cycle through the available options for the command, that would already
> > be a massive improvement over writing JSON manually (which you would
> > still have to do with your text-file based approach, without any
> > QMP-specific support).
> 
> Doing all that in a python process (i.e. an actual python shell with a
> bunch of qemu commands added) seems easyish.

It does. Polishing everything for the perfect experience could be a bit
more work, but just querying the schema and providing some
autocompletion from it sounds easy enough.

> > The other part that it needs to solve is how to be available by default
> > without specifying anything on the command line. Basically, if I press
> > Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> > implemented internally or by an external Python process, I don't mind.
> 
> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> use HMP on stdin).

I don't think it would be that hard, actually.

If you have a -qmp-shell option that takes a chardev and defaults to vc,
you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
to do is launch the Python child process, pass it a pair of pipes for
communication and forward everything between the pipes and the chardev.

(That's the theory anyway.)

Kevin

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
> > > The other part that it needs to solve is how to be available by default
> > > without specifying anything on the command line. Basically, if I press
> > > Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> > > implemented internally or by an external Python process, I don't mind.
> > 
> > That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> > use HMP on stdin).
> 
> I don't think it would be that hard, actually.
> 
> If you have a -qmp-shell option that takes a chardev and defaults to vc,
> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
> to do is launch the Python child process, pass it a pair of pipes for
> communication and forward everything between the pipes and the chardev.
> 
> (That's the theory anyway.)

If someone is interested, I did a quick proof-of-concept hack:

    https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell

It doesn't clean up anything properly (including the qmp-shell processes
it starts), but it spawns a usable qmp-shell on a user-specified
character device. stdio seems to work, though without readline
functionality (I suppose I still have line-buffering somewhere), vc
doesn't really work at all yet.

Try it out like this:

    $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
    monitor_qmp_event: 1
    Welcome to the QMP low-level shell!
    Connected to QEMU 4.2.50

    (QEMU) query-version
    {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
    (QEMU) quit

(Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
refactorings in the storage daemon branch, so why not try both at once?)

Polishing this to make it mergable would still require substantial work,
so at the moment I'm not planning to do this. But if someone wants to
pick it up, feel free (just let us know).

Hm, in fact... A qmp-shell GSoC project?

Kevin

qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications)

[John Snow]

I'm forking the subject as I believe Markus wanted to focus on the
machine interface aspect.

I feel that a new human interface is *related* to that goal: the
splitting of, and commitment to, separate human and machine interfaces
powered by a single root schema.

I am a big believer in making QEMU usable directly to human users as I
feel the pipeline of "tinker to deployment" is important for a
successful project, for many reasons:

- QEMU should be easy to pick up and learn.

- Supporting QEMU's use directly as an "end-user" program increases
proficiency in the user population at large, which (can) lead to better
answers and engagement on e.g. Reddit, StackOverflow, IRC

- Evolving deployments from QEMU-only to libvirt+ or above (RHV, oVirt,
kubevirt) should be a smooth and gradual process as additional
complexity is desired.

- Focusing on QEMU's usability allows our project to be consumed easier
by new cloud-focused projects. If they are already familiar with (and
happy with) our project, it is more likely to be used instead of seeking
out alternatives. This is about reducing friction.

So, for those reasons ... even though I feel that a machine-focused API
is our #1 priority as it caters to our existing users, we should also
focus on what it will take to grow mindshare for QEMU's value in the
ecosystem.

Slick interfaces and documentation go a long, long way to doing that.

So: I feel that any new machine-only paradigm or overhaul needs to be
accompanied with some new sugar to help the medicine go down, so-to-speak.

On 2/5/20 8:09 AM, Kevin Wolf wrote:
> Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
>>>> The other part that it needs to solve is how to be available by default
>>>> without specifying anything on the command line. Basically, if I press
>>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
>>>> implemented internally or by an external Python process, I don't mind.
>>>
>>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
>>> use HMP on stdin).
>>
>> I don't think it would be that hard, actually.
>>
>> If you have a -qmp-shell option that takes a chardev and defaults to vc,
>> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
>> to do is launch the Python child process, pass it a pair of pipes for
>> communication and forward everything between the pipes and the chardev.
>>
>> (That's the theory anyway.)
> 
> If someone is interested, I did a quick proof-of-concept hack:
> 
>     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> 
> It doesn't clean up anything properly (including the qmp-shell processes
> it starts), but it spawns a usable qmp-shell on a user-specified
> character device. stdio seems to work, though without readline
> functionality (I suppose I still have line-buffering somewhere), vc
> doesn't really work at all yet.
> 
> Try it out like this:
> 
>     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
>     monitor_qmp_event: 1
>     Welcome to the QMP low-level shell!
>     Connected to QEMU 4.2.50
> 
>     (QEMU) query-version
>     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
>     (QEMU) quit
> 
> (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> refactorings in the storage daemon branch, so why not try both at once?)
> 
> Polishing this to make it mergable would still require substantial work,
> so at the moment I'm not planning to do this. But if someone wants to
> pick it up, feel free (just let us know).
> 
> Hm, in fact... A qmp-shell GSoC project?
> 

That would be great. I worry that we should have a clear vision for the
syntax before we give this project to an intern, though. With a clear
vision and an outline for deliverables, it's an incredibly appropriate
project.

Some things I think we want to define before we start:

1. What are we trying to achieve with a standalone shell?
2. What syntax should it use?

I think those are the hardest parts.


Below, some musings:

- An integrated QMP shell would be a great usability boost to users of
bare QEMU.

- It is undesirable in general to support two interfaces. Feature
disparity is a problem, as is needing to document and test two separate
interfaces. The quality disparity between the two is also an issue.

- Offering HMP via the GTK interface but not QMP is a discoverability
problem. Unfamiliar users might assume that HMP is our flagship
interface. It is not.

- We are unlikely to re-expand HMP to cover everything QMP does; writing
a QMP shell that makes QMP easy to interface with is a better solution
for removing redundancy and complexity.

- I suspect that the target audience for users of naked QEMU are:
  - QEMU developers
  - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
researching, testing, and debugging integration.
  - Devops professionals testing, implementing and debugging
    configuration & infrastructure
  - Security/infosec researchers
  - Embedded platform developers
  - Academic researchers



So please correct me if I am off the mark;

Design Goals:
  - The removal of HMP
  - An easy-to-use interface that remains reasonably "close" to the
machine API such that it provides a smooth transition to scripting QEMU.
  - Integration with our GTK interface for discoverability and convenience

Syntax:
  - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
should be replaced? If yes, what should it look like?

Re: qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications)

[Dr. David Alan Gilbert]

* John Snow (jsnow@redhat.com) wrote:
> I'm forking the subject as I believe Markus wanted to focus on the
> machine interface aspect.
> 
> I feel that a new human interface is *related* to that goal: the
> splitting of, and commitment to, separate human and machine interfaces
> powered by a single root schema.
> 
> I am a big believer in making QEMU usable directly to human users as I
> feel the pipeline of "tinker to deployment" is important for a
> successful project, for many reasons:
> 
> - QEMU should be easy to pick up and learn.
> 
> - Supporting QEMU's use directly as an "end-user" program increases
> proficiency in the user population at large, which (can) lead to better
> answers and engagement on e.g. Reddit, StackOverflow, IRC
> 
> - Evolving deployments from QEMU-only to libvirt+ or above (RHV, oVirt,
> kubevirt) should be a smooth and gradual process as additional
> complexity is desired.
> 
> - Focusing on QEMU's usability allows our project to be consumed easier
> by new cloud-focused projects. If they are already familiar with (and
> happy with) our project, it is more likely to be used instead of seeking
> out alternatives. This is about reducing friction.
> 
> So, for those reasons ... even though I feel that a machine-focused API
> is our #1 priority as it caters to our existing users, we should also
> focus on what it will take to grow mindshare for QEMU's value in the
> ecosystem.
> 
> Slick interfaces and documentation go a long, long way to doing that.
> 
> So: I feel that any new machine-only paradigm or overhaul needs to be
> accompanied with some new sugar to help the medicine go down, so-to-speak.
> 
> On 2/5/20 8:09 AM, Kevin Wolf wrote:
> > Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
> >>>> The other part that it needs to solve is how to be available by default
> >>>> without specifying anything on the command line. Basically, if I press
> >>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> >>>> implemented internally or by an external Python process, I don't mind.
> >>>
> >>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> >>> use HMP on stdin).
> >>
> >> I don't think it would be that hard, actually.
> >>
> >> If you have a -qmp-shell option that takes a chardev and defaults to vc,
> >> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
> >> to do is launch the Python child process, pass it a pair of pipes for
> >> communication and forward everything between the pipes and the chardev.
> >>
> >> (That's the theory anyway.)
> > 
> > If someone is interested, I did a quick proof-of-concept hack:
> > 
> >     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> > 
> > It doesn't clean up anything properly (including the qmp-shell processes
> > it starts), but it spawns a usable qmp-shell on a user-specified
> > character device. stdio seems to work, though without readline
> > functionality (I suppose I still have line-buffering somewhere), vc
> > doesn't really work at all yet.
> > 
> > Try it out like this:
> > 
> >     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
> >     monitor_qmp_event: 1
> >     Welcome to the QMP low-level shell!
> >     Connected to QEMU 4.2.50
> > 
> >     (QEMU) query-version
> >     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
> >     (QEMU) quit
> > 
> > (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> > refactorings in the storage daemon branch, so why not try both at once?)
> > 
> > Polishing this to make it mergable would still require substantial work,
> > so at the moment I'm not planning to do this. But if someone wants to
> > pick it up, feel free (just let us know).
> > 
> > Hm, in fact... A qmp-shell GSoC project?
> > 
> 
> That would be great. I worry that we should have a clear vision for the
> syntax before we give this project to an intern, though. With a clear
> vision and an outline for deliverables, it's an incredibly appropriate
> project.
> 
> Some things I think we want to define before we start:
> 
> 1. What are we trying to achieve with a standalone shell?
> 2. What syntax should it use?
> 
> I think those are the hardest parts.
> 
> 
> Below, some musings:
> 
> - An integrated QMP shell would be a great usability boost to users of
> bare QEMU.
> 
> - It is undesirable in general to support two interfaces. Feature
> disparity is a problem, as is needing to document and test two separate
> interfaces. The quality disparity between the two is also an issue.
> 
> - Offering HMP via the GTK interface but not QMP is a discoverability
> problem. Unfamiliar users might assume that HMP is our flagship
> interface. It is not.
> 
> - We are unlikely to re-expand HMP to cover everything QMP does; writing
> a QMP shell that makes QMP easy to interface with is a better solution
> for removing redundancy and complexity.
> 
> - I suspect that the target audience for users of naked QEMU are:
>   - QEMU developers
>   - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
> researching, testing, and debugging integration.
>   - Devops professionals testing, implementing and debugging
>     configuration & infrastructure
>   - Security/infosec researchers
>   - Embedded platform developers
>   - Academic researchers
> 
> 
> 
> So please correct me if I am off the mark;
> 
> Design Goals:
>   - The removal of HMP
>   - An easy-to-use interface that remains reasonably "close" to the
> machine API such that it provides a smooth transition to scripting QEMU.
>   - Integration with our GTK interface for discoverability and convenience
> 
> Syntax:
>   - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
> should be replaced? If yes, what should it look like?

I believe it should be a python shell with added commands.

Simple things should be simple.
  e.g. adding a disk from a local file should be trivial.

Complex things can be complex - but it would be better if they were
simple.

  It's OK if the worst case of a blockdev is a bit hairy, but
  watch out for cases where the hairyness creeps in unnecessarily.

If the user screwsup, it should give an error that prompts the user
to the parameter they got wrong.

Output from commands should normally be pretty formatted (with an option
to display raw json for those needing it).
  e.g. that 'query-version' should give either just the package
  version (as info version currently does) or:
      4.2.50  Package: v4.2.0-1188-gd95a3885a9

We shouldn't lose any HMP commands that some people find useful
  Ditching HMP isn't an option until we've got almost all of it
  covered.

Dave

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: qmp-shell for GSoC/Outreachy?

[Markus Armbruster]

"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:

> * John Snow (jsnow@redhat.com) wrote:
>> I'm forking the subject as I believe Markus wanted to focus on the
>> machine interface aspect.
>> 
>> I feel that a new human interface is *related* to that goal: the
>> splitting of, and commitment to, separate human and machine interfaces
>> powered by a single root schema.

A bit of history.

QMP initially shared HMP's "schema": qemu-monitor.hx.  The command
handler was either a traditional HMP one, or a QMP-enabling pair (QMP
handler, HMP formatter).  The idea was to convert traditional HMP
handlers one by one, then ditch support for them.

With a QMP handler, HMP became a wrapper around QMP.  The input wrapping
was data-driven: @args_type specifies how to map HMP arguments to QMP.
The output wrapper was code, namely the HMP formatter.

This design turned out to tie QMP to HMP too tightly.  It assumes QMP
and HMP commands are identical apart from argument syntax and output
formatting.  They often should not be.  QMP wants building blocks:
simple commands with simple replies, in particular simple failure modes.
HMP wants convenience.  QMP wants rigor.  HMP has uses where that's a
painful and unnecessary.

So we split qemu-monitor.hx into hmp-commands.hx and qmp-commands.hx.
The former reverted back to the pre-QMP state, and the latter lost
support for HMP wrappers.  QMP was liberated from having to reimplement
HMP.  HMP was liberated from always having to do QMP first.

qmp-commands.hx was eventually replaced by the QAPI schema.

The lesson here is that to make "powered by a single root schema" work
well, we'll likely have to put in more smarts than we did back then.

More on that below in reply to David's reply.

>> I am a big believer in making QEMU usable directly to human users as I
>> feel the pipeline of "tinker to deployment" is important for a
>> successful project, for many reasons:
>> 
>> - QEMU should be easy to pick up and learn.
>> 
>> - Supporting QEMU's use directly as an "end-user" program increases
>> proficiency in the user population at large, which (can) lead to better
>> answers and engagement on e.g. Reddit, StackOverflow, IRC
>> 
>> - Evolving deployments from QEMU-only to libvirt+ or above (RHV, oVirt,
>> kubevirt) should be a smooth and gradual process as additional
>> complexity is desired.
>> 
>> - Focusing on QEMU's usability allows our project to be consumed easier
>> by new cloud-focused projects. If they are already familiar with (and
>> happy with) our project, it is more likely to be used instead of seeking
>> out alternatives. This is about reducing friction.
>> 
>> So, for those reasons ... even though I feel that a machine-focused API
>> is our #1 priority as it caters to our existing users, we should also
>> focus on what it will take to grow mindshare for QEMU's value in the
>> ecosystem.
>> 
>> Slick interfaces and documentation go a long, long way to doing that.
>> 
>> So: I feel that any new machine-only paradigm or overhaul needs to be
>> accompanied with some new sugar to help the medicine go down, so-to-speak.

Points taken.

>> On 2/5/20 8:09 AM, Kevin Wolf wrote:
>> > Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
>> >>>> The other part that it needs to solve is how to be available by default
>> >>>> without specifying anything on the command line. Basically, if I press
>> >>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
>> >>>> implemented internally or by an external Python process, I don't mind.
>> >>>
>> >>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
>> >>> use HMP on stdin).
>> >>
>> >> I don't think it would be that hard, actually.
>> >>
>> >> If you have a -qmp-shell option that takes a chardev and defaults to vc,
>> >> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
>> >> to do is launch the Python child process, pass it a pair of pipes for
>> >> communication and forward everything between the pipes and the chardev.
>> >>
>> >> (That's the theory anyway.)
>> > 
>> > If someone is interested, I did a quick proof-of-concept hack:
>> > 
>> >     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
>> > 
>> > It doesn't clean up anything properly (including the qmp-shell processes
>> > it starts), but it spawns a usable qmp-shell on a user-specified
>> > character device. stdio seems to work, though without readline
>> > functionality (I suppose I still have line-buffering somewhere), vc
>> > doesn't really work at all yet.
>> > 
>> > Try it out like this:
>> > 
>> >     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
>> >     monitor_qmp_event: 1
>> >     Welcome to the QMP low-level shell!
>> >     Connected to QEMU 4.2.50
>> > 
>> >     (QEMU) query-version
>> >     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
>> >     (QEMU) quit
>> > 
>> > (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
>> > refactorings in the storage daemon branch, so why not try both at once?)
>> > 
>> > Polishing this to make it mergable would still require substantial work,
>> > so at the moment I'm not planning to do this. But if someone wants to
>> > pick it up, feel free (just let us know).
>> > 
>> > Hm, in fact... A qmp-shell GSoC project?
>> > 
>> 
>> That would be great. I worry that we should have a clear vision for the
>> syntax before we give this project to an intern, though. With a clear
>> vision and an outline for deliverables, it's an incredibly appropriate
>> project.
>> 
>> Some things I think we want to define before we start:
>> 
>> 1. What are we trying to achieve with a standalone shell?

Projects without a clear goal rarely succeed.  Success within three
months is even rarer.

>> 2. What syntax should it use?

Leaving that to a GSoC student amounts to setting up for failure.

>> I think those are the hardest parts.
>>
>> Below, some musings:
>> 
>> - An integrated QMP shell would be a great usability boost to users of
>> bare QEMU.
>> 
>> - It is undesirable in general to support two interfaces. Feature
>> disparity is a problem, as is needing to document and test two separate
>> interfaces. The quality disparity between the two is also an issue.
>> 
>> - Offering HMP via the GTK interface but not QMP is a discoverability
>> problem. Unfamiliar users might assume that HMP is our flagship
>> interface. It is not.
>> 
>> - We are unlikely to re-expand HMP to cover everything QMP does; writing
>> a QMP shell that makes QMP easy to interface with is a better solution
>> for removing redundancy and complexity.
>> 
>> - I suspect that the target audience for users of naked QEMU are:
>>   - QEMU developers
>>   - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
>> researching, testing, and debugging integration.
>>   - Devops professionals testing, implementing and debugging
>>     configuration & infrastructure
>>   - Security/infosec researchers
>>   - Embedded platform developers
>>   - Academic researchers
>> 
>> 
>> 
>> So please correct me if I am off the mark;
>> 
>> Design Goals:
>>   - The removal of HMP
>>   - An easy-to-use interface that remains reasonably "close" to the
>> machine API such that it provides a smooth transition to scripting QEMU.
>>   - Integration with our GTK interface for discoverability and convenience
>> 
>> Syntax:
>>   - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
>> should be replaced? If yes, what should it look like?
>
> I believe it should be a python shell with added commands.
>
> Simple things should be simple.
>   e.g. adding a disk from a local file should be trivial.
>
> Complex things can be complex - but it would be better if they were
> simple.
>
>   It's OK if the worst case of a blockdev is a bit hairy, but
>   watch out for cases where the hairyness creeps in unnecessarily.

Designing interfaces to complex machinery is hard.  Experience tells
that we do okay when we focus on the building blocks first.  That's
-blockdev.  When we start with trying to make simple things simple, we
end in swamps.  That's -drive.

Focus on building blocks is of course no excuse for unnecessary
hairiness.

It's also no reason not to build more convenient things on top of the
building blocks.  I doubt they should go into QMP, though.

> If the user screwsup, it should give an error that prompts the user
> to the parameter they got wrong.
>
> Output from commands should normally be pretty formatted (with an option
> to display raw json for those needing it).
>   e.g. that 'query-version' should give either just the package
>   version (as info version currently does) or:
>       4.2.50  Package: v4.2.0-1188-gd95a3885a9
>
> We shouldn't lose any HMP commands that some people find useful
>   Ditching HMP isn't an option until we've got almost all of it
>   covered.

In particular, we currently use HMP for debugging and monitoring
purposes, where we don't need or want QMP's rigor, neither its rigorous
interface stability, nor its structured I/O.  We want the "whipuptitude"
we get from monitor_printf().  This is actually a point David has made
several times.

To have a qmp-shell replace HMP, I think it needs to be able to

* Go beyond 1:1

  We tried a 1:1 mapping between HMP and QMP commands, and it didn't
  work out.  HMP's replacement should let us build convenient commands
  from QMP building blocks.

  We tried a 1:1 mapping between HMP and QMP command arguments, guided
  by @args_type.  Worked out for simple cases, but was too constricting.

* Preserve "whipuptitude" [David]

  I figure that means allowing some in QMP.  Without compromising its
  core mission, of course.

* As discoverable as HMP is now [Kevin]

* Help, completion and such at least on par with what HMP provides now

Highly desirable:

* Support transitioning to the machine interface [John]

  Let humans start playing with the human interface, and when they feel
  the need to automate, help them transition to QMP.

Back to John's question on qmp-shell syntax, which hasn't been answered
so far.

JSON is a data-interchange format.  It doesn't try to be a configuration
format or programming language syntax for human use.  It gets pressed
into these roles with entirely predictable poor results.

Pain points of JSON include having to count parenthesises and having to
quote so bloody much.  Additional QMP pain points include long names and
excessive nesting.

For the configuration format role, more usable alternatives exist.  YAML
is a popular one.

qmp-shell is a REPL.  It needs a REPL-friendly syntax.  I doubt YAML is
or even tries to be REPL-friendly.  I'd love to be wrong; the first rule
of language design is "don't".

Other language suggestions?

On making JSON suck less in this role:

LISP REPLs demonstrate that computers can assist effectively with
counting parenthesises, and with completing long names.

We could make quoting optional for sufficiently nice object member
names.  QAPI naming rules ensure niceness, actually.

We could make quoting optional for certain string literals.  Simple
enough for literals that can only be a string, like abc.  For literals
that could be something else like 123 or true, omitting quotes creates
ambiguity.  When the schema accepts only one of the possible types, the
ambiguity goes away.  Complexity stays, however.

Excessive nesting should ideally be attacked in QMP itself, but backward
compatibility makes that hard.

Re: qmp-shell for GSoC/Outreachy?

[Daniel P. Berrangé]

On Thu, Feb 06, 2020 at 10:40:37AM +0100, Markus Armbruster wrote:
> > If the user screwsup, it should give an error that prompts the user
> > to the parameter they got wrong.
> >
> > Output from commands should normally be pretty formatted (with an option
> > to display raw json for those needing it).
> >   e.g. that 'query-version' should give either just the package
> >   version (as info version currently does) or:
> >       4.2.50  Package: v4.2.0-1188-gd95a3885a9
> >
> > We shouldn't lose any HMP commands that some people find useful
> >   Ditching HMP isn't an option until we've got almost all of it
> >   covered.
> 
> In particular, we currently use HMP for debugging and monitoring
> purposes, where we don't need or want QMP's rigor, neither its rigorous
> interface stability, nor its structured I/O.  We want the "whipuptitude"
> we get from monitor_printf().  This is actually a point David has made
> several times.

I'd like to argue that this decision to keep these debugging/monitoring
things in HMP only was a mistake, because it ensures that QEMU internals
need to keep HMP related code forever.

What we actually need is a part of QMP that does not have the long term
stability requirement, nor need for fully structured data. In fact this
pretty much already exists - we have declared the 'x-' prefix as a way
to model QMP commands which are experimental / suboptimal / subject
to change.

I suggest that every HMP command which does not have a QMP equivalent
should be turned into a QMP command with an "x-" prefix, with no
extra modelling applied

Take "info block"

(hmp) info block
ide1-cd0: [not inserted]
    Attached to:      /machine/unattached/device[23]
    Removable device: not locked, tray closed

floppy0: [not inserted]
    Attached to:      /machine/unattached/device[16]
    Removable device: not locked, tray closed

sd0: [not inserted]
    Removable device: not locked, tray closed


I suggest we support it as "x-query-block"

(qmp) x-query-block
{
    "return": {
        "info": "ide1-cd0: [not inserted]
    Attached to:      /machine/unattached/device[23]
    Removable device: not locked, tray closed

floppy0: [not inserted]
    Attached to:      /machine/unattached/device[16]
    Removable device: not locked, tray closed

sd0: [not inserted]
    Removable device: not locked, tray closed"
    }
}


Functionally we in fact already support pretty much exactly
that via the "human-monitor-command"  QMP command.

The difference is that with the latter, we will still have to
keep around the internal dispatching machinery for HMP inside
QEMU forever. If we transplant all remaining HMP commands with
an "x-" prefix, we open up the possibility of completely
separating HMP out and having QEMU work exclusively with QMP
internally.

This is complementary to an improved qmp-shell client.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: qmp-shell for GSoC/Outreachy?

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Feb 06, 2020 at 10:40:37AM +0100, Markus Armbruster wrote:
>> > If the user screwsup, it should give an error that prompts the user
>> > to the parameter they got wrong.
>> >
>> > Output from commands should normally be pretty formatted (with an option
>> > to display raw json for those needing it).
>> >   e.g. that 'query-version' should give either just the package
>> >   version (as info version currently does) or:
>> >       4.2.50  Package: v4.2.0-1188-gd95a3885a9
>> >
>> > We shouldn't lose any HMP commands that some people find useful
>> >   Ditching HMP isn't an option until we've got almost all of it
>> >   covered.
>> 
>> In particular, we currently use HMP for debugging and monitoring
>> purposes, where we don't need or want QMP's rigor, neither its rigorous
>> interface stability, nor its structured I/O.  We want the "whipuptitude"
>> we get from monitor_printf().  This is actually a point David has made
>> several times.
>
> I'd like to argue that this decision to keep these debugging/monitoring
> things in HMP only was a mistake, because it ensures that QEMU internals
> need to keep HMP related code forever.
>
> What we actually need is a part of QMP that does not have the long term
> stability requirement, nor need for fully structured data. In fact this
> pretty much already exists - we have declared the 'x-' prefix as a way
> to model QMP commands which are experimental / suboptimal / subject
> to change.
>
> I suggest that every HMP command which does not have a QMP equivalent
> should be turned into a QMP command with an "x-" prefix, with no
> extra modelling applied

Makes sense (see my point about "allowing some [whipuptitude] in QMP"),
except I disagree with your example:

> Take "info block"
>
> (hmp) info block
> ide1-cd0: [not inserted]
>     Attached to:      /machine/unattached/device[23]
>     Removable device: not locked, tray closed
>
> floppy0: [not inserted]
>     Attached to:      /machine/unattached/device[16]
>     Removable device: not locked, tray closed
>
> sd0: [not inserted]
>     Removable device: not locked, tray closed
>
>
> I suggest we support it as "x-query-block"
>
> (qmp) x-query-block
> {
>     "return": {
>         "info": "ide1-cd0: [not inserted]
>     Attached to:      /machine/unattached/device[23]
>     Removable device: not locked, tray closed
>
> floppy0: [not inserted]
>     Attached to:      /machine/unattached/device[16]
>     Removable device: not locked, tray closed
>
> sd0: [not inserted]
>     Removable device: not locked, tray closed"
>     }
> }

This commmand does have a QMP equivalent: query-block.

Hmm, no more.  It actually wraps around both query-block and
query-named-block-nodes now.  I think that makes it an example of "go
beyond 1:1".

A better example for "allowing whipuptitude" would be "info registers".

> Functionally we in fact already support pretty much exactly
> that via the "human-monitor-command"  QMP command.
>
> The difference is that with the latter, we will still have to
> keep around the internal dispatching machinery for HMP inside
> QEMU forever. If we transplant all remaining HMP commands with
> an "x-" prefix, we open up the possibility of completely
> separating HMP out and having QEMU work exclusively with QMP
> internally.
>
> This is complementary to an improved qmp-shell client.

Yes.

Re: qmp-shell for GSoC/Outreachy?

[Daniel P. Berrangé]

On Thu, Feb 06, 2020 at 01:11:58PM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Thu, Feb 06, 2020 at 10:40:37AM +0100, Markus Armbruster wrote:
> >> > If the user screwsup, it should give an error that prompts the user
> >> > to the parameter they got wrong.
> >> >
> >> > Output from commands should normally be pretty formatted (with an option
> >> > to display raw json for those needing it).
> >> >   e.g. that 'query-version' should give either just the package
> >> >   version (as info version currently does) or:
> >> >       4.2.50  Package: v4.2.0-1188-gd95a3885a9
> >> >
> >> > We shouldn't lose any HMP commands that some people find useful
> >> >   Ditching HMP isn't an option until we've got almost all of it
> >> >   covered.
> >> 
> >> In particular, we currently use HMP for debugging and monitoring
> >> purposes, where we don't need or want QMP's rigor, neither its rigorous
> >> interface stability, nor its structured I/O.  We want the "whipuptitude"
> >> we get from monitor_printf().  This is actually a point David has made
> >> several times.
> >
> > I'd like to argue that this decision to keep these debugging/monitoring
> > things in HMP only was a mistake, because it ensures that QEMU internals
> > need to keep HMP related code forever.
> >
> > What we actually need is a part of QMP that does not have the long term
> > stability requirement, nor need for fully structured data. In fact this
> > pretty much already exists - we have declared the 'x-' prefix as a way
> > to model QMP commands which are experimental / suboptimal / subject
> > to change.
> >
> > I suggest that every HMP command which does not have a QMP equivalent
> > should be turned into a QMP command with an "x-" prefix, with no
> > extra modelling applied
> 
> Makes sense (see my point about "allowing some [whipuptitude] in QMP"),
> except I disagree with your example:
> 
> > Take "info block"
> >
> > (hmp) info block
> > ide1-cd0: [not inserted]
> >     Attached to:      /machine/unattached/device[23]
> >     Removable device: not locked, tray closed
> >
> > floppy0: [not inserted]
> >     Attached to:      /machine/unattached/device[16]
> >     Removable device: not locked, tray closed
> >
> > sd0: [not inserted]
> >     Removable device: not locked, tray closed
> >
> >
> > I suggest we support it as "x-query-block"
> >
> > (qmp) x-query-block
> > {
> >     "return": {
> >         "info": "ide1-cd0: [not inserted]
> >     Attached to:      /machine/unattached/device[23]
> >     Removable device: not locked, tray closed
> >
> > floppy0: [not inserted]
> >     Attached to:      /machine/unattached/device[16]
> >     Removable device: not locked, tray closed
> >
> > sd0: [not inserted]
> >     Removable device: not locked, tray closed"
> >     }
> > }
> 
> This commmand does have a QMP equivalent: query-block.

Doh, I should have actually checked before picking a random
example :-)

> 
> Hmm, no more.  It actually wraps around both query-block and
> query-named-block-nodes now.  I think that makes it an example of "go
> beyond 1:1".
> 
> A better example for "allowing whipuptitude" would be "info registers".

Yep, that's a classic that would be horribly painful to try to represent
as a fully structured set of arrays & dicts for all architectures.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: qmp-shell for GSoC/Outreachy?

[Dr. David Alan Gilbert]

* Daniel P. Berrangé (berrange@redhat.com) wrote:
> On Thu, Feb 06, 2020 at 01:11:58PM +0100, Markus Armbruster wrote:
> > Daniel P. Berrangé <berrange@redhat.com> writes:
> > 
> > > On Thu, Feb 06, 2020 at 10:40:37AM +0100, Markus Armbruster wrote:
> > >> > If the user screwsup, it should give an error that prompts the user
> > >> > to the parameter they got wrong.
> > >> >
> > >> > Output from commands should normally be pretty formatted (with an option
> > >> > to display raw json for those needing it).
> > >> >   e.g. that 'query-version' should give either just the package
> > >> >   version (as info version currently does) or:
> > >> >       4.2.50  Package: v4.2.0-1188-gd95a3885a9
> > >> >
> > >> > We shouldn't lose any HMP commands that some people find useful
> > >> >   Ditching HMP isn't an option until we've got almost all of it
> > >> >   covered.
> > >> 
> > >> In particular, we currently use HMP for debugging and monitoring
> > >> purposes, where we don't need or want QMP's rigor, neither its rigorous
> > >> interface stability, nor its structured I/O.  We want the "whipuptitude"
> > >> we get from monitor_printf().  This is actually a point David has made
> > >> several times.
> > >
> > > I'd like to argue that this decision to keep these debugging/monitoring
> > > things in HMP only was a mistake, because it ensures that QEMU internals
> > > need to keep HMP related code forever.
> > >
> > > What we actually need is a part of QMP that does not have the long term
> > > stability requirement, nor need for fully structured data. In fact this
> > > pretty much already exists - we have declared the 'x-' prefix as a way
> > > to model QMP commands which are experimental / suboptimal / subject
> > > to change.
> > >
> > > I suggest that every HMP command which does not have a QMP equivalent
> > > should be turned into a QMP command with an "x-" prefix, with no
> > > extra modelling applied
> > 
> > Makes sense (see my point about "allowing some [whipuptitude] in QMP"),
> > except I disagree with your example:
> > 
> > > Take "info block"
> > >
> > > (hmp) info block
> > > ide1-cd0: [not inserted]
> > >     Attached to:      /machine/unattached/device[23]
> > >     Removable device: not locked, tray closed
> > >
> > > floppy0: [not inserted]
> > >     Attached to:      /machine/unattached/device[16]
> > >     Removable device: not locked, tray closed
> > >
> > > sd0: [not inserted]
> > >     Removable device: not locked, tray closed
> > >
> > >
> > > I suggest we support it as "x-query-block"
> > >
> > > (qmp) x-query-block
> > > {
> > >     "return": {
> > >         "info": "ide1-cd0: [not inserted]
> > >     Attached to:      /machine/unattached/device[23]
> > >     Removable device: not locked, tray closed
> > >
> > > floppy0: [not inserted]
> > >     Attached to:      /machine/unattached/device[16]
> > >     Removable device: not locked, tray closed
> > >
> > > sd0: [not inserted]
> > >     Removable device: not locked, tray closed"
> > >     }
> > > }
> > 
> > This commmand does have a QMP equivalent: query-block.
> 
> Doh, I should have actually checked before picking a random
> example :-)
> 
> > 
> > Hmm, no more.  It actually wraps around both query-block and
> > query-named-block-nodes now.  I think that makes it an example of "go
> > beyond 1:1".
> > 
> > A better example for "allowing whipuptitude" would be "info registers".
> 
> Yep, that's a classic that would be horribly painful to try to represent
> as a fully structured set of arrays & dicts for all architectures.

'info registers' is a challenge for JSON since it's all 64bit unsigned
int's;  you really want the result to be in hex, and definitely not to
have been randomly sign extended or truncated, or converted to float or
whatever other evils a json processor might do.  So the user wants
a nice consistent format.

'info qtree' and 'info mtree' are other interesting cases that I use
very frequently.

Dave



> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: qmp-shell for GSoC/Outreachy?

[John Snow]

On 2/6/20 7:11 AM, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
>> On Thu, Feb 06, 2020 at 10:40:37AM +0100, Markus Armbruster wrote:
>>>> If the user screwsup, it should give an error that prompts the user
>>>> to the parameter they got wrong.
>>>>
>>>> Output from commands should normally be pretty formatted (with an option
>>>> to display raw json for those needing it).
>>>>   e.g. that 'query-version' should give either just the package
>>>>   version (as info version currently does) or:
>>>>       4.2.50  Package: v4.2.0-1188-gd95a3885a9
>>>>
>>>> We shouldn't lose any HMP commands that some people find useful
>>>>   Ditching HMP isn't an option until we've got almost all of it
>>>>   covered.
>>>
>>> In particular, we currently use HMP for debugging and monitoring
>>> purposes, where we don't need or want QMP's rigor, neither its rigorous
>>> interface stability, nor its structured I/O.  We want the "whipuptitude"
>>> we get from monitor_printf().  This is actually a point David has made
>>> several times.
>>
>> I'd like to argue that this decision to keep these debugging/monitoring
>> things in HMP only was a mistake, because it ensures that QEMU internals
>> need to keep HMP related code forever.
>>
>> What we actually need is a part of QMP that does not have the long term
>> stability requirement, nor need for fully structured data. In fact this
>> pretty much already exists - we have declared the 'x-' prefix as a way
>> to model QMP commands which are experimental / suboptimal / subject
>> to change.
>>
>> I suggest that every HMP command which does not have a QMP equivalent
>> should be turned into a QMP command with an "x-" prefix, with no
>> extra modelling applied
> 
> Makes sense (see my point about "allowing some [whipuptitude] in QMP"),
> except I disagree with your example:
> 
>> Take "info block"
>>
>> (hmp) info block
>> ide1-cd0: [not inserted]
>>     Attached to:      /machine/unattached/device[23]
>>     Removable device: not locked, tray closed
>>
>> floppy0: [not inserted]
>>     Attached to:      /machine/unattached/device[16]
>>     Removable device: not locked, tray closed
>>
>> sd0: [not inserted]
>>     Removable device: not locked, tray closed
>>
>>
>> I suggest we support it as "x-query-block"
>>
>> (qmp) x-query-block
>> {
>>     "return": {
>>         "info": "ide1-cd0: [not inserted]
>>     Attached to:      /machine/unattached/device[23]
>>     Removable device: not locked, tray closed
>>
>> floppy0: [not inserted]
>>     Attached to:      /machine/unattached/device[16]
>>     Removable device: not locked, tray closed
>>
>> sd0: [not inserted]
>>     Removable device: not locked, tray closed"
>>     }
>> }
> 
> This commmand does have a QMP equivalent: query-block.
> 
> Hmm, no more.  It actually wraps around both query-block and
> query-named-block-nodes now.  I think that makes it an example of "go
> beyond 1:1".
> 
> A better example for "allowing whipuptitude" would be "info registers".
> 
>> Functionally we in fact already support pretty much exactly
>> that via the "human-monitor-command"  QMP command.
>>
>> The difference is that with the latter, we will still have to
>> keep around the internal dispatching machinery for HMP inside
>> QEMU forever. If we transplant all remaining HMP commands with
>> an "x-" prefix, we open up the possibility of completely
>> separating HMP out and having QEMU work exclusively with QMP
>> internally.
>>
>> This is complementary to an improved qmp-shell client.
> 
> Yes.
> 

A note:

qmp-shell could offer some sugared versions of things like "query-block"
that do some work understanding the reply from QEMU and printing it in a
nice human format.

This is generally what I have in mind for things when it's possible: use
"real" QMP interfaces to do the query, and then use qmp-shell code to
display "pretty" results.

e.g.

>> query-block

might execute the raw QMP command and give you back the giant textwall
of gibberish, but perhaps:

>> ?info block

would execute query-block behind the scenes, digest the results, and
pretty-print some results.

These shell-local commands, denoted by the "?" prefix, could be
user-extensible python extensions that simply react to incoming data,
optionally perform extra work, and display results.

Otherwise, for commands where this is too laborious, too intensive, or
just not worth it -- but we cannot be rid of them -- I like Dan's idea
of just offering e.g.

'x-debug-foo' and returning big text blobs, so we don't have to waste
any brain-calories on devising a properly structured response.

Actually, we can even use the featureflags to tag such commands as e.g.
"human-ui-only" and could conditionally compile them out for enterprise
deployments where poor behavior WRT synchronicity is a security liability.

--js

Re: qmp-shell for GSoC/Outreachy?

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> A note:
>
> qmp-shell could offer some sugared versions of things like "query-block"
> that do some work understanding the reply from QEMU and printing it in a
> nice human format.

This is exactly how HMP commands should work.

By moving them to qmp-shell, we get to code them in qmp-shell's language
(Python, I guess) rather than C.

I think we need to decide what qmp-shell's mission should be: is it a
better socat for the QMP monitor, or a replacement for the traditional
HMP monitor?

You may answer "both".  It'll complicate things, though.  More on that
below.

> This is generally what I have in mind for things when it's possible: use
> "real" QMP interfaces to do the query, and then use qmp-shell code to
> display "pretty" results.
>
> e.g.
>
>>> query-block
>
> might execute the raw QMP command and give you back the giant textwall
> of gibberish, but perhaps:
>
>>> ?info block
>
> would execute query-block behind the scenes, digest the results, and
> pretty-print some results.
>
> These shell-local commands, denoted by the "?" prefix, could be
> user-extensible python extensions that simply react to incoming data,
> optionally perform extra work, and display results.

As long as we develop qmp-shell and QMP together, we don't *have* to
separate name spaces.  It could simply be "info block".

However, consider chardev-add.  Example from char.json:

    -> { "execute" : "chardev-add",
         "arguments" : { "id" : "baz",
                         "backend" : { "type" : "pty", "data" : {} } } }
    <- { "return": { "pty" : "/dev/pty/42" } }

Here's the HMP equivalent:

    (qemu) chardev-add pty,id=baz
    char device redirected to /dev/pts/16 (label baz)

With your proposed "?" prefix, the straightforward way to run
chardev-add gives your the (thankfully non-giant) textwall of gibberish.
If you stick in a "?" and change the "-" to "_", you get the
pretty-printed version.  Not exactly the kind of UX we strive for, I'm
afraid, even if we fix the '-' vs. '_' nonsense.

If qmp-shell aims to be a better socat for the QMP monitor, then "?info
block" is out of scope, and so is pretty-printed chardev-add.

If qmp-shell aims to replace HMP, then query-block is out of scope, and
so is JSON-printing chardev-add.  Also, "qmp-shell" is a misnomer.

If qmp-shell aims to do both, we need to try harder :)

Kevin mentioned another way to provide HMP-like functionality in
qmp-shell: "have a QMP mode and an HMP mode and the user can switch
between these modes."

> Otherwise, for commands where this is too laborious, too intensive, or
> just not worth it -- but we cannot be rid of them -- I like Dan's idea
> of just offering e.g.
>
> 'x-debug-foo' and returning big text blobs, so we don't have to waste
> any brain-calories on devising a properly structured response.

To actually replace HMP, we'll want such commands.

> Actually, we can even use the featureflags to tag such commands as e.g.
> "human-ui-only" and could conditionally compile them out for enterprise
> deployments where poor behavior WRT synchronicity is a security liability.

Everybody knows enterprise deployments always work, and always perform
perfectly.  Commands for debugging and monitoring them are quite
superfluous ;-P

Re: qmp-shell for GSoC/Outreachy?

[Kevin Wolf]

Am 06.02.2020 um 10:40 hat Markus Armbruster geschrieben:
> >> On 2/5/20 8:09 AM, Kevin Wolf wrote:
> >> > Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
> >> >>>> The other part that it needs to solve is how to be available by default
> >> >>>> without specifying anything on the command line. Basically, if I press
> >> >>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> >> >>>> implemented internally or by an external Python process, I don't mind.
> >> >>>
> >> >>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> >> >>> use HMP on stdin).
> >> >>
> >> >> I don't think it would be that hard, actually.
> >> >>
> >> >> If you have a -qmp-shell option that takes a chardev and defaults to vc,
> >> >> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
> >> >> to do is launch the Python child process, pass it a pair of pipes for
> >> >> communication and forward everything between the pipes and the chardev.
> >> >>
> >> >> (That's the theory anyway.)
> >> > 
> >> > If someone is interested, I did a quick proof-of-concept hack:
> >> > 
> >> >     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> >> > 
> >> > It doesn't clean up anything properly (including the qmp-shell processes
> >> > it starts), but it spawns a usable qmp-shell on a user-specified
> >> > character device. stdio seems to work, though without readline
> >> > functionality (I suppose I still have line-buffering somewhere), vc
> >> > doesn't really work at all yet.
> >> > 
> >> > Try it out like this:
> >> > 
> >> >     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
> >> >     monitor_qmp_event: 1
> >> >     Welcome to the QMP low-level shell!
> >> >     Connected to QEMU 4.2.50
> >> > 
> >> >     (QEMU) query-version
> >> >     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
> >> >     (QEMU) quit
> >> > 
> >> > (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> >> > refactorings in the storage daemon branch, so why not try both at once?)
> >> > 
> >> > Polishing this to make it mergable would still require substantial work,
> >> > so at the moment I'm not planning to do this. But if someone wants to
> >> > pick it up, feel free (just let us know).
> >> > 
> >> > Hm, in fact... A qmp-shell GSoC project?
> >> > 
> >> 
> >> That would be great. I worry that we should have a clear vision for the
> >> syntax before we give this project to an intern, though. With a clear
> >> vision and an outline for deliverables, it's an incredibly appropriate
> >> project.
> >> 
> >> Some things I think we want to define before we start:
> >> 
> >> 1. What are we trying to achieve with a standalone shell?
> 
> Projects without a clear goal rarely succeed.  Success within three
> months is even rarer.
>
> >> 2. What syntax should it use?
> 
> Leaving that to a GSoC student amounts to setting up for failure.

I think this subthread shows that we actually have many separate
projects that people wish to have someone work on. Each of them is
probably a bit too small for a whole GSoC, but all of them together are
probably too much. So I'll guess the student would pick maybe two of
them, and if time is left at the end, more can be added as a bonus.

1. Something like --monitor mode=qmp-shell that just spawns an external
   Python script and passes it a QMP socket. This is the fundamental
   building block for having any kind of external monitor script
   actually integrated in QEMU, so I think just running the existing
   qmp-shell this way (with proper support for at least stdio and vc
   chardevs) would make sense as a first milestone.

2. Rewriting qmp-shell to use a better syntax for nested data
   structures. This would have to be defined before the project starts.

3. Improving qmp-shell UI-wise, e.g. by having better autocompletion,
   support for counting brackets, or whatever else was mentioned. We
   have a few ideas, and there's room for the student to add their own
   ideas, too.

4. Something HMP-like. This isn't QMP any more, so it could as well be a
   separate script (hmp-shell?). But it could also be integrated in
   qmp-shell in the form of additional commands that are implemented
   client-side. Or maybe have a single shell, but have a QMP mode and an
   HMP mode and the user can switch between these modes.

   The syntax for the HMP shell/mode could be the same or different from
   the QMP syntax. This would have to be defined beforehand, too.

5. Probably more that I just forgot now.

Suggesting the exact goals is part of the student application process,
but for fundamental things like the syntax we should probably already
know what we want.

> >> I think those are the hardest parts.
> >>
> >> Below, some musings:
> >> 
> >> - An integrated QMP shell would be a great usability boost to users of
> >> bare QEMU.
> >> 
> >> - It is undesirable in general to support two interfaces. Feature
> >> disparity is a problem, as is needing to document and test two separate
> >> interfaces. The quality disparity between the two is also an issue.
> >> 
> >> - Offering HMP via the GTK interface but not QMP is a discoverability
> >> problem. Unfamiliar users might assume that HMP is our flagship
> >> interface. It is not.
> >> 
> >> - We are unlikely to re-expand HMP to cover everything QMP does; writing
> >> a QMP shell that makes QMP easy to interface with is a better solution
> >> for removing redundancy and complexity.

I'm not entirely convinced of this because QMP is often too low-level to
actually address the practical high-level needs of users.

But these HMP-ish things are probably easier to maintain as scripts
outside of the QEMU binary, so I think some kind of "QMP with
extensions" for human could be the solution.

Once it's an external script, it will also be easy to exchange the shell
for another one depending on user preference, or to hack in whatever
functionality they are missing.

> >> - I suspect that the target audience for users of naked QEMU are:
> >>   - QEMU developers
> >>   - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
> >> researching, testing, and debugging integration.
> >>   - Devops professionals testing, implementing and debugging
> >>     configuration & infrastructure
> >>   - Security/infosec researchers
> >>   - Embedded platform developers
> >>   - Academic researchers

Maybe kernel developers should be mentioned separately, but yes, this
list looks plausible to me.

> >> So please correct me if I am off the mark;
> >> 
> >> Design Goals:
> >>   - The removal of HMP
> >>   - An easy-to-use interface that remains reasonably "close" to the
> >> machine API such that it provides a smooth transition to scripting QEMU.
> >>   - Integration with our GTK interface for discoverability and convenience

As I listed above, I think these are actually three separate projects,
rather than goals for a single big projects.

> >> Syntax:
> >>   - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
> >> should be replaced? If yes, what should it look like?
> >
> > I believe it should be a python shell with added commands.
> >
> > Simple things should be simple.
> >   e.g. adding a disk from a local file should be trivial.
> >
> > Complex things can be complex - but it would be better if they were
> > simple.
> >
> >   It's OK if the worst case of a blockdev is a bit hairy, but
> >   watch out for cases where the hairyness creeps in unnecessarily.
> 
> Designing interfaces to complex machinery is hard.  Experience tells
> that we do okay when we focus on the building blocks first.  That's
> -blockdev.  When we start with trying to make simple things simple, we
> end in swamps.  That's -drive.
> 
> Focus on building blocks is of course no excuse for unnecessary
> hairiness.
> 
> It's also no reason not to build more convenient things on top of the
> building blocks.  I doubt they should go into QMP, though.

Right, they should be implemented in that external script, which would
use the lower-level QMP building blocks to provide the functionality. I
also think it's a good idea to keep QMP accessible for more exotic use
cases when the simple thing just doesn't cut it any more.

> > If the user screwsup, it should give an error that prompts the user
> > to the parameter they got wrong.
> >
> > Output from commands should normally be pretty formatted (with an option
> > to display raw json for those needing it).
> >   e.g. that 'query-version' should give either just the package
> >   version (as info version currently does) or:
> >       4.2.50  Package: v4.2.0-1188-gd95a3885a9
> >
> > We shouldn't lose any HMP commands that some people find useful
> >   Ditching HMP isn't an option until we've got almost all of it
> >   covered.
> 
> In particular, we currently use HMP for debugging and monitoring
> purposes, where we don't need or want QMP's rigor, neither its rigorous
> interface stability, nor its structured I/O.  We want the "whipuptitude"
> we get from monitor_printf().  This is actually a point David has made
> several times.
> 
> To have a qmp-shell replace HMP, I think it needs to be able to
> 
> * Go beyond 1:1
> 
>   We tried a 1:1 mapping between HMP and QMP commands, and it didn't
>   work out.  HMP's replacement should let us build convenient commands
>   from QMP building blocks.
> 
>   We tried a 1:1 mapping between HMP and QMP command arguments, guided
>   by @args_type.  Worked out for simple cases, but was too constricting.

We need to go beyond 1:1, but we probably want to be able to offer 1:1
as a subset of commands accepted in that shell.

Offering only 1:1 in a good way might already be a step forward.

> * Preserve "whipuptitude" [David]
> 
>   I figure that means allowing some in QMP.  Without compromising its
>   core mission, of course.

As long as we confine it to x- commands, I think this is okay.

> * As discoverable as HMP is now [Kevin]
> 
> * Help, completion and such at least on par with what HMP provides now

Will we want to add new annotations in the schema for this?

For example, HMP has completion support for block device names. In the
QAPI schema, these are simply 'str'. We could bake the knowledge that
in command 'foo' the parameter 'bar' is a block device name, but that
would be a hack and would probably rarely be consistent with what QEMU
actually does. It's really something that schema introspection should be
able to tell us.

Kevin

Re: qmp-shell for GSoC/Outreachy?

[Dr. David Alan Gilbert]

* Kevin Wolf (kwolf@redhat.com) wrote:
> Am 06.02.2020 um 10:40 hat Markus Armbruster geschrieben:
> > >> On 2/5/20 8:09 AM, Kevin Wolf wrote:
> > >> > Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
> > >> >>>> The other part that it needs to solve is how to be available by default
> > >> >>>> without specifying anything on the command line. Basically, if I press
> > >> >>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> > >> >>>> implemented internally or by an external Python process, I don't mind.
> > >> >>>
> > >> >>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> > >> >>> use HMP on stdin).
> > >> >>
> > >> >> I don't think it would be that hard, actually.
> > >> >>
> > >> >> If you have a -qmp-shell option that takes a chardev and defaults to vc,
> > >> >> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
> > >> >> to do is launch the Python child process, pass it a pair of pipes for
> > >> >> communication and forward everything between the pipes and the chardev.
> > >> >>
> > >> >> (That's the theory anyway.)
> > >> > 
> > >> > If someone is interested, I did a quick proof-of-concept hack:
> > >> > 
> > >> >     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> > >> > 
> > >> > It doesn't clean up anything properly (including the qmp-shell processes
> > >> > it starts), but it spawns a usable qmp-shell on a user-specified
> > >> > character device. stdio seems to work, though without readline
> > >> > functionality (I suppose I still have line-buffering somewhere), vc
> > >> > doesn't really work at all yet.
> > >> > 
> > >> > Try it out like this:
> > >> > 
> > >> >     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
> > >> >     monitor_qmp_event: 1
> > >> >     Welcome to the QMP low-level shell!
> > >> >     Connected to QEMU 4.2.50
> > >> > 
> > >> >     (QEMU) query-version
> > >> >     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
> > >> >     (QEMU) quit
> > >> > 
> > >> > (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> > >> > refactorings in the storage daemon branch, so why not try both at once?)
> > >> > 
> > >> > Polishing this to make it mergable would still require substantial work,
> > >> > so at the moment I'm not planning to do this. But if someone wants to
> > >> > pick it up, feel free (just let us know).
> > >> > 
> > >> > Hm, in fact... A qmp-shell GSoC project?
> > >> > 
> > >> 
> > >> That would be great. I worry that we should have a clear vision for the
> > >> syntax before we give this project to an intern, though. With a clear
> > >> vision and an outline for deliverables, it's an incredibly appropriate
> > >> project.
> > >> 
> > >> Some things I think we want to define before we start:
> > >> 
> > >> 1. What are we trying to achieve with a standalone shell?
> > 
> > Projects without a clear goal rarely succeed.  Success within three
> > months is even rarer.
> >
> > >> 2. What syntax should it use?
> > 
> > Leaving that to a GSoC student amounts to setting up for failure.
> 
> I think this subthread shows that we actually have many separate
> projects that people wish to have someone work on. Each of them is
> probably a bit too small for a whole GSoC, but all of them together are
> probably too much. So I'll guess the student would pick maybe two of
> them, and if time is left at the end, more can be added as a bonus.
> 
> 1. Something like --monitor mode=qmp-shell that just spawns an external
>    Python script and passes it a QMP socket. This is the fundamental
>    building block for having any kind of external monitor script
>    actually integrated in QEMU, so I think just running the existing
>    qmp-shell this way (with proper support for at least stdio and vc
>    chardevs) would make sense as a first milestone.

I was originally going to suggest that should be sugar for a
-chardev filter  that takes an in/out chardev - but I don't know how
you'd handle tty stuff like formatting and tab completion etc.

> 2. Rewriting qmp-shell to use a better syntax for nested data
>    structures. This would have to be defined before the project starts.
> 
> 3. Improving qmp-shell UI-wise, e.g. by having better autocompletion,
>    support for counting brackets, or whatever else was mentioned. We
>    have a few ideas, and there's room for the student to add their own
>    ideas, too.
> 
> 4. Something HMP-like. This isn't QMP any more, so it could as well be a
>    separate script (hmp-shell?). But it could also be integrated in
>    qmp-shell in the form of additional commands that are implemented
>    client-side. Or maybe have a single shell, but have a QMP mode and an
>    HMP mode and the user can switch between these modes.
> 
>    The syntax for the HMP shell/mode could be the same or different from
>    the QMP syntax. This would have to be defined beforehand, too.

separate script sharing the qmp interface code?

Dave

> 5. Probably more that I just forgot now.
> 
> Suggesting the exact goals is part of the student application process,
> but for fundamental things like the syntax we should probably already
> know what we want.
> 
> > >> I think those are the hardest parts.
> > >>
> > >> Below, some musings:
> > >> 
> > >> - An integrated QMP shell would be a great usability boost to users of
> > >> bare QEMU.
> > >> 
> > >> - It is undesirable in general to support two interfaces. Feature
> > >> disparity is a problem, as is needing to document and test two separate
> > >> interfaces. The quality disparity between the two is also an issue.
> > >> 
> > >> - Offering HMP via the GTK interface but not QMP is a discoverability
> > >> problem. Unfamiliar users might assume that HMP is our flagship
> > >> interface. It is not.
> > >> 
> > >> - We are unlikely to re-expand HMP to cover everything QMP does; writing
> > >> a QMP shell that makes QMP easy to interface with is a better solution
> > >> for removing redundancy and complexity.
> 
> I'm not entirely convinced of this because QMP is often too low-level to
> actually address the practical high-level needs of users.
> 
> But these HMP-ish things are probably easier to maintain as scripts
> outside of the QEMU binary, so I think some kind of "QMP with
> extensions" for human could be the solution.
> 
> Once it's an external script, it will also be easy to exchange the shell
> for another one depending on user preference, or to hack in whatever
> functionality they are missing.
> 
> > >> - I suspect that the target audience for users of naked QEMU are:
> > >>   - QEMU developers
> > >>   - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
> > >> researching, testing, and debugging integration.
> > >>   - Devops professionals testing, implementing and debugging
> > >>     configuration & infrastructure
> > >>   - Security/infosec researchers
> > >>   - Embedded platform developers
> > >>   - Academic researchers
> 
> Maybe kernel developers should be mentioned separately, but yes, this
> list looks plausible to me.
> 
> > >> So please correct me if I am off the mark;
> > >> 
> > >> Design Goals:
> > >>   - The removal of HMP
> > >>   - An easy-to-use interface that remains reasonably "close" to the
> > >> machine API such that it provides a smooth transition to scripting QEMU.
> > >>   - Integration with our GTK interface for discoverability and convenience
> 
> As I listed above, I think these are actually three separate projects,
> rather than goals for a single big projects.
> 
> > >> Syntax:
> > >>   - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
> > >> should be replaced? If yes, what should it look like?
> > >
> > > I believe it should be a python shell with added commands.
> > >
> > > Simple things should be simple.
> > >   e.g. adding a disk from a local file should be trivial.
> > >
> > > Complex things can be complex - but it would be better if they were
> > > simple.
> > >
> > >   It's OK if the worst case of a blockdev is a bit hairy, but
> > >   watch out for cases where the hairyness creeps in unnecessarily.
> > 
> > Designing interfaces to complex machinery is hard.  Experience tells
> > that we do okay when we focus on the building blocks first.  That's
> > -blockdev.  When we start with trying to make simple things simple, we
> > end in swamps.  That's -drive.
> > 
> > Focus on building blocks is of course no excuse for unnecessary
> > hairiness.
> > 
> > It's also no reason not to build more convenient things on top of the
> > building blocks.  I doubt they should go into QMP, though.
> 
> Right, they should be implemented in that external script, which would
> use the lower-level QMP building blocks to provide the functionality. I
> also think it's a good idea to keep QMP accessible for more exotic use
> cases when the simple thing just doesn't cut it any more.
> 
> > > If the user screwsup, it should give an error that prompts the user
> > > to the parameter they got wrong.
> > >
> > > Output from commands should normally be pretty formatted (with an option
> > > to display raw json for those needing it).
> > >   e.g. that 'query-version' should give either just the package
> > >   version (as info version currently does) or:
> > >       4.2.50  Package: v4.2.0-1188-gd95a3885a9
> > >
> > > We shouldn't lose any HMP commands that some people find useful
> > >   Ditching HMP isn't an option until we've got almost all of it
> > >   covered.
> > 
> > In particular, we currently use HMP for debugging and monitoring
> > purposes, where we don't need or want QMP's rigor, neither its rigorous
> > interface stability, nor its structured I/O.  We want the "whipuptitude"
> > we get from monitor_printf().  This is actually a point David has made
> > several times.
> > 
> > To have a qmp-shell replace HMP, I think it needs to be able to
> > 
> > * Go beyond 1:1
> > 
> >   We tried a 1:1 mapping between HMP and QMP commands, and it didn't
> >   work out.  HMP's replacement should let us build convenient commands
> >   from QMP building blocks.
> > 
> >   We tried a 1:1 mapping between HMP and QMP command arguments, guided
> >   by @args_type.  Worked out for simple cases, but was too constricting.
> 
> We need to go beyond 1:1, but we probably want to be able to offer 1:1
> as a subset of commands accepted in that shell.
> 
> Offering only 1:1 in a good way might already be a step forward.
> 
> > * Preserve "whipuptitude" [David]
> > 
> >   I figure that means allowing some in QMP.  Without compromising its
> >   core mission, of course.
> 
> As long as we confine it to x- commands, I think this is okay.
> 
> > * As discoverable as HMP is now [Kevin]
> > 
> > * Help, completion and such at least on par with what HMP provides now
> 
> Will we want to add new annotations in the schema for this?
> 
> For example, HMP has completion support for block device names. In the
> QAPI schema, these are simply 'str'. We could bake the knowledge that
> in command 'foo' the parameter 'bar' is a block device name, but that
> would be a hack and would probably rarely be consistent with what QEMU
> actually does. It's really something that schema introspection should be
> able to tell us.
> 
> Kevin
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: qmp-shell for GSoC/Outreachy?

[Kevin Wolf]

Am 06.02.2020 um 19:26 hat Dr. David Alan Gilbert geschrieben:
> * Kevin Wolf (kwolf@redhat.com) wrote:
> > Am 06.02.2020 um 10:40 hat Markus Armbruster geschrieben:
> > > >> On 2/5/20 8:09 AM, Kevin Wolf wrote:
> > > >> > Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
> > > >> >>>> The other part that it needs to solve is how to be available by default
> > > >> >>>> without specifying anything on the command line. Basically, if I press
> > > >> >>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> > > >> >>>> implemented internally or by an external Python process, I don't mind.
> > > >> >>>
> > > >> >>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> > > >> >>> use HMP on stdin).
> > > >> >>
> > > >> >> I don't think it would be that hard, actually.
> > > >> >>
> > > >> >> If you have a -qmp-shell option that takes a chardev and defaults to vc,
> > > >> >> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
> > > >> >> to do is launch the Python child process, pass it a pair of pipes for
> > > >> >> communication and forward everything between the pipes and the chardev.
> > > >> >>
> > > >> >> (That's the theory anyway.)
> > > >> > 
> > > >> > If someone is interested, I did a quick proof-of-concept hack:
> > > >> > 
> > > >> >     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> > > >> > 
> > > >> > It doesn't clean up anything properly (including the qmp-shell processes
> > > >> > it starts), but it spawns a usable qmp-shell on a user-specified
> > > >> > character device. stdio seems to work, though without readline
> > > >> > functionality (I suppose I still have line-buffering somewhere), vc
> > > >> > doesn't really work at all yet.
> > > >> > 
> > > >> > Try it out like this:
> > > >> > 
> > > >> >     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
> > > >> >     monitor_qmp_event: 1
> > > >> >     Welcome to the QMP low-level shell!
> > > >> >     Connected to QEMU 4.2.50
> > > >> > 
> > > >> >     (QEMU) query-version
> > > >> >     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
> > > >> >     (QEMU) quit
> > > >> > 
> > > >> > (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> > > >> > refactorings in the storage daemon branch, so why not try both at once?)
> > > >> > 
> > > >> > Polishing this to make it mergable would still require substantial work,
> > > >> > so at the moment I'm not planning to do this. But if someone wants to
> > > >> > pick it up, feel free (just let us know).
> > > >> > 
> > > >> > Hm, in fact... A qmp-shell GSoC project?
> > > >> > 
> > > >> 
> > > >> That would be great. I worry that we should have a clear vision for the
> > > >> syntax before we give this project to an intern, though. With a clear
> > > >> vision and an outline for deliverables, it's an incredibly appropriate
> > > >> project.
> > > >> 
> > > >> Some things I think we want to define before we start:
> > > >> 
> > > >> 1. What are we trying to achieve with a standalone shell?
> > > 
> > > Projects without a clear goal rarely succeed.  Success within three
> > > months is even rarer.
> > >
> > > >> 2. What syntax should it use?
> > > 
> > > Leaving that to a GSoC student amounts to setting up for failure.
> > 
> > I think this subthread shows that we actually have many separate
> > projects that people wish to have someone work on. Each of them is
> > probably a bit too small for a whole GSoC, but all of them together are
> > probably too much. So I'll guess the student would pick maybe two of
> > them, and if time is left at the end, more can be added as a bonus.
> > 
> > 1. Something like --monitor mode=qmp-shell that just spawns an external
> >    Python script and passes it a QMP socket. This is the fundamental
> >    building block for having any kind of external monitor script
> >    actually integrated in QEMU, so I think just running the existing
> >    qmp-shell this way (with proper support for at least stdio and vc
> >    chardevs) would make sense as a first milestone.
> 
> I was originally going to suggest that should be sugar for a
> -chardev filter  that takes an in/out chardev - but I don't know how
> you'd handle tty stuff like formatting and tab completion etc.

Originally I thought the same, but it's actually not enough. The
external process needs not only stdio connected to the chardev, but also
another file descriptor for a QMP socket.

I guess tab completion should be possible if we can turn off line
buffering. Not sure what's the problem with formatting.

> > 2. Rewriting qmp-shell to use a better syntax for nested data
> >    structures. This would have to be defined before the project starts.
> > 
> > 3. Improving qmp-shell UI-wise, e.g. by having better autocompletion,
> >    support for counting brackets, or whatever else was mentioned. We
> >    have a few ideas, and there's room for the student to add their own
> >    ideas, too.
> > 
> > 4. Something HMP-like. This isn't QMP any more, so it could as well be a
> >    separate script (hmp-shell?). But it could also be integrated in
> >    qmp-shell in the form of additional commands that are implemented
> >    client-side. Or maybe have a single shell, but have a QMP mode and an
> >    HMP mode and the user can switch between these modes.
> > 
> >    The syntax for the HMP shell/mode could be the same or different from
> >    the QMP syntax. This would have to be defined beforehand, too.
> 
> separate script sharing the qmp interface code?

I would certainly expect that any QMP/HMP shell code uses the existing
QEMUMonitorProtocol Python class. qmp-shell already does.

Maybe they could share a bit more (command parsing?) if we want to have
separate scripts for HMP and QMP. Though actually I think I like the
idea of a combined QMP/HMP shell better at the moment.

Kevin

Re: qmp-shell for GSoC/Outreachy?

[John Snow]

On 2/6/20 9:21 AM, Kevin Wolf wrote:
> Am 06.02.2020 um 10:40 hat Markus Armbruster geschrieben:
>>>> On 2/5/20 8:09 AM, Kevin Wolf wrote:
>>>>> Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
>>>>>>>> The other part that it needs to solve is how to be available by default
>>>>>>>> without specifying anything on the command line. Basically, if I press
>>>>>>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
>>>>>>>> implemented internally or by an external Python process, I don't mind.
>>>>>>>
>>>>>>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
>>>>>>> use HMP on stdin).
>>>>>>
>>>>>> I don't think it would be that hard, actually.
>>>>>>
>>>>>> If you have a -qmp-shell option that takes a chardev and defaults to vc,
>>>>>> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
>>>>>> to do is launch the Python child process, pass it a pair of pipes for
>>>>>> communication and forward everything between the pipes and the chardev.
>>>>>>
>>>>>> (That's the theory anyway.)
>>>>>
>>>>> If someone is interested, I did a quick proof-of-concept hack:
>>>>>
>>>>>     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
>>>>>
>>>>> It doesn't clean up anything properly (including the qmp-shell processes
>>>>> it starts), but it spawns a usable qmp-shell on a user-specified
>>>>> character device. stdio seems to work, though without readline
>>>>> functionality (I suppose I still have line-buffering somewhere), vc
>>>>> doesn't really work at all yet.
>>>>>
>>>>> Try it out like this:
>>>>>
>>>>>     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
>>>>>     monitor_qmp_event: 1
>>>>>     Welcome to the QMP low-level shell!
>>>>>     Connected to QEMU 4.2.50
>>>>>
>>>>>     (QEMU) query-version
>>>>>     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
>>>>>     (QEMU) quit
>>>>>
>>>>> (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
>>>>> refactorings in the storage daemon branch, so why not try both at once?)
>>>>>
>>>>> Polishing this to make it mergable would still require substantial work,
>>>>> so at the moment I'm not planning to do this. But if someone wants to
>>>>> pick it up, feel free (just let us know).
>>>>>
>>>>> Hm, in fact... A qmp-shell GSoC project?
>>>>>
>>>>
>>>> That would be great. I worry that we should have a clear vision for the
>>>> syntax before we give this project to an intern, though. With a clear
>>>> vision and an outline for deliverables, it's an incredibly appropriate
>>>> project.
>>>>
>>>> Some things I think we want to define before we start:
>>>>
>>>> 1. What are we trying to achieve with a standalone shell?
>>
>> Projects without a clear goal rarely succeed.  Success within three
>> months is even rarer.
>>
>>>> 2. What syntax should it use?
>>
>> Leaving that to a GSoC student amounts to setting up for failure.
> 
> I think this subthread shows that we actually have many separate
> projects that people wish to have someone work on. Each of them is
> probably a bit too small for a whole GSoC, but all of them together are
> probably too much. So I'll guess the student would pick maybe two of
> them, and if time is left at the end, more can be added as a bonus.
> 

You're right, I am bundling together a few things.

> 1. Something like --monitor mode=qmp-shell that just spawns an external
>    Python script and passes it a QMP socket. This is the fundamental
>    building block for having any kind of external monitor script
>    actually integrated in QEMU, so I think just running the existing
>    qmp-shell this way (with proper support for at least stdio and vc
>    chardevs) would make sense as a first milestone.
> 

This is a good standalone task, but

> 2. Rewriting qmp-shell to use a better syntax for nested data
>    structures. This would have to be defined before the project starts.
> 

... Can't start until we define the proper interface, because then we
have to support it. Right now, qmp-shell is a developer toy because it's
hidden in the tree.

Promoting it to prime-time will be fruitless unless we come up with a
convincing TUI for it.

> 3. Improving qmp-shell UI-wise, e.g. by having better autocompletion,
>    support for counting brackets, or whatever else was mentioned. We
>    have a few ideas, and there's room for the student to add their own
>    ideas, too.
> 

Yes, this is all just great sugar. Completely optional, non-necessary.
Forgive me for bringing up things like this so often.

Not requisite to start on the project, but are great auxiliary tasks
that would make it really shiny. Perfect GSoC/outreachy material.

> 4. Something HMP-like. This isn't QMP any more, so it could as well be a
>    separate script (hmp-shell?). But it could also be integrated in
>    qmp-shell in the form of additional commands that are implemented
>    client-side. Or maybe have a single shell, but have a QMP mode and an
>    HMP mode and the user can switch between these modes.
> 
>    The syntax for the HMP shell/mode could be the same or different from
>    the QMP syntax. This would have to be defined beforehand, too.
> 

More or less my suggestion for offering e.g. "?info block" sugar
commands in the qmp-shell: an interface for executing sugar commands in
a unified human shell.

So, I don't know if "The final death of HMP" is something we want to
pursue as a project (Even though I bring it up often, I consider it an
open question -- do we want this? I actually don't know. I had assumed
that working on a fancy qmp-shell would only be interesting if it meant
the death of HMP, but perhaps I am wrong.)

Before we integrate qmp-shell into the GUI, I suspect we at least want
to answer ourselves what we want qmp-shell to be:

- QMP only, but nice for humans to interface with, or
- QMP with extras, like "?info block"

This mission statement will inform our basic premise for the syntax, and
getting the basic syntax right is important for having a successful
GSoC/outreachy project.


> 5. Probably more that I just forgot now.
> 
> Suggesting the exact goals is part of the student application process,
> but for fundamental things like the syntax we should probably already
> know what we want.
> 

The goals for the intership yes, but I am merely suggesting we come up
with a goal for what we want qmp-shell *to be* and what it is for, and
why it will benefit us.

Otherwise, it will be hard to offer direction to a student trying to
write code in a very political and opinionated area of our codebase.

So as requisites, I think we just need to answer:

- Do we want to get rid of HMP?

- Roughly, how do we want qmp-shell to act? Is it a REPL? Are we
convinced there is a workable QMP-compatible syntax that will be "easy
to use" (subjective) and "easy to parse" (subjective) in such a system?

- Do we like the existing syntax? (I assume we do not.)


And all the rest is details that can be worked out by discussions,
review, the student proposal, etc.

>>>> I think those are the hardest parts.
>>>>
>>>> Below, some musings:
>>>>
>>>> - An integrated QMP shell would be a great usability boost to users of
>>>> bare QEMU.
>>>>
>>>> - It is undesirable in general to support two interfaces. Feature
>>>> disparity is a problem, as is needing to document and test two separate
>>>> interfaces. The quality disparity between the two is also an issue.
>>>>
>>>> - Offering HMP via the GTK interface but not QMP is a discoverability
>>>> problem. Unfamiliar users might assume that HMP is our flagship
>>>> interface. It is not.
>>>>
>>>> - We are unlikely to re-expand HMP to cover everything QMP does; writing
>>>> a QMP shell that makes QMP easy to interface with is a better solution
>>>> for removing redundancy and complexity.
> 
> I'm not entirely convinced of this because QMP is often too low-level to
> actually address the practical high-level needs of users.
> 
> But these HMP-ish things are probably easier to maintain as scripts
> outside of the QEMU binary, so I think some kind of "QMP with
> extensions" for human could be the solution.
> 
> Once it's an external script, it will also be easy to exchange the shell
> for another one depending on user preference, or to hack in whatever
> functionality they are missing.
> 

Yes! This is exactly what I want with "?info block" style commands in
qmp-shell: standalone qmp-shell extensions that implement one nugget of
functionality that make a task easy at the human level.

Things that are important to "make easy" will get extensions.

Things that aren't, won't.

Let's say we do remove HMP: Anyone who misses some of that functionality
... Well, if you care, please do contribute a qmp-shell plugin that adds it.

>>>> - I suspect that the target audience for users of naked QEMU are:
>>>>   - QEMU developers
>>>>   - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
>>>> researching, testing, and debugging integration.
>>>>   - Devops professionals testing, implementing and debugging
>>>>     configuration & infrastructure
>>>>   - Security/infosec researchers
>>>>   - Embedded platform developers
>>>>   - Academic researchers
> 
> Maybe kernel developers should be mentioned separately, but yes, this
> list looks plausible to me.
> 

Oh, sure. This list is primarily here to state that there are very valid
reasons to make QEMU usable *directly* to people because there are
extremely valid reasons to be driving it directly.

I think the *exact nature* of all of these use cases is not as important
so much as they unambiguously exist.

These users are technical, but may or may not be intricately familiar
with QEMU inner workings.

Two takeaways:

- Having some sugar that doesn't assume they know *too much* is
important. A little bit of hand-holding is nice.

- Having an interface that does not get in your way when you know
exactly what you want is important to QEMU developers. (Allowing
raw/direct QMP input, showing raw/pretty-printed incoming and outgoing
QMP for writing documentation, etc.)



>>>> So please correct me if I am off the mark;
>>>>
>>>> Design Goals:
>>>>   - The removal of HMP
>>>>   - An easy-to-use interface that remains reasonably "close" to the
>>>> machine API such that it provides a smooth transition to scripting QEMU.
>>>>   - Integration with our GTK interface for discoverability and convenience
> 
> As I listed above, I think these are actually three separate projects,
> rather than goals for a single big projects.
> 

Yes, so let me say that the *deliverable* goals for a GSoC/Outreachy
project should just simply be *compatible* with our long-term roadmap/goals.

Having an idea for where we want to head will allow us to define a
tighter bound on what we want this summer without wasting effort.

(And will allow us to give more rapid feedback to the student without
arguing between ourselves what we even want. I just want us to agree on
a mission statement.)

>>>> Syntax:
>>>>   - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
>>>> should be replaced? If yes, what should it look like?
>>>
>>> I believe it should be a python shell with added commands.
>>>
>>> Simple things should be simple.
>>>   e.g. adding a disk from a local file should be trivial.
>>>
>>> Complex things can be complex - but it would be better if they were
>>> simple.
>>>
>>>   It's OK if the worst case of a blockdev is a bit hairy, but
>>>   watch out for cases where the hairyness creeps in unnecessarily.
>>
>> Designing interfaces to complex machinery is hard.  Experience tells
>> that we do okay when we focus on the building blocks first.  That's
>> -blockdev.  When we start with trying to make simple things simple, we
>> end in swamps.  That's -drive.
>>
>> Focus on building blocks is of course no excuse for unnecessary
>> hairiness.
>>
>> It's also no reason not to build more convenient things on top of the
>> building blocks.  I doubt they should go into QMP, though.
> 
> Right, they should be implemented in that external script, which would
> use the lower-level QMP building blocks to provide the functionality. I
> also think it's a good idea to keep QMP accessible for more exotic use
> cases when the simple thing just doesn't cut it any more.
> 
>>> If the user screwsup, it should give an error that prompts the user
>>> to the parameter they got wrong.
>>>
>>> Output from commands should normally be pretty formatted (with an option
>>> to display raw json for those needing it).
>>>   e.g. that 'query-version' should give either just the package
>>>   version (as info version currently does) or:
>>>       4.2.50  Package: v4.2.0-1188-gd95a3885a9
>>>
>>> We shouldn't lose any HMP commands that some people find useful
>>>   Ditching HMP isn't an option until we've got almost all of it
>>>   covered.
>>
>> In particular, we currently use HMP for debugging and monitoring
>> purposes, where we don't need or want QMP's rigor, neither its rigorous
>> interface stability, nor its structured I/O.  We want the "whipuptitude"
>> we get from monitor_printf().  This is actually a point David has made
>> several times.
>>
>> To have a qmp-shell replace HMP, I think it needs to be able to
>>
>> * Go beyond 1:1
>>
>>   We tried a 1:1 mapping between HMP and QMP commands, and it didn't
>>   work out.  HMP's replacement should let us build convenient commands
>>   from QMP building blocks.
>>
>>   We tried a 1:1 mapping between HMP and QMP command arguments, guided
>>   by @args_type.  Worked out for simple cases, but was too constricting.
> 
> We need to go beyond 1:1, but we probably want to be able to offer 1:1
> as a subset of commands accepted in that shell.
> 
> Offering only 1:1 in a good way might already be a step forward.
> 
>> * Preserve "whipuptitude" [David]
>>
>>   I figure that means allowing some in QMP.  Without compromising its
>>   core mission, of course.
> 
> As long as we confine it to x- commands, I think this is okay.
> 
>> * As discoverable as HMP is now [Kevin]
>>
>> * Help, completion and such at least on par with what HMP provides now
> 
> Will we want to add new annotations in the schema for this?
> 
> For example, HMP has completion support for block device names. In the
> QAPI schema, these are simply 'str'. We could bake the knowledge that
> in command 'foo' the parameter 'bar' is a block device name, but that
> would be a hack and would probably rarely be consistent with what QEMU
> actually does. It's really something that schema introspection should be
> able to tell us.
> 
> Kevin
> 

Re: qmp-shell for GSoC/Outreachy?

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 2/6/20 9:21 AM, Kevin Wolf wrote:
[...]
>> 2. Rewriting qmp-shell to use a better syntax for nested data
>>    structures. This would have to be defined before the project starts.
>> 
>
> ... Can't start until we define the proper interface, because then we
> have to support it. Right now, qmp-shell is a developer toy because it's
> hidden in the tree.
>
> Promoting it to prime-time will be fruitless unless we come up with a
> convincing TUI for it.

Oh, it'll bear fruit alright!  And that will be the problem.

Back to serious: I also object to the idea to expose end users to
qmp-shell in its current state.  Do not ship.

[...]

Re: qmp-shell for GSoC/Outreachy?

[Kevin Wolf]

Am 08.02.2020 um 08:25 hat Markus Armbruster geschrieben:
> John Snow <jsnow@redhat.com> writes:
> 
> > On 2/6/20 9:21 AM, Kevin Wolf wrote:
> [...]
> >> 2. Rewriting qmp-shell to use a better syntax for nested data
> >>    structures. This would have to be defined before the project starts.
> >> 
> >
> > ... Can't start until we define the proper interface, because then we
> > have to support it. Right now, qmp-shell is a developer toy because it's
> > hidden in the tree.
> >
> > Promoting it to prime-time will be fruitless unless we come up with a
> > convincing TUI for it.
> 
> Oh, it'll bear fruit alright!  And that will be the problem.
> 
> Back to serious: I also object to the idea to expose end users to
> qmp-shell in its current state.  Do not ship.

I don't think anyone suggested making the existing qmp-shell available
by default or shipping it in distro packages or anything. Being able to
configure qemu so that it does offer qmp-shell on a chardev seems like a
reasonable intermediate milestone to me, though. Users won't see this
unless they are actively trying to experiment with it.

Kevin

Re: qmp-shell for GSoC/Outreachy?

[Kevin Wolf]

Am 07.02.2020 um 22:23 hat John Snow geschrieben:
> On 2/6/20 9:21 AM, Kevin Wolf wrote:
> > I think this subthread shows that we actually have many separate
> > projects that people wish to have someone work on. Each of them is
> > probably a bit too small for a whole GSoC, but all of them together are
> > probably too much. So I'll guess the student would pick maybe two of
> > them, and if time is left at the end, more can be added as a bonus.
> > 
> 
> You're right, I am bundling together a few things.
> 
> > 1. Something like --monitor mode=qmp-shell that just spawns an external
> >    Python script and passes it a QMP socket. This is the fundamental
> >    building block for having any kind of external monitor script
> >    actually integrated in QEMU, so I think just running the existing
> >    qmp-shell this way (with proper support for at least stdio and vc
> >    chardevs) would make sense as a first milestone.
> 
> This is a good standalone task, but
> 
> > 2. Rewriting qmp-shell to use a better syntax for nested data
> >    structures. This would have to be defined before the project starts.
> > 
> 
> ... Can't start until we define the proper interface, because then we
> have to support it. Right now, qmp-shell is a developer toy because it's
> hidden in the tree.
> 
> Promoting it to prime-time will be fruitless unless we come up with a
> convincing TUI for it.

I'm not sure what you mean by "support". We wouldn't have to support it
any more than qmp-shell in the initial phase. It wouldn't even have to
be any less hidden.

And in fact, even in the long run, as it is meant for human use, the
compatibility promise (if this is what you mean) would be the same as
HMP - we change it whenever we think that this improves its usability
for humans.

There is a big difference between making qmp-shell more usable and
promoting it to prime-time.

> > 3. Improving qmp-shell UI-wise, e.g. by having better autocompletion,
> >    support for counting brackets, or whatever else was mentioned. We
> >    have a few ideas, and there's room for the student to add their own
> >    ideas, too.
> 
> Yes, this is all just great sugar. Completely optional, non-necessary.
> Forgive me for bringing up things like this so often.
> 
> Not requisite to start on the project, but are great auxiliary tasks
> that would make it really shiny. Perfect GSoC/outreachy material.

I think some amount of sugar will actually be necessary. We don't want
to have something that barely works, but something that people would
actually prefer to HMP.

> > 4. Something HMP-like. This isn't QMP any more, so it could as well be a
> >    separate script (hmp-shell?). But it could also be integrated in
> >    qmp-shell in the form of additional commands that are implemented
> >    client-side. Or maybe have a single shell, but have a QMP mode and an
> >    HMP mode and the user can switch between these modes.
> > 
> >    The syntax for the HMP shell/mode could be the same or different from
> >    the QMP syntax. This would have to be defined beforehand, too.
> > 
> 
> More or less my suggestion for offering e.g. "?info block" sugar
> commands in the qmp-shell: an interface for executing sugar commands in
> a unified human shell.
> 
> So, I don't know if "The final death of HMP" is something we want to
> pursue as a project

I would start with "Creating something that could eventually evolve into
something that makes HMP useless".

> (Even though I bring it up often, I consider it an open question -- do
> we want this? I actually don't know. I had assumed that working on a
> fancy qmp-shell would only be interesting if it meant the death of
> HMP, but perhaps I am wrong.)

I don't know either. I can tell you when I see the alternative. Having
the alternative implemented as an external script allows for some
experimentation without fully committing to it yet. qmp-shell has lived
for a long time in the tree without replacing HMP, so if we just end up
with a more usable qmp-shell that still doesn't replace HMP, I would
still see it as progress.

> Before we integrate qmp-shell into the GUI, I suspect we at least want
> to answer ourselves what we want qmp-shell to be:
> 
> - QMP only, but nice for humans to interface with, or
> - QMP with extras, like "?info block"
> 
> This mission statement will inform our basic premise for the syntax,
> and getting the basic syntax right is important for having a
> successful GSoC/outreachy project.

I think the former would be a subset of the latter, so starting with the
former shouldn't result in wasted work.

Personally, I like the idea of supporting some extras to perform high
level operations that QMP doesn't provide. I don't like the idea of
prefixing the extras with "?" to make them second-class commands.
Instead of having "?info block", I think "query-block" should be able to
print something readable by default and switching it to raw JSON would
be optional.

> > 5. Probably more that I just forgot now.
> > 
> > Suggesting the exact goals is part of the student application process,
> > but for fundamental things like the syntax we should probably already
> > know what we want.
> > 
> 
> The goals for the intership yes, but I am merely suggesting we come up
> with a goal for what we want qmp-shell *to be* and what it is for, and
> why it will benefit us.
> 
> Otherwise, it will be hard to offer direction to a student trying to
> write code in a very political and opinionated area of our codebase.
> 
> So as requisites, I think we just need to answer:
> 
> - Do we want to get rid of HMP?

Possibly.

> - Roughly, how do we want qmp-shell to act? Is it a REPL?

What else would it be?

> Are we convinced there is a workable QMP-compatible syntax that will
> be "easy to use" (subjective) and "easy to parse" (subjective) in such
> a system?

I think so. QMP-compatible basically just means that we need to be able
to distinguish QTypes (where the schema doesn't already disambiguate)
and to allow nesting.

> - Do we like the existing syntax? (I assume we do not.)

It makes sense for scalar options, so I think we'll want to keep these
cases working. We'll probably want something different for lists and
dicts.

Compared to HMP, a difference is that qmp-shell always requires you to
specify the option name. Introducing "positional arguments" like in HMP
could probably make things considerably easier to use.

> And all the rest is details that can be worked out by discussions,
> review, the student proposal, etc.
> 
> >>>> I think those are the hardest parts.
> >>>>
> >>>> Below, some musings:
> >>>>
> >>>> - An integrated QMP shell would be a great usability boost to users of
> >>>> bare QEMU.
> >>>>
> >>>> - It is undesirable in general to support two interfaces. Feature
> >>>> disparity is a problem, as is needing to document and test two separate
> >>>> interfaces. The quality disparity between the two is also an issue.
> >>>>
> >>>> - Offering HMP via the GTK interface but not QMP is a discoverability
> >>>> problem. Unfamiliar users might assume that HMP is our flagship
> >>>> interface. It is not.
> >>>>
> >>>> - We are unlikely to re-expand HMP to cover everything QMP does; writing
> >>>> a QMP shell that makes QMP easy to interface with is a better solution
> >>>> for removing redundancy and complexity.
> > 
> > I'm not entirely convinced of this because QMP is often too low-level to
> > actually address the practical high-level needs of users.
> > 
> > But these HMP-ish things are probably easier to maintain as scripts
> > outside of the QEMU binary, so I think some kind of "QMP with
> > extensions" for human could be the solution.
> > 
> > Once it's an external script, it will also be easy to exchange the shell
> > for another one depending on user preference, or to hack in whatever
> > functionality they are missing.
> > 
> 
> Yes! This is exactly what I want with "?info block" style commands in
> qmp-shell: standalone qmp-shell extensions that implement one nugget of
> functionality that make a task easy at the human level.
> 
> Things that are important to "make easy" will get extensions.
> 
> Things that aren't, won't.
> 
> Let's say we do remove HMP: Anyone who misses some of that functionality
> ... Well, if you care, please do contribute a qmp-shell plugin that adds it.

I think we should keep removing HMP completely out of this discussion.
If at some point we agree that qmp-shell has been improved to the extent
that none of us use HMP any more anyway, that's probably the point when
we should talk about removing HMP.

Let's just talk about improving qmp-shell for now.

> >>>> - I suspect that the target audience for users of naked QEMU are:
> >>>>   - QEMU developers
> >>>>   - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
> >>>> researching, testing, and debugging integration.
> >>>>   - Devops professionals testing, implementing and debugging
> >>>>     configuration & infrastructure
> >>>>   - Security/infosec researchers
> >>>>   - Embedded platform developers
> >>>>   - Academic researchers
> > 
> > Maybe kernel developers should be mentioned separately, but yes, this
> > list looks plausible to me.
> 
> Oh, sure. This list is primarily here to state that there are very valid
> reasons to make QEMU usable *directly* to people because there are
> extremely valid reasons to be driving it directly.
> 
> I think the *exact nature* of all of these use cases is not as important
> so much as they unambiguously exist.
> 
> These users are technical, but may or may not be intricately familiar
> with QEMU inner workings.

Yes, I agree.

> Two takeaways:
> 
> - Having some sugar that doesn't assume they know *too much* is
> important. A little bit of hand-holding is nice.
> 
> - Having an interface that does not get in your way when you know
> exactly what you want is important to QEMU developers. (Allowing
> raw/direct QMP input, showing raw/pretty-printed incoming and outgoing
> QMP for writing documentation, etc.)

Yes, even if we have human-oriented output for a command, there should
always be a way to see the raw JSON. And probably also a way to send raw
JSON.

> >>>> So please correct me if I am off the mark;
> >>>>
> >>>> Design Goals:
> >>>>   - The removal of HMP
> >>>>   - An easy-to-use interface that remains reasonably "close" to the
> >>>> machine API such that it provides a smooth transition to scripting QEMU.
> >>>>   - Integration with our GTK interface for discoverability and convenience
> > 
> > As I listed above, I think these are actually three separate projects,
> > rather than goals for a single big projects.
> 
> Yes, so let me say that the *deliverable* goals for a GSoC/Outreachy
> project should just simply be *compatible* with our long-term roadmap/goals.

That's a good way to phrase it.

> Having an idea for where we want to head will allow us to define a
> tighter bound on what we want this summer without wasting effort.
> 
> (And will allow us to give more rapid feedback to the student without
> arguing between ourselves what we even want. I just want us to agree on
> a mission statement.)

We should have a rough idea, but I don't think we need to know exactly
what the final state will look like. Let's keep some room for
experimentation. And even if we now implement something that turns out a
less than optional decision later, we can still change it. It's a human
interface, we don't have to be careful with respect to compatibility.

Kevin

Re: qmp-shell for GSoC/Outreachy?

[Dr. David Alan Gilbert]

* Markus Armbruster (armbru@redhat.com) wrote:
  That I wrote:
> >
> > I believe it should be a python shell with added commands.
> >
> > Simple things should be simple.
> >   e.g. adding a disk from a local file should be trivial.
> >
> > Complex things can be complex - but it would be better if they were
> > simple.
> >
> >   It's OK if the worst case of a blockdev is a bit hairy, but
> >   watch out for cases where the hairyness creeps in unnecessarily.
> 
> Designing interfaces to complex machinery is hard.  Experience tells
> that we do okay when we focus on the building blocks first.  That's
> -blockdev.  When we start with trying to make simple things simple, we
> end in swamps.  That's -drive.
> 
> Focus on building blocks is of course no excuse for unnecessary
> hairiness.
> 
> It's also no reason not to build more convenient things on top of the
> building blocks.  I doubt they should go into QMP, though.

I see where you're coming from, but I like -drive - it's simple for
simple things; maybe it's a nightmare for some others, but if I just
want a VM with a disk on virtio, it's easy.

But I agree if you have good building blocks and they're easy to
understand and easy to represent, it's not a bad start.

> > If the user screwsup, it should give an error that prompts the user
> > to the parameter they got wrong.
> >
> > Output from commands should normally be pretty formatted (with an option
> > to display raw json for those needing it).
> >   e.g. that 'query-version' should give either just the package
> >   version (as info version currently does) or:
> >       4.2.50  Package: v4.2.0-1188-gd95a3885a9
> >
> > We shouldn't lose any HMP commands that some people find useful
> >   Ditching HMP isn't an option until we've got almost all of it
> >   covered.
> 
> In particular, we currently use HMP for debugging and monitoring
> purposes, where we don't need or want QMP's rigor, neither its rigorous
> interface stability, nor its structured I/O.  We want the "whipuptitude"
> we get from monitor_printf().  This is actually a point David has made
> several times.
> 
> To have a qmp-shell replace HMP, I think it needs to be able to
> 
> * Go beyond 1:1
> 
>   We tried a 1:1 mapping between HMP and QMP commands, and it didn't
>   work out.  HMP's replacement should let us build convenient commands
>   from QMP building blocks.
> 
>   We tried a 1:1 mapping between HMP and QMP command arguments, guided
>   by @args_type.  Worked out for simple cases, but was too constricting.

Yes, it works for some things.

> * Preserve "whipuptitude" [David]
> 
>   I figure that means allowing some in QMP.  Without compromising its
>   core mission, of course.
> 
> * As discoverable as HMP is now [Kevin]
> 
> * Help, completion and such at least on par with what HMP provides now
> 
> Highly desirable:
> 
> * Support transitioning to the machine interface [John]
> 
>   Let humans start playing with the human interface, and when they feel
>   the need to automate, help them transition to QMP.
> 
> Back to John's question on qmp-shell syntax, which hasn't been answered
> so far.
> 
> JSON is a data-interchange format.  It doesn't try to be a configuration
> format or programming language syntax for human use.  It gets pressed
> into these roles with entirely predictable poor results.
> 
> Pain points of JSON include having to count parenthesises and having to
> quote so bloody much.  Additional QMP pain points include long names and
> excessive nesting.

Some other pet hates:
  a) Number formats are awful for our uses
  b) Lack of room for comments

> For the configuration format role, more usable alternatives exist.  YAML
> is a popular one.
> 
> qmp-shell is a REPL.  It needs a REPL-friendly syntax.  I doubt YAML is
> or even tries to be REPL-friendly.  I'd love to be wrong; the first rule
> of language design is "don't".
> 
> Other language suggestions?

While I hate XML, there's a certain niceness in using the same thing as
libvirt for places that mean the same thing; but that would have the bad
requirement that things meant *exactly* the same thing.

But, for machine representations, I'm not sure moving away from JSON is
a requirement.

Dave

> On making JSON suck less in this role:
> 
> LISP REPLs demonstrate that computers can assist effectively with
> counting parenthesises, and with completing long names.
> 
> We could make quoting optional for sufficiently nice object member
> names.  QAPI naming rules ensure niceness, actually.
> 
> We could make quoting optional for certain string literals.  Simple
> enough for literals that can only be a string, like abc.  For literals
> that could be something else like 123 or true, omitting quotes creates
> ambiguity.  When the schema accepts only one of the possible types, the
> ambiguity goes away.  Complexity stays, however.
> 
> Excessive nesting should ideally be attacked in QMP itself, but backward
> compatibility makes that hard.
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: qmp-shell for GSoC/Outreachy?

[Markus Armbruster]

"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:

> * Markus Armbruster (armbru@redhat.com) wrote:
>   That I wrote:
>> >
>> > I believe it should be a python shell with added commands.
>> >
>> > Simple things should be simple.
>> >   e.g. adding a disk from a local file should be trivial.
>> >
>> > Complex things can be complex - but it would be better if they were
>> > simple.
>> >
>> >   It's OK if the worst case of a blockdev is a bit hairy, but
>> >   watch out for cases where the hairyness creeps in unnecessarily.
>> 
>> Designing interfaces to complex machinery is hard.  Experience tells
>> that we do okay when we focus on the building blocks first.  That's
>> -blockdev.  When we start with trying to make simple things simple, we
>> end in swamps.  That's -drive.
>> 
>> Focus on building blocks is of course no excuse for unnecessary
>> hairiness.
>> 
>> It's also no reason not to build more convenient things on top of the
>> building blocks.  I doubt they should go into QMP, though.
>
> I see where you're coming from, but I like -drive - it's simple for
> simple things; maybe it's a nightmare for some others, but if I just
> want a VM with a disk on virtio, it's easy.
>
> But I agree if you have good building blocks and they're easy to
> understand and easy to represent, it's not a bad start.
>
>> > If the user screwsup, it should give an error that prompts the user
>> > to the parameter they got wrong.
>> >
>> > Output from commands should normally be pretty formatted (with an option
>> > to display raw json for those needing it).
>> >   e.g. that 'query-version' should give either just the package
>> >   version (as info version currently does) or:
>> >       4.2.50  Package: v4.2.0-1188-gd95a3885a9
>> >
>> > We shouldn't lose any HMP commands that some people find useful
>> >   Ditching HMP isn't an option until we've got almost all of it
>> >   covered.
>> 
>> In particular, we currently use HMP for debugging and monitoring
>> purposes, where we don't need or want QMP's rigor, neither its rigorous
>> interface stability, nor its structured I/O.  We want the "whipuptitude"
>> we get from monitor_printf().  This is actually a point David has made
>> several times.
>> 
>> To have a qmp-shell replace HMP, I think it needs to be able to
>> 
>> * Go beyond 1:1
>> 
>>   We tried a 1:1 mapping between HMP and QMP commands, and it didn't
>>   work out.  HMP's replacement should let us build convenient commands
>>   from QMP building blocks.
>> 
>>   We tried a 1:1 mapping between HMP and QMP command arguments, guided
>>   by @args_type.  Worked out for simple cases, but was too constricting.
>
> Yes, it works for some things.
>
>> * Preserve "whipuptitude" [David]
>> 
>>   I figure that means allowing some in QMP.  Without compromising its
>>   core mission, of course.
>> 
>> * As discoverable as HMP is now [Kevin]
>> 
>> * Help, completion and such at least on par with what HMP provides now
>> 
>> Highly desirable:
>> 
>> * Support transitioning to the machine interface [John]
>> 
>>   Let humans start playing with the human interface, and when they feel
>>   the need to automate, help them transition to QMP.
>> 
>> Back to John's question on qmp-shell syntax, which hasn't been answered
>> so far.
>> 
>> JSON is a data-interchange format.  It doesn't try to be a configuration
>> format or programming language syntax for human use.  It gets pressed
>> into these roles with entirely predictable poor results.
>> 
>> Pain points of JSON include having to count parenthesises and having to
>> quote so bloody much.  Additional QMP pain points include long names and
>> excessive nesting.
>
> Some other pet hates:
>   a) Number formats are awful for our uses

Yes, good point.

>   b) Lack of room for comments

An interactive REPL can do without comments, but for scripting and for
configuration files, they're essential.

>> For the configuration format role, more usable alternatives exist.  YAML
>> is a popular one.
>> 
>> qmp-shell is a REPL.  It needs a REPL-friendly syntax.  I doubt YAML is
>> or even tries to be REPL-friendly.  I'd love to be wrong; the first rule
>> of language design is "don't".
>> 
>> Other language suggestions?
>
> While I hate XML, there's a certain niceness in using the same thing as
> libvirt for places that mean the same thing; but that would have the bad
> requirement that things meant *exactly* the same thing.

I'm afraid such a requirement would first complicate things massively,
and then we'd accidentally mess up details anyway.

> But, for machine representations, I'm not sure moving away from JSON is
> a requirement.

QMP is working in its intended role "for use by machines".  JSON's
laughably weak specification of numbers is an issue, but it hardly
justifies starting over.

We're looking for a qmp-shell syntax that's more pleasant to use for
humans.

Re: qmp-shell for GSoC/Outreachy?

[Eric Blake]

On 2/6/20 12:18 PM, Dr. David Alan Gilbert wrote:

>>
>> Pain points of JSON include having to count parenthesises and having to
>> quote so bloody much.  Additional QMP pain points include long names and
>> excessive nesting.
> 
> Some other pet hates:
>    a) Number formats are awful for our uses
>    b) Lack of room for comments

Both permitted in JSON5.  Converting valid JSON5 into the stricter JSON 
is lossy when it comes to comments, but as comments don't affect machine 
meaning, maybe all the more it requires is an argument to 
qmp_capabilities stating whether the rest of the session will stick to 
strict JSON or prefer JSON5.

>> We could make quoting optional for certain string literals.

Hmm - JSON5 makes quoting optional for keys in { key:value }, but not 
values.

And one of the reasons why my QAPIfication of netdev was NOT committed 
was that the original code allowed both "item":1 and "item":"1" in the 
QemuOpts world, and we were torn on whether letting QMP accept both 
integer and string-that-parses-as-integer was acceptable.

>>  Simple
>> enough for literals that can only be a string, like abc.  For literals
>> that could be something else like 123 or true, omitting quotes creates
>> ambiguity.  When the schema accepts only one of the possible types, the
>> ambiguity goes away.  Complexity stays, however.

Do we even allow an alternate that permits both a string and a number at 
once? I thought you tightened qapi to reject that because of the 
potential for ambiguity.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: qmp-shell for GSoC/Outreachy?

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 2/6/20 12:18 PM, Dr. David Alan Gilbert wrote:
>
>>>
>>> Pain points of JSON include having to count parenthesises and having to
>>> quote so bloody much.  Additional QMP pain points include long names and
>>> excessive nesting.
>>
>> Some other pet hates:
>>    a) Number formats are awful for our uses
>>    b) Lack of room for comments
>
> Both permitted in JSON5.  Converting valid JSON5 into the stricter
> JSON is lossy when it comes to comments, but as comments don't affect
> machine meaning, maybe all the more it requires is an argument to
> qmp_capabilities stating whether the rest of the session will stick to
> strict JSON or prefer JSON5.
>
>>> We could make quoting optional for certain string literals.
>
> Hmm - JSON5 makes quoting optional for keys in { key:value }, but not
> values.
>
> And one of the reasons why my QAPIfication of netdev was NOT committed
> was that the original code allowed both "item":1 and "item":"1" in the
> QemuOpts world, and we were torn on whether letting QMP accept both
> integer and string-that-parses-as-integer was acceptable.

I think we were in perfect agreement that netdev_add accepting strings
that happen to contain integers as integers is a flaw, we just couldn't
muster the nerve to correct it.

We've since become less dogmatic about maintaining bug compatibility.

>>>  Simple
>>> enough for literals that can only be a string, like abc.  For literals
>>> that could be something else like 123 or true, omitting quotes creates
>>> ambiguity.  When the schema accepts only one of the possible types, the
>>> ambiguity goes away.  Complexity stays, however.
>
> Do we even allow an alternate that permits both a string and a number
> at once? I thought you tightened qapi to reject that because of the
> potential for ambiguity.

Correct.

The complexity enters through the 'any' backdoor.

Re: qmp-shell for GSoC/Outreachy?

[John Snow]

On 2/6/20 1:18 PM, Dr. David Alan Gilbert wrote:
> But, for machine representations, I'm not sure moving away from JSON is
> a requirement.

This is not really on the table.

--js

Re: qmp-shell for GSoC/Outreachy?

[John Snow]

On 2/6/20 4:40 AM, Markus Armbruster wrote:
> "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> 
>> * John Snow (jsnow@redhat.com) wrote:
>>> I'm forking the subject as I believe Markus wanted to focus on the
>>> machine interface aspect.
>>>
>>> I feel that a new human interface is *related* to that goal: the
>>> splitting of, and commitment to, separate human and machine interfaces
>>> powered by a single root schema.
> 
> A bit of history.
> 
> QMP initially shared HMP's "schema": qemu-monitor.hx.  The command
> handler was either a traditional HMP one, or a QMP-enabling pair (QMP
> handler, HMP formatter).  The idea was to convert traditional HMP
> handlers one by one, then ditch support for them.
> 
> With a QMP handler, HMP became a wrapper around QMP.  The input wrapping
> was data-driven: @args_type specifies how to map HMP arguments to QMP.
> The output wrapper was code, namely the HMP formatter.
> 
> This design turned out to tie QMP to HMP too tightly.  It assumes QMP
> and HMP commands are identical apart from argument syntax and output
> formatting.  They often should not be.  QMP wants building blocks:
> simple commands with simple replies, in particular simple failure modes.
> HMP wants convenience.  QMP wants rigor.  HMP has uses where that's a
> painful and unnecessary.
> 
> So we split qemu-monitor.hx into hmp-commands.hx and qmp-commands.hx.
> The former reverted back to the pre-QMP state, and the latter lost
> support for HMP wrappers.  QMP was liberated from having to reimplement
> HMP.  HMP was liberated from always having to do QMP first.
> 
> qmp-commands.hx was eventually replaced by the QAPI schema.
> 
> The lesson here is that to make "powered by a single root schema" work
> well, we'll likely have to put in more smarts than we did back then.
> 

Yes, understood: this is rigorous work that we've failed at before. It's
a big task; but I feel like we are increasingly hurt by our focus on QMP
and leaving HMP in the dust.

Unifying them might be the only reasonable forward-thinking thing, even
if it's hard.

> More on that below in reply to David's reply.
> 
>>> I am a big believer in making QEMU usable directly to human users as I
>>> feel the pipeline of "tinker to deployment" is important for a
>>> successful project, for many reasons:
>>>
>>> - QEMU should be easy to pick up and learn.
>>>
>>> - Supporting QEMU's use directly as an "end-user" program increases
>>> proficiency in the user population at large, which (can) lead to better
>>> answers and engagement on e.g. Reddit, StackOverflow, IRC
>>>
>>> - Evolving deployments from QEMU-only to libvirt+ or above (RHV, oVirt,
>>> kubevirt) should be a smooth and gradual process as additional
>>> complexity is desired.
>>>
>>> - Focusing on QEMU's usability allows our project to be consumed easier
>>> by new cloud-focused projects. If they are already familiar with (and
>>> happy with) our project, it is more likely to be used instead of seeking
>>> out alternatives. This is about reducing friction.
>>>
>>> So, for those reasons ... even though I feel that a machine-focused API
>>> is our #1 priority as it caters to our existing users, we should also
>>> focus on what it will take to grow mindshare for QEMU's value in the
>>> ecosystem.
>>>
>>> Slick interfaces and documentation go a long, long way to doing that.
>>>
>>> So: I feel that any new machine-only paradigm or overhaul needs to be
>>> accompanied with some new sugar to help the medicine go down, so-to-speak.
> 
> Points taken.
> 
>>> On 2/5/20 8:09 AM, Kevin Wolf wrote:
>>>> Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
>>>>>>> The other part that it needs to solve is how to be available by default
>>>>>>> without specifying anything on the command line. Basically, if I press
>>>>>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
>>>>>>> implemented internally or by an external Python process, I don't mind.
>>>>>>
>>>>>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
>>>>>> use HMP on stdin).
>>>>>
>>>>> I don't think it would be that hard, actually.
>>>>>
>>>>> If you have a -qmp-shell option that takes a chardev and defaults to vc,
>>>>> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
>>>>> to do is launch the Python child process, pass it a pair of pipes for
>>>>> communication and forward everything between the pipes and the chardev.
>>>>>
>>>>> (That's the theory anyway.)
>>>>
>>>> If someone is interested, I did a quick proof-of-concept hack:
>>>>
>>>>     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
>>>>
>>>> It doesn't clean up anything properly (including the qmp-shell processes
>>>> it starts), but it spawns a usable qmp-shell on a user-specified
>>>> character device. stdio seems to work, though without readline
>>>> functionality (I suppose I still have line-buffering somewhere), vc
>>>> doesn't really work at all yet.
>>>>
>>>> Try it out like this:
>>>>
>>>>     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
>>>>     monitor_qmp_event: 1
>>>>     Welcome to the QMP low-level shell!
>>>>     Connected to QEMU 4.2.50
>>>>
>>>>     (QEMU) query-version
>>>>     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": "v4.2.0-1188-gd95a3885a9"}}
>>>>     (QEMU) quit
>>>>
>>>> (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
>>>> refactorings in the storage daemon branch, so why not try both at once?)
>>>>
>>>> Polishing this to make it mergable would still require substantial work,
>>>> so at the moment I'm not planning to do this. But if someone wants to
>>>> pick it up, feel free (just let us know).
>>>>
>>>> Hm, in fact... A qmp-shell GSoC project?
>>>>
>>>
>>> That would be great. I worry that we should have a clear vision for the
>>> syntax before we give this project to an intern, though. With a clear
>>> vision and an outline for deliverables, it's an incredibly appropriate
>>> project.
>>>
>>> Some things I think we want to define before we start:
>>>
>>> 1. What are we trying to achieve with a standalone shell?
> 
> Projects without a clear goal rarely succeed.  Success within three
> months is even rarer.
> 
>>> 2. What syntax should it use?
> 
> Leaving that to a GSoC student amounts to setting up for failure.
> 
>>> I think those are the hardest parts.
>>>
>>> Below, some musings:
>>>
>>> - An integrated QMP shell would be a great usability boost to users of
>>> bare QEMU.
>>>
>>> - It is undesirable in general to support two interfaces. Feature
>>> disparity is a problem, as is needing to document and test two separate
>>> interfaces. The quality disparity between the two is also an issue.
>>>
>>> - Offering HMP via the GTK interface but not QMP is a discoverability
>>> problem. Unfamiliar users might assume that HMP is our flagship
>>> interface. It is not.
>>>
>>> - We are unlikely to re-expand HMP to cover everything QMP does; writing
>>> a QMP shell that makes QMP easy to interface with is a better solution
>>> for removing redundancy and complexity.
>>>
>>> - I suspect that the target audience for users of naked QEMU are:
>>>   - QEMU developers
>>>   - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
>>> researching, testing, and debugging integration.
>>>   - Devops professionals testing, implementing and debugging
>>>     configuration & infrastructure
>>>   - Security/infosec researchers
>>>   - Embedded platform developers
>>>   - Academic researchers
>>>
>>>
>>>
>>> So please correct me if I am off the mark;
>>>
>>> Design Goals:
>>>   - The removal of HMP
>>>   - An easy-to-use interface that remains reasonably "close" to the
>>> machine API such that it provides a smooth transition to scripting QEMU.
>>>   - Integration with our GTK interface for discoverability and convenience
>>>
>>> Syntax:
>>>   - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
>>> should be replaced? If yes, what should it look like?
>>
>> I believe it should be a python shell with added commands.
>>
>> Simple things should be simple.
>>   e.g. adding a disk from a local file should be trivial.
>>
>> Complex things can be complex - but it would be better if they were
>> simple.
>>
>>   It's OK if the worst case of a blockdev is a bit hairy, but
>>   watch out for cases where the hairyness creeps in unnecessarily.
> 
> Designing interfaces to complex machinery is hard.  Experience tells
> that we do okay when we focus on the building blocks first.  That's
> -blockdev.  When we start with trying to make simple things simple, we
> end in swamps.  That's -drive.
> 
> Focus on building blocks is of course no excuse for unnecessary
> hairiness.
> 
> It's also no reason not to build more convenient things on top of the
> building blocks.  I doubt they should go into QMP, though.
> 
>> If the user screwsup, it should give an error that prompts the user
>> to the parameter they got wrong.
>>
>> Output from commands should normally be pretty formatted (with an option
>> to display raw json for those needing it).
>>   e.g. that 'query-version' should give either just the package
>>   version (as info version currently does) or:
>>       4.2.50  Package: v4.2.0-1188-gd95a3885a9
>>
>> We shouldn't lose any HMP commands that some people find useful
>>   Ditching HMP isn't an option until we've got almost all of it
>>   covered.
> 
> In particular, we currently use HMP for debugging and monitoring
> purposes, where we don't need or want QMP's rigor, neither its rigorous
> interface stability, nor its structured I/O.  We want the "whipuptitude"
> we get from monitor_printf().  This is actually a point David has made
> several times.
> 

Yes, the "whipupitude" is something I am keen on preserving -- I just
want that whipupitude to be based on a schema.

Can we accomplish this with qmp-shell plugins that use special QMP
interfaces? I think so.

I'm imagining a special feature flag for "experimental" interfaces in
QMP that aren't thought through enough to become real interfaces that
these sugar commands -- implemented primarily in the shell - use.

They might do some bad things: they're synchronous, they might return a
bit too much data, they might return too much data. But it's only for
debug use, and we declare the API unstable/fluid, so we don't care.

Maybe this is a fool's errand, though, and we SHOULD just keep HMP.
Maybe we just need to QAPI-fy our HMP shell so that we can document it
better, and expand QAPI power to describe the format of HMP commands.

The core issue I take with HMP is that it's underdocumented, tends to
rot, and rarely keeps up with configuration idiom changes in QEMU -- it
*feels* like a legacy interface, not an up-to-date debugging interface.

However we fix this is actually not personally important to me.


> To have a qmp-shell replace HMP, I think it needs to be able to
> 
> * Go beyond 1:1
> 
>   We tried a 1:1 mapping between HMP and QMP commands, and it didn't
>   work out.  HMP's replacement should let us build convenient commands
>   from QMP building blocks.
> 
>   We tried a 1:1 mapping between HMP and QMP command arguments, guided
>   by @args_type.  Worked out for simple cases, but was too constricting.
> 
> * Preserve "whipuptitude" [David]
> 
>   I figure that means allowing some in QMP.  Without compromising its
>   core mission, of course.
> 
> * As discoverable as HMP is now [Kevin]
> 
> * Help, completion and such at least on par with what HMP provides now
> 
> Highly desirable:
> 
> * Support transitioning to the machine interface [John]
> 
>   Let humans start playing with the human interface, and when they feel
>   the need to automate, help them transition to QMP.
> 
> Back to John's question on qmp-shell syntax, which hasn't been answered
> so far.
> 
> JSON is a data-interchange format.  It doesn't try to be a configuration
> format or programming language syntax for human use.  It gets pressed
> into these roles with entirely predictable poor results.
> 

More or less; but it's still pretty decent as a format if you're editing
it in an IDE that can assist you in doing so.

No editor I am aware of at present is capable of editing JSON or YAML
against a schema, though: the prevailing paradigm appears to be:

- Write file in editor
- Run some validator command
- Deploy and pray

From what I can tell, this is how it works in the cloud world, too.

> Pain points of JSON include having to count parenthesises and having to
> quote so bloody much.  Additional QMP pain points include long names and
> excessive nesting.
> 
> For the configuration format role, more usable alternatives exist.  YAML
> is a popular one.
> 

Yes, but doesn't solve our parsing issue for interactive shells.

> qmp-shell is a REPL.  It needs a REPL-friendly syntax.  I doubt YAML is
> or even tries to be REPL-friendly.  I'd love to be wrong; the first rule
> of language design is "don't".
> 

Yes.

> Other language suggestions?
> 
> On making JSON suck less in this role:
> 
> LISP REPLs demonstrate that computers can assist effectively with
> counting parenthesises, and with completing long names.
> 
> We could make quoting optional for sufficiently nice object member
> names.  QAPI naming rules ensure niceness, actually.
> 
> We could make quoting optional for certain string literals.  Simple
> enough for literals that can only be a string, like abc.  For literals
> that could be something else like 123 or true, omitting quotes creates
> ambiguity.  When the schema accepts only one of the possible types, the
> ambiguity goes away.  Complexity stays, however.
> 
> Excessive nesting should ideally be attacked in QMP itself, but backward
> compatibility makes that hard.
> 

I wish I had a good idea on how to make a QMP wrapper that works in a
REPL that didn't involve copy/pasting JSON. I guess it's why so many
people *do* just cat data into the socket.

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/27/20 9:35 AM, Kevin Wolf wrote:
> Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
>> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
>>> Well, sure. The context of this email was qmp-shell though, which is
>>> meant to help facilitate the entry of JSON commands so that you *can*
>>> indeed just forego the CLI/HMP entirely.
>>>
>>> If you are of the opinion that every user of QEMU should be copy/pasting
>>> JSON straight into a socket and we should delete qmp-shell, that's
>>> certainly a fine opinion.
>>
>> I think part of the pain of qmp-shell comes from the very fact that
>> it is trying to be an interactive shell. This points people towards
>> interactively typing in the commands, which is horrific when you get
>> anywhere near the JSON, or even dot-notation traditional commands.
>>
>> If it was just a qmp-client that was single shot, we'd encourage
>> people to create the JSON in a sensible way - vim/emacs/whatever.
> 
> I don't see how this is sensible. QMP commands are something that I
> reuse even less than VM configurations, so creating a one-off file for
> each would take me a lot more time and I would still have to type the
> same JSON code that I have to type with -qmp stdio.
> 
> The reason it is and should be an interactive shell is that I'm
> interacting with it. Switching back and forth between a text editor and
> a shell to actually send the command to QEMU would make things only even
> more cumbersome than they already are.
> 
>> Bash/dash/zsh/$whatever is their interactive shell, with massively
>> more features than qmp-shell. You have command history, autocomplete,
>> conditional and looping constructs, and everything a normal shell
>> offers.
> 
> If I wanted to program a QMP client, I would use Python. For me,
> conditionals and loops are completely out of scope for a QMP shell. I
> just want an easy way to tell QEMU to do something specific.
> 
> A command history already exists for qmp-shell. It's better than bash
> because it doesn't mix QMP history with whatever else I do on my
> computer.
> 
> Autocomplete in qmp-shell doesn't exist, as far as I know, but if
> implemented, it could be a lot more useful than bash completion because
> it could offer key completion based on the QMP schema.
> 

It does have tab completion for command names, but it does not know
about or remember argument fields. It does not have autocomplete or
typing hints like FiSH or bash ^r.

I would like to change this, actually, by making the docstrings in QAPI
schema a first class citizen of the spec and allowing them to be
introspectable via the socket directly.

(I.e., you can get the list of arguments and the docstrings that
accompany them over the wire so you can display it in the client.)

Problem I'm having with qmp-shell is, like Kevin says below ...

> This is in fact a big part of the problem that qmp-shell really needs to
> solve before it can replace HMP: How to make writing commands at least
> almost as simple as with HMP. If I can just press tab a few times to
> cycle through the available options for the command, that would already
> be a massive improvement over writing JSON manually (which you would
> still have to do with your text-file based approach, without any
> QMP-specific support).
> 

... I can't figure out how to make writing commands simple.

When you have a "simple" command, the abstraction works OK; you can type
key=val pairs and go about your way.

As soon as you have anything nested, the gossamer-thin illusion is
destroyed. I investigated making this a little easier by adding a parser
that could read directly from stdin and would allow multi-line JSON
inputs as arguments.

(Like the python shell does it, for example: When you have a dictionary
opening brace, it lets you continue to the next line.)

I was a little disheartened that most JSON parsers in python expect to
consume buffered objects and generally consume the entire buffer -- it
didn't seem to play nice with the idea of wanting to parse from STDIN
directly.


So:

- I think qmp-shell is worth having, especially if polished
(autocomplete, docstrings, argument hints, etc).

- Kevin mentioned getting this into the GTK shell. I think that would be
great, as a step to help phase out HMP.

- I think getting rid of HMP is good because I don't care for the idea
of supporting two monitor protocols. One schema, one parser, one truth.

- I do, however, like the idea of having a non-rigorous monitor that
lets us get away with murder when we need to. HMP is useful for
debugging, prototypes and other things where the rigor and permanence of
a QAPI schema feels too burdensome.

- So: maybe a turbocharged qmp-shell can offer some similar kinds of
convenience commands that are build on top of real QMP. Sugar logic and
other fanciful things could be implemented there in qmp-shell as
extensions. You'd get a stock pile of them with your QEMU install that
help you do certain tasks quickly and trivially.

- Given all the above, I am willing to try to save, polish, or re-design
qmp-shell; but am a bit starved for ideas on the syntax... This is why I
was spending a bit of time talking about our flattening to dot syntax,
and other projects related to representing hierarchical data.

Would really love to hear ideas on what a good interactive shell syntax
for a JSON-fueled schema would look like.

Any prior art, other projects, and reading anyone can recommend would be
nice.

> The other part that it needs to solve is how to be available by default
> without specifying anything on the command line. Basically, if I press
> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> implemented internally or by an external Python process, I don't mind.
> 
>> The only strong reason for qmp-shell to be interactive would be if
>> the initial protoocl handshake was too slow. I can't see that being
>> a problem with QMP.
> 
> Speed would be the least of my concerns. This is about manual use, and
> it already takes me a while to type in my commands.
> 
>> Example usage:
>>
>> 1. Launch the QEMU runtime for the desired target
>>
>>      $ qemu-runtime-x86_64 myvm.sock
>>
>> 2. Load the configuration to define the VM
>>
>>    $ cat myvm.yaml
>>    commands:
>>      - machine_declare:
>>          name: pc-q35-5.0
>> 	 ...
>>      - blockdev_add:
>>          ...
>>      - device_add:
>>          ...
>>      - blockdev_add:
>>          ...
>>      - device_add:
>>          ...
>>    $ qemu-client myvm.sock myvm.yaml
>>
>>
>> 3. Hotplug a disk
>>
>>    $ cat mynewdisk.yaml
>>    commands:
>>      - blockdev_add:
>>          ...
>>      - device_add:
>>          ...
>>    $ qemu-client myvm.sock mynewdisk.yaml
>>
>>
>> 3. Hotunplug a disk
>>
>>    $ cat myolddisk.yaml
>>    commands:
>>      - device_del:
>>          ...
>>      - blockdev_del:
>>          ...
>>    $ qemu-client myvm.sock myolddisk.yaml
> 
> Just to compare, this is what the human user oriented flow looks like
> today:
> 
> 1. qemu-system-x86_64 -M pc-q35-5.0 -drive if=virtio,... -cdrom ...
> 
> 2. <Press Ctrl-Alt-2 to get to the HMP shell>
>    (qemu) drive_add ...
>    <Press Ctrl-Alt-1 to get back to the guest>
> 
> 3. <Press Ctrl-Alt-2 to get to the HMP shell>
>    (qemu) device_del ...
>    <Press Ctrl-Alt-1 to get back to the guest>
> 
> This is what we're competing with, and honestly I don't see how your
> qemu-runtime-*/qemu-client based flow comes even close to it in terms of
> usability.
> 
> QMP, JSON and YAML may be nice machine interfaces, but having nice
> machine interfaces doesn't mean that you shouldn't also have something
> that is suitable for humans. qmp-shell is trying to be that, and while
> it leaves much to be desired in its current state, replacing it with
> even more machine-friendly stuff that is cumbersome for humans isn't the
> right answer.
> 

This is why I have two things I want to do:

Part A:
- Remove the CLI as it exists today*
- Introduce a machine-readable, but human-friendly format, like YAML (or
JSON) that's based directly on QAPI.

Part B:
- Improve qmp-shell such that it provides a sufficient usability layer
on top of QMP
- Make this tool prominent to users; e.g. putting it in the GTK
interface would be nice.
- Reduce prominence of HMP and works towards its removal.

--js

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 1/27/20 9:35 AM, Kevin Wolf wrote:
>> Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
>>> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
>>>> Well, sure. The context of this email was qmp-shell though, which is
>>>> meant to help facilitate the entry of JSON commands so that you *can*
>>>> indeed just forego the CLI/HMP entirely.
>>>>
>>>> If you are of the opinion that every user of QEMU should be copy/pasting
>>>> JSON straight into a socket and we should delete qmp-shell, that's
>>>> certainly a fine opinion.
>>>
>>> I think part of the pain of qmp-shell comes from the very fact that
>>> it is trying to be an interactive shell. This points people towards
>>> interactively typing in the commands, which is horrific when you get
>>> anywhere near the JSON, or even dot-notation traditional commands.
>>>
>>> If it was just a qmp-client that was single shot, we'd encourage
>>> people to create the JSON in a sensible way - vim/emacs/whatever.
>> 
>> I don't see how this is sensible. QMP commands are something that I
>> reuse even less than VM configurations, so creating a one-off file for
>> each would take me a lot more time and I would still have to type the
>> same JSON code that I have to type with -qmp stdio.
>> 
>> The reason it is and should be an interactive shell is that I'm
>> interacting with it. Switching back and forth between a text editor and
>> a shell to actually send the command to QEMU would make things only even
>> more cumbersome than they already are.
>> 
>>> Bash/dash/zsh/$whatever is their interactive shell, with massively
>>> more features than qmp-shell. You have command history, autocomplete,
>>> conditional and looping constructs, and everything a normal shell
>>> offers.
>> 
>> If I wanted to program a QMP client, I would use Python. For me,
>> conditionals and loops are completely out of scope for a QMP shell. I
>> just want an easy way to tell QEMU to do something specific.
>> 
>> A command history already exists for qmp-shell. It's better than bash
>> because it doesn't mix QMP history with whatever else I do on my
>> computer.
>> 
>> Autocomplete in qmp-shell doesn't exist, as far as I know, but if
>> implemented, it could be a lot more useful than bash completion because
>> it could offer key completion based on the QMP schema.
>> 
>
> It does have tab completion for command names, but it does not know
> about or remember argument fields. It does not have autocomplete or
> typing hints like FiSH or bash ^r.
>
> I would like to change this, actually, by making the docstrings in QAPI
> schema a first class citizen of the spec and allowing them to be
> introspectable via the socket directly.
>
> (I.e., you can get the list of arguments and the docstrings that
> accompany them over the wire so you can display it in the client.)

You need doc strings for help, but not for completion.

query-qmp-schema's reply is already rather big: 130KiB.

The size of docs in an JSON encoding useful to qmp-shell we can only
estimate.  Plain text docs/interop/qemu-qmp-ref.txt is 505KiB.

I guess we'd make the doc strings optional.

> Problem I'm having with qmp-shell is, like Kevin says below ...
>
>> This is in fact a big part of the problem that qmp-shell really needs to
>> solve before it can replace HMP: How to make writing commands at least
>> almost as simple as with HMP. If I can just press tab a few times to
>> cycle through the available options for the command, that would already
>> be a massive improvement over writing JSON manually (which you would
>> still have to do with your text-file based approach, without any
>> QMP-specific support).
>> 
>
> ... I can't figure out how to make writing commands simple.
>
> When you have a "simple" command, the abstraction works OK; you can type
> key=val pairs and go about your way.

qmp-shell tries to improve on the JSON syntax by inventing its own,
but...

> As soon as you have anything nested, the gossamer-thin illusion is
> destroyed.

... it does a halfhearted job.

I'm deeply skeptical of any solution that starts with inventing a
new language.  Start with examining existing ones instead.

>            I investigated making this a little easier by adding a parser
> that could read directly from stdin and would allow multi-line JSON
> inputs as arguments.
>
> (Like the python shell does it, for example: When you have a dictionary
> opening brace, it lets you continue to the next line.)
>
> I was a little disheartened that most JSON parsers in python expect to
> consume buffered objects and generally consume the entire buffer -- it
> didn't seem to play nice with the idea of wanting to parse from STDIN
> directly.
>
>
> So:
>
> - I think qmp-shell is worth having, especially if polished
> (autocomplete, docstrings, argument hints, etc).
>
> - Kevin mentioned getting this into the GTK shell. I think that would be
> great, as a step to help phase out HMP.
>
> - I think getting rid of HMP is good because I don't care for the idea
> of supporting two monitor protocols. One schema, one parser, one truth.
>
> - I do, however, like the idea of having a non-rigorous monitor that
> lets us get away with murder when we need to. HMP is useful for
> debugging, prototypes and other things where the rigor and permanence of
> a QAPI schema feels too burdensome.

We discussed relaxing the rules for such QMP commands.  In particular,
they could return just a string.  Fine with me as long as we make
perfectly clear these commands are not stable interfaces.

> - So: maybe a turbocharged qmp-shell can offer some similar kinds of
> convenience commands that are build on top of real QMP. Sugar logic and
> other fanciful things could be implemented there in qmp-shell as
> extensions. You'd get a stock pile of them with your QEMU install that
> help you do certain tasks quickly and trivially.
>
> - Given all the above, I am willing to try to save, polish, or re-design
> qmp-shell; but am a bit starved for ideas on the syntax... This is why I
> was spending a bit of time talking about our flattening to dot syntax,
> and other projects related to representing hierarchical data.
>
> Would really love to hear ideas on what a good interactive shell syntax
> for a JSON-fueled schema would look like.

We can't possibly be the first ones asking this question.

> Any prior art, other projects, and reading anyone can recommend would be
> nice.
[...]

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/28/20 5:16 AM, Markus Armbruster wrote:
> I'm deeply skeptical of any solution that starts with inventing a
> new language.  Start with examining existing ones instead.

I promise that we are in violent agreement.

Later in the mail:

"Any prior art, other projects, and reading anyone can recommend would
be nice."

--js

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 1/23/20 2:19 AM, Markus Armbruster wrote:
[...]
>> Any conventional parser can be made streaming by turning it into a
>> coroutine.  This is probably the simplest solution for handwritten
>> streaming LL parsers, because it permits recursive descent.  In Python,
>> I'd try a generator.
>> 
>> Our actual solution for QMP predates coroutine support in QEMU, and is
>> rather hamfisted:
>> 
>> * Streaming lexer: it gets fed characters one at a time, and when its
>>   state machine says "token complete", it feeds the token to the
>>   "streamer".
>> 
>> * "Streamer": gets fed tokens one at a time, buffers them up counting
>>   curly and square bracket nesting until the nesting is zero, then
>>   passes the buffered tokens to the parser.
>> 
>> * Non-streaming parser: it gets fed a sequence of tokens that constitute
>>   a full expression.
>> 
>> The best I can say about this is that it works.  The streamer's token
>> buffer eats a lot of memory compared to a real streaming parser, but in
>> practice, it's a drop in the bucket.
>> 
>
> I looked into this at one point. I forget why I didn't like it. I had

Because it's confusing?  Took me a while to get used to it.

> some notion that I should replace this one too, but forget exactly why.
> Maybe it wasn't that bad, if I've forgotten.

Bigger fish to fry.

[...]

Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin]

> On 23 Jan 2020, at 18:58, John Snow <jsnow@redhat.com> wrote:
> 
> 
> 
> On 1/23/20 2:19 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>>>>> * scripts/qmp/qmp-shell
>>>>> 
>>>>>  Half-hearted attempt at a human-friendly wrapper around the JSON
>>>>>  syntax.  I have no use for this myself.
>>>> I use this fairly often as its a useful debugging / experimentation
>>>> / trouble shooting tool. There's similar ish functionality in
>>>> virsh qemu-monitor-command. I think there's scope of a supported
>>>> tool here that can talk to libvirt or a UNIX socket for doing
>>>> QMP commands, with a friendlier syntax & pretty printing. 
>>>> 
>>> 
>>> qmp-shell is one of my go-to tools for working through bitmap workflows
>>> where we don't have convenience commands yet, as some of the setups
>>> required for fleecing et al involve quite a number of steps.
>>> 
>>> I can copy-paste raw JSON into a socket, but personally I like seeing my
>>> commands neatly organized in a format where I can visually reduce them
>>> to their components at a glance.
>>> 
>>> (What I mean is: It's hard to remember which QMP commands you've barfed
>>> into a terminal because JSON is hard to read and looks very visually
>>> repetitive.)
>>> 
>>> I tried to rewrite qmp-shell late last year, actually. I wanted to write
>>> a new REPL that was json-aware in some manner such that you could write
>>> multi-line commands like this:
>>> 
>>>> example-command arg={
>>>  "hello": "world"
>>> }
>>> 
>>> This requires, sadly, a streamable JSON parser. Most JSON parsers built
>>> into Python as-is simply take a file pointer and consume the entirety of
>>> the rest of the stream -- they don't play very nice with incomplete
>>> input or input that may have trailing data, e.g.:
>>> 
>>>> example-command arg={
>>>  "hello": "world"
>>> } arg2={
>>>  "oops!": "more json!"
>>> }
>> 
>> QMP is in the same boat: it needs to process input that isn't
>> necessarily full expressions (JSON-text in the RFC's grammar).
>> 
>> Any conventional parser can be made streaming by turning it into a
>> coroutine.  This is probably the simplest solution for handwritten
>> streaming LL parsers, because it permits recursive descent.  In Python,
>> I'd try a generator.
>> 
>> Our actual solution for QMP predates coroutine support in QEMU, and is
>> rather hamfisted:
>> 
>> * Streaming lexer: it gets fed characters one at a time, and when its
>>  state machine says "token complete", it feeds the token to the
>>  "streamer".
>> 
>> * "Streamer": gets fed tokens one at a time, buffers them up counting
>>  curly and square bracket nesting until the nesting is zero, then
>>  passes the buffered tokens to the parser.
>> 
>> * Non-streaming parser: it gets fed a sequence of tokens that constitute
>>  a full expression.
>> 
>> The best I can say about this is that it works.  The streamer's token
>> buffer eats a lot of memory compared to a real streaming parser, but in
>> practice, it's a drop in the bucket.
>> 
> 
> I looked into this at one point. I forget why I didn't like it. I had
> some notion that I should replace this one too, but forget exactly why.
> Maybe it wasn't that bad, if I've forgotten.
> 
>>> Also, due to the nature of JSON as being a single discrete object and
>>> never a stream of objects, no existing JSON parser really supports the
>>> idea of ever seeing more than one object per buffer.
>> 
>> That plainly sucks.
>> 
>>> ...So I investigated writing a proper grammar for qmp-shell.
>> 
>> Any parser must start with a proper grammar.  If it doesn't, it's a toy,
>> or a highway to madness.
>> 
>>> Unfortunately, this basically means including the JSON grammar as a
>>> subset of the shell grammar and writing your own parser for it entirely.
>> 
>> Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
>> paper, so JSON sticks out left and right.
>> 
>> Scrap and start over.
>> 
>>> I looked into using Python's own lexer; but it's designed to lex
>>> *python*, not *json*. I got a prototype lexer working for this purpose
>>> under a grammar that I think reflects JSON, but I got that sinking
>>> feeling that it was all more trouble than it was worth, and scrapped
>>> working on it any further.
>> 
>> Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
>> weird derivative of JSON in 239 SLOC.
>> 
>>> I did not find any other flex/yacc-like tools that seemed properly
>>> idiomatic or otherwise heavily specialized. I gave up on the idea of
>>> writing a new parser.
>> 
>> While I recommend use of tools for parsing non-trivial grammars (you'll
>> screw up, they won't), they're massive overkill for JSON.
>> 
>>> I'd love to offer a nice robust QMP shell that is available for use by
>>> end users, but the syntax of the shell will need some major considerations.
>> 
>> Scrap and start over.
>> 
>> [...]
>> 
> 
> Yes, I agree: Scrap and start over.
> 
> What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
> is that it offers a convenient way to enter the top-level keys of the
> arguments dict. This works absolutely fine right up until you need to
> start providing nested definitions.

Well, if you are really ready to start from scratch, I might offer the XL syntax
as a starting point for a discussion of a user-visible syntax that is also
applicable for text-based or binary API exchanges.

I’m going to talk about it at FOSDEM in the “minimalist languages” design.
Those who are in Brussels might want to attend to get a better feel.
Source code is here: https://github.com/c3d/xl, but the only part you
care about for this discussion is src/{parser,scanner}.{c,h} and the
syntax configuration file src/xl.syntax. As well as renderer styles
src/xl.stylesheet, src/html.stylesheet, etc.

Key points for the use case considered:
- Tiny (~2000 lines of code for parser/scanner, a C and a C++ implementation)
- Fully introspectable, serializable in a cross-platform way, printable (with styles)
- Character-precise position tracking for error printing
- Parser preserves comments (for documentation generators)
- Small, if slow, interpreter in about 20K lines of code (~bash speed on some tests)
  meaning we would get a “qemu scripting language” with loops, tests, arithmetic, etc.

More detailed discussion at end of this mail if you think it warrants a second look.
In any case, if it helps, I’d be happy to help connecting it to qemu…

> 
> For the nesting, we say: "Go ahead and use JSON, but you have to take
> all the spaces out."

Here, that would be A.B.C, which parses as
(infix.
 (infix.
  A
  B)
 C
)

(result of `xl -nobuiltins -parse test.xl -style debug -show`)

Also, an example given earlier:

 { 'command': 'iothread-set-poll-params',
   'data': {
       'id': 'str',
	'*max-ns': 'uint64',
	'*grow': 'uint64',
	'*shrink': 'uint64'
   },
   'map-to-qom-set': 'IOThread'
 }

could be written as:

command iothread_set_poll_params
	data
		id : str
		*max_ns : uint64
		*grow : uint64
		*shrink : uint64
	map_to_qom_set IOThread

But if you want to keep the original syntax, it seems to parse and render practically OK:

% cat /tmp/a.xl 
{ 'command': 'iothread-set-poll-params',
   'data': {
       'id': 'str',
        '*max-ns': 'uint64',
        '*grow': 'uint64',
        '*shrink': 'uint64'
   },
   'map-to-qom-set': 'IOThread'
 }

%xl -nobuiltins -parse /tmp/a.xl  -show            
{ 'command':'iothread-set-poll-params', 'data': { 'id':'str', 
    '*max-ns':'uint64', '*grow':'uint64', '*shrink':'uint64'
}, 'map-to-qom-set':'IOThread' }

This is with no change to the XL parser / scanner code
whatsoever, not even to the syntax file. So that gives me hope
that we could have a “reasonably good” compatibility mode
that transforms the quasi-JSON format into the new form,
with a single parser accepting both.

> 
> This... works, charitably, but is hardly what I would call usable.
> 
> For the CLI, we offer a dot syntax notation that resembles nothing in
> particular. It often seems the case that it isn't expressive enough to
> map losslessly to JSON. I suspect it doesn't handle siblings very well.
> 
> A proper HMP-esque TUI would likely have need of coming up with its own
> pet syntax for commands that avoid complicated nested JSON definitions,
> but for effort:value ratio, having a QMP shorthand shell that works
> arbitrarily with any command might be a better win.

The XL proposal here would be to have a single format shared by
- The source definitions used to generate C code
- The monitor / internal shell syntax
- The command-line syntax
- The API data (possibly in serialized form for compactness)

> 
> Do we still have a general-case problem of how to represent QAPI
> structures in plaintext? Will this need to be solved for the CLI, too?
> 
> --js

More info below.

Here are some aspects that I think are interesting about it:

- Tiny (2000 lines of code for scanner and parser, ~20K for a full interpreter)
C:  wc parser.c parser.h scanner.c scanner.h 
     716    2183   26702 parser.c
     100     440    3372 parser.h
     926    2966   30537 scanner.c
     206     945    8249 scanner.h
    1948    6534   68860 total

C++:
     726    2372   26918 parser.cpp
     885    2480   26363 scanner.cpp
     248    1025    8867 ../include/scanner.h
     166     687    5958 ../include/parser.h
    2025    6564   68106 total

- Simple (parse tree with 8 node types, integer, real, name/symbol, text, infix, prefix, postfix and block)

	+ integer, e.g. 12, 1_000_000 or 16#33A or 2#10101
	+ real, e.g. 11.3, 16#1.FFF#e-3, 2#1.01
	+ text, e.g. “ABC”, ‘ABC’, <<Long text, multi-lines>> (configurable separators)
	+ name/symbols, a.g. Foo_Bar, +, <=, (precedence and spelling configurable)

	+ infix, e.g. A+B, A and B
	+ prefix, e.g. +3, sin X
	+ postfix, e.g. 3%, 3 km
	+ block, e.g. [A], (A), {A} and indentation blocks

- Fully introspectable (mostly because the parse tree is simple)
- Reversible, i.e. can be printed, including with formatting, e.g.:
	% xl -nobuiltins -parse demo/1-hello.xl -show
	tell "localhost”, 
	    print "Hello World”
	% xl -nobuiltins -parse demo/1-hello.xl -style debug -show   
	(prefix
	 tell
	 (infix,
	  “localhost"
	  (block indent
	   (prefix
	    print
	    "Hello World”
	   ))))
- Also has a binary serializer that produces a platform-independent format
- Has multiple implementations, notably C and C++ implementation (and even one in XL :-)
- Validated on thousands of lines of input, with various language styles (e.g. Ada-like or functional)
- Character-level position tracking for error messages in scripts / config files:
	/tmp/xl.xl:1007:8: Mismatched identation, expected “)"
	/tmp/xl.xl:2409:23: Mismatched identation, expected ""
- Designed to be easy to read and write
- Powerful enough to parse itself (https://github.com/c3d/xl/blob/master/xl2/native/xl.parser.xl)
- Dynamically configurable syntax (spelling and precedence of operators)
- Multi-line text with configurable separators, e.g. the following can be made a text constant by having XML and END_XML as text separators:
     XML
         <stuff>
            Insert your XML here
        </stuff> 
    END_XML
- Based-numbers in any base, e.g. 8#777, 16#FFFF_FFFF and 2#1.001 as valid numbers
- Has essentially a single contributor (me), so easy to relicense as needed
- There is an interpreter, e.g. potential evaluate expressions like 2+3*A
- Relatively fast (6.1s to parse 1M lines of code representing 40M of code, cpp ~1s)
% wc /tmp/tmp.xl
 1000000 3893922 41679700 /tmp/tmp.xl
% time xl -parse /tmp/tmp.xl
 6.10s user 0.21s system 99% cpu 6.346 total
- Support multiple styles, e.g. using { } for blocks or indentation, parentheses or not, etc.

Cons (but I’m not the better person to come up with cons on this pet project of mine ;-):
- Idiosyncratic
- Single contributor
- Not well maintained
- Definitely not production quality (even the makefiles are broken ;-)
- Has some CI testing, but it fails, and it’s totally insufficient
- Interpreter far from perfect
- Designed with another purpose in mind (a programming language)
- Syntax is not C-centric, e.g. 16#FFFF instead of 0xFFFF.
- Name syntax does not allow -, i.e. max-ns is “max minus ns”, max_ns OK.
- [insert probably about a thousand others here]

Precedences and other stuff can be configured dynamically, through a file
in the current implementations, eg. https://github.com/c3d/xl-c/blob/master/xl.syntax.
So that means we can have a “nice” syntax for the commands and objects,
and a format that can serve both as a config file format, as a command
language, and as a full shell-style language with if, loops, etc.

It also supports nested syntaxes, i.e. dynamic changes of precedence
between selected separators. Used to support simplified C syntax
with “extern int foo();”, where the “C” syntax is active between “extern”
and “;”. Could be useful for compatibility.

Parse tree is simple enough that it’s fully introspectable.

There is a (configurable) renderer, so you can generate source from the
internal data structure. The renderer can generate colorized source
code in HTML, so I guess we could generate C data structures relatively
easily.

I believe that it is relatively trivial to configure the parser syntax file 
to accept the QEMU quasi-JSON. (some code changes required to teach
it to totally ignore whitespace, toi avoid error messages).

More complete documentation about the language is here:
https://c3d.github.io/xl, but it’s quite light on implementation details.
So read only if you have a bit of time.

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 24/12/19 14:41, Daniel P. Berrangé wrote:
> I'm wondering if we need to think about a temporary *throwaway* fork of
> QEMU. I wasn't involved in the work, but as an outside observer I think
> it is interesting to see how the NEMU project fork ultimately led to QEMU
> moving forward with a new KConfig approach, and the integration of microvm.
> They had the freedom to develop these new approaches with zero regard
> to back compat or broader QEMU needs that might otherwise hold back dev.

Just a quick note: none of these were developed first in NEMU.  KConfig
was completed *for* NEMU (to make their lives easier) but the code
already existed, and microvm is inspired by Firecracker.  NEMU's main
merit in my opinion was to make QEMU's perception issues clearer, and to
cause more thought on those issues and how to approach it.

That said, I agree that a throwaway experiment on command line and
configuration is worthwhile.  The main thing to think about is to make
convenience options syntactic sugar for something else.  For example, a
couple ideas mentioned in the thread or elsewhere in hallway discussions:

- Make default devices expressible as QemuOpts, so that -writeconfig
able to write out the default devices as well and -readconfig would
imply -nodefaults.

- Reduce the scope of -drive by dropping if=none in favor of -blockdev.
 Leave -drive as a quick way of configuring frontend and backend.

- And somewhat unrelated to configuration files: generate bindings for
QMP clients from the schema.

In the meanwhile, we should deprecate -readconfig/-writeconfig.  Can't
go wrong with that.

Paolo

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4227 bytes --]

On Sat, Dec 21, 2019 at 10:02:23AM +0100, Markus Armbruster wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> 
> > Hi,
> > QEMU presents a command-line interface and QMP monitor for
> > applications to interact with.  Applications actually need API
> > bindings in their programming language.  Bindings avoid reimplementing
> > code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> > lazy and de facto relies on libvirt for API bindings.
> >
> > Is it time for better QEMU APIs?
> >
> > 1. We have qapi-schema.json.  Let's render to HTML and publish
> > versioned documentation online.
> 
> Make can build docs/interop/qemu-qmp-ref.{7,html,info,pdf,txt}.  Grab
> the .html and go for it.  There's also qmp-ga-ref.

The missing step here is to integrate doc generation and upload into the
release process so the documentation is published.

Once my recent documentation publishing patches have been merged I'll
see if Mike Roth wants to extend the release scripts.

> > 2. scripts/qmp/ contains command-line tools for QMP communication.
> > They could use some polish and then be shipped.
> 
> MAINTAINERS blames them on me, but they're effectively unmaintained.
> Prerequisite for shipping: having a maintainer who actually gives a
> damn.
...
> * scripts/qmp/qmp-shell
> 
>   Half-hearted attempt at a human-friendly wrapper around the JSON
>   syntax.  I have no use for this myself.

I think this one is used by people.  John Snow comes to mind.

> > 3. python/qemu/ contains Python modules for managing a QEMU process
> > and QMP communication.  This should be packaged in distros and
> > available on PyPI.
> 
> Currently maintained by Eduardo and Cleber (cc'ed) under "Python
> scripts".
> 
> > 4. Go and Rust bindings would also be useful.  There is
> > https://github.com/intel/govmm but I think it makes sense to keep it
> > in qemu.git and provide an interface similar to our Python modules.
> 
> Mapping QAPI/QMP commands and events to function signatures isn't hard
> (the QAPI code generator does).  Two problems (at least):
> 
> 1. Leads to some pretty ridiculous functions.  Here's one:
> 
>     void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
>                              const char *device,
>                              const char *target,
>                              bool has_replaces, const char *replaces,
>                              MirrorSyncMode sync,
>                              bool has_speed, int64_t speed,
>                              bool has_granularity, uint32_t granularity,
>                              bool has_buf_size, int64_t buf_size,
>                              bool has_on_source_error,
>                              BlockdevOnError on_source_error,
>                              bool has_on_target_error, BlockdevOnError on_target_error,
>                              bool has_filter_node_name, const char *filter_node_name,
>                              bool has_copy_mode, MirrorCopyMode copy_mode, 
>                              bool has_auto_finalize, bool auto_finalize,
>                              bool has_auto_dismiss, bool auto_dismiss,
>                              Error **errp);
> 
>   We commonly use 'boxed': true for such beasts, which results in
>   functions like this one:
> 
>     void qmp_blockdev_add(BlockdevOptions *arg, Error **errp);
> 
> 2. Many schema changes that are nicely backward compatible in QMP are
>    anything but in such an "obvious" C API.  Adding optional arguments,
>    for instance, or changing integer type width.  The former is less of
>    an issue with 'boxed': true.
> 
> Perhaps less of an issue with dynamic languages.
> 
> I figure a static language would need much more expressive oomph than C
> to be a good target.  No idea how well Go or Rust bindings can work.

Most of what govmm does is build QEMU command-lines, not wrap QMP APIs.

Native QMP API bindings are harder and we'd need to see who really needs
this.  Languages with a built-in dict type like Python make working with
JSON-level QMP feasible.  In C it's painful though and native API
bindings are desirable.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Making QEMU easier for management tools and applications

[Kashyap Chamarthy]

On Thu, Jan 02, 2020 at 02:47:22PM +0000, Stefan Hajnoczi wrote:
> On Sat, Dec 21, 2019 at 10:02:23AM +0100, Markus Armbruster wrote:
> > Stefan Hajnoczi <stefanha@gmail.com> writes:

[...]

> > > 2. scripts/qmp/ contains command-line tools for QMP communication.
> > > They could use some polish and then be shipped.
> > 
> > MAINTAINERS blames them on me, but they're effectively unmaintained.
> > Prerequisite for shipping: having a maintainer who actually gives a
> > damn.
> ...
> > * scripts/qmp/qmp-shell
> > 
> >   Half-hearted attempt at a human-friendly wrapper around the JSON
> >   syntax.  I have no use for this myself.
> 
> I think this one is used by people.  John Snow comes to mind.

FWIW I too frequently use 'qmp-shell'.  And some of the examples in this
document[1] are demonstrated with it.

I'm reasonably happy with it (particularly the persistent history
captured in ~/.qmp-shell_history), and it has some "known issues" that
can trip up a new user.  The one that immediately jumps to mind:
asynchronous events won't be printed without a prompt from the user --
e.g. after a `blockdev-commit`, you won't see BLOCK_JOB_{READY,
COMPLETED} events printed unless you manually hit enter from the
'qmp-shell'.

(Not complaining; I have a long-standing TODO to make time to
investigate this.)

[1] https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html

> > > 3. python/qemu/ contains Python modules for managing a QEMU process
> > > and QMP communication.  This should be packaged in distros and
> > > available on PyPI.
> > 
> > Currently maintained by Eduardo and Cleber (cc'ed) under "Python
> > scripts".

Yeah, that'd certainly be useful.  I vaguely recall a discussion on this
topic here on the list (or probably some hallway discussion, can't
recall.)


[...]


-- 
/kashyap

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

On Thu, Jan 16, 2020 at 12:03:14PM +0100, Kashyap Chamarthy wrote:
> On Thu, Jan 02, 2020 at 02:47:22PM +0000, Stefan Hajnoczi wrote:
> > On Sat, Dec 21, 2019 at 10:02:23AM +0100, Markus Armbruster wrote:
> > > Stefan Hajnoczi <stefanha@gmail.com> writes:
> 
> [...]
> 
> > > > 2. scripts/qmp/ contains command-line tools for QMP communication.
> > > > They could use some polish and then be shipped.
> > > 
> > > MAINTAINERS blames them on me, but they're effectively unmaintained.
> > > Prerequisite for shipping: having a maintainer who actually gives a
> > > damn.
> > ...
> > > * scripts/qmp/qmp-shell
> > > 
> > >   Half-hearted attempt at a human-friendly wrapper around the JSON
> > >   syntax.  I have no use for this myself.
> > 
> > I think this one is used by people.  John Snow comes to mind.
> 
> FWIW I too frequently use 'qmp-shell'.  And some of the examples in this
> document[1] are demonstrated with it.
> 
> I'm reasonably happy with it (particularly the persistent history
> captured in ~/.qmp-shell_history), and it has some "known issues" that
> can trip up a new user.  The one that immediately jumps to mind:
> asynchronous events won't be printed without a prompt from the user --
> e.g. after a `blockdev-commit`, you won't see BLOCK_JOB_{READY,
> COMPLETED} events printed unless you manually hit enter from the
> 'qmp-shell'.
> 
> (Not complaining; I have a long-standing TODO to make time to
> investigate this.)
> 
> [1] https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html

John and I discussed async events in the past.  qmp-shell currently uses
the input() built-in function.  If we modify it with a
select(2)/poll(2)-style function that monitors both stdin and the QMP
socket then it could print QMP events as soon as they are received.

There might be a nicer way of doing it, but pseudo-code for the idea is:

  def input_with_events(prompt):
      while True:
          print(prompt, end='', flush=True)
          readable_files = select([sys.stdin, qmp_socket])
          if qmp_socket in readable_files:
              print_qmp_events()

      # stdin is ready, read a line
      return input()

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Making QEMU easier for management tools and applications

[Kashyap Chamarthy]

On Mon, Jan 20, 2020 at 09:55:54AM +0000, Stefan Hajnoczi wrote:
> On Thu, Jan 16, 2020 at 12:03:14PM +0100, Kashyap Chamarthy wrote:

[...]

> > I'm reasonably happy with it (particularly the persistent history
> > captured in ~/.qmp-shell_history), and it has some "known issues" that
> > can trip up a new user.  The one that immediately jumps to mind:
> > asynchronous events won't be printed without a prompt from the user --
> > e.g. after a `blockdev-commit`, you won't see BLOCK_JOB_{READY,
> > COMPLETED} events printed unless you manually hit enter from the
> > 'qmp-shell'.

[...]

> John and I discussed async events in the past.  qmp-shell currently uses
> the input() built-in function.  If we modify it with a
> select(2)/poll(2)-style function that monitors both stdin and the QMP
> socket then it could print QMP events as soon as they are received.
> 
> There might be a nicer way of doing it, but pseudo-code for the idea is:
> 
>   def input_with_events(prompt):
>       while True:
>           print(prompt, end='', flush=True)
>           readable_files = select([sys.stdin, qmp_socket])
>           if qmp_socket in readable_files:
>               print_qmp_events()
> 
>       # stdin is ready, read a line
>       return input()

Thanks for the suggestion.  The Python 'select' module[1] indeed seems
to provide access to select() and poll() Linux system calls.

[1] https://docs.python.org/3/library/select.html

-- 
/kashyap

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 20/01/20 10:55, Stefan Hajnoczi wrote:
>>
>> [1] https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html
> John and I discussed async events in the past.  qmp-shell currently uses
> the input() built-in function.  If we modify it with a
> select(2)/poll(2)-style function that monitors both stdin and the QMP
> socket then it could print QMP events as soon as they are received.

I think it should be rewritten using async/await.  A simple example:

    import asyncio
    import sys
    from concurrent.futures import ThreadPoolExecutor

    async def ainput(prompt: str = ""):
        with ThreadPoolExecutor(1, "ainput") as executor:
            return (await asyncio.get_event_loop().run_in_executor(
                executor, sys.stdin.readline
            )).rstrip()

    async def numbers():
        i = 1
        while True:
            print(i)
            i = i + 1
            await asyncio.sleep(1)

    async def main():
        name = await ainput("What's your name? ")
        print("Hello, {}!".format(name))

    asyncio.get_event_loop().create_task(numbers())
    asyncio.get_event_loop().run_until_complete(main())

This would be a great Summer of Code project.  Even an autocompletion
interface using readline should be possible.

Paolo

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/25/20 6:41 AM, Paolo Bonzini wrote:
> On 20/01/20 10:55, Stefan Hajnoczi wrote:
>>>
>>> [1] https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html
>> John and I discussed async events in the past.  qmp-shell currently uses
>> the input() built-in function.  If we modify it with a
>> select(2)/poll(2)-style function that monitors both stdin and the QMP
>> socket then it could print QMP events as soon as they are received.
> 
> I think it should be rewritten using async/await.  A simple example:
> 
>     import asyncio
>     import sys
>     from concurrent.futures import ThreadPoolExecutor
> 
>     async def ainput(prompt: str = ""):
>         with ThreadPoolExecutor(1, "ainput") as executor:
>             return (await asyncio.get_event_loop().run_in_executor(
>                 executor, sys.stdin.readline
>             )).rstrip()
> 
>     async def numbers():
>         i = 1
>         while True:
>             print(i)
>             i = i + 1
>             await asyncio.sleep(1)
> 
>     async def main():
>         name = await ainput("What's your name? ")
>         print("Hello, {}!".format(name))
> 
>     asyncio.get_event_loop().create_task(numbers())
>     asyncio.get_event_loop().run_until_complete(main())
> 
> This would be a great Summer of Code project.  Even an autocompletion
> interface using readline should be possible.
> 
> Paolo
> 

I wrote an async version at one point; I had problems integrating
asyncio with readline functionality.

--js

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* Markus Armbruster (armbru@redhat.com) wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> > 4. Go and Rust bindings would also be useful.  There is
> > https://github.com/intel/govmm but I think it makes sense to keep it
> > in qemu.git and provide an interface similar to our Python modules.
> 
> Mapping QAPI/QMP commands and events to function signatures isn't hard
> (the QAPI code generator does).  Two problems (at least):
> 
> 1. Leads to some pretty ridiculous functions.  Here's one:
> 
>     void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
>                              const char *device,
>                              const char *target,
>                              bool has_replaces, const char *replaces,
>                              MirrorSyncMode sync,
>                              bool has_speed, int64_t speed,
>                              bool has_granularity, uint32_t granularity,
>                              bool has_buf_size, int64_t buf_size,
>                              bool has_on_source_error,
>                              BlockdevOnError on_source_error,
>                              bool has_on_target_error, BlockdevOnError on_target_error,
>                              bool has_filter_node_name, const char *filter_node_name,
>                              bool has_copy_mode, MirrorCopyMode copy_mode, 
>                              bool has_auto_finalize, bool auto_finalize,
>                              bool has_auto_dismiss, bool auto_dismiss,
>                              Error **errp);

Those might not be as bad when mapped to other languages, all the
bool/value pairs would become Option<...>  so that removes that doubling.
The Error ** mechanism should somehow map onto functions returning a
normal Rust Result<> type.

Dave

>   We commonly use 'boxed': true for such beasts, which results in
>   functions like this one:
> 
>     void qmp_blockdev_add(BlockdevOptions *arg, Error **errp);
> 
> 2. Many schema changes that are nicely backward compatible in QMP are
>    anything but in such an "obvious" C API.  Adding optional arguments,
>    for instance, or changing integer type width.  The former is less of
>    an issue with 'boxed': true.
> 
> Perhaps less of an issue with dynamic languages.
> 
> I figure a static language would need much more expressive oomph than C
> to be a good target.  No idea how well Go or Rust bindings can work.
> 
> > 5. A jailer is needed to isolate the QEMU process and vhost-user
> > device backends using seccomp, Linux namespaces, and maybe
> > SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> > security, but it's becoming a common task for any device backend and
> > IMO should be its own launcher tool.
> 
> Perhaps the libvirt refactoring effort can give us one.
> 
> > 6. A configuration file format is sorely needed so that guest
> > configuration can be persisted and easily launched.  Today you have to
> > create a shell script that executes a QEMU command-line, but this is
> > suboptimal because sharing executable scripts is dangerous from a
> > security perspective and is hard to parse or modify programmatically.
> 
> No argument.  There is -readconfig, but it falls way short.
> 
> With command line QAPIfication, a real configuration file will be quite
> feasible.
> 
> The main reason for the lack of progress there is our dedication to
> backward compatibility.  A functional replacement of our CLI is a huge
> task already.  Throwing in backward compatibility makes it a daunting
> one.  Not least because nobody fully understands all the quirks.
> 
> > In many of these areas we already have a partial solution.  It just
> > needs more work.  I think it would be worth the effort and the mental
> > shift to really providing APIs that are easy to use by applications.
> >
> > What do you think?
> >
> > Have I missed things that are needed?
> >
> > Have I included things that are unnecessary?
> 
> I feel we need to make up our minds what kind of interface(s) we want to
> provide.
> 
> We provide a low-level interface for management applications.  I feel we
> need to nail this one.  Its QMP part is okay, I think, the CLI part is
> not.
> 
> We also try to provide friendlier interfaces for human users.  Not only
> do we suck at it, we're also prone to create liabilities for later: much
> of the compatibility woes that have been holding us back are rooted in
> "convenience" features.  And don't get me started on the folly of us
> dabbling in GUI.
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:

> * Markus Armbruster (armbru@redhat.com) wrote:
>> Stefan Hajnoczi <stefanha@gmail.com> writes:
>> > 4. Go and Rust bindings would also be useful.  There is
>> > https://github.com/intel/govmm but I think it makes sense to keep it
>> > in qemu.git and provide an interface similar to our Python modules.
>> 
>> Mapping QAPI/QMP commands and events to function signatures isn't hard
>> (the QAPI code generator does).  Two problems (at least):
>> 
>> 1. Leads to some pretty ridiculous functions.  Here's one:
>> 
>>     void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
>>                              const char *device,
>>                              const char *target,
>>                              bool has_replaces, const char *replaces,
>>                              MirrorSyncMode sync,
>>                              bool has_speed, int64_t speed,
>>                              bool has_granularity, uint32_t granularity,
>>                              bool has_buf_size, int64_t buf_size,
>>                              bool has_on_source_error,
>>                              BlockdevOnError on_source_error,
>>                              bool has_on_target_error, BlockdevOnError on_target_error,
>>                              bool has_filter_node_name, const char *filter_node_name,
>>                              bool has_copy_mode, MirrorCopyMode copy_mode, 
>>                              bool has_auto_finalize, bool auto_finalize,
>>                              bool has_auto_dismiss, bool auto_dismiss,
>>                              Error **errp);
>
> Those might not be as bad when mapped to other languages, all the
> bool/value pairs would become Option<...>  so that removes that doubling.
> The Error ** mechanism should somehow map onto functions returning a
> normal Rust Result<> type.

Fifteen parameters even with the has_FOO and the errp dropped.
Still ridiculous enough for me.

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Fri, Dec 20, 2019 at 04:13:59PM +0000, Stefan Hajnoczi wrote:
> Hi,
> QEMU presents a command-line interface and QMP monitor for
> applications to interact with.  Applications actually need API
> bindings in their programming language.  Bindings avoid reimplementing
> code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> lazy and de facto relies on libvirt for API bindings.
> 
> Is it time for better QEMU APIs?
> 
> 1. We have qapi-schema.json.  Let's render to HTML and publish
> versioned documentation online.

You have patches to publish this to the website already which is
good for git master. We still need to get this published for
stable releases, ideally we would have up to publish it for our
pre-existing stable releases back a year or two too.

At the same time though this is insufficient. It is *really* hard
to understand how QAPI schema maps to either JSON or CLI args for
either QMP or the CLI. Most docs examples just talk in terms of
-drive. I think the qapi schema docs could benefit from more
work to provide real world example usage inline.


> 2. scripts/qmp/ contains command-line tools for QMP communication.
> They could use some polish and then be shipped.

The qmp-shell in particular is interesting. Libvirt has a similar
need for a simpler way to deal with QMP and "virsh qemu-monitor-command"
has notable overlap with functionality of qmp-shell.

I wonder if there's a way to spin this off so that we have have a
standard QMP client shell that can either talk direct to a UNIX
socket, or indirect via libvirt's QMP command passthrough APIs.

> 3. python/qemu/ contains Python modules for managing a QEMU process
> and QMP communication.  This should be packaged in distros and
> available on PyPI.
>
> 4. Go and Rust bindings would also be useful.  There is
> https://github.com/intel/govmm but I think it makes sense to keep it
> in qemu.git and provide an interface similar to our Python modules.

Presumably you mean bindings for the QMP commands ?  This will
quickly expand to mean bindings for every aspect of QEMU configuration
that's currently QAPI-ified. It will quickly become a very big job
to design, develop & maintain, especially if you want to support
multiple languages in a natural way.

> 5. A jailer is needed to isolate the QEMU process and vhost-user
> device backends using seccomp, Linux namespaces, and maybe
> SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> security, but it's becoming a common task for any device backend and
> IMO should be its own launcher tool.

Based on experiance in libvirt, this is an even larger job than (4),
as the feature set here is huge.  Much of it directly ties into the
config problem, as to deal with SELinux / namespace setup the code
needs to understand what resources to provide access to. This
requires a way to express 100% coverage of all QEMU configuration
in use & analyse it to determine what resources it implies. So this
ties strongly into QAPI-ification completion.


On the libvirt side, we've a strong long term desire to move our
codebase from C to either Go or Rust. In doing this refactoring
and rewriting, we would quite likely end up with self-contained
packages / modules / APIs that can be used independantly of the
main libvirt API. Thus there's probably scope here to collaborate
between libvirt & QEMU communities to develop new APIs.

Some of this code could be directly QEMU related, but other parts
of the code are likely to be generic & not tied to QEMU at all.
For example libvirt's SELinux, namespace, cgroup setup code is used
by LXC too. Thus from the libvirt POV, some libraries might be
deliverable by QEMU project, but others might be deliverable by
libvirt project. Or might be scope for a indepent new project to
deliver pieces.

Essentially we can consider a stack

     +--------------+
     |  Libvirt API |
     +--------------+
           |
	   V
     +--------------+
     | Helper APIs  |
     +--------------+
           |
	   V
     +--------------+
     |  QEMU        |
     +--------------+

Right now, everything in "Helper APIs" is hidden inside libvirt.
Some of that conceptually ought to be deliverable by QEMU. Some
of that ought to be spun out as separate deliverables by libvirt
or another independant project.

> 6. A configuration file format is sorely needed so that guest
> configuration can be persisted and easily launched.  Today you have to
> create a shell script that executes a QEMU command-line, but this is
> suboptimal because sharing executable scripts is dangerous from a
> security perspective and is hard to parse or modify programmatically.

A configuration file is just one aspect of our broader configuration
problem.

Our CLI in particular suffers from trying to be "all things to all people"
and IMHO is a direct cause of the perception of QEMU as being overly
complex, hard to understand & burdened by legacy support. Of course we
didn't set out intentionally on this path, things just grew organically
we were not succesfully enough at tackling technical debt over time.

To pick on blockdev, as it has had the most growth, IIUC we have

  $QEMU FILENAME
  $QEMU -hda FILENAME
  $QEMU -drive if=ide,file=FILENAME
  $QEMU -drive if=none,file=FILENAME,id=disk0 -device virtio-blk,drive=disk0
  $QEMU -blockdev driver=raw,file.filename=FILENAME,id=disk0 -device virtio-blk,drive=disk0

IIUC, in (all/most) cases "FILENAME" can be either a plain filename,
or a "json:{....}" string, so that's giving us possibly as many as
10 ways to configure disks, with increasing generality.

If you were designing QEMU from a clean slate it is unlikely we'd
plan to have 10 syntaxes. A clean slate would probably provide
2 different syntaxes at most - one simple but convenient, and one
complex but fully expressive.

For every one there's going to be some set of users who think
their chosen syntax is ideal, and will thus be unhappy if we
cull one choice.  For the health of QEMU long term though, I
think we're going to have to accept the need to make some people
temporarily unhappy in order to become sustaininable over the
long term.


This ultimately all stems from the fact that the CLI design has
aimed to provide 100% coverage of all QEMU features for mgmt apps,
while at the same time as providing convenient syntax for humans.
Except it has actually failed in the former, because there's some
stuff you can only configured after launching QEMU via QMP commands.

We can reduce some of the maint burden by mapping "convenient syntax"
onto the more expressive (usually QAPI modelled) syntax internally,
but I think that alone is insufficient.

We should step back & consider the bigger question of what we want
QEMU's approach to configuration to be.

With my libvirt hat on, we have to support JSON for a large portion
of config because we need to use QMP hotplug. Thus if we could eliminate
the need to use the CLI syntax entirely, it would remove / simplify
libvirt's code base to work exclusively in terms of JSON.

With this in mind, I think we could reasonably declare a high level
goal that

 - CLI arg syntax is *exclusively* for human targetted convenient syntax
 - CLI arg syntax will *not* aim for 100% feature coverage
 - CLI arg syntax will not guarantee strict back compat but will not
   gratituitously break human usage
 - JSON syntax will provide 100% feature coverage for mmgmt apps and humans
 - JSON syntax will not be exposed via ARGV
 - JSON syntax will be usable via a config file
 - JSON syntax will be usable via QMP
 - JSON syntax will provide strict back compat, but with deprecations
   policy followed

This is more or less what our long term goal with QAPI has always
been. The sticking point has always been the tension with CLI arg
syntax back-compat. We need to decide how to cut the knot so that
we can complete the QAPI-ification in a forseeable amount of time.

> In many of these areas we already have a partial solution.  It just
> needs more work.  I think it would be worth the effort and the mental
> shift to really providing APIs that are easy to use by applications.
> 
> What do you think?

Personally I think QEMU configuration is the single biggest pain point
for dealing with QEMU that we're not investing enough resources in.

> Have I missed things that are needed?
> 
> Have I included things that are unnecessary?

Many of the things you list above are conceptually nice, but imply a
lot of extra work for maintainers, as they are expanding the scope of
the QEMU project. There's a risk this could divert resources away from
tackling technical debt in things we're already maintaining. The flip
side of course is that some of the things might attract new contributors
to the project, because new concepts are inevitably more interesting to
some people than tackling technical debt. IOW both are important and
we need to find a balance between them.

One of the serious long term issues we've had in QEMU is our inability
to finish jobs that we start. The configuration problem is the biggest
example or something we started JSON-ifying 10 years ago, but are still
an unknowable amount of time away from finishing.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 11722 bytes --]

On Tue, Dec 24, 2019 at 01:00:35PM +0000, Daniel P. Berrangé wrote:
> On Fri, Dec 20, 2019 at 04:13:59PM +0000, Stefan Hajnoczi wrote:
> > Hi,
> > QEMU presents a command-line interface and QMP monitor for
> > applications to interact with.  Applications actually need API
> > bindings in their programming language.  Bindings avoid reimplementing
> > code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> > lazy and de facto relies on libvirt for API bindings.
> > 
> > Is it time for better QEMU APIs?
> > 
> > 1. We have qapi-schema.json.  Let's render to HTML and publish
> > versioned documentation online.
> 
> You have patches to publish this to the website already which is
> good for git master. We still need to get this published for
> stable releases, ideally we would have up to publish it for our
> pre-existing stable releases back a year or two too.
> 
> At the same time though this is insufficient. It is *really* hard
> to understand how QAPI schema maps to either JSON or CLI args for
> either QMP or the CLI. Most docs examples just talk in terms of
> -drive. I think the qapi schema docs could benefit from more
> work to provide real world example usage inline.
> 
> 
> > 2. scripts/qmp/ contains command-line tools for QMP communication.
> > They could use some polish and then be shipped.
> 
> The qmp-shell in particular is interesting. Libvirt has a similar
> need for a simpler way to deal with QMP and "virsh qemu-monitor-command"
> has notable overlap with functionality of qmp-shell.
> 
> I wonder if there's a way to spin this off so that we have have a
> standard QMP client shell that can either talk direct to a UNIX
> socket, or indirect via libvirt's QMP command passthrough APIs.

A standard (and easier-to-use and powerful) QMP shell would be nice.

> > 3. python/qemu/ contains Python modules for managing a QEMU process
> > and QMP communication.  This should be packaged in distros and
> > available on PyPI.
> >
> > 4. Go and Rust bindings would also be useful.  There is
> > https://github.com/intel/govmm but I think it makes sense to keep it
> > in qemu.git and provide an interface similar to our Python modules.
> 
> Presumably you mean bindings for the QMP commands ?  This will
> quickly expand to mean bindings for every aspect of QEMU configuration
> that's currently QAPI-ified. It will quickly become a very big job
> to design, develop & maintain, especially if you want to support
> multiple languages in a natural way.

No, I mean just launching QEMU and communicating over QMP.

I think creating bindings for QMP commands is too much effort.  It would
duplicate the QAPI schema documentation because we'd need native API
documentation too.

I think clients should work at the QMP JSON level.

> > 5. A jailer is needed to isolate the QEMU process and vhost-user
> > device backends using seccomp, Linux namespaces, and maybe
> > SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> > security, but it's becoming a common task for any device backend and
> > IMO should be its own launcher tool.
> 
> Based on experiance in libvirt, this is an even larger job than (4),
> as the feature set here is huge.  Much of it directly ties into the
> config problem, as to deal with SELinux / namespace setup the code
> needs to understand what resources to provide access to. This
> requires a way to express 100% coverage of all QEMU configuration
> in use & analyse it to determine what resources it implies. So this
> ties strongly into QAPI-ification completion.

I don't have a clear design in mind yet but I've learnt a little from
writing the virtiofsd namespace isolation and seccomp code.  The task is
really hard for a general-purpose QEMU binary that supports all
features, because configuring the sandbox with the minimal attack
surface requires knowledge of the feature set being used in this QEMU
invocation :(.

> On the libvirt side, we've a strong long term desire to move our
> codebase from C to either Go or Rust. In doing this refactoring
> and rewriting, we would quite likely end up with self-contained
> packages / modules / APIs that can be used independantly of the
> main libvirt API. Thus there's probably scope here to collaborate
> between libvirt & QEMU communities to develop new APIs.
> 
> Some of this code could be directly QEMU related, but other parts
> of the code are likely to be generic & not tied to QEMU at all.
> For example libvirt's SELinux, namespace, cgroup setup code is used
> by LXC too. Thus from the libvirt POV, some libraries might be
> deliverable by QEMU project, but others might be deliverable by
> libvirt project. Or might be scope for a indepent new project to
> deliver pieces.

I think duplication is in the nature of these isolation mechanisms.
systemd also supports most of the features necessary for isolation but
neither QEMU nor libvirt take advantage of systemd and it's not easy to
do so either.  I think it will be necessary to be pragmatic and look at
it on a case-by-case basis.  There will be some duplication.

> Essentially we can consider a stack
> 
>      +--------------+
>      |  Libvirt API |
>      +--------------+
>            |
> 	   V
>      +--------------+
>      | Helper APIs  |
>      +--------------+
>            |
> 	   V
>      +--------------+
>      |  QEMU        |
>      +--------------+
> 
> Right now, everything in "Helper APIs" is hidden inside libvirt.
> Some of that conceptually ought to be deliverable by QEMU. Some
> of that ought to be spun out as separate deliverables by libvirt
> or another independant project.

I would like QEMU to directly expose APIs without the need for another
level of APIs, where this makes sense.  It's inefficient and slow to
develop stable APIs in QEMU and then repeat that work again in higher
layers of the stack.  To some extent this is unavoidable but I think
QEMU is close to providing usable APIs and tools itself in the areas
I've outlined in this email thread.

By doing this we'll create a better experience for Kata Containers and
other programs that directly use QEMU.

> > 6. A configuration file format is sorely needed so that guest
> > configuration can be persisted and easily launched.  Today you have to
> > create a shell script that executes a QEMU command-line, but this is
> > suboptimal because sharing executable scripts is dangerous from a
> > security perspective and is hard to parse or modify programmatically.
> 
> A configuration file is just one aspect of our broader configuration
> problem.
> 
> Our CLI in particular suffers from trying to be "all things to all people"
> and IMHO is a direct cause of the perception of QEMU as being overly
> complex, hard to understand & burdened by legacy support. Of course we
> didn't set out intentionally on this path, things just grew organically
> we were not succesfully enough at tackling technical debt over time.
> 
> To pick on blockdev, as it has had the most growth, IIUC we have
> 
>   $QEMU FILENAME
>   $QEMU -hda FILENAME
>   $QEMU -drive if=ide,file=FILENAME
>   $QEMU -drive if=none,file=FILENAME,id=disk0 -device virtio-blk,drive=disk0
>   $QEMU -blockdev driver=raw,file.filename=FILENAME,id=disk0 -device virtio-blk,drive=disk0
> 
> IIUC, in (all/most) cases "FILENAME" can be either a plain filename,
> or a "json:{....}" string, so that's giving us possibly as many as
> 10 ways to configure disks, with increasing generality.
> 
> If you were designing QEMU from a clean slate it is unlikely we'd
> plan to have 10 syntaxes. A clean slate would probably provide
> 2 different syntaxes at most - one simple but convenient, and one
> complex but fully expressive.
> 
> For every one there's going to be some set of users who think
> their chosen syntax is ideal, and will thus be unhappy if we
> cull one choice.  For the health of QEMU long term though, I
> think we're going to have to accept the need to make some people
> temporarily unhappy in order to become sustaininable over the
> long term.

Agreed :).

> This ultimately all stems from the fact that the CLI design has
> aimed to provide 100% coverage of all QEMU features for mgmt apps,
> while at the same time as providing convenient syntax for humans.
> Except it has actually failed in the former, because there's some
> stuff you can only configured after launching QEMU via QMP commands.
> 
> We can reduce some of the maint burden by mapping "convenient syntax"
> onto the more expressive (usually QAPI modelled) syntax internally,
> but I think that alone is insufficient.
> 
> We should step back & consider the bigger question of what we want
> QEMU's approach to configuration to be.
> 
> With my libvirt hat on, we have to support JSON for a large portion
> of config because we need to use QMP hotplug. Thus if we could eliminate
> the need to use the CLI syntax entirely, it would remove / simplify
> libvirt's code base to work exclusively in terms of JSON.
> 
> With this in mind, I think we could reasonably declare a high level
> goal that
> 
>  - CLI arg syntax is *exclusively* for human targetted convenient syntax
>  - CLI arg syntax will *not* aim for 100% feature coverage
>  - CLI arg syntax will not guarantee strict back compat but will not
>    gratituitously break human usage
>  - JSON syntax will provide 100% feature coverage for mmgmt apps and humans
>  - JSON syntax will not be exposed via ARGV
>  - JSON syntax will be usable via a config file
>  - JSON syntax will be usable via QMP
>  - JSON syntax will provide strict back compat, but with deprecations
>    policy followed
> 
> This is more or less what our long term goal with QAPI has always
> been. The sticking point has always been the tension with CLI arg
> syntax back-compat. We need to decide how to cut the knot so that
> we can complete the QAPI-ification in a forseeable amount of time.

Sounds good.

> > In many of these areas we already have a partial solution.  It just
> > needs more work.  I think it would be worth the effort and the mental
> > shift to really providing APIs that are easy to use by applications.
> > 
> > What do you think?
> 
> Personally I think QEMU configuration is the single biggest pain point
> for dealing with QEMU that we're not investing enough resources in.
> 
> > Have I missed things that are needed?
> > 
> > Have I included things that are unnecessary?
> 
> Many of the things you list above are conceptually nice, but imply a
> lot of extra work for maintainers, as they are expanding the scope of
> the QEMU project. There's a risk this could divert resources away from
> tackling technical debt in things we're already maintaining. The flip
> side of course is that some of the things might attract new contributors
> to the project, because new concepts are inevitably more interesting to
> some people than tackling technical debt. IOW both are important and
> we need to find a balance between them.
> 
> One of the serious long term issues we've had in QEMU is our inability
> to finish jobs that we start. The configuration problem is the biggest
> example or something we started JSON-ifying 10 years ago, but are still
> an unknowable amount of time away from finishing.

I think now is the right time to do it because there is awareness and
willingness to find solutions both in libvirt and QEMU.  I'm optimistic
that progress can be made here.

The next step is a more detailed plan so that multiple people can work
on these tasks.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Making QEMU easier for management tools and applications

[John Snow]

On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
> Based on experiance in libvirt, this is an even larger job than (4),
> as the feature set here is huge.  Much of it directly ties into the
> config problem, as to deal with SELinux / namespace setup the code
> needs to understand what resources to provide access to. This
> requires a way to express 100% coverage of all QEMU configuration
> in use & analyse it to determine what resources it implies. So this
> ties strongly into QAPI-ification completion.

Is it totally bonkers to suggest that QEMU provide a method of digesting
a given configuration and returning a configuration object that a
standalone jailer can use?

So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
manager cooperate to produce the jailing configuration, and the jailer
does what we ask it to.

Nuts?

--js

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
>> Based on experiance in libvirt, this is an even larger job than (4),
>> as the feature set here is huge.  Much of it directly ties into the
>> config problem, as to deal with SELinux / namespace setup the code
>> needs to understand what resources to provide access to. This
>> requires a way to express 100% coverage of all QEMU configuration
>> in use & analyse it to determine what resources it implies. So this
>> ties strongly into QAPI-ification completion.
>
> Is it totally bonkers to suggest that QEMU provide a method of digesting
> a given configuration and returning a configuration object that a
> standalone jailer can use?
>
> So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
> manager cooperate to produce the jailing configuration, and the jailer
> does what we ask it to.
>
> Nuts?

With the nuts-o-meter calibrated for QEMU CLI: nope, this hardly moves
the needle.

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Wed, Jan 22, 2020 at 05:42:10PM -0500, John Snow wrote:
> 
> 
> On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
> > Based on experiance in libvirt, this is an even larger job than (4),
> > as the feature set here is huge.  Much of it directly ties into the
> > config problem, as to deal with SELinux / namespace setup the code
> > needs to understand what resources to provide access to. This
> > requires a way to express 100% coverage of all QEMU configuration
> > in use & analyse it to determine what resources it implies. So this
> > ties strongly into QAPI-ification completion.
> 
> Is it totally bonkers to suggest that QEMU provide a method of digesting
> a given configuration and returning a configuration object that a
> standalone jailer can use?
> 
> So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
> manager cooperate to produce the jailing configuration, and the jailer
> does what we ask it to.

It isn't clear what you mean by "QEMU" here. If this QEMU, the system
emulator process, then this is the untrustworthy part of the stack,
so the jailer must not use any data that QEMU is providing. In fact
during startup the jailer does its work before QEMU even exists.

There are aspects to the confinement that use / rely on knowledge that
QEMU doesn't normally have, or are expressed in a different way that
which QEMU uses, or needs to take a different imlpementation approach to
that which QEMU normally has.

For networking, for example, from QEMU's config POV, there's just a
TAP file descriptor. There are then a huge number of ways in which
that TAP FD has been connected to the network in the host that are
invisible to QEMU. Plain bridge, openvswitch bridge, macvtap device
all with varying configs. Knowledge of this is relevant to the manager
process and the jailer but irrelevant to QEMU.

When configuring disks we have technical issues. For example we need
to identify the full backing chain and grant the appropriate permissions
on this. Even if there was a libqemublock.so, libvirt would not use this
because the QEMU storage code design is not reliable & minimal enough.
For example to just query the backing file, QEMU opens the qcow2 and
parses all the data about it, building up L1/L2 tables, and other
data structures involved. It is trivial to create qcow2 files which
result in both memory and CPU denial of service merely from opening
the file.  Libvirt's approach to this is minimalist just having a
data table of offsets to the key fields in each file format. So we
can extract the backing file & its format without reading anything
else from the disk.

When configuring chardevs there is a choice of how to do it - we
could just pass the UNIX socket path in, or we could create the
UNIX socket ourselves & pass in the pre-opened FD. Both are equally
functional from QEMU's POV and the end user's POV, but passing a
pre-opened FD is more convenient for libvirt's needs as it allowed
for race-free startups sychronization between libvirt & QEMU, or
rather QMP.  The different options here though, have different
needs on the jailer, because extra steps are needed when passing
pre-opened FD to get the SELinux labelling right. QEMU doesn't
know which approach the mgmt app will want to take, so we can't
ask QEMU how the jailer should be configured - the mgmt app needs
to make that decision.

Essentially we have 2 configuration formats - the high level one
that the mgmt app layer uses & the low level one that QEMU uses.
The component in the stack which maps between the two config
formats, is that one that has the knowledge to configure the
jailer. This isn't QEMU. It is whatever is immediately above QEMU,
currently libvirt, but something conceptually equivalent to the
role libvirt's QEMU driver impl fills.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/23/20 5:27 AM, Daniel P. Berrangé wrote:
> On Wed, Jan 22, 2020 at 05:42:10PM -0500, John Snow wrote:
>>
>>
>> On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
>>> Based on experiance in libvirt, this is an even larger job than (4),
>>> as the feature set here is huge.  Much of it directly ties into the
>>> config problem, as to deal with SELinux / namespace setup the code
>>> needs to understand what resources to provide access to. This
>>> requires a way to express 100% coverage of all QEMU configuration
>>> in use & analyse it to determine what resources it implies. So this
>>> ties strongly into QAPI-ification completion.
>>
>> Is it totally bonkers to suggest that QEMU provide a method of digesting
>> a given configuration and returning a configuration object that a
>> standalone jailer can use?
>>
>> So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
>> manager cooperate to produce the jailing configuration, and the jailer
>> does what we ask it to.
> 
> It isn't clear what you mean by "QEMU" here. If this QEMU, the system
> emulator process, then this is the untrustworthy part of the stack,
> so the jailer must not use any data that QEMU is providing. In fact
> during startup the jailer does its work before QEMU even exists.
> 

I worried about this. Hence the "Nuts?" ask. It sounds like the ultimate
problem is nobody can know -- except QEMU -- what permissions are truly
needed for a given configuration. Even if we had an immaculate API, how
would anyone except QEMU developers know?

Trial and error, perhaps, on behalf of the jailer developers. Trial and
error is not the greatest feature of a security mechanism. Clearly, a
lot of effort has been spent to get libvirt's implementation correct,
but Stefan raises the idea that other projects have need of
understanding how to map QEMU configurations to appropriate jails.

Worse, it could still change on a whim. We (QEMU developers) probably
are not used to thinking of permitted syscall lists as ABI that we
strive to maintain. It can change.

How do we make this easier in a way that doesn't trust QEMU? I feel like
QEMU needs to provide *some* kind of information that can be used to
build better jailing configurations...

> There are aspects to the confinement that use / rely on knowledge that
> QEMU doesn't normally have, or are expressed in a different way that
> which QEMU uses, or needs to take a different imlpementation approach to
> that which QEMU normally has.
> 
> For networking, for example, from QEMU's config POV, there's just a
> TAP file descriptor. There are then a huge number of ways in which
> that TAP FD has been connected to the network in the host that are
> invisible to QEMU. Plain bridge, openvswitch bridge, macvtap device
> all with varying configs. Knowledge of this is relevant to the manager
> process and the jailer but irrelevant to QEMU.
> 
> When configuring disks we have technical issues. For example we need
> to identify the full backing chain and grant the appropriate permissions
> on this. Even if there was a libqemublock.so, libvirt would not use this
> because the QEMU storage code design is not reliable & minimal enough.
> For example to just query the backing file, QEMU opens the qcow2 and
> parses all the data about it, building up L1/L2 tables, and other
> data structures involved. It is trivial to create qcow2 files which
> result in both memory and CPU denial of service merely from opening
> the file.  Libvirt's approach to this is minimalist just having a
> data table of offsets to the key fields in each file format. So we
> can extract the backing file & its format without reading anything
> else from the disk.
> 
> When configuring chardevs there is a choice of how to do it - we
> could just pass the UNIX socket path in, or we could create the
> UNIX socket ourselves & pass in the pre-opened FD. Both are equally
> functional from QEMU's POV and the end user's POV, but passing a
> pre-opened FD is more convenient for libvirt's needs as it allowed
> for race-free startups sychronization between libvirt & QEMU, or
> rather QMP.  The different options here though, have different
> needs on the jailer, because extra steps are needed when passing
> pre-opened FD to get the SELinux labelling right. QEMU doesn't
> know which approach the mgmt app will want to take, so we can't
> ask QEMU how the jailer should be configured - the mgmt app needs
> to make that decision.
> 
> Essentially we have 2 configuration formats - the high level one
> that the mgmt app layer uses & the low level one that QEMU uses.
> The component in the stack which maps between the two config
> formats, is that one that has the knowledge to configure the
> jailer. This isn't QEMU. It is whatever is immediately above QEMU,
> currently libvirt, but something conceptually equivalent to the
> role libvirt's QEMU driver impl fills.
> 
> Regards,
> Daniel
> 

-- 
—js

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Thu, Jan 23, 2020 at 01:13:33PM -0500, John Snow wrote:
> 
> 
> On 1/23/20 5:27 AM, Daniel P. Berrangé wrote:
> > On Wed, Jan 22, 2020 at 05:42:10PM -0500, John Snow wrote:
> >>
> >>
> >> On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
> >>> Based on experiance in libvirt, this is an even larger job than (4),
> >>> as the feature set here is huge.  Much of it directly ties into the
> >>> config problem, as to deal with SELinux / namespace setup the code
> >>> needs to understand what resources to provide access to. This
> >>> requires a way to express 100% coverage of all QEMU configuration
> >>> in use & analyse it to determine what resources it implies. So this
> >>> ties strongly into QAPI-ification completion.
> >>
> >> Is it totally bonkers to suggest that QEMU provide a method of digesting
> >> a given configuration and returning a configuration object that a
> >> standalone jailer can use?
> >>
> >> So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
> >> manager cooperate to produce the jailing configuration, and the jailer
> >> does what we ask it to.
> > 
> > It isn't clear what you mean by "QEMU" here. If this QEMU, the system
> > emulator process, then this is the untrustworthy part of the stack,
> > so the jailer must not use any data that QEMU is providing. In fact
> > during startup the jailer does its work before QEMU even exists.
> > 
> 
> I worried about this. Hence the "Nuts?" ask. It sounds like the ultimate
> problem is nobody can know -- except QEMU -- what permissions are truly
> needed for a given configuration. Even if we had an immaculate API, how
> would anyone except QEMU developers know?
> 
> Trial and error, perhaps, on behalf of the jailer developers. Trial and
> error is not the greatest feature of a security mechanism. Clearly, a
> lot of effort has been spent to get libvirt's implementation correct,
> but Stefan raises the idea that other projects have need of
> understanding how to map QEMU configurations to appropriate jails.
> 
> Worse, it could still change on a whim. We (QEMU developers) probably
> are not used to thinking of permitted syscall lists as ABI that we
> strive to maintain. It can change.
> 
> How do we make this easier in a way that doesn't trust QEMU? I feel like
> QEMU needs to provide *some* kind of information that can be used to
> build better jailing configurations...

I think the key thing to understand is that once you involve a "jailer"
then "QEMU" is no longer a single thing we can refer to. It is immediately
split into two pieces, "the launcher" and "the runtime", with a jailer
sitting in between the pair.

The configuration information needed by the launcher is a superset
of the information needed by the runtime. The launcher is trustworthy
and the runtime is untrustworthy.

In the current world qemu-system-* is the runtime and libvirt is
the launcher, and they each have their own completely different
vocabulary for configuration.

Essentially what you're talking about is a way to bring the concept
of the launcher under the responsibility of the QEMU project, and
have much closer alignment between the configuration for the launcher
and the runtime. One would still need to be the superset of the
other, but they could be 90% common. 

Having "the launcher" be part of the QEMU project deliverables is
a completely reasonable concept. Libvirt only took on that role
itself because QEMU has never provided any solution for this
problem itself.  I don't think this is a technically difficult
task. The issues are simply around the practicality of the size/
scope of the work required in order to achieve deliver it for the
diverse range of QEMU functionality & use cases.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@gmail.com) wrote:

> 5. A jailer is needed to isolate the QEMU process and vhost-user
> device backends using seccomp, Linux namespaces, and maybe
> SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> security, but it's becoming a common task for any device backend and
> IMO should be its own launcher tool.

It feels like there's two parts to this:
  a) The sandboxing at startup
  b) Then whatever comms is needed to allow things like hotplugging
    afterwards, passing fd's etc

you'd think (a) would be some code that would be shared between logs of
projects.

Dave

> 6. A configuration file format is sorely needed so that guest
> configuration can be persisted and easily launched.  Today you have to
> create a shell script that executes a QEMU command-line, but this is
> suboptimal because sharing executable scripts is dangerous from a
> security perspective and is hard to parse or modify programmatically.
> 
> In many of these areas we already have a partial solution.  It just
> needs more work.  I think it would be worth the effort and the mental
> shift to really providing APIs that are easy to use by applications.
> 
> What do you think?
> 
> Have I missed things that are needed?
> 
> Have I included things that are unnecessary?
> 
> Stefan
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin]

Hi Stefan,



> On 20 Dec 2019, at 17:13, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
> Hi,
> QEMU presents a command-line interface and QMP monitor for
> applications to interact with.  Applications actually need API
> bindings in their programming language.  Bindings avoid reimplementing
> code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> lazy and de facto relies on libvirt for API bindings.
> 
> Is it time for better QEMU APIs?

I hear that often ;-)

After catching up with this thread, I think that there is a common,
intuitive sense that a number of topics are interconnected and
relate to the QEMU API, from command-line to language bindings
to sandboxing to configuration files, and even to rewriting in Rust…

A good question is: why do we believe that? Thinking about it, it’s
not entirely obvious, is it, that a C API has anything to do with a
configuration file, is it?

(I _believe_ that it is true, but I’m trying to explore why our intuition
tells us so)

My gut feeling at this point is that we are not talking about an API,
we are first talking about a _language_, with relatively complicated
and loosely defined semantics. There are signs of this language
everywhere, from some custom object-model to several distinct
syntaxen (from ever-more-complex command-line options to JSON).
In order to see what I mean with “relatively complicated semantics”,
see the recent discussions about -hda vs. -drive vs. -device vs. -blockdev…

It does not help that a large fraction of this language is used to describe
graphs, e.g. network or bus topology. Textual languages make it
notoriously difficult to describe graphs, forcing us to add intermediate
user-provided names e.g. to interconnect a guest device and the
associated host resource, or to precisely define how guest NICs
connect to various virtual networks or host interface cards.

We then have several dialects of what should be, at least conceptually,
the same language, including, but not limited, to an XML dialect, a virsh
dialect, a command-line dialect, a qmp dialect, a JSON dialect that is
partially incorporated in several of the above, etc…

Of note, several tools, including virsh, git, OpenShift or dnf, have started
moving away from the simple “list of switches” towards having sub-commands.
Some, like virsh, even switch to “shell mode” if called without arguments.
So maybe qemu should enter “shell mode” if given no argument, and
accept a command file as input… If only there was no backward
compatibility issue…

So I think that it might help, in the long run, to start defining the
language in question in some abstract way, and then to have rules
for how to transform that abstract language into concrete bindings.
This definition itself is not obvious (at least not to me). For
example, do we have, anywhere but in the C code, the specification
of how one can add a disk to qemu, and what it means?
Say, looking at qemu-options.def, how do I tell that -hda has
anything to do with -device or -blockdev or -help?

I think that the following piece of code from vl.c is revealing:

            case QEMU_OPTION_hda:
            case QEMU_OPTION_hdb:
            case QEMU_OPTION_hdc:
            case QEMU_OPTION_hdd:
                drive_add(IF_DEFAULT, popt->index - QEMU_OPTION_hda, optarg,
                          HD_OPTS);
                break;
            case QEMU_OPTION_blockdev:
                {
                    Visitor *v;
                    BlockdevOptionsQueueEntry *bdo;

                    v = qobject_input_visitor_new_str(optarg, "driver",
                                                      &error_fatal);

                    bdo = g_new(BlockdevOptionsQueueEntry, 1);
                    visit_type_BlockdevOptions(v, NULL, &bdo->bdo,
                                               &error_fatal);
                    visit_free(v);
                    loc_save(&bdo->loc);
                    QSIMPLEQ_INSERT_TAIL(&bdo_queue, bdo, entry);
                    break;
                }
            case QEMU_OPTION_drive:
                if (drive_def(optarg) == NULL) {
                    exit(1);
                }
                break;

Here, we have three cases related to disks in a way or another,
and three entirely different ways of doing things.

AFAICT, qemu already created several meta-languages to define
several aspects of the API, from qemu-options.def to qapi-schema.json.
But maybe at some point we need to go meta once more, and define
a language defining the API from which we could automatically
derive the various bindings, including FFI-style bindings for Rust and Go,
as well as some internal data structures. Ideally, that meta-definition
is something that could be shared between libvirt and qemu so that they
literally speak the same language. Or that could be used to automatically
build a REST interface.

A big issue, though, is that of compatibility. Doing the above starting
from scratch does not seem that complicated. Doing it in a way that
preserves a minimum of interoperability with earlier-generation
software is another ordeal.

So I think that Daniel is right. We may need at some point to start
a NEMU-style offshoot that does not attempt to be compatible,
but explores describing an increasing surface of the API using a
new meta-language from which we can generate, in a consistent
way, at least:

- C bindings
- Command-line options
- Shell bindings (or “HMP”)
- JSON schema or qom description
- Bindings in other languages (Rust, Go, Python)
- Networked versions of the API (socket, REST)
- Client-side code e.g. for libvirt.
- Serialization / deserialization, e.g. for configuration files
- Documentation, including man page and API docs
- Command-line help

At the most fundamental level, I think we need to describe:

- Values, e.g. how we represent names, sizes, paths, etc, possibly
with some user-friendly aspects, e.g. path shortcuts, memory units,
spelling shortcuts (e.g. being able to consistently say -blo for -blockdev
if that’s the shortest option that matches)
- Relations, e.g. how we represent “contains”, “derives from”, “needs”,
“one of”, “one or several”, “attaches to”…
- States, e.g. how do we represent the machine configuration,
or the desired new disk setting
- Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
“start”, “notify”, etc. and how we describe the kind of input they need.
- Possibly more subtle things like support for transactions, commit/rollback,
i.e. “I want to add connect a virtual nic to some host vf, but if anything
along the way fails, I’d like all the cleanup to happen automatically)


> 
> 1. We have qapi-schema.json.  Let's render to HTML and publish
> versioned documentation online.

Added “Documentation” above to make sure this is preserved.
I agree with the assessment that the existing commenting state
is not good enough to make this that valuable at this point.
Notably missing are all the relation aspects.


> 
> 2. scripts/qmp/ contains command-line tools for QMP communication.
> They could use some polish and then be shipped.

Shouldn’t we first identify what they do and how to use them? ;-)

> 
> 3. python/qemu/ contains Python modules for managing a QEMU process
> and QMP communication.  This should be packaged in distros and
> available on PyPI.

That is probably somewhat independent from the rest, although
in my “meta” model, we should certainly be able to also generate
some python bindings.

> 
> 4. Go and Rust bindings would also be useful.  There is
> https://github.com/intel/govmm but I think it makes sense to keep it
> in qemu.git and provide an interface similar to our Python modules.

Same comment as above.

> 
> 5. A jailer is needed to isolate the QEMU process and vhost-user
> device backends using seccomp, Linux namespaces, and maybe
> SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> security, but it's becoming a common task for any device backend and
> IMO should be its own launcher tool.

I believe that this “jailer” comment is interesting. It’s not really
related to a refactoring of APIs, it looks more like an API that we
forgot because libvirt did it, and we now realize that this may not
be the right place for some use cases.

> 
> 6. A configuration file format is sorely needed so that guest
> configuration can be persisted and easily launched.  Today you have to
> create a shell script that executes a QEMU command-line, but this is
> suboptimal because sharing executable scripts is dangerous from a
> security perspective and is hard to parse or modify programmatically.

Agreed. That’s one reason why I started talking about a “language” above.


> In many of these areas we already have a partial solution.  It just
> needs more work.  I think it would be worth the effort and the mental
> shift to really providing APIs that are easy to use by applications.
> 
> What do you think?

Overall, I think this is not a simple problem.

> 
> Have I missed things that are needed?

The main one for me is the meta-language I talked about above.

> 
> Have I included things that are unnecessary?

No, but some of the things that you included surprised me in an “API”
discussion, and that’s what launched me in this “meta-language” direction.


> 
> Stefan
> 

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 07.01.2020 um 18:11 hat Christophe de Dinechin geschrieben:
> So I think that it might help, in the long run, to start defining the
> language in question in some abstract way, and then to have rules
> for how to transform that abstract language into concrete bindings.

I think this abstract language is QAPI. The problem is that we're not
even close to using QAPI for everything. Adding a new language on top of
QAPI instead isn't going to make the conversion process any faster.

> This definition itself is not obvious (at least not to me). For
> example, do we have, anywhere but in the C code, the specification
> of how one can add a disk to qemu, and what it means?
> Say, looking at qemu-options.def, how do I tell that -hda has
> anything to do with -device or -blockdev or -help?

BlockdevOptions in the QAPI schema is what tells you how it _really_
works. The connection to the various command line syntaxes isn't defined
in a declarative way because we don't have a QAPIfied command line yet.
I know that Markus wants to work on this, but I don't know how much time
he actually has to invest in it.

> I think that the following piece of code from vl.c is revealing:
> 
>             case QEMU_OPTION_hda:
>             case QEMU_OPTION_hdb:
>             case QEMU_OPTION_hdc:
>             case QEMU_OPTION_hdd:
>                 drive_add(IF_DEFAULT, popt->index - QEMU_OPTION_hda, optarg,
>                           HD_OPTS);
>                 break;
>             case QEMU_OPTION_blockdev:
>                 {
>                     Visitor *v;
>                     BlockdevOptionsQueueEntry *bdo;
> 
>                     v = qobject_input_visitor_new_str(optarg, "driver",
>                                                       &error_fatal);
> 
>                     bdo = g_new(BlockdevOptionsQueueEntry, 1);
>                     visit_type_BlockdevOptions(v, NULL, &bdo->bdo,
>                                                &error_fatal);
>                     visit_free(v);
>                     loc_save(&bdo->loc);
>                     QSIMPLEQ_INSERT_TAIL(&bdo_queue, bdo, entry);
>                     break;
>                 }
>             case QEMU_OPTION_drive:
>                 if (drive_def(optarg) == NULL) {
>                     exit(1);
>                 }
>                 break;
> 
> Here, we have three cases related to disks in a way or another,
> and three entirely different ways of doing things.

I would say two different ways because drive_add() is just a small
wrapper around drive_def() that overrides a few options.

Describing the semantics of the -drive way is hard. This is one of the
reasons why I would love to get rid of it and replace it with a new
user-friendly option that has a more direct mapping to the -blockdev
way, which in turn just is BlockdevOptions mapped 1:1 to the command
line.

> AFAICT, qemu already created several meta-languages to define
> several aspects of the API, from qemu-options.def to qapi-schema.json.
> But maybe at some point we need to go meta once more, and define
> a language defining the API from which we could automatically
> derive the various bindings, including FFI-style bindings for Rust and Go,
> as well as some internal data structures. Ideally, that meta-definition
> is something that could be shared between libvirt and qemu so that they
> literally speak the same language. Or that could be used to automatically
> build a REST interface.

I think adding an output for additional languages to the QAPI generator
shouldn't be too hard. It already creates multiple things from a single
schema (C data structures and command wrappers, schema introspection
data, documentation, and probably other things that I forgot).

libvirt already speaks QAPI, however without reusing the schema and the
generator from QEMU.

> A big issue, though, is that of compatibility. Doing the above starting
> from scratch does not seem that complicated. Doing it in a way that
> preserves a minimum of interoperability with earlier-generation
> software is another ordeal.

Indeed, this is the major reason why QAPI isn't as pervasive as it
should be.

> So I think that Daniel is right. We may need at some point to start
> a NEMU-style offshoot that does not attempt to be compatible,
> but explores describing an increasing surface of the API using a
> new meta-language from which we can generate, in a consistent
> way, at least:
> 
> - C bindings
> - Command-line options
> - Shell bindings (or “HMP”)
> - JSON schema or qom description
> - Bindings in other languages (Rust, Go, Python)
> - Networked versions of the API (socket, REST)
> - Client-side code e.g. for libvirt.
> - Serialization / deserialization, e.g. for configuration files
> - Documentation, including man page and API docs
> - Command-line help

I think the only thing in this list that can't obviously be covered
easily by QAPI is QOM. Or rather, it's covered by passing through
key=value lists without describing their structure - which, as far as I
understand, is mainly because QOM properties aren't necessarily static,
so we can't provide a statically defined interface for them. Probably
solvable in QEMU, but not without a major effort. In a fork that doesn't
care about compatibility, it should be easier.

> At the most fundamental level, I think we need to describe:
> 
> - Values, e.g. how we represent names, sizes, paths, etc, possibly
> with some user-friendly aspects, e.g. path shortcuts, memory units,
> spelling shortcuts (e.g. being able to consistently say -blo for -blockdev
> if that’s the shortest option that matches)

I don't think user-friendly shortcuts on the command line are "most
fundamental". Whether to accept -blo is an implementation detail of the
command line parser which translates a bunch of strings into QAPI
objects.

> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
> “one of”, “one or several”, “attaches to”…
> - States, e.g. how do we represent the machine configuration,
> or the desired new disk setting
> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
> “start”, “notify”, etc. and how we describe the kind of input they need.
> - Possibly more subtle things like support for transactions, commit/rollback,
> i.e. “I want to add connect a virtual nic to some host vf, but if anything
> along the way fails, I’d like all the cleanup to happen automatically)

This sounds like a different approach from our current QAPI command set
(use a smaller set of operations that can work with a greater variety of
objects).

Does it actually provide more functionality, though?

Kevin

Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin]

> On 8 Jan 2020, at 11:43, Kevin Wolf <kwolf@redhat.com> wrote:
> 
> Am 07.01.2020 um 18:11 hat Christophe de Dinechin geschrieben:
>> So I think that it might help, in the long run, to start defining the
>> language in question in some abstract way, and then to have rules
>> for how to transform that abstract language into concrete bindings.
> 
> I think this abstract language is QAPI.

Currently, yes, that’s the closest we have to a well-defined language,
I pointed it out in my mail, and there is qapi-code-gen.txt to prove it.

So it would certainly make sense for the language I’m describing to
be a forward evolution of QAPI, or something that is trivially transformed
to / from the existing json schema.

> The problem is that we're not even close to using QAPI for everything.

That’s the symptom of the problem. The problem as I see it is that
we could not. at least not with today’s QAPI (maybe I’m wrong).
A good example is shortcut command-line options. I don’t think
you can, today, derive a user-friendly command-line syntax from
qapi-schema.json, and even less so one that is compatible with what
we have today.

> Adding a new language on top of QAPI instead isn't going to make the
> conversion process any faster.

I fully agree that whatever language we select has to have an easy,
systematic forward path from QAPI, but also from the other
existing meta-languages, like the .def files used for options,
or build configuration (there is a hint of the connection between
the two in the option ‘if’ member in the .json files).

In other words, the language I have in mind would be something
that I could auto-generate from, say, the current qapi-schema.json
and the qemu-options.def. Maybe in the end, that would be simply
by having qemu-options.def being used to build qemu-options.json,
and then add the relevant entries in qemu-options.json. Maybe
there is a better way.

> 
>> This definition itself is not obvious (at least not to me). For
>> example, do we have, anywhere but in the C code, the specification
>> of how one can add a disk to qemu, and what it means?
>> Say, looking at qemu-options.def, how do I tell that -hda has
>> anything to do with -device or -blockdev or -help?
> 
> BlockdevOptions in the QAPI schema is what tells you how it _really_
> works.

Not really, IMO, in the sense that it gives me no clue as to how -hda
or -device relate to it.

An important point that you raise through your response, though, is
that the QAPI schema is only an IDL (interface definition language).
In other words, it can tell me what a BlockdevOptions look like on
the wire, but at least in the current state, it cannot tell me what happens
with it.

This is probably a good thing (it’s already complicated enough as is),
but it’s worth pointing out that, even sticking to a declarative language,
one could add constraints, as hinted by HAS_ARG in the .def file
(but we could have a richer constraints language, e.g. describing
-m option so that there can be at most one, the arg is a number,
can’t be negative or floating-point, and so on.

And of course, we could decide to not stick to a declarative language,
but to have a way to have some, possibly limited, processing
described in the language. I’m thinking notably of what is called
“desugaring” in other threads.

> The connection to the various command line syntaxes isn't defined
> in a declarative way because we don't have a QAPIfied command line yet.
> I know that Markus wants to work on this, but I don't know how much time
> he actually has to invest in it.

Agreed. As I said, that’s a symptom that QAPI is not the language
I am talking about, but at best a subset of it.

> 
>> I think that the following piece of code from vl.c is revealing:
>> 
>>            case QEMU_OPTION_hda:
>>            case QEMU_OPTION_hdb:
>>            case QEMU_OPTION_hdc:
>>            case QEMU_OPTION_hdd:
>>                drive_add(IF_DEFAULT, popt->index - QEMU_OPTION_hda, optarg,
>>                          HD_OPTS);
>>                break;
>>            case QEMU_OPTION_blockdev:
>>                {
>>                    Visitor *v;
>>                    BlockdevOptionsQueueEntry *bdo;
>> 
>>                    v = qobject_input_visitor_new_str(optarg, "driver",
>>                                                      &error_fatal);
>> 
>>                    bdo = g_new(BlockdevOptionsQueueEntry, 1);
>>                    visit_type_BlockdevOptions(v, NULL, &bdo->bdo,
>>                                               &error_fatal);
>>                    visit_free(v);
>>                    loc_save(&bdo->loc);
>>                    QSIMPLEQ_INSERT_TAIL(&bdo_queue, bdo, entry);
>>                    break;
>>                }
>>            case QEMU_OPTION_drive:
>>                if (drive_def(optarg) == NULL) {
>>                    exit(1);
>>                }
>>                break;
>> 
>> Here, we have three cases related to disks in a way or another,
>> and three entirely different ways of doing things.
> 
> I would say two different ways because drive_add() is just a small
> wrapper around drive_def() that overrides a few options.

Well, I stick to three, because one option has a test and an exit(1),
and the other does not. Which is the kind of inconsistencies a
well-defined language would get rid of.

> 
> Describing the semantics of the -drive way is hard. This is one of the
> reasons why I would love to get rid of it and replace it with a new
> user-friendly option that has a more direct mapping to the -blockdev
> way, which in turn just is BlockdevOptions mapped 1:1 to the command
> line.

I would love that too :-)

> 
>> AFAICT, qemu already created several meta-languages to define
>> several aspects of the API, from qemu-options.def to qapi-schema.json.
>> But maybe at some point we need to go meta once more, and define
>> a language defining the API from which we could automatically
>> derive the various bindings, including FFI-style bindings for Rust and Go,
>> as well as some internal data structures. Ideally, that meta-definition
>> is something that could be shared between libvirt and qemu so that they
>> literally speak the same language. Or that could be used to automatically
>> build a REST interface.
> 
> I think adding an output for additional languages to the QAPI generator
> shouldn't be too hard. It already creates multiple things from a single
> schema (C data structures and command wrappers, schema introspection
> data, documentation, and probably other things that I forgot).

Agreed.

> libvirt already speaks QAPI, however without reusing the schema and the
> generator from QEMU.

That’s another thread ;-)

> 
>> A big issue, though, is that of compatibility. Doing the above starting
>> from scratch does not seem that complicated. Doing it in a way that
>> preserves a minimum of interoperability with earlier-generation
>> software is another ordeal.
> 
> Indeed, this is the major reason why QAPI isn't as pervasive as it
> should be.

Probably, but also because QAPI does not address the needs of
some of the things a slightly more general language could do.

> 
>> So I think that Daniel is right. We may need at some point to start
>> a NEMU-style offshoot that does not attempt to be compatible,
>> but explores describing an increasing surface of the API using a
>> new meta-language from which we can generate, in a consistent
>> way, at least:
>> 
>> - C bindings
>> - Command-line options
>> - Shell bindings (or “HMP”)
>> - JSON schema or qom description
>> - Bindings in other languages (Rust, Go, Python)
>> - Networked versions of the API (socket, REST)
>> - Client-side code e.g. for libvirt.
>> - Serialization / deserialization, e.g. for configuration files
>> - Documentation, including man page and API docs
>> - Command-line help
> 
> I think the only thing in this list that can't obviously be covered
> easily by QAPI is QOM. Or rather, it's covered by passing through
> key=value lists without describing their structure

That’s close enough to me. (In my mind, that part was already “done”
by QAPI, even if in the convoluted way you describe)

> - which, as far as I
> understand, is mainly because QOM properties aren't necessarily static,
> so we can't provide a statically defined interface for them. Probably
> solvable in QEMU, but not without a major effort.

Or maybe extend the language so that it’s internal semantics
knows about this aspect of QOM?

> In a fork that doesn’t care about compatibility, it should be easier.
> 
>> At the most fundamental level, I think we need to describe:
>> 
>> - Values, e.g. how we represent names, sizes, paths, etc, possibly
>> with some user-friendly aspects, e.g. path shortcuts, memory units,
>> spelling shortcuts (e.g. being able to consistently say -blo for -blockdev
>> if that’s the shortest option that matches)
> 
> I don't think user-friendly shortcuts on the command line are "most
> fundamental".

I agree, my phrasing was poor. What I meant is that the language in
question should include a user-convenience dimension that is
not necessary (and not present) in QAPI. Agree the details are not
fundamental, I meant them as an illustration.

> Whether to accept -blo is an implementation detail of the
> command line parser which translates a bunch of strings into QAPI
> objects.

Yes.

> 
>> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
>> “one of”, “one or several”, “attaches to”…
>> - States, e.g. how do we represent the machine configuration,
>> or the desired new disk setting
>> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
>> “start”, “notify”, etc. and how we describe the kind of input they need.
>> - Possibly more subtle things like support for transactions, commit/rollback,
>> i.e. “I want to add connect a virtual nic to some host vf, but if anything
>> along the way fails, I’d like all the cleanup to happen automatically)
> 
> This sounds like a different approach from our current QAPI command set

Well, except for purposefully trying to use a different wording to avoid
the risk of getting your mind stuck in one of the particular existing
meta-languages in QEMU, the approach is not very different.

- Relations already exist in QAPI for example: ’struct’, ‘union’, ‘alternate’

- States exist, today represented using JSON

- Verbs are called “commands” in the QAPI json files, and are what
follows DEF( in the .def file

- Transactions do not exist today that I know of, although we see
signs of them in discussions about the fact that this options destroys
that back end but that option does not.

> (use a smaller set of operations that can work with a greater variety of
> objects).

More like trying to find terminology that lets you reason about it
without being too mind-constrainted by existing stuff.


> Does it actually provide more functionality, though?

It’s not intended to provide more, but to require less to do the same thing.


Thanks,
Christophe

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 08.01.2020 um 12:40 hat Christophe de Dinechin geschrieben:
> > On 8 Jan 2020, at 11:43, Kevin Wolf <kwolf@redhat.com> wrote:
> > Am 07.01.2020 um 18:11 hat Christophe de Dinechin geschrieben:
> >> So I think that it might help, in the long run, to start defining the
> >> language in question in some abstract way, and then to have rules
> >> for how to transform that abstract language into concrete bindings.
> > 
> > I think this abstract language is QAPI.
> 
> Currently, yes, that’s the closest we have to a well-defined language,
> I pointed it out in my mail, and there is qapi-code-gen.txt to prove it.

Oh, did you? I must have missed that point, sorry.

> So it would certainly make sense for the language I’m describing to
> be a forward evolution of QAPI, or something that is trivially transformed
> to / from the existing json schema.

I guess I need to be more specific about what I mean with "QAPI". Of
course, many things you can't currently do in QAPI because they are not
implemented. When I'm talking about that QAPI should be used more, I'm
mostly talking about applying its core concepts and infrastructure with
the obvious extensions in some places, not about the existing
implementation.

So I guess "a forward evoluion of QAPI" is what I really have in mind.

> > The problem is that we're not even close to using QAPI for everything.
> 
> That’s the symptom of the problem. The problem as I see it is that
> we could not. at least not with today’s QAPI (maybe I’m wrong).
> A good example is shortcut command-line options. I don’t think
> you can, today, derive a user-friendly command-line syntax from
> qapi-schema.json, and even less so one that is compatible with what
> we have today.

Well, you can't because it's not implemented. Implementing it from
scratch without compatibility requirements would be relatively easy, and
we're in fact going to try it out with the qemu-storage-daemon.

But there is nothing in defining a command line option that would
inherently be different in structure from defining a QMP command, so I
would consider providing a command line option that is described by a
QAPI type not a problem, even though it's not implemented yet.

Compatibility is much harder, which is not the least because the
existing command line options are bad (primarily: using inconsistent
syntax). The question is if we should make bad options work unchanged in
the new infrastructure or if it wouldn't be better to replace them with
something that behaves the same as the rest.

> > Adding a new language on top of QAPI instead isn't going to make the
> > conversion process any faster.
> 
> I fully agree that whatever language we select has to have an easy,
> systematic forward path from QAPI, but also from the other
> existing meta-languages, like the .def files used for options,
> or build configuration (there is a hint of the connection between
> the two in the option ‘if’ member in the .json files).
> 
> In other words, the language I have in mind would be something
> that I could auto-generate from, say, the current qapi-schema.json
> and the qemu-options.def. Maybe in the end, that would be simply
> by having qemu-options.def being used to build qemu-options.json,
> and then add the relevant entries in qemu-options.json. Maybe
> there is a better way.

I would rather say that .def should go away and its content should
be converted to a JSON schema that becomes the new source format rather
than an intermediate generated file. There is nothing in .def files that
couldn't be represented in the schema.

> >> This definition itself is not obvious (at least not to me). For
> >> example, do we have, anywhere but in the C code, the specification
> >> of how one can add a disk to qemu, and what it means?
> >> Say, looking at qemu-options.def, how do I tell that -hda has
> >> anything to do with -device or -blockdev or -help?
> > 
> > BlockdevOptions in the QAPI schema is what tells you how it _really_
> > works.
> 
> Not really, IMO, in the sense that it gives me no clue as to how -hda
> or -device relate to it.
> 
> An important point that you raise through your response, though, is
> that the QAPI schema is only an IDL (interface definition language).
> In other words, it can tell me what a BlockdevOptions look like on
> the wire, but at least in the current state, it cannot tell me what happens
> with it.
> 
> This is probably a good thing (it’s already complicated enough as is),
> but it’s worth pointing out that, even sticking to a declarative language,
> one could add constraints, as hinted by HAS_ARG in the .def file
> (but we could have a richer constraints language, e.g. describing
> -m option so that there can be at most one, the arg is a number,
> can’t be negative or floating-point, and so on.

To a certain degree, QAPI does just that, by supporting different data
types for options. If necessary, I'm sure the type system could be
extended, but it's not clear to me to which degree we actually need
this.

Just drawing parallels from what we do for QMP commands, I imagine
something like this to describe the -m option as it actually exists in
qemu-options.def today:

##
# @m:
#
# Configure guest RAM.
#
# @size: initial amount of guest memory
# @slots: number of hotplug slots (default: none)
# @maxmem: maximum amount of guest memory (default: none)
##
{ 'cmdline-option': 'm',
  'arguments': {
    'size': 'size',
    'slots': '*int',
    'maxmem': '*size'
  },
  'default-key': 'size',
  'repeat': 'OVERRIDE' }

HAS_ARG is automatically covered simply by 'arguments' being non-empty.

(Markus probably already has a prototype with slightly different syntax,
but the idea should be similar.)

> And of course, we could decide to not stick to a declarative language,
> but to have a way to have some, possibly limited, processing
> described in the language. I’m thinking notably of what is called
> “desugaring” in other threads.

We could. But is it actually worth inventing a new programming language?
I think this is something that should be done in C code even in the
future. I think what we're looking for is a way to describe interfaces,
not implementations.

Even just for adding more sophisticated constraints, like in the example
above that maxmem >= size, it's questionable whether doing this in the
schema provides enough value for actually implementing it there.

> >> A big issue, though, is that of compatibility. Doing the above starting
> >> from scratch does not seem that complicated. Doing it in a way that
> >> preserves a minimum of interoperability with earlier-generation
> >> software is another ordeal.
> > 
> > Indeed, this is the major reason why QAPI isn't as pervasive as it
> > should be.
> 
> Probably, but also because QAPI does not address the needs of
> some of the things a slightly more general language could do.

So what are the things that you would like to describe, but can't see an
easy way to actually describe them with QAPI concepts?

> >> So I think that Daniel is right. We may need at some point to start
> >> a NEMU-style offshoot that does not attempt to be compatible,
> >> but explores describing an increasing surface of the API using a
> >> new meta-language from which we can generate, in a consistent
> >> way, at least:
> >> 
> >> - C bindings
> >> - Command-line options
> >> - Shell bindings (or “HMP”)
> >> - JSON schema or qom description
> >> - Bindings in other languages (Rust, Go, Python)
> >> - Networked versions of the API (socket, REST)
> >> - Client-side code e.g. for libvirt.
> >> - Serialization / deserialization, e.g. for configuration files
> >> - Documentation, including man page and API docs
> >> - Command-line help
> > 
> > I think the only thing in this list that can't obviously be covered
> > easily by QAPI is QOM. Or rather, it's covered by passing through
> > key=value lists without describing their structure
> 
> That’s close enough to me. (In my mind, that part was already “done”
> by QAPI, even if in the convoluted way you describe)

That's not really close enough. If you are happy with this level of
abstraction, you can define any command line option to be either a flag
or take a string as its argument and be done. This is obviously not very
helpful because it says nothing about the structure of that string. In
the same way, QAPI can't say anything about the structure of a QOM
object, and I think that's a problem.

> > - which, as far as I
> > understand, is mainly because QOM properties aren't necessarily static,
> > so we can't provide a statically defined interface for them. Probably
> > solvable in QEMU, but not without a major effort.
> 
> Or maybe extend the language so that it’s internal semantics
> knows about this aspect of QOM?

My point is that for example you can't generate a C struct (which is
statically defined) from something that has a dynamic structure. The
best you could do is fall back to key=value even in the C source, both
key and value being strings. But then you still have to parse these
strings manually, so it doesn't actually help much compared to a state
without C bindings.

Maybe what could be done is covering at least the static properties and
then having key=value for the dynamic part (which should be the
exception).

> >> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
> >> “one of”, “one or several”, “attaches to”…
> >> - States, e.g. how do we represent the machine configuration,
> >> or the desired new disk setting
> >> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
> >> “start”, “notify”, etc. and how we describe the kind of input they need.
> >> - Possibly more subtle things like support for transactions, commit/rollback,
> >> i.e. “I want to add connect a virtual nic to some host vf, but if anything
> >> along the way fails, I’d like all the cleanup to happen automatically)
> > 
> > This sounds like a different approach from our current QAPI command set
> 
> Well, except for purposefully trying to use a different wording to avoid
> the risk of getting your mind stuck in one of the particular existing
> meta-languages in QEMU, the approach is not very different.

I didn't necessarily mean relations/state/verbs, which obviously exist,
but the examples that seemed to express things in a deliberately
different way from what we have today.

> - Transactions do not exist today that I know of, although we see
> signs of them in discussions about the fact that this options destroys
> that back end but that option does not.

We have a 'transaction' QMP command, but they are not an integral part
of the language. Considering the trouble to implement transactional
commands, I suppose we don't even want it to be a fundamental part of
the language.

> > Does it actually provide more functionality, though?
> 
> It’s not intended to provide more, but to require less to do the same thing.

Though we always need to keep in mind that if requiring less for future
additions requires a huge effort now, the investment may pay off only in
a very distant future (if the abstraction we build even survives until
then).

Kevin

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Kevin Wolf <kwolf@redhat.com> writes:

> Am 08.01.2020 um 12:40 hat Christophe de Dinechin geschrieben:
>> > On 8 Jan 2020, at 11:43, Kevin Wolf <kwolf@redhat.com> wrote:
>> > Am 07.01.2020 um 18:11 hat Christophe de Dinechin geschrieben:
>> >> So I think that it might help, in the long run, to start defining the
>> >> language in question in some abstract way, and then to have rules
>> >> for how to transform that abstract language into concrete bindings.
>> > 
>> > I think this abstract language is QAPI.
>> 
>> Currently, yes, that’s the closest we have to a well-defined language,
>> I pointed it out in my mail, and there is qapi-code-gen.txt to prove it.
>
> Oh, did you? I must have missed that point, sorry.
>
>> So it would certainly make sense for the language I’m describing to
>> be a forward evolution of QAPI, or something that is trivially transformed
>> to / from the existing json schema.
>
> I guess I need to be more specific about what I mean with "QAPI". Of
> course, many things you can't currently do in QAPI because they are not
> implemented. When I'm talking about that QAPI should be used more, I'm
> mostly talking about applying its core concepts and infrastructure with
> the obvious extensions in some places, not about the existing
> implementation.
>
> So I guess "a forward evoluion of QAPI" is what I really have in mind.
>
>> > The problem is that we're not even close to using QAPI for everything.
>> 
>> That’s the symptom of the problem. The problem as I see it is that
>> we could not. at least not with today’s QAPI (maybe I’m wrong).
>> A good example is shortcut command-line options. I don’t think
>> you can, today, derive a user-friendly command-line syntax from
>> qapi-schema.json, and even less so one that is compatible with what
>> we have today.
>
> Well, you can't because it's not implemented. Implementing it from
> scratch without compatibility requirements would be relatively easy, and
> we're in fact going to try it out with the qemu-storage-daemon.

Yes.

Prior art:

    Presentation
    KVM Forum 2017: Towards a More Expressive and Introspectable QEMU
    Command Line
    https://www.youtube.com/watch?v=gtpOLQgnwug
    https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf

    RFC patches
    https://lists.nongnu.org/archive/html/qemu-devel/2017-10/msg00209.html
    Message-Id: <20171002152552.27999-1-armbru@redhat.com>
    https://repo.or.cz/qemu/armbru.git/shortlog/refs/heads/qapi-cmdline

> But there is nothing in defining a command line option that would
> inherently be different in structure from defining a QMP command, so I
> would consider providing a command line option that is described by a
> QAPI type not a problem, even though it's not implemented yet.
>
> Compatibility is much harder, which is not the least because the
> existing command line options are bad (primarily: using inconsistent
> syntax).

Semantics are just as problematic.

>          The question is if we should make bad options work unchanged in
> the new infrastructure or if it wouldn't be better to replace them with
> something that behaves the same as the rest.
>
>> > Adding a new language on top of QAPI instead isn't going to make the
>> > conversion process any faster.
>> 
>> I fully agree that whatever language we select has to have an easy,
>> systematic forward path from QAPI, but also from the other
>> existing meta-languages, like the .def files used for options,

You mean the qemu-options.hx.  qemu-options.def is generated from it.

We used to have qmp-commands.hx, until QAPI swallowed it.  Command line
QAPIfication will swallow qemu-options.hx.

>> or build configuration (there is a hint of the connection between
>> the two in the option ‘if’ member in the .json files).

I'm not sure what exactly you have in mind for build configuration.

>> In other words, the language I have in mind would be something
>> that I could auto-generate from, say, the current qapi-schema.json
>> and the qemu-options.def. Maybe in the end, that would be simply
>> by having qemu-options.def being used to build qemu-options.json,
>> and then add the relevant entries in qemu-options.json. Maybe
>> there is a better way.
>
> I would rather say that .def should go away and its content should
> be converted to a JSON schema that becomes the new source format rather
> than an intermediate generated file. There is nothing in .def files that
> couldn't be represented in the schema.

Yes.

>> >> This definition itself is not obvious (at least not to me). For
>> >> example, do we have, anywhere but in the C code, the specification
>> >> of how one can add a disk to qemu, and what it means?
>> >> Say, looking at qemu-options.def, how do I tell that -hda has
>> >> anything to do with -device or -blockdev or -help?
>> > 
>> > BlockdevOptions in the QAPI schema is what tells you how it _really_
>> > works.
>> 
>> Not really, IMO, in the sense that it gives me no clue as to how -hda
>> or -device relate to it.
>> 
>> An important point that you raise through your response, though, is
>> that the QAPI schema is only an IDL (interface definition language).
>> In other words, it can tell me what a BlockdevOptions look like on
>> the wire, but at least in the current state, it cannot tell me what happens
>> with it.

True.

>> This is probably a good thing (it’s already complicated enough as is),
>> but it’s worth pointing out that, even sticking to a declarative language,
>> one could add constraints, as hinted by HAS_ARG in the .def file
>> (but we could have a richer constraints language, e.g. describing
>> -m option so that there can be at most one, the arg is a number,
>> can’t be negative or floating-point, and so on.
>
> To a certain degree, QAPI does just that, by supporting different data
> types for options. If necessary, I'm sure the type system could be
> extended, but it's not clear to me to which degree we actually need
> this.

QAPI is much more expressive than qemu-options.hx.  All the latter can
do is "has option argument" and "option is arch-specific".  QAPI lets
you specify an option argument's structure, and supports compile-time
conditionals.  It can't do things like "must specify either argument A
or B", or "numeric argument C must not exceed argument D", or "multiple
options combine vs. last one wins".  Mostly because "it’s already
complicated enough as is".

> Just drawing parallels from what we do for QMP commands, I imagine
> something like this to describe the -m option as it actually exists in
> qemu-options.def today:
>
> ##
> # @m:
> #
> # Configure guest RAM.
> #
> # @size: initial amount of guest memory
> # @slots: number of hotplug slots (default: none)
> # @maxmem: maximum amount of guest memory (default: none)
> ##
> { 'cmdline-option': 'm',
>   'arguments': {
>     'size': 'size',
>     'slots': '*int',
>     'maxmem': '*size'
>   },
>   'default-key': 'size',
>   'repeat': 'OVERRIDE' }
>
> HAS_ARG is automatically covered simply by 'arguments' being non-empty.
>
> (Markus probably already has a prototype with slightly different syntax,
> but the idea should be similar.)

A similar example from my RFC patches:

##
# @--add-fd:
# !texinfo
# -add-fd fd=@var{fd},set=@var{set}[,opaque=@var{opaque}]@*
#
# Add a file descriptor to an fd set.  Valid options are:
#
# @table @option
# @item fd=@var{fd}
# This option defines the file descriptor of which a duplicate is added to fd set.
# The file descriptor cannot be stdin, stdout, or stderr.
# @item set=@var{set}
# This option defines the ID of the fd set to add the file descriptor to.
# @item opaque=@var{opaque}
# This option defines a free-form string that can be used to describe @var{fd}.
# @end table
#
# You can open an image using pre-opened file descriptors from an fd set:
# @example
# qemu-system-i386
# -add-fd fd=3,set=2,opaque="rdwr:/path/to/file"
# -add-fd fd=4,set=2,opaque="rdonly:/path/to/file"
# -drive file=/dev/fdset/2,index=0,media=disk
# @end example
# !end texinfo
##
{ 'option': '--add-fd',
  'data': { 'fd': 'int', 'set': 'int', '*opaque': 'str' },
  'help': [
"-add-fd fd=fd,set=set[,opaque=opaque]",
"                Add 'fd' to fd 'set'"] }

>> And of course, we could decide to not stick to a declarative language,
>> but to have a way to have some, possibly limited, processing
>> described in the language. I’m thinking notably of what is called
>> “desugaring” in other threads.
>
> We could. But is it actually worth inventing a new programming language?
> I think this is something that should be done in C code even in the
> future. I think what we're looking for is a way to describe interfaces,
> not implementations.

Sane sugar can be described declaratively as macro expansion.

Less than sane sugar takes code.

Desugaring in code promotes (accidental) abandonment of sanity.

That said, we have to keep things simple to succeed.  Declarative
desugaring seems beyond our reach.

> Even just for adding more sophisticated constraints, like in the example
> above that maxmem >= size, it's questionable whether doing this in the
> schema provides enough value for actually implementing it there.

Yes.

>> >> A big issue, though, is that of compatibility. Doing the above starting
>> >> from scratch does not seem that complicated. Doing it in a way that
>> >> preserves a minimum of interoperability with earlier-generation
>> >> software is another ordeal.
>> > 
>> > Indeed, this is the major reason why QAPI isn't as pervasive as it
>> > should be.
>> 
>> Probably, but also because QAPI does not address the needs of
>> some of the things a slightly more general language could do.
>
> So what are the things that you would like to describe, but can't see an
> easy way to actually describe them with QAPI concepts?
>
>> >> So I think that Daniel is right. We may need at some point to start
>> >> a NEMU-style offshoot that does not attempt to be compatible,
>> >> but explores describing an increasing surface of the API using a
>> >> new meta-language from which we can generate, in a consistent
>> >> way, at least:
>> >> 
>> >> - C bindings
>> >> - Command-line options
>> >> - Shell bindings (or “HMP”)
>> >> - JSON schema or qom description
>> >> - Bindings in other languages (Rust, Go, Python)
>> >> - Networked versions of the API (socket, REST)
>> >> - Client-side code e.g. for libvirt.
>> >> - Serialization / deserialization, e.g. for configuration files
>> >> - Documentation, including man page and API docs
>> >> - Command-line help
>> > 
>> > I think the only thing in this list that can't obviously be covered
>> > easily by QAPI is QOM. Or rather, it's covered by passing through
>> > key=value lists without describing their structure
>> 
>> That’s close enough to me. (In my mind, that part was already “done”
>> by QAPI, even if in the convoluted way you describe)
>
> That's not really close enough. If you are happy with this level of
> abstraction, you can define any command line option to be either a flag
> or take a string as its argument and be done. This is obviously not very
> helpful because it says nothing about the structure of that string.

Similar to all QMP commands taking a single 'any' argument.

>                                                                     In
> the same way, QAPI can't say anything about the structure of a QOM
> object, and I think that's a problem.
>
>> > - which, as far as I
>> > understand, is mainly because QOM properties aren't necessarily static,
>> > so we can't provide a statically defined interface for them. Probably
>> > solvable in QEMU, but not without a major effort.
>> 
>> Or maybe extend the language so that it’s internal semantics
>> knows about this aspect of QOM?
>
> My point is that for example you can't generate a C struct (which is
> statically defined) from something that has a dynamic structure. The
> best you could do is fall back to key=value even in the C source, both
> key and value being strings. But then you still have to parse these
> strings manually, so it doesn't actually help much compared to a state
> without C bindings.

QOM and QAPI sabotage each other.  Ironic, considering they were dreamed
up by the same guy :)

QAPI is compile-time static by design.

QOM is run-time dynamic by design.  Some support for static definitions
has been grafted on since.

We use QAPI type system escapes for QOM.  Defeats QAPI introspection and
doc generation.  We provide separate QOM introspection instead, which is
clumsier and less expressive.  QOM documentation doesn't exist.

> Maybe what could be done is covering at least the static properties and
> then having key=value for the dynamic part (which should be the
> exception).

To make this worthwhile, we'd have to replace dynamic QOM properties by
static ones when possible.  Monumental task.

>> >> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
>> >> “one of”, “one or several”, “attaches to”…
>> >> - States, e.g. how do we represent the machine configuration,
>> >> or the desired new disk setting
>> >> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
>> >> “start”, “notify”, etc. and how we describe the kind of input they need.
>> >> - Possibly more subtle things like support for transactions, commit/rollback,
>> >> i.e. “I want to add connect a virtual nic to some host vf, but if anything
>> >> along the way fails, I’d like all the cleanup to happen automatically)
>> > 
>> > This sounds like a different approach from our current QAPI command set
>> 
>> Well, except for purposefully trying to use a different wording to avoid
>> the risk of getting your mind stuck in one of the particular existing
>> meta-languages in QEMU, the approach is not very different.
>
> I didn't necessarily mean relations/state/verbs, which obviously exist,
> but the examples that seemed to express things in a deliberately
> different way from what we have today.
>
>> - Transactions do not exist today that I know of, although we see
>> signs of them in discussions about the fact that this options destroys
>> that back end but that option does not.
>
> We have a 'transaction' QMP command, but they are not an integral part
> of the language. Considering the trouble to implement transactional
> commands, I suppose we don't even want it to be a fundamental part of
> the language.
>
>> > Does it actually provide more functionality, though?
>> 
>> It’s not intended to provide more, but to require less to do the same thing.
>
> Though we always need to keep in mind that if requiring less for future
> additions requires a huge effort now, the investment may pay off only in
> a very distant future (if the abstraction we build even survives until
> then).

Worse is better.

http://dreamsongs.com/RiseOfWorseIsBetter.html

Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin]

I started cutting some stuff out.

> On 14 Jan 2020, at 14:04, Markus Armbruster <armbru@redhat.com> wrote:
> 
> Prior art:
> 
>    Presentation
>    KVM Forum 2017: Towards a More Expressive and Introspectable QEMU
>    Command Line
>    https://www.youtube.com/watch?v=gtpOLQgnwug
>    https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf
> 
>    RFC patches
>    https://lists.nongnu.org/archive/html/qemu-devel/2017-10/msg00209.html
>    Message-Id: <20171002152552.27999-1-armbru@redhat.com>
>    https://repo.or.cz/qemu/armbru.git/shortlog/refs/heads/qapi-cmdline

Very useful, thanks.

> 
>> Compatibility is much harder, which is not the least because the
>> existing command line options are bad (primarily: using inconsistent
>> syntax).
> 
> Semantics are just as problematic.

Yes, but often you can leverage something (e.g. JSON, XML, etc)

We could decide that the new QAPI language is a custom implementation
of Python ;-) Or more seriously, we could have some part of the JSON
schema that are “escapes” to C code that we put in some generated file.
I used {{ }} below for this escape, in which case the semantics would be
that of C plus some specific macros we insert in it.

{ 'option': '--add-fd',
 'data': { 'fd': 'int', 'set': 'int', '*opaque': 'str' },
 'help': [
"-add-fd fd=fd,set=set[,opaque=opaque]",
"                Add 'fd' to fd 'set'”] 
 ‘validate’ : {{
    error_if(QOPT.fd < 0, “fd option cannot be negative”);
    error_if(QOPT.fd <= STDERR_FILENO, “fd cannot be a standard I/O stream”);
 }},
 ‘exec’ : {{
    if (do_add_fd(QOPT.fd, QOPT.fset, QOPT.opaque) < 0) {
        exit(1);
    }
 }},

I’m not necessarily saying this is a good idea, mind you.

> 
> You mean the qemu-options.hx.  qemu-options.def is generated from it.

Oh, I had missed that, thanks.

>>> or build configuration (there is a hint of the connection between
>>> the two in the option ‘if’ member in the .json files).
> 
> I'm not sure what exactly you have in mind for build configuration.

I meant what you pointed out wrt. options, that this “language” needs to
know about the qemu configuration. There is ‘if’ today, you pointed
out arch-specific for options. But the configuration itself is yet
another “language” with its own semantics… which JSON knows
nothing about.

> 
>>> In other words, the language I have in mind would be something
>>> that I could auto-generate from, say, the current qapi-schema.json
>>> and the qemu-options.def. Maybe in the end, that would be simply
>>> by having qemu-options.def being used to build qemu-options.json,
>>> and then add the relevant entries in qemu-options.json. Maybe
>>> there is a better way.
>> 
>> I would rather say that .def should go away and its content should
>> be converted to a JSON schema that becomes the new source format rather
>> than an intermediate generated file. There is nothing in .def files that
>> couldn't be represented in the schema.
> 
> Yes.

Agreed too. The meta-schema would need to be extended.

>> 
>> To a certain degree, QAPI does just that, by supporting different data
>> types for options. If necessary, I'm sure the type system could be
>> extended, but it's not clear to me to which degree we actually need
>> this.
> 
> QAPI is much more expressive than qemu-options.hx.  All the latter can
> do is "has option argument" and "option is arch-specific".  QAPI lets
> you specify an option argument's structure, and supports compile-time
> conditionals.  It can't do things like "must specify either argument A
> or B", or "numeric argument C must not exceed argument D", or "multiple
> options combine vs. last one wins".  Mostly because "it’s already
> complicated enough as is".

Understood.


>> 
>> We could. But is it actually worth inventing a new programming language?
>> I think this is something that should be done in C code even in the
>> future. I think what we're looking for is a way to describe interfaces,
>> not implementations.
> 
> Sane sugar can be described declaratively as macro expansion.
> 
> Less than sane sugar takes code.
> 
> Desugaring in code promotes (accidental) abandonment of sanity.
> 
> That said, we have to keep things simple to succeed.  Declarative
> desugaring seems beyond our reach.

Well, “keeping things simple” is a complicated thing.

> 
>> Even just for adding more sophisticated constraints, like in the example
>> above that maxmem >= size, it's questionable whether doing this in the
>> schema provides enough value for actually implementing it there.
> 
> Yes.

If we do it, yes. If we delegate to an existing language (I showed C above),
maybe it adds clarity without too much of an additional cost.

> 
>>>> I think the only thing in this list that can't obviously be covered
>>>> easily by QAPI is QOM. Or rather, it's covered by passing through
>>>> key=value lists without describing their structure
>>> 
>>> That’s close enough to me. (In my mind, that part was already “done”
>>> by QAPI, even if in the convoluted way you describe)
>> 
>> That's not really close enough. If you are happy with this level of
>> abstraction, you can define any command line option to be either a flag
>> or take a string as its argument and be done. This is obviously not very
>> helpful because it says nothing about the structure of that string.
> 
> Similar to all QMP commands taking a single 'any' argument.

Hmmm, I guess I have not understood what you meant by
"QOM could not be covered easily by QAPI" then.

We don’t want the QAPI to let arbitrary fields of a QOM object
be modified, do we?

As for the “public” aspects of a QOM object, that is static if it
comes from QAPI, so why couldn’t we define it there?


> 
>>                                                                    In
>> the same way, QAPI can't say anything about the structure of a QOM
>> object, and I think that's a problem.
>> 
>>>> - which, as far as I
>>>> understand, is mainly because QOM properties aren't necessarily static,
>>>> so we can't provide a statically defined interface for them. Probably
>>>> solvable in QEMU, but not without a major effort.
>>> 
>>> Or maybe extend the language so that it’s internal semantics
>>> knows about this aspect of QOM?
>> 
>> My point is that for example you can't generate a C struct (which is
>> statically defined) from something that has a dynamic structure. The
>> best you could do is fall back to key=value even in the C source, both
>> key and value being strings. But then you still have to parse these
>> strings manually, so it doesn't actually help much compared to a state
>> without C bindings.

I really don’t understand that point. To me, all these operations
seem relatively simple to generate.

I think that what confuses me is when you write “something that has a
dynamic structure”. I understand that as referring to the structure
defined in the schema. But the schema itself is static. So you can
generate static code from it, and it’s already done today.

Another hypothesis on my side is that we don’t want, ever, to
have the API provide for example the option to create its own
arbitrary QOM classes, or to tag arbitrary properties to an object,
where by “arbitrary” I mean not explicitly mentioned somewhere in
something like the schema.

So I suspect you are talking about something else.


> 
> QOM and QAPI sabotage each other.  Ironic, considering they were dreamed
> up by the same guy :)
> 
> QAPI is compile-time static by design.
> 
> QOM is run-time dynamic by design.  Some support for static definitions
> has been grafted on since.
> 
> We use QAPI type system escapes for QOM.  Defeats QAPI introspection and
> doc generation.  We provide separate QOM introspection instead, which is
> clumsier and less expressive.  QOM documentation doesn't exist.

But back to the original discussion about management tools,
do we let upper layers tag their own arbitrary stuff in QOM objects?


> 
>> Maybe what could be done is covering at least the static properties and
>> then having key=value for the dynamic part (which should be the
>> exception).
> 
> To make this worthwhile, we'd have to replace dynamic QOM properties by
> static ones when possible.  Monumental task.

I’m sure you are right, but it’s hard for me to evaluate, given how
many ways there are to access an object. Naively, grepping for
set_prop and for new_with_prop does not give me that many hits.


> 
>>>>> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
>>>>> “one of”, “one or several”, “attaches to”…
>>>>> - States, e.g. how do we represent the machine configuration,
>>>>> or the desired new disk setting
>>>>> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
>>>>> “start”, “notify”, etc. and how we describe the kind of input they need.
>>>>> - Possibly more subtle things like support for transactions, commit/rollback,
>>>>> i.e. “I want to add connect a virtual nic to some host vf, but if anything
>>>>> along the way fails, I’d like all the cleanup to happen automatically)
>>>> 
>>>> This sounds like a different approach from our current QAPI command set
>>> 
>>> Well, except for purposefully trying to use a different wording to avoid
>>> the risk of getting your mind stuck in one of the particular existing
>>> meta-languages in QEMU, the approach is not very different.
>> 
>> I didn't necessarily mean relations/state/verbs, which obviously exist,
>> but the examples that seemed to express things in a deliberately
>> different way from what we have today.
>> 
>>> - Transactions do not exist today that I know of, although we see
>>> signs of them in discussions about the fact that this options destroys
>>> that back end but that option does not.
>> 
>> We have a 'transaction' QMP command, but they are not an integral part
>> of the language. Considering the trouble to implement transactional
>> commands, I suppose we don't even want it to be a fundamental part of
>> the language.
>> 
>>>> Does it actually provide more functionality, though?
>>> 
>>> It’s not intended to provide more, but to require less to do the same thing.
>> 
>> Though we always need to keep in mind that if requiring less for future
>> additions requires a huge effort now, the investment may pay off only in
>> a very distant future (if the abstraction we build even survives until
>> then).
> 
> Worse is better.
> 
> http://dreamsongs.com/RiseOfWorseIsBetter.html

You know that I positively hate this ;-)

Well, I guess we can expand the schema. #ILoveJSON.


Thanks
Christophe

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Christophe de Dinechin <dinechin@redhat.com> writes:

> I started cutting some stuff out.
>
>> On 14 Jan 2020, at 14:04, Markus Armbruster <armbru@redhat.com> wrote:
>> 
>> Prior art:
>> 
>>    Presentation
>>    KVM Forum 2017: Towards a More Expressive and Introspectable QEMU
>>    Command Line
>>    https://www.youtube.com/watch?v=gtpOLQgnwug
>>    https://www.linux-kvm.org/images/f/f2/Armbru-qapi-cmdline_1.pdf
>> 
>>    RFC patches
>>    https://lists.nongnu.org/archive/html/qemu-devel/2017-10/msg00209.html
>>    Message-Id: <20171002152552.27999-1-armbru@redhat.com>
>>    https://repo.or.cz/qemu/armbru.git/shortlog/refs/heads/qapi-cmdline
>
> Very useful, thanks.
>
>> 
>>> Compatibility is much harder, which is not the least because the
>>> existing command line options are bad (primarily: using inconsistent
>>> syntax).
>> 
>> Semantics are just as problematic.
>
> Yes, but often you can leverage something (e.g. JSON, XML, etc)
>
> We could decide that the new QAPI language is a custom implementation
> of Python ;-) Or more seriously, we could have some part of the JSON
> schema that are “escapes” to C code that we put in some generated file.
> I used {{ }} below for this escape, in which case the semantics would be
> that of C plus some specific macros we insert in it.
>
> { 'option': '--add-fd',
>  'data': { 'fd': 'int', 'set': 'int', '*opaque': 'str' },
>  'help': [
> "-add-fd fd=fd,set=set[,opaque=opaque]",
> "                Add 'fd' to fd 'set'”] 
>  ‘validate’ : {{
>     error_if(QOPT.fd < 0, “fd option cannot be negative”);
>     error_if(QOPT.fd <= STDERR_FILENO, “fd cannot be a standard I/O stream”);
>  }},
>  ‘exec’ : {{
>     if (do_add_fd(QOPT.fd, QOPT.fset, QOPT.opaque) < 0) {
>         exit(1);
>     }
>  }},
>
> I’m not necessarily saying this is a good idea, mind you.

Precedence: the value of 'if' is also a C snippet.

>> You mean the qemu-options.hx.  qemu-options.def is generated from it.
>
> Oh, I had missed that, thanks.
>
>>>> or build configuration (there is a hint of the connection between
>>>> the two in the option ‘if’ member in the .json files).
>> 
>> I'm not sure what exactly you have in mind for build configuration.
>
> I meant what you pointed out wrt. options, that this “language” needs to
> know about the qemu configuration. There is ‘if’ today, you pointed
> out arch-specific for options. But the configuration itself is yet
> another “language” with its own semantics… which JSON knows
> nothing about.

QAPI 'if' is a pragmatic design balancing simplicty with expressiveness.

It's part of the QAPI language except for the actual conditions.  The
QAPI generator blindly wraps whatever it generates for the thing
carrying an 'if': 'COND' in #if COND ... #endif /* COND */.  In other
words, we express "what is conditional" in the QAPI language, but resort
to the C language for expressing the actual conditionals.

PRO:
* Supports arbitrary #if.
* QAPI language impact is light: just another (string-valued) property
  of definitions and members.
* Generated code is the same regardless of QEMU configuration.

CON:
* Generating anything but C is problematic.  Currently that's just
  documentation.  Having C conditionals there isn't exactly great, but
  tolerable.

>>>> In other words, the language I have in mind would be something
>>>> that I could auto-generate from, say, the current qapi-schema.json
>>>> and the qemu-options.def. Maybe in the end, that would be simply
>>>> by having qemu-options.def being used to build qemu-options.json,
>>>> and then add the relevant entries in qemu-options.json. Maybe
>>>> there is a better way.
>>> 
>>> I would rather say that .def should go away and its content should
>>> be converted to a JSON schema that becomes the new source format rather
>>> than an intermediate generated file. There is nothing in .def files that
>>> couldn't be represented in the schema.
>> 
>> Yes.
>
> Agreed too. The meta-schema would need to be extended.
>
>>> 
>>> To a certain degree, QAPI does just that, by supporting different data
>>> types for options. If necessary, I'm sure the type system could be
>>> extended, but it's not clear to me to which degree we actually need
>>> this.
>> 
>> QAPI is much more expressive than qemu-options.hx.  All the latter can
>> do is "has option argument" and "option is arch-specific".  QAPI lets
>> you specify an option argument's structure, and supports compile-time
>> conditionals.  It can't do things like "must specify either argument A
>> or B", or "numeric argument C must not exceed argument D", or "multiple
>> options combine vs. last one wins".  Mostly because "it’s already
>> complicated enough as is".
>
> Understood.
>
>
>>> 
>>> We could. But is it actually worth inventing a new programming language?
>>> I think this is something that should be done in C code even in the
>>> future. I think what we're looking for is a way to describe interfaces,
>>> not implementations.
>> 
>> Sane sugar can be described declaratively as macro expansion.
>> 
>> Less than sane sugar takes code.
>> 
>> Desugaring in code promotes (accidental) abandonment of sanity.
>> 
>> That said, we have to keep things simple to succeed.  Declarative
>> desugaring seems beyond our reach.
>
> Well, “keeping things simple” is a complicated thing.

True!

>>> Even just for adding more sophisticated constraints, like in the example
>>> above that maxmem >= size, it's questionable whether doing this in the
>>> schema provides enough value for actually implementing it there.
>> 
>> Yes.
>
> If we do it, yes. If we delegate to an existing language (I showed C above),
> maybe it adds clarity without too much of an additional cost.

Possible.

>>>>> I think the only thing in this list that can't obviously be covered
>>>>> easily by QAPI is QOM. Or rather, it's covered by passing through
>>>>> key=value lists without describing their structure
>>>> 
>>>> That’s close enough to me. (In my mind, that part was already “done”
>>>> by QAPI, even if in the convoluted way you describe)
>>> 
>>> That's not really close enough. If you are happy with this level of
>>> abstraction, you can define any command line option to be either a flag
>>> or take a string as its argument and be done. This is obviously not very
>>> helpful because it says nothing about the structure of that string.
>> 
>> Similar to all QMP commands taking a single 'any' argument.
>
> Hmmm, I guess I have not understood what you meant by
> "QOM could not be covered easily by QAPI" then.

I think Kevin's point is that QAPI can't fully describe QOM.

Let's talk specifics.  Here's how QAPI defines blockdev-add:

    { 'command': 'blockdev-add', 'data': 'BlockdevOptions', 'boxed': true }
    { 'union': 'BlockdevOptions',
      'base': { 'driver': 'BlockdevDriver',
                [...]
                '*detect-zeroes': 'BlockdevDetectZeroesOptions' },
      'discriminator': 'driver',
      'data': {
          'blkdebug':   'BlockdevOptionsBlkdebug',
          'blklogwrites':'BlockdevOptionsBlklogwrites',
          [...]
          'vxhs':       'BlockdevOptionsVxHS'
      } }
    { 'enum': 'BlockdevDriver',
      'data': [ 'blkdebug', 'blklogwrites', [...], 'vxhs' ] }

QAPI specifies the available blockdev object types and their properties,
with types.

Compare object-add:

    { 'command': 'object-add',
      'data': {'qom-type': 'str', 'id': 'str', '*props': 'any'} }

QAPI specifies neither the available QOM object types, nor their
properties.  Consequently, QAPI/QMP introspection tells you nothing
about them.  You need to use QOM introspection instead.  qom-list-types
shows what QOM object types are available, and qom-list-properties shows
an object type's properties.  Except for dynamic ones created only after
object creation).  With undocumented type name strings instead of types.

> We don’t want the QAPI to let arbitrary fields of a QOM object
> be modified, do we?

We already do: QMP command qom-set.  If it breaks your guest, you get to
keep the pieces.

> As for the “public” aspects of a QOM object, that is static if it
> comes from QAPI, so why couldn’t we define it there?

QAPI specifies a certain kind of data types.

QOM "specifies" data types as an imperative program for creating them.
Maximally flexible, and fundamentally at odds with static analysis.
I've hated this since day one.

There's no hard reason why QOM could not specify static aspects
declaratively.  It just doesn't, and changing it now would be a
monumental task.

The imperative program mostly creates identical data types every time.
In other words, the data types are static.  Two problems.  One,
converting such a program to an equivalent declaration takes manual
labor.  Two, there are exceptions, and identifying them is more manual
labor.

>>> the same way, QAPI can't say anything about the structure of a QOM
>>> object, and I think that's a problem.
>>> 
>>>>> - which, as far as I
>>>>> understand, is mainly because QOM properties aren't necessarily static,
>>>>> so we can't provide a statically defined interface for them. Probably
>>>>> solvable in QEMU, but not without a major effort.
>>>> 
>>>> Or maybe extend the language so that it’s internal semantics
>>>> knows about this aspect of QOM?
>>> 
>>> My point is that for example you can't generate a C struct (which is
>>> statically defined) from something that has a dynamic structure. The
>>> best you could do is fall back to key=value even in the C source, both
>>> key and value being strings. But then you still have to parse these
>>> strings manually, so it doesn't actually help much compared to a state
>>> without C bindings.
>
> I really don’t understand that point. To me, all these operations
> seem relatively simple to generate.

Yes, if the QAPI fairy gives us a declarative specification equivalent
to the existing imperative one, then replacing the hand-written
imperative code by code generated from the declarative specification
would be relatively simple.

> I think that what confuses me is when you write “something that has a
> dynamic structure”. I understand that as referring to the structure
> defined in the schema. But the schema itself is static. So you can
> generate static code from it, and it’s already done today.

QAPI data types are static.

QOM types are not.  They're effectively almost static in practice.

> Another hypothesis on my side is that we don’t want, ever, to
> have the API provide for example the option to create its own
> arbitrary QOM classes, or to tag arbitrary properties to an object,
> where by “arbitrary” I mean not explicitly mentioned somewhere in
> something like the schema.
>
> So I suspect you are talking about something else.

Kevin's talking about the imperative code creating different QOM
properties depending on how and in what context it is run.

QOM is perfectly capable of supporting a QMP command to add arbitrary
QOM properties to an object at run time, or even add arbitrary QOM
types, but as you wrote we don't want that.

>> QOM and QAPI sabotage each other.  Ironic, considering they were dreamed
>> up by the same guy :)
>> 
>> QAPI is compile-time static by design.
>> 
>> QOM is run-time dynamic by design.  Some support for static definitions
>> has been grafted on since.
>> 
>> We use QAPI type system escapes for QOM.  Defeats QAPI introspection and
>> doc generation.  We provide separate QOM introspection instead, which is
>> clumsier and less expressive.  QOM documentation doesn't exist.
>
> But back to the original discussion about management tools,
> do we let upper layers tag their own arbitrary stuff in QOM objects?

We don't want management applications to add QOM properties for their
own purposes.

>>> Maybe what could be done is covering at least the static properties and
>>> then having key=value for the dynamic part (which should be the
>>> exception).
>> 
>> To make this worthwhile, we'd have to replace dynamic QOM properties by
>> static ones when possible.  Monumental task.
>
> I’m sure you are right, but it’s hard for me to evaluate, given how
> many ways there are to access an object. Naively, grepping for
> set_prop and for new_with_prop does not give me that many hits.

Look for object_property_add*().  Some 450 hits.

See also object_class_property_add(), grafted on in

commit 16bf7f522a2ff68993f80631ed86254c71eaf5d4
Author: Daniel P. Berrange <berrange@redhat.com>
Date:   Tue Oct 13 13:37:46 2015 +0100

    qom: Allow properties to be registered against classes
    
    When there are many instances of a given class, registering
    properties against the instance is wasteful of resources. The
    majority of objects have a statically defined list of possible
    properties, so most of the properties are easily registerable
    against the class. Only those properties which are conditionally
    registered at runtime need be recorded against the klass.
    
    Registering properties against classes also makes it possible
    to provide static introspection of QOM - currently introspection
    is only possible after creating an instance of a class, which
    severely limits its usefulness.
    
    This impl only supports simple scalar properties. It does not
    attempt to allow child object / link object properties against
    the class. There are ways to support those too, but it would
    make this patch more complicated, so it is left as an exercise
    for the future.
    
    There is no equivalent to object_property_del() provided, since
    classes must be immutable once they are defined.
    
    Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
    Signed-off-by: Andreas Färber <afaerber@suse.de>

>>>>>> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
>>>>>> “one of”, “one or several”, “attaches to”…
>>>>>> - States, e.g. how do we represent the machine configuration,
>>>>>> or the desired new disk setting
>>>>>> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
>>>>>> “start”, “notify”, etc. and how we describe the kind of input they need.
>>>>>> - Possibly more subtle things like support for transactions, commit/rollback,
>>>>>> i.e. “I want to add connect a virtual nic to some host vf, but if anything
>>>>>> along the way fails, I’d like all the cleanup to happen automatically)
>>>>> 
>>>>> This sounds like a different approach from our current QAPI command set
>>>> 
>>>> Well, except for purposefully trying to use a different wording to avoid
>>>> the risk of getting your mind stuck in one of the particular existing
>>>> meta-languages in QEMU, the approach is not very different.
>>> 
>>> I didn't necessarily mean relations/state/verbs, which obviously exist,
>>> but the examples that seemed to express things in a deliberately
>>> different way from what we have today.
>>> 
>>>> - Transactions do not exist today that I know of, although we see
>>>> signs of them in discussions about the fact that this options destroys
>>>> that back end but that option does not.
>>> 
>>> We have a 'transaction' QMP command, but they are not an integral part
>>> of the language. Considering the trouble to implement transactional
>>> commands, I suppose we don't even want it to be a fundamental part of
>>> the language.
>>> 
>>>>> Does it actually provide more functionality, though?
>>>> 
>>>> It’s not intended to provide more, but to require less to do the same thing.
>>> 
>>> Though we always need to keep in mind that if requiring less for future
>>> additions requires a huge effort now, the investment may pay off only in
>>> a very distant future (if the abstraction we build even survives until
>>> then).
>> 
>> Worse is better.
>> 
>> http://dreamsongs.com/RiseOfWorseIsBetter.html
>
> You know that I positively hate this ;-)

It's been a tough lesson for me, too.

> Well, I guess we can expand the schema. #ILoveJSON.

Basing the QAPI language on JSON was a poor choice.  Not sure that's
fixable at a reasonable cost.

Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin]

Thanks a bunch. This clarifies a number of my misconceptions about
how this is currently used. Most notably this one:

> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
> 
>> We don’t want the QAPI to let arbitrary fields of a QOM object
>> be modified, do we?
> 
> We already do: QMP command qom-set.  If it breaks your guest, you get to
> keep the pieces.

Ouch. I certainly did not expect that.

"It is not what you don’t know that kills you, it is what you know that ain’t so".

> 
>>> http://dreamsongs.com/RiseOfWorseIsBetter.html
>> 
>> You know that I positively hate this ;-)
> 
> It's been a tough lesson for me, too.

Not sure I can call it a “lesson”. Much like a philosophy to fight against, IMO.


>> Well, I guess we can expand the schema. #ILoveJSON.
> 
> Basing the QAPI language on JSON was a poor choice.  Not sure that's
> fixable at a reasonable cost.

Well, at least now I have a slightly better understanding of the related costs
and trade-offs. Thanks a lot for explaining.


Christophe

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Christophe de Dinechin <dinechin@redhat.com> writes:

> Thanks a bunch. This clarifies a number of my misconceptions about
> how this is currently used. Most notably this one:
>
>> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
>> 
>>> We don’t want the QAPI to let arbitrary fields of a QOM object
>>> be modified, do we?
>> 
>> We already do: QMP command qom-set.  If it breaks your guest, you get to
>> keep the pieces.
>
> Ouch. I certainly did not expect that.
>
> "It is not what you don’t know that kills you, it is what you know that ain’t so".

Do we have a legitimate use for qom-set right now?  Hmm, let's check
libvirt... aha:

* qemuMonitorJSONSetMemoryStatsPeriod() uses it to control
  virtio-balloon's guest-stats-polling-interval property, in accordance
  with docs/virtio-balloon-stats.txt.

* qemuMonitorJSONSetIOThread() uses it to control iothread's properties
  poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
  documented (in qemu-options.hx), their use with qom-set is not.

Oh well.

>>>> http://dreamsongs.com/RiseOfWorseIsBetter.html
>>> 
>>> You know that I positively hate this ;-)
>> 
>> It's been a tough lesson for me, too.
>
> Not sure I can call it a “lesson”. Much like a philosophy to fight against, IMO.

The tough lesson isn't philosophy, it's an observation: "I believe that
worse-is-better [...] has better survival characteristics than
the-right-thing".

>>> Well, I guess we can expand the schema. #ILoveJSON.
>> 
>> Basing the QAPI language on JSON was a poor choice.  Not sure that's
>> fixable at a reasonable cost.
>
> Well, at least now I have a slightly better understanding of the related costs
> and trade-offs. Thanks a lot for explaining.

You're welcome!

Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> Christophe de Dinechin <dinechin@redhat.com> writes:
> 
> > Thanks a bunch. This clarifies a number of my misconceptions about
> > how this is currently used. Most notably this one:
> >
> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
> >> 
> >>> We don’t want the QAPI to let arbitrary fields of a QOM object
> >>> be modified, do we?
> >> 
> >> We already do: QMP command qom-set.  If it breaks your guest, you get to
> >> keep the pieces.
> >
> > Ouch. I certainly did not expect that.
> >
> > "It is not what you don’t know that kills you, it is what you know that ain’t so".
> 
> Do we have a legitimate use for qom-set right now?  Hmm, let's check
> libvirt... aha:
> 
> * qemuMonitorJSONSetMemoryStatsPeriod() uses it to control
>   virtio-balloon's guest-stats-polling-interval property, in accordance
>   with docs/virtio-balloon-stats.txt.
> 
> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>   documented (in qemu-options.hx), their use with qom-set is not.
> 
> Oh well.

Libvirt is of course happy to switch to something else instead of
qom-set for these features if QEMU wants to provide a safer
alternative.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
>> Christophe de Dinechin <dinechin@redhat.com> writes:
>> 
>> > Thanks a bunch. This clarifies a number of my misconceptions about
>> > how this is currently used. Most notably this one:
>> >
>> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
>> >> 
>> >>> We don’t want the QAPI to let arbitrary fields of a QOM object
>> >>> be modified, do we?
>> >> 
>> >> We already do: QMP command qom-set.  If it breaks your guest, you get to
>> >> keep the pieces.
>> >
>> > Ouch. I certainly did not expect that.
>> >
>> > "It is not what you don’t know that kills you, it is what you know that ain’t so".
>> 
>> Do we have a legitimate use for qom-set right now?  Hmm, let's check
>> libvirt... aha:
>> 
>> * qemuMonitorJSONSetMemoryStatsPeriod() uses it to control
>>   virtio-balloon's guest-stats-polling-interval property, in accordance
>>   with docs/virtio-balloon-stats.txt.
>> 
>> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>>   documented (in qemu-options.hx), their use with qom-set is not.
>> 
>> Oh well.
>
> Libvirt is of course happy to switch to something else instead of
> qom-set for these features if QEMU wants to provide a safer
> alternative.

Noted.

libvirt's use of qom-set is okay.  What's not okay is the near-complete
lack of QOM documentation, its poor QMP interface, in part due to
non-integration with QAPI, and last but not least the lack of QOM
leadership leaving it adrift.

Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Kashyap Chamarthy]

On Wed, Jan 15, 2020 at 03:02:48PM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:

[Changed the subject-line to indicate deviation from the original
topic.]

[...]

> > Libvirt is of course happy to switch to something else instead of
> > qom-set for these features if QEMU wants to provide a safer
> > alternative.
> 
> Noted.
> 
> libvirt's use of qom-set is okay.  What's not okay is the near-complete
> lack of QOM documentation, its poor QMP interface, in part due to
> non-integration with QAPI, and last but not least the lack of QOM
> leadership leaving it adrift.

What can be done to improve QOM documentation (or lack thereof)?

From a couple of hurried `grep` queries in the QEMU tree, there seems to
be no explicit qom.rst|txt, or qemu-object-model.txt|rst or some such.
(I hope I haven't missed any other files.)

Let's dig further.  Ah, I come across this helpful 2016 blog post[1]
("An incomplete list of QEMU APIs") by Eduardo from my bookmarks.  Here
I get some clues:

(a) In the section titled "QOM", Eduardo writes:

        "QOM is short for QEMU Object Model and was introduced in 2011.
        It is heavily documented on its header file
        [include/qom/object.h]" 

    Opening qom/object.h[2], indeed there is copious amounts of docs,
    expressed as commented-out text.  Two questions:

    - How much of this is still accurate?  (Sorry, if that's a loaded
      question.)

    - If at least 60% is still accurate, is it valuable to extract and
      publish it as rendered rST, as part of the on-going QEMU Docs
      improvement?

(b) The other clue is also from the same post, where Eduardo provides
    pointers to past KVM Forum presentations by MarkusA, PaoloB,
    AndreasF on QOM, Qdev et al.

    Is it worth slapping all these references (with a clear intro and
    outro) into a qom.rst file in QEMU tree, even if only for
    reference/context?  Or are these references, in-tree docs in
    object.h out-of-date beyond repair?  

If it is useful, I'm happy to get the initial doc going, secure in the
knowledge that more clueful people than me will chip in during the
review :-)

[1] https://habkost.net/posts/2016/11/incomplete-list-of-qemu-apis.html
[2] https://git.qemu.org/?p=qemu.git;a=blob;f=include/qom/object.h
[3] http://www.linux-kvm.org/images/9/90/Kvmforum14-qom.pdf

-- 
/kashyap

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Kashyap Chamarthy <kchamart@redhat.com> writes:

> On Wed, Jan 15, 2020 at 03:02:48PM +0100, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>
> [Changed the subject-line to indicate deviation from the original
> topic.]
>
> [...]
>
>> > Libvirt is of course happy to switch to something else instead of
>> > qom-set for these features if QEMU wants to provide a safer
>> > alternative.
>> 
>> Noted.
>> 
>> libvirt's use of qom-set is okay.  What's not okay is the near-complete
>> lack of QOM documentation, its poor QMP interface, in part due to
>> non-integration with QAPI, and last but not least the lack of QOM
>> leadership leaving it adrift.
>
> What can be done to improve QOM documentation (or lack thereof)?

Are you trying to push us from idle grousing to actually improve things?
No fair!

> From a couple of hurried `grep` queries in the QEMU tree, there seems to
> be no explicit qom.rst|txt, or qemu-object-model.txt|rst or some such.
> (I hope I haven't missed any other files.)

As far as I know, all we have is the lovingly[*] written comments in
include/qom/object.h.  Sadly, we've let them rot in places.  In
particular, many newer functions are undocumented.

This is *reference* documentation.  What we lack (sorely!) is an
overview / friendly introduction, and a design document with rationale.
Reconstructing rationale now would involve guesswork.

> Let's dig further.  Ah, I come across this helpful 2016 blog post[1]
> ("An incomplete list of QEMU APIs") by Eduardo from my bookmarks.  Here
> I get some clues:
>
> (a) In the section titled "QOM", Eduardo writes:
>
>         "QOM is short for QEMU Object Model and was introduced in 2011.
>         It is heavily documented on its header file
>         [include/qom/object.h]" 
>
>     Opening qom/object.h[2], indeed there is copious amounts of docs,
>     expressed as commented-out text.  Two questions:
>
>     - How much of this is still accurate?  (Sorry, if that's a loaded
>       question.)

May I present you Armbru's Comment Trust Levels:

ACTL2: The comment may be overly terse or incomplete, but the
probability for it to be outright wrong is low.

ACTL1: Treat as helpful guidance (with gratitude), but trust only the
code.

ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
Signifying nothing.

Most comments in decently maintained code are at ACTL1.

Around the time initial QOM development solidified, object.h's comments
were ACTL2.  The neglect that is now clearly visible there makes me
downgrade to ACTL1.

Paolo will have a more informed and possibly different opinion.

>     - If at least 60% is still accurate, is it valuable to extract and
>       publish it as rendered rST, as part of the on-going QEMU Docs
>       improvement?

Beware, personal opinion.

When you put documentation next to the code it documents (which you
absolutely should, because it's your only realistic chance to keep the
two in sync), then extracting API comments is useful, because it
collects them in one place.

It's of next to no use to me when the comments are all in the same place
already, namely the header.

> (b) The other clue is also from the same post, where Eduardo provides
>     pointers to past KVM Forum presentations by MarkusA, PaoloB,
>     AndreasF on QOM, Qdev et al.
>
>     Is it worth slapping all these references (with a clear intro and
>     outro) into a qom.rst file in QEMU tree, even if only for
>     reference/context?  Or are these references, in-tree docs in
>     object.h out-of-date beyond repair?  

Uff.

My qdev talks predate the rebase onto QOM.  They may well confuse /
mislead as much as inform now.

> If it is useful, I'm happy to get the initial doc going, secure in the
> knowledge that more clueful people than me will chip in during the
> review :-)

Ha, nerd sniping!

> [1] https://habkost.net/posts/2016/11/incomplete-list-of-qemu-apis.html
> [2] https://git.qemu.org/?p=qemu.git;a=blob;f=include/qom/object.h
> [3] http://www.linux-kvm.org/images/9/90/Kvmforum14-qom.pdf


[*] Absolutely no irony intended.  Honest, officer!

[**] Don't take it personally, we're all part-time idiots.  Besides,
this is literature.

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

On 31/01/20 07:11, Markus Armbruster wrote:
> May I present you Armbru's Comment Trust Levels:
> 
> ACTL2: The comment may be overly terse or incomplete, but the
> probability for it to be outright wrong is low.
> 
> ACTL1: Treat as helpful guidance (with gratitude), but trust only the
> code.
> 
> ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
> Signifying nothing.
> 
> Most comments in decently maintained code are at ACTL1.
> 
> Around the time initial QOM development solidified, object.h's comments
> were ACTL2.  The neglect that is now clearly visible there makes me
> downgrade to ACTL1.
> 
> Paolo will have a more informed and possibly different opinion.

I think around initial development it was ACTL3, now it's around 1.8.

Paolo

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Christophe de Dinechin]

> On 31 Jan 2020, at 08:46, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 31/01/20 07:11, Markus Armbruster wrote:
>> May I present you Armbru's Comment Trust Levels:
>> 
>> ACTL2: The comment may be overly terse or incomplete, but the
>> probability for it to be outright wrong is low.
>> 
>> ACTL1: Treat as helpful guidance (with gratitude), but trust only the
>> code.
>> 
>> ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
>> Signifying nothing.
>> 
>> Most comments in decently maintained code are at ACTL1.
>> 
>> Around the time initial QOM development solidified, object.h's comments
>> were ACTL2.  The neglect that is now clearly visible there makes me
>> downgrade to ACTL1.
>> 
>> Paolo will have a more informed and possibly different opinion.
> 
> I think around initial development it was ACTL3, now it's around 1.8.

Interestingly, the initial doc suggested ACTL to be an whole value between
0 and 2. Now it’s fractional and value 3 has a meaning you can guess
to be “above 2”…

How fast this happened tells you everything you need to know about
documentation/specifications and actual implementations :-)

> 
> Paolo
> 

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

On 31/01/20 16:37, Christophe de Dinechin wrote:
> 
> 
>> On 31 Jan 2020, at 08:46, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>> On 31/01/20 07:11, Markus Armbruster wrote:
>>> May I present you Armbru's Comment Trust Levels:
>>>
>>> ACTL2: The comment may be overly terse or incomplete, but the
>>> probability for it to be outright wrong is low.
>>>
>>> ACTL1: Treat as helpful guidance (with gratitude), but trust only the
>>> code.
>>>
>>> ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
>>> Signifying nothing.
>>>
>>> Most comments in decently maintained code are at ACTL1.
>>>
>>> Around the time initial QOM development solidified, object.h's comments
>>> were ACTL2.  The neglect that is now clearly visible there makes me
>>> downgrade to ACTL1.
>>>
>>> Paolo will have a more informed and possibly different opinion.
>>
>> I think around initial development it was ACTL3, now it's around 1.8.
> 
> Interestingly, the initial doc suggested ACTL to be an whole value between
> 0 and 2. Now it’s fractional and value 3 has a meaning you can guess
> to be “above 2”…

I won't say if this was accidental or not. :
Paolo

> How fast this happened tells you everything you need to know about
> documentation/specifications and actual implementations :-)

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Kashyap Chamarthy]

On Fri, Jan 31, 2020 at 07:11:15AM +0100, Markus Armbruster wrote:
> Kashyap Chamarthy <kchamart@redhat.com> writes:

[...]

> > What can be done to improve QOM documentation (or lack thereof)?
> 
> Are you trying to push us from idle grousing to actually improve things?
> No fair!

I first wrote it as a semi-complaint, and then I caught myself: "you're
not helping"; so I rephrased it to be more constructive. :-)

> > From a couple of hurried `grep` queries in the QEMU tree, there seems to
> > be no explicit qom.rst|txt, or qemu-object-model.txt|rst or some such.
> > (I hope I haven't missed any other files.)
> 
> As far as I know, all we have is the lovingly[*] written comments in
> include/qom/object.h.  Sadly, we've let them rot in places.  In
> particular, many newer functions are undocumented.
> 
> This is *reference* documentation.  What we lack (sorely!) is an
> overview / friendly introduction, and a design document with rationale.
> Reconstructing rationale now would involve guesswork.

Me nods; that (guesswork in retroactive rationale construction) makes
matters slightly more difficult.

[...]

> >     Opening qom/object.h[2], indeed there is copious amounts of docs,
> >     expressed as commented-out text.  Two questions:
> >
> >     - How much of this is still accurate?  (Sorry, if that's a loaded
> >       question.)
> 
> May I present you Armbru's Comment Trust Levels:
>
> ACTL2: The comment may be overly terse or incomplete, but the
> probability for it to be outright wrong is low.
> 
> ACTL1: Treat as helpful guidance (with gratitude), but trust only the
> code.
> 
> ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
> Signifying nothing.
> 
> Most comments in decently maintained code are at ACTL1.

Noted.  (And thanks for the useful reference scale :-))

> Around the time initial QOM development solidified, object.h's comments
> were ACTL2.  The neglect that is now clearly visible there makes me
> downgrade to ACTL1.
> 
> Paolo will have a more informed and possibly different opinion.
> 
> >     - If at least 60% is still accurate, is it valuable to extract and
> >       publish it as rendered rST, as part of the on-going QEMU Docs
> >       improvement?
> 
> Beware, personal opinion.
> 
> When you put documentation next to the code it documents (which you
> absolutely should, because it's your only realistic chance to keep the
> two in sync), then extracting API comments is useful, because it
> collects them in one place.
> 
> It's of next to no use to me when the comments are all in the same place
> already, namely the header.

Yeah, reasonable point.

> > (b) The other clue is also from the same post, where Eduardo provides
> >     pointers to past KVM Forum presentations by MarkusA, PaoloB,
> >     AndreasF on QOM, Qdev et al.
> >
> >     Is it worth slapping all these references (with a clear intro and
> >     outro) into a qom.rst file in QEMU tree, even if only for
> >     reference/context?  Or are these references, in-tree docs in
> >     object.h out-of-date beyond repair?  
> 
> Uff.
> 
> My qdev talks predate the rebase onto QOM.  They may well confuse /
> mislead as much as inform now.

Good to know.  (We don't want to add additional sources of confusion.)

> > If it is useful, I'm happy to get the initial doc going, secure in the
> > knowledge that more clueful people than me will chip in during the
> > review :-)
> 
> Ha, nerd sniping!

:-)

Thanks for the response; it was useful.

[...]

-- 
/kashyap

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Peter Maydell]

On Fri, 31 Jan 2020 at 06:11, Markus Armbruster <armbru@redhat.com> wrote:
> Beware, personal opinion.
>
> When you put documentation next to the code it documents (which you
> absolutely should, because it's your only realistic chance to keep the
> two in sync), then extracting API comments is useful, because it
> collects them in one place.
>
> It's of next to no use to me when the comments are all in the same place
> already, namely the header.

To throw in a personal opinion on the other side, API comments
should be in the header, not the .c file, because they're
your external interface and as an external consumer of that
interface I shouldn't have to go digging around in your
implementation source file to find the documentation.

Since Paolo put in the effort to upstream the kerneldoc
Sphinx plugin, it's now relatively simple to pull in
the doc comments into a rST file, so you might as well I
guess, though I agree that the cumulative benefit over
just reading the .h file is not enormous.

I definitely agree that the overview/introduction/conventions
side of things is where we'd benefit most if somebody wanted
to try to tackle that. We could roll
https://wiki.qemu.org/Documentation/QOMConventions
into that if we had a better place to put that info.

thanks
-- PMM

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

Il ven 31 gen 2020, 11:36 Peter Maydell <peter.maydell@linaro.org> ha
scritto:

> On Fri, 31 Jan 2020 at 06:11, Markus Armbruster <armbru@redhat.com> wrote:
> > Beware, personal opinion.
> >
> > When you put documentation next to the code it documents (which you
> > absolutely should, because it's your only realistic chance to keep the
> > two in sync), then extracting API comments is useful, because it
> > collects them in one place.
> >
> > It's of next to no use to me when the comments are all in the same place
> > already, namely the header.
>

The advantage of putting them in the header is that you have them all in
one place (inline functions and structs must be in the header). In practice
that balances for me the disadvantage of having some comments far from the
code they document, which increases the risk of bitrot especially for
comments such as "called with lock X held".

I definitely agree that the overview/introduction/conventions
> side of things is where we'd benefit most if somebody wanted
> to try to tackle that. We could roll
> https://wiki.qemu.org/Documentation/QOMConventions
> into that if we had a better place to put that info.
>

I am travelling this weekend so I might try to do some kind of thread
summary and brain dump in the wiki. I'll leave to Kashyap to do the rST
conversion and patch submission. ;-)

Paolo


> thanks
> -- PMM
>
>

[-- Attachment #2: Type: text/html, Size: 2480 bytes --]

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Kashyap Chamarthy]

On Fri, Jan 31, 2020 at 12:02:05PM +0100, Paolo Bonzini wrote:
> Il ven 31 gen 2020, 11:36 Peter Maydell <peter.maydell@linaro.org> ha
> scritto:

[...]

> The advantage of putting them in the header is that you have them all in
> one place (inline functions and structs must be in the header). In practice
> that balances for me the disadvantage of having some comments far from the
> code they document, which increases the risk of bitrot especially for
> comments such as "called with lock X held".
> 
> I definitely agree that the overview/introduction/conventions
> > side of things is where we'd benefit most if somebody wanted
> > to try to tackle that. We could roll
> > https://wiki.qemu.org/Documentation/QOMConventions
> > into that if we had a better place to put that info.
> >
> 
> I am travelling this weekend so I might try to do some kind of thread
> summary and brain dump in the wiki. I'll leave to Kashyap to do the rST
> conversion and patch submission. ;-)

Thanks!  Happy to be the 'scribe' ;-)  I have a skeltal
qemu-object-model.rst file sitting with some initial content based on
various sources, including one of your presentations[*] from 2014.
I'll wait for your new Wiki link to incorporate that content.

(Minor aside: I'm not sure if this file should be in docs/interop/ dir,
which IIRC, is for things that are 'external' interfaces.  And I learn
that QOM is used both internally in and as an external interface, e.g.
whenever a device is being created, machine types, CPU config, etc.)

            - - -

I've re-skimmed your scarily-titled "QOM exegesis and apocalypse" 2014
KVM Forum talk slides[*], where the "Why QOM?" slide says:

    All device creation, device configuration, backend creation and
    backed configuration done through a single interface
    
    Rigorous support for introspection both of runtime objects and type
    capabilities

Me wonders how much of the above "Why" still holds true today.  Although
further slides give more clues on what worked and what didn't.

I'll wait for fresher details from your upcoming Wiki :-)

[*] http://www.linux-kvm.org/images/9/90/Kvmforum14-qom.pdf

-- 
/kashyap

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Kashyap Chamarthy <kchamart@redhat.com> writes:

> On Fri, Jan 31, 2020 at 12:02:05PM +0100, Paolo Bonzini wrote:
>> Il ven 31 gen 2020, 11:36 Peter Maydell <peter.maydell@linaro.org> ha
>> scritto:
>
> [...]
>
>> The advantage of putting them in the header is that you have them all in
>> one place (inline functions and structs must be in the header). In practice
>> that balances for me the disadvantage of having some comments far from the
>> code they document, which increases the risk of bitrot especially for
>> comments such as "called with lock X held".
>> 
>> I definitely agree that the overview/introduction/conventions
>> > side of things is where we'd benefit most if somebody wanted
>> > to try to tackle that. We could roll
>> > https://wiki.qemu.org/Documentation/QOMConventions
>> > into that if we had a better place to put that info.
>> >
>> 
>> I am travelling this weekend so I might try to do some kind of thread
>> summary and brain dump in the wiki. I'll leave to Kashyap to do the rST
>> conversion and patch submission. ;-)
>
> Thanks!  Happy to be the 'scribe' ;-)  I have a skeltal
> qemu-object-model.rst file sitting with some initial content based on
> various sources, including one of your presentations[*] from 2014.
> I'll wait for your new Wiki link to incorporate that content.
>
> (Minor aside: I'm not sure if this file should be in docs/interop/ dir,
> which IIRC, is for things that are 'external' interfaces.  And I learn
> that QOM is used both internally in and as an external interface, e.g.
> whenever a device is being created, machine types, CPU config, etc.)

docs/devel/qapi-code-gen.txt has the same problem: it's mostly internal
stuff, but there's also introspection, which is an external interface.

>
>             - - -
>
> I've re-skimmed your scarily-titled "QOM exegesis and apocalypse" 2014
> KVM Forum talk slides[*], where the "Why QOM?" slide says:
>
>     All device creation, device configuration, backend creation and
>     backed configuration done through a single interface
>     
>     Rigorous support for introspection both of runtime objects and type
>     capabilities

For a value of "rigorous".

Let me propose QAPI's query-qmp-schema as the tin standard[*] of
introspection:

* It's documented

* It comes with something that can pass as a type system

* It actually tells you the full truth.

Now compare to QOM:

* Documentation

  QAPI: docs/devel/qapi-code-gen.txt section "Client JSON Protocol
  introspection"

  QOM: Nada

* Type system

  QAPI: A few built-in types specified in the documentation, type
  constructors for complex types.

  QOM: Types are strings, and you just need to know what they mean.
  Some string patterns are special: link<STR>, child<STR>, STR[INT], and
  you just need to know what that means, too.

* Full truth

  QAPI: If you can access it at the interface, you can also see it in
  introspection.

  QOM: Type introspection can show you only the properties of a freshly
  created object.  Properties that get created only later are invisible.
  Properties that depend on global state are unreliable.  Object
  introspection is reliable, but only for that object in its current
  state.

> Me wonders how much of the above "Why" still holds true today.  Although
> further slides give more clues on what worked and what didn't.
>
> I'll wait for fresher details from your upcoming Wiki :-)
>
> [*] http://www.linux-kvm.org/images/9/90/Kvmforum14-qom.pdf

[*] Gold was too expensive, but we had some tin on hand, so...

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

On 31/01/20 18:23, Markus Armbruster wrote:
> Kashyap Chamarthy <kchamart@redhat.com> writes:
> docs/devel/qapi-code-gen.txt has the same problem: it's mostly internal
> stuff, but there's also introspection, which is an external interface.

We should move introspection to docs/interop.  Any takers?

>>     Rigorous support for introspection both of runtime objects and type
>>     capabilities
> 
> For a value of "rigorous".
> 
> Let me propose QAPI's query-qmp-schema as the tin standard[*] of
> introspection:
> 
> * It's documented
> 
> * It comes with something that can pass as a type system
> 
> * It actually tells you the full truth.

Well, not all statements age equally well.  But compared to netdev_add 
and device_add, it was still an improvement. :)

It's certainly worse than QAPI *now*, but it's not nonexistent:

> * Documentation
> 
>   QAPI: docs/devel/qapi-code-gen.txt section "Client JSON Protocol
>   introspection"
> 
>   QOM: Nada

Well, there is qom.json.  So slightly more than nothing, though I have 
already found an off-by-one error and it could definitely be improved:

 @type: the type of the property.  This will typically come in one of four
        forms:

        1) A primitive type such as 'u8', 'u16', 'bool', 'str', or 'double'.
           These types are mapped to the appropriate JSON type.

        2) A child type in the form 'child<subtype>' where subtype is a qdev
           device type name.  Child properties create the composition tree.

        3) A link type in the form 'link<subtype>' where subtype is a qdev
           device type name.  Link properties form the device model graph.


> * Type system
> 
>   QAPI: A few built-in types specified in the documentation, type
>   constructors for complex types.
> 
>   QOM: Types are strings, and you just need to know what they mean.
>   Some string patterns are special: link<STR>, child<STR>, STR[INT], and
>   you just need to know what that means, too.

str[int] is not a type as far as I understood it, it's a property name.  
Types are documented as above; however types other than link<> and 
child<>, which are QAPI types, can be user-defined types (structs, 
enums) and this is not included in (1).

> * Full truth
> 
>   QAPI: If you can access it at the interface, you can also see it in
>   introspection.
> 
>   QOM: Type introspection can show you only the properties of a freshly
>   created object.  Properties that get created only later are invisible.
>   Properties that depend on global state are unreliable.  Object
>   introspection is reliable, but only for that object in its current
>   state.

Right, that's array properties and at least theoretically child 
properties (I don't know if there are examples).

Paolo

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 31/01/20 18:23, Markus Armbruster wrote:
>> Kashyap Chamarthy <kchamart@redhat.com> writes:
>> docs/devel/qapi-code-gen.txt has the same problem: it's mostly internal
>> stuff, but there's also introspection, which is an external interface.
>
> We should move introspection to docs/interop.  Any takers?
>
>>>     Rigorous support for introspection both of runtime objects and type
>>>     capabilities
>> 
>> For a value of "rigorous".
>> 
>> Let me propose QAPI's query-qmp-schema as the tin standard[*] of
>> introspection:
>> 
>> * It's documented
>> 
>> * It comes with something that can pass as a type system
>> 
>> * It actually tells you the full truth.
>
> Well, not all statements age equally well.  But compared to netdev_add 
> and device_add, it was still an improvement. :)

I think "rigorous" is to be read as ambition, not description.  Sadly,
QOM stagnated before getting close to realizing its ambitions.

> It's certainly worse than QAPI *now*, but it's not nonexistent:
>
>> * Documentation
>> 
>>   QAPI: docs/devel/qapi-code-gen.txt section "Client JSON Protocol
>>   introspection"
>> 
>>   QOM: Nada
>
> Well, there is qom.json.  So slightly more than nothing, though I have 
> already found an off-by-one error and it could definitely be improved:
>
>  @type: the type of the property.  This will typically come in one of four
>         forms:
>
>         1) A primitive type such as 'u8', 'u16', 'bool', 'str', or 'double'.
>            These types are mapped to the appropriate JSON type.
>
>         2) A child type in the form 'child<subtype>' where subtype is a qdev
>            device type name.  Child properties create the composition tree.
>
>         3) A link type in the form 'link<subtype>' where subtype is a qdev
>            device type name.  Link properties form the device model graph.

If the off-by-one you found is "four forms" followed by a list of three,
then the error isn't "four", it's the omitted fourth list item.

Anyway, I grant you "nada" was an exaggeration for effect.

Note that the doc comments are reference documentation.  QAPI got that,
too.  Reference documentation is useful, but no replacement for an
explainer like qapi-code-gen.txt.

>> * Type system
>> 
>>   QAPI: A few built-in types specified in the documentation, type
>>   constructors for complex types.
>> 
>>   QOM: Types are strings, and you just need to know what they mean.
>>   Some string patterns are special: link<STR>, child<STR>, STR[INT], and
>>   you just need to know what that means, too.
>
> str[int] is not a type as far as I understood it, it's a property name.  

You're right.

> Types are documented as above; however types other than link<> and 
> child<>, which are QAPI types, can be user-defined types (structs, 
> enums) and this is not included in (1).

Specifically, three of four kinds of type names are documented:
primitive, child, scalar.  The fourth kind is not, and it can be
anything.  It need not be a QAPI type name.  In any case, you just have
to know what the type name means.

In contrast, QAPI introspection defines all types in terms of type
constructors and primitive types.  It completely hides user-defined QAPI
type names, so they don't become ABI.

Do you think we could somehow appropriate QAPI's type system for QOM
introspection?

>> * Full truth
>> 
>>   QAPI: If you can access it at the interface, you can also see it in
>>   introspection.
>> 
>>   QOM: Type introspection can show you only the properties of a freshly
>>   created object.  Properties that get created only later are invisible.
>>   Properties that depend on global state are unreliable.  Object
>>   introspection is reliable, but only for that object in its current
>>   state.
>
> Right, that's array properties and at least theoretically child 
> properties (I don't know if there are examples).
>
> Paolo

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

On 03/02/20 10:54, Markus Armbruster wrote:
>> Types are documented as above; however types other than link<> and 
>> child<>, which are QAPI types, can be user-defined types (structs, 
>> enums) and this is not included in (1).
> Specifically, three of four kinds of type names are documented:
> primitive, child, scalar.  The fourth kind is not, and it can be
> anything.  It need not be a QAPI type name.  In any case, you just have
> to know what the type name means.

It is not enforced, but it is supposed to be only QAPI type names
(primitive or not), child or link.

Paolo

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 03/02/20 10:54, Markus Armbruster wrote:
>>> Types are documented as above; however types other than link<> and 
>>> child<>, which are QAPI types, can be user-defined types (structs, 
>>> enums) and this is not included in (1).
>> Specifically, three of four kinds of type names are documented:
>> primitive, child, scalar.  The fourth kind is not, and it can be
>> anything.  It need not be a QAPI type name.  In any case, you just have
>> to know what the type name means.
>
> It is not enforced, but it is supposed to be only QAPI type names
> (primitive or not), child or link.

It's not even documented.

Actual names include "struct tm", "guest statistics", and (my
favourite) "struct".

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Il ven 31 gen 2020, 11:36 Peter Maydell <peter.maydell@linaro.org> ha
> scritto:
>
>> On Fri, 31 Jan 2020 at 06:11, Markus Armbruster <armbru@redhat.com> wrote:
>> > Beware, personal opinion.
>> >
>> > When you put documentation next to the code it documents (which you
>> > absolutely should, because it's your only realistic chance to keep the
>> > two in sync), then extracting API comments is useful, because it
>> > collects them in one place.
>> >
>> > It's of next to no use to me when the comments are all in the same place
>> > already, namely the header.
>>
>
> The advantage of putting them in the header is that you have them all in
> one place (inline functions and structs must be in the header). In practice
> that balances for me the disadvantage of having some comments far from the
> code they document, which increases the risk of bitrot especially for
> comments such as "called with lock X held".

With suitable doc generation from source, we can have them next to the
code *and* all in one place, namely the generated interface docs.

>> I definitely agree that the overview/introduction/conventions
>> side of things is where we'd benefit most if somebody wanted
>> to try to tackle that. We could roll
>> https://wiki.qemu.org/Documentation/QOMConventions
>> into that if we had a better place to put that info.
>>
>
> I am travelling this weekend so I might try to do some kind of thread
> summary and brain dump in the wiki. I'll leave to Kashyap to do the rST
> conversion and patch submission. ;-)

That would be awesome!

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> Christophe de Dinechin <dinechin@redhat.com> writes:
> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>   documented (in qemu-options.hx), their use with qom-set is not.

I'm happy to use a different interface.

Writing a boilerplate "iothread-set-poll-params" QMP command in C would
be a step backwards.

Maybe the QAPI code generator could map something like this:

  { 'command': 'iothread-set-poll-params',
    'data': {
        'id': 'str',
	'*max-ns': 'uint64',
	'*grow': 'uint64',
	'*shrink': 'uint64'
    },
    'map-to-qom-set': 'IOThread'
  }

And turn it into QOM accessors on the IOThread object.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
>> Christophe de Dinechin <dinechin@redhat.com> writes:
>> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
>> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>>   documented (in qemu-options.hx), their use with qom-set is not.
>
> I'm happy to use a different interface.
>
> Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> be a step backwards.

No argument.

> Maybe the QAPI code generator could map something like this:
>
>   { 'command': 'iothread-set-poll-params',
>     'data': {
>         'id': 'str',
> 	'*max-ns': 'uint64',
> 	'*grow': 'uint64',
> 	'*shrink': 'uint64'
>     },
>     'map-to-qom-set': 'IOThread'
>   }
>
> And turn it into QOM accessors on the IOThread object.

I think a generic "set this configuration to that value" command is just
fine.  qom-set fails on several counts, though:

* Tolerable: qom-set is not actually generic, it applies only to QOM.

* qom-set lets you set tons of stuff that is not meant to be changed at
  run time.  If it breaks your guest, you get to keep the pieces.

* There is virtually no documentation on what can be set to what values,
  and their semantics.

In its current state, QOM is a user interface superfund site.

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> 
> > On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> >> Christophe de Dinechin <dinechin@redhat.com> writes:
> >> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
> >> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
> >>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
> >>   documented (in qemu-options.hx), their use with qom-set is not.
> >
> > I'm happy to use a different interface.
> >
> > Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> > be a step backwards.
> 
> No argument.
> 
> > Maybe the QAPI code generator could map something like this:
> >
> >   { 'command': 'iothread-set-poll-params',
> >     'data': {
> >         'id': 'str',
> > 	'*max-ns': 'uint64',
> > 	'*grow': 'uint64',
> > 	'*shrink': 'uint64'
> >     },
> >     'map-to-qom-set': 'IOThread'
> >   }
> >
> > And turn it into QOM accessors on the IOThread object.
> 
> I think a generic "set this configuration to that value" command is just
> fine.  qom-set fails on several counts, though:
> 
> * Tolerable: qom-set is not actually generic, it applies only to QOM.
> 
> * qom-set lets you set tons of stuff that is not meant to be changed at
>   run time.  If it breaks your guest, you get to keep the pieces.
> 
> * There is virtually no documentation on what can be set to what values,
>   and their semantics.
> 
> In its current state, QOM is a user interface superfund site.

Thoughts about a solution:

Static QOM properties should be declared via QAPI instead of
imperatively via QOM APIs.  That way they are introspectable and type
information is present in the schema.

The QAPI code generator could emit a function that is callable from
.class_init().  This eliminates the need to manually call
object_class_property_add().

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Making QEMU easier for management tools and applications

[Marc-André Lureau]

Hi

On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
> > Stefan Hajnoczi <stefanha@gmail.com> writes:
> >
> > > On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> > >> Christophe de Dinechin <dinechin@redhat.com> writes:
> > >> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
> > >> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
> > >>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
> > >>   documented (in qemu-options.hx), their use with qom-set is not.
> > >
> > > I'm happy to use a different interface.
> > >
> > > Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> > > be a step backwards.
> >
> > No argument.
> >
> > > Maybe the QAPI code generator could map something like this:
> > >
> > >   { 'command': 'iothread-set-poll-params',
> > >     'data': {
> > >         'id': 'str',
> > >     '*max-ns': 'uint64',
> > >     '*grow': 'uint64',
> > >     '*shrink': 'uint64'
> > >     },
> > >     'map-to-qom-set': 'IOThread'
> > >   }
> > >
> > > And turn it into QOM accessors on the IOThread object.
> >
> > I think a generic "set this configuration to that value" command is just
> > fine.  qom-set fails on several counts, though:
> >
> > * Tolerable: qom-set is not actually generic, it applies only to QOM.
> >
> > * qom-set lets you set tons of stuff that is not meant to be changed at
> >   run time.  If it breaks your guest, you get to keep the pieces.
> >
> > * There is virtually no documentation on what can be set to what values,
> >   and their semantics.
> >
> > In its current state, QOM is a user interface superfund site.
>
> Thoughts about a solution:
>
> Static QOM properties should be declared via QAPI instead of
> imperatively via QOM APIs.  That way they are introspectable and type
> information is present in the schema.
>
> The QAPI code generator could emit a function that is callable from
> .class_init().  This eliminates the need to manually call
> object_class_property_add().

I have this in mind too. First step is probably to move as much as
possible as class properties. Please review
https://patchew.org/QEMU/20200110153039.1379601-1-marcandre.lureau@redhat.com/,
I have more of this kind of qdev/qom cleanups pending.



-- 
Marc-André Lureau

Integrating QOM into QAPI (was: Making QEMU easier for management tools and applications)

[Markus Armbruster]

Marc-André Lureau <marcandre.lureau@gmail.com> writes:

> Hi
>
> On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>
>> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
>> > Stefan Hajnoczi <stefanha@gmail.com> writes:
>> >
>> > > On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
>> > >> Christophe de Dinechin <dinechin@redhat.com> writes:
>> > >> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
>> > >> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>> > >>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>> > >>   documented (in qemu-options.hx), their use with qom-set is not.
>> > >
>> > > I'm happy to use a different interface.
>> > >
>> > > Writing a boilerplate "iothread-set-poll-params" QMP command in C would
>> > > be a step backwards.
>> >
>> > No argument.
>> >
>> > > Maybe the QAPI code generator could map something like this:
>> > >
>> > >   { 'command': 'iothread-set-poll-params',
>> > >     'data': {
>> > >         'id': 'str',
>> > >     '*max-ns': 'uint64',
>> > >     '*grow': 'uint64',
>> > >     '*shrink': 'uint64'
>> > >     },
>> > >     'map-to-qom-set': 'IOThread'
>> > >   }
>> > >
>> > > And turn it into QOM accessors on the IOThread object.
>> >
>> > I think a generic "set this configuration to that value" command is just
>> > fine.  qom-set fails on several counts, though:
>> >
>> > * Tolerable: qom-set is not actually generic, it applies only to QOM.
>> >
>> > * qom-set lets you set tons of stuff that is not meant to be changed at
>> >   run time.  If it breaks your guest, you get to keep the pieces.
>> >
>> > * There is virtually no documentation on what can be set to what values,
>> >   and their semantics.
>> >
>> > In its current state, QOM is a user interface superfund site.
>>
>> Thoughts about a solution:
>>
>> Static QOM properties should be declared via QAPI instead of
>> imperatively via QOM APIs.  That way they are introspectable and type
>> information is present in the schema.
>>
>> The QAPI code generator could emit a function that is callable from
>> .class_init().  This eliminates the need to manually call
>> object_class_property_add().

We need to make up our minds what exactly we want generated.  Then we
can design the QAPI language, and code up the generator.

Skeleton QOM type, to help with the discussion:

    #define TYPE_FOO "foo"

    #define FOO(obj) OBJECT_CHECK(Foo, (obj), TYPE_FOO)
    #define FOO_CLASS(klass) \
        OBJECT_CLASS_CHECK(FooClass, (klass), TYPE_FOO)
    #define FOO_GET_CLASS(obj) \
        OBJECT_GET_CLASS(FooClass, (obj), TYPE_FOO)

    typedef FooClass {
        ParentClass parent_class;
        ... // hand-written per-class state
    }

    struct Chardev {
        ParentObject parent_obj;
        ... // hand-written instance (per-object) state
    };

    static const TypeInfo char_type_info = {
        .name = TYPE_FOO,
        .parent = TYPE_OBJECT,
        .instance_size = sizeof(Foo),
        .instance_init = ...,                   // methods to initialize
        .instance_post_init = ...,              // and finalize instances,
        .instance_finalize = ...,               // all optional
        .abstract = ...,                        // true or false (d'oh)
        .class_size = sizeof(FooClass),
        .class_init = ...,                      // methods to initialize
        .class_base_init = ...,                 // classes, optional
        .class_data = ...,                      // extra argument for them
        .interfaces = ...
    };

There's substantial boilerplate, with plenty of hand-written code in the
gaps.  What of the boilerplate do we plan to generate?  How do we plan
to fill the gaps, if any?

A "property" is always a property of an instance.  It can either be
registered with the instance (with object_property_add() & friends) or
the class (with object_class_property_add() & friends).  Yes, the naming
is confusing.

To have QAPI register properties with the class, it needs to generate
a variation of

    object_class_property_add(cls, prop_name, type_name,
                              get, set, release, opaque, &err)

for each property "somewhere", so it runs within .class_init().  For
real QAPI integration, the property's type must be a QAPI type.  QAPI
needs to generate the callbacks get(), set() and release() then.

Existing properties could conceivably do funny things in their
handwritten callbacks, defying conversion to this scheme.  There's just
one way to find out: attempt to convert all of them.

> I have this in mind too. First step is probably to move as much as
> possible as class properties.

We don't *have* to move "as much as possible" to support such a QAPI
integration, only the ones we want to expose to the user via QAPI.  But
we *want* to, because its an improvement on its own.

>                               Please review
> https://patchew.org/QEMU/20200110153039.1379601-1-marcandre.lureau@redhat.com/,
> I have more of this kind of qdev/qom cleanups pending.

Please lend Marc-André a hand whenever this work touches your area of
expertise :)

Re: Integrating QOM into QAPI (was: Making QEMU easier for management tools and applications)

[Daniel P. Berrangé]

On Tue, Jan 21, 2020 at 02:36:17PM +0100, Markus Armbruster wrote:
> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
> 
> > Hi
> >
> > On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >>
> >> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
> >> > Stefan Hajnoczi <stefanha@gmail.com> writes:
> >> >
> >> > > On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> >> > >> Christophe de Dinechin <dinechin@redhat.com> writes:
> >> > >> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
> >> > >> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
> >> > >>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
> >> > >>   documented (in qemu-options.hx), their use with qom-set is not.
> >> > >
> >> > > I'm happy to use a different interface.
> >> > >
> >> > > Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> >> > > be a step backwards.
> >> >
> >> > No argument.
> >> >
> >> > > Maybe the QAPI code generator could map something like this:
> >> > >
> >> > >   { 'command': 'iothread-set-poll-params',
> >> > >     'data': {
> >> > >         'id': 'str',
> >> > >     '*max-ns': 'uint64',
> >> > >     '*grow': 'uint64',
> >> > >     '*shrink': 'uint64'
> >> > >     },
> >> > >     'map-to-qom-set': 'IOThread'
> >> > >   }
> >> > >
> >> > > And turn it into QOM accessors on the IOThread object.
> >> >
> >> > I think a generic "set this configuration to that value" command is just
> >> > fine.  qom-set fails on several counts, though:
> >> >
> >> > * Tolerable: qom-set is not actually generic, it applies only to QOM.
> >> >
> >> > * qom-set lets you set tons of stuff that is not meant to be changed at
> >> >   run time.  If it breaks your guest, you get to keep the pieces.
> >> >
> >> > * There is virtually no documentation on what can be set to what values,
> >> >   and their semantics.
> >> >
> >> > In its current state, QOM is a user interface superfund site.
> >>
> >> Thoughts about a solution:
> >>
> >> Static QOM properties should be declared via QAPI instead of
> >> imperatively via QOM APIs.  That way they are introspectable and type
> >> information is present in the schema.
> >>
> >> The QAPI code generator could emit a function that is callable from
> >> .class_init().  This eliminates the need to manually call
> >> object_class_property_add().
> 
> We need to make up our minds what exactly we want generated.  Then we
> can design the QAPI language, and code up the generator.
> 
> Skeleton QOM type, to help with the discussion:
> 
>     #define TYPE_FOO "foo"
> 
>     #define FOO(obj) OBJECT_CHECK(Foo, (obj), TYPE_FOO)
>     #define FOO_CLASS(klass) \
>         OBJECT_CLASS_CHECK(FooClass, (klass), TYPE_FOO)
>     #define FOO_GET_CLASS(obj) \
>         OBJECT_GET_CLASS(FooClass, (obj), TYPE_FOO)
> 
>     typedef FooClass {
>         ParentClass parent_class;
>         ... // hand-written per-class state
>     }
> 
>     struct Chardev {
>         ParentObject parent_obj;
>         ... // hand-written instance (per-object) state
>     };
> 
>     static const TypeInfo char_type_info = {
>         .name = TYPE_FOO,
>         .parent = TYPE_OBJECT,
>         .instance_size = sizeof(Foo),
>         .instance_init = ...,                   // methods to initialize
>         .instance_post_init = ...,              // and finalize instances,
>         .instance_finalize = ...,               // all optional
>         .abstract = ...,                        // true or false (d'oh)
>         .class_size = sizeof(FooClass),
>         .class_init = ...,                      // methods to initialize
>         .class_base_init = ...,                 // classes, optional
>         .class_data = ...,                      // extra argument for them
>         .interfaces = ...
>     };
> 
> There's substantial boilerplate, with plenty of hand-written code in the
> gaps.  What of the boilerplate do we plan to generate?  How do we plan
> to fill the gaps, if any?

FWIW, even without a QOM generator, we can do waaaaaaay better on the
amount of boilerplate needed for QOM without very much work. It just
needs a few convenience macros writing.

QOM is not GObject, but is heavily inspired by it and so looking at
GObject gives us a design pattern we can aim to match in terms of
amount of boilerplate.

What we do manually with TypeInfo struct there has essentially always
been done by a 1 line macro in GObject:

  G_DEFINE_TYPE(virIdentity, vir_identity, G_TYPE_OBJECT)

If implementing interfaces, there's 1 extra line needed per interface
to associate them.

  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DEFINE-TYPE:CAPS
  

And what we do in the header file to add the 4 or more FOO_XXX macros,
and the class struct and the object struct has recently been turned
into a 2-liner:

  #define VIR_TYPE_IDENTITY vir_identity_get_type()
  G_DECLARE_FINAL_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);

  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-FINAL-TYPE:CAPS

Or 

  #define VIR_TYPE_IDENTITY vir_identity_get_type()
  G_DECLARE_DERIVABLE_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);

  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-DERIVABLE-TYPE:CAPS


It would be nice to have a QOM code generator so that we can statically
declare properties & parent/child/interface relationships, but for an
immediate low cost win, better macros would be very useful IMHO.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Integrating QOM into QAPI

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Jan 21, 2020 at 02:36:17PM +0100, Markus Armbruster wrote:
>> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
>> 
>> > Hi
>> >
>> > On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> >>
>> >> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
>> >> > Stefan Hajnoczi <stefanha@gmail.com> writes:
>> >> >
>> >> > > On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
>> >> > >> Christophe de Dinechin <dinechin@redhat.com> writes:
>> >> > >> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
>> >> > >> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>> >> > >>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>> >> > >>   documented (in qemu-options.hx), their use with qom-set is not.
>> >> > >
>> >> > > I'm happy to use a different interface.
>> >> > >
>> >> > > Writing a boilerplate "iothread-set-poll-params" QMP command in C would
>> >> > > be a step backwards.
>> >> >
>> >> > No argument.
>> >> >
>> >> > > Maybe the QAPI code generator could map something like this:
>> >> > >
>> >> > >   { 'command': 'iothread-set-poll-params',
>> >> > >     'data': {
>> >> > >         'id': 'str',
>> >> > >     '*max-ns': 'uint64',
>> >> > >     '*grow': 'uint64',
>> >> > >     '*shrink': 'uint64'
>> >> > >     },
>> >> > >     'map-to-qom-set': 'IOThread'
>> >> > >   }
>> >> > >
>> >> > > And turn it into QOM accessors on the IOThread object.
>> >> >
>> >> > I think a generic "set this configuration to that value" command is just
>> >> > fine.  qom-set fails on several counts, though:
>> >> >
>> >> > * Tolerable: qom-set is not actually generic, it applies only to QOM.
>> >> >
>> >> > * qom-set lets you set tons of stuff that is not meant to be changed at
>> >> >   run time.  If it breaks your guest, you get to keep the pieces.
>> >> >
>> >> > * There is virtually no documentation on what can be set to what values,
>> >> >   and their semantics.
>> >> >
>> >> > In its current state, QOM is a user interface superfund site.
>> >>
>> >> Thoughts about a solution:
>> >>
>> >> Static QOM properties should be declared via QAPI instead of
>> >> imperatively via QOM APIs.  That way they are introspectable and type
>> >> information is present in the schema.
>> >>
>> >> The QAPI code generator could emit a function that is callable from
>> >> .class_init().  This eliminates the need to manually call
>> >> object_class_property_add().
>> 
>> We need to make up our minds what exactly we want generated.  Then we
>> can design the QAPI language, and code up the generator.
>> 
>> Skeleton QOM type, to help with the discussion:
>> 
>>     #define TYPE_FOO "foo"
>> 
>>     #define FOO(obj) OBJECT_CHECK(Foo, (obj), TYPE_FOO)
>>     #define FOO_CLASS(klass) \
>>         OBJECT_CLASS_CHECK(FooClass, (klass), TYPE_FOO)
>>     #define FOO_GET_CLASS(obj) \
>>         OBJECT_GET_CLASS(FooClass, (obj), TYPE_FOO)
>> 
>>     typedef FooClass {
>>         ParentClass parent_class;
>>         ... // hand-written per-class state
>>     }
>> 
>>     struct Chardev {
>>         ParentObject parent_obj;
>>         ... // hand-written instance (per-object) state
>>     };
>> 
>>     static const TypeInfo char_type_info = {
>>         .name = TYPE_FOO,
>>         .parent = TYPE_OBJECT,
>>         .instance_size = sizeof(Foo),
>>         .instance_init = ...,                   // methods to initialize
>>         .instance_post_init = ...,              // and finalize instances,
>>         .instance_finalize = ...,               // all optional
>>         .abstract = ...,                        // true or false (d'oh)
>>         .class_size = sizeof(FooClass),
>>         .class_init = ...,                      // methods to initialize
>>         .class_base_init = ...,                 // classes, optional
>>         .class_data = ...,                      // extra argument for them
>>         .interfaces = ...
>>     };
>> 
>> There's substantial boilerplate, with plenty of hand-written code in the
>> gaps.  What of the boilerplate do we plan to generate?  How do we plan
>> to fill the gaps, if any?
>
> FWIW, even without a QOM generator, we can do waaaaaaay better on the
> amount of boilerplate needed for QOM without very much work. It just
> needs a few convenience macros writing.
>
> QOM is not GObject, but is heavily inspired by it and so looking at
> GObject gives us a design pattern we can aim to match in terms of
> amount of boilerplate.
>
> What we do manually with TypeInfo struct there has essentially always
> been done by a 1 line macro in GObject:
>
>   G_DEFINE_TYPE(virIdentity, vir_identity, G_TYPE_OBJECT)
>
> If implementing interfaces, there's 1 extra line needed per interface
> to associate them.
>
>   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DEFINE-TYPE:CAPS
>   
>
> And what we do in the header file to add the 4 or more FOO_XXX macros,
> and the class struct and the object struct has recently been turned
> into a 2-liner:
>
>   #define VIR_TYPE_IDENTITY vir_identity_get_type()
>   G_DECLARE_FINAL_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
>
>   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-FINAL-TYPE:CAPS
>
> Or 
>
>   #define VIR_TYPE_IDENTITY vir_identity_get_type()
>   G_DECLARE_DERIVABLE_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
>
>   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-DERIVABLE-TYPE:CAPS
>
>
> It would be nice to have a QOM code generator so that we can statically
> declare properties & parent/child/interface relationships, but for an
> immediate low cost win, better macros would be very useful IMHO.

Volunteers?

Re: Integrating QOM into QAPI

[Marc-André Lureau]

Hi

On Tue, Jan 21, 2020 at 7:01 PM Markus Armbruster <armbru@redhat.com> wrote:
>
> Daniel P. Berrangé <berrange@redhat.com> writes:
>
> > On Tue, Jan 21, 2020 at 02:36:17PM +0100, Markus Armbruster wrote:
> >> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
> >>
> >> > Hi
> >> >
> >> > On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >> >>
> >> >> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
> >> >> > Stefan Hajnoczi <stefanha@gmail.com> writes:
> >> >> >
> >> >> > > On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> >> >> > >> Christophe de Dinechin <dinechin@redhat.com> writes:
> >> >> > >> >> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
> >> >> > >> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
> >> >> > >>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
> >> >> > >>   documented (in qemu-options.hx), their use with qom-set is not.
> >> >> > >
> >> >> > > I'm happy to use a different interface.
> >> >> > >
> >> >> > > Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> >> >> > > be a step backwards.
> >> >> >
> >> >> > No argument.
> >> >> >
> >> >> > > Maybe the QAPI code generator could map something like this:
> >> >> > >
> >> >> > >   { 'command': 'iothread-set-poll-params',
> >> >> > >     'data': {
> >> >> > >         'id': 'str',
> >> >> > >     '*max-ns': 'uint64',
> >> >> > >     '*grow': 'uint64',
> >> >> > >     '*shrink': 'uint64'
> >> >> > >     },
> >> >> > >     'map-to-qom-set': 'IOThread'
> >> >> > >   }
> >> >> > >
> >> >> > > And turn it into QOM accessors on the IOThread object.
> >> >> >
> >> >> > I think a generic "set this configuration to that value" command is just
> >> >> > fine.  qom-set fails on several counts, though:
> >> >> >
> >> >> > * Tolerable: qom-set is not actually generic, it applies only to QOM.
> >> >> >
> >> >> > * qom-set lets you set tons of stuff that is not meant to be changed at
> >> >> >   run time.  If it breaks your guest, you get to keep the pieces.
> >> >> >
> >> >> > * There is virtually no documentation on what can be set to what values,
> >> >> >   and their semantics.
> >> >> >
> >> >> > In its current state, QOM is a user interface superfund site.
> >> >>
> >> >> Thoughts about a solution:
> >> >>
> >> >> Static QOM properties should be declared via QAPI instead of
> >> >> imperatively via QOM APIs.  That way they are introspectable and type
> >> >> information is present in the schema.
> >> >>
> >> >> The QAPI code generator could emit a function that is callable from
> >> >> .class_init().  This eliminates the need to manually call
> >> >> object_class_property_add().
> >>
> >> We need to make up our minds what exactly we want generated.  Then we
> >> can design the QAPI language, and code up the generator.
> >>
> >> Skeleton QOM type, to help with the discussion:
> >>
> >>     #define TYPE_FOO "foo"
> >>
> >>     #define FOO(obj) OBJECT_CHECK(Foo, (obj), TYPE_FOO)
> >>     #define FOO_CLASS(klass) \
> >>         OBJECT_CLASS_CHECK(FooClass, (klass), TYPE_FOO)
> >>     #define FOO_GET_CLASS(obj) \
> >>         OBJECT_GET_CLASS(FooClass, (obj), TYPE_FOO)
> >>
> >>     typedef FooClass {
> >>         ParentClass parent_class;
> >>         ... // hand-written per-class state
> >>     }
> >>
> >>     struct Chardev {
> >>         ParentObject parent_obj;
> >>         ... // hand-written instance (per-object) state
> >>     };
> >>
> >>     static const TypeInfo char_type_info = {
> >>         .name = TYPE_FOO,
> >>         .parent = TYPE_OBJECT,
> >>         .instance_size = sizeof(Foo),
> >>         .instance_init = ...,                   // methods to initialize
> >>         .instance_post_init = ...,              // and finalize instances,
> >>         .instance_finalize = ...,               // all optional
> >>         .abstract = ...,                        // true or false (d'oh)
> >>         .class_size = sizeof(FooClass),
> >>         .class_init = ...,                      // methods to initialize
> >>         .class_base_init = ...,                 // classes, optional
> >>         .class_data = ...,                      // extra argument for them
> >>         .interfaces = ...
> >>     };
> >>
> >> There's substantial boilerplate, with plenty of hand-written code in the
> >> gaps.  What of the boilerplate do we plan to generate?  How do we plan
> >> to fill the gaps, if any?
> >
> > FWIW, even without a QOM generator, we can do waaaaaaay better on the
> > amount of boilerplate needed for QOM without very much work. It just
> > needs a few convenience macros writing.
> >
> > QOM is not GObject, but is heavily inspired by it and so looking at
> > GObject gives us a design pattern we can aim to match in terms of
> > amount of boilerplate.
> >
> > What we do manually with TypeInfo struct there has essentially always
> > been done by a 1 line macro in GObject:
> >
> >   G_DEFINE_TYPE(virIdentity, vir_identity, G_TYPE_OBJECT)
> >
> > If implementing interfaces, there's 1 extra line needed per interface
> > to associate them.
> >
> >   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DEFINE-TYPE:CAPS
> >
> >
> > And what we do in the header file to add the 4 or more FOO_XXX macros,
> > and the class struct and the object struct has recently been turned
> > into a 2-liner:
> >
> >   #define VIR_TYPE_IDENTITY vir_identity_get_type()
> >   G_DECLARE_FINAL_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
> >
> >   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-FINAL-TYPE:CAPS
> >
> > Or
> >
> >   #define VIR_TYPE_IDENTITY vir_identity_get_type()
> >   G_DECLARE_DERIVABLE_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
> >
> >   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-DERIVABLE-TYPE:CAPS
> >
> >
> > It would be nice to have a QOM code generator so that we can statically
> > declare properties & parent/child/interface relationships, but for an
> > immediate low cost win, better macros would be very useful IMHO.
>
> Volunteers?
>

Actually, we are not that far off from being able to use GObject
altogether (I hacked something like that to play with), but I
disgress...

So introducing GObject-like macros? sure!

There are plenty of refactoring to do. The problem when touching the
whole code-base, imho, is review time. It may take a couple of
hours/days to come up with a cocci/spatch, and make various patches
here and there. But it takes often weeks and a lot of constant push to
various folks to get all the reviews (as seens by the qdev prop-ptr
series earlier for example). How can we better address whole code-base
changes?



-- 
Marc-André Lureau

Re: Integrating QOM into QAPI

[Peter Maydell]

On Tue, 21 Jan 2020 at 15:11, Marc-André Lureau
<marcandre.lureau@gmail.com> wrote:
> There are plenty of refactoring to do. The problem when touching the
> whole code-base, imho, is review time. It may take a couple of
> hours/days to come up with a cocci/spatch, and make various patches
> here and there. But it takes often weeks and a lot of constant push to
> various folks to get all the reviews (as seens by the qdev prop-ptr
> series earlier for example). How can we better address whole code-base
> changes?

It depends. If it's literally just a cocci/spatch mechanical
transformation, I think we should be OK with reviewing that
transform and then applying it; we don't need to get acks
from every submaintainer for that kind of thing. The
prop-ptr series was harder work because it required making
active non-mechanical changes to various devices, which
means the specific changes needed review.

PS: thanks for pushing that prop-ptr work through and
especially the serial device part which was both awkward
and definitely needed.

thanks
-- PMM

Getting whole-tree patches reviewed and merged (was: Integrating QOM into QAPI)

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Tue, 21 Jan 2020 at 15:11, Marc-André Lureau
> <marcandre.lureau@gmail.com> wrote:
>> There are plenty of refactoring to do. The problem when touching the
>> whole code-base, imho, is review time. It may take a couple of
>> hours/days to come up with a cocci/spatch, and make various patches
>> here and there. But it takes often weeks and a lot of constant push to
>> various folks to get all the reviews (as seens by the qdev prop-ptr
>> series earlier for example). How can we better address whole code-base
>> changes?
>
> It depends. If it's literally just a cocci/spatch mechanical
> transformation, I think we should be OK with reviewing that
> transform and then applying it; we don't need to get acks
> from every submaintainer for that kind of thing.

I go one step further: I prefer mechanical changes committed together,
not torn apart and routed through N+1 trees, where N is the number of
active maintainers picking patches from the series, and 1 is the
maintainer taking pity of the inevitable leftovers.

Tearing a patch series apart may be in order when it's invasive enough
to risk many conflicts.  The subsystem maintainer may need tighter
control over merging order then, and routing picked patches through his
own tree may be the practical way to get it.

[...]

Re: Getting whole-tree patches reviewed and merged

[Eric Blake]

On 1/21/20 11:16 PM, Markus Armbruster wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
>> On Tue, 21 Jan 2020 at 15:11, Marc-André Lureau
>> <marcandre.lureau@gmail.com> wrote:
>>> There are plenty of refactoring to do. The problem when touching the
>>> whole code-base, imho, is review time. It may take a couple of
>>> hours/days to come up with a cocci/spatch, and make various patches
>>> here and there. But it takes often weeks and a lot of constant push to
>>> various folks to get all the reviews (as seens by the qdev prop-ptr
>>> series earlier for example). How can we better address whole code-base
>>> changes?
>>
>> It depends. If it's literally just a cocci/spatch mechanical
>> transformation, I think we should be OK with reviewing that
>> transform and then applying it; we don't need to get acks
>> from every submaintainer for that kind of thing.
> 
> I go one step further: I prefer mechanical changes committed together,
> not torn apart and routed through N+1 trees, where N is the number of
> active maintainers picking patches from the series, and 1 is the
> maintainer taking pity of the inevitable leftovers.
> 
> Tearing a patch series apart may be in order when it's invasive enough
> to risk many conflicts.  The subsystem maintainer may need tighter
> control over merging order then, and routing picked patches through his
> own tree may be the practical way to get it.

And the pending work on ERRP_AUTO_PROPAGATE is an example of that - 
Vladimir has been trying to get the improvements in for some time, but 
it touches so many files, and is awkward to review whether it is split 
into over 100 patches per maintainer area or when it is consolidates 
into few but large patches.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: Getting whole-tree patches reviewed and merged

[Paolo Bonzini]

On 07/02/20 22:53, Eric Blake wrote:
> On 1/21/20 11:16 PM, Markus Armbruster wrote:
>> Peter Maydell <peter.maydell@linaro.org> writes:
>>
>>> On Tue, 21 Jan 2020 at 15:11, Marc-André Lureau
>>> <marcandre.lureau@gmail.com> wrote:
>>>> There are plenty of refactoring to do. The problem when touching the
>>>> whole code-base, imho, is review time. It may take a couple of
>>>> hours/days to come up with a cocci/spatch, and make various patches
>>>> here and there. But it takes often weeks and a lot of constant push to
>>>> various folks to get all the reviews (as seens by the qdev prop-ptr
>>>> series earlier for example). How can we better address whole code-base
>>>> changes?
>>>
>>> It depends. If it's literally just a cocci/spatch mechanical
>>> transformation, I think we should be OK with reviewing that
>>> transform and then applying it; we don't need to get acks
>>> from every submaintainer for that kind of thing.
>>
>> I go one step further: I prefer mechanical changes committed together,
>> not torn apart and routed through N+1 trees, where N is the number of
>> active maintainers picking patches from the series, and 1 is the
>> maintainer taking pity of the inevitable leftovers.
>>
>> Tearing a patch series apart may be in order when it's invasive enough
>> to risk many conflicts.  The subsystem maintainer may need tighter
>> control over merging order then, and routing picked patches through his
>> own tree may be the practical way to get it.
> 
> And the pending work on ERRP_AUTO_PROPAGATE is an example of that -
> Vladimir has been trying to get the improvements in for some time, but
> it touches so many files, and is awkward to review whether it is split
> into over 100 patches per maintainer area or when it is consolidates
> into few but large patches.

If I can help with ERRP_AUTO_PROPAGATE, I can merge it through my tree.

Paolo

Re: Getting whole-tree patches reviewed and merged

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 07/02/20 22:53, Eric Blake wrote:
>> On 1/21/20 11:16 PM, Markus Armbruster wrote:
>>> Peter Maydell <peter.maydell@linaro.org> writes:
>>>
>>>> On Tue, 21 Jan 2020 at 15:11, Marc-André Lureau
>>>> <marcandre.lureau@gmail.com> wrote:
>>>>> There are plenty of refactoring to do. The problem when touching the
>>>>> whole code-base, imho, is review time. It may take a couple of
>>>>> hours/days to come up with a cocci/spatch, and make various patches
>>>>> here and there. But it takes often weeks and a lot of constant push to
>>>>> various folks to get all the reviews (as seens by the qdev prop-ptr
>>>>> series earlier for example). How can we better address whole code-base
>>>>> changes?
>>>>
>>>> It depends. If it's literally just a cocci/spatch mechanical
>>>> transformation, I think we should be OK with reviewing that
>>>> transform and then applying it; we don't need to get acks
>>>> from every submaintainer for that kind of thing.
>>>
>>> I go one step further: I prefer mechanical changes committed together,
>>> not torn apart and routed through N+1 trees, where N is the number of
>>> active maintainers picking patches from the series, and 1 is the
>>> maintainer taking pity of the inevitable leftovers.
>>>
>>> Tearing a patch series apart may be in order when it's invasive enough
>>> to risk many conflicts.  The subsystem maintainer may need tighter
>>> control over merging order then, and routing picked patches through his
>>> own tree may be the practical way to get it.
>> 
>> And the pending work on ERRP_AUTO_PROPAGATE is an example of that -
>> Vladimir has been trying to get the improvements in for some time, but
>> it touches so many files, and is awkward to review whether it is split
>> into over 100 patches per maintainer area or when it is consolidates
>> into few but large patches.
>
> If I can help with ERRP_AUTO_PROPAGATE, I can merge it through my tree.

Right now I'm to blame: I need to review the infrastructure and its
pattern of use.

Once that's done, the next questions are what level of review we want
for the instances of the pattern, and how to merge the thing.

One way is to split along subsystems boundaries, and ask subsystem
maintainers to merge their part.  Infrastructure has to go first, of
course.  That would be my job as error subsystem maintainer.  Getting
everything merged via pretty much every subsystem we have will likely be
a drawn out affair.

Another way is to go ahead and merge everything, damn the torpedoes.

Re: Getting whole-tree patches reviewed and merged

[Peter Maydell]

On Mon, 10 Feb 2020 at 16:04, Markus Armbruster <armbru@redhat.com> wrote:
> Once that's done, the next questions are what level of review we want
> for the instances of the pattern, and how to merge the thing.
>
> One way is to split along subsystems boundaries, and ask subsystem
> maintainers to merge their part.  Infrastructure has to go first, of
> course.  That would be my job as error subsystem maintainer.  Getting
> everything merged via pretty much every subsystem we have will likely be
> a drawn out affair.
>
> Another way is to go ahead and merge everything, damn the torpedoes.

For this sort of thing, provided that somebody has reviewed the
overall approach, and somebody has reviewed each patch at least
vaguely, and we've given subsystem maintainers a chance
to say "I would prefer to take the patches that apply to the
subsystems I care for via my tree", in my view we should default
to "take the patches via one tree/pullreq" (in this case that
would be one of yours I guess). I suspect most of the time
most submaintainers won't care and will be happier not having
to actively deal with the change...

thanks
-- PMM

Re: Integrating QOM into QAPI

[Alex Bennée]

Marc-André Lureau <marcandre.lureau@gmail.com> writes:

> Hi
>
> On Tue, Jan 21, 2020 at 7:01 PM Markus Armbruster <armbru@redhat.com> wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> > On Tue, Jan 21, 2020 at 02:36:17PM +0100, Markus Armbruster wrote:
<snip>
>> >>     static const TypeInfo char_type_info = {
>> >>         .name = TYPE_FOO,
>> >>         .parent = TYPE_OBJECT,
>> >>         .instance_size = sizeof(Foo),
>> >>         .instance_init = ...,                   // methods to initialize
>> >>         .instance_post_init = ...,              // and finalize instances,
>> >>         .instance_finalize = ...,               // all optional
>> >>         .abstract = ...,                        // true or false (d'oh)
>> >>         .class_size = sizeof(FooClass),
>> >>         .class_init = ...,                      // methods to initialize
>> >>         .class_base_init = ...,                 // classes, optional
>> >>         .class_data = ...,                      // extra argument for them
>> >>         .interfaces = ...
>> >>     };
>> >>
>> >> There's substantial boilerplate, with plenty of hand-written code in the
>> >> gaps.  What of the boilerplate do we plan to generate?  How do we plan
>> >> to fill the gaps, if any?
>> >
>> > FWIW, even without a QOM generator, we can do waaaaaaay better on the
>> > amount of boilerplate needed for QOM without very much work. It just
>> > needs a few convenience macros writing.
>> >
>> > QOM is not GObject, but is heavily inspired by it and so looking at
>> > GObject gives us a design pattern we can aim to match in terms of
>> > amount of boilerplate.
>> >
>> > What we do manually with TypeInfo struct there has essentially always
>> > been done by a 1 line macro in GObject:
>> >
>> >   G_DEFINE_TYPE(virIdentity, vir_identity, G_TYPE_OBJECT)
>> >
>> > If implementing interfaces, there's 1 extra line needed per interface
>> > to associate them.
>> >
>> >   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DEFINE-TYPE:CAPS
>> >
>> >
>> > And what we do in the header file to add the 4 or more FOO_XXX macros,
>> > and the class struct and the object struct has recently been turned
>> > into a 2-liner:
>> >
>> >   #define VIR_TYPE_IDENTITY vir_identity_get_type()
>> >   G_DECLARE_FINAL_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
>> >
>> >   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-FINAL-TYPE:CAPS
>> >
>> > Or
>> >
>> >   #define VIR_TYPE_IDENTITY vir_identity_get_type()
>> >   G_DECLARE_DERIVABLE_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
>> >
>> >   https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-DERIVABLE-TYPE:CAPS
>> >
>> >
>> > It would be nice to have a QOM code generator so that we can statically
>> > declare properties & parent/child/interface relationships, but for an
>> > immediate low cost win, better macros would be very useful IMHO.
>>
>> Volunteers?
>>
>
> Actually, we are not that far off from being able to use GObject
> altogether (I hacked something like that to play with), but I
> disgress...

As a mostly hands off observer who mainly c&p's QOM code when he has to
I have to ask is this a long term plan?

I've always found having our own hand rolled object system a little
incongruous given we lean heavily on the rest of glib.

> So introducing GObject-like macros? sure!
>
> There are plenty of refactoring to do. The problem when touching the
> whole code-base, imho, is review time. It may take a couple of
> hours/days to come up with a cocci/spatch, and make various patches
> here and there. But it takes often weeks and a lot of constant push to
> various folks to get all the reviews (as seens by the qdev prop-ptr
> series earlier for example). How can we better address whole code-base
> changes?

The problem with review time - especially for QOM - is having domain
knowledge to understand what is happening.

Are we happy that the existing qdev/qmp tests sufficiently exercise
QEMU's object model? Perhaps with a little extra tweaking of the tests
we could dump the object hierarchy and then compare it to the hierarchy
presented after modification. That might make it easier to have
confidence that these large scale but mostly mechanical changes don't
change anything externally visible?

--
Alex Bennée

Re: Integrating QOM into QAPI

[Markus Armbruster]

Alex Bennée <alex.bennee@linaro.org> writes:

> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
>> Actually, we are not that far off from being able to use GObject
>> altogether (I hacked something like that to play with), but I
>> disgress...
>
> As a mostly hands off observer who mainly c&p's QOM code when he has to
> I have to ask is this a long term plan?
>
> I've always found having our own hand rolled object system a little
> incongruous given we lean heavily on the rest of glib.

I vaguely remember claims that GObject falls short of our needs.  Sadly,
I don't remember the details.  This is why major features should come
with a design document.

https://wiki.qemu.org/Features/QOM ain't: it does not mention GObject.
I'm afraid that page has fallen too far behind the code to be useful to
anyone not familiar with the code.

>> So introducing GObject-like macros? sure!
>>
>> There are plenty of refactoring to do. The problem when touching the
>> whole code-base, imho, is review time. It may take a couple of
>> hours/days to come up with a cocci/spatch, and make various patches
>> here and there. But it takes often weeks and a lot of constant push to
>> various folks to get all the reviews (as seens by the qdev prop-ptr
>> series earlier for example). How can we better address whole code-base
>> changes?
>
> The problem with review time - especially for QOM - is having domain
> knowledge to understand what is happening.
>
> Are we happy that the existing qdev/qmp tests sufficiently exercise
> QEMU's object model? Perhaps with a little extra tweaking of the tests
> we could dump the object hierarchy and then compare it to the hierarchy
> presented after modification. That might make it easier to have
> confidence that these large scale but mostly mechanical changes don't
> change anything externally visible?

Comparing the composition tree complete with properties and property
values before and after feels like a useful regression test.  Any
takers?

Re: Integrating QOM into QAPI

[Marc-André Lureau]

Hi

On Wed, Jan 22, 2020 at 4:25 PM Markus Armbruster <armbru@redhat.com> wrote:
>
> Alex Bennée <alex.bennee@linaro.org> writes:
>
> > Marc-André Lureau <marcandre.lureau@gmail.com> writes:
> >> Actually, we are not that far off from being able to use GObject
> >> altogether (I hacked something like that to play with), but I
> >> disgress...
> >
> > As a mostly hands off observer who mainly c&p's QOM code when he has to
> > I have to ask is this a long term plan?
> >
> > I've always found having our own hand rolled object system a little
> > incongruous given we lean heavily on the rest of glib.
>
> I vaguely remember claims that GObject falls short of our needs.  Sadly,
> I don't remember the details.  This is why major features should come
> with a design document.
>
> https://wiki.qemu.org/Features/QOM ain't: it does not mention GObject.
> I'm afraid that page has fallen too far behind the code to be useful to
> anyone not familiar with the code.

From the top of my mind, this is the pain point when trying to use GObject:
- static/inlined object, not supported by GObject, unlikely to ever be
- few users in qemu, transition possible.
- 64k limit of GObject, for some reason, unlikely to change but I will
take a look. Some users in qemu, code adaptation possible.
- dynamic properties, possible in GObject with hacks, but not
recommended and going to be deprecated from what I remember
- "array" properties - would need extra layer/tweaks for compatibility
- link properties - would need special handling
- different limitations for type names and properties names

A possible initial approach is to have all the type system and object
allocation done by GObject under the hood (what I hacked), while
keeping all the properties handled by QOM. Then, figure out a
migration to GObject properties (which are also being refactored a bit
upstream). If there is enough interest, I will keep investigating. But
for now, helping with meson seems more urgent.

>
> >> So introducing GObject-like macros? sure!
> >>
> >> There are plenty of refactoring to do. The problem when touching the
> >> whole code-base, imho, is review time. It may take a couple of
> >> hours/days to come up with a cocci/spatch, and make various patches
> >> here and there. But it takes often weeks and a lot of constant push to
> >> various folks to get all the reviews (as seens by the qdev prop-ptr
> >> series earlier for example). How can we better address whole code-base
> >> changes?
> >
> > The problem with review time - especially for QOM - is having domain
> > knowledge to understand what is happening.
> >
> > Are we happy that the existing qdev/qmp tests sufficiently exercise
> > QEMU's object model? Perhaps with a little extra tweaking of the tests
> > we could dump the object hierarchy and then compare it to the hierarchy
> > presented after modification. That might make it easier to have
> > confidence that these large scale but mostly mechanical changes don't
> > change anything externally visible?
>
> Comparing the composition tree complete with properties and property
> values before and after feels like a useful regression test.  Any
> takers?
>


-- 
Marc-André Lureau

Re: Integrating QOM into QAPI

[Peter Maydell]

On Wed, 22 Jan 2020 at 12:42, Marc-André Lureau
<marcandre.lureau@gmail.com> wrote:
> From the top of my mind, this is the pain point when trying to use GObject:
> - static/inlined object, not supported by GObject, unlikely to ever be
> - few users in qemu, transition possible.

Isn't there lots of use of this in the device emulation, or am I
misunderstanding what the restriction is?

thanks
-- PMM

Re: Integrating QOM into QAPI

[Marc-André Lureau]

Hi

On Wed, Jan 22, 2020 at 5:28 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Wed, 22 Jan 2020 at 12:42, Marc-André Lureau
> <marcandre.lureau@gmail.com> wrote:
> > From the top of my mind, this is the pain point when trying to use GObject:
> > - static/inlined object, not supported by GObject, unlikely to ever be
> > - few users in qemu, transition possible.
>
> Isn't there lots of use of this in the device emulation, or am I
> misunderstanding what the restriction is?

There are a bunch, from what I remember. Yes, that's a bit painful,
but spatch managed to rewrite most of dev.subdev.foo to
dev.subdev->foo iirc.


-- 
Marc-André Lureau

Re: Integrating QOM into QAPI

[Markus Armbruster]

Marc-André Lureau <marcandre.lureau@gmail.com> writes:

> Hi
>
> On Wed, Jan 22, 2020 at 4:25 PM Markus Armbruster <armbru@redhat.com> wrote:
>>
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>
>> > Marc-André Lureau <marcandre.lureau@gmail.com> writes:
>> >> Actually, we are not that far off from being able to use GObject
>> >> altogether (I hacked something like that to play with), but I
>> >> disgress...
>> >
>> > As a mostly hands off observer who mainly c&p's QOM code when he has to
>> > I have to ask is this a long term plan?
>> >
>> > I've always found having our own hand rolled object system a little
>> > incongruous given we lean heavily on the rest of glib.
>>
>> I vaguely remember claims that GObject falls short of our needs.  Sadly,
>> I don't remember the details.  This is why major features should come
>> with a design document.
>>
>> https://wiki.qemu.org/Features/QOM ain't: it does not mention GObject.
>> I'm afraid that page has fallen too far behind the code to be useful to
>> anyone not familiar with the code.
>
>>From the top of my mind, this is the pain point when trying to use GObject:
> - static/inlined object, not supported by GObject, unlikely to ever be

Lame.

Okay for us as long as the pointer chasing stays off the hot paths.  But
that's not obvious.

> - few users in qemu, transition possible.

Peter challenged this idea.

> - 64k limit of GObject, for some reason, unlikely to change but I will
> take a look. Some users in qemu, code adaptation possible.

Also lame.  I figure code adaption will involve pointer chasing.  Again,
beware of hot paths.  This one feels less likely to be a problem,
though.

> - dynamic properties, possible in GObject with hacks, but not
> recommended and going to be deprecated from what I remember

Never understood our need for them for longer than half a day after
getting it explained to me.  I guess that's my fault.  Anyway, the less
we use them, the better.

Hacks exploiting deprecated GObject features would be a bad idea.
Perhaps we can hack something up that doesn't.

> - "array" properties - would need extra layer/tweaks for compatibility

I hated them from day one, tried to kill them, lost the argument.

> - link properties - would need special handling

Hard or not?

> - different limitations for type names and properties names

What to do?

Our naming rules and conventions are weakly documented.

> A possible initial approach is to have all the type system and object
> allocation done by GObject under the hood (what I hacked), while
> keeping all the properties handled by QOM. Then, figure out a
> migration to GObject properties (which are also being refactored a bit
> upstream).

So most QOM properties would be based on a GObject property, but some
wouldn't be, correct?

>            If there is enough interest, I will keep investigating. But
> for now, helping with meson seems more urgent.

Yes.

/me warily eyes the monster lurking in the review queue...

[...]

Re: Integrating QOM into QAPI

[Paolo Bonzini]

On 22/01/20 13:42, Marc-André Lureau wrote:
> From the top of my mind, this is the pain point when trying to use GObject:
> - static/inlined object, not supported by GObject, unlikely to ever be
> - few users in qemu, transition possible.
> - 64k limit of GObject, for some reason, unlikely to change but I will
> take a look. Some users in qemu, code adaptation possible.
> - dynamic properties, possible in GObject with hacks, but not
> recommended and going to be deprecated from what I remember
> - "array" properties - would need extra layer/tweaks for compatibility
> - link properties - would need special handling
> - different limitations for type names and properties names

The properties in general are very different between QOM and QAPI.  They
have different limitations and features as Marc-André mentioned, but an
especially important one is the integration with QAPI visitors.  This is
what allows us to support -object and object-add with the same code, and
is what separates QOM from GObject the most.

Maybe it would be possible to build an adapter, but having written in
the past code that uses GType to do marshalling and unmarshalling, I'm
not really fond of repeating the experience...

Paolo

Re: Integrating QOM into QAPI

[Marc-André Lureau]

[-- Attachment #1: Type: text/plain, Size: 1691 bytes --]

Hi

Le ven. 24 janv. 2020 à 19:32, Paolo Bonzini <pbonzini@redhat.com> a écrit :

> On 22/01/20 13:42, Marc-André Lureau wrote:
> > From the top of my mind, this is the pain point when trying to use
> GObject:
> > - static/inlined object, not supported by GObject, unlikely to ever be
> > - few users in qemu, transition possible.
> > - 64k limit of GObject, for some reason, unlikely to change but I will
> > take a look. Some users in qemu, code adaptation possible.
> > - dynamic properties, possible in GObject with hacks, but not
> > recommended and going to be deprecated from what I remember
> > - "array" properties - would need extra layer/tweaks for compatibility
> > - link properties - would need special handling
> > - different limitations for type names and properties names
>
> The properties in general are very different between QOM and QAPI.  They
> have different limitations and features as Marc-André mentioned, but an
> especially important one is the integration with QAPI visitors.  This is
> what allows us to support -object and object-add with the same code, and
> is what separates QOM from GObject the most.
>
> Maybe it would be possible to build an adapter, but having written in
> the past code that uses GType to do marshalling and unmarshalling, I'm
> not really fond of repeating the experience...
>

I agree it is one of the things that look very different from gobject. At
the same time, I think defining conventions/types or interface to describe
hierarchy isn't so difficult, and then adapting the visitors shouldn't be
either.

I try to find a good reason qom was chosen over gobject, and I can't find
it.

>

[-- Attachment #2: Type: text/html, Size: 2305 bytes --]

Re: Integrating QOM into QAPI

[Paolo Bonzini]

On 25/01/20 05:44, Marc-André Lureau wrote:
>     On 22/01/20 13:42, Marc-André Lureau wrote:
>     > From the top of my mind, this is the pain point when trying to use
>     GObject:
>     > - static/inlined object, not supported by GObject, unlikely to ever be
>     > - few users in qemu, transition possible.
>     > - 64k limit of GObject, for some reason, unlikely to change but I will
>     > take a look. Some users in qemu, code adaptation possible.
>     > - dynamic properties, possible in GObject with hacks, but not
>     > recommended and going to be deprecated from what I remember
>     > - "array" properties - would need extra layer/tweaks for compatibility
>     > - link properties - would need special handling
>     > - different limitations for type names and properties names
> 
>     The properties in general are very different between QOM and QAPI.  They
>     have different limitations and features as Marc-André mentioned, but an
>     especially important one is the integration with QAPI visitors.  This is
>     what allows us to support -object and object-add with the same code, and
>     is what separates QOM from GObject the most.
> 
>     Maybe it would be possible to build an adapter, but having written in
>     the past code that uses GType to do marshalling and unmarshalling, I'm
>     not really fond of repeating the experience...
> 
> I agree it is one of the things that look very different from gobject.
> At the same time, I think defining conventions/types or interface to
> describe hierarchy isn't so difficult, and then adapting the visitors
> shouldn't be either.

Note that there are a few "structured" properties that export a QAPI
struct rather than a scalar.  Those would be a bit more complex.  Links
might also be tricky.

Another small difference that came to mind is the different object
liveness model; GObject is mostly plain old reference counting (plus the
"floating reference" idea), while in QOM reference counting is secondary
and lifetime is mostly dictated by the object tree and "unparenting".
This is more of a conceptual difference; it could be easily retrofitted
in GObject.

> I try to find a good reason qom was chosen over gobject, and I can't
> find it.

The main reasons were integration with QAPI, and the object tree.
Though everything I say here is a kind of reverse engineering of
Anthony's brain because there aren't really any design documents besides
what's in include/qom/object.h (and he overlooked some aspects, for
example "unparent" was introduced a few months later).

Overall I don't think there would be much benefit in reusing GObject.
It's about 3k lines of code, quite a few would stay (those implementing
the tree) and a bunch more would have to be rewritten.  I don't think
we'd have any use for most of the features that QOM lacks, such as
signals.  Also, QOM is quite well documented and we should include its
documentation in docs/devel instead.

That said, coccinell-ing efforts on QOM code are definitely a good idea
since it's quite mature now and we know better what we need.  And there
are practical advantages too, it's not just code cleanups---see how your
series shows the default value of the properties in -device foo,help.

Paolo

Re: Integrating QOM into QAPI

[Peter Maydell]

On Sat, 25 Jan 2020 at 09:28, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 25/01/20 05:44, Marc-André Lureau wrote:
> > I try to find a good reason qom was chosen over gobject, and I can't
> > find it.
>
> The main reasons were integration with QAPI, and the object tree.
> Though everything I say here is a kind of reverse engineering of
> Anthony's brain because there aren't really any design documents besides
> what's in include/qom/object.h (and he overlooked some aspects, for
> example "unparent" was introduced a few months later).

I vagely recall that back at that time we were a lot less heavy
in our usage of glib also, so "just use the glib version of whatever"
would not have been quite as easy a sell as it might be today.
Anthony's original RFC email lists some "key differences" between
QOM and GObject, which presumably seemed to him at the time
to be sufficient to justify not using GObject:
https://lists.gnu.org/archive/html/qemu-devel/2011-07/msg01673.html

thanks
-- PMM

Re: Integrating QOM into QAPI

[Christophe de Dinechin]

> On 21 Jan 2020, at 16:11, Marc-André Lureau <marcandre.lureau@gmail.com> wrote:
> 
> Hi
> 
> On Tue, Jan 21, 2020 at 7:01 PM Markus Armbruster <armbru@redhat.com> wrote:
>> 
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>>> On Tue, Jan 21, 2020 at 02:36:17PM +0100, Markus Armbruster wrote:
>>>> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
>>>> 
>>>>> Hi
>>>>> 
>>>>> On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>>>> 
>>>>>> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
>>>>>>> Stefan Hajnoczi <stefanha@gmail.com> writes:
>>>>>>> 
>>>>>>>> On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
>>>>>>>>> Christophe de Dinechin <dinechin@redhat.com> writes:
>>>>>>>>>>> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
>>>>>>>>> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>>>>>>>>>  poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>>>>>>>>>  documented (in qemu-options.hx), their use with qom-set is not.
>>>>>>>> 
>>>>>>>> I'm happy to use a different interface.
>>>>>>>> 
>>>>>>>> Writing a boilerplate "iothread-set-poll-params" QMP command in C would
>>>>>>>> be a step backwards.
>>>>>>> 
>>>>>>> No argument.
>>>>>>> 
>>>>>>>> Maybe the QAPI code generator could map something like this:
>>>>>>>> 
>>>>>>>>  { 'command': 'iothread-set-poll-params',
>>>>>>>>    'data': {
>>>>>>>>        'id': 'str',
>>>>>>>>    '*max-ns': 'uint64',
>>>>>>>>    '*grow': 'uint64',
>>>>>>>>    '*shrink': 'uint64'
>>>>>>>>    },
>>>>>>>>    'map-to-qom-set': 'IOThread'
>>>>>>>>  }
>>>>>>>> 
>>>>>>>> And turn it into QOM accessors on the IOThread object.
>>>>>>> 
>>>>>>> I think a generic "set this configuration to that value" command is just
>>>>>>> fine.  qom-set fails on several counts, though:
>>>>>>> 
>>>>>>> * Tolerable: qom-set is not actually generic, it applies only to QOM.
>>>>>>> 
>>>>>>> * qom-set lets you set tons of stuff that is not meant to be changed at
>>>>>>>  run time.  If it breaks your guest, you get to keep the pieces.
>>>>>>> 
>>>>>>> * There is virtually no documentation on what can be set to what values,
>>>>>>>  and their semantics.
>>>>>>> 
>>>>>>> In its current state, QOM is a user interface superfund site.
>>>>>> 
>>>>>> Thoughts about a solution:
>>>>>> 
>>>>>> Static QOM properties should be declared via QAPI instead of
>>>>>> imperatively via QOM APIs.  That way they are introspectable and type
>>>>>> information is present in the schema.
>>>>>> 
>>>>>> The QAPI code generator could emit a function that is callable from
>>>>>> .class_init().  This eliminates the need to manually call
>>>>>> object_class_property_add().
>>>> 
>>>> We need to make up our minds what exactly we want generated.  Then we
>>>> can design the QAPI language, and code up the generator.
>>>> 
>>>> Skeleton QOM type, to help with the discussion:
>>>> 
>>>>    #define TYPE_FOO "foo"
>>>> 
>>>>    #define FOO(obj) OBJECT_CHECK(Foo, (obj), TYPE_FOO)
>>>>    #define FOO_CLASS(klass) \
>>>>        OBJECT_CLASS_CHECK(FooClass, (klass), TYPE_FOO)
>>>>    #define FOO_GET_CLASS(obj) \
>>>>        OBJECT_GET_CLASS(FooClass, (obj), TYPE_FOO)
>>>> 
>>>>    typedef FooClass {
>>>>        ParentClass parent_class;
>>>>        ... // hand-written per-class state
>>>>    }
>>>> 
>>>>    struct Chardev {
>>>>        ParentObject parent_obj;
>>>>        ... // hand-written instance (per-object) state
>>>>    };
>>>> 
>>>>    static const TypeInfo char_type_info = {
>>>>        .name = TYPE_FOO,
>>>>        .parent = TYPE_OBJECT,
>>>>        .instance_size = sizeof(Foo),
>>>>        .instance_init = ...,                   // methods to initialize
>>>>        .instance_post_init = ...,              // and finalize instances,
>>>>        .instance_finalize = ...,               // all optional
>>>>        .abstract = ...,                        // true or false (d'oh)
>>>>        .class_size = sizeof(FooClass),
>>>>        .class_init = ...,                      // methods to initialize
>>>>        .class_base_init = ...,                 // classes, optional
>>>>        .class_data = ...,                      // extra argument for them
>>>>        .interfaces = ...
>>>>    };
>>>> 
>>>> There's substantial boilerplate, with plenty of hand-written code in the
>>>> gaps.  What of the boilerplate do we plan to generate?  How do we plan
>>>> to fill the gaps, if any?
>>> 
>>> FWIW, even without a QOM generator, we can do waaaaaaay better on the
>>> amount of boilerplate needed for QOM without very much work. It just
>>> needs a few convenience macros writing.
>>> 
>>> QOM is not GObject, but is heavily inspired by it and so looking at
>>> GObject gives us a design pattern we can aim to match in terms of
>>> amount of boilerplate.
>>> 
>>> What we do manually with TypeInfo struct there has essentially always
>>> been done by a 1 line macro in GObject:
>>> 
>>>  G_DEFINE_TYPE(virIdentity, vir_identity, G_TYPE_OBJECT)
>>> 
>>> If implementing interfaces, there's 1 extra line needed per interface
>>> to associate them.
>>> 
>>>  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DEFINE-TYPE:CAPS
>>> 
>>> 
>>> And what we do in the header file to add the 4 or more FOO_XXX macros,
>>> and the class struct and the object struct has recently been turned
>>> into a 2-liner:
>>> 
>>>  #define VIR_TYPE_IDENTITY vir_identity_get_type()
>>>  G_DECLARE_FINAL_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
>>> 
>>>  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-FINAL-TYPE:CAPS
>>> 
>>> Or
>>> 
>>>  #define VIR_TYPE_IDENTITY vir_identity_get_type()
>>>  G_DECLARE_DERIVABLE_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
>>> 
>>>  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-DERIVABLE-TYPE:CAPS
>>> 
>>> 
>>> It would be nice to have a QOM code generator so that we can statically
>>> declare properties & parent/child/interface relationships, but for an
>>> immediate low cost win, better macros would be very useful IMHO.
>> 
>> Volunteers?
>> 
> 
> Actually, we are not that far off from being able to use GObject
> altogether (I hacked something like that to play with), but I
> disgress...
> 
> So introducing GObject-like macros? sure!

I’m still puzzled as to why anybody would switch to something like
GObject when there is C++.

I’m serious.

> 
> There are plenty of refactoring to do. The problem when touching the
> whole code-base, imho, is review time. It may take a couple of
> hours/days to come up with a cocci/spatch, and make various patches
> here and there. But it takes often weeks and a lot of constant push to
> various folks to get all the reviews (as seens by the qdev prop-ptr
> series earlier for example). How can we better address whole code-base
> changes?
> 
> 
> 
> -- 
> Marc-André Lureau
> 

Re: Integrating QOM into QAPI

[Marc-André Lureau]

Hi

On Sun, Jan 26, 2020 at 9:10 AM Christophe de Dinechin
<dinechin@redhat.com> wrote:
>
>
>
> > On 21 Jan 2020, at 16:11, Marc-André Lureau <marcandre.lureau@gmail.com> wrote:
> >
> > Hi
> >
> > On Tue, Jan 21, 2020 at 7:01 PM Markus Armbruster <armbru@redhat.com> wrote:
> >>
> >> Daniel P. Berrangé <berrange@redhat.com> writes:
> >>
> >>> On Tue, Jan 21, 2020 at 02:36:17PM +0100, Markus Armbruster wrote:
> >>>> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
> >>>>
> >>>>> Hi
> >>>>>
> >>>>> On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >>>>>>
> >>>>>> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
> >>>>>>> Stefan Hajnoczi <stefanha@gmail.com> writes:
> >>>>>>>
> >>>>>>>> On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> >>>>>>>>> Christophe de Dinechin <dinechin@redhat.com> writes:
> >>>>>>>>>>> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
> >>>>>>>>> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
> >>>>>>>>>  poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
> >>>>>>>>>  documented (in qemu-options.hx), their use with qom-set is not.
> >>>>>>>>
> >>>>>>>> I'm happy to use a different interface.
> >>>>>>>>
> >>>>>>>> Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> >>>>>>>> be a step backwards.
> >>>>>>>
> >>>>>>> No argument.
> >>>>>>>
> >>>>>>>> Maybe the QAPI code generator could map something like this:
> >>>>>>>>
> >>>>>>>>  { 'command': 'iothread-set-poll-params',
> >>>>>>>>    'data': {
> >>>>>>>>        'id': 'str',
> >>>>>>>>    '*max-ns': 'uint64',
> >>>>>>>>    '*grow': 'uint64',
> >>>>>>>>    '*shrink': 'uint64'
> >>>>>>>>    },
> >>>>>>>>    'map-to-qom-set': 'IOThread'
> >>>>>>>>  }
> >>>>>>>>
> >>>>>>>> And turn it into QOM accessors on the IOThread object.
> >>>>>>>
> >>>>>>> I think a generic "set this configuration to that value" command is just
> >>>>>>> fine.  qom-set fails on several counts, though:
> >>>>>>>
> >>>>>>> * Tolerable: qom-set is not actually generic, it applies only to QOM.
> >>>>>>>
> >>>>>>> * qom-set lets you set tons of stuff that is not meant to be changed at
> >>>>>>>  run time.  If it breaks your guest, you get to keep the pieces.
> >>>>>>>
> >>>>>>> * There is virtually no documentation on what can be set to what values,
> >>>>>>>  and their semantics.
> >>>>>>>
> >>>>>>> In its current state, QOM is a user interface superfund site.
> >>>>>>
> >>>>>> Thoughts about a solution:
> >>>>>>
> >>>>>> Static QOM properties should be declared via QAPI instead of
> >>>>>> imperatively via QOM APIs.  That way they are introspectable and type
> >>>>>> information is present in the schema.
> >>>>>>
> >>>>>> The QAPI code generator could emit a function that is callable from
> >>>>>> .class_init().  This eliminates the need to manually call
> >>>>>> object_class_property_add().
> >>>>
> >>>> We need to make up our minds what exactly we want generated.  Then we
> >>>> can design the QAPI language, and code up the generator.
> >>>>
> >>>> Skeleton QOM type, to help with the discussion:
> >>>>
> >>>>    #define TYPE_FOO "foo"
> >>>>
> >>>>    #define FOO(obj) OBJECT_CHECK(Foo, (obj), TYPE_FOO)
> >>>>    #define FOO_CLASS(klass) \
> >>>>        OBJECT_CLASS_CHECK(FooClass, (klass), TYPE_FOO)
> >>>>    #define FOO_GET_CLASS(obj) \
> >>>>        OBJECT_GET_CLASS(FooClass, (obj), TYPE_FOO)
> >>>>
> >>>>    typedef FooClass {
> >>>>        ParentClass parent_class;
> >>>>        ... // hand-written per-class state
> >>>>    }
> >>>>
> >>>>    struct Chardev {
> >>>>        ParentObject parent_obj;
> >>>>        ... // hand-written instance (per-object) state
> >>>>    };
> >>>>
> >>>>    static const TypeInfo char_type_info = {
> >>>>        .name = TYPE_FOO,
> >>>>        .parent = TYPE_OBJECT,
> >>>>        .instance_size = sizeof(Foo),
> >>>>        .instance_init = ...,                   // methods to initialize
> >>>>        .instance_post_init = ...,              // and finalize instances,
> >>>>        .instance_finalize = ...,               // all optional
> >>>>        .abstract = ...,                        // true or false (d'oh)
> >>>>        .class_size = sizeof(FooClass),
> >>>>        .class_init = ...,                      // methods to initialize
> >>>>        .class_base_init = ...,                 // classes, optional
> >>>>        .class_data = ...,                      // extra argument for them
> >>>>        .interfaces = ...
> >>>>    };
> >>>>
> >>>> There's substantial boilerplate, with plenty of hand-written code in the
> >>>> gaps.  What of the boilerplate do we plan to generate?  How do we plan
> >>>> to fill the gaps, if any?
> >>>
> >>> FWIW, even without a QOM generator, we can do waaaaaaay better on the
> >>> amount of boilerplate needed for QOM without very much work. It just
> >>> needs a few convenience macros writing.
> >>>
> >>> QOM is not GObject, but is heavily inspired by it and so looking at
> >>> GObject gives us a design pattern we can aim to match in terms of
> >>> amount of boilerplate.
> >>>
> >>> What we do manually with TypeInfo struct there has essentially always
> >>> been done by a 1 line macro in GObject:
> >>>
> >>>  G_DEFINE_TYPE(virIdentity, vir_identity, G_TYPE_OBJECT)
> >>>
> >>> If implementing interfaces, there's 1 extra line needed per interface
> >>> to associate them.
> >>>
> >>>  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DEFINE-TYPE:CAPS
> >>>
> >>>
> >>> And what we do in the header file to add the 4 or more FOO_XXX macros,
> >>> and the class struct and the object struct has recently been turned
> >>> into a 2-liner:
> >>>
> >>>  #define VIR_TYPE_IDENTITY vir_identity_get_type()
> >>>  G_DECLARE_FINAL_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
> >>>
> >>>  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-FINAL-TYPE:CAPS
> >>>
> >>> Or
> >>>
> >>>  #define VIR_TYPE_IDENTITY vir_identity_get_type()
> >>>  G_DECLARE_DERIVABLE_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
> >>>
> >>>  https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-DERIVABLE-TYPE:CAPS
> >>>
> >>>
> >>> It would be nice to have a QOM code generator so that we can statically
> >>> declare properties & parent/child/interface relationships, but for an
> >>> immediate low cost win, better macros would be very useful IMHO.
> >>
> >> Volunteers?
> >>
> >
> > Actually, we are not that far off from being able to use GObject
> > altogether (I hacked something like that to play with), but I
> > disgress...
> >
> > So introducing GObject-like macros? sure!
>
> I’m still puzzled as to why anybody would switch to something like
> GObject when there is C++.

C++ is another level of complexity.

Replacing QOM with GObject would mainly bring us a more solid type
system with better tooling/features, gobject-introspection support,
and remove the burden of having our own OO from QEMU code base.

It sufficiently hard for GObject developers to allow writing GObjects
from Rust, I don't think anyone want to repeat that work for QOM/QDev.
I don't know how c++ and rust would interoperate, but that seems even
more complicated to me.


-- 
Marc-André Lureau

Re: Integrating QOM into QAPI

[Paolo Bonzini]

On 26/01/20 10:11, Marc-André Lureau wrote:
>> I’m still puzzled as to why anybody would switch to something like
>> GObject when there is C++.
> C++ is another level of complexity.
> 
> Replacing QOM with GObject would mainly bring us a more solid type
> system with better tooling/features, gobject-introspection support,
> and remove the burden of having our own OO from QEMU code base.

In fact, C++ doesn't solve any of the problems that either QOM or
GObject try to solve.  (Neither does Rust for that matter).
Nevertheless, there is no stupid question, only stupid answers, and I
think Christophe's remark is an example of a common misconception.  In
the hope of not making this a stupid answer, let my try to formulate
succinctly what I think the differences are between QOM, GObject and the
C++ object model:

- the C++ object model (at least "old-style" C++ with virtual functions
and the like) provides you with _the intersection_ of what QOM and
GObject try to solve.  This is what Marc-André calls "OO", and it's
essentially virtual functions and dynamic casts.  It's a relatively
small part of both QOM and GObject, and unfortunately a wheel that
almost every large C program ends up reinventing.

- Marc-André also described above what GObject provides: a fully
introspectable type system and the tools so that _libraries_ can define
_types that will be used from multiple programming languages_.

- QOM also provides a fully introspectable type system, but with a
different focus: it's so that _objects_ can expose _properties that will
be accessed from multiple channels_.


Everything else in both GObject and QOM follows from this core purpose,
and the differences between the two follow from the differences.  For
example:

- GObject's focus on multiple programming languages:
gobject-introspection, GClosure, support for non-object types (scalar
and GBoxed)

- QOM's focus on objects: dynamic properties, object tree, all types are
classes

- QOM's focus on properties: no introspection of methods

- QOM's support for multiple channels: visitors

Paolo

Re: Integrating QOM into QAPI

[Christophe de Dinechin]

> On 26 Jan 2020, at 17:47, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 26/01/20 10:11, Marc-André Lureau wrote:
>>> I’m still puzzled as to why anybody would switch to something like
>>> GObject when there is C++.
>> C++ is another level of complexity.
>> 
>> Replacing QOM with GObject would mainly bring us a more solid type
>> system with better tooling/features, gobject-introspection support,
>> and remove the burden of having our own OO from QEMU code base.
> 
> In fact, C++ doesn't solve any of the problems that either QOM or
> GObject try to solve.  (Neither does Rust for that matter).

It does not solve all of them _easily_. I do believe that any solution
to these problems can be engineered in C++, if only because C++
is essentially a superset of C. The question is whether the result can
be made easier / safer to use and generally more elegant. I believe
that the answer is yes, see proof at end of mail.

However, before going further, glib offers way more than gobject.
So there’s that too… And I’m not saying migrating to C++ is a good
idea, I’m just trying to evaluate the various options fairly.


> Nevertheless, there is no stupid question, only stupid answers, and I
> think Christophe's remark is an example of a common misconception.  In
> the hope of not making this a stupid answer, let my try to formulate
> succinctly what I think the differences are between QOM, GObject and the
> C++ object model:

Thank you for this remarkably non-stupid answer ;-) You have really
isolated, in very clear terms, the essence of the discussion.

> 
> - the C++ object model (at least "old-style" C++ with virtual functions
> and the like) provides you with _the intersection_ of what QOM and
> GObject try to solve.  This is what Marc-André calls "OO", and it's
> essentially virtual functions and dynamic casts.  It's a relatively
> small part of both QOM and GObject, and unfortunately a wheel that
> almost every large C program ends up reinventing.

This was the part I was pointing to in my initial comment.

C++ solves that basic “OO” stuff well, and goes a little beyond QOM
or GObject in offering IMO many other benefits, e.g. wrt type safety.

> 
> - Marc-André also described above what GObject provides: a fully
> introspectable type system and the tools so that _libraries_ can define
> _types that will be used from multiple programming languages_.

This kind of things has existed for a very long time. CORBA dates back
to 1991. Also, I’m not sure how important multiple programming languages
are for the qemu use-case. I believe that what matters more is remotely
accessible objects (e.g. over a socket), which in turn makes it almost
trivial to call from another language as long as you accept some kind
of serialization / deserialization process along the way.

GObject only seems marginally better for the “in same process” use case,
in the sense that it makes “objects” that can be used from any language,
indeed, but at the cost of being somewhat foreign and weird in all languages,
including C.

Look at the GClosure marshalling, for example, and compare it with the
example I give you at end of this email, and you tell me which one looks
easier and safer to use.

> 
> - QOM also provides a fully introspectable type system, but with a
> different focus: it's so that _objects_ can expose _properties that will
> be accessed from multiple channels_.

Exposing the properties and making them introspectable are the
fundamental feature we need to discuss. So agreement here.

What you have not explained to my satisfaction
yet is how GObject is a better starting point for re-creating a new
externally-accessible API than some kind of wrapper built in C++
and taking advantage of modern C++ features.


> Everything else in both GObject and QOM follows from this core purpose,
> and the differences between the two follow from the differences.  For
> example:
> 
> - GObject's focus on multiple programming languages:
> gobject-introspection, GClosure, support for non-object types (scalar
> and GBoxed)

How much of that is actually useful to create a new usable qemu API?

> 
> - QOM's focus on objects: dynamic properties, object tree, all types are
> classes

That, to me, looks fundamental, since not having it would require
a total re-architecture of the rest of qemu.

But it looks also somewhat trivial to implement in C++.
For example, obj[“id”] could return a PropertyAccessor<T>
that lets you read or write the object knowing that it is of type
T, so you could write:

	if (my_object[“id”] < 3) // Automatically checks the type

or 

	my_object[“id”] = 42;

The latter would call PropertyAccessor<int>::operator=(int), which
in turn would check if property “id” exists in my_object, if it has type
“int”, and so on.

Implementation-wise, a simple std::map of BaseProperty pointers,
where each would be an instance of some

	template<class T>
	class Property : public BaseProperty
	{
		operator T(); // get
		T& operator=(const T&); // set
	};

	template<class T, class Index>
	class PropertyAccessor
	{
		PropertyAccessor(PropertyOwner<T> &owner, Index &index) : owner(owner), index(index) {}
		operator T() { return owner[index]; }
		T& operator =(const T&val) { return owner[index] = val; }
	};

> - QOM's focus on properties: no introspection of methods

OK. So I’m giving a bad example below, because I’m introspecting
a method, but… ;-)

> - QOM's support for multiple channels: visitors

How different are visitors from iterators in C++? Sorry, showing my
ignorance here… 


I do not fully understand all the subtleties of QOM, nor all its use cases.
However, my own experience with “modern” C++ shows that it’s pretty
straightforward to get it to do extremely interesting introspective work.

As an illustration, which happens to be directly related to the problem
of interconnecting with another language, I would like to share the following
tidbit of code (https://github.com/c3d/xl/blob/master/src/runtime.cpp#L73).

Tree *  xl_evaluate(Scope *scope, Tree *tree)
// ----------------------------------------------------------------------------
//   Dispatch evaluation to the main entry point
// ----------------------------------------------------------------------------
{
    return MAIN->Evaluate(scope, tree);
}
XL_NATIVE(evaluate);

With this little XL_NATIVE macro, the compiler automatically makes the
function xl_evaluate available to the XL language as “evaluate”, with
the correct parameter sets. In particular:

- It inserts an object representing xl_evaluate in a list of all native functions
- It generates a function that generates the source-code interface,
  which in XL would look like `evaluate A:scope, B:tree as tree`,
  see record #28601 in the output below. Note that this includes 
  all the required type conversions (e.g. from `Tree *` to `tree`).
- It generates a function that generates a machine-level prototype
  using LLVM, i.e. it automatically all the LLVM calls required to declare
    Tree *xl_evaluate(Scope *scope, Tree *tree);
- It generates a function that acts as a wrapper around it and calls
  the prototype above, doing the necessary type conversions.

Here is an example of output at run-time:
> xl -tnative -O3
[…]
[26809 0.007432] native: Entering prototype for same_text shape 0x111a3f540 [same_text A:tree, B:text as boolean]
[27021 0.007512] native: Shape 1 infix 0x111615d28 [tree] : 0x111ac8838 [A] = 0x111a94af0 [A:tree]
[27213 0.007630] native: Return shape (one) 0x111615d28 [tree]
[27274 0.007651] native: Native shape 1 0x111a94aa8 [A:tree as tree] return 0x111615d28 [tree]
[27447 0.007708] native: Entering prototype for stack_overflow shape 0x111a3f510 [stack_overflow A:tree as tree]
[27595 0.007784] native: Shape 1 infix 0x111615c10 [scope] : 0x111ac87c8 [A] = 0x111a94a60 [A:scope]
[27795 0.007844] native: Shape 2 infix 0x111615d28 [tree] : 0x111ac8790 [B] = 0x111a94a18 [B:tree]
[27991 0.007897] native: Shape 2 (...) 0x111a94a60 [A:scope],0x111a94a18 [B:tree] = 0x111a949d0 [A:scope, B:tree]
[28303 0.007985] native: Return shape (...) 0x111615d28 [tree]
[28364 0.008014] native: Native shape 2 0x111a94988 [A:scope, B:tree as tree] return 0x111615d28 [tree]
[28601 0.008077] native: Entering prototype for evaluate shape 0x111a3f4e0 [evaluate A:scope, B:tree as tree]
[…]

What I believe makes my point is that this is achieved with 476 lines of C++
(admittedly not the simplest C++ on earth, but still not super-complex either)
If you are curious, this is done by https://github.com/c3d/xl/blob/master/include/native.h.

So in 500 lines of C++ code, we have something that automatically turns any C function into
a wrapper object that:
- Gives you introspected type information about the parameters and return type of the function
- Gives you directly-accessible methods that operate on that function in a type-safe way

The same approach could of course offer serialization, construction of objects, etc.

So to me, this proves that introspecting an API in C++ is not just “easy”, it is highly usable.


Cheers,
Christophe

PS: Again, I’m not necessarily advocating we start writing any C++ code.
I’m just trying to understand the value proposition of GObject, QOM, etc,
in order to make a more informed decision before doing anything.

> 
> Paolo
> 

Re: Integrating QOM into QAPI

[Christophe de Dinechin]

> On 26 Jan 2020, at 10:11, Marc-André Lureau <marcandre.lureau@gmail.com> wrote:
> 
> Hi
> 
> On Sun, Jan 26, 2020 at 9:10 AM Christophe de Dinechin
> <dinechin@redhat.com> wrote:
>> 
>> 
>> 
>>> On 21 Jan 2020, at 16:11, Marc-André Lureau <marcandre.lureau@gmail.com> wrote:
>>> 
>>> Hi
>>> 
>>> On Tue, Jan 21, 2020 at 7:01 PM Markus Armbruster <armbru@redhat.com> wrote:
>>>> 
>>>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>>> 
>>>>> On Tue, Jan 21, 2020 at 02:36:17PM +0100, Markus Armbruster wrote:
>>>>>> Marc-André Lureau <marcandre.lureau@gmail.com> writes:
>>>>>> 
>>>>>>> Hi
>>>>>>> 
>>>>>>> On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>>>>>> 
>>>>>>>> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
>>>>>>>>> Stefan Hajnoczi <stefanha@gmail.com> writes:
>>>>>>>>> 
>>>>>>>>>> On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
>>>>>>>>>>> Christophe de Dinechin <dinechin@redhat.com> writes:
>>>>>>>>>>>>> On 15 Jan 2020, at 10:20, Markus Armbruster <armbru@redhat.com> wrote:
>>>>>>>>>>> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>>>>>>>>>>> poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>>>>>>>>>>> documented (in qemu-options.hx), their use with qom-set is not.
>>>>>>>>>> 
>>>>>>>>>> I'm happy to use a different interface.
>>>>>>>>>> 
>>>>>>>>>> Writing a boilerplate "iothread-set-poll-params" QMP command in C would
>>>>>>>>>> be a step backwards.
>>>>>>>>> 
>>>>>>>>> No argument.
>>>>>>>>> 
>>>>>>>>>> Maybe the QAPI code generator could map something like this:
>>>>>>>>>> 
>>>>>>>>>> { 'command': 'iothread-set-poll-params',
>>>>>>>>>>   'data': {
>>>>>>>>>>       'id': 'str',
>>>>>>>>>>   '*max-ns': 'uint64',
>>>>>>>>>>   '*grow': 'uint64',
>>>>>>>>>>   '*shrink': 'uint64'
>>>>>>>>>>   },
>>>>>>>>>>   'map-to-qom-set': 'IOThread'
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> And turn it into QOM accessors on the IOThread object.
>>>>>>>>> 
>>>>>>>>> I think a generic "set this configuration to that value" command is just
>>>>>>>>> fine.  qom-set fails on several counts, though:
>>>>>>>>> 
>>>>>>>>> * Tolerable: qom-set is not actually generic, it applies only to QOM.
>>>>>>>>> 
>>>>>>>>> * qom-set lets you set tons of stuff that is not meant to be changed at
>>>>>>>>> run time.  If it breaks your guest, you get to keep the pieces.
>>>>>>>>> 
>>>>>>>>> * There is virtually no documentation on what can be set to what values,
>>>>>>>>> and their semantics.
>>>>>>>>> 
>>>>>>>>> In its current state, QOM is a user interface superfund site.
>>>>>>>> 
>>>>>>>> Thoughts about a solution:
>>>>>>>> 
>>>>>>>> Static QOM properties should be declared via QAPI instead of
>>>>>>>> imperatively via QOM APIs.  That way they are introspectable and type
>>>>>>>> information is present in the schema.
>>>>>>>> 
>>>>>>>> The QAPI code generator could emit a function that is callable from
>>>>>>>> .class_init().  This eliminates the need to manually call
>>>>>>>> object_class_property_add().
>>>>>> 
>>>>>> We need to make up our minds what exactly we want generated.  Then we
>>>>>> can design the QAPI language, and code up the generator.
>>>>>> 
>>>>>> Skeleton QOM type, to help with the discussion:
>>>>>> 
>>>>>>   #define TYPE_FOO "foo"
>>>>>> 
>>>>>>   #define FOO(obj) OBJECT_CHECK(Foo, (obj), TYPE_FOO)
>>>>>>   #define FOO_CLASS(klass) \
>>>>>>       OBJECT_CLASS_CHECK(FooClass, (klass), TYPE_FOO)
>>>>>>   #define FOO_GET_CLASS(obj) \
>>>>>>       OBJECT_GET_CLASS(FooClass, (obj), TYPE_FOO)
>>>>>> 
>>>>>>   typedef FooClass {
>>>>>>       ParentClass parent_class;
>>>>>>       ... // hand-written per-class state
>>>>>>   }
>>>>>> 
>>>>>>   struct Chardev {
>>>>>>       ParentObject parent_obj;
>>>>>>       ... // hand-written instance (per-object) state
>>>>>>   };
>>>>>> 
>>>>>>   static const TypeInfo char_type_info = {
>>>>>>       .name = TYPE_FOO,
>>>>>>       .parent = TYPE_OBJECT,
>>>>>>       .instance_size = sizeof(Foo),
>>>>>>       .instance_init = ...,                   // methods to initialize
>>>>>>       .instance_post_init = ...,              // and finalize instances,
>>>>>>       .instance_finalize = ...,               // all optional
>>>>>>       .abstract = ...,                        // true or false (d'oh)
>>>>>>       .class_size = sizeof(FooClass),
>>>>>>       .class_init = ...,                      // methods to initialize
>>>>>>       .class_base_init = ...,                 // classes, optional
>>>>>>       .class_data = ...,                      // extra argument for them
>>>>>>       .interfaces = ...
>>>>>>   };
>>>>>> 
>>>>>> There's substantial boilerplate, with plenty of hand-written code in the
>>>>>> gaps.  What of the boilerplate do we plan to generate?  How do we plan
>>>>>> to fill the gaps, if any?
>>>>> 
>>>>> FWIW, even without a QOM generator, we can do waaaaaaay better on the
>>>>> amount of boilerplate needed for QOM without very much work. It just
>>>>> needs a few convenience macros writing.
>>>>> 
>>>>> QOM is not GObject, but is heavily inspired by it and so looking at
>>>>> GObject gives us a design pattern we can aim to match in terms of
>>>>> amount of boilerplate.
>>>>> 
>>>>> What we do manually with TypeInfo struct there has essentially always
>>>>> been done by a 1 line macro in GObject:
>>>>> 
>>>>> G_DEFINE_TYPE(virIdentity, vir_identity, G_TYPE_OBJECT)
>>>>> 
>>>>> If implementing interfaces, there's 1 extra line needed per interface
>>>>> to associate them.
>>>>> 
>>>>> https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DEFINE-TYPE:CAPS
>>>>> 
>>>>> 
>>>>> And what we do in the header file to add the 4 or more FOO_XXX macros,
>>>>> and the class struct and the object struct has recently been turned
>>>>> into a 2-liner:
>>>>> 
>>>>> #define VIR_TYPE_IDENTITY vir_identity_get_type()
>>>>> G_DECLARE_FINAL_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
>>>>> 
>>>>> https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-FINAL-TYPE:CAPS
>>>>> 
>>>>> Or
>>>>> 
>>>>> #define VIR_TYPE_IDENTITY vir_identity_get_type()
>>>>> G_DECLARE_DERIVABLE_TYPE(virIdentity, vir_identity, VIR, IDENTITY, GObject);
>>>>> 
>>>>> https://developer.gnome.org/gobject/stable/gobject-Type-Information.html#G-DECLARE-DERIVABLE-TYPE:CAPS
>>>>> 
>>>>> 
>>>>> It would be nice to have a QOM code generator so that we can statically
>>>>> declare properties & parent/child/interface relationships, but for an
>>>>> immediate low cost win, better macros would be very useful IMHO.
>>>> 
>>>> Volunteers?
>>>> 
>>> 
>>> Actually, we are not that far off from being able to use GObject
>>> altogether (I hacked something like that to play with), but I
>>> disgress...
>>> 
>>> So introducing GObject-like macros? sure!
>> 
>> I’m still puzzled as to why anybody would switch to something like
>> GObject when there is C++.
> 
> C++ is another level of complexity.

Migrating an existing code base to C++ object, I agree.

In the absolute, however, the boilerplate code is clearly
another level of complexity in qom or gobject. And I don’t
think it’s particularly hard to add dynamic properties to a
Object class either, including in a form that would call the
existing C functions.

> 
> Replacing QOM with GObject would mainly bring us a more solid type
> system with better tooling/features, gobject-introspection support,
> and remove the burden of having our own OO from QEMU code base.

Agree with that.

> 
> It sufficiently hard for GObject developers to allow writing GObjects
> from Rust, I don't think anyone want to repeat that work for QOM/QDev.
> I don't know how c++ and rust would interoperate, but that seems even
> more complicated to me.

Good point. But C++ has tools to solve that. Please see my response
to Paolo.

> 
> 
> -- 
> Marc-André Lureau

Re: Integrating QOM into QAPI

[Peter Maydell]

On Sun, 26 Jan 2020 at 08:10, Christophe de Dinechin
<dinechin@redhat.com> wrote:
> I’m still puzzled as to why anybody would switch to something like
> GObject when there is C++.

I'm fairly strongly against using C++. C++'s language design
is an "everything including the kitchen sink, lots of "this
is here for back compat but it's a bear trap", lots of new
stuff arriving all the time. It's just too big to keep in
your head all at once. C has its faults, absolutely, but at
least it tries to be a reasonably sized vaguely coherent
language.

You'd have more luck persuading me we should move to Rust:
at least then we'd get some clear benefits (no more buffer
overrun security bugs) for the upheaval :-)

thanks
-- PMM

Re: Integrating QOM into QAPI

[Christophe de Dinechin]

> On 26 Jan 2020, at 16:04, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On Sun, 26 Jan 2020 at 08:10, Christophe de Dinechin
> <dinechin@redhat.com> wrote:
>> I’m still puzzled as to why anybody would switch to something like
>> GObject when there is C++.
> 
> I'm fairly strongly against using C++.

Just to be clear, so am I ;-)

> C++'s language design
> is an "everything including the kitchen sink, lots of "this
> is here for back compat but it's a bear trap", lots of new
> stuff arriving all the time.

Actually, the new stuff is not that bad, overall.

I do agree C++ is an overly complicated language, and that in
practice, there is zero chance of qemu moving to it. But that does
not invalidate my point that creating a class in C++ is easier
than creating a class in any C-based macro-heavy reinvention
of basic OO concepts.

(I write this after having read Paolo’s response, which points
out IMO better reasons for GObject, which I will discuss there).

> It's just too big to keep in
> your head all at once. C has its faults, absolutely, but at
> least it tries to be a reasonably sized vaguely coherent
> language.
> 
> You'd have more luck persuading me we should move to Rust:
> at least then we'd get some clear benefits (no more buffer
> overrun security bugs) for the upheaval :-)

This is largely a myth as soon as you need to do “your own stuff”.
Example: CVE-2019-18960, https://seclists.org/oss-sec/2019/q4/141.

> 
> thanks
> -- PMM
> 

Re: Integrating QOM into QAPI

[Markus Armbruster]

Christophe de Dinechin <dinechin@redhat.com> writes:

>> On 26 Jan 2020, at 16:04, Peter Maydell <peter.maydell@linaro.org> wrote:
>> 
>> On Sun, 26 Jan 2020 at 08:10, Christophe de Dinechin
>> <dinechin@redhat.com> wrote:
[...]
>> You'd have more luck persuading me we should move to Rust:
>> at least then we'd get some clear benefits (no more buffer
>> overrun security bugs) for the upheaval :-)
>
> This is largely a myth as soon as you need to do “your own stuff”.
> Example: CVE-2019-18960, https://seclists.org/oss-sec/2019/q4/141.

I think "largely a myth" is too harsh.  Yes, it's not a silver bullet to
insta-slay all memory and concurrency safety vampires.  It does provide
useful guarantees, though.  How useful exactly in practice time will
tell.

Re: Integrating QOM into QAPI

[Daniel P. Berrangé]

On Mon, Jan 27, 2020 at 08:05:36PM +0100, Christophe de Dinechin wrote:
> 
> 
> > On 26 Jan 2020, at 16:04, Peter Maydell <peter.maydell@linaro.org> wrote:
> > 
> > On Sun, 26 Jan 2020 at 08:10, Christophe de Dinechin
> > <dinechin@redhat.com> wrote:
> >> I’m still puzzled as to why anybody would switch to something like
> >> GObject when there is C++.
> > 
> > I'm fairly strongly against using C++.
> 
> Just to be clear, so am I ;-)
> 
> > C++'s language design
> > is an "everything including the kitchen sink, lots of "this
> > is here for back compat but it's a bear trap", lots of new
> > stuff arriving all the time.
> 
> Actually, the new stuff is not that bad, overall.
> 
> I do agree C++ is an overly complicated language, and that in
> practice, there is zero chance of qemu moving to it. But that does
> not invalidate my point that creating a class in C++ is easier
> than creating a class in any C-based macro-heavy reinvention
> of basic OO concepts.
> 
> (I write this after having read Paolo’s response, which points
> out IMO better reasons for GObject, which I will discuss there).
> 
> > It's just too big to keep in
> > your head all at once. C has its faults, absolutely, but at
> > least it tries to be a reasonably sized vaguely coherent
> > language.
> > 
> > You'd have more luck persuading me we should move to Rust:
> > at least then we'd get some clear benefits (no more buffer
> > overrun security bugs) for the upheaval :-)
> 
> This is largely a myth as soon as you need to do “your own stuff”.
> Example: CVE-2019-18960, https://seclists.org/oss-sec/2019/q4/141.

Calling it a "myth" from from that one data point is not really credible.

No language is perfect & such that it can eliminate all possible CVEs.
Rust *can*, however, eliminate a very large set of bugs which lead to 
memory corruption in unchecked languages like C/C++.  You'll still have
CVEs to deal with, but they'll be different classes of bugs, or rare
edge cases like the one you show above.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Integrating QOM into QAPI

[Christophe de Dinechin]

> On 28 Jan 2020, at 11:03, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Mon, Jan 27, 2020 at 08:05:36PM +0100, Christophe de Dinechin wrote:
>> 
>> 
>>> On 26 Jan 2020, at 16:04, Peter Maydell <peter.maydell@linaro.org> wrote:
>>> 
>>> On Sun, 26 Jan 2020 at 08:10, Christophe de Dinechin
>>> <dinechin@redhat.com> wrote:
>>>> I’m still puzzled as to why anybody would switch to something like
>>>> GObject when there is C++.
>>> 
>>> I'm fairly strongly against using C++.
>> 
>> Just to be clear, so am I ;-)
>> 
>>> C++'s language design
>>> is an "everything including the kitchen sink, lots of "this
>>> is here for back compat but it's a bear trap", lots of new
>>> stuff arriving all the time.
>> 
>> Actually, the new stuff is not that bad, overall.
>> 
>> I do agree C++ is an overly complicated language, and that in
>> practice, there is zero chance of qemu moving to it. But that does
>> not invalidate my point that creating a class in C++ is easier
>> than creating a class in any C-based macro-heavy reinvention
>> of basic OO concepts.
>> 
>> (I write this after having read Paolo’s response, which points
>> out IMO better reasons for GObject, which I will discuss there).
>> 
>>> It's just too big to keep in
>>> your head all at once. C has its faults, absolutely, but at
>>> least it tries to be a reasonably sized vaguely coherent
>>> language.
>>> 
>>> You'd have more luck persuading me we should move to Rust:
>>> at least then we'd get some clear benefits (no more buffer
>>> overrun security bugs) for the upheaval :-)
>> 
>> This is largely a myth as soon as you need to do “your own stuff”.
>> Example: CVE-2019-18960, https://seclists.org/oss-sec/2019/q4/141.
> 
> Calling it a "myth" from from that one data point is not really credible.

A single failure is enough to credibly counter any “no more X” claim ;-)

Also, I carefully qualified that as for “your own stuff”. IOW, if you write
your own buffer management, Rust does not help. Well, dealing with
guest memory forces us to have our own buffer management.

> 
> No language is perfect & such that it can eliminate all possible CVEs.
> Rust *can*, however, eliminate a very large set of bugs which lead to 
> memory corruption in unchecked languages like C/C++.

The Rust hype today eerily reminds me so strongly of the C++ hype
20 years ago. C++ too was supposed to be a safer C. And indeed,
if you stick to the standard library, you _do_ eliminate a very large set
of bugs which lead to memory corruption, leaks, etc in C. So the
claim C++ made was not wrong. The claim Rust makes is not wrong
either. The false sense of safety that some people get from it is.


>  You'll still have
> CVEs to deal with, but they'll be different classes of bugs, or rare
> edge cases like the one you show above.

How exactly was it an “edge case”? From what I saw, it was the most
basic kind of range check, exactly the kind that you see in 99.9% of all
range checks.

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Making QEMU easier for management tools and applications

[Marc-André Lureau]

Hi

On Wed, Jan 15, 2020 at 1:21 PM Markus Armbruster <armbru@redhat.com> wrote:
>
> Christophe de Dinechin <dinechin@redhat.com> writes:
>
> >> To make this worthwhile, we'd have to replace dynamic QOM properties by
> >> static ones when possible.  Monumental task.
> >
> > I’m sure you are right, but it’s hard for me to evaluate, given how
> > many ways there are to access an object. Naively, grepping for
> > set_prop and for new_with_prop does not give me that many hits.
>
> Look for object_property_add*().  Some 450 hits.

fwiw, I have started tackling that.

Easy first step is to move all QDev properties to class properties,
and this is done in :
https://patchew.org/QEMU/20200110153039.1379601-1-marcandre.lureau@redhat.com/

Moving from instance to class properties is straightforward many times
(when the property is unconditonally added in instance init for
example). There are a few complicated cases though.

To me, the most problematic is reviewer-time and willingness to do
such low-benefits changes.

> Basing the QAPI language on JSON was a poor choice.  Not sure that's
> fixable at a reasonable cost.

Translating it to another language should be relatively easy, but to what?

-- 
Marc-André Lureau

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Marc-André Lureau <marcandre.lureau@gmail.com> writes:

> Hi
>
> On Wed, Jan 15, 2020 at 1:21 PM Markus Armbruster <armbru@redhat.com> wrote:
>>
>> Christophe de Dinechin <dinechin@redhat.com> writes:
>>
>> >> To make this worthwhile, we'd have to replace dynamic QOM properties by
>> >> static ones when possible.  Monumental task.
>> >
>> > I’m sure you are right, but it’s hard for me to evaluate, given how
>> > many ways there are to access an object. Naively, grepping for
>> > set_prop and for new_with_prop does not give me that many hits.
>>
>> Look for object_property_add*().  Some 450 hits.
>
> fwiw, I have started tackling that.

Laudable.

> Easy first step is to move all QDev properties to class properties,
> and this is done in :
> https://patchew.org/QEMU/20200110153039.1379601-1-marcandre.lureau@redhat.com/

Easy because qdev properties are declarative by design.  Imperative came
in when they got rebased onto QOM.

> Moving from instance to class properties is straightforward many times
> (when the property is unconditonally added in instance init for
> example). There are a few complicated cases though.
>
> To me, the most problematic is reviewer-time and willingness to do
> such low-benefits changes.

Understand.

>> Basing the QAPI language on JSON was a poor choice.

Aggravated by immediately extending it in ways that defeat all common
JSON tools.

>>                                                      Not sure that's
>> fixable at a reasonable cost.
>
> Translating it to another language should be relatively easy, but to what?

The QAPI language is layered on top of JSON (see
docs/devel/qapi-code-gen.txt section "Schema syntax").  Swapping out
JSON for another language that is at least as expressive is an
relatively simple change.  Obvious candidates: s-expressions, TOML,
YAML.  The latter is far too complex for my taste.

The expensive part is dealing with the conflicts during the transition,
and having everybody relearn QAPI schema syntax.

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 9278 bytes --]

Il mer 15 gen 2020, 10:21 Markus Armbruster <armbru@redhat.com> ha scritto:

> > We don’t want the QAPI to let arbitrary fields of a QOM object
> > be modified, do we?
>
> We already do: QMP command qom-set.  If it breaks your guest, you get to
> keep the pieces.
>

That's not true. We chose not to make that a "recommended" interface, and
instead we add new commands. However that's mostly to avoid tying our hands
and not making too much of QOM part of the API. But I would be very
surprised if a guest could be broken with qom-set.

This was definitely not the case when QOM was introduced in a half-baked
state, but let's not indulge in self-flagellation more than it's actually
necessary.



> > As for the “public” aspects of a QOM object, that is static if it
> > comes from QAPI, so why couldn’t we define it there?
>
> QAPI specifies a certain kind of data types.
>
> QOM "specifies" data types as an imperative program for creating them.
> Maximally flexible, and fundamentally at odds with static analysis.
> I've hated this since day one.
>
> There's no hard reason why QOM could not specify static aspects
> declaratively.  It just doesn't, and changing it now would be a
> monumental task.
>
> The imperative program mostly creates identical data types every time.
> In other words, the data types are static.  Two problems.  One,
> converting such a program to an equivalent declaration takes manual
> labor.  Two, there are exceptions, and identifying them is more manual
> labor.
>
> >>> the same way, QAPI can't say anything about the structure of a QOM
> >>> object, and I think that's a problem.
> >>>
> >>>>> - which, as far as I
> >>>>> understand, is mainly because QOM properties aren't necessarily
> static,
> >>>>> so we can't provide a statically defined interface for them. Probably
> >>>>> solvable in QEMU, but not without a major effort.
> >>>>
> >>>> Or maybe extend the language so that it’s internal semantics
> >>>> knows about this aspect of QOM?
> >>>
> >>> My point is that for example you can't generate a C struct (which is
> >>> statically defined) from something that has a dynamic structure. The
> >>> best you could do is fall back to key=value even in the C source, both
> >>> key and value being strings. But then you still have to parse these
> >>> strings manually, so it doesn't actually help much compared to a state
> >>> without C bindings.
> >
> > I really don’t understand that point. To me, all these operations
> > seem relatively simple to generate.
>
> Yes, if the QAPI fairy gives us a declarative specification equivalent
> to the existing imperative one, then replacing the hand-written
> imperative code by code generated from the declarative specification
> would be relatively simple.
>
> > I think that what confuses me is when you write “something that has a
> > dynamic structure”. I understand that as referring to the structure
> > defined in the schema. But the schema itself is static. So you can
> > generate static code from it, and it’s already done today.
>
> QAPI data types are static.
>
> QOM types are not.  They're effectively almost static in practice.
>
> > Another hypothesis on my side is that we don’t want, ever, to
> > have the API provide for example the option to create its own
> > arbitrary QOM classes, or to tag arbitrary properties to an object,
> > where by “arbitrary” I mean not explicitly mentioned somewhere in
> > something like the schema.
> >
> > So I suspect you are talking about something else.
>
> Kevin's talking about the imperative code creating different QOM
> properties depending on how and in what context it is run.
>
> QOM is perfectly capable of supporting a QMP command to add arbitrary
> QOM properties to an object at run time, or even add arbitrary QOM
> types, but as you wrote we don't want that.
>
> >> QOM and QAPI sabotage each other.  Ironic, considering they were dreamed
> >> up by the same guy :)
> >>
> >> QAPI is compile-time static by design.
> >>
> >> QOM is run-time dynamic by design.  Some support for static definitions
> >> has been grafted on since.
> >>
> >> We use QAPI type system escapes for QOM.  Defeats QAPI introspection and
> >> doc generation.  We provide separate QOM introspection instead, which is
> >> clumsier and less expressive.  QOM documentation doesn't exist.
> >
> > But back to the original discussion about management tools,
> > do we let upper layers tag their own arbitrary stuff in QOM objects?
>
> We don't want management applications to add QOM properties for their
> own purposes.
>
> >>> Maybe what could be done is covering at least the static properties and
> >>> then having key=value for the dynamic part (which should be the
> >>> exception).
> >>
> >> To make this worthwhile, we'd have to replace dynamic QOM properties by
> >> static ones when possible.  Monumental task.
> >
> > I’m sure you are right, but it’s hard for me to evaluate, given how
> > many ways there are to access an object. Naively, grepping for
> > set_prop and for new_with_prop does not give me that many hits.
>
> Look for object_property_add*().  Some 450 hits.
>
> See also object_class_property_add(), grafted on in
>
> commit 16bf7f522a2ff68993f80631ed86254c71eaf5d4
> Author: Daniel P. Berrange <berrange@redhat.com>
> Date:   Tue Oct 13 13:37:46 2015 +0100
>
>     qom: Allow properties to be registered against classes
>
>     When there are many instances of a given class, registering
>     properties against the instance is wasteful of resources. The
>     majority of objects have a statically defined list of possible
>     properties, so most of the properties are easily registerable
>     against the class. Only those properties which are conditionally
>     registered at runtime need be recorded against the klass.
>
>     Registering properties against classes also makes it possible
>     to provide static introspection of QOM - currently introspection
>     is only possible after creating an instance of a class, which
>     severely limits its usefulness.
>
>     This impl only supports simple scalar properties. It does not
>     attempt to allow child object / link object properties against
>     the class. There are ways to support those too, but it would
>     make this patch more complicated, so it is left as an exercise
>     for the future.
>
>     There is no equivalent to object_property_del() provided, since
>     classes must be immutable once they are defined.
>
>     Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
>     Signed-off-by: Andreas Färber <afaerber@suse.de>
>
> >>>>>> - Relations, e.g. how we represent “contains”, “derives from”,
> “needs”,
> >>>>>> “one of”, “one or several”, “attaches to”…
> >>>>>> - States, e.g. how do we represent the machine configuration,
> >>>>>> or the desired new disk setting
> >>>>>> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
> >>>>>> “start”, “notify”, etc. and how we describe the kind of input they
> need.
> >>>>>> - Possibly more subtle things like support for transactions,
> commit/rollback,
> >>>>>> i.e. “I want to add connect a virtual nic to some host vf, but if
> anything
> >>>>>> along the way fails, I’d like all the cleanup to happen
> automatically)
> >>>>>
> >>>>> This sounds like a different approach from our current QAPI command
> set
> >>>>
> >>>> Well, except for purposefully trying to use a different wording to
> avoid
> >>>> the risk of getting your mind stuck in one of the particular existing
> >>>> meta-languages in QEMU, the approach is not very different.
> >>>
> >>> I didn't necessarily mean relations/state/verbs, which obviously exist,
> >>> but the examples that seemed to express things in a deliberately
> >>> different way from what we have today.
> >>>
> >>>> - Transactions do not exist today that I know of, although we see
> >>>> signs of them in discussions about the fact that this options destroys
> >>>> that back end but that option does not.
> >>>
> >>> We have a 'transaction' QMP command, but they are not an integral part
> >>> of the language. Considering the trouble to implement transactional
> >>> commands, I suppose we don't even want it to be a fundamental part of
> >>> the language.
> >>>
> >>>>> Does it actually provide more functionality, though?
> >>>>
> >>>> It’s not intended to provide more, but to require less to do the same
> thing.
> >>>
> >>> Though we always need to keep in mind that if requiring less for future
> >>> additions requires a huge effort now, the investment may pay off only
> in
> >>> a very distant future (if the abstraction we build even survives until
> >>> then).
> >>
> >> Worse is better.
> >>
> >> http://dreamsongs.com/RiseOfWorseIsBetter.html
> >
> > You know that I positively hate this ;-)
>
> It's been a tough lesson for me, too.
>
> > Well, I guess we can expand the schema. #ILoveJSON.
>
> Basing the QAPI language on JSON was a poor choice.  Not sure that's
> fixable at a reasonable cost.
>
>

[-- Attachment #2: Type: text/html, Size: 11624 bytes --]

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Il mer 15 gen 2020, 10:21 Markus Armbruster <armbru@redhat.com> ha scritto:
>
>> > We don’t want the QAPI to let arbitrary fields of a QOM object
>> > be modified, do we?
>>
>> We already do: QMP command qom-set.  If it breaks your guest, you get to
>> keep the pieces.
>>
>
> That's not true. We chose not to make that a "recommended" interface, and
> instead we add new commands. However that's mostly to avoid tying our hands
> and not making too much of QOM part of the API. But I would be very
> surprised if a guest could be broken with qom-set.
>
> This was definitely not the case when QOM was introduced in a half-baked
> state, but let's not indulge in self-flagellation more than it's actually
> necessary.

Nobody thinks of qom-set when adding a QOM property.

qom-set runs a setter function that can do anything.  Is that safe at
any point of time?  Depends.

Fortunately, most setters merely write to some object instance variable.

Often, the object's code reads the instance variable only in realize(),
and never writes to it.  qom-set is safe then, but not terribly useful.

More interesting are the ones that are read and written throughout the
object's life cycle.  Is qom-set safe at any point of time?  Depends.

Your belief qom-set can't break a guest in practice is reassuring.

However, the point I was trying to make with "you get to keep the
pieces" is precisely that we're "not making too much of QOM part of the
API".

An additional point I'd like to repeat now: which parts of QOM are is
pretty much anybody's guess.

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2552 bytes --]

On Tue, Jan 07, 2020 at 06:11:13PM +0100, Christophe de Dinechin wrote:
> > On 20 Dec 2019, at 17:13, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> So I think that Daniel is right. We may need at some point to start
> a NEMU-style offshoot that does not attempt to be compatible,
> but explores describing an increasing surface of the API using a
> new meta-language from which we can generate, in a consistent
> way, at least:
> 
> - C bindings
> - Command-line options
> - Shell bindings (or “HMP”)
> - JSON schema or qom description
> - Bindings in other languages (Rust, Go, Python)
> - Networked versions of the API (socket, REST)
> - Client-side code e.g. for libvirt.
> - Serialization / deserialization, e.g. for configuration files
> - Documentation, including man page and API docs
> - Command-line help
> 
> At the most fundamental level, I think we need to describe:
> 
> - Values, e.g. how we represent names, sizes, paths, etc, possibly
> with some user-friendly aspects, e.g. path shortcuts, memory units,
> spelling shortcuts (e.g. being able to consistently say -blo for -blockdev
> if that’s the shortest option that matches)
> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
> “one of”, “one or several”, “attaches to”…
> - States, e.g. how do we represent the machine configuration,
> or the desired new disk setting
> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
> “start”, “notify”, etc. and how we describe the kind of input they need.
> - Possibly more subtle things like support for transactions, commit/rollback,
> i.e. “I want to add connect a virtual nic to some host vf, but if anything
> along the way fails, I’d like all the cleanup to happen automatically)

Extending QAPI to achieve these things is a possibility.

If we afford ourselves the luxury of breaking backwards compatibility
then I would instead use the opportunity to eliminate complexity:
1. Get rid of the CLI
2. Get rid of HMP
3. No per-command C bindings, just a qmp_call() API
4. No configuration file, just a sequence of QMP commands

The new QEMU would be very different and solely focussed on QMP (or a
standards-based RPC system).

It's not very fun working on projects that have a lot of custom
infrastructure.  Making a one-time change requires a lot of learning
weird infrastructure that you won't use often.  We already have too much
of this and it slows down QEMU development.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Summary of Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

This thread has accumulated more than a hundred messages over seven
weeks.  We need a summary.

The conversation has gone in several directions.  In this message, I'll
cover just the one I consider most important: machine-friendly initial
configuration.  I'll do the rest later, if I have stamina left.

= QMP is fine for machines, CLI is not =

We have two major external host interfaces: the CLI and QMP.  Compare:

QMP
* Purpose: control at run-time
* Commands with argument and return value
* Events with argument value
* Simple type system
* Based on QAPI
* Regular, well-defined syntax: QAPI schema on top of JSON
* Documentation is required; actual documentation is of mixed quality
* Introspectable, except for a few mostly QOM-related holes where we
  bypass QAPI
* Machine-friendly, complemented by human-only HMP

CLI
* Purpose: initial configuration
* Options with argument
* Based on a crazy zoo of QemuOpts (with and without dotted keys), QAPI
  (only a few recent options), ad hoc parsers
* Option argument syntax is mostly (variations of) key=value,...
* QemuOpts' "type system" is "list of key-value pairs, where value can
  be string, bool, (unsigned) integer, or size", optionally restricted
  to known keys with known value types.
* QAPI type system is the same as in QMP
* Introspection is completely inadequate: misses options, incorrect
  option names, misses option arguments partly or completely
* Configuration files with INI-like syntax, completely inadequate: can't
  do most options
* Not machine-friendly
* Maintaining it is a pain, evolving it is worse

The inadequacy of the CLI has become a serious issue.

= Ways to provide machine-friendly initial configuration =

Two ways to provide machine-friendly initial configuration on par with
QMP have been proposed:

1. Extend QMP

   Machines use the CLI only to configure a QMP socket.  The remainder
   of the CLI becomes human-only, with much relaxed compatibility rules.

2. QAPIfy the CLI

   Provide a machine-friendly CLI based on QAPI and JSON.  The current
   CLI becomes human-only, with much relaxed compatibility rules.

   Aside: I looked into cleaning up the human-only CLI at the same time,
   but the need to maintain compatibility until the transition to the
   machine-friendly CLI is complete makes this hard.  It needs to be
   cleaned up, though.  More on that below.

To extend QMP, we wrap QMP commands around the internal initial
configuration interfaces.  These QMP commands take arguments, but don't
return anything.  Many of them will only make sense during initial
configuration.  We'll want to express that in the schema, and enforce it
in the QMP core.  Others will behave differently, e.g. cold plug during
initial configuration, hot plug once the guest runs.

Configuration files are just QMP commands from the initial configuration
subset read from a file instead of a chardev.  JSON is a poor choice for
configuration files, and QMP's verbosity makes it poorer.  We'll want
more suitable concrete syntax for configuration files.

To improve the CLI, we wrap QAPI-based CLI options around the internal
initial configuration interfaces.  We'll want some infrastructure to
generate CLI option boilerplate, just like we generate QMP command
boilerplate.

Configuration files are just CLI options read from a file instead of
argv[].  Again, we'll want a more suitable concrete syntax there.

With an improved CLI, I'd expect machines to use configuration files so
they don't have to mess with shell quoting.  With extended QMP, they'd
perhaps rather reuse their existing QMP code to send the configuration
down a socket.  Less efficient, but I doubt it'll matter.

In both cases, the internal configuration interfaces need to be
converted from QemuOpts to QAPI types.

The one clear advantage of extending QMP is the ability to mix query
commands with initial configuration.

This is also a clear disadvantage: we need to make it work.  Existing
QMP commands may rely on initial configuration to be complete.  We may
have to allow only QMP commands we carefully checked.

How important is the advantage?

= Cleaning up the human-only CLI =

In both cases, the existing CLI remains in a human-only role.  That's a
truckload of messy code to maintain.  Not good.  Can we replace it by
thin wrappers around the machine-friendly alternative, ideally
incrementally?  Can we generate a useful part of these wrappers?

If we QAPIfy the CLI, the QAPI generator already knows the CLI, and only
has to be taught the general human-friendly key=value,... syntax, plus a
way to specify exceptions.

If we extend QMP, we'll additionally need some of the CLI QAPIfication
infrastructure for this.  Hmm.

Re: Summary of Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Markus Armbruster <armbru@redhat.com> writes:

[...]
> = Ways to provide machine-friendly initial configuration =
>
> Two ways to provide machine-friendly initial configuration on par with
> QMP have been proposed:
>
> 1. Extend QMP
>
>    Machines use the CLI only to configure a QMP socket.  The remainder
>    of the CLI becomes human-only, with much relaxed compatibility rules.
>
> 2. QAPIfy the CLI
>
>    Provide a machine-friendly CLI based on QAPI and JSON.  The current
>    CLI becomes human-only, with much relaxed compatibility rules.
>
>    Aside: I looked into cleaning up the human-only CLI at the same time,
>    but the need to maintain compatibility until the transition to the
>    machine-friendly CLI is complete makes this hard.  It needs to be
>    cleaned up, though.  More on that below.

Forgot to write down: we're talking not just about qemu-system-FOO, but
any executable with a non-trivial command line for use by machines.
This includes qemu-img, future qemu-storage-daemon, possibly qemu-nbd
and more.  Of these, qemu-storage-daemon will have a QMP monitor.  The
others don't have one so far.

[...]

Re: Summary of Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster <armbru@redhat.com> wrote:
> = Ways to provide machine-friendly initial configuration =
>
> Two ways to provide machine-friendly initial configuration on par with
> QMP have been proposed:
>
> 1. Extend QMP
>
>    Machines use the CLI only to configure a QMP socket.  The remainder
>    of the CLI becomes human-only, with much relaxed compatibility rules.
>
> 2. QAPIfy the CLI
>
>    Provide a machine-friendly CLI based on QAPI and JSON.  The current
>    CLI becomes human-only, with much relaxed compatibility rules.

Do we keep the existing CLI around in both cases?  I'm concerned that
we're still following the HMP/QMP approach, which has left QEMU with
the legacy HMP monitor that we still haven't removed.

I'm in favor of simplifying QEMU at the expense of an incompatible CLI
change in QEMU 6.0.

A project like this could prototype incompatible CLI changes in a
separate git tree.  If it achieves the desired unification (CLI, QMP,
configuration file) and simplification (less code, legacy removal)
then it can be merged for an upcoming QEMU major release.

Stefan

Re: Summary of Re: Making QEMU easier for management tools and applications

[Peter Maydell]

On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> change in QEMU 6.0.

If we want to do wholesale incompatible changes to the CLI
I think we definitely need some kind of tool where a user
can say "here's my old command line, what's the new style
equivalent?". Otherwise we're going to have a deluge of
user issues where their old working setups broke and
QEMU didn't give them any useful hints about why.

thanks
-- PMM

Re: Summary of Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Mon, Feb 10, 2020 at 11:01:48AM +0000, Peter Maydell wrote:
> On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> > change in QEMU 6.0.
> 
> If we want to do wholesale incompatible changes to the CLI
> I think we definitely need some kind of tool where a user
> can say "here's my old command line, what's the new style
> equivalent?". Otherwise we're going to have a deluge of
> user issues where their old working setups broke and
> QEMU didn't give them any useful hints about why.

There is a risk that if we promise to have a fully automated conversion
that it will be alot of work, and could force us to introduce hacks into
the new impl just to satisfy conversion.  IMHO we shouldn't be afraid of
declaring that some parts of the old syntax can NOT be directly transformed
into new syntax, simply for the sake of making a new impl more practical
to move forward with.

An alternative approach to mitigate the disruption is to *not* make any
incompatible changes to qemu-system-XXXX. Instead introduce new binaries
with the new syntax and any other architectural  changes we wish to make.
The old binaries can be deprecated but remain around for an extended
period of time, to give people and apps time to migrate. We can provide
rough guidance and perhaps partially automated conversion to help people
move, but not aim for a 100% automated conversion. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Summary of Re: Making QEMU easier for management tools and applications

[Peter Maydell]

On Mon, 10 Feb 2020 at 11:08, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Mon, Feb 10, 2020 at 11:01:48AM +0000, Peter Maydell wrote:
> > On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> > > change in QEMU 6.0.
> >
> > If we want to do wholesale incompatible changes to the CLI
> > I think we definitely need some kind of tool where a user
> > can say "here's my old command line, what's the new style
> > equivalent?". Otherwise we're going to have a deluge of
> > user issues where their old working setups broke and
> > QEMU didn't give them any useful hints about why.
>
> There is a risk that if we promise to have a fully automated conversion
> that it will be alot of work, and could force us to introduce hacks into
> the new impl just to satisfy conversion.  IMHO we shouldn't be afraid of
> declaring that some parts of the old syntax can NOT be directly transformed
> into new syntax, simply for the sake of making a new impl more practical
> to move forward with.

Agreed, but we should at least be able to handle the easy
stuff and say "this is the general kind of new option syntax
and set of options you want" for most of the rest.

> An alternative approach to mitigate the disruption is to *not* make any
> incompatible changes to qemu-system-XXXX. Instead introduce new binaries
> with the new syntax and any other architectural  changes we wish to make.
> The old binaries can be deprecated but remain around for an extended
> period of time, to give people and apps time to migrate. We can provide
> rough guidance and perhaps partially automated conversion to help people
> move, but not aim for a 100% automated conversion.

I think our history of failing to actually complete transitions
would predict that we'd end up with both the old and the new
binaries essentially forever.

thanks
-- PMM

Re: Summary of Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 10/02/20 11:56, Stefan Hajnoczi wrote:
> On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster <armbru@redhat.com> wrote:
>> = Ways to provide machine-friendly initial configuration =
>>
>> Two ways to provide machine-friendly initial configuration on par with
>> QMP have been proposed:
>>
>> 1. Extend QMP
>>
>>    Machines use the CLI only to configure a QMP socket.  The remainder
>>    of the CLI becomes human-only, with much relaxed compatibility rules.
>>
>> 2. QAPIfy the CLI
>>
>>    Provide a machine-friendly CLI based on QAPI and JSON.  The current
>>    CLI becomes human-only, with much relaxed compatibility rules.
> 
> Do we keep the existing CLI around in both cases?  I'm concerned that
> we're still following the HMP/QMP approach, which has left QEMU with
> the legacy HMP monitor that we still haven't removed.
> 
> I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> change in QEMU 6.0.
> 
> A project like this could prototype incompatible CLI changes in a
> separate git tree.  If it achieves the desired unification (CLI, QMP,
> configuration file) and simplification (less code, legacy removal)
> then it can be merged for an upcoming QEMU major release.

I think Daniel had a good point in suggesting a (possibly) throwaway
fork for either (1) or (2).  Let's see what kind of change is needed to
do 100% QMP-based configuration of guests (or at least to QMP-ify
configuration of devices and backends---things that can already have an
*-add command now); then we can figure out which subset of the current
CLI can be mapped to it.

Paolo

Re: Summary of Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster <armbru@redhat.com> wrote:
>> = Ways to provide machine-friendly initial configuration =
>>
>> Two ways to provide machine-friendly initial configuration on par with
>> QMP have been proposed:
>>
>> 1. Extend QMP
>>
>>    Machines use the CLI only to configure a QMP socket.  The remainder
>>    of the CLI becomes human-only, with much relaxed compatibility rules.
>>
>> 2. QAPIfy the CLI
>>
>>    Provide a machine-friendly CLI based on QAPI and JSON.  The current
>>    CLI becomes human-only, with much relaxed compatibility rules.
>
> Do we keep the existing CLI around in both cases?  I'm concerned that
> we're still following the HMP/QMP approach, which has left QEMU with
> the legacy HMP monitor that we still haven't removed.

The "HMP is legacy" idea is relatively recent.

I think having separate interfaces for humans and machines makes sense,
we just need to give both the attention and care they need and deserve.

I think a human-friendly monitor is has its use, but it should ideally
be done differently than we do HMP now.

Likewise, human-friendly initial configuration should exist, but it
should ideally be done differently than we do HMP now.

> I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> change in QEMU 6.0.

I'm convinced the current CLI needs cleanup badly, and that means
incompatible change.  The question is how and when to change it.

Here's how I'd like us to do it:

1. Create machine-friendly initial configuration interface separate from
   the existing CLI

   Doesn't mean it cannot be a CLI.

2. Develop it step by step to feature parity with existing CLI

   If we identify misfeatures we don't want anymore, we should
   immediately deprecate them in the existing CLI instead.

2. Transition machine users to this new interface

3. Declare the existing CLI to be like HMP: for humans, may change
   incompatibly

4. Clean up existing CLI step by step to wrap around the
   machine-friendly interface

   Whatever we deprecated in step 2 goes to the bit bucket instead.

   I'm open to replacing the existing CLI by a separate wrapper process
   instead.

   Capability to translate to the machine-friendly interface is
   desirable, so human users can easily transition to the
   machine-friendly interface when they run into a need to automate.

The risk is of course that we fail at step 4 and remain stuck with the
CLI mess we've made.

> A project like this could prototype incompatible CLI changes in a
> separate git tree.  If it achieves the desired unification (CLI, QMP,
> configuration file) and simplification (less code, legacy removal)
> then it can be merged for an upcoming QEMU major release.

That's effectively a really long-lived feature branch.  Painful.  If it
it what it takes, we do it.  I hope the process I just sketched permits
more incremental development.

We can explore conflict-free in qemu-storage-daemon.

Re: Summary of Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3277 bytes --]

On Mon, Feb 10, 2020 at 05:43:13PM +0100, Markus Armbruster wrote:
> Stefan Hajnoczi <stefanha@gmail.com> writes:
> 
> > On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster <armbru@redhat.com> wrote:
> >> = Ways to provide machine-friendly initial configuration =
> >>
> >> Two ways to provide machine-friendly initial configuration on par with
> >> QMP have been proposed:
> >>
> >> 1. Extend QMP
> >>
> >>    Machines use the CLI only to configure a QMP socket.  The remainder
> >>    of the CLI becomes human-only, with much relaxed compatibility rules.
> >>
> >> 2. QAPIfy the CLI
> >>
> >>    Provide a machine-friendly CLI based on QAPI and JSON.  The current
> >>    CLI becomes human-only, with much relaxed compatibility rules.
> >
> > Do we keep the existing CLI around in both cases?  I'm concerned that
> > we're still following the HMP/QMP approach, which has left QEMU with
> > the legacy HMP monitor that we still haven't removed.
> 
> The "HMP is legacy" idea is relatively recent.
> 
> I think having separate interfaces for humans and machines makes sense,
> we just need to give both the attention and care they need and deserve.
> 
> I think a human-friendly monitor is has its use, but it should ideally
> be done differently than we do HMP now.
> 
> Likewise, human-friendly initial configuration should exist, but it
> should ideally be done differently than we do HMP now.
> 
> > I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> > change in QEMU 6.0.
> 
> I'm convinced the current CLI needs cleanup badly, and that means
> incompatible change.  The question is how and when to change it.
> 
> Here's how I'd like us to do it:
> 
> 1. Create machine-friendly initial configuration interface separate from
>    the existing CLI
> 
>    Doesn't mean it cannot be a CLI.
> 
> 2. Develop it step by step to feature parity with existing CLI
> 
>    If we identify misfeatures we don't want anymore, we should
>    immediately deprecate them in the existing CLI instead.
> 
> 2. Transition machine users to this new interface
> 
> 3. Declare the existing CLI to be like HMP: for humans, may change
>    incompatibly
> 
> 4. Clean up existing CLI step by step to wrap around the
>    machine-friendly interface
> 
>    Whatever we deprecated in step 2 goes to the bit bucket instead.
> 
>    I'm open to replacing the existing CLI by a separate wrapper process
>    instead.
> 
>    Capability to translate to the machine-friendly interface is
>    desirable, so human users can easily transition to the
>    machine-friendly interface when they run into a need to automate.
> 
> The risk is of course that we fail at step 4 and remain stuck with the
> CLI mess we've made.

Yes, QEMU does not have a good track record of successfully converting
to new APIs and then removing old code.

My worry is that this effort will result in the addition of even more
code but we'll still be stuck with the old cruft (both in the user
visible interface and in the implementation).

But we won't get anywhere if we don't try :).  This sounds like a
significant project and I wonder if others would be willing to help if
you can break down the tasks for them.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Summary of Re: Making QEMU easier for management tools and applications

[Daniel P. Berrangé]

On Wed, Feb 12, 2020 at 01:54:42PM +0000, Stefan Hajnoczi wrote:
> On Mon, Feb 10, 2020 at 05:43:13PM +0100, Markus Armbruster wrote:
> > Stefan Hajnoczi <stefanha@gmail.com> writes:
> > 
> > > On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster <armbru@redhat.com> wrote:
> > >> = Ways to provide machine-friendly initial configuration =
> > >>
> > >> Two ways to provide machine-friendly initial configuration on par with
> > >> QMP have been proposed:
> > >>
> > >> 1. Extend QMP
> > >>
> > >>    Machines use the CLI only to configure a QMP socket.  The remainder
> > >>    of the CLI becomes human-only, with much relaxed compatibility rules.
> > >>
> > >> 2. QAPIfy the CLI
> > >>
> > >>    Provide a machine-friendly CLI based on QAPI and JSON.  The current
> > >>    CLI becomes human-only, with much relaxed compatibility rules.
> > >
> > > Do we keep the existing CLI around in both cases?  I'm concerned that
> > > we're still following the HMP/QMP approach, which has left QEMU with
> > > the legacy HMP monitor that we still haven't removed.
> > 
> > The "HMP is legacy" idea is relatively recent.
> > 
> > I think having separate interfaces for humans and machines makes sense,
> > we just need to give both the attention and care they need and deserve.
> > 
> > I think a human-friendly monitor is has its use, but it should ideally
> > be done differently than we do HMP now.
> > 
> > Likewise, human-friendly initial configuration should exist, but it
> > should ideally be done differently than we do HMP now.
> > 
> > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> > > change in QEMU 6.0.
> > 
> > I'm convinced the current CLI needs cleanup badly, and that means
> > incompatible change.  The question is how and when to change it.
> > 
> > Here's how I'd like us to do it:
> > 
> > 1. Create machine-friendly initial configuration interface separate from
> >    the existing CLI
> > 
> >    Doesn't mean it cannot be a CLI.
> > 
> > 2. Develop it step by step to feature parity with existing CLI
> > 
> >    If we identify misfeatures we don't want anymore, we should
> >    immediately deprecate them in the existing CLI instead.
> > 
> > 2. Transition machine users to this new interface
> > 
> > 3. Declare the existing CLI to be like HMP: for humans, may change
> >    incompatibly
> > 
> > 4. Clean up existing CLI step by step to wrap around the
> >    machine-friendly interface
> > 
> >    Whatever we deprecated in step 2 goes to the bit bucket instead.
> > 
> >    I'm open to replacing the existing CLI by a separate wrapper process
> >    instead.
> > 
> >    Capability to translate to the machine-friendly interface is
> >    desirable, so human users can easily transition to the
> >    machine-friendly interface when they run into a need to automate.
> > 
> > The risk is of course that we fail at step 4 and remain stuck with the
> > CLI mess we've made.
> 
> Yes, QEMU does not have a good track record of successfully converting
> to new APIs and then removing old code.
> 
> My worry is that this effort will result in the addition of even more
> code but we'll still be stuck with the old cruft (both in the user
> visible interface and in the implementation).

This is why I think any new CLI ought to be done in a new binary,
not qemu-system-XXXX. I think it is an easier proposition to
sell to people that this is a clean break if we make it a new
binary. The mere fact the binary exists will make people curious
about it. If we add new stuff to existing binaries, it is
essentially invisible unless you look for it.  Separate binaries
would also make life better for documentation IMHO, as we can
clearly distinguish legacy and modern in the docs. Indeed the
new binary doc shoudl be completely separate, so when people
learn about it, they're not distracted by legacy.

This way, even if we don't delete qemu-system-XXXX for a long time,
the new binary would not be polluted by the legacy cruft, even if
it still exists in some internal places.

Ideally the goal would be that QemuOpts be entirely missing from
any code linked into the new binary. This will be challenging given
some of the places QemuOpts embeds itself. Perhaps we can split some
of the source files to isolate the QemuOpts usage. The block layer
is the biggest challenge here.

> But we won't get anywhere if we don't try :).  This sounds like a
> significant project and I wonder if others would be willing to help if
> you can break down the tasks for them.



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|