KVM call for agenda for 2020-10-06

KVM call for agenda for 2020-10-06

[Juan Quintela]

Hi

Please, send any topic that you are interested in covering.

At the end of Monday I will send an email with the agenda or the
cancellation of the call, so hurry up.


For this call, we have agenda!!

John Snow wants to talk about his new (and excting) developments with
x-configure.  Stay tuned.


After discussions on the QEMU Summit, we are going to have always open a
KVM call where you can add topics.

 Call details:

By popular demand, a google calendar public entry with it

  https://www.google.com/calendar/embed?src=dG9iMXRqcXAzN3Y4ZXZwNzRoMHE4a3BqcXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

(Let me know if you have any problems with the calendar entry.  I just
gave up about getting right at the same time CEST, CET, EDT and DST).

If you need phone number details,  contact me privately

Thanks, Juan.

Re: KVM call for agenda for 2020-10-06

[John Snow]

On 10/2/20 5:09 AM, Juan Quintela wrote:
> 
> 
> Hi
> 
> Please, send any topic that you are interested in covering.
> 
> At the end of Monday I will send an email with the agenda or the
> cancellation of the call, so hurry up.
> 
> 
> For this call, we have agenda!!
> 
> John Snow wants to talk about his new (and excting) developments with
> x-configure.  Stay tuned.
> 

I'm working on an email to qemu-devel now detailing some of our work 
trying to make good on a renewed effort for better APIs for QEMU.

In short, I'd like to discuss a roadmap for converting our CLI to 
something QAPI-based, and discuss ways to coordinate and distribute this 
work to interested maintainers and developers.

Look out for it!

--js

> 
> After discussions on the QEMU Summit, we are going to have always open a
> KVM call where you can add topics.
> 
>   Call details:
> 
> By popular demand, a google calendar public entry with it
> 
>    https://www.google.com/calendar/embed?src=dG9iMXRqcXAzN3Y4ZXZwNzRoMHE4a3BqcXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
> 
> (Let me know if you have any problems with the calendar entry.  I just
> gave up about getting right at the same time CEST, CET, EDT and DST).
> 
> If you need phone number details,  contact me privately
> 
> Thanks, Juan.
> 

Re: KVM call for agenda for 2020-10-06

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

On Fri, Oct 02, 2020 at 11:09:44AM +0200, Juan Quintela wrote:
> For this call, we have agenda!!
> 
> John Snow wants to talk about his new (and excting) developments with
> x-configure.  Stay tuned.

Hi,
Juan will be offline tomorrow so I will moderate the call.

You can join the call from your web browser here:
https://bluejeans.com/497377100

Meeting ID: 497377100
Phone numbers: https://www.redhat.com/en/conference-numbers

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: KVM call for agenda for 2020-10-06

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

On Fri, Oct 02, 2020 at 11:09:44AM +0200, Juan Quintela wrote:
> For this call, we have agenda!!
> 
> John Snow wants to talk about his new (and excting) developments with
> x-configure.  Stay tuned.

Hi,
Juan will be offline tomorrow so I will moderate the call.

You can join the call from your web browser here:
https://bluejeans.com/497377100

Meeting ID: 497377100
Phone numbers: https://www.redhat.com/en/conference-numbers

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: KVM call for agenda for 2020-10-06

[Stefan Hajnoczi]

Thank you everyone who joined the call. The meeting notes are below.

Stefan
---
*KVM Community Call - Tue Oct 6th
*Topic: QEMU CLI QAPI conversion

    * John Snow's summary of command-line options:
https://docs.google.com/spreadsheets/d/1OJvzDwLpo2XsGytNp9Pr-vYIrb0pWrFKBNPtOyH80ew/edit?usp=sharing
    * What is the first milestone?
        * Is it QemuOpts for everything? Straight to QAPI? something else?
        * Markus: The goal is to represent the configuration interface
in QAPI types
            * Don't parse QemuOpts, go straight to QAPI
    * How do we distribute this work to multiple engineers?
        * Examples:
            * --blockdev API is used on the command-line
            * --display
            * qemu-storage-daemon command-line is largely QAPI-fied
    * Alex Bennee:
        * We should have a gold-standard reference with documentation
if we are to expect maintainers to convert their own flags
        * -> John Snow will work on this document
    * Do we have good examples of turning QemuOpts to QAPI?
        * 53 of our 93 CLI flags that take arguments are QemuOpts, so
this is a major component
        * Kevin: -monitor for the Qemu Storage Daemon, recently
    * John Snow: Final milestone might be an automated QAPI-based CLI
parser, but only once QAPI types have been defined

    * Does command-line order matter?
        * Two options: allow any order OR left-to-right ordering
        * Andrea Bolognani: Most users expect left-to-right ordering,
why allow any order?
        * Eduardo Habkost: Can we enforce left-to-right ordering or do
we need to follow the deprecation process?
        * Daniel Berrange: Solve compability by introducing new
binaries without the burden of backwards compability
            * Unclear whether we will reach consensus on this call
about this. Maybe raise it on qemu-devel. [stefanha]
            * Philippe: Easy command-line options (-drive) and
managent-friendly options (-blockdev) could be offered by different
binaries
                * Daniel Berrange: Focussing on one new binary is more
achievable

    * Board defaults, configuration file options
        * How to set properties on existing objects (e.g. board defaults)?
        * Andrea Bolognani: Perhaps represent the board defaults as a
list of ordered options, append user-provided options, and *only then*
create the object?
            * Currently the boards create QOM objects directly, they
don't involve QAPI
        * Stefan: How do QOM objects/properties fit into QAPI
CLI/configuration parsing?
            * QAPI objects are processed by functions that will create
QOM objects
        * Markus: -global is broken

    * Eduardo: Long-term goal to describe QOM properties in QAPI
        * Daniel Berrange: eliminate QOM boilerplate by describing
object properties in QAPI
        * Markus: It's hard to use QAPI because QOM properties are
dynamic and can change at runtime

    * Next steps
        * John Snow and Markus will work on reference documentation



Bluejeans Chat Log
    [ 9:02] Stefan Hajnoczi: https://etherpad.opendev.org/p/QEMUCLI
    [ 9:05] John Ferlan: @stefan - there are some people in a different room....
    [ 9:08] Daniel Berrange: if you're not talking please mute
    [ 9:24] Alex Bennée: I ran into this ordering stuff w.r.t
semihosting and chardevs so knowing how to properly order things in
the "new world" would be useful
    [ 9:33] Phil: YAML &anchor symbol is helpful to use a definition
from a previous layer
    [ 9:34] Mdasoh: It makes sense to have ordered options when you're
talking about putting objects within objects; at the same time it
doesn't make so much sense to order them when you're talking about
running them all through a BNF parser (flex+yacc?) for user-friendly
configuration. So of course there would be two layers, and you would
translate the unordered options into a set of encapsulating or ordered
options.
    [ 9:49] Andrea Bolognani: It Will Be Totally Different This Time™
    [ 9:50] John Snow: ^ that's partly why I wanted to discuss this,
to set concrete goals and to be able to measure progress
    [ 9:51] Alex Bennée: I'm afraid I have to drop off for the school
run - look forward to reading the reference docs ;-)
    [ 9:51] John Snow: ^ ty Alex
    [10:00] Stefan Hajnoczi: I need to drop now. I'll send the meeting
notes to the mailing list. Bye!
    [10:00] Kevin: Same for me
    [10:00] John Snow: OK!

Re: KVM call for agenda for 2020-10-06

[Stefan Hajnoczi]

Thank you everyone who joined the call. The meeting notes are below.

Stefan
---
*KVM Community Call - Tue Oct 6th
*Topic: QEMU CLI QAPI conversion

    * John Snow's summary of command-line options:
https://docs.google.com/spreadsheets/d/1OJvzDwLpo2XsGytNp9Pr-vYIrb0pWrFKBNPtOyH80ew/edit?usp=sharing
    * What is the first milestone?
        * Is it QemuOpts for everything? Straight to QAPI? something else?
        * Markus: The goal is to represent the configuration interface
in QAPI types
            * Don't parse QemuOpts, go straight to QAPI
    * How do we distribute this work to multiple engineers?
        * Examples:
            * --blockdev API is used on the command-line
            * --display
            * qemu-storage-daemon command-line is largely QAPI-fied
    * Alex Bennee:
        * We should have a gold-standard reference with documentation
if we are to expect maintainers to convert their own flags
        * -> John Snow will work on this document
    * Do we have good examples of turning QemuOpts to QAPI?
        * 53 of our 93 CLI flags that take arguments are QemuOpts, so
this is a major component
        * Kevin: -monitor for the Qemu Storage Daemon, recently
    * John Snow: Final milestone might be an automated QAPI-based CLI
parser, but only once QAPI types have been defined

    * Does command-line order matter?
        * Two options: allow any order OR left-to-right ordering
        * Andrea Bolognani: Most users expect left-to-right ordering,
why allow any order?
        * Eduardo Habkost: Can we enforce left-to-right ordering or do
we need to follow the deprecation process?
        * Daniel Berrange: Solve compability by introducing new
binaries without the burden of backwards compability
            * Unclear whether we will reach consensus on this call
about this. Maybe raise it on qemu-devel. [stefanha]
            * Philippe: Easy command-line options (-drive) and
managent-friendly options (-blockdev) could be offered by different
binaries
                * Daniel Berrange: Focussing on one new binary is more
achievable

    * Board defaults, configuration file options
        * How to set properties on existing objects (e.g. board defaults)?
        * Andrea Bolognani: Perhaps represent the board defaults as a
list of ordered options, append user-provided options, and *only then*
create the object?
            * Currently the boards create QOM objects directly, they
don't involve QAPI
        * Stefan: How do QOM objects/properties fit into QAPI
CLI/configuration parsing?
            * QAPI objects are processed by functions that will create
QOM objects
        * Markus: -global is broken

    * Eduardo: Long-term goal to describe QOM properties in QAPI
        * Daniel Berrange: eliminate QOM boilerplate by describing
object properties in QAPI
        * Markus: It's hard to use QAPI because QOM properties are
dynamic and can change at runtime

    * Next steps
        * John Snow and Markus will work on reference documentation



Bluejeans Chat Log
    [ 9:02] Stefan Hajnoczi: https://etherpad.opendev.org/p/QEMUCLI
    [ 9:05] John Ferlan: @stefan - there are some people in a different room....
    [ 9:08] Daniel Berrange: if you're not talking please mute
    [ 9:24] Alex Bennée: I ran into this ordering stuff w.r.t
semihosting and chardevs so knowing how to properly order things in
the "new world" would be useful
    [ 9:33] Phil: YAML &anchor symbol is helpful to use a definition
from a previous layer
    [ 9:34] Mdasoh: It makes sense to have ordered options when you're
talking about putting objects within objects; at the same time it
doesn't make so much sense to order them when you're talking about
running them all through a BNF parser (flex+yacc?) for user-friendly
configuration. So of course there would be two layers, and you would
translate the unordered options into a set of encapsulating or ordered
options.
    [ 9:49] Andrea Bolognani: It Will Be Totally Different This Time™
    [ 9:50] John Snow: ^ that's partly why I wanted to discuss this,
to set concrete goals and to be able to measure progress
    [ 9:51] Alex Bennée: I'm afraid I have to drop off for the school
run - look forward to reading the reference docs ;-)
    [ 9:51] John Snow: ^ ty Alex
    [10:00] Stefan Hajnoczi: I need to drop now. I'll send the meeting
notes to the mailing list. Bye!
    [10:00] Kevin: Same for me
    [10:00] John Snow: OK!

Re: KVM call for agenda for 2020-10-06

[Paolo Bonzini]

On 06/10/20 20:21, Stefan Hajnoczi wrote:
>     * Does command-line order matter?
>         * Two options: allow any order OR left-to-right ordering
>         * Andrea Bolognani: Most users expect left-to-right ordering,
> why allow any order?
>         * Eduardo Habkost: Can we enforce left-to-right ordering or do
> we need to follow the deprecation process?
>         * Daniel Berrange: Solve compability by introducing new
> binaries without the burden of backwards compability

I think "new binaries" shouldn't even have a command line; all
configuration should happen through QMP commands.  Those are naturally
time-ordered, which is equivalent to left-to-right, and therefore the
question is sidestepped.  Perhaps even having a command line in
qemu-storage-daemon was a mistake.

For "old binaries" we are not adding too many options, so apart from the
nasty distinction between early and late objects we're at least not
making it worse.

The big question to me is whether the configuration should be
QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
"object-add" (and to a lesser extent "device-add") are fine mechanisms
for configuration.  There is still need for better QOM introspection,
but it would be much simpler than doing QOM object creation via QAPI
struct, if at all possible.						

Paolo

Re: KVM call for agenda for 2020-10-06

[Paolo Bonzini]

On 06/10/20 20:21, Stefan Hajnoczi wrote:
>     * Does command-line order matter?
>         * Two options: allow any order OR left-to-right ordering
>         * Andrea Bolognani: Most users expect left-to-right ordering,
> why allow any order?
>         * Eduardo Habkost: Can we enforce left-to-right ordering or do
> we need to follow the deprecation process?
>         * Daniel Berrange: Solve compability by introducing new
> binaries without the burden of backwards compability

I think "new binaries" shouldn't even have a command line; all
configuration should happen through QMP commands.  Those are naturally
time-ordered, which is equivalent to left-to-right, and therefore the
question is sidestepped.  Perhaps even having a command line in
qemu-storage-daemon was a mistake.

For "old binaries" we are not adding too many options, so apart from the
nasty distinction between early and late objects we're at least not
making it worse.

The big question to me is whether the configuration should be
QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
"object-add" (and to a lesser extent "device-add") are fine mechanisms
for configuration.  There is still need for better QOM introspection,
but it would be much simpler than doing QOM object creation via QAPI
struct, if at all possible.						

Paolo

Re: KVM call for agenda for 2020-10-06

[Daniel P. Berrangé]

On Wed, Oct 07, 2020 at 07:50:20PM +0200, Paolo Bonzini wrote:
> On 06/10/20 20:21, Stefan Hajnoczi wrote:
> >     * Does command-line order matter?
> >         * Two options: allow any order OR left-to-right ordering
> >         * Andrea Bolognani: Most users expect left-to-right ordering,
> > why allow any order?
> >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
> > we need to follow the deprecation process?
> >         * Daniel Berrange: Solve compability by introducing new
> > binaries without the burden of backwards compability
> 
> I think "new binaries" shouldn't even have a command line; all
> configuration should happen through QMP commands.  Those are naturally
> time-ordered, which is equivalent to left-to-right, and therefore the
> question is sidestepped.  Perhaps even having a command line in
> qemu-storage-daemon was a mistake.

Non-interactive configuration is a nice property for simpler integration
use cases. eg launching from the shell is tedious with QMP compared to
CLI args.

This could be addressed though by having a configuration file to load
config from, where the config entries can be mapped 1-1 onto QMP commands,
essentially making the config file a non-interactive QMP.

> The big question to me is whether the configuration should be
> QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
> "object-add" (and to a lesser extent "device-add") are fine mechanisms
> for configuration.  There is still need for better QOM introspection,
> but it would be much simpler than doing QOM object creation via QAPI
> struct, if at all possible.



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: KVM call for agenda for 2020-10-06

[Daniel P. Berrangé]

On Wed, Oct 07, 2020 at 07:50:20PM +0200, Paolo Bonzini wrote:
> On 06/10/20 20:21, Stefan Hajnoczi wrote:
> >     * Does command-line order matter?
> >         * Two options: allow any order OR left-to-right ordering
> >         * Andrea Bolognani: Most users expect left-to-right ordering,
> > why allow any order?
> >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
> > we need to follow the deprecation process?
> >         * Daniel Berrange: Solve compability by introducing new
> > binaries without the burden of backwards compability
> 
> I think "new binaries" shouldn't even have a command line; all
> configuration should happen through QMP commands.  Those are naturally
> time-ordered, which is equivalent to left-to-right, and therefore the
> question is sidestepped.  Perhaps even having a command line in
> qemu-storage-daemon was a mistake.

Non-interactive configuration is a nice property for simpler integration
use cases. eg launching from the shell is tedious with QMP compared to
CLI args.

This could be addressed though by having a configuration file to load
config from, where the config entries can be mapped 1-1 onto QMP commands,
essentially making the config file a non-interactive QMP.

> The big question to me is whether the configuration should be
> QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
> "object-add" (and to a lesser extent "device-add") are fine mechanisms
> for configuration.  There is still need for better QOM introspection,
> but it would be much simpler than doing QOM object creation via QAPI
> struct, if at all possible.



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: KVM call for agenda for 2020-10-06

[Kevin Wolf]

Am 07.10.2020 um 19:50 hat Paolo Bonzini geschrieben:
> On 06/10/20 20:21, Stefan Hajnoczi wrote:
> >     * Does command-line order matter?
> >         * Two options: allow any order OR left-to-right ordering
> >         * Andrea Bolognani: Most users expect left-to-right ordering,
> > why allow any order?
> >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
> > we need to follow the deprecation process?
> >         * Daniel Berrange: Solve compability by introducing new
> > binaries without the burden of backwards compability
> 
> I think "new binaries" shouldn't even have a command line; all
> configuration should happen through QMP commands.  Those are naturally
> time-ordered, which is equivalent to left-to-right, and therefore the
> question is sidestepped.  Perhaps even having a command line in
> qemu-storage-daemon was a mistake.
> 
> For "old binaries" we are not adding too many options, so apart from the
> nasty distinction between early and late objects we're at least not
> making it worse.
> 
> The big question to me is whether the configuration should be
> QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
> "object-add" (and to a lesser extent "device-add") are fine mechanisms
> for configuration.  There is still need for better QOM introspection,
> but it would be much simpler than doing QOM object creation via QAPI
> struct, if at all possible.

I would strongly vote for QAPI-based. It doesn't have to be fully based
on QAPI structs internally, but the defining property for me is that the
external interface is described in the QAPI schema (which implies using
QAPI structs for the external facing code).

Not only is it a PITA to work with things like "gen": false or "props":
"any", but having two systems to configure things side by side is also
highly inconsistent.

I have recently discussed object-add with Markus, or to be more precise,
a QAPIfied --object in qsd wrapping it. This doesn't work well without
having a schema. I believe the right thing to do there is build a QAPI
schema describing the existing QOM properties in a first step (which
already gives you all of the advantages of QAPI like introspection), and
then in a second step generate the respective QOM code for initialising
the properties from the schema instead of duplicating it.

This can get challenging with dynamic properties, but as far as I can
see, user creatable objects only have class properties or object
properties created right in .instance_init (which should be equivalent).

As the number of user creatable objects isn't too large, this shouldn't
be too hard. I'm less sure about device-add, though in theory the same
approch would probably result in the best interface.

Kevin

Re: KVM call for agenda for 2020-10-06

[Kevin Wolf]

Am 07.10.2020 um 19:50 hat Paolo Bonzini geschrieben:
> On 06/10/20 20:21, Stefan Hajnoczi wrote:
> >     * Does command-line order matter?
> >         * Two options: allow any order OR left-to-right ordering
> >         * Andrea Bolognani: Most users expect left-to-right ordering,
> > why allow any order?
> >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
> > we need to follow the deprecation process?
> >         * Daniel Berrange: Solve compability by introducing new
> > binaries without the burden of backwards compability
> 
> I think "new binaries" shouldn't even have a command line; all
> configuration should happen through QMP commands.  Those are naturally
> time-ordered, which is equivalent to left-to-right, and therefore the
> question is sidestepped.  Perhaps even having a command line in
> qemu-storage-daemon was a mistake.
> 
> For "old binaries" we are not adding too many options, so apart from the
> nasty distinction between early and late objects we're at least not
> making it worse.
> 
> The big question to me is whether the configuration should be
> QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
> "object-add" (and to a lesser extent "device-add") are fine mechanisms
> for configuration.  There is still need for better QOM introspection,
> but it would be much simpler than doing QOM object creation via QAPI
> struct, if at all possible.

I would strongly vote for QAPI-based. It doesn't have to be fully based
on QAPI structs internally, but the defining property for me is that the
external interface is described in the QAPI schema (which implies using
QAPI structs for the external facing code).

Not only is it a PITA to work with things like "gen": false or "props":
"any", but having two systems to configure things side by side is also
highly inconsistent.

I have recently discussed object-add with Markus, or to be more precise,
a QAPIfied --object in qsd wrapping it. This doesn't work well without
having a schema. I believe the right thing to do there is build a QAPI
schema describing the existing QOM properties in a first step (which
already gives you all of the advantages of QAPI like introspection), and
then in a second step generate the respective QOM code for initialising
the properties from the schema instead of duplicating it.

This can get challenging with dynamic properties, but as far as I can
see, user creatable objects only have class properties or object
properties created right in .instance_init (which should be equivalent).

As the number of user creatable objects isn't too large, this shouldn't
be too hard. I'm less sure about device-add, though in theory the same
approch would probably result in the best interface.

Kevin

Re: KVM call for agenda for 2020-10-06

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Oct 07, 2020 at 07:50:20PM +0200, Paolo Bonzini wrote:
>> On 06/10/20 20:21, Stefan Hajnoczi wrote:
>> >     * Does command-line order matter?
>> >         * Two options: allow any order OR left-to-right ordering
>> >         * Andrea Bolognani: Most users expect left-to-right ordering,
>> > why allow any order?
>> >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
>> > we need to follow the deprecation process?
>> >         * Daniel Berrange: Solve compability by introducing new
>> > binaries without the burden of backwards compability
>> 
>> I think "new binaries" shouldn't even have a command line; all
>> configuration should happen through QMP commands.  Those are naturally
>> time-ordered, which is equivalent to left-to-right, and therefore the
>> question is sidestepped.  Perhaps even having a command line in
>> qemu-storage-daemon was a mistake.
>
> Non-interactive configuration is a nice property for simpler integration
> use cases. eg launching from the shell is tedious with QMP compared to
> CLI args.

Yes.

CLI, config file and QMP are differently convenient for different use
cases.

When you want to tweak just a few things for a single run, or a few in a
row, nothing beats the CLI.

A bespoke, long-lived configuration you really want to keep in a file.

When you use a program to run QEMU, and that program does QMP anyway,
doing *everything* in QMP can be just what you want.

If we could afford just one of the three, we'd probably want to pick
QMP, because it's the most flexible (it's supports queries naturally),
and because picking something else can't eliminate QMP.  Fortunately, we
don't have to pick just one if we base on initial configuration on QAPI.

A QMP command is "send an instance of a QAPI type to QEMU, get an
instance of a QAPI type or an error back".

A CLI option or configuration file entry is the same with the return
type constrained to "nothing".

The hard part is designing the QAPI type and putting it to use in QEMU.
Wrapping it in a QMP command is utterly trivial.

Wrapping it in a CLI option is no harder (we've done it a few times).
Right now, it takes some boilerplate.  We can eliminate the boilerplate
the exact same way we did for QMP commands: generate it.

Wrapping a bunch of QAPI types in a configuration file format is a bit
more work, mostly because the bikeshed will need a coat of paint.

> This could be addressed though by having a configuration file to load
> config from, where the config entries can be mapped 1-1 onto QMP commands,
> essentially making the config file a non-interactive QMP.
>
>> The big question to me is whether the configuration should be
>> QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
>> "object-add" (and to a lesser extent "device-add") are fine mechanisms
>> for configuration.  There is still need for better QOM introspection,
>> but it would be much simpler than doing QOM object creation via QAPI
>> struct, if at all possible.

object-add and device-add are the last two genuine users of 'gen': false
(query-qmp-schema does it out of laziness, and my "Configurable policy
for handling deprecated interfaces" series makes it stop).

I'd like to take a serious swing at QAPIfying them, with a loose schema.
Good enough for QAPI-based initial configuration interfaces.  Not good
enough for introspection, but a better QOM introspection could fill that
gap.

Re: KVM call for agenda for 2020-10-06

[Paolo Bonzini]

On 08/10/20 13:25, Markus Armbruster wrote:
> CLI, config file and QMP are differently convenient for different use
> cases. [...]
> 
> If we could afford just one of the three, we'd probably want to pick
> QMP, because it's the most flexible (it's supports queries naturally),
> and because picking something else can't eliminate QMP.  Fortunately, we
> don't have to pick just one if we base on initial configuration on QAPI.

On the other hand, we don't have to pick just one because we already
have a CLI (though one that is full of warts) so the question is not
whether we want to have a CLI, but whether we want to have a *second* CLI.

So my point is essentially that:

* as you said, you cannot get rid of QMP

* we can make the existing CLI a QMP wrapper just like we did with HMP

* any work on QMP-based configuration would apply just as well to both
binaries, so developers could still mix CLI+QMP when (or if) desirable

* once you have a (warty but well-known) CLI and QMP, there are
diminishing returns in going all the way down to QAPI even for the two
hardest commands (device-add and object-add).  That time is better
invested in minimizing the differences between the two binaries, because
we all know that you won't pry the qemu-system-* command line from the
cold dead hands of users and developers.

(not coincidentially, this goes from least to most controversial).

Of course you may say this is "whataboutism", on the other hand time is
limited so I prefer to make the interesting tasks clear from the
beginning and allow better collaboration.

> I'd like to take a serious swing at QAPIfying them, with a loose schema.

What do you mean by "loose schema"?  Is it anything other than
"represent a QDict with a QAPI list of key-value pairs"?

Paolo

> Good enough for QAPI-based initial configuration interfaces.  Not good
> enough for introspection, but a better QOM introspection could fill that
> gap.
> 

Re: KVM call for agenda for 2020-10-06

[Eduardo Habkost]

On Thu, Oct 08, 2020 at 10:03:45AM +0200, Kevin Wolf wrote:
> Am 07.10.2020 um 19:50 hat Paolo Bonzini geschrieben:
> > On 06/10/20 20:21, Stefan Hajnoczi wrote:
> > >     * Does command-line order matter?
> > >         * Two options: allow any order OR left-to-right ordering
> > >         * Andrea Bolognani: Most users expect left-to-right ordering,
> > > why allow any order?
> > >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
> > > we need to follow the deprecation process?
> > >         * Daniel Berrange: Solve compability by introducing new
> > > binaries without the burden of backwards compability
> > 
> > I think "new binaries" shouldn't even have a command line; all
> > configuration should happen through QMP commands.  Those are naturally
> > time-ordered, which is equivalent to left-to-right, and therefore the
> > question is sidestepped.  Perhaps even having a command line in
> > qemu-storage-daemon was a mistake.
> > 
> > For "old binaries" we are not adding too many options, so apart from the
> > nasty distinction between early and late objects we're at least not
> > making it worse.
> > 
> > The big question to me is whether the configuration should be
> > QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
> > "object-add" (and to a lesser extent "device-add") are fine mechanisms
> > for configuration.  There is still need for better QOM introspection,
> > but it would be much simpler than doing QOM object creation via QAPI
> > struct, if at all possible.
> 
> I would strongly vote for QAPI-based. It doesn't have to be fully based
> on QAPI structs internally, but the defining property for me is that the
> external interface is described in the QAPI schema (which implies using
> QAPI structs for the external facing code).
> 
> Not only is it a PITA to work with things like "gen": false or "props":
> "any", but having two systems to configure things side by side is also
> highly inconsistent.
> 
> I have recently discussed object-add with Markus, or to be more precise,
> a QAPIfied --object in qsd wrapping it. This doesn't work well without
> having a schema. I believe the right thing to do there is build a QAPI
> schema describing the existing QOM properties in a first step (which
> already gives you all of the advantages of QAPI like introspection), and
> then in a second step generate the respective QOM code for initialising
> the properties from the schema instead of duplicating it.
> 
> This can get challenging with dynamic properties, but as far as I can
> see, user creatable objects only have class properties or object
> properties created right in .instance_init (which should be equivalent).

I've just submitted a series to ensure 100% of
TYPE_USER_CREATABLE types have only class properties:

https://lore.kernel.org/qemu-devel/20201009160122.1662082-1-ehabkost@redhat.com

> 
> As the number of user creatable objects isn't too large, this shouldn't
> be too hard. I'm less sure about device-add, though in theory the same
> approch would probably result in the best interface.

Doing the same for all user creatable device types would be nice
too.  We can use the property locking mechanism from the series above
to find out how bad the situation is.

-- 
Eduardo

Re: KVM call for agenda for 2020-10-06

[Eduardo Habkost]

On Thu, Oct 08, 2020 at 10:03:45AM +0200, Kevin Wolf wrote:
> Am 07.10.2020 um 19:50 hat Paolo Bonzini geschrieben:
> > On 06/10/20 20:21, Stefan Hajnoczi wrote:
> > >     * Does command-line order matter?
> > >         * Two options: allow any order OR left-to-right ordering
> > >         * Andrea Bolognani: Most users expect left-to-right ordering,
> > > why allow any order?
> > >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
> > > we need to follow the deprecation process?
> > >         * Daniel Berrange: Solve compability by introducing new
> > > binaries without the burden of backwards compability
> > 
> > I think "new binaries" shouldn't even have a command line; all
> > configuration should happen through QMP commands.  Those are naturally
> > time-ordered, which is equivalent to left-to-right, and therefore the
> > question is sidestepped.  Perhaps even having a command line in
> > qemu-storage-daemon was a mistake.
> > 
> > For "old binaries" we are not adding too many options, so apart from the
> > nasty distinction between early and late objects we're at least not
> > making it worse.
> > 
> > The big question to me is whether the configuration should be
> > QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
> > "object-add" (and to a lesser extent "device-add") are fine mechanisms
> > for configuration.  There is still need for better QOM introspection,
> > but it would be much simpler than doing QOM object creation via QAPI
> > struct, if at all possible.
> 
> I would strongly vote for QAPI-based. It doesn't have to be fully based
> on QAPI structs internally, but the defining property for me is that the
> external interface is described in the QAPI schema (which implies using
> QAPI structs for the external facing code).
> 
> Not only is it a PITA to work with things like "gen": false or "props":
> "any", but having two systems to configure things side by side is also
> highly inconsistent.
> 
> I have recently discussed object-add with Markus, or to be more precise,
> a QAPIfied --object in qsd wrapping it. This doesn't work well without
> having a schema. I believe the right thing to do there is build a QAPI
> schema describing the existing QOM properties in a first step (which
> already gives you all of the advantages of QAPI like introspection), and
> then in a second step generate the respective QOM code for initialising
> the properties from the schema instead of duplicating it.
> 
> This can get challenging with dynamic properties, but as far as I can
> see, user creatable objects only have class properties or object
> properties created right in .instance_init (which should be equivalent).

I've just submitted a series to ensure 100% of
TYPE_USER_CREATABLE types have only class properties:

https://lore.kernel.org/qemu-devel/20201009160122.1662082-1-ehabkost@redhat.com

> 
> As the number of user creatable objects isn't too large, this shouldn't
> be too hard. I'm less sure about device-add, though in theory the same
> approch would probably result in the best interface.

Doing the same for all user creatable device types would be nice
too.  We can use the property locking mechanism from the series above
to find out how bad the situation is.

-- 
Eduardo

Re: KVM call for agenda for 2020-10-06

[Markus Armbruster]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Oct 08, 2020 at 10:03:45AM +0200, Kevin Wolf wrote:
>> Am 07.10.2020 um 19:50 hat Paolo Bonzini geschrieben:
>> > On 06/10/20 20:21, Stefan Hajnoczi wrote:
>> > >     * Does command-line order matter?
>> > >         * Two options: allow any order OR left-to-right ordering
>> > >         * Andrea Bolognani: Most users expect left-to-right ordering,
>> > > why allow any order?
>> > >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
>> > > we need to follow the deprecation process?
>> > >         * Daniel Berrange: Solve compability by introducing new
>> > > binaries without the burden of backwards compability
>> > 
>> > I think "new binaries" shouldn't even have a command line; all
>> > configuration should happen through QMP commands.  Those are naturally
>> > time-ordered, which is equivalent to left-to-right, and therefore the
>> > question is sidestepped.  Perhaps even having a command line in
>> > qemu-storage-daemon was a mistake.
>> > 
>> > For "old binaries" we are not adding too many options, so apart from the
>> > nasty distinction between early and late objects we're at least not
>> > making it worse.
>> > 
>> > The big question to me is whether the configuration should be
>> > QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
>> > "object-add" (and to a lesser extent "device-add") are fine mechanisms
>> > for configuration.  There is still need for better QOM introspection,
>> > but it would be much simpler than doing QOM object creation via QAPI
>> > struct, if at all possible.
>> 
>> I would strongly vote for QAPI-based. It doesn't have to be fully based
>> on QAPI structs internally, but the defining property for me is that the
>> external interface is described in the QAPI schema (which implies using
>> QAPI structs for the external facing code).
>> 
>> Not only is it a PITA to work with things like "gen": false or "props":
>> "any", but having two systems to configure things side by side is also
>> highly inconsistent.
>> 
>> I have recently discussed object-add with Markus, or to be more precise,
>> a QAPIfied --object in qsd wrapping it. This doesn't work well without
>> having a schema. I believe the right thing to do there is build a QAPI
>> schema describing the existing QOM properties in a first step (which
>> already gives you all of the advantages of QAPI like introspection), and
>> then in a second step generate the respective QOM code for initialising
>> the properties from the schema instead of duplicating it.
>> 
>> This can get challenging with dynamic properties, but as far as I can
>> see, user creatable objects only have class properties or object
>> properties created right in .instance_init (which should be equivalent).
>
> I've just submitted a series to ensure 100% of
> TYPE_USER_CREATABLE types have only class properties:
>
> https://lore.kernel.org/qemu-devel/20201009160122.1662082-1-ehabkost@redhat.com

Lovely idea!

Additional benefit: QOM introspection becomes more useful.

>> As the number of user creatable objects isn't too large, this shouldn't
>> be too hard. I'm less sure about device-add, though in theory the same
>> approch would probably result in the best interface.
>
> Doing the same for all user creatable device types would be nice
> too.  We can use the property locking mechanism from the series above
> to find out how bad the situation is.

Yes, please!

Re: KVM call for agenda for 2020-10-06

[Markus Armbruster]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Oct 08, 2020 at 10:03:45AM +0200, Kevin Wolf wrote:
>> Am 07.10.2020 um 19:50 hat Paolo Bonzini geschrieben:
>> > On 06/10/20 20:21, Stefan Hajnoczi wrote:
>> > >     * Does command-line order matter?
>> > >         * Two options: allow any order OR left-to-right ordering
>> > >         * Andrea Bolognani: Most users expect left-to-right ordering,
>> > > why allow any order?
>> > >         * Eduardo Habkost: Can we enforce left-to-right ordering or do
>> > > we need to follow the deprecation process?
>> > >         * Daniel Berrange: Solve compability by introducing new
>> > > binaries without the burden of backwards compability
>> > 
>> > I think "new binaries" shouldn't even have a command line; all
>> > configuration should happen through QMP commands.  Those are naturally
>> > time-ordered, which is equivalent to left-to-right, and therefore the
>> > question is sidestepped.  Perhaps even having a command line in
>> > qemu-storage-daemon was a mistake.
>> > 
>> > For "old binaries" we are not adding too many options, so apart from the
>> > nasty distinction between early and late objects we're at least not
>> > making it worse.
>> > 
>> > The big question to me is whether the configuration should be
>> > QAPI-based, that is based on QAPI structs, or QMP-based.  If the latter,
>> > "object-add" (and to a lesser extent "device-add") are fine mechanisms
>> > for configuration.  There is still need for better QOM introspection,
>> > but it would be much simpler than doing QOM object creation via QAPI
>> > struct, if at all possible.
>> 
>> I would strongly vote for QAPI-based. It doesn't have to be fully based
>> on QAPI structs internally, but the defining property for me is that the
>> external interface is described in the QAPI schema (which implies using
>> QAPI structs for the external facing code).
>> 
>> Not only is it a PITA to work with things like "gen": false or "props":
>> "any", but having two systems to configure things side by side is also
>> highly inconsistent.
>> 
>> I have recently discussed object-add with Markus, or to be more precise,
>> a QAPIfied --object in qsd wrapping it. This doesn't work well without
>> having a schema. I believe the right thing to do there is build a QAPI
>> schema describing the existing QOM properties in a first step (which
>> already gives you all of the advantages of QAPI like introspection), and
>> then in a second step generate the respective QOM code for initialising
>> the properties from the schema instead of duplicating it.
>> 
>> This can get challenging with dynamic properties, but as far as I can
>> see, user creatable objects only have class properties or object
>> properties created right in .instance_init (which should be equivalent).
>
> I've just submitted a series to ensure 100% of
> TYPE_USER_CREATABLE types have only class properties:
>
> https://lore.kernel.org/qemu-devel/20201009160122.1662082-1-ehabkost@redhat.com

Lovely idea!

Additional benefit: QOM introspection becomes more useful.

>> As the number of user creatable objects isn't too large, this shouldn't
>> be too hard. I'm less sure about device-add, though in theory the same
>> approch would probably result in the best interface.
>
> Doing the same for all user creatable device types would be nice
> too.  We can use the property locking mechanism from the series above
> to find out how bad the situation is.

Yes, please!