Redesign of QEMU startup & initial configuration

Redesign of QEMU startup & initial configuration

[Markus Armbruster]

= Motivation =

QEMU startup and initial configuration were designed many years ago for
a much, much simpler QEMU.  They have since changed beyond recognition
to adapt to new needs.  There was no real redesign.  Adaption to new
needs has become more and more difficult.  A recent example for
"difficult" is Damien's "[RFC PATCH v3 0/5] QMP support for
cold-plugging devices".

I think it's time for a redesign.


= What users want for initial configuration =

1. QMP only

   Management applications need to use QMP for monitoring anyway.  They
   may want to use it for initial configuration, too.  Libvirt does.

   They still need to bootstrap a QMP monitor, and for that, CLI is fine
   as long as it's simple and stable.

2. CLI and configuration files

   Human users want a CLI and configuration files.

   CLI is good for quick tweaks, and to explore.

   For more permanent, non-trivial configuration, configuration files
   are more suitable, because they are easier to read, edit, and
   document than long command lines.


= What we have for initial configuration =

Half of 1. and half of 2., satisfying nobody's needs.

Management applications need to do a lot of non-trivial initial
configuration with the CLI.

Human users struggle with inconsistent syntax, insufficiently expressive
configuration files, and huge command lines.


= Why startup matters for initial configuration =

Startup and initial configuration are joined at the hip: the way startup
works limits how initial configuration can work.

Consider the startup we had before -prefconfig:

0. Basic, configuration-independent initialization

1. Parse and partially process CLI

2. Startup with the remaining CLI processing mixed in

3. Startup complete, run main loop

Note:

* CLI is processed out of order.

  Few people understand the order, and only while starting at the code.
  Pretty much every time we mess with the order, we break something.

* QMP monitors become available only at 3, i.e. after startup is
  complete.  Precludes use of QMP for initial configuration.

The current startup with -preconfig permits a bit of initial
configuration with QMP, at the cost of complicating startup code and
interfaces.  More in "Appendix: Why further incremental change feels
impractical".


= Ways forward =

0. This is too hard, let's go shopping

   I can't really blame people for not wanting to mess with startup.  It
   *is* hard, and there's plenty of other useful stuff to do.

   However, I believe we have to mess with it, i.e. this is not actually
   a way forward.

1. Improve what we have step by step, managing incompatible changes

   We tried.  It has eaten substantial time and energy, it has
   complicated code and interfaces further, and we're still nowhere near
   where we need to be.  Does not feel like a practical way forward to
   me.

   If you don't believe me, read the long version in "Appendix: Why
   further incremental change feels impractical".

2. Start over with a QMP-only QEMU, keep the traditional QEMU forever

   A QMP-only QEMU isn't just a matter of providing an alternate main()
   where we ditch most of the CLI for QMP replacements.  It takes a
   rework of the startup code as "phase transitions" triggered by QMP
   commands.  Similar to how -preconfig triggers some startup work.

   To reduce code duplication, we'll probably want to rework the
   traditional QEMU's startup code some, too.

3. Start over with the aim to replace the traditional QEMU

   I don't want an end game where we have two QEMUs, one with bad CLI
   and limited QMP, and one without real CLI and decent QMP.  I want one
   QEMU with decent CLI and decent QMP.

   I propose that a QEMU with decent CLI and decent QMP isn't really
   harder than a QMP-only QEMU.  CLI is just another transport that
   happens to be one-way, i.e. the QMP commands encoded as CLI options
   can't return a value.  Same for configuration files.  I believe the
   parts that are actually hard are shared with QMP-only: QAPIfying
   initial configuration and reworking the startup code as phase
   transitions.

   When that decent QEMU is ready, we can deprecate and later drop the
   traditional one.


= A design proposal =

0. Basic, configuration-independent initialization

1. Start main loop

2. Feed it CLI left to right.  Each option runs a handler just like each
   QMP command does.

   Options that read a configuration file inject the file into the feed.

   Options that create a monitor create it suspended.

   Options may advance the phase / run state, they may require certain
   phase(s), and semantics may depend on the phase.

3. When we're done with CLI, resume any monitors we created.

   As a convenience, provide an easy way to auto-advance phase / run
   state to "guest runs" right here.  The traditional way would be to
   auto-advance unless the user gives -S.

4. Monitors now feed commands to the main loop.  Commands may advance
   the phase / run state, and they may require certain phase(s), and
   semantics may depend on the phase.

   In particular, command "cont" advances phase / run state to "guest
   runs".

Users can do as much or as little with the CLI as they want.  A
management application might want to set up a QMP monitor and no more.


= A way to get to the proposed design =

Create an alternate QEMU binary with the CLI taken out and shot:

0. Basic, configuration-independent initialization

2. Startup

3. Startup complete, run main loop

Maybe hardcode a few things so we can actually run something without any
configuration.  To be ditched when we can configure again.

Rework 2. Startup as a sequence of state transitions we can later
trigger from commands / options.

Create a quasi-monitor so we can feed QMP commands to the main loop.

Make QMP command "cont" transition to "guest runs".

Feed "cont" to the main loop, and drop 2. Startup.

Create QAPI/CLI infrastructure, so we can define CLI options like QMP
commands.  I have (dusty and likely quite bit-rotten) RFC patches.

Feed CLI options to the main loop, and only then feed "cont".

Create option -S to suppress feeding "cont".

Create / reuse / rework QMP commands and CLI options to create QMP
monitors.  Monitors created by CLI start suspended, and get resumed only
when we're done with the CLI.  This is so monitor commands can't race
with the CLI.

Create CLI option to read a configuration file, and feed its contents to
the main loop.  Recursively.  We'll likely want nicer syntax than CLI or
QMP for configuration files.

At this point, we have all the infrastructure we need.  What's still
missing is filling gaps in QMP and in CLI.

I believe "QMP only" would only be marginally easier.  Basically, it
processes (a minimal) CLI before the main loop, not in it.  We could
leave it unQAPIfied.  I can't quite see offhand how to do configuration
files sanely.

Stretch goals:

* Throw in QAPIfying HMP.

* Dust with syntactic sugar (sparingly, please) to make CLI/HMP easier
  to use for humans.


= Initial exploration, a.k.a. talk's cheap, show me the code =

I'm going to post "[PATCH RFC 00/11] vl: Explore redesign of startup"
shortly.


= Appendix: Why further incremental change feels impractical =

Recall from "Why startup matters for initial configuration" how the
traditional startup code made initial configuration with QMP impossible.

We added -preconfig (v3.0.0 in 2018) to support a bit of it.  If given,
we delay part of the startup until QMP command x-exit-preconfig, like
this:

0. Basic, configuration-independent initialization

1. Parse and partially process CLI left to right

2. Some startup with some of the remaining CLI processing mixed in

3. Run main loop

3.1. Execute QMP commands that are allowed before x-exit-preconfig

3.2. x-exit-preconfig: Remaining startup with the remaining CLI
     processing mixed in

3.3. Startup complete

Note:

* (Out of order) CLI processing is interleaved with (in-order) QMP
  execution.

  One word: ugh!

* QMP becomes available earlier, but still not early enough to support
  "QMP only".

* The initial implementation of -preconfig was a hack.  Getting to the
  current implementation took effort.

  Actually, *any* non-trivial change in this area now takes more effort
  than it should.  See also "pretty much every time [...] we break
  something" above.

We can try to push more and more startup work from 2. to 3 until "QMP
only" becomes possible.

Damien's "[RFC PATCH v3 0/5] QMP support for cold-plugging devices" is a
modest step in this direction.  Since the work to be pushed needs to
move to a different point than x-exit-preconfig, we need another command
x-machine-init to provide a suitable point.

At this point, we complicated startup code and interfaces (and arguably
compromised CLI sanity further), with more of the same needed to get
anywhere near "QMP only".

There is plenty of doubt on the interfaces going around, and that's why
-preconfig is still not a stable interface.

Crafting a big change in small steps sounds great.  It isn't when we
have to make things worse before they can get better, and every step is
painfully slow because it's just too hard, and the end state we aim for
isn't really what we want.

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Thu, Dec 02, 2021 at 07:57:38AM +0100, Markus Armbruster wrote:
> = Motivation =
> 
> QEMU startup and initial configuration were designed many years ago for
> a much, much simpler QEMU.  They have since changed beyond recognition
> to adapt to new needs.  There was no real redesign.  Adaption to new
> needs has become more and more difficult.  A recent example for
> "difficult" is Damien's "[RFC PATCH v3 0/5] QMP support for
> cold-plugging devices".
> 
> I think it's time for a redesign.
> 
> 
> = What users want for initial configuration =
> 
> 1. QMP only
> 
>    Management applications need to use QMP for monitoring anyway.  They
>    may want to use it for initial configuration, too.  Libvirt does.

Essentially, as soon as you need to deal with hotplug, you need QMP/QAPI.
As soon as you need QMP/QAPI, it is horrible to also need to use something
that isn't JSON for the coldplug configuration approach, as you've doubled
the number of things to write + test.

>    They still need to bootstrap a QMP monitor, and for that, CLI is fine
>    as long as it's simple and stable.

Since you mentioned 'simple', allow me to go down a slight
tangent...

Incidentally, I often wish we didn't use -chardevs anywhere near
as much as we do. Chardev was OK when we used it for backing
simple guest devices that just had a single host endpoint that
was considered permanently connected.

Everywhere we've used it for things that really want socket
semantics we've created so much pain. The things we've done
with chardevs for vhostuser in particular horrify me. I'm
glad the block layer resisted the tempetation and just
directly used the SocketAddress QAPI type with a QIOChannel
objects.

I would say the monitor would be better without chardevs
too. Having to create multiple monitor instances so that
you can have multiple clients is insane. The complexity
of course is the need for 'mux' with HMP. If it were not
for that, we could use QIOChannel and SocketAddress for
the monitor code.

Maybe if we one day get HMP fully  separated from QMP such
that it is independant code, we can simplify internally.
Meanwhile, I think we should consider the QMP CLI at least
to only use SocketAddress for config, and secretly turn
it into a chardev internally. 

> 2. CLI and configuration files
> 
>    Human users want a CLI and configuration files.
> 
>    CLI is good for quick tweaks, and to explore.
> 
>    For more permanent, non-trivial configuration, configuration files
>    are more suitable, because they are easier to read, edit, and
>    document than long command lines.

And you can start doing things like "includes" or templating
to make the config more manageable / scalable.


It is also nice to not have human and machines in completely
separate worlds. Humans will often look at what machines do and
then try to replicate that. eg people often ask to see the libvirt
QEMU config and then want to run that directly themselves, or use
libvirt CLI passthrough to add on features that libvirt does not
native support yet.

As libvirt makes more & more use of QAPI, we are increasing the
divide betweeen what machines and humans target.

I'd note that Kubernetes doesn't bother with a human CLI at
all, and just expects everyone/everything to use JSON/YAML
config files. So there's no divide between what syntax humans
and machines use - humans just send the config via the CLI
tool which uploads it to the REST API that machines use
directly. 


> = What we have for initial configuration =
> 
> Half of 1. and half of 2., satisfying nobody's needs.
> 
> Management applications need to do a lot of non-trivial initial
> configuration with the CLI.
> 
> Human users struggle with inconsistent syntax, insufficiently expressive
> configuration files, and huge command lines.

The QEMU CLI was nice because you could historically do simple
stuff really simply eg

  qemu-system-x86_64 mydisk.img

or

  qemu-system-x86_64 -hda mydisk.img -cdrom mydistro.iso

The challenge is how CLI usage evolves as you need to finese
the config you're using. Using this simple CLI approach, our
defaults still largely give you a VM from 1995. If people
want a modern guest setup, the simple syntax quickly stops
being simple.

You very easily get to a point where passing stuff on the
CLI gets out of control. IMHO if the CLI is over 100 characters
long, a config file is the way to go. Given that I really
wonder whether direct configuration of hardware on the CLI
is worthwhile at all, and users should instead /always/ be
using a config file, albeit indirectly.


This doesn't mean simple things become harder.


I'm thinking of a config file that supports a standard
template language supporting variable substitution,
loops, conditionals. The CLI then does not need to
represent anything related to hardware config schemas
at all. Instead it just needs to take the name of a
config file and ability to set variables.  We could
then ship  some standard configs for the simple cases,
which would solve the problem of us being stuck on
ancient defaults for the simple CLI configs.

  $QEMU  virtiovm.cfg  root=mydisk.img cdrom=mydistro.iso

where virtiovm.cfg contains

  {
     "machine": {
       "type": "q35",
       "memory": "2G",
     },
     "disks": [
       {
         "type": "virtio-blk",
	 "file": @@ VAR root@@
       },
       @@ IF defined cdrom @@
       {
         "type": "sata",
	 "media": "cdroM"
	 "file": @@ VAR cdrom @@
       }
       @@ END @@
     ]
  }

Don't pay close attention to my suggesting JSON syntax
here. I just invented something for sake of illustration.
A real config would be more complex  than this and follow
the QAPI schemas.

The key is that the complexity of the config file does
not matter so much, because the user isn't directly
interacting with it for simple tasks. They just have
a nice high level CLI taking a few useful variables.
The simple CLI becomes massively more useful than
our current simple CLI, because it can support modern
VMs of arbitrary complexity while still remaining
simple.


> = Why startup matters for initial configuration =
> 
> Startup and initial configuration are joined at the hip: the way startup
> works limits how initial configuration can work.
> 
> Consider the startup we had before -prefconfig:
> 
> 0. Basic, configuration-independent initialization
> 
> 1. Parse and partially process CLI
> 
> 2. Startup with the remaining CLI processing mixed in
> 
> 3. Startup complete, run main loop
> 
> Note:
> 
> * CLI is processed out of order.
> 
>   Few people understand the order, and only while starting at the code.
>   Pretty much every time we mess with the order, we break something.
> 
> * QMP monitors become available only at 3, i.e. after startup is
>   complete.  Precludes use of QMP for initial configuration.
> 
> The current startup with -preconfig permits a bit of initial
> configuration with QMP, at the cost of complicating startup code and
> interfaces.  More in "Appendix: Why further incremental change feels
> impractical".
> 
> 
> = Ways forward =

Just to clarify or remind ourselves of scope.

When I think about startup/cli in QEMU, I essentially consider
this to mean all softmmu/vl.c code.  Of course there are
supporting pieces splattered around, but IMHO 90% of the pain
is resulting from the code in the vl.c file.

> 0. This is too hard, let's go shopping
> 
>    I can't really blame people for not wanting to mess with startup.  It
>    *is* hard, and there's plenty of other useful stuff to do.
> 
>    However, I believe we have to mess with it, i.e. this is not actually
>    a way forward.

Yeah, we hurt ourselves by not being able to move forward with
a better approach to startup and configuration.

> 1. Improve what we have step by step, managing incompatible changes
> 
>    We tried.  It has eaten substantial time and energy, it has
>    complicated code and interfaces further, and we're still nowhere near
>    where we need to be.  Does not feel like a practical way forward to
>    me.
> 
>    If you don't believe me, read the long version in "Appendix: Why
>    further incremental change feels impractical".

Essentially death by 1000 cuts.

> 2. Start over with a QMP-only QEMU, keep the traditional QEMU forever
> 
>    A QMP-only QEMU isn't just a matter of providing an alternate main()
>    where we ditch most of the CLI for QMP replacements.  It takes a
>    rework of the startup code as "phase transitions" triggered by QMP
>    commands.  Similar to how -preconfig triggers some startup work.
> 
>    To reduce code duplication, we'll probably want to rework the
>    traditional QEMU's startup code some, too.

Refering back to my earlier note, when I've suggested this approach
in the past, I've basically considered it to mean stop touching
softmmu/vl.c, and create a softmmu/vl2.c. The existing binaries
remain using vl.c, we get legacy free binaries built from vl2.c

Overtime we might get confidence to refactor bits of vl.c to
reduce duplication, but I would consider that not a priority
at least in the short term.

Part of the reason why vl.c is so painful is danger of breaking
existing apps, especially mgmt tools like libvirt. If we did
have a vl.c and vl2.c in parallel, creating separate binaries
for each we could finese our support guarantees.

ie once we consider the new vl2.c to be feature complete for
typical production usage, we could declare that mgmt apps
should exclusively use the new binaries, and the old binaries
are no longer guaranteed CLI stable. This would unlock thue
ability to make more risky changes to vl.c to reduce the
duplication / maint burden of two binaries.

On the other hand if we do a good job with the new binaries
maybe there ceases to be any reason to keep the old binaries
at all after a few years.

> 3. Start over with the aim to replace the traditional QEMU
> 
>    I don't want an end game where we have two QEMUs, one with bad CLI
>    and limited QMP, and one without real CLI and decent QMP.  I want one
>    QEMU with decent CLI and decent QMP.
> 
>    I propose that a QEMU with decent CLI and decent QMP isn't really
>    harder than a QMP-only QEMU.  CLI is just another transport that
>    happens to be one-way, i.e. the QMP commands encoded as CLI options
>    can't return a value.  Same for configuration files.  I believe the
>    parts that are actually hard are shared with QMP-only: QAPIfying
>    initial configuration and reworking the startup code as phase
>    transitions.
> 
>    When that decent QEMU is ready, we can deprecate and later drop the
>    traditional one.

Right, so this is essentially what I've just suggested as a possible
evoluation of (2). The only difference here is that you're setting
the death of the old QEMU as an explicit end gaol in (3), while
in (2) it is merely a nice-to-have.

In practical terms I don't think (2) and (3) are that different
from each other for the first 2 years or so.  A lot can change in
that time, so I don't think we need to fixate on a choice of
(2) vs (3) upfront, just make it clear that the death of the old
binaries is "on the table" as an outcome if we do well.

> = A design proposal =
> 
> 0. Basic, configuration-independent initialization
> 
> 1. Start main loop
> 
> 2. Feed it CLI left to right.  Each option runs a handler just like each
>    QMP command does.
> 
>    Options that read a configuration file inject the file into the feed.
> 
>    Options that create a monitor create it suspended.
> 
>    Options may advance the phase / run state, they may require certain
>    phase(s), and semantics may depend on the phase.
> 
> 3. When we're done with CLI, resume any monitors we created.
> 
>    As a convenience, provide an easy way to auto-advance phase / run
>    state to "guest runs" right here.  The traditional way would be to
>    auto-advance unless the user gives -S.
> 
> 4. Monitors now feed commands to the main loop.  Commands may advance
>    the phase / run state, and they may require certain phase(s), and
>    semantics may depend on the phase.
> 
>    In particular, command "cont" advances phase / run state to "guest
>    runs".
> 
> Users can do as much or as little with the CLI as they want.  A
> management application might want to set up a QMP monitor and no more.

As illustrated earlier, I'd really like us to consider being a bit
more adventurous on the CLI side. I'm convinced that a CLI for
directly configurable hardware is doomed to be horrible no matter
what, if you try to directly expose all QAPI configuration
flexibilty. Whether key/value, JSON, whatever, it will become
unmanagable on the CLI because VM hardware config is inherantly
complicated.

Thus my though that config files or QMP should be the only two
places where the full power of QAPI config is exposed. Use CLI
as just a way to interact with config files in a simple way
with templates.

We can really think of QEMU's original simple CLI from 15
years ago as being a kind of templating frontend for a config
file. The config file just happened to be written as C code,
and so not user edittable if going outside the common cases.

> = A way to get to the proposed design =
> 
> Create an alternate QEMU binary with the CLI taken out and shot:
> 
> 0. Basic, configuration-independent initialization
> 
> 2. Startup
> 
> 3. Startup complete, run main loop
> 
> Maybe hardcode a few things so we can actually run something without any
> configuration.  To be ditched when we can configure again.

I wouldn't bother trying to keep things working in the intermediate
steps, as if we go with "new binary" we basically assume a clean
slate and can just build things up in understandable chunks  even
if they're not runnable initially.

> Rework 2. Startup as a sequence of state transitions we can later
> trigger from commands / options.
> 
> Create a quasi-monitor so we can feed QMP commands to the main loop.
> 
> Make QMP command "cont" transition to "guest runs".
> 
> Feed "cont" to the main loop, and drop 2. Startup.
> 
> Create QAPI/CLI infrastructure, so we can define CLI options like QMP
> commands.  I have (dusty and likely quite bit-rotten) RFC patches.
> 
> Feed CLI options to the main loop, and only then feed "cont".
> 
> Create option -S to suppress feeding "cont".
> 
> Create / reuse / rework QMP commands and CLI options to create QMP
> monitors.  Monitors created by CLI start suspended, and get resumed only
> when we're done with the CLI.  This is so monitor commands can't race
> with the CLI.
> 
> Create CLI option to read a configuration file, and feed its contents to
> the main loop.  Recursively.  We'll likely want nicer syntax than CLI or
> QMP for configuration files.
> 
> At this point, we have all the infrastructure we need.  What's still
> missing is filling gaps in QMP and in CLI.
> 
> I believe "QMP only" would only be marginally easier.  Basically, it
> processes (a minimal) CLI before the main loop, not in it.  We could
> leave it unQAPIfied.  I can't quite see offhand how to do configuration
> files sanely.
> 
> Stretch goals:
> 
> * Throw in QAPIfying HMP.
> 
> * Dust with syntactic sugar (sparingly, please) to make CLI/HMP easier
>   to use for humans.
> 
> 
> = Initial exploration, a.k.a. talk's cheap, show me the code =
> 
> I'm going to post "[PATCH RFC 00/11] vl: Explore redesign of startup"
> shortly.

So in your RFC you've basically modified the existing startup
code in vl.c, but I presume that was just expediant for the
purpose of a quickish illustration.

None the less I feel that all the patches cutting cruft are
perhaps a distraction to understanding the proposal. Might
have been clearer to just start from a vl2.c file and build
up from nothing.  Anyway, I don't want was our time arguing
over that approach for an RFC, so I won't say more here
at least :-)

> = Appendix: Why further incremental change feels impractical =
> 
> Recall from "Why startup matters for initial configuration" how the
> traditional startup code made initial configuration with QMP impossible.

snip

> Crafting a big change in small steps sounds great.  It isn't when we
> have to make things worse before they can get better, and every step is
> painfully slow because it's just too hard, and the end state we aim for
> isn't really what we want.

I can't disagree with this. If we carry on trying to evolve vl.c
incrementally we are doomed to be stuck with a horrible starstup
process for enternity (or at least as long as I'll still be
working as QEMU maintainer).


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

I’ll take the liberty to cut one part (I agree with much of what you say elsewhere)

> On 9 Dec 2021, at 20:11, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> As illustrated earlier, I'd really like us to consider being a bit
> more adventurous on the CLI side. I'm convinced that a CLI for
> directly configurable hardware is doomed to be horrible no matter
> what, if you try to directly expose all QAPI configuration
> flexibilty. Whether key/value, JSON, whatever, it will become
> unmanagable on the CLI because VM hardware config is inherantly
> complicated.
> 

I absolutely agree, but reach a slightly different conclusion

> Thus my though that config files or QMP should be the only two
> places where the full power of QAPI config is exposed. Use CLI
> as just a way to interact with config files in a simple way
> with templates.

I would countenance that we choose only one place to ‘support’ an interface. Either “Yet Another Hardware Configuration Language” or QAPI. Rather than re-inventing that wheel I would simply suggest that we leave that to the relevant ‘user’ community (libvirt, whatever), who have specific requirements and/or existing solutions. Leaving QEMU itself to focus on improving QAPI (and migrating the CLI). 

Cheers
Mark.


[-- Attachment #2: Type: text/html, Size: 9828 bytes --]

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Thu, Dec 09, 2021 at 09:01:24PM +0100, Mark Burton wrote:
> I’ll take the liberty to cut one part (I agree with much of what you say elsewhere)
> 
> > On 9 Dec 2021, at 20:11, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > As illustrated earlier, I'd really like us to consider being a bit
> > more adventurous on the CLI side. I'm convinced that a CLI for
> > directly configurable hardware is doomed to be horrible no matter
> > what, if you try to directly expose all QAPI configuration
> > flexibilty. Whether key/value, JSON, whatever, it will become
> > unmanagable on the CLI because VM hardware config is inherantly
> > complicated.
> > 
> 
> I absolutely agree, but reach a slightly different conclusion
> 
> > Thus my though that config files or QMP should be the only two
> > places where the full power of QAPI config is exposed. Use CLI
> > as just a way to interact with config files in a simple way
> > with templates.
> 
> I would countenance that we choose only one place to ‘support’ an interface. Either “Yet Another Hardware Configuration Language” or QAPI. Rather than re-inventing that wheel I would simply suggest that we leave that to the relevant ‘user’ community (libvirt, whatever), who have specific requirements and/or existing solutions. Leaving QEMU itself to focus on improving QAPI (and migrating the CLI). 

Yes, indeed, the logical extension of my idea is that the 'simple'
CLI + templating thing doesn't atually have to be in the main QEMU
binary at all. We could in fact ship a bare '/usr/bin/qemu' which
does the config file templating and spawns whatever full QEMU
binary (/usr/bin/qemu-system-blah) does the real VM.  The key is
just that we have something simple for users, who don't want a
full mgmt layer and like the historical QEMU simple configs.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/9/21 20:11, Daniel P. Berrangé wrote:
>>     They still need to bootstrap a QMP monitor, and for that, CLI is fine
>>     as long as it's simple and stable.

I would go a step further and say that the QMP monitor socket should be 
created by whoever invoked QEMU and passed down via systemd's socket 
activation protocol, with a fallback to stdin/stdout.

>> 2. CLI and configuration files
>>
>>     Human users want a CLI and configuration files.
> 
> I'd note that Kubernetes doesn't bother with a human CLI at
> all, and just expects everyone/everything to use JSON/YAML
> config files. So there's no divide between what syntax humans
> and machines use - humans just send the config via the CLI
> tool which uploads it to the REST API that machines use
> directly.

I agree with Mark here that configuration is beyond QEMU's role.  QEMU's 
JSON-based API isn't quite REST, but the idea is the same: the user 
talks to a tool which converts the configuration to QAPI commands.  The 
QAPI commands can be:

- invocations of qmp_* functions in the case of a command-line based 
configuration

- invocations of QMP commands on a socket if startup is mediated by a 
management tool

Where Markus and I disagree is whether the two cases should be covered 
by one binary or two.  Personally I don't see much advantage in having a 
single binary, just like I don't care if /bin/ls and /bin/dir are 
symlinks, or they're separate binaries, or one invokes the other.

What matters to me is the maintainability of the command-line based 
startup code.  Actual experience says that, as more and more options are 
supported via QMP-based configuration, the command-line based startup 
code gets simpler or at least more self-contained.

In the terms of my KVM Forum talk, this is obtained by converting 
"one-off" options (typically global variables) to a combination of a 
"shortcut" option (typically a qemu_opts_set or qemu_opts_parse 
invocation) and a QOM property or a QMP command.  Examples from recent 
QEMU versions include -no-shutdown, -no-hpet, -no-acpi.

>> = Ways forward =
> 
> Just to clarify or remind ourselves of scope.
> 
> When I think about startup/cli in QEMU, I essentially consider
> this to mean all softmmu/vl.c code.  Of course there are
> supporting pieces splattered around, but IMHO 90% of the pain
> is resulting from the code in the vl.c file.
>
>> 1. Improve what we have step by step, managing incompatible changes
>>
>>     We tried.  It has eaten substantial time and energy, it has
>>     complicated code and interfaces further, and we're still nowhere near
>>     where we need to be.  Does not feel like a practical way forward to
>>     me.
>>
>>     If you don't believe me, read the long version in "Appendix: Why
>>     further incremental change feels impractical".
> 
> Essentially death by 1000 cuts.

There are two kinds of incremental changes.  Incremental changes to the 
UI are one thing; incremental changes to the implementations aka 
refactoring is another.

I agree that further incremental changes to the UI (in the style of 
-preconfig) are impractical, but incremental changes to the 
implementation are totally practical.  The reason for that is that there 
*is* a way forward for decoupling the options from the command line 
parsing code, and this way forward has *already* proved itself.

> Refering back to my earlier note, when I've suggested this approach
> in the past, I've basically considered it to mean stop touching
> softmmu/vl.c, and create a softmmu/vl2.c. The existing binaries
> remain using vl.c, we get legacy free binaries built from vl2.c
> 
> Overtime we might get confidence to refactor bits of vl.c to
> reduce duplication, but I would consider that not a priority
> at least in the short term.

This is what I have in my tree, though it's called climain.c and 
qmpmain.c. :)  There is one major difference though: refactoring bits of 
vl.c to use more of QMP/QOM *is* the way you add functionality in qmpmain.c.

I disagree that vl.c is hard to refactor.  Sure there were some issues 
in 6.0/6.1, but at this point a lot of the hard parts have been done. 
The conversion of -smp to a QOM option, for example, was largely painless.

> On the other hand if we do a good job with the new binaries
> maybe there ceases to be any reason to keep the old binaries
> at all after a few years.

I'm not sure if this is possible if the new binaries provide exclusively 
a machine-oriented (JSON) interface.  But as you said, the pressure to 
keep a backwards-compatible command line would be much less with a good 
machine-oriented binary.

>> 3. Start over with the aim to replace the traditional QEMU
>>
>>     I don't want an end game where we have two QEMUs, one with bad CLI
>>     and limited QMP, and one without real CLI and decent QMP.  I want one
>>     QEMU with decent CLI and decent QMP.
>
> Right, so this is essentially what I've just suggested as a possible
> evoluation of (2). The only difference here is that you're setting
> the death of the old QEMU as an explicit end gaol in (3), while
> in (2) it is merely a nice-to-have.

I agree.  Though in my case I'd still go for two binaries: one with good 
CLI and limited QMP, and one with limited CLI and good QMP.

> In practical terms I don't think (2) and (3) are that different
> from each other for the first 2 years or so.  A lot can change in
> that time, so I don't think we need to fixate on a choice of
> (2) vs (3) upfront, just make it clear that the death of the old
> binaries is "on the table" as an outcome if we do well.

Agreed, but I would set the beginning of the "2 years or so" at last year...

>> = Appendix: Why further incremental change feels impractical =
>>
>> Crafting a big change in small steps sounds great.  It isn't when we
>> have to make things worse before they can get better, and every step is
>> painfully slow because it's just too hard, and the end state we aim for
>> isn't really what we want.
> 
> I can't disagree with this. If we carry on trying to evolve vl.c
> incrementally we are doomed to be stuck with a horrible starstup
> process for enternity (or at least as long as I'll still be
> working as QEMU maintainer).

... and if you compare vl.c in 5.2 and now, and consider current vl.c to 
be horrible, my knowedge of English does not include an adjective to 
describe the 5.2 state.  Some incremental work _is_ possible or even 
necessary, and has been done already.

Paolo

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Fri, Dec 10, 2021 at 09:34:41AM +0100, Paolo Bonzini wrote:
> On 12/9/21 20:11, Daniel P. Berrangé wrote:
> > >     They still need to bootstrap a QMP monitor, and for that, CLI is fine
> > >     as long as it's simple and stable.
> 
> I would go a step further and say that the QMP monitor socket should be
> created by whoever invoked QEMU and passed down via systemd's socket
> activation protocol, with a fallback to stdin/stdout.

That's an interesting idea, firmly relegating any "human friendly"
targetted CLI to a separate program, that in turn invokes this
low level QEMU binary. I do like the simplicity of this and the
strict division of the layers it provides us, as it will help keep
us honest when designing human friendly interfaces.

To be clear, I do think the QEMU project should be delivering a
nice simple human targetted interface, and ideally using the
'/usr/bin/qemu' binary name, and able to deliver users a machines
with a modern hardware config that can evolve over time.

> > > = Appendix: Why further incremental change feels impractical =
> > > 
> > > Crafting a big change in small steps sounds great.  It isn't when we
> > > have to make things worse before they can get better, and every step is
> > > painfully slow because it's just too hard, and the end state we aim for
> > > isn't really what we want.
> > 
> > I can't disagree with this. If we carry on trying to evolve vl.c
> > incrementally we are doomed to be stuck with a horrible starstup
> > process for enternity (or at least as long as I'll still be
> > working as QEMU maintainer).
> 
> ... and if you compare vl.c in 5.2 and now, and consider current vl.c to be
> horrible, my knowedge of English does not include an adjective to describe
> the 5.2 state.  Some incremental work _is_ possible or even necessary, and
> has been done already.

Right, I'm not saying vl.c hasn't improved, but we're never going
to get out of the peculiar historical startup ordering rules we
have today by incremental fixes, without breaking people.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Dec 02, 2021 at 07:57:38AM +0100, Markus Armbruster wrote:
>> = Motivation =
>> 
>> QEMU startup and initial configuration were designed many years ago for
>> a much, much simpler QEMU.  They have since changed beyond recognition
>> to adapt to new needs.  There was no real redesign.  Adaption to new
>> needs has become more and more difficult.  A recent example for
>> "difficult" is Damien's "[RFC PATCH v3 0/5] QMP support for
>> cold-plugging devices".
>> 
>> I think it's time for a redesign.
>> 
>> 
>> = What users want for initial configuration =
>> 
>> 1. QMP only
>> 
>>    Management applications need to use QMP for monitoring anyway.  They
>>    may want to use it for initial configuration, too.  Libvirt does.
>
> Essentially, as soon as you need to deal with hotplug, you need QMP/QAPI.
> As soon as you need QMP/QAPI, it is horrible to also need to use something
> that isn't JSON for the coldplug configuration approach, as you've doubled
> the number of things to write + test.

Yes, having to encode the same information in multiple non-trivial ways
is horrible.

When using nothing but QMP, you deal with just one encoding.

With QMP-like configuration files[1], it's one encoding over two
transports.

A QAPIfied CLI where the option arguments can be JSON would offer yet
another transport with only a slight variation of the encoding.

Any extras need testing.  Okay if their benefits are worth it.

>>    They still need to bootstrap a QMP monitor, and for that, CLI is fine
>>    as long as it's simple and stable.
>
> Since you mentioned 'simple', allow me to go down a slight
> tangent...
>
> Incidentally, I often wish we didn't use -chardevs anywhere near
> as much as we do. Chardev was OK when we used it for backing
> simple guest devices that just had a single host endpoint that
> was considered permanently connected.
>
> Everywhere we've used it for things that really want socket
> semantics we've created so much pain. The things we've done
> with chardevs for vhostuser in particular horrify me. I'm
> glad the block layer resisted the tempetation and just
> directly used the SocketAddress QAPI type with a QIOChannel
> objects.
>
> I would say the monitor would be better without chardevs
> too. Having to create multiple monitor instances so that
> you can have multiple clients is insane. The complexity
> of course is the need for 'mux' with HMP. If it were not
> for that, we could use QIOChannel and SocketAddress for
> the monitor code.
>
> Maybe if we one day get HMP fully  separated from QMP such
> that it is independant code, we can simplify internally.
> Meanwhile, I think we should consider the QMP CLI at least
> to only use SocketAddress for config, and secretly turn
> it into a chardev internally. 

QMP is used in anger pretty much only with Unix domain sockets, and
maybe Internet domain sockets.  I figure people also use stdio for
messing around, but something like

    $ socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>" UNIX-CONNECT:/path/to/qmp-socket

is actually nicer to use (mostly because we restrict readline to HMP,
but our readline is homegrown and can't hold a candle to the real thing
anyway).

>> 2. CLI and configuration files
>> 
>>    Human users want a CLI and configuration files.
>> 
>>    CLI is good for quick tweaks, and to explore.
>> 
>>    For more permanent, non-trivial configuration, configuration files
>>    are more suitable, because they are easier to read, edit, and
>>    document than long command lines.
>
> And you can start doing things like "includes" or templating
> to make the config more manageable / scalable.

A single configuration file would already be an improvement over today's
use of CLI, but as configuration gets more complex, a way to stitch
together multiple configuration files becomes a necessity real fast..
Even now, we keep a few -readconfig files in docs/config/.

Configuration will get a lot more complex once we can wire up devices
from components in configuration instead of C code.

> It is also nice to not have human and machines in completely
> separate worlds. Humans will often look at what machines do and
> then try to replicate that. eg people often ask to see the libvirt
> QEMU config and then want to run that directly themselves, or use
> libvirt CLI passthrough to add on features that libvirt does not
> native support yet.

Yes!  Obvious and easy paths between the different ways to configure
stuff are empowering.  I very much want that.

QemuOpts provides one from CLI to -readconfig: -writeconfig.  Falls well
short of needs mostly because our needs have long outgrown QemuOpts.

> As libvirt makes more & more use of QAPI, we are increasing the
> divide betweeen what machines and humans target.
>
> I'd note that Kubernetes doesn't bother with a human CLI at
> all, and just expects everyone/everything to use JSON/YAML
> config files. So there's no divide between what syntax humans
> and machines use - humans just send the config via the CLI
> tool which uploads it to the REST API that machines use
> directly. 
>
>> = What we have for initial configuration =
>> 
>> Half of 1. and half of 2., satisfying nobody's needs.
>> 
>> Management applications need to do a lot of non-trivial initial
>> configuration with the CLI.
>> 
>> Human users struggle with inconsistent syntax, insufficiently expressive
>> configuration files, and huge command lines.
>
> The QEMU CLI was nice because you could historically do simple
> stuff really simply eg
>
>   qemu-system-x86_64 mydisk.img
>
> or
>
>   qemu-system-x86_64 -hda mydisk.img -cdrom mydistro.iso
>
> The challenge is how CLI usage evolves as you need to finese
> the config you're using. Using this simple CLI approach, our
> defaults still largely give you a VM from 1995. If people
> want a modern guest setup, the simple syntax quickly stops
> being simple.
>
> You very easily get to a point where passing stuff on the
> CLI gets out of control. IMHO if the CLI is over 100 characters
> long, a config file is the way to go.

Yes, we want simple things to be simple, and complicated things to be
possible.

Keeping defaults that have long gone bad just to avoid user-visible
change is an issue that won't go away, but can perhaps be mitigated a
bit.  More on that below.

>                                       Given that I really
> wonder whether direct configuration of hardware on the CLI
> is worthwhile at all, and users should instead /always/ be
> using a config file, albeit indirectly.
>
>
> This doesn't mean simple things become harder.

While ridiculously complicated CLI hardware configuration certainly
exists and is a problem, dead simple CLI hardware configuration also
exists.  Doesn't mean your idea is bad, it just raises the bar for it :)

> I'm thinking of a config file that supports a standard
> template language supporting variable substitution,
> loops, conditionals. The CLI then does not need to
> represent anything related to hardware config schemas
> at all. Instead it just needs to take the name of a
> config file and ability to set variables.  We could
> then ship  some standard configs for the simple cases,
> which would solve the problem of us being stuck on
> ancient defaults for the simple CLI configs.
>
>   $QEMU  virtiovm.cfg  root=mydisk.img cdrom=mydistro.iso
>
> where virtiovm.cfg contains
>
>   {
>      "machine": {
>        "type": "q35",
>        "memory": "2G",
>      },
>      "disks": [
>        {
>          "type": "virtio-blk",
> 	 "file": @@ VAR root@@
>        },
>        @@ IF defined cdrom @@
>        {
>          "type": "sata",
> 	 "media": "cdroM"
> 	 "file": @@ VAR cdrom @@
>        }
>        @@ END @@
>      ]
>   }
>
> Don't pay close attention to my suggesting JSON syntax
> here. I just invented something for sake of illustration.
> A real config would be more complex  than this and follow
> the QAPI schemas.
>
> The key is that the complexity of the config file does
> not matter so much, because the user isn't directly
> interacting with it for simple tasks. They just have
> a nice high level CLI taking a few useful variables.
> The simple CLI becomes massively more useful than
> our current simple CLI, because it can support modern
> VMs of arbitrary complexity while still remaining
> simple.

There are also cases where the user's configuration is structurally
simple, just too long to do comfortably in CLI.  A configuration file
with comments is better.

For persistent configuration, writing a configuration file beats writing
a shell script for me.

Not arguments against your templating idea.  I'm merely pointing out
human end users will find (simple!) configuration files useful, too.

>> = Why startup matters for initial configuration =
>> 
>> Startup and initial configuration are joined at the hip: the way startup
>> works limits how initial configuration can work.
>> 
>> Consider the startup we had before -prefconfig:
>> 
>> 0. Basic, configuration-independent initialization
>> 
>> 1. Parse and partially process CLI
>> 
>> 2. Startup with the remaining CLI processing mixed in
>> 
>> 3. Startup complete, run main loop
>> 
>> Note:
>> 
>> * CLI is processed out of order.
>> 
>>   Few people understand the order, and only while starting at the code.
>>   Pretty much every time we mess with the order, we break something.
>> 
>> * QMP monitors become available only at 3, i.e. after startup is
>>   complete.  Precludes use of QMP for initial configuration.
>> 
>> The current startup with -preconfig permits a bit of initial
>> configuration with QMP, at the cost of complicating startup code and
>> interfaces.  More in "Appendix: Why further incremental change feels
>> impractical".
>> 
>> 
>> = Ways forward =
>
> Just to clarify or remind ourselves of scope.
>
> When I think about startup/cli in QEMU, I essentially consider
> this to mean all softmmu/vl.c code.  Of course there are
> supporting pieces splattered around, but IMHO 90% of the pain
> is resulting from the code in the vl.c file.
>
>> 0. This is too hard, let's go shopping
>> 
>>    I can't really blame people for not wanting to mess with startup.  It
>>    *is* hard, and there's plenty of other useful stuff to do.
>> 
>>    However, I believe we have to mess with it, i.e. this is not actually
>>    a way forward.
>
> Yeah, we hurt ourselves by not being able to move forward with
> a better approach to startup and configuration.
>
>> 1. Improve what we have step by step, managing incompatible changes
>> 
>>    We tried.  It has eaten substantial time and energy, it has
>>    complicated code and interfaces further, and we're still nowhere near
>>    where we need to be.  Does not feel like a practical way forward to
>>    me.
>> 
>>    If you don't believe me, read the long version in "Appendix: Why
>>    further incremental change feels impractical".
>
> Essentially death by 1000 cuts.
>
>> 2. Start over with a QMP-only QEMU, keep the traditional QEMU forever
>> 
>>    A QMP-only QEMU isn't just a matter of providing an alternate main()
>>    where we ditch most of the CLI for QMP replacements.  It takes a
>>    rework of the startup code as "phase transitions" triggered by QMP
>>    commands.  Similar to how -preconfig triggers some startup work.
>> 
>>    To reduce code duplication, we'll probably want to rework the
>>    traditional QEMU's startup code some, too.
>
> Refering back to my earlier note, when I've suggested this approach
> in the past, I've basically considered it to mean stop touching
> softmmu/vl.c, and create a softmmu/vl2.c. The existing binaries
> remain using vl.c, we get legacy free binaries built from vl2.c
>
> Overtime we might get confidence to refactor bits of vl.c to
> reduce duplication, but I would consider that not a priority
> at least in the short term.
>
> Part of the reason why vl.c is so painful is danger of breaking
> existing apps, especially mgmt tools like libvirt. If we did
> have a vl.c and vl2.c in parallel, creating separate binaries
> for each we could finese our support guarantees.
>
> ie once we consider the new vl2.c to be feature complete for
> typical production usage, we could declare that mgmt apps
> should exclusively use the new binaries, and the old binaries
> are no longer guaranteed CLI stable. This would unlock thue
> ability to make more risky changes to vl.c to reduce the
> duplication / maint burden of two binaries.
>
> On the other hand if we do a good job with the new binaries
> maybe there ceases to be any reason to keep the old binaries
> at all after a few years.

I want an open path to a single binary.  Taking years to get there is
fine.

>> 3. Start over with the aim to replace the traditional QEMU
>> 
>>    I don't want an end game where we have two QEMUs, one with bad CLI
>>    and limited QMP, and one without real CLI and decent QMP.  I want one
>>    QEMU with decent CLI and decent QMP.
>> 
>>    I propose that a QEMU with decent CLI and decent QMP isn't really
>>    harder than a QMP-only QEMU.  CLI is just another transport that
>>    happens to be one-way, i.e. the QMP commands encoded as CLI options
>>    can't return a value.  Same for configuration files.  I believe the
>>    parts that are actually hard are shared with QMP-only: QAPIfying
>>    initial configuration and reworking the startup code as phase
>>    transitions.
>> 
>>    When that decent QEMU is ready, we can deprecate and later drop the
>>    traditional one.
>
> Right, so this is essentially what I've just suggested as a possible
> evoluation of (2). The only difference here is that you're setting
> the death of the old QEMU as an explicit end gaol in (3), while
> in (2) it is merely a nice-to-have.

I'm not sure we all agree it's nice to have.  See Paolo's reply.

> In practical terms I don't think (2) and (3) are that different
> from each other for the first 2 years or so.  A lot can change in
> that time, so I don't think we need to fixate on a choice of
> (2) vs (3) upfront, just make it clear that the death of the old
> binaries is "on the table" as an outcome if we do well.

Yes, the overlap between (2) and (3) is big enough to defer the decision
some.

However, I'd like to start working on (3) earlier, time permitting.

>> = A design proposal =
>> 
>> 0. Basic, configuration-independent initialization
>> 
>> 1. Start main loop
>> 
>> 2. Feed it CLI left to right.  Each option runs a handler just like each
>>    QMP command does.
>> 
>>    Options that read a configuration file inject the file into the feed.
>> 
>>    Options that create a monitor create it suspended.
>> 
>>    Options may advance the phase / run state, they may require certain
>>    phase(s), and semantics may depend on the phase.
>> 
>> 3. When we're done with CLI, resume any monitors we created.
>> 
>>    As a convenience, provide an easy way to auto-advance phase / run
>>    state to "guest runs" right here.  The traditional way would be to
>>    auto-advance unless the user gives -S.
>> 
>> 4. Monitors now feed commands to the main loop.  Commands may advance
>>    the phase / run state, and they may require certain phase(s), and
>>    semantics may depend on the phase.
>> 
>>    In particular, command "cont" advances phase / run state to "guest
>>    runs".
>> 
>> Users can do as much or as little with the CLI as they want.  A
>> management application might want to set up a QMP monitor and no more.
>
> As illustrated earlier, I'd really like us to consider being a bit
> more adventurous on the CLI side. I'm convinced that a CLI for
> directly configurable hardware is doomed to be horrible no matter
> what, if you try to directly expose all QAPI configuration
> flexibilty. Whether key/value, JSON, whatever, it will become
> unmanagable on the CLI because VM hardware config is inherantly
> complicated.
>
> Thus my though that config files or QMP should be the only two
> places where the full power of QAPI config is exposed. Use CLI
> as just a way to interact with config files in a simple way
> with templates.
>
> We can really think of QEMU's original simple CLI from 15
> years ago as being a kind of templating frontend for a config
> file. The config file just happened to be written as C code,
> and so not user edittable if going outside the common cases.

Right.

Painted with a big brush, there are two kinds of code in hw/: actual
device emulation, and "wiring".  Both in C, and sometimes in the same .c
file.

Doing the "wiring" in configuration instead is less powerful (no longer
Turing complete[2]), but easier to reason about, maintain and change.
Change is possible even in the field.  The obvious separation between
emulation and wiring is a nice bonus.

The wiring C code supports something not unlike templating: we have a
number of configuration knobs that deposit something for the wiring C
code to pick up.

It's rather limited, though: the "variables" are fixed at compile time.

Aside: the handling of variables that doesn't get substituted is wildly
inconsistent.

Perhaps doing wiring in configuration reduces the maintenance burden to
a degree where we can mitigate the "defaults gone bad, but no want
change" issue by offering additional, better things instead of changing
existing, bad things.  I don't know.

>> = A way to get to the proposed design =
>> 
>> Create an alternate QEMU binary with the CLI taken out and shot:
>> 
>> 0. Basic, configuration-independent initialization
>> 
>> 2. Startup
>> 
>> 3. Startup complete, run main loop
>> 
>> Maybe hardcode a few things so we can actually run something without any
>> configuration.  To be ditched when we can configure again.
>
> I wouldn't bother trying to keep things working in the intermediate
> steps, as if we go with "new binary" we basically assume a clean
> slate and can just build things up in understandable chunks  even
> if they're not runnable initially.
>
>> Rework 2. Startup as a sequence of state transitions we can later
>> trigger from commands / options.
>> 
>> Create a quasi-monitor so we can feed QMP commands to the main loop.
>> 
>> Make QMP command "cont" transition to "guest runs".
>> 
>> Feed "cont" to the main loop, and drop 2. Startup.
>> 
>> Create QAPI/CLI infrastructure, so we can define CLI options like QMP
>> commands.  I have (dusty and likely quite bit-rotten) RFC patches.
>> 
>> Feed CLI options to the main loop, and only then feed "cont".
>> 
>> Create option -S to suppress feeding "cont".
>> 
>> Create / reuse / rework QMP commands and CLI options to create QMP
>> monitors.  Monitors created by CLI start suspended, and get resumed only
>> when we're done with the CLI.  This is so monitor commands can't race
>> with the CLI.
>> 
>> Create CLI option to read a configuration file, and feed its contents to
>> the main loop.  Recursively.  We'll likely want nicer syntax than CLI or
>> QMP for configuration files.
>> 
>> At this point, we have all the infrastructure we need.  What's still
>> missing is filling gaps in QMP and in CLI.
>> 
>> I believe "QMP only" would only be marginally easier.  Basically, it
>> processes (a minimal) CLI before the main loop, not in it.  We could
>> leave it unQAPIfied.  I can't quite see offhand how to do configuration
>> files sanely.
>> 
>> Stretch goals:
>> 
>> * Throw in QAPIfying HMP.
>> 
>> * Dust with syntactic sugar (sparingly, please) to make CLI/HMP easier
>>   to use for humans.
>> 
>> 
>> = Initial exploration, a.k.a. talk's cheap, show me the code =
>> 
>> I'm going to post "[PATCH RFC 00/11] vl: Explore redesign of startup"
>> shortly.
>
> So in your RFC you've basically modified the existing startup
> code in vl.c, but I presume that was just expediant for the
> purpose of a quickish illustration.

Yes.  I wasn't in the mood to figure out the Meson magic to link a new
binary, so I took a shortcut.

> None the less I feel that all the patches cutting cruft are
> perhaps a distraction to understanding the proposal. Might
> have been clearer to just start from a vl2.c file and build
> up from nothing.  Anyway, I don't want was our time arguing
> over that approach for an RFC, so I won't say more here
> at least :-)

Easier to understand, possibly.  Slower to post, certainly.  Happy to
iterate.

Also: cutting the cruft like this demonstrates just how much of it
there is.

>> = Appendix: Why further incremental change feels impractical =
>> 
>> Recall from "Why startup matters for initial configuration" how the
>> traditional startup code made initial configuration with QMP impossible.
>
> snip
>
>> Crafting a big change in small steps sounds great.  It isn't when we
>> have to make things worse before they can get better, and every step is
>> painfully slow because it's just too hard, and the end state we aim for
>> isn't really what we want.
>
> I can't disagree with this. If we carry on trying to evolve vl.c
> incrementally we are doomed to be stuck with a horrible starstup
> process for enternity (or at least as long as I'll still be
> working as QEMU maintainer).
>
>
> Regards,
> Daniel


[1] Probably in a superset of JSON, because bare JSON is an awful
configuration file format.

[2] Unless we go apeshit.  Which we shouldn't.

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

> On 10 Dec 2021, at 12:25, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Fri, Dec 10, 2021 at 09:34:41AM +0100, Paolo Bonzini wrote:
>> On 12/9/21 20:11, Daniel P. Berrangé wrote:
>>>>    They still need to bootstrap a QMP monitor, and for that, CLI is fine
>>>>    as long as it's simple and stable.
>> 
>> I would go a step further and say that the QMP monitor socket should be
>> created by whoever invoked QEMU and passed down via systemd's socket
>> activation protocol, with a fallback to stdin/stdout.

Could we take one more small step …. 
(Sorry - I’m sure you’ll all hate me for pointing at the elephant in the room….)

Why should QEMU itself handle this? You may want to use systemd socket activation, I may be happier with a different approach. The commonality is surely at the level of the underlying QAPI.
Being able to build QEMU as a ….. library, with a single entry point to access the QAPI would allow the QEMU community to focus on it’s key ‘kernel’, while others are able to propose integrated solutions like activation through systemd an/or whatever libvirt does etc etc…. By all means there can be a systemd-qemu project…. But does that have to be baked into QEMU?
I know there’s a history on the use of the “Library” word - equally there is a notion of a library needing a static interface etc - I propose we agree upon a single access mechanism to the QAPI - who’s existence and stability we have already (I think) agreed upon.

This requires a ‘full’ QAPI, of course, and a bit of care in the build system. It allows total flexibility, and the guarantees of stability reach no further than what we're proposing anyway. The existing CLI can migrate (fast or slow) to using QAPI….

Cheers
Mark.


> 
> That's an interesting idea, firmly relegating any "human friendly"
> targetted CLI to a separate program, that in turn invokes this
> low level QEMU binary. I do like the simplicity of this and the
> strict division of the layers it provides us, as it will help keep
> us honest when designing human friendly interfaces.
> 
> To be clear, I do think the QEMU project should be delivering a
> nice simple human targetted interface, and ideally using the
> '/usr/bin/qemu' binary name, and able to deliver users a machines
> with a modern hardware config that can evolve over time.
> 
>>>> = Appendix: Why further incremental change feels impractical =
>>>> 
>>>> Crafting a big change in small steps sounds great.  It isn't when we
>>>> have to make things worse before they can get better, and every step is
>>>> painfully slow because it's just too hard, and the end state we aim for
>>>> isn't really what we want.
>>> 
>>> I can't disagree with this. If we carry on trying to evolve vl.c
>>> incrementally we are doomed to be stuck with a horrible starstup
>>> process for enternity (or at least as long as I'll still be
>>> working as QEMU maintainer).
>> 
>> ... and if you compare vl.c in 5.2 and now, and consider current vl.c to be
>> horrible, my knowedge of English does not include an adjective to describe
>> the 5.2 state.  Some incremental work _is_ possible or even necessary, and
>> has been done already.
> 
> Right, I'm not saying vl.c hasn't improved, but we're never going
> to get out of the peculiar historical startup ordering rules we
> have today by incremental fixes, without breaking people.
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Fri, Dec 10, 2021 at 03:15:50PM +0100, Mark Burton wrote:
> 
> 
> > On 10 Dec 2021, at 12:25, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Fri, Dec 10, 2021 at 09:34:41AM +0100, Paolo Bonzini wrote:
> >> On 12/9/21 20:11, Daniel P. Berrangé wrote:
> >>>>    They still need to bootstrap a QMP monitor, and for that, CLI is fine
> >>>>    as long as it's simple and stable.
> >> 
> >> I would go a step further and say that the QMP monitor socket should be
> >> created by whoever invoked QEMU and passed down via systemd's socket
> >> activation protocol, with a fallback to stdin/stdout.
> 
> Could we take one more small step …. 
> (Sorry - I’m sure you’ll all hate me for pointing at the elephant in the room….)
> 
> Why should QEMU itself handle this? You may want to use systemd
> socket activation, I may be happier with a different approach.
> The commonality is surely at the level of the underlying QAPI.
> Being able to build QEMU as a ….. library, with a single entry
> point to access the QAPI would allow the QEMU community to focus
> on it’s key ‘kernel’, while others are able to propose integrated
> solutions like activation through systemd an/or whatever libvirt
> does etc etc…. By all means there can be a systemd-qemu project….
> But does that have to be baked into QEMU?

Systemd activation doesn't really tie QEMU into systemd at
all. The socket passing scheme is trivial and both sides are
easily implemented by any application. It is reasonable to
use in QEMU on any UNIX platform at least. Windows is probably
the only complication here.

> I know there’s a history on the use of the “Library” word - equally
> there is a notion of a library needing a static interface etc - I
> propose we agree upon a single access mechanism to the QAPI - who’s
> existence and stability we have already (I think) agreed upon.

A stable/static interface is not hard - it doesn't require all
that much more than exposing a few APIs related to input and
output of QAPI based JSON docs. This all exists already, you
wwould just get skipping thue sockets serialization of QMP.

The biggest stumbling block for QEMU as a library is actually
the licensing mess. Too much of our code is GPLv2-only, which
makes it impractical to use as a library in too many use cases.
Any app that is not GPLv2-only compatible would have to isolate
QEMU in its own process and talk to it over RPC, at which point
it has just reinventing QMP again.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

> On 10 Dec 2021, at 15:26, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Fri, Dec 10, 2021 at 03:15:50PM +0100, Mark Burton wrote:
>> 
>> 
>>> On 10 Dec 2021, at 12:25, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> 
>>> On Fri, Dec 10, 2021 at 09:34:41AM +0100, Paolo Bonzini wrote:
>>>> On 12/9/21 20:11, Daniel P. Berrangé wrote:
>>>>>>   They still need to bootstrap a QMP monitor, and for that, CLI is fine
>>>>>>   as long as it's simple and stable.
>>>> 
>>>> I would go a step further and say that the QMP monitor socket should be
>>>> created by whoever invoked QEMU and passed down via systemd's socket
>>>> activation protocol, with a fallback to stdin/stdout.
>> 
>> Could we take one more small step …. 
>> (Sorry - I’m sure you’ll all hate me for pointing at the elephant in the room….)
>> 
>> Why should QEMU itself handle this? You may want to use systemd
>> socket activation, I may be happier with a different approach.
>> The commonality is surely at the level of the underlying QAPI.
>> Being able to build QEMU as a ….. library, with a single entry
>> point to access the QAPI would allow the QEMU community to focus
>> on it’s key ‘kernel’, while others are able to propose integrated
>> solutions like activation through systemd an/or whatever libvirt
>> does etc etc…. By all means there can be a systemd-qemu project….
>> But does that have to be baked into QEMU?
> 
> Systemd activation doesn't really tie QEMU into systemd at
> all. The socket passing scheme is trivial and both sides are
> easily implemented by any application. It is reasonable to
> use in QEMU on any UNIX platform at least. Windows is probably
> the only complication here.
> 
>> I know there’s a history on the use of the “Library” word - equally
>> there is a notion of a library needing a static interface etc - I
>> propose we agree upon a single access mechanism to the QAPI - who’s
>> existence and stability we have already (I think) agreed upon.
> 
> A stable/static interface is not hard - it doesn't require all
> that much more than exposing a few APIs related to input and
> output of QAPI based JSON docs. This all exists already, you
> wwould just get skipping thue sockets serialization of QMP.
> 

Yes - good - seems we agree on that part :-)

> The biggest stumbling block for QEMU as a library is actually
> the licensing mess. Too much of our code is GPLv2-only, which
> makes it impractical to use as a library in too many use cases.
> Any app that is not GPLv2-only compatible would have to isolate
> QEMU in its own process and talk to it over RPC, at which point
> it has just reinventing QMP again.

To be clear I am not suggesting that we touch, in any way, the license. Stipulating that the application that uses QEMU is GPLv2 is FINE with me. All I’m saying is that If you want to put it in a different process - the way you do that (QMP or other) is then your choice.

Cheers
Mark.


> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/10/21 12:25, Daniel P. Berrangé wrote:
>>> I can't disagree with this. If we carry on trying to evolve vl.c
>>> incrementally we are doomed to be stuck with a horrible starstup
>>> process for enternity (or at least as long as I'll still be
>>> working as QEMU maintainer).
>> ... and if you compare vl.c in 5.2 and now, and consider current vl.c to be
>> horrible, my knowedge of English does not include an adjective to describe
>> the 5.2 state.  Some incremental work_is_  possible or even necessary, and
>> has been done already.
> Right, I'm not saying vl.c hasn't improved, but we're never going
> to get out of the peculiar historical startup ordering rules we
> have today by incremental fixes, without breaking people.

Ok, so the confusion is between a horrible startup process and a 
horrible startup interface.  The latter cannot be improved 
incrementally, the former can (to the point where adding a new 
non-horrible frontend is "trivial").

Thanks,

Paolo

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 12/9/21 20:11, Daniel P. Berrangé wrote:
>>>     They still need to bootstrap a QMP monitor, and for that, CLI is fine
>>>     as long as it's simple and stable.
>
> I would go a step further and say that the QMP monitor socket should
> be created by whoever invoked QEMU and passed down via systemd's
> socket activation protocol, with a fallback to stdin/stdout.
>
>>> 2. CLI and configuration files
>>>
>>>     Human users want a CLI and configuration files.
>>
>> I'd note that Kubernetes doesn't bother with a human CLI at
>> all, and just expects everyone/everything to use JSON/YAML
>> config files. So there's no divide between what syntax humans
>> and machines use - humans just send the config via the CLI
>> tool which uploads it to the REST API that machines use
>> directly.
>
> I agree with Mark here that configuration is beyond QEMU's role.
> QEMU's JSON-based API isn't quite REST, but the idea is the same: the
> user talks to a tool which converts the configuration to QAPI
> commands.  The QAPI commands can be:
>
> - invocations of qmp_* functions in the case of a command-line based
>   configuration
>
> - invocations of QMP commands on a socket if startup is mediated by a
>   management tool
>
> Where Markus and I disagree is whether the two cases should be covered
> by one binary or two.  Personally I don't see much advantage in having
> a single binary, just like I don't care if /bin/ls and /bin/dir are 
> symlinks, or they're separate binaries, or one invokes the other.
>
> What matters to me is the maintainability of the command-line based
> startup code.  Actual experience says that, as more and more options
> are supported via QMP-based configuration, the command-line based
> startup code gets simpler or at least more self-contained.
>
> In the terms of my KVM Forum talk, this is obtained by converting
> "one-off" options (typically global variables) to a combination of a 
> "shortcut" option (typically a qemu_opts_set or qemu_opts_parse
> invocation) and a QOM property or a QMP command.  Examples from recent 
> QEMU versions include -no-shutdown, -no-hpet, -no-acpi.
>
>>> = Ways forward =
>>
>> Just to clarify or remind ourselves of scope.
>> When I think about startup/cli in QEMU, I essentially consider
>> this to mean all softmmu/vl.c code.  Of course there are
>> supporting pieces splattered around, but IMHO 90% of the pain
>> is resulting from the code in the vl.c file.
>>
>>> 1. Improve what we have step by step, managing incompatible changes
>>>
>>>     We tried.  It has eaten substantial time and energy, it has
>>>     complicated code and interfaces further, and we're still nowhere near
>>>     where we need to be.  Does not feel like a practical way forward to
>>>     me.
>>>
>>>     If you don't believe me, read the long version in "Appendix: Why
>>>     further incremental change feels impractical".
>>
>> Essentially death by 1000 cuts.
>
> There are two kinds of incremental changes.  Incremental changes to
> the UI are one thing; incremental changes to the implementations aka 
> refactoring is another.
>
> I agree that further incremental changes to the UI (in the style of
> -preconfig) are impractical, but incremental changes to the 
> implementation are totally practical.  The reason for that is that
> there *is* a way forward for decoupling the options from the command
> line parsing code, and this way forward has *already* proved itself.
>
>> Refering back to my earlier note, when I've suggested this approach
>> in the past, I've basically considered it to mean stop touching
>> softmmu/vl.c, and create a softmmu/vl2.c. The existing binaries
>> remain using vl.c, we get legacy free binaries built from vl2.c
>> Overtime we might get confidence to refactor bits of vl.c to
>> reduce duplication, but I would consider that not a priority
>> at least in the short term.
>
> This is what I have in my tree, though it's called climain.c and
> qmpmain.c. :)  There is one major difference though: refactoring bits
> of vl.c to use more of QMP/QOM *is* the way you add functionality in
> qmpmain.c.
>
> I disagree that vl.c is hard to refactor.  Sure there were some issues
> in 6.0/6.1, but at this point a lot of the hard parts have been done. 
> The conversion of -smp to a QOM option, for example, was largely painless.

I posit that "largely painless to refactor for Paolo" does not imply
"refactoring is largely painless (or even feasible) for mere mortals".

Kidding aside, I acknowledge the progress in vl.c.  My RFC series was
made quite a bit easier by it.

>> On the other hand if we do a good job with the new binaries
>> maybe there ceases to be any reason to keep the old binaries
>> at all after a few years.
>
> I'm not sure if this is possible if the new binaries provide
> exclusively a machine-oriented (JSON) interface.  But as you said, the
> pressure to keep a backwards-compatible command line would be much
> less with a good machine-oriented binary.
>
>>> 3. Start over with the aim to replace the traditional QEMU
>>>
>>>     I don't want an end game where we have two QEMUs, one with bad CLI
>>>     and limited QMP, and one without real CLI and decent QMP.  I want one
>>>     QEMU with decent CLI and decent QMP.
>>
>> Right, so this is essentially what I've just suggested as a possible
>> evoluation of (2). The only difference here is that you're setting
>> the death of the old QEMU as an explicit end gaol in (3), while
>> in (2) it is merely a nice-to-have.
>
> I agree.  Though in my case I'd still go for two binaries: one with
> good CLI and limited QMP, and one with limited CLI and good QMP.

The existing binary provides bad CLI and limited QMP.

Going from limited to good QMP involves reworking the startup code.  I
believe that's easier in a new binary.

Going from bad CLI to good CLI involves incompatible change.
Impractical as long as the CLI is a stable interface.  I believe the
sane way out is a new binary.

However, I can't see why we'd want to put a good CLI in the old binary
then.  We could just as well put it in the new binary, or in a wrapper
program around the new binary.

In both cases, the CLI is layered on top of a QAPI interface.  For an
integrated CLI, it's the C interface specified in the QAPI schema.  For
a wrapper program, it's the QMP interface specified in the schema, which
is in turn layered on top of the C interface specified in the schema.

Or, as ASCII art, first for the wrapper program:

    command line
         |
         |  CLI parser
         v
    internal CLI representation
         |
         |  CLI to QMP translator
         v
    internal QMP representation
         |
         |  QMP formatter
         v                                   wrapper program
--- QMP interface ------------------------------------------
         |                                   wrapped program
         |  QMP parser
         v
    internal QMP representation
         |
         |  QMP dispatcher
         v
    C interface

Cutting the detour from internal QMP representation via QMP interface
right back to internal QMP representation leads to the intergrated CLI:

    command line
         |
         |  CLI parser
         v
    internal CLI representation
         |
         |  CLI to QMP translator
         v
    internal QMP representation
         |
         |  QMP dispatcher
         v
    C interface

Fewer moving parts, easier to debug, same clean layering.

>> In practical terms I don't think (2) and (3) are that different
>> from each other for the first 2 years or so.  A lot can change in
>> that time, so I don't think we need to fixate on a choice of
>> (2) vs (3) upfront, just make it clear that the death of the old
>> binaries is "on the table" as an outcome if we do well.
>
> Agreed, but I would set the beginning of the "2 years or so" at last year...
>
>>> = Appendix: Why further incremental change feels impractical =
>>>
>>> Crafting a big change in small steps sounds great.  It isn't when we
>>> have to make things worse before they can get better, and every step is
>>> painfully slow because it's just too hard, and the end state we aim for
>>> isn't really what we want.
>>
>> I can't disagree with this. If we carry on trying to evolve vl.c
>> incrementally we are doomed to be stuck with a horrible starstup
>> process for enternity (or at least as long as I'll still be
>> working as QEMU maintainer).
>
> ... and if you compare vl.c in 5.2 and now, and consider current vl.c
> to be horrible, my knowedge of English does not include an adjective
> to describe the 5.2 state.  Some incremental work _is_ possible or
> even necessary, and has been done already.

Once again, I acknowledge the progress in vl.c.  I still find it pretty
horrible, but certainly less horrible than it used to be.

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/10/21 14:54, Markus Armbruster wrote:
> I want an open path to a single binary.  Taking years to get there is
> fine.

The single binary is a distraction in my opinion.  Imagine
instead of vl.c you have this in your second binary:

/*
  * This copyright line means that at some point the below actually compiled
  * in my tree (though it was only a stub); I am not fully making it up.
  *
  * Copyright (c) 2020 Red Hat, Inc.
  *
  * SPDX-License-Identifier: GPL-2.0-or-later
  */

#include "qemu/osdep.h"
#include "qemu/rcu.h"
#include "qemu-common.h"
#include "chardev/char.h"
#include "monitor/monitor.h"
#include "qapi/error.h"
#include "qapi/qapi-commands-misc.h"
#include "qapi/qapi-commands-ui.h"
#include "qemu/systemd.h"
#include "sysemu/cpu-timers.h"
#include "sysemu/sysemu.h"
#include "ui/console.h"
#include "hw/qdev-core.h"

static void open_socket_and_monitor(void)
{
     int nfds = check_socket_activation();
     Chardev *chardev;
     if (nfds > 1) {
         error_report("QEMU only supports listening on one socket");
         exit(1);
     }
     if (!nfds) {
         ChardevBackend backend = {
             .type = CHARDEV_BACKEND_KIND_STDIO,
             .u.stdio.data = &(ChardevStdio) {
                 .has_signal = true,
                 .signal = false
             }
         };
         chardev = qemu_chardev_new("#qmp0", TYPE_CHARDEV_STDIO, &backend, NULL, &error_fatal);
     } else {
         ChardevBackend backend = {
            .type = CHARDEV_BACKEND_KIND_SOCKET,
            .u.socket.data = &(ChardevSocket) {
                .addr = &(SocketAddressLegacy) {
                    .type = SOCKET_ADDRESS_LEGACY_KIND_FD,
                    .u.fd.data = &(String){
                        .str = (char *) stringify(FIRST_SOCKET_ACTIVATION_FD)
                    }
                }
            }
         };
         chardev = qemu_chardev_new("#qmp0", TYPE_CHARDEV_SOCKET, &backend, NULL, &error_fatal);
     }
     monitor_init_qmp(chardev, true, &error_fatal);
}

void qemu_init(int argc, char **argv, char **envp)
{
     error_init(argv[0]);
     qemu_init_exec_dir(argv[0]);
     qemu_init_subsystems();

     /* Missing: parse -name, -sandbox, -trace, -L */

     loc_set_none();
     rcu_disable_atfork();
     qemu_init_main_loop(&error_fatal);
     cpu_timers_init();
     open_socket_and_monitor();
     init_displaystate();
     os_setup_signal_handling();
}

This is the ultimate QEMU startup code.  If we can get this code to
actually build a machine, you've reached the point where you don't care
about what is in the command line parser; and consequently you don't care
if there is one binary or two.

Paolo

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Fri, Dec 10, 2021 at 04:26:20PM +0100, Markus Armbruster wrote:
> 
> The existing binary provides bad CLI and limited QMP.
> 
> Going from limited to good QMP involves reworking the startup code.  I
> believe that's easier in a new binary.
> 
> Going from bad CLI to good CLI involves incompatible change.
> Impractical as long as the CLI is a stable interface.  I believe the
> sane way out is a new binary.
> 
> However, I can't see why we'd want to put a good CLI in the old binary
> then.  We could just as well put it in the new binary, or in a wrapper
> program around the new binary.

Having good CLI in a completely new binary is likely to be easier
for users to understand too. The typical pitfall with our existing
binaries is that they provide 4 ways to do the same thing, from
the different points in QEMU's life. This constantly trips up
unsuspecting users and also makes our docs task way more complicated
to think about.

If we're going to have a good CLI, it would ideally only have
one way to do each given task.

No matter what we do we're fighting against a mass of docs
all over the internet talking about 15 years of old QEMU
syntax. If we do a good CLI in a newly named binary, at
least when reading docs, it'll be pretty clear whether
it is talking about the old QEMU or new QEMU binaries,
reducing liklihood of mixing things up.



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Damien Hedde]

On 12/10/21 14:54, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
>> On Thu, Dec 02, 2021 at 07:57:38AM +0100, Markus Armbruster wrote:
>>> = Motivation =
>>>
>>> QEMU startup and initial configuration were designed many years ago for
>>> a much, much simpler QEMU.  They have since changed beyond recognition
>>> to adapt to new needs.  There was no real redesign.  Adaption to new
>>> needs has become more and more difficult.  A recent example for
>>> "difficult" is Damien's "[RFC PATCH v3 0/5] QMP support for
>>> cold-plugging devices".
>>>
>>> I think it's time for a redesign.
>>>
>>>
>>> = What users want for initial configuration =
>>>
>>> 1. QMP only
>>>
>>>     Management applications need to use QMP for monitoring anyway.  They
>>>     may want to use it for initial configuration, too.  Libvirt does.
>>
>> Essentially, as soon as you need to deal with hotplug, you need QMP/QAPI.
>> As soon as you need QMP/QAPI, it is horrible to also need to use something
>> that isn't JSON for the coldplug configuration approach, as you've doubled
>> the number of things to write + test.
> 
> Yes, having to encode the same information in multiple non-trivial ways
> is horrible.
> 
> When using nothing but QMP, you deal with just one encoding.
> 
> With QMP-like configuration files[1], it's one encoding over two
> transports.
> 
> A QAPIfied CLI where the option arguments can be JSON would offer yet
> another transport with only a slight variation of the encoding.
> 
> Any extras need testing.  Okay if their benefits are worth it.
> 
>>>     They still need to bootstrap a QMP monitor, and for that, CLI is fine
>>>     as long as it's simple and stable.
>>
>> Since you mentioned 'simple', allow me to go down a slight
>> tangent...
>>
>> Incidentally, I often wish we didn't use -chardevs anywhere near
>> as much as we do. Chardev was OK when we used it for backing
>> simple guest devices that just had a single host endpoint that
>> was considered permanently connected.
>>
>> Everywhere we've used it for things that really want socket
>> semantics we've created so much pain. The things we've done
>> with chardevs for vhostuser in particular horrify me. I'm
>> glad the block layer resisted the tempetation and just
>> directly used the SocketAddress QAPI type with a QIOChannel
>> objects.
>>
>> I would say the monitor would be better without chardevs
>> too. Having to create multiple monitor instances so that
>> you can have multiple clients is insane. The complexity
>> of course is the need for 'mux' with HMP. If it were not
>> for that, we could use QIOChannel and SocketAddress for
>> the monitor code.
>>
>> Maybe if we one day get HMP fully  separated from QMP such
>> that it is independant code, we can simplify internally.
>> Meanwhile, I think we should consider the QMP CLI at least
>> to only use SocketAddress for config, and secretly turn
>> it into a chardev internally.
> 
> QMP is used in anger pretty much only with Unix domain sockets, and
> maybe Internet domain sockets.  I figure people also use stdio for
> messing around, but something like
> 
>      $ socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>" UNIX-CONNECT:/path/to/qmp-socket
> 
> is actually nicer to use (mostly because we restrict readline to HMP,
> but our readline is homegrown and can't hold a candle to the real thing
> anyway).
> 
>>> 2. CLI and configuration files
>>>
>>>     Human users want a CLI and configuration files.
>>>
>>>     CLI is good for quick tweaks, and to explore.
>>>
>>>     For more permanent, non-trivial configuration, configuration files
>>>     are more suitable, because they are easier to read, edit, and
>>>     document than long command lines.
>>
>> And you can start doing things like "includes" or templating
>> to make the config more manageable / scalable.
> 
> A single configuration file would already be an improvement over today's
> use of CLI, but as configuration gets more complex, a way to stitch
> together multiple configuration files becomes a necessity real fast..
> Even now, we keep a few -readconfig files in docs/config/.
> 
> Configuration will get a lot more complex once we can wire up devices
> from components in configuration instead of C code.
> 
>> It is also nice to not have human and machines in completely
>> separate worlds. Humans will often look at what machines do and
>> then try to replicate that. eg people often ask to see the libvirt
>> QEMU config and then want to run that directly themselves, or use
>> libvirt CLI passthrough to add on features that libvirt does not
>> native support yet.
> 
> Yes!  Obvious and easy paths between the different ways to configure
> stuff are empowering.  I very much want that.
> 
> QemuOpts provides one from CLI to -readconfig: -writeconfig.  Falls well
> short of needs mostly because our needs have long outgrown QemuOpts.
> 
>> As libvirt makes more & more use of QAPI, we are increasing the
>> divide betweeen what machines and humans target.
>>
>> I'd note that Kubernetes doesn't bother with a human CLI at
>> all, and just expects everyone/everything to use JSON/YAML
>> config files. So there's no divide between what syntax humans
>> and machines use - humans just send the config via the CLI
>> tool which uploads it to the REST API that machines use
>> directly.
>>
>>> = What we have for initial configuration =
>>>
>>> Half of 1. and half of 2., satisfying nobody's needs.
>>>
>>> Management applications need to do a lot of non-trivial initial
>>> configuration with the CLI.
>>>
>>> Human users struggle with inconsistent syntax, insufficiently expressive
>>> configuration files, and huge command lines.
>>
>> The QEMU CLI was nice because you could historically do simple
>> stuff really simply eg
>>
>>    qemu-system-x86_64 mydisk.img
>>
>> or
>>
>>    qemu-system-x86_64 -hda mydisk.img -cdrom mydistro.iso
>>
>> The challenge is how CLI usage evolves as you need to finese
>> the config you're using. Using this simple CLI approach, our
>> defaults still largely give you a VM from 1995. If people
>> want a modern guest setup, the simple syntax quickly stops
>> being simple.
>>
>> You very easily get to a point where passing stuff on the
>> CLI gets out of control. IMHO if the CLI is over 100 characters
>> long, a config file is the way to go.
> 
> Yes, we want simple things to be simple, and complicated things to be
> possible.
> 
> Keeping defaults that have long gone bad just to avoid user-visible
> change is an issue that won't go away, but can perhaps be mitigated a
> bit.  More on that below.
> 
>>                                        Given that I really
>> wonder whether direct configuration of hardware on the CLI
>> is worthwhile at all, and users should instead /always/ be
>> using a config file, albeit indirectly.
>>
>>
>> This doesn't mean simple things become harder.
> 
> While ridiculously complicated CLI hardware configuration certainly
> exists and is a problem, dead simple CLI hardware configuration also
> exists.  Doesn't mean your idea is bad, it just raises the bar for it :)
> 
>> I'm thinking of a config file that supports a standard
>> template language supporting variable substitution,
>> loops, conditionals. The CLI then does not need to
>> represent anything related to hardware config schemas
>> at all. Instead it just needs to take the name of a
>> config file and ability to set variables.  We could
>> then ship  some standard configs for the simple cases,
>> which would solve the problem of us being stuck on
>> ancient defaults for the simple CLI configs.
>>
>>    $QEMU  virtiovm.cfg  root=mydisk.img cdrom=mydistro.iso
>>
>> where virtiovm.cfg contains
>>
>>    {
>>       "machine": {
>>         "type": "q35",
>>         "memory": "2G",
>>       },
>>       "disks": [
>>         {
>>           "type": "virtio-blk",
>> 	 "file": @@ VAR root@@
>>         },
>>         @@ IF defined cdrom @@
>>         {
>>           "type": "sata",
>> 	 "media": "cdroM"
>> 	 "file": @@ VAR cdrom @@
>>         }
>>         @@ END @@
>>       ]
>>    }
>>
>> Don't pay close attention to my suggesting JSON syntax
>> here. I just invented something for sake of illustration.
>> A real config would be more complex  than this and follow
>> the QAPI schemas.
>>
>> The key is that the complexity of the config file does
>> not matter so much, because the user isn't directly
>> interacting with it for simple tasks. They just have
>> a nice high level CLI taking a few useful variables.
>> The simple CLI becomes massively more useful than
>> our current simple CLI, because it can support modern
>> VMs of arbitrary complexity while still remaining
>> simple.
> 
> There are also cases where the user's configuration is structurally
> simple, just too long to do comfortably in CLI.  A configuration file
> with comments is better.
> 
> For persistent configuration, writing a configuration file beats writing
> a shell script for me.
> 
> Not arguments against your templating idea.  I'm merely pointing out
> human end users will find (simple!) configuration files useful, too.
> 
>>> = Why startup matters for initial configuration =
>>>
>>> Startup and initial configuration are joined at the hip: the way startup
>>> works limits how initial configuration can work.
>>>
>>> Consider the startup we had before -prefconfig:
>>>
>>> 0. Basic, configuration-independent initialization
>>>
>>> 1. Parse and partially process CLI
>>>
>>> 2. Startup with the remaining CLI processing mixed in
>>>
>>> 3. Startup complete, run main loop
>>>
>>> Note:
>>>
>>> * CLI is processed out of order.
>>>
>>>    Few people understand the order, and only while starting at the code.
>>>    Pretty much every time we mess with the order, we break something.
>>>
>>> * QMP monitors become available only at 3, i.e. after startup is
>>>    complete.  Precludes use of QMP for initial configuration.
>>>
>>> The current startup with -preconfig permits a bit of initial
>>> configuration with QMP, at the cost of complicating startup code and
>>> interfaces.  More in "Appendix: Why further incremental change feels
>>> impractical".
>>>
>>>
>>> = Ways forward =
>>
>> Just to clarify or remind ourselves of scope.
>>
>> When I think about startup/cli in QEMU, I essentially consider
>> this to mean all softmmu/vl.c code.  Of course there are
>> supporting pieces splattered around, but IMHO 90% of the pain
>> is resulting from the code in the vl.c file.
>>
>>> 0. This is too hard, let's go shopping
>>>
>>>     I can't really blame people for not wanting to mess with startup.  It
>>>     *is* hard, and there's plenty of other useful stuff to do.
>>>
>>>     However, I believe we have to mess with it, i.e. this is not actually
>>>     a way forward.
>>
>> Yeah, we hurt ourselves by not being able to move forward with
>> a better approach to startup and configuration.
>>
>>> 1. Improve what we have step by step, managing incompatible changes
>>>
>>>     We tried.  It has eaten substantial time and energy, it has
>>>     complicated code and interfaces further, and we're still nowhere near
>>>     where we need to be.  Does not feel like a practical way forward to
>>>     me.
>>>
>>>     If you don't believe me, read the long version in "Appendix: Why
>>>     further incremental change feels impractical".
>>
>> Essentially death by 1000 cuts.
>>
>>> 2. Start over with a QMP-only QEMU, keep the traditional QEMU forever
>>>
>>>     A QMP-only QEMU isn't just a matter of providing an alternate main()
>>>     where we ditch most of the CLI for QMP replacements.  It takes a
>>>     rework of the startup code as "phase transitions" triggered by QMP
>>>     commands.  Similar to how -preconfig triggers some startup work.
>>>
>>>     To reduce code duplication, we'll probably want to rework the
>>>     traditional QEMU's startup code some, too.
>>
>> Refering back to my earlier note, when I've suggested this approach
>> in the past, I've basically considered it to mean stop touching
>> softmmu/vl.c, and create a softmmu/vl2.c. The existing binaries
>> remain using vl.c, we get legacy free binaries built from vl2.c
>>
>> Overtime we might get confidence to refactor bits of vl.c to
>> reduce duplication, but I would consider that not a priority
>> at least in the short term.
>>
>> Part of the reason why vl.c is so painful is danger of breaking
>> existing apps, especially mgmt tools like libvirt. If we did
>> have a vl.c and vl2.c in parallel, creating separate binaries
>> for each we could finese our support guarantees.
>>
>> ie once we consider the new vl2.c to be feature complete for
>> typical production usage, we could declare that mgmt apps
>> should exclusively use the new binaries, and the old binaries
>> are no longer guaranteed CLI stable. This would unlock thue
>> ability to make more risky changes to vl.c to reduce the
>> duplication / maint burden of two binaries.
>>
>> On the other hand if we do a good job with the new binaries
>> maybe there ceases to be any reason to keep the old binaries
>> at all after a few years.
> 
> I want an open path to a single binary.  Taking years to get there is
> fine.
> 
>>> 3. Start over with the aim to replace the traditional QEMU
>>>
>>>     I don't want an end game where we have two QEMUs, one with bad CLI
>>>     and limited QMP, and one without real CLI and decent QMP.  I want one
>>>     QEMU with decent CLI and decent QMP.
>>>
>>>     I propose that a QEMU with decent CLI and decent QMP isn't really
>>>     harder than a QMP-only QEMU.  CLI is just another transport that
>>>     happens to be one-way, i.e. the QMP commands encoded as CLI options
>>>     can't return a value.  Same for configuration files.  I believe the
>>>     parts that are actually hard are shared with QMP-only: QAPIfying
>>>     initial configuration and reworking the startup code as phase
>>>     transitions.
>>>
>>>     When that decent QEMU is ready, we can deprecate and later drop the
>>>     traditional one.
>>
>> Right, so this is essentially what I've just suggested as a possible
>> evoluation of (2). The only difference here is that you're setting
>> the death of the old QEMU as an explicit end gaol in (3), while
>> in (2) it is merely a nice-to-have.
> 
> I'm not sure we all agree it's nice to have.  See Paolo's reply.
> 
>> In practical terms I don't think (2) and (3) are that different
>> from each other for the first 2 years or so.  A lot can change in
>> that time, so I don't think we need to fixate on a choice of
>> (2) vs (3) upfront, just make it clear that the death of the old
>> binaries is "on the table" as an outcome if we do well.
> 
> Yes, the overlap between (2) and (3) is big enough to defer the decision
> some.
> 
> However, I'd like to start working on (3) earlier, time permitting.
> 
>>> = A design proposal =
>>>
>>> 0. Basic, configuration-independent initialization
>>>
>>> 1. Start main loop
>>>
>>> 2. Feed it CLI left to right.  Each option runs a handler just like each
>>>     QMP command does.
>>>
>>>     Options that read a configuration file inject the file into the feed.
>>>
>>>     Options that create a monitor create it suspended.
>>>
>>>     Options may advance the phase / run state, they may require certain
>>>     phase(s), and semantics may depend on the phase.
>>>
>>> 3. When we're done with CLI, resume any monitors we created.
>>>
>>>     As a convenience, provide an easy way to auto-advance phase / run
>>>     state to "guest runs" right here.  The traditional way would be to
>>>     auto-advance unless the user gives -S.
>>>
>>> 4. Monitors now feed commands to the main loop.  Commands may advance
>>>     the phase / run state, and they may require certain phase(s), and
>>>     semantics may depend on the phase.
>>>
>>>     In particular, command "cont" advances phase / run state to "guest
>>>     runs".
>>>
>>> Users can do as much or as little with the CLI as they want.  A
>>> management application might want to set up a QMP monitor and no more.
>>
>> As illustrated earlier, I'd really like us to consider being a bit
>> more adventurous on the CLI side. I'm convinced that a CLI for
>> directly configurable hardware is doomed to be horrible no matter
>> what, if you try to directly expose all QAPI configuration
>> flexibilty. Whether key/value, JSON, whatever, it will become
>> unmanagable on the CLI because VM hardware config is inherantly
>> complicated.
>>
>> Thus my though that config files or QMP should be the only two
>> places where the full power of QAPI config is exposed. Use CLI
>> as just a way to interact with config files in a simple way
>> with templates.
>>
>> We can really think of QEMU's original simple CLI from 15
>> years ago as being a kind of templating frontend for a config
>> file. The config file just happened to be written as C code,
>> and so not user edittable if going outside the common cases.
> 
> Right.
> 
> Painted with a big brush, there are two kinds of code in hw/: actual
> device emulation, and "wiring".  Both in C, and sometimes in the same .c
> file.
> 
> Doing the "wiring" in configuration instead is less powerful (no longer
> Turing complete[2]), but easier to reason about, maintain and change.
> Change is possible even in the field.  The obvious separation between
> emulation and wiring is a nice bonus.
> 
> The wiring C code supports something not unlike templating: we have a
> number of configuration knobs that deposit something for the wiring C
> code to pick up.
> 
> It's rather limited, though: the "variables" are fixed at compile time.
> 
> Aside: the handling of variables that doesn't get substituted is wildly
> inconsistent.
> 
> Perhaps doing wiring in configuration reduces the maintenance burden to
> a degree where we can mitigate the "defaults gone bad, but no want
> change" issue by offering additional, better things instead of changing
> existing, bad things.  I don't know.
> 

We should be careful with config files, because the configuration parser 
can become a real mess. There are 2 kinds of config files:

1. Imperative: an equivalent of QAPI/QMP command script. (eg like CLI 
config files Markus proposed). These are easy to handle because they 
follow the QMP flow. We do one command after the other, the ordering of 
tasks is clear.

2. Descriptive: A description of the configuration where we describe the 
components and the wiring. This can be really complicated because we 
easily end up with ordering/determinism issues when instantiating them 
afterwards. For example the  configuration parser may have to solve : 
Which device do I create first ? I don't think a configuration parser 
should have to solve such issues, but I'm not sure if we can avoid it.

--
Damien

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Fri, Dec 10, 2021 at 04:26:20PM +0100, Markus Armbruster wrote:
>> 
>> The existing binary provides bad CLI and limited QMP.
>> 
>> Going from limited to good QMP involves reworking the startup code.  I
>> believe that's easier in a new binary.
>> 
>> Going from bad CLI to good CLI involves incompatible change.
>> Impractical as long as the CLI is a stable interface.  I believe the
>> sane way out is a new binary.
>> 
>> However, I can't see why we'd want to put a good CLI in the old binary
>> then.  We could just as well put it in the new binary, or in a wrapper
>> program around the new binary.
>
> Having good CLI in a completely new binary is likely to be easier
> for users to understand too. The typical pitfall with our existing
> binaries is that they provide 4 ways to do the same thing, from
> the different points in QEMU's life. This constantly trips up
> unsuspecting users and also makes our docs task way more complicated
> to think about.

I think it's more often just three: the long one that can do everything,
the short one that can do simple things (and doesn't tell you anything
about the long one), and the bad one you shouldn't use.

> If we're going to have a good CLI, it would ideally only have
> one way to do each given task.

Ideally, the long one plus good defaults suffices.

When we must also have a short one, it should macro-expand into long
one(s), and the user should be able to see the expansion.

> No matter what we do we're fighting against a mass of docs
> all over the internet talking about 15 years of old QEMU
> syntax. If we do a good CLI in a newly named binary, at
> least when reading docs, it'll be pretty clear whether
> it is talking about the old QEMU or new QEMU binaries,
> reducing liklihood of mixing things up.

An advantage that truly matters.

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 12/10/21 14:54, Markus Armbruster wrote:
>> I want an open path to a single binary.  Taking years to get there is
>> fine.
>
> The single binary is a distraction in my opinion.  Imagine
> instead of vl.c you have this in your second binary:

[...]

> static void open_socket_and_monitor(void)
> {
>     int nfds = check_socket_activation();
>     Chardev *chardev;
>     if (nfds > 1) {
>         error_report("QEMU only supports listening on one socket");
>         exit(1);
>     }
>     if (!nfds) {
>         ChardevBackend backend = {
>             .type = CHARDEV_BACKEND_KIND_STDIO,
>             .u.stdio.data = &(ChardevStdio) {
>                 .has_signal = true,
>                 .signal = false
>             }
>         };
>         chardev = qemu_chardev_new("#qmp0", TYPE_CHARDEV_STDIO, &backend, NULL, &error_fatal);
>     } else {
>         ChardevBackend backend = {
>            .type = CHARDEV_BACKEND_KIND_SOCKET,
>            .u.socket.data = &(ChardevSocket) {
>                .addr = &(SocketAddressLegacy) {
>                    .type = SOCKET_ADDRESS_LEGACY_KIND_FD,
>                    .u.fd.data = &(String){
>                        .str = (char *) stringify(FIRST_SOCKET_ACTIVATION_FD)
>                    }
>                }
>            }
>         };
>         chardev = qemu_chardev_new("#qmp0", TYPE_CHARDEV_SOCKET, &backend, NULL, &error_fatal);
>     }
>     monitor_init_qmp(chardev, true, &error_fatal);
> }
>
> void qemu_init(int argc, char **argv, char **envp)
> {
>     error_init(argv[0]);
>     qemu_init_exec_dir(argv[0]);
>     qemu_init_subsystems();
>
>     /* Missing: parse -name, -sandbox, -trace, -L */
>
>     loc_set_none();
>     rcu_disable_atfork();
>     qemu_init_main_loop(&error_fatal);
>     cpu_timers_init();
>     open_socket_and_monitor();
>     init_displaystate();
>     os_setup_signal_handling();
> }
>
> This is the ultimate QEMU startup code.  If we can get this code to
> actually build a machine, you've reached the point where you don't care
> about what is in the command line parser; and consequently you don't care
> if there is one binary or two.

Define "you".  Also explain why it should include me, because I think it
doesn't :)

By when can we have this second binary in master?  Opinion, please, not
promise.

Would you object to me expanding the CLI here to the point where I think
we can deprecate the old binary?

If yes, why?

If no, the file names climain.c and qmpmain.c you mentioned upthread
won't last.  Recommend to pick better names.

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Damien Hedde <damien.hedde@greensocs.com> writes:


[...]

>> Painted with a big brush, there are two kinds of code in hw/: actual
>> device emulation, and "wiring".  Both in C, and sometimes in the same .c
>> file.
>> 
>> Doing the "wiring" in configuration instead is less powerful (no longer
>> Turing complete[2]), but easier to reason about, maintain and change.
>> Change is possible even in the field.  The obvious separation between
>> emulation and wiring is a nice bonus.
>> 
>> The wiring C code supports something not unlike templating: we have a
>> number of configuration knobs that deposit something for the wiring C
>> code to pick up.
>> 
>> It's rather limited, though: the "variables" are fixed at compile time.
>> 
>> Aside: the handling of variables that doesn't get substituted is wildly
>> inconsistent.
>> 
>> Perhaps doing wiring in configuration reduces the maintenance burden to
>> a degree where we can mitigate the "defaults gone bad, but no want
>> change" issue by offering additional, better things instead of changing
>> existing, bad things.  I don't know.
>
> We should be careful with config files, because the configuration
> parser can become a real mess.

Oh yes.

>                                There are 2 kinds of config files:
>
> 1. Imperative: an equivalent of QAPI/QMP command script. (eg like CLI
> config files Markus proposed). These are easy to handle because they 
> follow the QMP flow. We do one command after the other, the ordering
> of tasks is clear.
>
> 2. Descriptive: A description of the configuration where we describe
> the components and the wiring. This can be really complicated because
> we easily end up with ordering/determinism issues when instantiating
> them afterwards. For example the  configuration parser may have to
> solve : Which device do I create first ? I don't think a configuration
> parser should have to solve such issues, but I'm not sure if we can
> avoid it.

Actual instantiation necessarily happens in some order[*].

An imperative configuration dictates the order.

A descriptive configuration leaves it to some planner software.

Any descriptive configuration can therefore be transformed into an
imperative one.  *Unless* the planner *also* breaks up components into
smaller ones that aren't accessible in the imperative configuration.

Example: say the configuration specifies a device and its connections as
a unit.  Say we have two devices A and B, where A has a connection A->B,
and B a connection B->A.  A sufficiently smart planner can then create
A, B, A->B, B->A.  You can't express this imperatively unless connection
B->A can be specified separately from device A.

I propose to start stupid, i.e. with an imperative, low-level
configuration.  Then add smarts as we need them.


[*] Even if you instantiate stuff concurrently for some reason, there
should be at least some conceptual order.

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/13/21 16:19, Markus Armbruster wrote:
> I think it's more often just three: the long one that can do everything,
> the short one that can do simple things (and doesn't tell you anything
> about the long one), and the bad one you shouldn't use.
> 
>> If we're going to have a good CLI, it would ideally only have
>> one way to do each given task.
> Ideally, the long one plus good defaults suffices.

That's hard to do, because the short one typically creates both a 
backend and a frontend.  It should be possible to macro-expand into the 
long one + a -machine property, but that's extremely tedious work.

Paolo

> When we must also have a short one, it should macro-expand into long
> one(s), and the user should be able to see the expansion.
> 

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/13/21 16:28, Markus Armbruster wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>> On 12/10/21 14:54, Markus Armbruster wrote:
>>> I want an open path to a single binary.  Taking years to get there is
>>> fine.
>>
>> The single binary is a distraction in my opinion.  Imagine
>> instead of vl.c you have this in your second binary:
> 
> [...]
> 
>> static void open_socket_and_monitor(void)
>> {
>>      int nfds = check_socket_activation();
>>      Chardev *chardev;
>>      if (nfds > 1) {
>>          error_report("QEMU only supports listening on one socket");
>>          exit(1);
>>      }
>>      if (!nfds) {
>>          ChardevBackend backend = {
>>              .type = CHARDEV_BACKEND_KIND_STDIO,
>>              .u.stdio.data = &(ChardevStdio) {
>>                  .has_signal = true,
>>                  .signal = false
>>              }
>>          };
>>          chardev = qemu_chardev_new("#qmp0", TYPE_CHARDEV_STDIO, &backend, NULL, &error_fatal);
>>      } else {
>>          ChardevBackend backend = {
>>             .type = CHARDEV_BACKEND_KIND_SOCKET,
>>             .u.socket.data = &(ChardevSocket) {
>>                 .addr = &(SocketAddressLegacy) {
>>                     .type = SOCKET_ADDRESS_LEGACY_KIND_FD,
>>                     .u.fd.data = &(String){
>>                         .str = (char *) stringify(FIRST_SOCKET_ACTIVATION_FD)
>>                     }
>>                 }
>>             }
>>          };
>>          chardev = qemu_chardev_new("#qmp0", TYPE_CHARDEV_SOCKET, &backend, NULL, &error_fatal);
>>      }
>>      monitor_init_qmp(chardev, true, &error_fatal);
>> }
>>
>> void qemu_init(int argc, char **argv, char **envp)
>> {
>>      error_init(argv[0]);
>>      qemu_init_exec_dir(argv[0]);
>>      qemu_init_subsystems();
>>
>>      /* Missing: parse -name, -sandbox, -trace, -L */
>>
>>      loc_set_none();
>>      rcu_disable_atfork();
>>      qemu_init_main_loop(&error_fatal);
>>      cpu_timers_init();
>>      open_socket_and_monitor();
>>      init_displaystate();
>>      os_setup_signal_handling();
>> }
>>
>> This is the ultimate QEMU startup code.  If we can get this code to
>> actually build a machine, you've reached the point where you don't care
>> about what is in the command line parser; and consequently you don't care
>> if there is one binary or two.
> 
> Define "you".  Also explain why it should include me, because I think it
> doesn't :)

Impersonal you. :)

> By when can we have this second binary in master?  Opinion, please, not
> promise.

Define "have":

- a binary that builds

- a binary that builds a bootable guest

- a binary that builds any guest that the current well-maintained 
targets can build, using a given (but roughly full-featured) subset of 
options

Estimates for the first are easy (it's in my tree), estimates for the 
second depends on somebody helping (upstreaming -M smp took months 
between me being busy, reviewers being busy, and releases freezing 
development), estimates for the third are hard.

> Would you object to me expanding the CLI here to the point where I think
> we can deprecate the old binary?
> 
> If yes, why?

Yes, for two reasons.

First, because there will be usually differences between the command 
lines as mentioned elsewhere in the thread.  qemu-system-* is a good 
name, but one that is already taken by 15 years of docs using the 
existing command line.

Second, because a command line is really hard to get right as complexity 
increases.  QMP is the way to go to get as clean as possible a 
configuration mechanism.  There *will* be a second set of warts layered 
on top of the above code, and I don't want that.

Paolo

> If no, the file names climain.c and qmpmain.c you mentioned upthread
> won't last.  Recommend to pick better names.
> 
> 

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Mon, Dec 13, 2021 at 06:30:45PM +0100, Paolo Bonzini wrote:
> On 12/13/21 16:19, Markus Armbruster wrote:
> > I think it's more often just three: the long one that can do everything,
> > the short one that can do simple things (and doesn't tell you anything
> > about the long one), and the bad one you shouldn't use.
> > 
> > > If we're going to have a good CLI, it would ideally only have
> > > one way to do each given task.
> > Ideally, the long one plus good defaults suffices.
> 
> That's hard to do, because the short one typically creates both a backend
> and a frontend.  It should be possible to macro-expand into the long one + a
> -machine property, but that's extremely tedious work.

I don't think it has to be hard, we just need to approach the
problem differently to how we've done in the past.

As a general point, I think we should probably change our terminology
here. IMHO using "long" and "short" options leads to the idea that
they are alternatives that are both supported directly by the same
binary and users switch between them depending on what they need.
That's true with current QEMU impl, but I don't think we should go
down that same route again, as it is not at all user friendly and
leads to confusing docs due to multiple ways todo the same thing.

I'd call them low-level and high-level config, to make it clearer
that they're operating at different conceptual levels, and potentially
(ideally even) separate binaries.

I'd like a situation where machines always use low level config
and humans can *always* use the high level config. This implies
the high level config needs to cope with everything that the
low level config can. This might sound contradictory but it is
possible, if the user has ability to customize the mapping.


If we assume the low-level config is the QAPI based JSON, then a
high level config is essentially just a mapping of some strings
into JSON. Looking at it in this light, we can then consider the
high level config problem to simply be a JSON templating problem
and pick an off the shelf solution for that such as Jinja. 


To create our so called "short option" high level API we need to
be able to ship a collection of useful JSON config snippets, and
have a tool which reads some CLI args using them as input variables
to something like the Jinja templating engine, and then launching
the low level QEMU using the template output.


The benefit of using a general purpose JSON templating system in a
separate binary is that we no longer have to solve everything
ourselves. We can provide templates that handle some obvious common
cases. If they don't do quite what a user needs, then they can just
clone our template, tweak it, and carry on using the high level
tool.

IOW, the ability to customize the templates, means that when
launching QEMU the user only ever needs to use the high level
config on the CLI, and yet they still have ability to express
every possible QEMU config. THis is quite a different and better
experiance than today where if a short option doesn't do what
you need, you have to stop using it and pick a completely
different low level config instead.

It also means we can ship a variety of templates with mutually
exclusive defaults eg a template targetting a 1990's machine
defaults, a template targetting defaults for a general purpose
maximum portability VM, and a template targetting defaults
for virtio.

Since it would be separate from the low level QEMU, we can
also decide to use a non-C language for the high level tool.
eg write it in python if we want to use Jinja as templating,
which would also open it up as an interesting thing for a
new set of contributors who would not touch C code.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Mon, Dec 13, 2021 at 06:37:44PM +0100, Paolo Bonzini wrote:
> On 12/13/21 16:28, Markus Armbruster wrote:
> > Would you object to me expanding the CLI here to the point where I think
> > we can deprecate the old binary?
> > 
> > If yes, why?
> 
> Yes, for two reasons.
> 
> First, because there will be usually differences between the command lines
> as mentioned elsewhere in the thread.  qemu-system-* is a good name, but one
> that is already taken by 15 years of docs using the existing command line.

T

Lets pick naming to make it clearer who/what each binary is targetted
towards. e.g.

  - /usr/bin/qemu-buildvm-$TARGET   for the low level binary that just
    speaks QMP on stdio / passed in pre-opened socket, targetted
    at mgmt apps and needs a series of commands to build a VM up
    from scratch

  - /usr/bin/qemu (or /usr/bin/qemu-vm) - for a high level binary that
    targets humans and uses a templating system to expose a friendly
    simple config, that internally invokes whichever target specific
    /usr/bin/qemu-buildvm-$TARGET is implied, plus any other vhost-user
    backends, or whatever other helper processes it needs

> Second, because a command line is really hard to get right as complexity
> increases.  QMP is the way to go to get as clean as possible a configuration
> mechanism.  There *will* be a second set of warts layered on top of the
> above code, and I don't want that.

Turning the high level / short config into a general purpose templating
problem, strictly separated from the low level binary using QMP, means
gives us a more flexible way to live with the warts IMHO.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/13/21 19:07, Daniel P. Berrangé wrote:
>    - /usr/bin/qemu (or /usr/bin/qemu-vm) - for a high level binary that
>      targets humans and uses a templating system to expose a friendly
>      simple config, that internally invokes whichever target specific
>      /usr/bin/qemu-buildvm-$TARGET is implied, plus any other vhost-user
>      backends, or whatever other helper processes it needs

Adding vhost-user backends and helper processes means one of two things: 
either you are not going to support hotplug, or you are going to redo 
libvirtd with a QMP-based RPC.

Paolo

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Mon, Dec 13, 2021 at 07:37:49PM +0100, Paolo Bonzini wrote:
> On 12/13/21 19:07, Daniel P. Berrangé wrote:
> >    - /usr/bin/qemu (or /usr/bin/qemu-vm) - for a high level binary that
> >      targets humans and uses a templating system to expose a friendly
> >      simple config, that internally invokes whichever target specific
> >      /usr/bin/qemu-buildvm-$TARGET is implied, plus any other vhost-user
> >      backends, or whatever other helper processes it needs
> 
> Adding vhost-user backends and helper processes means one of two things:
> either you are not going to support hotplug, or you are going to redo
> libvirtd with a QMP-based RPC.

I can't say I thought about the helper process idea too much. I was not
trying to imply anything beyond the fact that I think at the high level
human users should only have interact with a single QEMU binary, not
per-target binaries, and also not worry about helper binaries if they
happen to be used as impl details.

If it were possible to keep auto-spawning of helpers at the high level
that feels cleaner, so that the low level only has to worry about a
single way of doing things. If that is too hard for hotplug though,
so be it, leave auto-spawning in the low level.

Any high level thing would need a monitor of some kind since there'll
always be a need for humans to interrogate the QEMU to some degree. If
we're trying to keep the monitor high level though, we'd want something
closer to HMP. Perhaps again have an HMP that's based around a template
engine that spits out QMP commands, and can extract bits of the reply
for pretty printing, so again we're not writing C code for each new
command that we care to support, just simple template snippets, that
users can again customize if needed.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

[-- Attachment #1: Type: text/plain, Size: 2287 bytes --]



> On 13 Dec 2021, at 18:59, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> …. we no longer have to solve everything
> Ourselves. 

I support this sentiment.

Lets re-factor the code so people can build what they need using an API.
Actually, ‘QEMU’ only need support the existing CLI, and provide a suitable internal API.
If that API was relatively stable, that would help those (few) who maintain a different startup mechanism (but its only a ’nice to have’). (Making that convenient, as Paolo has show, would also be ’nice to have’).

If Paolo (sorry to pick on you :-) ) wants a pink spotted dog, then - surely - let him have it. YES it’s a separate binary. YES it has a different CLI…. So what? It has nothing to do with QEMU itself, Paolo can maintain it, out of the QEMU tree. QEMU should not attempt to directly support all use cases -  those who have the use case should support them. (e.g. We should not be arguing about what JSON parsing mechanism to use, how command lines could be better,  or configuration languages - thats for the use case provider to worry about. )

That leaves us needing a ‘full’ API. Surely we should be arguing about that - what “full” means, how the QAPI should work, specifically how phases should work, etc. How to move QAPI forward.

To help, I do think it’s important that we include use cases into QEMU (as it stands) in order to build test cases and to effectively have the use cases coded; but only as a ’step’. The plan to refactor needs to be there from the start. [Once refactored, ‘unstable’ features used to demonstrate use cases can be removed of course]

Of course, ‘Refactoring the code’ and getting the CLI to work with the ‘new’ QAPI will be hard  - but each and every scheme I’ve seen so far has some pain involved. If we know this is the goal, then we can start planning how to get there. (And GreenSocs will commit to help if we can).

(Of course, one day, a ‘use case provider’ might come along with the best CLI ever, and the best config language ever, and we will all bow down and use it…. Until that day…. Can we focus on the key for QEMU, providing a ‘full’ QAPI tat can support all use cases (and the existing CLI)).

Cheers
Mark.


[-- Attachment #2: Type: text/html, Size: 5220 bytes --]

Meeting today?

[Mark Burton]

I realise it’s very short notice, but what about having a discussion today at 15:00 ?
Cheers
Mark.


> On 13 Dec 2021, at 19:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Mon, Dec 13, 2021 at 07:37:49PM +0100, Paolo Bonzini wrote:
>> On 12/13/21 19:07, Daniel P. Berrangé wrote:
>>>   - /usr/bin/qemu (or /usr/bin/qemu-vm) - for a high level binary that
>>>     targets humans and uses a templating system to expose a friendly
>>>     simple config, that internally invokes whichever target specific
>>>     /usr/bin/qemu-buildvm-$TARGET is implied, plus any other vhost-user
>>>     backends, or whatever other helper processes it needs
>> 
>> Adding vhost-user backends and helper processes means one of two things:
>> either you are not going to support hotplug, or you are going to redo
>> libvirtd with a QMP-based RPC.
> 
> I can't say I thought about the helper process idea too much. I was not
> trying to imply anything beyond the fact that I think at the high level
> human users should only have interact with a single QEMU binary, not
> per-target binaries, and also not worry about helper binaries if they
> happen to be used as impl details.
> 
> If it were possible to keep auto-spawning of helpers at the high level
> that feels cleaner, so that the low level only has to worry about a
> single way of doing things. If that is too hard for hotplug though,
> so be it, leave auto-spawning in the low level.
> 
> Any high level thing would need a monitor of some kind since there'll
> always be a need for humans to interrogate the QEMU to some degree. If
> we're trying to keep the monitor high level though, we'd want something
> closer to HMP. Perhaps again have an HMP that's based around a template
> engine that spits out QMP commands, and can extract bits of the reply
> for pretty printing, so again we're not writing C code for each new
> command that we care to support, just simple template snippets, that
> users can again customize if needed.
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Meeting today?

[Markus Armbruster]

Mark Burton <mark.burton@greensocs.com> writes:

> I realise it’s very short notice, but what about having a discussion today at 15:00 ?

I have a conflict today.  I could try to reschedule, but I'd prefer to
talk next week instead.  Less stress, better prep.

Re: Meeting today?

[Mark Burton]

Works for me
Cheers
Mark.


> On 14 Dec 2021, at 12:37, Markus Armbruster <armbru@redhat.com> wrote:
> 
> Mark Burton <mark.burton@greensocs.com> writes:
> 
>> I realise it’s very short notice, but what about having a discussion today at 15:00 ?
> 
> I have a conflict today.  I could try to reschedule, but I'd prefer to
> talk next week instead.  Less stress, better prep.
> 

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 12/13/21 16:28, Markus Armbruster wrote:
>> Paolo Bonzini <pbonzini@redhat.com> writes:
>> 
>>> On 12/10/21 14:54, Markus Armbruster wrote:
>>>> I want an open path to a single binary.  Taking years to get there is
>>>> fine.
>>>
>>> The single binary is a distraction in my opinion.  Imagine
>>> instead of vl.c you have this in your second binary:

[...]

>>> This is the ultimate QEMU startup code.  If we can get this code to
>>> actually build a machine, you've reached the point where you don't care
>>> about what is in the command line parser; and consequently you don't care
>>> if there is one binary or two.
>> 
>> Define "you".  Also explain why it should include me, because I think it
>> doesn't :)
>
> Impersonal you. :)

Unfortunate choice of a word.

>> By when can we have this second binary in master?  Opinion, please, not
>> promise.
>
> Define "have":
>
> - a binary that builds
>
> - a binary that builds a bootable guest
>
> - a binary that builds any guest that the current well-maintained
>   targets can build, using a given (but roughly full-featured) subset
>  of options
>
> Estimates for the first are easy (it's in my tree), estimates for the
> second depends on somebody helping (upstreaming -M smp took months 
> between me being busy, reviewers being busy, and releases freezing
> development), estimates for the third are hard.

Thanks.

>> Would you object to me expanding the CLI here to the point where I think
>> we can deprecate the old binary?
>> 
>> If yes, why?
>
> Yes, for two reasons.
>
> First, because there will be usually differences between the command
> lines as mentioned elsewhere in the thread.  qemu-system-* is a good 
> name, but one that is already taken by 15 years of docs using the
> existing command line.

A new CLI is pointless unless there are differences to the old one.

It is unadvisable unless we can eventually retire the old one.

While they coexist, the old binary name should use the old CLI, to
reduce confusion.

> Second, because a command line is really hard to get right as
> complexity increases.  QMP is the way to go to get as clean as
> possible a configuration mechanism.  There *will* be a second set of
> warts layered on top of the above code, and I don't want that.

We do not have consensus.  We may have misunderstandings.

Let's start with where we (hopefully) agree:

* We need a single, cohesive, low-level interface suitable for
  management applications.

* The existing interface is specified in QAPI.  Its concrete transport
  is QMP.

* The existing interface is not complete: certain things can only be
  done with the CLI.

* The existing transport is not available early enough to permit
  completing the interface.

* Fixing that involves a rework of startup.

* Reworking the existing startup and managing incompatible changes is
  impractical, and likely to make the mess we have on our hands worse.

* A new binary sidesteps the need to manage incompatible change.

Any objections so far?

Now let me make a few more points:

* Single, cohesive interface does not require single transport.  In
  fact, we already have two: QMP and the (internal) C interface.

* QMP encodes the abstract interface in JSON, and offers the result on a
  Unix domain socket[1].

* The (internal) C interface encodes the abstract interface as a set of
  C data types and functions.

* Consider a configuration file transport that encodes the abstract
  interface in JSON.  The only wart this adds is syntax that is
  arguiably ill-suited to the purpose.  More suitable syntax exists.

* Similar for CLI.

* To get a "a second set of warts layered on top", we actually have to
  layer something on top that isn't utterly trivial.  Like a
  higher-level interface.  The "second set of warts" objection does not
  apply to (sane) transports.

* We already layer an interface on top: HMP[2].  It has its warts.

* The old CLI is partly layered on QMP, partly on HMP, and partly on
  internal C interfaces.  It's full of warts.

* Management applications are not the only users that matter.  Humans
  matter.  Simple programs like ad hoc scripts matter.

Objections to these?

[...]


[1] Actually a QEMU character device now, but let's ignore that.

[2] Except where we choose to bypass QMP, but that's unimportant here.

Re: Meeting today?

[Daniel P. Berrangé]

On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
> Mark Burton <mark.burton@greensocs.com> writes:
> 
> > I realise it’s very short notice, but what about having a discussion today at 15:00 ?
> 
> I have a conflict today.  I could try to reschedule, but I'd prefer to
> talk next week instead.  Less stress, better prep.

I fear we've run out of time for this year if we want all interested
parties to be able to attend.  I'll be off on PTO from end of this
week until the new year, and I know alot of folk are doing similar.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

> On 14 Dec 2021, at 12:48, Markus Armbruster <armbru@redhat.com> wrote:
> 
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
>> On 12/13/21 16:28, Markus Armbruster wrote:
>>> Paolo Bonzini <pbonzini@redhat.com> writes:
>>> 
>>>> On 12/10/21 14:54, Markus Armbruster wrote:
>>>>> I want an open path to a single binary.  Taking years to get there is
>>>>> fine.
>>>> 
>>>> The single binary is a distraction in my opinion.  Imagine
>>>> instead of vl.c you have this in your second binary:
> 
> [...]
> 
>>>> This is the ultimate QEMU startup code.  If we can get this code to
>>>> actually build a machine, you've reached the point where you don't care
>>>> about what is in the command line parser; and consequently you don't care
>>>> if there is one binary or two.
>>> 
>>> Define "you".  Also explain why it should include me, because I think it
>>> doesn't :)
>> 
>> Impersonal you. :)
> 
> Unfortunate choice of a word.
> 
>>> By when can we have this second binary in master?  Opinion, please, not
>>> promise.
>> 
>> Define "have":
>> 
>> - a binary that builds
>> 
>> - a binary that builds a bootable guest
>> 
>> - a binary that builds any guest that the current well-maintained
>>  targets can build, using a given (but roughly full-featured) subset
>> of options
>> 
>> Estimates for the first are easy (it's in my tree), estimates for the
>> second depends on somebody helping (upstreaming -M smp took months 
>> between me being busy, reviewers being busy, and releases freezing
>> development), estimates for the third are hard.
> 
> Thanks.
> 
>>> Would you object to me expanding the CLI here to the point where I think
>>> we can deprecate the old binary?
>>> 
>>> If yes, why?
>> 
>> Yes, for two reasons.
>> 
>> First, because there will be usually differences between the command
>> lines as mentioned elsewhere in the thread.  qemu-system-* is a good 
>> name, but one that is already taken by 15 years of docs using the
>> existing command line.
> 
> A new CLI is pointless unless there are differences to the old one.
> 
> It is unadvisable unless we can eventually retire the old one.
> 
> While they coexist, the old binary name should use the old CLI, to
> reduce confusion.
> 
>> Second, because a command line is really hard to get right as
>> complexity increases.  QMP is the way to go to get as clean as
>> possible a configuration mechanism.  There *will* be a second set of
>> warts layered on top of the above code, and I don't want that.
> 
> We do not have consensus.  We may have misunderstandings.
> 
> Let's start with where we (hopefully) agree:
> 
> * We need a single, cohesive, low-level interface suitable for
>  management applications.
> 
> * The existing interface is specified in QAPI.  Its concrete transport
>  is QMP.
> 
> * The existing interface is not complete: certain things can only be
>  done with the CLI.
> 
> * The existing transport is not available early enough to permit
>  completing the interface.
> 
> * Fixing that involves a rework of startup.
> 
> * Reworking the existing startup and managing incompatible changes is
>  impractical, and likely to make the mess we have on our hands worse.

For “Completing” the interface, I agree. 
To add a certain number of use cases - many of those can be (have been - aka preconfig) done, if with some degree of unpleasant-ness NOW without full re-working. That would give us test cases that we can subsequently use to test against as we move forward.

> 
> * A new binary sidesteps the need to manage incompatible change.
> 
> Any objections so far?
> 
> Now let me make a few more points:
> 
> * Single, cohesive interface does not require single transport.  In
>  fact, we already have two: QMP and the (internal) C interface.
> 
> * QMP encodes the abstract interface in JSON, and offers the result on a
>  Unix domain socket[1].
> 
> * The (internal) C interface encodes the abstract interface as a set of
>  C data types and functions.
> 
> * Consider a configuration file transport that encodes the abstract
>  interface in JSON.  The only wart this adds is syntax that is
>  arguiably ill-suited to the purpose.  More suitable syntax exists.
> 
> * Similar for CLI.
> 
> * To get a "a second set of warts layered on top", we actually have to
>  layer something on top that isn't utterly trivial.  Like a
>  higher-level interface.  The "second set of warts" objection does not
>  apply to (sane) transports.
> 
> * We already layer an interface on top: HMP[2].  It has its warts.
> 
> * The old CLI is partly layered on QMP, partly on HMP, and partly on
>  internal C interfaces.  It's full of warts.
> 
> * Management applications are not the only users that matter.  Humans
>  matter.  Simple programs like ad hoc scripts matter.
> 
(Unless one considers that a ‘human’ and/or ’script’ interface would just be ‘yet another management interface’…. And can/should be relegated to Somebody Else’s Problem)

Cheers
Mark.


> Objections to these?
> 
> [...]
> 
> 
> [1] Actually a QEMU character device now, but let's ignore that.
> 
> [2] Except where we choose to bypass QMP, but that's unimportant here.
> 

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Mon, Dec 13, 2021 at 09:22:14PM +0100, Mark Burton wrote:
> 
> 
> > On 13 Dec 2021, at 18:59, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > …. we no longer have to solve everything
> > Ourselves. 
> 
> I support this sentiment.
> 
> Lets re-factor the code so people can build what they need using an API.
> Actually, ‘QEMU’ only need support the existing CLI, and provide a suitable internal API.
> If that API was relatively stable, that would help those (few) who maintain a different startup mechanism (but its only a ’nice to have’). (Making that convenient, as Paolo has show, would also be ’nice to have’).

To be clear I do strongly believe that the QEMU project needs
to deliver the higher level simplified interface too. I just
want that higher level interface to be flexible enough to
let end users expand on what it offers, without having to
write C code nor having to switch entirely to the low level
interface like we do today.

IOW, QEMU needs to deliver more than just a low level building
block API.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

> On 14 Dec 2021, at 14:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Mon, Dec 13, 2021 at 09:22:14PM +0100, Mark Burton wrote:
>> 
>> 
>>> On 13 Dec 2021, at 18:59, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> 
>>> …. we no longer have to solve everything
>>> Ourselves. 
>> 
>> I support this sentiment.
>> 
>> Lets re-factor the code so people can build what they need using an API.
>> Actually, ‘QEMU’ only need support the existing CLI, and provide a suitable internal API.
>> If that API was relatively stable, that would help those (few) who maintain a different startup mechanism (but its only a ’nice to have’). (Making that convenient, as Paolo has show, would also be ’nice to have’).
> 
> To be clear I do strongly believe that the QEMU project needs
> to deliver the higher level simplified interface too. I just
> want that higher level interface to be flexible enough to
> let end users expand on what it offers, without having to
> write C code nor having to switch entirely to the low level
> interface like we do today.
> 
> IOW, QEMU needs to deliver more than just a low level building
> block API.

Why?
Clearly it would be nice if “higher level” interfaceS existed in the world. Clearly QEMU could provide one, two, or many. But, why do you think QEMU ‘must’ provide them?

Cheers
Mark.


> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Tue, Dec 14, 2021 at 02:11:11PM +0100, Mark Burton wrote:
> 
> 
> > On 14 Dec 2021, at 14:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Mon, Dec 13, 2021 at 09:22:14PM +0100, Mark Burton wrote:
> >> 
> >> 
> >>> On 13 Dec 2021, at 18:59, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>> 
> >>> …. we no longer have to solve everything
> >>> Ourselves. 
> >> 
> >> I support this sentiment.
> >> 
> >> Lets re-factor the code so people can build what they need using an API.
> >> Actually, ‘QEMU’ only need support the existing CLI, and provide a suitable internal API.
> >> If that API was relatively stable, that would help those (few) who maintain a different startup mechanism (but its only a ’nice to have’). (Making that convenient, as Paolo has show, would also be ’nice to have’).
> > 
> > To be clear I do strongly believe that the QEMU project needs
> > to deliver the higher level simplified interface too. I just
> > want that higher level interface to be flexible enough to
> > let end users expand on what it offers, without having to
> > write C code nor having to switch entirely to the low level
> > interface like we do today.
> > 
> > IOW, QEMU needs to deliver more than just a low level building
> > block API.
> 
> Why?
> Clearly it would be nice if “higher level” interfaceS existed in
> the world. Clearly QEMU could provide one, two, or many. But, why
> do you think QEMU ‘must’ provide them?

To serve our users who are not all wanting to be use a management
layer. They want to be using a simple binary to spin up adhoc
VMs. This is the reason why we've kept most of the short option
CLI args in the existing QEMU binaries, despite having more
comprehensive low level replacement args. 

If we just declare we're not going to provide this simple binary
any more, then we're just casting these users adrift. This in
effect means they'll just carry on existing the historical QEMU
binaries and we'll never be able to eliminate them, so we'll be
maintaining two things forever.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

> On 14 Dec 2021, at 14:21, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Tue, Dec 14, 2021 at 02:11:11PM +0100, Mark Burton wrote:
>> 
>> 
>>> On 14 Dec 2021, at 14:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> 
>>> On Mon, Dec 13, 2021 at 09:22:14PM +0100, Mark Burton wrote:
>>>> 
>>>> 
>>>>> On 13 Dec 2021, at 18:59, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>>> 
>>>>> …. we no longer have to solve everything
>>>>> Ourselves. 
>>>> 
>>>> I support this sentiment.
>>>> 
>>>> Lets re-factor the code so people can build what they need using an API.
>>>> Actually, ‘QEMU’ only need support the existing CLI, and provide a suitable internal API.
>>>> If that API was relatively stable, that would help those (few) who maintain a different startup mechanism (but its only a ’nice to have’). (Making that convenient, as Paolo has show, would also be ’nice to have’).
>>> 
>>> To be clear I do strongly believe that the QEMU project needs
>>> to deliver the higher level simplified interface too. I just
>>> want that higher level interface to be flexible enough to
>>> let end users expand on what it offers, without having to
>>> write C code nor having to switch entirely to the low level
>>> interface like we do today.
>>> 
>>> IOW, QEMU needs to deliver more than just a low level building
>>> block API.
>> 
>> Why?
>> Clearly it would be nice if “higher level” interfaceS existed in
>> the world. Clearly QEMU could provide one, two, or many. But, why
>> do you think QEMU ‘must’ provide them?
> 
> To serve our users who are not all wanting to be use a management
> layer. They want to be using a simple binary to spin up adhoc
> VMs. This is the reason why we've kept most of the short option
> CLI args in the existing QEMU binaries, despite having more
> comprehensive low level replacement args. 

So - there are
a) uses today that use the CLI as it exists.
b) users who might prefer a better interface if that was made available - but, again, that doesn’t require QEMU itself to do anything. If we provide a low-level API, and somebody else (you for instance) provides a ’nice’ ‘friendly’ CLI or config file reader - those users would be happy.

I still dont see why QEMU itself needs to provide this ‘high level’ thing. 

I think we all agree (correct me if I’m wrong) :
* We all want a low level interface
* We all want the current CLI to be supported (at least for now, though it could change in time)
* We all want the CLI to be based on the low level interface

I’m just asking why we ALSO want to support “yet another high level interface”….

Is the argument about whether we re-implement the existing CLI ontop of the ‘low level’ API (aka QAPI)?

> 
> If we just declare we're not going to provide this simple binary
> any more, then we're just casting these users adrift. This in

“Any more” - Are you talking about the existing CLI users?

> effect means they'll just carry on existing the historical QEMU
> binaries and we'll never be able to eliminate them, so we'll be
> maintaining two things forever.

A CLI and the low level interface? - Yes? Can we remove the CLI and only support the low-level interface ? But here you seem to be arguing against yourself, so I guess I misunderstood.

Cheers
Mark.

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Tue, Dec 14, 2021 at 02:36:26PM +0100, Mark Burton wrote:
> 
> 
> > On 14 Dec 2021, at 14:21, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Tue, Dec 14, 2021 at 02:11:11PM +0100, Mark Burton wrote:
> >> 
> >> 
> >>> On 14 Dec 2021, at 14:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>> 
> >>> On Mon, Dec 13, 2021 at 09:22:14PM +0100, Mark Burton wrote:
> >>>> 
> >>>> 
> >>>>> On 13 Dec 2021, at 18:59, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>>>> 
> >>>>> …. we no longer have to solve everything
> >>>>> Ourselves. 
> >>>> 
> >>>> I support this sentiment.
> >>>> 
> >>>> Lets re-factor the code so people can build what they need using an API.
> >>>> Actually, ‘QEMU’ only need support the existing CLI, and provide a suitable internal API.
> >>>> If that API was relatively stable, that would help those (few) who maintain a different startup mechanism (but its only a ’nice to have’). (Making that convenient, as Paolo has show, would also be ’nice to have’).
> >>> 
> >>> To be clear I do strongly believe that the QEMU project needs
> >>> to deliver the higher level simplified interface too. I just
> >>> want that higher level interface to be flexible enough to
> >>> let end users expand on what it offers, without having to
> >>> write C code nor having to switch entirely to the low level
> >>> interface like we do today.
> >>> 
> >>> IOW, QEMU needs to deliver more than just a low level building
> >>> block API.
> >> 
> >> Why?
> >> Clearly it would be nice if “higher level” interfaceS existed in
> >> the world. Clearly QEMU could provide one, two, or many. But, why
> >> do you think QEMU ‘must’ provide them?
> > 
> > To serve our users who are not all wanting to be use a management
> > layer. They want to be using a simple binary to spin up adhoc
> > VMs. This is the reason why we've kept most of the short option
> > CLI args in the existing QEMU binaries, despite having more
> > comprehensive low level replacement args. 
> 
> So - there are
> a) uses today that use the CLI as it exists.
> b) users who might prefer a better interface if that was made available - but, again, that doesn’t require QEMU itself to do anything. If we provide a low-level API, and somebody else (you for instance) provides a ’nice’ ‘friendly’ CLI or config file reader - those users would be happy.
> 
> I still dont see why QEMU itself needs to provide this ‘high level’ thing.

The QEMU project has provided this high level CLI itself historically.
In previous discussions about dropping the simplified options, leaving
only the QAPI based options, that idea has always been rejected by a
non-trivial number of QEMU maintainers who consider it a core part of
our deliverable as a project.

> 
> I think we all agree (correct me if I’m wrong) :
> * We all want a low level interface
> * We all want the current CLI to be supported (at least for now, though it could change in time)
> * We all want the CLI to be based on the low level interface
> 
> I’m just asking why we ALSO want to support “yet another high level interface”….

You're arguing for a significant change in the scope of what QEMU
delivers to its users, punting a bunch of functionality to be
someone else's problem.


> > If we just declare we're not going to provide this simple binary
> > any more, then we're just casting these users adrift. This in
> 
> “Any more” - Are you talking about the existing CLI users?

Yes.

> > effect means they'll just carry on existing the historical QEMU
> > binaries and we'll never be able to eliminate them, so we'll be
> > maintaining two things forever.
> 
> A CLI and the low level interface? - Yes? Can we remove the CLI
> and only support the low-level interface ? But here you seem to
> be arguing against yourself, so I guess I misunderstood.

The current qemu-system-$TARGET emulators have a variety of
CLI args, some low level QAPI driven, and some higher level
simplified args. Mgmt app consumers tend to use the former,
while our humans userbase tends to use the latter.  I'm
saying we need to carry on serving both userbases, unless
we get the QEMU maintainers in general to agree to a pretty
significant change in what the project delivers. Based on
previous discussions, I'm doubtful we'll get agreement to
do that.

So if we're going to successfully introduce replacement
for qemu-system-$TARGET, we need to cater for both userbases,
albeit we can do so with separate binaries, rather than trying
to cram low level and high level into the same.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

I think we’re talking at cross purposes, and probably we agree (not sure). I’ll top quote to try and explain my point of view.

I think there are two discussions being mixed:
1/ A discussion about improving the CLI. (Having a new one, etc etc)
2/ A discussion about supporting a low level, and complete, API that can be used by “management layers” (QAPI).

I think this also gets mixed up with the discussion on migrating the CLI to use the low level API.

I want to focus on the low level API. 

I dont see why we’re discussing a ‘high level’ thing when, for now, we have to support the CLI, and we have work to do on QAPI. 

If somebody wants to build a new CLI, with a new ‘high level’ interface, using QAPI - let them!

Cheers
Mark.



> On 14 Dec 2021, at 14:48, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Tue, Dec 14, 2021 at 02:36:26PM +0100, Mark Burton wrote:
>> 
>> 
>>> On 14 Dec 2021, at 14:21, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> 
>>> On Tue, Dec 14, 2021 at 02:11:11PM +0100, Mark Burton wrote:
>>>> 
>>>> 
>>>>> On 14 Dec 2021, at 14:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>>> 
>>>>> On Mon, Dec 13, 2021 at 09:22:14PM +0100, Mark Burton wrote:
>>>>>> 
>>>>>> 
>>>>>>> On 13 Dec 2021, at 18:59, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>>>>> 
>>>>>>> …. we no longer have to solve everything
>>>>>>> Ourselves. 
>>>>>> 
>>>>>> I support this sentiment.
>>>>>> 
>>>>>> Lets re-factor the code so people can build what they need using an API.
>>>>>> Actually, ‘QEMU’ only need support the existing CLI, and provide a suitable internal API.
>>>>>> If that API was relatively stable, that would help those (few) who maintain a different startup mechanism (but its only a ’nice to have’). (Making that convenient, as Paolo has show, would also be ’nice to have’).
>>>>> 
>>>>> To be clear I do strongly believe that the QEMU project needs
>>>>> to deliver the higher level simplified interface too. I just
>>>>> want that higher level interface to be flexible enough to
>>>>> let end users expand on what it offers, without having to
>>>>> write C code nor having to switch entirely to the low level
>>>>> interface like we do today.
>>>>> 
>>>>> IOW, QEMU needs to deliver more than just a low level building
>>>>> block API.
>>>> 
>>>> Why?
>>>> Clearly it would be nice if “higher level” interfaceS existed in
>>>> the world. Clearly QEMU could provide one, two, or many. But, why
>>>> do you think QEMU ‘must’ provide them?
>>> 
>>> To serve our users who are not all wanting to be use a management
>>> layer. They want to be using a simple binary to spin up adhoc
>>> VMs. This is the reason why we've kept most of the short option
>>> CLI args in the existing QEMU binaries, despite having more
>>> comprehensive low level replacement args. 
>> 
>> So - there are
>> a) uses today that use the CLI as it exists.
>> b) users who might prefer a better interface if that was made available - but, again, that doesn’t require QEMU itself to do anything. If we provide a low-level API, and somebody else (you for instance) provides a ’nice’ ‘friendly’ CLI or config file reader - those users would be happy.
>> 
>> I still dont see why QEMU itself needs to provide this ‘high level’ thing.
> 
> The QEMU project has provided this high level CLI itself historically.
> In previous discussions about dropping the simplified options, leaving
> only the QAPI based options, that idea has always been rejected by a
> non-trivial number of QEMU maintainers who consider it a core part of
> our deliverable as a project.
> 
>> 
>> I think we all agree (correct me if I’m wrong) :
>> * We all want a low level interface
>> * We all want the current CLI to be supported (at least for now, though it could change in time)
>> * We all want the CLI to be based on the low level interface
>> 
>> I’m just asking why we ALSO want to support “yet another high level interface”….
> 
> You're arguing for a significant change in the scope of what QEMU
> delivers to its users, punting a bunch of functionality to be
> someone else's problem.
> 
> 
>>> If we just declare we're not going to provide this simple binary
>>> any more, then we're just casting these users adrift. This in
>> 
>> “Any more” - Are you talking about the existing CLI users?
> 
> Yes.
> 
>>> effect means they'll just carry on existing the historical QEMU
>>> binaries and we'll never be able to eliminate them, so we'll be
>>> maintaining two things forever.
>> 
>> A CLI and the low level interface? - Yes? Can we remove the CLI
>> and only support the low-level interface ? But here you seem to
>> be arguing against yourself, so I guess I misunderstood.
> 
> The current qemu-system-$TARGET emulators have a variety of
> CLI args, some low level QAPI driven, and some higher level
> simplified args. Mgmt app consumers tend to use the former,
> while our humans userbase tends to use the latter.  I'm
> saying we need to carry on serving both userbases, unless
> we get the QEMU maintainers in general to agree to a pretty
> significant change in what the project delivers. Based on
> previous discussions, I'm doubtful we'll get agreement to
> do that.
> 
> So if we're going to successfully introduce replacement
> for qemu-system-$TARGET, we need to cater for both userbases,
> albeit we can do so with separate binaries, rather than trying
> to cram low level and high level into the same.
> 
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Meeting today?

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
>> Mark Burton <mark.burton@greensocs.com> writes:
>> 
>> > I realise it’s very short notice, but what about having a discussion today at 15:00 ?
>> 
>> I have a conflict today.  I could try to reschedule, but I'd prefer to
>> talk next week instead.  Less stress, better prep.
>
> I fear we've run out of time for this year if we want all interested
> parties to be able to attend.  I'll be off on PTO from end of this
> week until the new year, and I know alot of folk are doing similar.

Right.  I'll be off from Dec 23 to Jan 9.  Can we all make Jan 11?

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Mark Burton <mark.burton@greensocs.com> writes:

>> On 14 Dec 2021, at 12:48, Markus Armbruster <armbru@redhat.com> wrote:
>> 
>> Paolo Bonzini <pbonzini@redhat.com> writes:
>> 
>>> On 12/13/21 16:28, Markus Armbruster wrote:
>>>> Paolo Bonzini <pbonzini@redhat.com> writes:
>>>> 
>>>>> On 12/10/21 14:54, Markus Armbruster wrote:
>>>>>> I want an open path to a single binary.  Taking years to get there is
>>>>>> fine.
>>>>> 
>>>>> The single binary is a distraction in my opinion.  Imagine
>>>>> instead of vl.c you have this in your second binary:
>> 
>> [...]
>> 
>>>>> This is the ultimate QEMU startup code.  If we can get this code to
>>>>> actually build a machine, you've reached the point where you don't care
>>>>> about what is in the command line parser; and consequently you don't care
>>>>> if there is one binary or two.
>>>> 
>>>> Define "you".  Also explain why it should include me, because I think it
>>>> doesn't :)
>>> 
>>> Impersonal you. :)
>> 
>> Unfortunate choice of a word.
>> 
>>>> By when can we have this second binary in master?  Opinion, please, not
>>>> promise.
>>> 
>>> Define "have":
>>> 
>>> - a binary that builds
>>> 
>>> - a binary that builds a bootable guest
>>> 
>>> - a binary that builds any guest that the current well-maintained
>>>  targets can build, using a given (but roughly full-featured) subset
>>> of options
>>> 
>>> Estimates for the first are easy (it's in my tree), estimates for the
>>> second depends on somebody helping (upstreaming -M smp took months 
>>> between me being busy, reviewers being busy, and releases freezing
>>> development), estimates for the third are hard.
>> 
>> Thanks.
>> 
>>>> Would you object to me expanding the CLI here to the point where I think
>>>> we can deprecate the old binary?
>>>> 
>>>> If yes, why?
>>> 
>>> Yes, for two reasons.
>>> 
>>> First, because there will be usually differences between the command
>>> lines as mentioned elsewhere in the thread.  qemu-system-* is a good 
>>> name, but one that is already taken by 15 years of docs using the
>>> existing command line.
>> 
>> A new CLI is pointless unless there are differences to the old one.
>> 
>> It is unadvisable unless we can eventually retire the old one.
>> 
>> While they coexist, the old binary name should use the old CLI, to
>> reduce confusion.
>> 
>>> Second, because a command line is really hard to get right as
>>> complexity increases.  QMP is the way to go to get as clean as
>>> possible a configuration mechanism.  There *will* be a second set of
>>> warts layered on top of the above code, and I don't want that.
>> 
>> We do not have consensus.  We may have misunderstandings.
>> 
>> Let's start with where we (hopefully) agree:
>> 
>> * We need a single, cohesive, low-level interface suitable for
>>  management applications.
>> 
>> * The existing interface is specified in QAPI.  Its concrete transport
>>  is QMP.
>> 
>> * The existing interface is not complete: certain things can only be
>>  done with the CLI.
>> 
>> * The existing transport is not available early enough to permit
>>  completing the interface.
>> 
>> * Fixing that involves a rework of startup.
>> 
>> * Reworking the existing startup and managing incompatible changes is
>>  impractical, and likely to make the mess we have on our hands worse.
>
> For “Completing” the interface, I agree. 
> To add a certain number of use cases - many of those can be (have been - aka preconfig) done, if with some degree of unpleasant-ness NOW without full re-working. That would give us test cases that we can subsequently use to test against as we move forward.

I'd be okay with hacking up the current mess some more so it can serve
as a test bed, as long as we all understand that the hacks are to be
reverted.

>> * A new binary sidesteps the need to manage incompatible change.
>> 
>> Any objections so far?
>> 
>> Now let me make a few more points:
>> 
>> * Single, cohesive interface does not require single transport.  In
>>  fact, we already have two: QMP and the (internal) C interface.
>> 
>> * QMP encodes the abstract interface in JSON, and offers the result on a
>>  Unix domain socket[1].
>> 
>> * The (internal) C interface encodes the abstract interface as a set of
>>  C data types and functions.
>> 
>> * Consider a configuration file transport that encodes the abstract
>>  interface in JSON.  The only wart this adds is syntax that is
>>  arguiably ill-suited to the purpose.  More suitable syntax exists.
>> 
>> * Similar for CLI.
>> 
>> * To get a "a second set of warts layered on top", we actually have to
>>  layer something on top that isn't utterly trivial.  Like a
>>  higher-level interface.  The "second set of warts" objection does not
>>  apply to (sane) transports.
>> 
>> * We already layer an interface on top: HMP[2].  It has its warts.
>> 
>> * The old CLI is partly layered on QMP, partly on HMP, and partly on
>>  internal C interfaces.  It's full of warts.
>> 
>> * Management applications are not the only users that matter.  Humans
>>  matter.  Simple programs like ad hoc scripts matter.
>> 
> (Unless one considers that a ‘human’ and/or ’script’ interface would just be ‘yet another management interface’…. And can/should be relegated to Somebody Else’s Problem)

I really hate relying on this Somebody guy, he never gets anything done.

[...]

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Tue, Dec 14, 2021 at 03:42:52PM +0100, Mark Burton wrote:
> I think we’re talking at cross purposes, and probably we agree (not sure). I’ll top quote to try and explain my point of view.
> 
> I think there are two discussions being mixed:
> 1/ A discussion about improving the CLI. (Having a new one, etc etc)
> 2/ A discussion about supporting a low level, and complete, API that can be used by “management layers” (QAPI).
> 
> I think this also gets mixed up with the discussion on migrating the CLI to use the low level API.
> 
> I want to focus on the low level API. 
> 
> I dont see why we’re discussing a ‘high level’ thing when, for now, we have to support the CLI, and we have work to do on QAPI.

We're discussing both because we're setting out what our end goal is
to be, and that end goal should be expected to cover both use cases.

> If somebody wants to build a new CLI, with a new ‘high level’
> interface, using QAPI - let them!

This is too weak of a statement, as it implies that a replacement
high level interface is optional and not important for the overall
project. I don't believe that to be the case, so I'm saying that
our design & impl plan has to demonstrate how we intend to cover
both deliverables or use cases. We can't simply ignore the high
level API saying it is someone else's problem to worry about.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Dec 14, 2021 at 03:42:52PM +0100, Mark Burton wrote:
>> I think we’re talking at cross purposes, and probably we agree (not sure). I’ll top quote to try and explain my point of view.
>> 
>> I think there are two discussions being mixed:
>> 1/ A discussion about improving the CLI. (Having a new one, etc etc)
>> 2/ A discussion about supporting a low level, and complete, API that can be used by “management layers” (QAPI).
>> 
>> I think this also gets mixed up with the discussion on migrating the CLI to use the low level API.
>> 
>> I want to focus on the low level API. 
>> 
>> I dont see why we’re discussing a ‘high level’ thing when, for now, we have to support the CLI, and we have work to do on QAPI.
>
> We're discussing both because we're setting out what our end goal is
> to be, and that end goal should be expected to cover both use cases.
>
>> If somebody wants to build a new CLI, with a new ‘high level’
>> interface, using QAPI - let them!
>
> This is too weak of a statement, as it implies that a replacement
> high level interface is optional and not important for the overall
> project. I don't believe that to be the case, so I'm saying that
> our design & impl plan has to demonstrate how we intend to cover
> both deliverables or use cases. We can't simply ignore the high
> level API saying it is someone else's problem to worry about.

Seconded.

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

> On 14 Dec 2021, at 16:12, Markus Armbruster <armbru@redhat.com> wrote:
> 
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
>> On Tue, Dec 14, 2021 at 03:42:52PM +0100, Mark Burton wrote:
>>> I think we’re talking at cross purposes, and probably we agree (not sure). I’ll top quote to try and explain my point of view.
>>> 
>>> I think there are two discussions being mixed:
>>> 1/ A discussion about improving the CLI. (Having a new one, etc etc)
>>> 2/ A discussion about supporting a low level, and complete, API that can be used by “management layers” (QAPI).
>>> 
>>> I think this also gets mixed up with the discussion on migrating the CLI to use the low level API.
>>> 
>>> I want to focus on the low level API. 
>>> 
>>> I dont see why we’re discussing a ‘high level’ thing when, for now, we have to support the CLI, and we have work to do on QAPI.
>> 
>> We're discussing both because we're setting out what our end goal is
>> to be, and that end goal should be expected to cover both use cases.
>> 
>>> If somebody wants to build a new CLI, with a new ‘high level’
>>> interface, using QAPI - let them!
>> 
>> This is too weak of a statement, as it implies that a replacement

“Replacement” 
	So long as thats really what you want. 

Cheers
Mark.

>> high level interface is optional and not important for the overall
>> project. I don't believe that to be the case, so I'm saying that
>> our design & impl plan has to demonstrate how we intend to cover
>> both deliverables or use cases. We can't simply ignore the high
>> level API saying it is someone else's problem to worry about.
> 
> Seconded.
> 

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/13/21 19:53, Daniel P. Berrangé wrote:
>> Adding vhost-user backends and helper processes means one of two things:
>> either you are not going to support hotplug, or you are going to redo
>> libvirtd with a QMP-based RPC.
> 
> If it were possible to keep auto-spawning of helpers at the high level
> that feels cleaner, so that the low level only has to worry about a
> single way of doing things. If that is too hard for hotplug though,
> so be it, leave auto-spawning in the low level.

OTOH, autospawning in the low-level saves hotplugging but it kills 
sandboxing; the seccomp filter prohibits forking.

The libvirt model is the only good one once you care about separation of 
privilege.  The idea of moving large parts of libvirt's domain driver 
into a new QEMU-level binary was floated around in the past by Andrea 
Bolognani, and I don't dislike it; but I don't believe anybody will have 
time to actually realize it, much less to bring it to feature parity.

Paolo

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Wed, Dec 15, 2021 at 07:46:37PM +0100, Paolo Bonzini wrote:
> On 12/13/21 19:53, Daniel P. Berrangé wrote:
> > > Adding vhost-user backends and helper processes means one of two things:
> > > either you are not going to support hotplug, or you are going to redo
> > > libvirtd with a QMP-based RPC.
> > 
> > If it were possible to keep auto-spawning of helpers at the high level
> > that feels cleaner, so that the low level only has to worry about a
> > single way of doing things. If that is too hard for hotplug though,
> > so be it, leave auto-spawning in the low level.
> 
> OTOH, autospawning in the low-level saves hotplugging but it kills
> sandboxing; the seccomp filter prohibits forking.


I think the kind of users we expect to leverage the high level interface
don't especially need sandboxing. They're more the people doing adhoc
virtualization or emulation tasks, not production deployments of VMs.
If they need strong security they'd be better off using a layer like
libvirt. 

> The libvirt model is the only good one once you care about separation of
> privilege.  The idea of moving large parts of libvirt's domain driver into a
> new QEMU-level binary was floated around in the past by Andrea Bolognani,
> and I don't dislike it; but I don't believe anybody will have time to
> actually realize it, much less to bring it to feature parity.

Yep, lets not create masses more work for ourselves, by expanding the
scope of this new design. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/14/21 12:48, Markus Armbruster wrote:
> Let's start with where we (hopefully) agree:

More or less I do agree with this, except for a couple points below 
where I think we disagree.

> * We need a single, cohesive, low-level interface suitable for
>    management applications.
> 
> * The existing interface is specified in QAPI.  Its concrete transport
>    is QMP.
> 
> * The existing interface is not complete: certain things can only be
>    done with the CLI.
>
> * The existing transport is not available early enough to permit
>    completing the interface.

So far so good.

> * Fixing that involves a rework of startup.
> * Reworking the existing startup and managing incompatible changes is
>    impractical, and likely to make the mess we have on our hands worse.

Not really, in particular the startup has been mostly reworked already 
and I disagree that it is messy.  softmmu/vl.c is messy, sure: it has N 
different command line parser for command line options, magic related to 
default devices, and complicated ordering of -object creation.

But the building of emulator data structures is not messy; only the code 
that transforms the user's instructions into startup commands.  The 
messy parts are almost entirely contained within softmmu/vl.c.

The one (and important, but fixable) exception is backends for on-board 
devices: serial_hd, drive_get, and nd_table.

> * A new binary sidesteps the need to manage incompatible change.

More precisely, a new binary sidesteps the need to integrate an existing 
mechanism with a new transport, and deal with the incompatibilities that 
arise.

> Any objections so far?
> 
> Now let me make a few more points:
> 
> * Single, cohesive interface does not require single transport.  In
>    fact, we already have two: QMP and the (internal) C interface.
> 
> * QMP encodes the abstract interface in JSON, and offers the result on a
>    Unix domain socket[1].
> 
> * The (internal) C interface encodes the abstract interface as a set of
>    C data types and functions.
> 
> * Consider a configuration file transport that encodes the abstract
>    interface in JSON.  The only wart this adds is syntax that is
>    arguiably ill-suited to the purpose.  More suitable syntax exists.
> 
> * Similar for CLI.
> 
> * To get a "a second set of warts layered on top", we actually have to
>    layer something on top that isn't utterly trivial.  Like a
>    higher-level interface.  The "second set of warts" objection does not
>    apply to (sane) transports.

The problem is that CLI and HMP, being targeted to humans (and as you 
say below humans matter), are not necessarily trivial transports.  If 
people find the trivial transport unusable, we will not be able to 
retire the old CLI.

Bad CLI is also very hard to deprecate because, unlike QMP (for which 
you can delegate the workarounds to Libvirt & friends) and HMP (for 
which people can just learn the new thing and type it), it is baked in 
countless scripts.  People hate it when scripts break.

> * The old CLI is partly layered on QMP, partly on HMP, and partly on
>    internal C interfaces.  It's full of warts.

I've worked on moving it more towards QMP or at least QOM, and much less 
on internal C interfaces.  It still has warts but they are 
self-contained and due to the baroque ordering of options.  My point is 
that we can continue this work to the point that having separate entry 
points (one CLI-centered, one QMP-only) is not a problem.

The issues with the CLI then can be completely self-contained within 
softmmu/vl.c, and will not influence the innards of QEMU.

Paolo

> * Management applications are not the only users that matter.  Humans
>    matter.  Simple programs like ad hoc scripts matter.

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

FWIW I Agree.

(Which probably means somethings hiding somewhere :-) )

Cheers
Mark.

> On 15 Dec 2021, at 21:00, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 12/14/21 12:48, Markus Armbruster wrote:
>> Let's start with where we (hopefully) agree:
> 
> More or less I do agree with this, except for a couple points below where I think we disagree.
> 
>> * We need a single, cohesive, low-level interface suitable for
>>   management applications.
>> * The existing interface is specified in QAPI.  Its concrete transport
>>   is QMP.
>> * The existing interface is not complete: certain things can only be
>>   done with the CLI.
>> 
>> * The existing transport is not available early enough to permit
>>   completing the interface.
> 
> So far so good.
> 
>> * Fixing that involves a rework of startup.
>> * Reworking the existing startup and managing incompatible changes is
>>   impractical, and likely to make the mess we have on our hands worse.
> 
> Not really, in particular the startup has been mostly reworked already and I disagree that it is messy.  softmmu/vl.c is messy, sure: it has N different command line parser for command line options, magic related to default devices, and complicated ordering of -object creation.
> 
> But the building of emulator data structures is not messy; only the code that transforms the user's instructions into startup commands.  The messy parts are almost entirely contained within softmmu/vl.c.
> 
> The one (and important, but fixable) exception is backends for on-board devices: serial_hd, drive_get, and nd_table.
> 
>> * A new binary sidesteps the need to manage incompatible change.
> 
> More precisely, a new binary sidesteps the need to integrate an existing mechanism with a new transport, and deal with the incompatibilities that arise.
> 
>> Any objections so far?
>> Now let me make a few more points:
>> * Single, cohesive interface does not require single transport.  In
>>   fact, we already have two: QMP and the (internal) C interface.
>> * QMP encodes the abstract interface in JSON, and offers the result on a
>>   Unix domain socket[1].
>> * The (internal) C interface encodes the abstract interface as a set of
>>   C data types and functions.
>> * Consider a configuration file transport that encodes the abstract
>>   interface in JSON.  The only wart this adds is syntax that is
>>   arguiably ill-suited to the purpose.  More suitable syntax exists.
>> * Similar for CLI.
>> * To get a "a second set of warts layered on top", we actually have to
>>   layer something on top that isn't utterly trivial.  Like a
>>   higher-level interface.  The "second set of warts" objection does not
>>   apply to (sane) transports.
> 
> The problem is that CLI and HMP, being targeted to humans (and as you say below humans matter), are not necessarily trivial transports.  If people find the trivial transport unusable, we will not be able to retire the old CLI.
> 
> Bad CLI is also very hard to deprecate because, unlike QMP (for which you can delegate the workarounds to Libvirt & friends) and HMP (for which people can just learn the new thing and type it), it is baked in countless scripts.  People hate it when scripts break.
> 
>> * The old CLI is partly layered on QMP, partly on HMP, and partly on
>>   internal C interfaces.  It's full of warts.
> 
> I've worked on moving it more towards QMP or at least QOM, and much less on internal C interfaces.  It still has warts but they are self-contained and due to the baroque ordering of options.  My point is that we can continue this work to the point that having separate entry points (one CLI-centered, one QMP-only) is not a problem.
> 
> The issues with the CLI then can be completely self-contained within softmmu/vl.c, and will not influence the innards of QEMU.
> 
> Paolo
> 
>> * Management applications are not the only users that matter.  Humans
>>   matter.  Simple programs like ad hoc scripts matter.

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 12/14/21 12:48, Markus Armbruster wrote:
>> Let's start with where we (hopefully) agree:
>
> More or less I do agree with this, except for a couple points below
> where I think we disagree.
>
>> * We need a single, cohesive, low-level interface suitable for
>>   management applications.
>>
>> * The existing interface is specified in QAPI.  Its concrete transport
>>   is QMP.
>>
>> * The existing interface is not complete: certain things can only be
>>   done with the CLI.
>>
>> * The existing transport is not available early enough to permit
>>   completing the interface.
>
> So far so good.
>
>> * Fixing that involves a rework of startup.
>>
>> * Reworking the existing startup and managing incompatible changes is
>>   impractical, and likely to make the mess we have on our hands worse.
>
> Not really, in particular the startup has been mostly reworked already
> and I disagree that it is messy.  softmmu/vl.c is messy, sure: it has
> N different command line parser for command line options, magic
> related to default devices, and complicated ordering of -object
> creation.
>
> But the building of emulator data structures is not messy; only the
> code that transforms the user's instructions into startup commands.
> The messy parts are almost entirely contained within softmmu/vl.c.

In my opinion, the worst mess is the reordering and the (commonly
unstated, sometimes unknown) dependencies that govern it.

The reordering is part of the stable interface.  Its finer points
basically nobody understands, at least not without staring at the code.

When we mess with the order, we commonly break things.

This is what leads me to my "still a mess" and "impractical" verdicts.

> The one (and important, but fixable) exception is backends for
> on-board devices: serial_hd, drive_get, and nd_table.

Practical ideas on fixing it?

>> * A new binary sidesteps the need to manage incompatible change.
>
> More precisely, a new binary sidesteps the need to integrate an
> existing mechanism with a new transport, and deal with the
> incompatibilities that arise.

I'm not sure I got this.

>> Any objections so far?
>>
>> Now let me make a few more points:
>>
>> * Single, cohesive interface does not require single transport.  In
>>   fact, we already have two: QMP and the (internal) C interface.
>>
>> * QMP encodes the abstract interface in JSON, and offers the result on a
>>   Unix domain socket[1].
>>
>> * The (internal) C interface encodes the abstract interface as a set of
>>   C data types and functions.
>>
>> * Consider a configuration file transport that encodes the abstract
>>   interface in JSON.  The only wart this adds is syntax that is
>>   arguiably ill-suited to the purpose.  More suitable syntax exists.
>>
>> * Similar for CLI.
>>
>> * To get a "a second set of warts layered on top", we actually have to
>>   layer something on top that isn't utterly trivial.  Like a
>>   higher-level interface.  The "second set of warts" objection does not
>>   apply to (sane) transports.
>
> The problem is that CLI and HMP, being targeted to humans (and as you
> say below humans matter), are not necessarily trivial transports.  If 
> people find the trivial transport unusable, we will not be able to
> retire the old CLI.

Yes, a trivial CLI transport alone may not suffice to retire the old
CLI.  By itself, that doesn't mean trivial transports must be avoided.

Do I have to argue the benefits of a trivial configuration file
transport?

Do I have to argue the benefits of a trivial CLI transport for use by
relatively unsophisticated programs / relatively sophisticated humans
(such as developers)?

Can we agree these benefits are not zero?

If yes, we can move on to debate whether such trivial transports are
worth their (modest) keep.

> Bad CLI is also very hard to deprecate because, unlike QMP (for which
> you can delegate the workarounds to Libvirt & friends) and HMP (for 
> which people can just learn the new thing and type it), it is baked in
> countless scripts.  People hate it when scripts break.

I assure you that bad QMP is plenty hard to deprecate, even when libvirt
can be updated to cope.

I think HMP is easier to change than QMP and CLI mostly because we've
spent years guiding people in need of a stable interface towards QMP.

>> * The old CLI is partly layered on QMP, partly on HMP, and partly on
>>   internal C interfaces.  It's full of warts.
>
> I've worked on moving it more towards QMP or at least QOM, and much
> less on internal C interfaces.  It still has warts but they are 
> self-contained and due to the baroque ordering of options.  My point
> is that we can continue this work to the point that having separate
> entry points (one CLI-centered, one QMP-only) is not a problem.
>
> The issues with the CLI then can be completely self-contained within
> softmmu/vl.c, and will not influence the innards of QEMU.

The issues with the CLI will still influence its users.

Can we agree that human users deserve something better than the current
CLI?

If no, then I doubt consensus is possible.

If yes, then we need to discuss how to build a better CLI for humans.
No "somebody could" cop outs, please.

I think we can learn from our experience with HMP/QMP.

Good:

* Separate interfaces for machines and for humans, because their needs
  are different: QMP and HMP.

* The stable interface is clear: QMP unless explicitly marked unstable.

* Layering the human interface on top of the machine interface: HMP
  commands implemented on top of QMP's internal C interface.

Bad:

* Not layering the human interface on top of the machine interface: HMP
  commands bypassing QMP's internal C interface.

* Proper layering is too much work: writing HMP commands that way should
  be easier, not harder.

Debatable:

* Some functionality is only available in HMP, mostly because that's
  less work.  Can't really argue against "this is more trouble than it's
  worth".

* Some functionality is only available in QMP, mostly because providing
  equivalent HMP commands is too much work.  Sometimes, equivalent HMP
  commands are awkwardly low level for humans, but "nope, go use QMP" is
  at least as awkward.

There's obviously more, but I think these are the points that matter
here.

> Paolo
>
>> * Management applications are not the only users that matter.  Humans
>>   matter.  Simple programs like ad hoc scripts matter.

Re: Redesign of QEMU startup & initial configuration

[Paolo Bonzini]

On 12/16/21 11:24, Markus Armbruster wrote:
>> Not really, in particular the startup has been mostly reworked already
>> and I disagree that it is messy.  softmmu/vl.c is messy, sure: it has
>> N different command line parser for command line options, magic
>> related to default devices, and complicated ordering of -object
>> creation.
>>
>> But the building of emulator data structures is not messy; only the
>> code that transforms the user's instructions into startup commands.
>> The messy parts are almost entirely contained within softmmu/vl.c.
> 
> In my opinion, the worst mess is the reordering and the (commonly
> unstated, sometimes unknown) dependencies that govern it.
> The reordering is part of the stable interface.  Its finer points
> basically nobody understands, at least not without staring at the code.

Then we agree, because that's what I meant by "the messy parts are 
almost entirely contained within softmmu/vl.c".

>> The one (and important, but fixable) exception is backends for
>> on-board devices: serial_hd, drive_get, and nd_table.
> 
> Practical ideas on fixing it?

What you did with pflash, turned up to 11.

>>> * A new binary sidesteps the need to manage incompatible change.
>>
>> More precisely, a new binary sidesteps the need to integrate an
>> existing mechanism with a new transport, and deal with the
>> incompatibilities that arise.
> 
> I'm not sure I got this.

Configuring the VM part in CLI and part in QMP is not something anybody 
needs nor should desire.  A new binary can use the CLI only for things 
that really have to be done before QMP is up, for example the 
configuration of sandboxing or tracing; and even that is only 
nice-to-have and not absolutely necessary.

>> The problem is that CLI and HMP, being targeted to humans (and as you
>> say below humans matter), are not necessarily trivial transports.  If
>> people find the trivial transport unusable, we will not be able to
>> retire the old CLI.
> 
> Yes, a trivial CLI transport alone may not suffice to retire the old
> CLI.  By itself, that doesn't mean trivial transports must be avoided.
> 
> Do I have to argue the benefits of a trivial configuration file
> transport?

I think you do; I'm not sure that a trivial configuration file transport 
is useful.  It's a middle ground in sophistication that I'm not sure 
would serve many people.  The only source of bug reports for -readconfig 
has been lxd, and I think we agree that they would be served better by QMP.

> Do I have to argue the benefits of a trivial CLI transport for use by
> relatively unsophisticated programs / relatively sophisticated humans
> (such as developers)? Can we agree these benefits are not zero?

We can.  But again I think you're misunderstanding the pain that the 
existing CLI inflicts on users.  Most users do not find the ordering 
complicated in practice; they do not even know that the issue exists. 
The problem that users have is the 100+ character command lines, and 
that can be fixed in two ways:

- with a trivial configuration file transport

- with a management tool such as virt-manager or virsh.

And I maintain that most users would be better served by the latter.  In 
fact, I constantly wonder how much we're overestimating the amount of 
people that are using QEMU.  In every post (Reddit, HN, wherever) that 
mentions QEMU being too complex and not having a front-end like 
VirtualBox, there's always someone that mentions virt-manager.

For developers it's different of course.

>> Bad CLI is also very hard to deprecate because, unlike QMP (for which
>> you can delegate the workarounds to Libvirt & friends) and HMP (for
>> which people can just learn the new thing and type it), it is baked in
>> countless scripts.  People hate it when scripts break.
> 
> I assure you that bad QMP is plenty hard to deprecate, even when libvirt
> can be updated to cope.

Right, and CLI is worse. :)

>> The issues with the CLI then can be completely self-contained within
>> softmmu/vl.c, and will not influence the innards of QEMU.
> 
> The issues with the CLI will still influence its users. Can we
> agree that human users deserve something better than the current
> CLI?

Deserve, yes.  Need, not sure.  Do you agree that a lot of clients of 
QEMU would be better served by Libvirt (programs) and virt-manager (humans)?

So, if I have to choose between better QMP now and better CLI now, I 
choose better QMP now.  Exactly to avoid the "somebody could" trap and 
focus on something that "we can" do now.

> I think we can learn from our experience with HMP/QMP.
> 
> Good:
> 
> * Separate interfaces for machines and for humans
> * Layering the human interface on top of the machine interface: HMP
>    commands implemented on top of QMP's internal C interface.

Agreed.  CLI should likewise be implemented on top of QMP's internal C 
interface, the same way non-preconfig mode concludes startup with 
qmp_x_exit_preconfig(NULL).  Second choice (inferior but sometimes 
acceptable): implement it on top of the same QOM interfaces as the QOM 
commands.

> * The stable interface is clear: QMP unless explicitly marked unstable.

Agreed, the problem here is that CLI is harder to evolve.

Paolo

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Thu, Dec 16, 2021 at 04:28:29PM +0100, Paolo Bonzini wrote:
> On 12/16/21 11:24, Markus Armbruster wrote:
> > > Not really, in particular the startup has been mostly reworked already
> > > and I disagree that it is messy.  softmmu/vl.c is messy, sure: it has
> > > N different command line parser for command line options, magic
> > > related to default devices, and complicated ordering of -object
> > > creation.
> > > 
> > > But the building of emulator data structures is not messy; only the
> > > code that transforms the user's instructions into startup commands.
> > > The messy parts are almost entirely contained within softmmu/vl.c.
> > 
> > In my opinion, the worst mess is the reordering and the (commonly
> > unstated, sometimes unknown) dependencies that govern it.
> > The reordering is part of the stable interface.  Its finer points
> > basically nobody understands, at least not without staring at the code.
> 
> Then we agree, because that's what I meant by "the messy parts are almost
> entirely contained within softmmu/vl.c".
> 
> > > The one (and important, but fixable) exception is backends for
> > > on-board devices: serial_hd, drive_get, and nd_table.
> > 
> > Practical ideas on fixing it?
> 
> What you did with pflash, turned up to 11.
> 
> > > > * A new binary sidesteps the need to manage incompatible change.
> > > 
> > > More precisely, a new binary sidesteps the need to integrate an
> > > existing mechanism with a new transport, and deal with the
> > > incompatibilities that arise.
> > 
> > I'm not sure I got this.
> 
> Configuring the VM part in CLI and part in QMP is not something anybody
> needs nor should desire.  A new binary can use the CLI only for things that
> really have to be done before QMP is up, for example the configuration of
> sandboxing or tracing; and even that is only nice-to-have and not absolutely
> necessary.

I wouldn't special case sandboxing at least. It should be something that
can be activated via a QMP command too. That way you can blockdev-add
a bunch of disks and other backends while you are still relatively
unconfined, and then lock it down with a sandbox before starting vCPUs.

Or you can perhaps start with a coarse sandbox and then switch to
a stronger sandbox part way through config (though can't remember
offhand if that's possible with seccomp, or whether the first
seccomp profile applied, prevents you adding stronger ones later).

Anyway sandboxing doesn't need to be active before QMP IMHO, because
our primary goal with it is to mitigate guest exploits from untrusted
code - the initial startup is largely trustworthy. Starting guest CPUs,
or reading VMState is where it becomes dangerous generally.

I think you can probably argue that even tracing doesn't hugely
need special casing if you can get QMP started early enough that
there's little else that can go wrong before you get a chance to
send a QMP 'trace' command.

> 
> > > The problem is that CLI and HMP, being targeted to humans (and as you
> > > say below humans matter), are not necessarily trivial transports.  If
> > > people find the trivial transport unusable, we will not be able to
> > > retire the old CLI.
> > 
> > Yes, a trivial CLI transport alone may not suffice to retire the old
> > CLI.  By itself, that doesn't mean trivial transports must be avoided.
> > 
> > Do I have to argue the benefits of a trivial configuration file
> > transport?
> 
> I think you do; I'm not sure that a trivial configuration file transport is
> useful.  It's a middle ground in sophistication that I'm not sure would
> serve many people.  The only source of bug reports for -readconfig has been
> lxd, and I think we agree that they would be served better by QMP.
> 
> > Do I have to argue the benefits of a trivial CLI transport for use by
> > relatively unsophisticated programs / relatively sophisticated humans
> > (such as developers)? Can we agree these benefits are not zero?
> 
> We can.  But again I think you're misunderstanding the pain that the
> existing CLI inflicts on users.  Most users do not find the ordering
> complicated in practice; they do not even know that the issue exists. The
> problem that users have is the 100+ character command lines, and that can be
> fixed in two ways:
> 
> - with a trivial configuration file transport
> 
> - with a management tool such as virt-manager or virsh.
> 
> And I maintain that most users would be better served by the latter.  In
> fact, I constantly wonder how much we're overestimating the amount of people
> that are using QEMU.  In every post (Reddit, HN, wherever) that mentions
> QEMU being too complex and not having a front-end like VirtualBox, there's
> always someone that mentions virt-manager.

An important thing to note is that while libvirt is theoretically
general purpose for any aspect of QEMU, practically speaking our
coverage of QEMU features is very much skewed to virtualization
use cases. The emulation use cases don't get anywher near as much
love & attention, especially for architectures lacking KVM, or for
machine types not used with KVM.

Also although libvirt tries to focus on just exposing a mechanism
and not imposing a policy there are some aspects where we don't
achieve that due to our architecture design. For example, if you
want QEMU connected to stdio that's not possible with libvirt
directly. There's always a level of indirection there because
we unconditionally daemonize QEMU.

> > > Bad CLI is also very hard to deprecate because, unlike QMP (for which
> > > you can delegate the workarounds to Libvirt & friends) and HMP (for
> > > which people can just learn the new thing and type it), it is baked in
> > > countless scripts.  People hate it when scripts break.
> > 
> > I assure you that bad QMP is plenty hard to deprecate, even when libvirt
> > can be updated to cope.
> 
> Right, and CLI is worse. :)
> 
> > > The issues with the CLI then can be completely self-contained within
> > > softmmu/vl.c, and will not influence the innards of QEMU.
> > 
> > The issues with the CLI will still influence its users. Can we
> > agree that human users deserve something better than the current
> > CLI?
> 
> Deserve, yes.  Need, not sure.  Do you agree that a lot of clients of QEMU
> would be better served by Libvirt (programs) and virt-manager (humans)?
> 
> So, if I have to choose between better QMP now and better CLI now, I choose
> better QMP now.  Exactly to avoid the "somebody could" trap and focus on
> something that "we can" do now.
> 
> > I think we can learn from our experience with HMP/QMP.
> > 
> > Good:
> > 
> > * Separate interfaces for machines and for humans
> > * Layering the human interface on top of the machine interface: HMP
> >    commands implemented on top of QMP's internal C interface.
> 
> Agreed.  CLI should likewise be implemented on top of QMP's internal C
> interface, the same way non-preconfig mode concludes startup with
> qmp_x_exit_preconfig(NULL).  Second choice (inferior but sometimes
> acceptable): implement it on top of the same QOM interfaces as the QOM
> commands.
> 
> > * The stable interface is clear: QMP unless explicitly marked unstable.
> 
> Agreed, the problem here is that CLI is harder to evolve.
> 
> Paolo
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

> On 16 Dec 2021, at 16:40, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> On Thu, Dec 16, 2021 at 04:28:29PM +0100, Paolo Bonzini wrote:
>> On 12/16/21 11:24, Markus Armbruster wrote:
>>>> Not really, in particular the startup has been mostly reworked already
>>>> and I disagree that it is messy.  softmmu/vl.c is messy, sure: it has
>>>> N different command line parser for command line options, magic
>>>> related to default devices, and complicated ordering of -object
>>>> creation.
>>>> 
>>>> But the building of emulator data structures is not messy; only the
>>>> code that transforms the user's instructions into startup commands.
>>>> The messy parts are almost entirely contained within softmmu/vl.c.
>>> 
>>> In my opinion, the worst mess is the reordering and the (commonly
>>> unstated, sometimes unknown) dependencies that govern it.
>>> The reordering is part of the stable interface.  Its finer points
>>> basically nobody understands, at least not without staring at the code.
>> 
>> Then we agree, because that's what I meant by "the messy parts are almost
>> entirely contained within softmmu/vl.c".
>> 
>>>> The one (and important, but fixable) exception is backends for
>>>> on-board devices: serial_hd, drive_get, and nd_table.
>>> 
>>> Practical ideas on fixing it?
>> 
>> What you did with pflash, turned up to 11.
>> 
>>>>> * A new binary sidesteps the need to manage incompatible change.
>>>> 
>>>> More precisely, a new binary sidesteps the need to integrate an
>>>> existing mechanism with a new transport, and deal with the
>>>> incompatibilities that arise.
>>> 
>>> I'm not sure I got this.
>> 
>> Configuring the VM part in CLI and part in QMP is not something anybody
>> needs nor should desire.  A new binary can use the CLI only for things that
>> really have to be done before QMP is up, for example the configuration of
>> sandboxing or tracing; and even that is only nice-to-have and not absolutely
>> necessary.
> 
> I wouldn't special case sandboxing at least. It should be something that
> can be activated via a QMP command too. That way you can blockdev-add
> a bunch of disks and other backends while you are still relatively
> unconfined, and then lock it down with a sandbox before starting vCPUs.
> 
> Or you can perhaps start with a coarse sandbox and then switch to
> a stronger sandbox part way through config (though can't remember
> offhand if that's possible with seccomp, or whether the first
> seccomp profile applied, prevents you adding stronger ones later).
> 
> Anyway sandboxing doesn't need to be active before QMP IMHO, because
> our primary goal with it is to mitigate guest exploits from untrusted
> code - the initial startup is largely trustworthy. Starting guest CPUs,
> or reading VMState is where it becomes dangerous generally.
> 
> I think you can probably argue that even tracing doesn't hugely
> need special casing if you can get QMP started early enough that
> there's little else that can go wrong before you get a chance to
> send a QMP 'trace' command.
> 
>> 
>>>> The problem is that CLI and HMP, being targeted to humans (and as you
>>>> say below humans matter), are not necessarily trivial transports.  If
>>>> people find the trivial transport unusable, we will not be able to
>>>> retire the old CLI.
>>> 
>>> Yes, a trivial CLI transport alone may not suffice to retire the old
>>> CLI.  By itself, that doesn't mean trivial transports must be avoided.
>>> 
>>> Do I have to argue the benefits of a trivial configuration file
>>> transport?
>> 
>> I think you do; I'm not sure that a trivial configuration file transport is
>> useful.  It's a middle ground in sophistication that I'm not sure would
>> serve many people.  The only source of bug reports for -readconfig has been
>> lxd, and I think we agree that they would be served better by QMP.
>> 
>>> Do I have to argue the benefits of a trivial CLI transport for use by
>>> relatively unsophisticated programs / relatively sophisticated humans
>>> (such as developers)? Can we agree these benefits are not zero?
>> 
>> We can.  But again I think you're misunderstanding the pain that the
>> existing CLI inflicts on users.  Most users do not find the ordering
>> complicated in practice; they do not even know that the issue exists. The
>> problem that users have is the 100+ character command lines, and that can be
>> fixed in two ways:
>> 
>> - with a trivial configuration file transport
>> 
>> - with a management tool such as virt-manager or virsh.
>> 
>> And I maintain that most users would be better served by the latter.  In
>> fact, I constantly wonder how much we're overestimating the amount of people
>> that are using QEMU.  In every post (Reddit, HN, wherever) that mentions
>> QEMU being too complex and not having a front-end like VirtualBox, there's
>> always someone that mentions virt-manager.

I totally agree with Paolo of course - thats what I was saying before. You can call “libvirt” somebody else’s problem if you wish, but it seems to me a more sensible route….

> 
> An important thing to note is that while libvirt is theoretically
> general purpose for any aspect of QEMU, practically speaking our
> coverage of QEMU features is very much skewed to virtualization
> use cases. The emulation use cases don't get anywher near as much
> love & attention, especially for architectures lacking KVM, or for
> machine types not used with KVM.

Totally agree on this (of course).

Thats why I’m here - I care about the people who care about emulation  :-)

In general, what we are working on is exactly the ability to service the ‘complex’ emulation use case. No CLI, nor single ‘config’ file will be good enough, in all cases we will need to drive directly into QMP.

Cheers
Mark.



> 
> Also although libvirt tries to focus on just exposing a mechanism
> and not imposing a policy there are some aspects where we don't
> achieve that due to our architecture design. For example, if you
> want QEMU connected to stdio that's not possible with libvirt
> directly. There's always a level of indirection there because
> we unconditionally daemonize QEMU.
> 
>>>> Bad CLI is also very hard to deprecate because, unlike QMP (for which
>>>> you can delegate the workarounds to Libvirt & friends) and HMP (for
>>>> which people can just learn the new thing and type it), it is baked in
>>>> countless scripts.  People hate it when scripts break.
>>> 
>>> I assure you that bad QMP is plenty hard to deprecate, even when libvirt
>>> can be updated to cope.
>> 
>> Right, and CLI is worse. :)
>> 
>>>> The issues with the CLI then can be completely self-contained within
>>>> softmmu/vl.c, and will not influence the innards of QEMU.
>>> 
>>> The issues with the CLI will still influence its users. Can we
>>> agree that human users deserve something better than the current
>>> CLI?
>> 
>> Deserve, yes.  Need, not sure.  Do you agree that a lot of clients of QEMU
>> would be better served by Libvirt (programs) and virt-manager (humans)?
>> 
>> So, if I have to choose between better QMP now and better CLI now, I choose
>> better QMP now.  Exactly to avoid the "somebody could" trap and focus on
>> something that "we can" do now.
>> 
>>> I think we can learn from our experience with HMP/QMP.
>>> 
>>> Good:
>>> 
>>> * Separate interfaces for machines and for humans
>>> * Layering the human interface on top of the machine interface: HMP
>>>   commands implemented on top of QMP's internal C interface.
>> 
>> Agreed.  CLI should likewise be implemented on top of QMP's internal C
>> interface, the same way non-preconfig mode concludes startup with
>> qmp_x_exit_preconfig(NULL).  Second choice (inferior but sometimes
>> acceptable): implement it on top of the same QOM interfaces as the QOM
>> commands.
>> 
>>> * The stable interface is clear: QMP unless explicitly marked unstable.
>> 
>> Agreed, the problem here is that CLI is harder to evolve.
>> 
>> Paolo
>> 
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Redesign of QEMU startup & initial configuration

[Daniel P. Berrangé]

On Thu, Dec 16, 2021 at 05:00:55PM +0100, Mark Burton wrote:
> 
> 
> > On 16 Dec 2021, at 16:40, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > 
> > On Thu, Dec 16, 2021 at 04:28:29PM +0100, Paolo Bonzini wrote:
> >> On 12/16/21 11:24, Markus Armbruster wrote:
> >>>> Not really, in particular the startup has been mostly reworked already
> >>>> and I disagree that it is messy.  softmmu/vl.c is messy, sure: it has
> >>>> N different command line parser for command line options, magic
> >>>> related to default devices, and complicated ordering of -object
> >>>> creation.
> >>>> 
> >>>> But the building of emulator data structures is not messy; only the
> >>>> code that transforms the user's instructions into startup commands.
> >>>> The messy parts are almost entirely contained within softmmu/vl.c.
> >>> 
> >>> In my opinion, the worst mess is the reordering and the (commonly
> >>> unstated, sometimes unknown) dependencies that govern it.
> >>> The reordering is part of the stable interface.  Its finer points
> >>> basically nobody understands, at least not without staring at the code.
> >> 
> >> Then we agree, because that's what I meant by "the messy parts are almost
> >> entirely contained within softmmu/vl.c".
> >> 
> >>>> The one (and important, but fixable) exception is backends for
> >>>> on-board devices: serial_hd, drive_get, and nd_table.
> >>> 
> >>> Practical ideas on fixing it?
> >> 
> >> What you did with pflash, turned up to 11.
> >> 
> >>>>> * A new binary sidesteps the need to manage incompatible change.
> >>>> 
> >>>> More precisely, a new binary sidesteps the need to integrate an
> >>>> existing mechanism with a new transport, and deal with the
> >>>> incompatibilities that arise.
> >>> 
> >>> I'm not sure I got this.
> >> 
> >> Configuring the VM part in CLI and part in QMP is not something anybody
> >> needs nor should desire.  A new binary can use the CLI only for things that
> >> really have to be done before QMP is up, for example the configuration of
> >> sandboxing or tracing; and even that is only nice-to-have and not absolutely
> >> necessary.
> > 
> > I wouldn't special case sandboxing at least. It should be something that
> > can be activated via a QMP command too. That way you can blockdev-add
> > a bunch of disks and other backends while you are still relatively
> > unconfined, and then lock it down with a sandbox before starting vCPUs.
> > 
> > Or you can perhaps start with a coarse sandbox and then switch to
> > a stronger sandbox part way through config (though can't remember
> > offhand if that's possible with seccomp, or whether the first
> > seccomp profile applied, prevents you adding stronger ones later).
> > 
> > Anyway sandboxing doesn't need to be active before QMP IMHO, because
> > our primary goal with it is to mitigate guest exploits from untrusted
> > code - the initial startup is largely trustworthy. Starting guest CPUs,
> > or reading VMState is where it becomes dangerous generally.
> > 
> > I think you can probably argue that even tracing doesn't hugely
> > need special casing if you can get QMP started early enough that
> > there's little else that can go wrong before you get a chance to
> > send a QMP 'trace' command.
> > 
> >> 
> >>>> The problem is that CLI and HMP, being targeted to humans (and as you
> >>>> say below humans matter), are not necessarily trivial transports.  If
> >>>> people find the trivial transport unusable, we will not be able to
> >>>> retire the old CLI.
> >>> 
> >>> Yes, a trivial CLI transport alone may not suffice to retire the old
> >>> CLI.  By itself, that doesn't mean trivial transports must be avoided.
> >>> 
> >>> Do I have to argue the benefits of a trivial configuration file
> >>> transport?
> >> 
> >> I think you do; I'm not sure that a trivial configuration file transport is
> >> useful.  It's a middle ground in sophistication that I'm not sure would
> >> serve many people.  The only source of bug reports for -readconfig has been
> >> lxd, and I think we agree that they would be served better by QMP.
> >> 
> >>> Do I have to argue the benefits of a trivial CLI transport for use by
> >>> relatively unsophisticated programs / relatively sophisticated humans
> >>> (such as developers)? Can we agree these benefits are not zero?
> >> 
> >> We can.  But again I think you're misunderstanding the pain that the
> >> existing CLI inflicts on users.  Most users do not find the ordering
> >> complicated in practice; they do not even know that the issue exists. The
> >> problem that users have is the 100+ character command lines, and that can be
> >> fixed in two ways:
> >> 
> >> - with a trivial configuration file transport
> >> 
> >> - with a management tool such as virt-manager or virsh.
> >> 
> >> And I maintain that most users would be better served by the latter.  In
> >> fact, I constantly wonder how much we're overestimating the amount of people
> >> that are using QEMU.  In every post (Reddit, HN, wherever) that mentions
> >> QEMU being too complex and not having a front-end like VirtualBox, there's
> >> always someone that mentions virt-manager.
> 
> I totally agree with Paolo of course - thats what I was saying before. You can call “libvirt” somebody else’s problem if you wish, but it seems to me a more sensible route….
> 
> > 
> > An important thing to note is that while libvirt is theoretically
> > general purpose for any aspect of QEMU, practically speaking our
> > coverage of QEMU features is very much skewed to virtualization
> > use cases. The emulation use cases don't get anywher near as much
> > love & attention, especially for architectures lacking KVM, or for
> > machine types not used with KVM.
> 
> Totally agree on this (of course).
> 
> Thats why I’m here - I care about the people who care about emulation  :-)
> 
> In general, what we are working on is exactly the ability to service the ‘complex’ emulation use case. No CLI, nor single ‘config’ file will be good enough, in all cases we will need to drive directly into QMP.

Can you clarify when you say 'what we are working on' here who & what
are you refering to ? Something Greensocs is developing or am I
mis-interpreting ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Mark Burton]

>> 
>> Totally agree on this (of course).
>> 
>> Thats why I’m here - I care about the people who care about emulation  :-)
>> 
>> In general, what we are working on is exactly the ability to service the ‘complex’ emulation use case. No CLI, nor single ‘config’ file will be good enough, in all cases we will need to drive directly into QMP.
> 
> Can you clarify when you say 'what we are working on' here who & what
> are you refering to ? Something Greensocs is developing or am I
> mis-interpreting ?

Yes - on behalf of Edgar and others.
There are various Device tree (or device tree like) syntaxes that people want to use to ‘configure’ a machine. A typical use case could potentially also include ‘CLI’ features as well for users to select files or whatever. (Wether that CLI is interpreted by QEMU or another mechanism isn’t very important).
Equally, GreenSocs itself would also like a ‘cleaner’ way to ‘instantiate’ QEMU within a different simulation environment - for that, we want to strip off the startup and essentially provide our own (hence Paolo’s rather simple code is very appealing). So those are 2 use cases we care about on our side.

Cheers
Mark

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

Re: Meeting today?

[Edgar E. Iglesias]

[-- Attachment #1: Type: text/plain, Size: 851 bytes --]

On Tue, Dec 14, 2021 at 3:49 PM Markus Armbruster <armbru@redhat.com> wrote:

> Daniel P. Berrangé <berrange@redhat.com> writes:
>
> > On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
> >> Mark Burton <mark.burton@greensocs.com> writes:
> >>
> >> > I realise it’s very short notice, but what about having a discussion
> today at 15:00 ?
> >>
> >> I have a conflict today.  I could try to reschedule, but I'd prefer to
> >> talk next week instead.  Less stress, better prep.
> >
> > I fear we've run out of time for this year if we want all interested
> > parties to be able to attend.  I'll be off on PTO from end of this
> > week until the new year, and I know alot of folk are doing similar.
>
> Right.  I'll be off from Dec 23 to Jan 9.  Can we all make Jan 11?
>

Jan 11th works for me!

Thanks,
Edgar

[-- Attachment #2: Type: text/html, Size: 1443 bytes --]

Re: Redesign of QEMU startup & initial configuration

[Richard W.M. Jones]

Sorry for very delayed reply ...

On Thu, Dec 02, 2021 at 07:57:38AM +0100, Markus Armbruster wrote:
> 1. QMP only
> 
>    Management applications need to use QMP for monitoring anyway.  They
>    may want to use it for initial configuration, too.  Libvirt does.
> 
>    They still need to bootstrap a QMP monitor, and for that, CLI is fine
>    as long as it's simple and stable.

libguestfs actually does not use the QMP monitor but manages to
configure eveything from the command line just fine.  I've attached
below a typical example.  (Of course we can use libvirt too, but still
for many configurations libvirt causes problems unfortunately).

> Human users struggle with inconsistent syntax, insufficiently expressive
> configuration files, and huge command lines.

One advantage of "huge command lines" is that they document the
configuration of qemu quite well.  I know it's only an approximation,
but in many cases it's exactly what we want.  It is frequently the
case when troubleshooting that we ask the user "what is the qemu
command line", and they can get that from the libvirt log file or
through "ps".

So we need to consider this and make sure that everything is changed
so we don't regress.  libguestfs will need substantial changes and
libvirt must dump the full configuration (qinfo or whatever) to the
log file.

I don't really disagreee with anything else you wrote however.

Rich.


libguestfs example:

/usr/bin/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -no-user-config \
    -nodefaults \
    -display none \
    -machine accel=kvm:tcg,graphics=off \
    -cpu max \
    -m 1280 \
    -no-reboot \
    -rtc driftfix=slew \
    -no-hpet \
    -global kvm-pit.lost_tick_policy=discard \
    -kernel /var/tmp/.guestfs-1000/appliance.d/kernel \
    -initrd /var/tmp/.guestfs-1000/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/tmp/libguestfs9bBO1w/scratch1.img,cache=unsafe,format=raw,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-1000/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -chardev socket,path=/run/user/1000/libguestfsGIlAlu/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -append "panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=9e6e8889-f991-45a3-bb41-67acebe7b062 selinux=0 guestfs_verbose=1 TERM=xterm-256color"


-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

"Startup" meeting (was Re: Meeting today?)

[Mark Burton]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

Can we confirm the 11th for this meeting?

Cheers
Mark.


> On 4 Jan 2022, at 10:29, Edgar E. Iglesias <edgar.iglesias@gmail.com> wrote:
> 
> 
> 
> On Tue, Dec 14, 2021 at 3:49 PM Markus Armbruster <armbru@redhat.com <mailto:armbru@redhat.com>> wrote:
> Daniel P. Berrangé <berrange@redhat.com <mailto:berrange@redhat.com>> writes:
> 
> > On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
> >> Mark Burton <mark.burton@greensocs.com <mailto:mark.burton@greensocs.com>> writes:
> >> 
> >> > I realise it’s very short notice, but what about having a discussion today at 15:00 ?
> >> 
> >> I have a conflict today.  I could try to reschedule, but I'd prefer to
> >> talk next week instead.  Less stress, better prep.
> >
> > I fear we've run out of time for this year if we want all interested
> > parties to be able to attend.  I'll be off on PTO from end of this
> > week until the new year, and I know alot of folk are doing similar.
> 
> Right.  I'll be off from Dec 23 to Jan 9.  Can we all make Jan 11?
> 
> Jan 11th works for me!
> 
> Thanks,
> Edgar


[-- Attachment #2: Type: text/html, Size: 2385 bytes --]

Re: "Startup" meeting (was Re: Meeting today?)

[Daniel P. Berrangé]

No one objected, so I think we can go for the 11th.

On Thu, Jan 06, 2022 at 12:21:56PM +0100, Mark Burton wrote:
> Can we confirm the 11th for this meeting?
> 
> Cheers
> Mark.
> 
> 
> > On 4 Jan 2022, at 10:29, Edgar E. Iglesias <edgar.iglesias@gmail.com> wrote:
> > 
> > 
> > 
> > On Tue, Dec 14, 2021 at 3:49 PM Markus Armbruster <armbru@redhat.com <mailto:armbru@redhat.com>> wrote:
> > Daniel P. Berrangé <berrange@redhat.com <mailto:berrange@redhat.com>> writes:
> > 
> > > On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
> > >> Mark Burton <mark.burton@greensocs.com <mailto:mark.burton@greensocs.com>> writes:
> > >> 
> > >> > I realise it’s very short notice, but what about having a discussion today at 15:00 ?
> > >> 
> > >> I have a conflict today.  I could try to reschedule, but I'd prefer to
> > >> talk next week instead.  Less stress, better prep.
> > >
> > > I fear we've run out of time for this year if we want all interested
> > > parties to be able to attend.  I'll be off on PTO from end of this
> > > week until the new year, and I know alot of folk are doing similar.
> > 
> > Right.  I'll be off from Dec 23 to Jan 9.  Can we all make Jan 11?
> > 
> > Jan 11th works for me!
> > 
> > Thanks,
> > Edgar
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: "Startup" meeting (was Re: Meeting today?)

[Philippe Mathieu-Daudé]

Hi,

Just checking in, this call is scheduled for today, 3pm CEST, right?

Here is the KVM call calendar:
https://www.google.com/calendar/embed?src=dG9iMXRqcXAzN3Y4ZXZwNzRoMHE4a3BqcXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

On 1/6/22 12:23, Daniel P. Berrangé wrote:
> No one objected, so I think we can go for the 11th.
> 
> On Thu, Jan 06, 2022 at 12:21:56PM +0100, Mark Burton wrote:
>> Can we confirm the 11th for this meeting?
>>
>> Cheers
>> Mark.
>>
>>
>>> On 4 Jan 2022, at 10:29, Edgar E. Iglesias <edgar.iglesias@gmail.com> wrote:
>>>
>>>
>>>
>>> On Tue, Dec 14, 2021 at 3:49 PM Markus Armbruster <armbru@redhat.com <mailto:armbru@redhat.com>> wrote:
>>> Daniel P. Berrangé <berrange@redhat.com <mailto:berrange@redhat.com>> writes:
>>>
>>>> On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
>>>>> Mark Burton <mark.burton@greensocs.com <mailto:mark.burton@greensocs.com>> writes:
>>>>>
>>>>>> I realise it’s very short notice, but what about having a discussion today at 15:00 ?
>>>>>
>>>>> I have a conflict today.  I could try to reschedule, but I'd prefer to
>>>>> talk next week instead.  Less stress, better prep.
>>>>
>>>> I fear we've run out of time for this year if we want all interested
>>>> parties to be able to attend.  I'll be off on PTO from end of this
>>>> week until the new year, and I know alot of folk are doing similar.
>>>
>>> Right.  I'll be off from Dec 23 to Jan 9.  Can we all make Jan 11?
>>>
>>> Jan 11th works for me!
>>>
>>> Thanks,
>>> Edgar
>>
> 
> Regards,
> Daniel

Re: "Startup" meeting (was Re: Meeting today?)

[Mark Burton]

That is my understanding… 
See you there!
Cheers
Mark.


> On 11 Jan 2022, at 11:20, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> 
> Hi,
> 
> Just checking in, this call is scheduled for today, 3pm CEST, right?
> 
> Here is the KVM call calendar:
> https://www.google.com/calendar/embed?src=dG9iMXRqcXAzN3Y4ZXZwNzRoMHE4a3BqcXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
> 
> On 1/6/22 12:23, Daniel P. Berrangé wrote:
>> No one objected, so I think we can go for the 11th.
>> 
>> On Thu, Jan 06, 2022 at 12:21:56PM +0100, Mark Burton wrote:
>>> Can we confirm the 11th for this meeting?
>>> 
>>> Cheers
>>> Mark.
>>> 
>>> 
>>>> On 4 Jan 2022, at 10:29, Edgar E. Iglesias <edgar.iglesias@gmail.com> wrote:
>>>> 
>>>> 
>>>> 
>>>> On Tue, Dec 14, 2021 at 3:49 PM Markus Armbruster <armbru@redhat.com <mailto:armbru@redhat.com>> wrote:
>>>> Daniel P. Berrangé <berrange@redhat.com <mailto:berrange@redhat.com>> writes:
>>>> 
>>>>> On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
>>>>>> Mark Burton <mark.burton@greensocs.com <mailto:mark.burton@greensocs.com>> writes:
>>>>>> 
>>>>>>> I realise it’s very short notice, but what about having a discussion today at 15:00 ?
>>>>>> 
>>>>>> I have a conflict today.  I could try to reschedule, but I'd prefer to
>>>>>> talk next week instead.  Less stress, better prep.
>>>>> 
>>>>> I fear we've run out of time for this year if we want all interested
>>>>> parties to be able to attend.  I'll be off on PTO from end of this
>>>>> week until the new year, and I know alot of folk are doing similar.
>>>> 
>>>> Right.  I'll be off from Dec 23 to Jan 9.  Can we all make Jan 11?
>>>> 
>>>> Jan 11th works for me!
>>>> 
>>>> Thanks,
>>>> Edgar
>>> 
>> 
>> Regards,
>> Daniel

Re: "Startup" meeting (was Re: Meeting today?)

[Daniel P. Berrangé]

On Tue, Jan 11, 2022 at 11:20:54AM +0100, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> Just checking in, this call is scheduled for today, 3pm CEST, right?
> 
> Here is the KVM call calendar:
> https://www.google.com/calendar/embed?src=dG9iMXRqcXAzN3Y4ZXZwNzRoMHE4a3BqcXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

Hmm, no dial-in details in that calender, so where are we
holding the call.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Redesign of QEMU startup & initial configuration

[Markus Armbruster]

"Richard W.M. Jones" <rjones@redhat.com> writes:

> Sorry for very delayed reply ...
>
> On Thu, Dec 02, 2021 at 07:57:38AM +0100, Markus Armbruster wrote:
>> 1. QMP only
>> 
>>    Management applications need to use QMP for monitoring anyway.  They
>>    may want to use it for initial configuration, too.  Libvirt does.
>> 
>>    They still need to bootstrap a QMP monitor, and for that, CLI is fine
>>    as long as it's simple and stable.
>
> libguestfs actually does not use the QMP monitor but manages to
> configure eveything from the command line just fine.  I've attached
> below a typical example.  (Of course we can use libvirt too, but still
> for many configurations libvirt causes problems unfortunately).

This is just the example I've been looking for: an application that
doesn't really need "QMP only".  Thanks!

>> Human users struggle with inconsistent syntax, insufficiently expressive
>> configuration files, and huge command lines.
>
> One advantage of "huge command lines" is that they document the
> configuration of qemu quite well.  I know it's only an approximation,
> but in many cases it's exactly what we want.  It is frequently the
> case when troubleshooting that we ask the user "what is the qemu
> command line", and they can get that from the libvirt log file or
> through "ps".

Yes.

> So we need to consider this and make sure that everything is changed
> so we don't regress.  libguestfs will need substantial changes and

I think we should try to accomodate how libguestfs wants to use QEMU.

> libvirt must dump the full configuration (qinfo or whatever) to the
> log file.

I believe the entire QMP traffic needs to be logged.

CLI + configuration files are all or nothing: if QEMU starts, all, else
nothing.  When you configure via QMP, any command may fail, and these
failures are not fatal.

In general, we can also have run-time configuration changes via QMP,
such as hot plug.  Another reason to log all QMP traffic.

> I don't really disagreee with anything else you wrote however.

Thanks!

Re: "Startup" meeting (was Re: Meeting today?)

[Kevin Wolf]

Am 11.01.2022 um 11:22 hat Mark Burton geschrieben:
> That is my understanding… 
> See you there!

Unfortunately, I missed this whole thread until now.

If the meeting did happen, has anyone taken notes? And if so, where
could I find them?

Kevin

> > On 11 Jan 2022, at 11:20, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> > 
> > Hi,
> > 
> > Just checking in, this call is scheduled for today, 3pm CEST, right?
> > 
> > Here is the KVM call calendar:
> > https://www.google.com/calendar/embed?src=dG9iMXRqcXAzN3Y4ZXZwNzRoMHE4a3BqcXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
> > 
> > On 1/6/22 12:23, Daniel P. Berrangé wrote:
> >> No one objected, so I think we can go for the 11th.
> >> 
> >> On Thu, Jan 06, 2022 at 12:21:56PM +0100, Mark Burton wrote:
> >>> Can we confirm the 11th for this meeting?
> >>> 
> >>> Cheers
> >>> Mark.
> >>> 
> >>> 
> >>>> On 4 Jan 2022, at 10:29, Edgar E. Iglesias <edgar.iglesias@gmail.com> wrote:
> >>>> 
> >>>> 
> >>>> 
> >>>> On Tue, Dec 14, 2021 at 3:49 PM Markus Armbruster <armbru@redhat.com <mailto:armbru@redhat.com>> wrote:
> >>>> Daniel P. Berrangé <berrange@redhat.com <mailto:berrange@redhat.com>> writes:
> >>>> 
> >>>>> On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
> >>>>>> Mark Burton <mark.burton@greensocs.com <mailto:mark.burton@greensocs.com>> writes:
> >>>>>> 
> >>>>>>> I realise it’s very short notice, but what about having a discussion today at 15:00 ?
> >>>>>> 
> >>>>>> I have a conflict today.  I could try to reschedule, but I'd prefer to
> >>>>>> talk next week instead.  Less stress, better prep.
> >>>>> 
> >>>>> I fear we've run out of time for this year if we want all interested
> >>>>> parties to be able to attend.  I'll be off on PTO from end of this
> >>>>> week until the new year, and I know alot of folk are doing similar.
> >>>> 
> >>>> Right.  I'll be off from Dec 23 to Jan 9.  Can we all make Jan 11?
> >>>> 
> >>>> Jan 11th works for me!
> >>>> 
> >>>> Thanks,
> >>>> Edgar
> >>> 
> >> 
> >> Regards,
> >> Daniel
> 
> 

Re: "Startup" meeting (was Re: Meeting today?)

[Markus Armbruster]

Kevin Wolf <kwolf@redhat.com> writes:

> Am 11.01.2022 um 11:22 hat Mark Burton geschrieben:
>> That is my understanding… 
>> See you there!
>
> Unfortunately, I missed this whole thread until now.
>
> If the meeting did happen, has anyone taken notes? And if so, where
> could I find them?

Yes, it did.  Juan, did you take notes?

Re: "Startup" meeting (was Re: Meeting today?)

[Mark Burton]

All, I believe we will have a followup meeting this coming Tuesday 25th January, at 15:00 (presumably using the same link: https://redhat.bluejeans.com/5402697718).

We (GreenSocs/Xilinx) would like to quickly show what now ‘works’, and to give an update on the patches.

Cheers
Mark.




> On 17 Jan 2022, at 18:13, Kevin Wolf <kwolf@redhat.com> wrote:
> 
> Am 11.01.2022 um 11:22 hat Mark Burton geschrieben:
>> That is my understanding… 
>> See you there!
> 
> Unfortunately, I missed this whole thread until now.
> 
> If the meeting did happen, has anyone taken notes? And if so, where
> could I find them?
> 
> Kevin
> 
>>> On 11 Jan 2022, at 11:20, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>>> 
>>> Hi,
>>> 
>>> Just checking in, this call is scheduled for today, 3pm CEST, right?
>>> 
>>> Here is the KVM call calendar:
>>> https://www.google.com/calendar/embed?src=dG9iMXRqcXAzN3Y4ZXZwNzRoMHE4a3BqcXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
>>> 
>>> On 1/6/22 12:23, Daniel P. Berrangé wrote:
>>>> No one objected, so I think we can go for the 11th.
>>>> 
>>>> On Thu, Jan 06, 2022 at 12:21:56PM +0100, Mark Burton wrote:
>>>>> Can we confirm the 11th for this meeting?
>>>>> 
>>>>> Cheers
>>>>> Mark.
>>>>> 
>>>>> 
>>>>>> On 4 Jan 2022, at 10:29, Edgar E. Iglesias <edgar.iglesias@gmail.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Dec 14, 2021 at 3:49 PM Markus Armbruster <armbru@redhat.com <mailto:armbru@redhat.com>> wrote:
>>>>>> Daniel P. Berrangé <berrange@redhat.com <mailto:berrange@redhat.com>> writes:
>>>>>> 
>>>>>>> On Tue, Dec 14, 2021 at 12:37:43PM +0100, Markus Armbruster wrote:
>>>>>>>> Mark Burton <mark.burton@greensocs.com <mailto:mark.burton@greensocs.com>> writes:
>>>>>>>> 
>>>>>>>>> I realise it’s very short notice, but what about having a discussion today at 15:00 ?
>>>>>>>> 
>>>>>>>> I have a conflict today.  I could try to reschedule, but I'd prefer to
>>>>>>>> talk next week instead.  Less stress, better prep.
>>>>>>> 
>>>>>>> I fear we've run out of time for this year if we want all interested
>>>>>>> parties to be able to attend.  I'll be off on PTO from end of this
>>>>>>> week until the new year, and I know alot of folk are doing similar.
>>>>>> 
>>>>>> Right.  I'll be off from Dec 23 to Jan 9.  Can we all make Jan 11?
>>>>>> 
>>>>>> Jan 11th works for me!
>>>>>> 
>>>>>> Thanks,
>>>>>> Edgar
>>>>> 
>>>> 
>>>> Regards,
>>>> Daniel
>> 
>> 
> 

Re: "Startup" meeting (was Re: Meeting today?)

[Juan Quintela]

Mark Burton <mark.burton@greensocs.com> wrote:
> All, I believe we will have a followup meeting this coming Tuesday
> 25th January, at 15:00 (presumably using the same link:
> https://redhat.bluejeans.com/5402697718).
>
> We (GreenSocs/Xilinx) would like to quickly show what now ‘works’, and to give an update on the patches.

I send the call for agenda already.

We are having the meeting.

If you can add anything to my [very] incomplete minutes for last meeting
that I sent with the new invite, it would be great.

Later, Juan.

Re: "Startup" meeting (was Re: Meeting today?)

[Philippe Mathieu-Daudé via]

On 1/25/22 09:50, Juan Quintela wrote:
> Mark Burton <mark.burton@greensocs.com> wrote:
>> All, I believe we will have a followup meeting this coming Tuesday
>> 25th January, at 15:00 (presumably using the same link:
>> https://redhat.bluejeans.com/5402697718).
>>
>> We (GreenSocs/Xilinx) would like to quickly show what now ‘works’, and to give an update on the patches.
> 
> I send the call for agenda already.
> 
> We are having the meeting.

Do we need to stick to bluejeans, or can we switch to something more
generic to easily record the call, and be able to start when Juan is
not available?

Re: "Startup" meeting (was Re: Meeting today?)

[Juan Quintela]

Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> On 1/25/22 09:50, Juan Quintela wrote:
>> Mark Burton <mark.burton@greensocs.com> wrote:
>>> All, I believe we will have a followup meeting this coming Tuesday
>>> 25th January, at 15:00 (presumably using the same link:
>>> https://redhat.bluejeans.com/5402697718).
>>>
>>> We (GreenSocs/Xilinx) would like to quickly show what now ‘works’, and to give an update on the patches.
>> 
>> I send the call for agenda already.
>> 
>> We are having the meeting.
>
> Do we need to stick to bluejeans, or can we switch to something more
> generic to easily record the call, and be able to start when Juan is
> not available?

Hi

Anyone from inside Red Hat can start the call.  So I think that starting
the call shouldn't be a problem.

If I remember correctly, one of the reasons for using bluejeans was that
in the past we used to be around 40 people on the call, and that was not
easy to setup.  Perhaps today it is different.

About recording, I will ask everybody if they agree on the call and we
can record it.  On the past we _didn't_ want recordings, but I can't
remember the reason.

Later, Juan.

Re: "Startup" meeting (was Re: Meeting today?)

[Mark Burton]

Hi Juan, is there a meeting today? I think the plan was to talk about ’startup’ itself ?
Cheers
Mark.


> On 25 Jan 2022, at 11:58, Juan Quintela <quintela@redhat.com> wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>> On 1/25/22 09:50, Juan Quintela wrote:
>>> Mark Burton <mark.burton@greensocs.com> wrote:
>>>> All, I believe we will have a followup meeting this coming Tuesday
>>>> 25th January, at 15:00 (presumably using the same link:
>>>> https://redhat.bluejeans.com/5402697718).
>>>> 
>>>> We (GreenSocs/Xilinx) would like to quickly show what now ‘works’, and to give an update on the patches.
>>> 
>>> I send the call for agenda already.
>>> 
>>> We are having the meeting.
>> 
>> Do we need to stick to bluejeans, or can we switch to something more
>> generic to easily record the call, and be able to start when Juan is
>> not available?
> 
> Hi
> 
> Anyone from inside Red Hat can start the call.  So I think that starting
> the call shouldn't be a problem.
> 
> If I remember correctly, one of the reasons for using bluejeans was that
> in the past we used to be around 40 people on the call, and that was not
> easy to setup.  Perhaps today it is different.
> 
> About recording, I will ask everybody if they agree on the call and we
> can record it.  On the past we _didn't_ want recordings, but I can't
> remember the reason.
> 
> Later, Juan.
> 

Re: "Startup" meeting (was Re: Meeting today?)

[Juan Quintela]

Mark Burton <mark.burton@greensocs.com> wrote:
> Hi Juan, is there a meeting today? I think the plan was to talk about ’startup’ itself ?
> Cheers
> Mark.

Hi Mark

Yeap.  I asked for it on last meeting.  The call for topics is on the
list (not that anyone else has answered)

Later, Juan.



>
>> On 25 Jan 2022, at 11:58, Juan Quintela <quintela@redhat.com> wrote:
>> 
>> Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>>> On 1/25/22 09:50, Juan Quintela wrote:
>>>> Mark Burton <mark.burton@greensocs.com> wrote:
>>>>> All, I believe we will have a followup meeting this coming Tuesday
>>>>> 25th January, at 15:00 (presumably using the same link:
>>>>> https://redhat.bluejeans.com/5402697718).
>>>>> 
>>>>> We (GreenSocs/Xilinx) would like to quickly show what now ‘works’, and to give an update on the patches.
>>>> 
>>>> I send the call for agenda already.
>>>> 
>>>> We are having the meeting.
>>> 
>>> Do we need to stick to bluejeans, or can we switch to something more
>>> generic to easily record the call, and be able to start when Juan is
>>> not available?
>> 
>> Hi
>> 
>> Anyone from inside Red Hat can start the call.  So I think that starting
>> the call shouldn't be a problem.
>> 
>> If I remember correctly, one of the reasons for using bluejeans was that
>> in the past we used to be around 40 people on the call, and that was not
>> easy to setup.  Perhaps today it is different.
>> 
>> About recording, I will ask everybody if they agree on the call and we
>> can record it.  On the past we _didn't_ want recordings, but I can't
>> remember the reason.
>> 
>> Later, Juan.
>>