Re: Making QEMU easier for management tools and applications

[John Snow <jsnow@redhat.com>]

On 1/27/20 9:35 AM, Kevin Wolf wrote:
> Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
>> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
>>> Well, sure. The context of this email was qmp-shell though, which is
>>> meant to help facilitate the entry of JSON commands so that you *can*
>>> indeed just forego the CLI/HMP entirely.
>>>
>>> If you are of the opinion that every user of QEMU should be copy/pasting
>>> JSON straight into a socket and we should delete qmp-shell, that's
>>> certainly a fine opinion.
>>
>> I think part of the pain of qmp-shell comes from the very fact that
>> it is trying to be an interactive shell. This points people towards
>> interactively typing in the commands, which is horrific when you get
>> anywhere near the JSON, or even dot-notation traditional commands.
>>
>> If it was just a qmp-client that was single shot, we'd encourage
>> people to create the JSON in a sensible way - vim/emacs/whatever.
> 
> I don't see how this is sensible. QMP commands are something that I
> reuse even less than VM configurations, so creating a one-off file for
> each would take me a lot more time and I would still have to type the
> same JSON code that I have to type with -qmp stdio.
> 
> The reason it is and should be an interactive shell is that I'm
> interacting with it. Switching back and forth between a text editor and
> a shell to actually send the command to QEMU would make things only even
> more cumbersome than they already are.
> 
>> Bash/dash/zsh/$whatever is their interactive shell, with massively
>> more features than qmp-shell. You have command history, autocomplete,
>> conditional and looping constructs, and everything a normal shell
>> offers.
> 
> If I wanted to program a QMP client, I would use Python. For me,
> conditionals and loops are completely out of scope for a QMP shell. I
> just want an easy way to tell QEMU to do something specific.
> 
> A command history already exists for qmp-shell. It's better than bash
> because it doesn't mix QMP history with whatever else I do on my
> computer.
> 
> Autocomplete in qmp-shell doesn't exist, as far as I know, but if
> implemented, it could be a lot more useful than bash completion because
> it could offer key completion based on the QMP schema.
> 

It does have tab completion for command names, but it does not know
about or remember argument fields. It does not have autocomplete or
typing hints like FiSH or bash ^r.

I would like to change this, actually, by making the docstrings in QAPI
schema a first class citizen of the spec and allowing them to be
introspectable via the socket directly.

(I.e., you can get the list of arguments and the docstrings that
accompany them over the wire so you can display it in the client.)

Problem I'm having with qmp-shell is, like Kevin says below ...

> This is in fact a big part of the problem that qmp-shell really needs to
> solve before it can replace HMP: How to make writing commands at least
> almost as simple as with HMP. If I can just press tab a few times to
> cycle through the available options for the command, that would already
> be a massive improvement over writing JSON manually (which you would
> still have to do with your text-file based approach, without any
> QMP-specific support).
> 

... I can't figure out how to make writing commands simple.

When you have a "simple" command, the abstraction works OK; you can type
key=val pairs and go about your way.

As soon as you have anything nested, the gossamer-thin illusion is
destroyed. I investigated making this a little easier by adding a parser
that could read directly from stdin and would allow multi-line JSON
inputs as arguments.

(Like the python shell does it, for example: When you have a dictionary
opening brace, it lets you continue to the next line.)

I was a little disheartened that most JSON parsers in python expect to
consume buffered objects and generally consume the entire buffer -- it
didn't seem to play nice with the idea of wanting to parse from STDIN
directly.


So:

- I think qmp-shell is worth having, especially if polished
(autocomplete, docstrings, argument hints, etc).

- Kevin mentioned getting this into the GTK shell. I think that would be
great, as a step to help phase out HMP.

- I think getting rid of HMP is good because I don't care for the idea
of supporting two monitor protocols. One schema, one parser, one truth.

- I do, however, like the idea of having a non-rigorous monitor that
lets us get away with murder when we need to. HMP is useful for
debugging, prototypes and other things where the rigor and permanence of
a QAPI schema feels too burdensome.

- So: maybe a turbocharged qmp-shell can offer some similar kinds of
convenience commands that are build on top of real QMP. Sugar logic and
other fanciful things could be implemented there in qmp-shell as
extensions. You'd get a stock pile of them with your QEMU install that
help you do certain tasks quickly and trivially.

- Given all the above, I am willing to try to save, polish, or re-design
qmp-shell; but am a bit starved for ideas on the syntax... This is why I
was spending a bit of time talking about our flattening to dot syntax,
and other projects related to representing hierarchical data.

Would really love to hear ideas on what a good interactive shell syntax
for a JSON-fueled schema would look like.

Any prior art, other projects, and reading anyone can recommend would be
nice.

> The other part that it needs to solve is how to be available by default
> without specifying anything on the command line. Basically, if I press
> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> implemented internally or by an external Python process, I don't mind.
> 
>> The only strong reason for qmp-shell to be interactive would be if
>> the initial protoocl handshake was too slow. I can't see that being
>> a problem with QMP.
> 
> Speed would be the least of my concerns. This is about manual use, and
> it already takes me a while to type in my commands.
> 
>> Example usage:
>>
>> 1. Launch the QEMU runtime for the desired target
>>
>>      $ qemu-runtime-x86_64 myvm.sock
>>
>> 2. Load the configuration to define the VM
>>
>>    $ cat myvm.yaml
>>    commands:
>>      - machine_declare:
>>          name: pc-q35-5.0
>> 	 ...
>>      - blockdev_add:
>>          ...
>>      - device_add:
>>          ...
>>      - blockdev_add:
>>          ...
>>      - device_add:
>>          ...
>>    $ qemu-client myvm.sock myvm.yaml
>>
>>
>> 3. Hotplug a disk
>>
>>    $ cat mynewdisk.yaml
>>    commands:
>>      - blockdev_add:
>>          ...
>>      - device_add:
>>          ...
>>    $ qemu-client myvm.sock mynewdisk.yaml
>>
>>
>> 3. Hotunplug a disk
>>
>>    $ cat myolddisk.yaml
>>    commands:
>>      - device_del:
>>          ...
>>      - blockdev_del:
>>          ...
>>    $ qemu-client myvm.sock myolddisk.yaml
> 
> Just to compare, this is what the human user oriented flow looks like
> today:
> 
> 1. qemu-system-x86_64 -M pc-q35-5.0 -drive if=virtio,... -cdrom ...
> 
> 2. <Press Ctrl-Alt-2 to get to the HMP shell>
>    (qemu) drive_add ...
>    <Press Ctrl-Alt-1 to get back to the guest>
> 
> 3. <Press Ctrl-Alt-2 to get to the HMP shell>
>    (qemu) device_del ...
>    <Press Ctrl-Alt-1 to get back to the guest>
> 
> This is what we're competing with, and honestly I don't see how your
> qemu-runtime-*/qemu-client based flow comes even close to it in terms of
> usability.
> 
> QMP, JSON and YAML may be nice machine interfaces, but having nice
> machine interfaces doesn't mean that you shouldn't also have something
> that is suitable for humans. qmp-shell is trying to be that, and while
> it leaves much to be desired in its current state, replacing it with
> even more machine-friendly stuff that is cumbersome for humans isn't the
> right answer.
> 

This is why I have two things I want to do:

Part A:
- Remove the CLI as it exists today*
- Introduce a machine-readable, but human-friendly format, like YAML (or
JSON) that's based directly on QAPI.

Part B:
- Improve qmp-shell such that it provides a sufficient usability layer
on top of QMP
- Make this tool prominent to users; e.g. putting it in the GTK
interface would be nice.
- Reduce prominence of HMP and works towards its removal.

--js