Fwd: GSoC Intro - TUI interface for QMP

Fwd: GSoC Intro - TUI interface for QMP

[Niteesh G. S.]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]

Hello all,

I am Niteesh, one of the students selected under Google Summer of Code 2021
to work on the project* Interactive, asynchronous QMP TUI.*
Link to my proposal:
https://docs.google.com/document/d/1o_U9txCyqZDYIqqhj4V0IEO9-20KY2pb4egi75ueXco/edit?usp=sharing

The goal of this project is to get an interactive and asynchronous TUI
interface
that is capable of connecting to the AQMP server and communicate with it.

My experience with async stuff and AQMP is writing a chat application to
better understand python's asyncio and trying to connect the UI to AQMP.
The UI built for chatting basically worked as a primitive interface to send
and
receive QMP commands. The below video demonstrates this.
https://github.com/gs-niteesh/Async-Chat-App/blob/master/qemu.mp4

By end of this summer, I would like to get a basic TUI with some desirable
features working. Some of the features I would like to get working are
1) Syntax checking
2) Syntax highlighting
3) Code completion
4) Logging

I would like to hear some of the features you would like to have and also
your
advice's on implementation.

PS: If you are seeing the mail twice, sorry, I forgot to include the
mailing list
in the first time.


Thanks,
Niteesh

[-- Attachment #2: Type: text/html, Size: 3055 bytes --]

Re: Fwd: GSoC Intro - TUI interface for QMP

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

On Sat, May 22, 2021 at 12:32:00AM +0530, Niteesh G. S. wrote:
> By end of this summer, I would like to get a basic TUI with some desirable
> features working. Some of the features I would like to get working are
> 1) Syntax checking
> 2) Syntax highlighting
> 3) Code completion
> 4) Logging
> 
> I would like to hear some of the features you would like to have and also
> your
> advice's on implementation.

Welcome Niteesh!

It would be great to have an QMP API documentation viewer built into the
TUI. When you enter a command like 'migrate' it would automatically
display the associated QMP command documentation from qapi/*.json.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: GSoC Intro - TUI interface for QMP

[John Snow]

On 5/24/21 9:32 AM, Stefan Hajnoczi wrote:
> On Sat, May 22, 2021 at 12:32:00AM +0530, Niteesh G. S. wrote:

Welcome Niteesh :) I look forward to working with you this summer.

>> By end of this summer, I would like to get a basic TUI with some desirable
>> features working. Some of the features I would like to get working are

As a reminder to anyone reading this thread, the goal is to create a 
qmp-shell that functions more as a TUI, akin to mutt, irssi, or (my 
favorite example) mitmproxy. The idea is that there will be, at minimum, 
a history panel showing QMP messages that have occurred so far and a 
text entry panel for entering new commands.

This shell can then be augmented with various other features to 
facilitate testing, debugging, etc. One of the core upgrades over the 
existing qmp-shell will be the featuring of truly asynchronous events 
which will appear in the history panel without requiring the human user 
to press <enter> to allow them to display. This will use a new 
Asynchronous QMP library to facilitate this feature, bringing with it 
fixes over our current use of undocumented Python features abusing 
non-blocking sockets in the old QMP library.

My plan is to worry about implementing the very basics of the shell 
first, and then to add more features on as we feel comfortable with the 
basics. We can discuss what we consider to be the bare minimum for this 
project and lay out the feature requirements that define a successful 
minimum.

>> 1) Syntax checking

To a limited extent. I don't want to disallow the user from sending 
commands that are invalid in the event that we want to test the server's 
ability to cope with and reply to invalid commands.

However, if the syntax is malformed enough that we can't understand it 
to send it to the server, good error messages that point out what 
exactly went wrong are helpful.

>> 2) Syntax highlighting
>> 3) Code completion
>> 4) Logging
>>
>> I would like to hear some of the features you would like to have and also
>> your
>> advice's on implementation.
> 
> Welcome Niteesh!
> 
> It would be great to have an QMP API documentation viewer built into the
> TUI. When you enter a command like 'migrate' it would automatically
> display the associated QMP command documentation from qapi/*.json.
> 

Stefan's suggestion is an important one to me. The current qmp-shell 
does have an auto-complete list of available commands (press <tab> after 
connecting to see the list), but it offers no explanation of what the 
commands do, what their arguments are, etc.

Being able to get interactive help explaining the commands, their 
arguments, etc would be a massive usability improvement over the old shell.

> Stefan
>
Additionally, there's a few threads we've had before with people's 
stretch goals, dreams, nice-to-haves and so on. I owe you a bit of a 
compilation of those ideas. Here are some that I happen to remember.

(Keep in mind that not all of these have been discussed or vetted 
thoroughly on list, some may be vastly harder than others, and not all 
of them may be mutually compatible. There may even be objections to some 
of these. We will definitely not do every last thing on this list, and 
it is important we stay focused on the core task and get code merged 
this summer. However, some of these stretch goals might be fun and 
provide you with extreme leeway to investigate features that interest 
you personally, so I am listing them now.)


1. Utilize or otherwise re-implement readline's history feature, such 
that when opening qmp-shell, you can press up/down to recall previously 
issued commands. (The usage of 'readline' library itself may be 
complicated if we wish to use a TUI library like urwid. Some re-creation 
of features may be necessary.)

2. Implement tab-complete for command names, command arguments, etc.

3. Implement a FiSH-like suggestion mode where, without tab complete, 
the text entry window shows (in e.g. a gray font) the most likely 
auto-complete. You can press <tab> to accept this auto-completion.

4. Add the ability to save a log file for a given session, recording 
timestamps, incoming messages and outgoing messages. Sessions could 
possibly be automatically saved and cycled such that the last 10 or so 
are kept, or maybe up 10MB, or some other kind of limit.

5. Add the ability to load an old log file and "replay" it to a server. 
Optionally, the "replay" might be able to use the log timestamps to 
replay the commands with an identical timing. After a replay, it might 
be nice to be able to use a diff viewer of some kind to show differences 
between the original script and the replay.

6. Add the ability to "attach" qmp-shell to an existing instance of a 
libvirt-managed QEMU and have qmp-shell behave in a "watch" mode, 
showing commands sent back and forth between QEMU and libvirt, to help 
facilitate live debugging of libvirt.

7. Add the ability to "hide" certain events from the history view. By 
pressing a certain hotkey, we can enter a command mode and start 
filtering certain events. For example, we may wish to hide from view all 
JOB_STATUS_CHANGE events from view. Certain filtering modes may warrant 
their own explicit mode available directly in the shell, but we may wish 
to provide a more powerful filtering/scripting mechanism.

8. Similar to above, add the ability to change the "verbosity" of events 
in the history panel live at runtime. For example:

   - Show the full QMP message, pretty-printed
   - Show the full QMP message, condensed with no spaces.
   - Show the raw byte sequence of each QMP message.
   - Show an abbreviated summary; just the command name or event name.
   - Show or hide timestamps
   - Show outgoing commands in terms of what the user typed in qmp-shell.
     (e.g., if we offer a simple syntax, show the simple-syntax version.)

   Being able to toggle between these at-will may be nice to help see
   more or less information dynamically as the situation calls for it.


9. The ability to generate an iotest stub from the current session 
history or a subsection thereof. It's OK if the test isn't fully 
functional on its own, leaving some sections as stubs. Generating the 
commands, waiting for the commands to complete, and generating an '.out' 
could be a nice feature that helps qmp-shell users quickly write tests.

Some thoughts on this:

- iotests usually launch a VM to connect to. qmp-shell will not know the 
arguments QEMU was launched with. The generated test might attempt to 
launch QEMU with a minimum viable configuration, leaving other arguments 
as a stub marked #FIXME.

- The generated iotest may be able to automatically generate event 
waits. For instance, if a JOB_STATUS_CHANGE event occurs prior to the 
user entering the next command, the generated iotest could automatically 
insert a delay that waits for the same event.

- Generated tests should always end with a VM shutdown sequence, either 
explicitly (user sends a 'quit' command) or implicitly (The python 
iotest library calls vm.shutdown())


10. The ability to detect a disconnection and attempt to reconnect 
automatically, similar to how an IRC client would.

11. The ability to define macros to perform multi-step, complex routines 
with a single command. Something like a ~/.config/qemu/qmp-shell.conf 
file could possibly define these macros. The ability to create/save/edit 
these macros directly from the shell could be nice.

12. Integrating tools like Vladimir's block graph visualization tool. 
There are complex memory structures in QEMU that can be hard to 
understand quickly by reading large volumes of text. qmp-shell could be 
augmented to help render and visualize them quickly, facilitating 
debugging greatly.


I'll update this thread with more ideas as I remember them when going 
back through previous upstream discussions we've had on the topic. We 
may want to create a wiki page https://wiki.qemu.org/Documentation and 
start updating it with various ideas, stretch goals, and so on, to keep 
our planning organized. If you do not have an account on this wiki yet, 
I believe you'll need to ask the list for an account, letting us know 
what email and username you want to use for it.


...I also need to warn you about one big problem that looms over this 
project: QMP is a syntax designed for machines, but it's difficult to 
write by hand. The old qmp-shell has a simplistic syntax that was 
designed to make entering commands easier, but it isn't a suitable 
syntax for entering more complex, hierarchical data.

One of the decisions we'll need to make is if we simply REMOVE the 
convenient syntax (allowing only the raw input of QMP), or if we attempt 
to design a new shorthand syntax, simply use the old shorthand syntax, etc.

This is a complex area (and one I haven't fully solved for myself), so 
if this project involves only the input of *raw QMP commands* into the 
shell, I would still consider that a success if we got some of the other 
features worked into it. We will have to discuss the pros/cons of adding 
an imperfect shorthand syntax to this shell; there has been some 
reluctance to that idea in the past.


So, your first homework assignment from me is:

- Create a QEMU wiki account
- Port your GSoC proposal to the wiki
- Begin recording stretch goals to that wiki page
- Set up an (at least) weekly meeting time with me to discuss progress, 
problems, etc. We can chat on IRC, via google meet/zoom, etc at your 
preference.
- Make sure you have a gitlab account. I will add you as a contributor 
to my fork of QEMU and we can use the issue and milestone tracker to 
help set expectations and stay focused.


Thanks! (And welcome!)
--js

Re: Fwd: GSoC Intro - TUI interface for QMP

[Niteesh G. S.]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

Hello Stefan,

On Mon, May 24, 2021 at 7:02 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Sat, May 22, 2021 at 12:32:00AM +0530, Niteesh G. S. wrote:
> > By end of this summer, I would like to get a basic TUI with some
> desirable
> > features working. Some of the features I would like to get working are
> > 1) Syntax checking
> > 2) Syntax highlighting
> > 3) Code completion
> > 4) Logging
> >
> > I would like to hear some of the features you would like to have and also
> > your
> > advice's on implementation.
>
> Welcome Niteesh!
>
> It would be great to have an QMP API documentation viewer built into the
> TUI. When you enter a command like 'migrate' it would automatically
> display the associated QMP command documentation from qapi/*.json.
>
That's a great idea.
Do you want the documentation as a popup or in a separate tab?
By separate tab what I mean is a full window dedicated to documentation
with some kind of search functionality.

Thanks,
Niteesh.

>
> Stefan
>

[-- Attachment #2: Type: text/html, Size: 1993 bytes --]

Re: GSoC Intro - TUI interface for QMP

[John Snow]

On 5/26/21 11:35 AM, Niteesh G. S. wrote:
> Hello Stefan,
> 
> On Mon, May 24, 2021 at 7:02 PM Stefan Hajnoczi <stefanha@redhat.com 
> <mailto:stefanha@redhat.com>> wrote:
> 
>     On Sat, May 22, 2021 at 12:32:00AM +0530, Niteesh G. S. wrote:
>      > By end of this summer, I would like to get a basic TUI with some
>     desirable
>      > features working. Some of the features I would like to get
>     working are
>      > 1) Syntax checking
>      > 2) Syntax highlighting
>      > 3) Code completion
>      > 4) Logging
>      >
>      > I would like to hear some of the features you would like to have
>     and also
>      > your
>      > advice's on implementation.
> 
>     Welcome Niteesh!
> 
>     It would be great to have an QMP API documentation viewer built into the
>     TUI. When you enter a command like 'migrate' it would automatically
>     display the associated QMP command documentation from qapi/*.json.
> 
> That's a great idea.
> Do you want the documentation as a popup or in a separate tab?
> By separate tab what I mean is a full window dedicated to documentation
> with some kind of search functionality.
> 
> Thanks,
> Niteesh.
> 
> 

If we're talking about an urwid interface ...

There are a few ways to do this.

1. Explicitly, if you type something like "/help [command-name]" in the 
command bar, you could create a panel that temporarily replaces the 
history panel that shows the arguments and usage of the requested 
command name.

You could use page-up/page-down to scroll the help up and down accordingly.

To exit the help window, you could issue some keypress (like Ctrl+X? If 
you prototype this, just make it configurable -- I am sure people will 
have opinions.)

While the help window is up, the command entry bar could stay active -- 
i.e., you can type your command while the help is visible on-screen.

(As an aside: If we expect that we will have more than just one or two 
slash commands, it may be nice to have urwid create a popup suggestion 
box when you type a '/', hinting to the user what local shell commands 
are available.)


2. You could just display the help output as a flow item directly in the 
history pane, but if QEMU is receiving lots of events, you run the risk 
of having the help message flood off the screen.

Still, it might be a very simple way to implement it -- and the user can 
always just scroll up.


3. If we get around to implementing any kind of fancy syntax for the 
command entry bar, it may be possible to have the shell read the 
partially-entered command and guess which command we want to see help 
for -- maybe by pressing Ctrl+H?


A challenge will be where to pull the help text from. The QEMU server is 
not capable (today) of sending help information over the QMP socket itself.

We will need to implement a QMP command inside of QEMU directly that is 
capable of delivering this information to the client so that it can 
render it.

Since not all versions of QEMU will have this feature, the qmp-shell 
will need to be able to gracefully deal with the lack of help text, 
displaying an error indicating that this version of QEMU does not have 
help information compiled into it.

--js

Re: GSoC Intro - TUI interface for QMP

[Bruno Piazera Larsen]

[-- Attachment #1: Type: text/plain, Size: 3176 bytes --]


On 24/05/2021 14:34, John Snow wrote:
> On 5/24/21 9:32 AM, Stefan Hajnoczi wrote:
>> On Sat, May 22, 2021 at 12:32:00AM +0530, Niteesh G. S. wrote:
>
> Welcome Niteesh :) I look forward to working with you this summer.
>
>>> By end of this summer, I would like to get a basic TUI with some 
>>> desirable
>>> features working. Some of the features I would like to get working are
>
> As a reminder to anyone reading this thread, the goal is to create a 
> qmp-shell that functions more as a TUI, akin to mutt, irssi, or (my 
> favorite example) mitmproxy. The idea is that there will be, at 
> minimum, a history panel showing QMP messages that have occurred so 
> far and a text entry panel for entering new commands.
>
> This shell can then be augmented with various other features to 
> facilitate testing, debugging, etc. One of the core upgrades over the 
> existing qmp-shell will be the featuring of truly asynchronous events 
> which will appear in the history panel without requiring the human 
> user to press <enter> to allow them to display. This will use a new 
> Asynchronous QMP library to facilitate this feature, bringing with it 
> fixes over our current use of undocumented Python features abusing 
> non-blocking sockets in the old QMP library.
>
> My plan is to worry about implementing the very basics of the shell 
> first, and then to add more features on as we feel comfortable with 
> the basics. We can discuss what we consider to be the bare minimum for 
> this project and lay out the feature requirements that define a 
> successful minimum.
>
>>> 1) Syntax checking
>
> To a limited extent. I don't want to disallow the user from sending 
> commands that are invalid in the event that we want to test the 
> server's ability to cope with and reply to invalid commands.
>
> However, if the syntax is malformed enough that we can't understand it 
> to send it to the server, good error messages that point out what 
> exactly went wrong are helpful.
>
I would actually suggest going the other way around. If there is a plan 
to test a server's ability to deal with invalid commands, it should be 
very obviously malformed, and when a user is trying to enter a command 
but has a small mistake (like a typo or something) telling the user that 
this was the likely mistake would make it more user-friendly.

A way to implement this would be to calculate a "proximity" value to all 
likely commands, and if it is very far from all known commands you send 
it to test, if it is very close to one or more, you print some helpful 
information. Other option is that you always show the closest command, 
say there's a mistake, and ask if the command should be sent anyway, 
which is easier to implement, but is a worse UX.

I admit this is pretty counter intuitive, but I think if it is well 
documented, it could be a better way to implement the feature

-- 
Bruno Piazera Larsen
Instituto de Pesquisas ELDORADO 
<https://www.eldorado.org.br/?utm_campaign=assinatura_de_e-mail&utm_medium=email&utm_source=RD+Station>
Departamento Computação Embarcada
Analista de Software Trainee
Aviso Legal - Disclaimer <https://www.eldorado.org.br/disclaimer.html>

[-- Attachment #2: Type: text/html, Size: 4334 bytes --]

Re: GSoC Intro - TUI interface for QMP

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

[...]

> A challenge will be where to pull the help text from. The QEMU server
> is not capable (today) of sending help information over the QMP socket
> itself.
>
> We will need to implement a QMP command inside of QEMU directly that
> is capable of delivering this information to the client so that it can 
> render it.
>
> Since not all versions of QEMU will have this feature, the qmp-shell
> will need to be able to gracefully deal with the lack of help text, 
> displaying an error indicating that this version of QEMU does not have
> help information compiled into it.

The doc text is bulky: my bld/docs/manual/interop/qemu-qmp-ref.html is
1.7 MiB and growing.  Less lavish markup results in smaller data.  We
may want to store it compressed, or load it on demand.  We might even
have to make it compile-time optional for some use cases.

Re: GSoC Intro - TUI interface for QMP

[John Snow]

On 6/8/21 11:01 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
> [...]
> 
>> A challenge will be where to pull the help text from. The QEMU server
>> is not capable (today) of sending help information over the QMP socket
>> itself.
>>
>> We will need to implement a QMP command inside of QEMU directly that
>> is capable of delivering this information to the client so that it can
>> render it.
>>
>> Since not all versions of QEMU will have this feature, the qmp-shell
>> will need to be able to gracefully deal with the lack of help text,
>> displaying an error indicating that this version of QEMU does not have
>> help information compiled into it.
> 
> The doc text is bulky: my bld/docs/manual/interop/qemu-qmp-ref.html is
> 1.7 MiB and growing.  Less lavish markup results in smaller data.  We
> may want to store it compressed, or load it on demand.  We might even
> have to make it compile-time optional for some use cases.
> 
> 

ACK, understood.

raw QAPI directory, including only the json files, is "only" 551.3 kB.

I assume we can compile help text to something json (or json-like) and 
then compress it. Perhaps we could compile something like 
qapi-help-introspect.json.tgz and load it on-demand from the QEMU binary 
when help text is requested.

We could prototype under the experimental QMP command x-help, and limit 
it to sending help for just one command at a time to limit data transfer.

The client could cache the information. (Against what kind of an 
identifier? Can QEMU report some kind of token that uniquely identifies 
its binary or uniquely identifies the set of QAPI commands it supports?)

This has the potential to exceed our capacity this summer, but a 
prototype experiment might be helpful to inform future work anyway.

--js

Re: GSoC Intro - TUI interface for QMP

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 6/8/21 11:01 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> [...]
>> 
>>> A challenge will be where to pull the help text from. The QEMU server
>>> is not capable (today) of sending help information over the QMP socket
>>> itself.
>>>
>>> We will need to implement a QMP command inside of QEMU directly that
>>> is capable of delivering this information to the client so that it can
>>> render it.
>>>
>>> Since not all versions of QEMU will have this feature, the qmp-shell
>>> will need to be able to gracefully deal with the lack of help text,
>>> displaying an error indicating that this version of QEMU does not have
>>> help information compiled into it.
>> The doc text is bulky: my bld/docs/manual/interop/qemu-qmp-ref.html
>> is
>> 1.7 MiB and growing.  Less lavish markup results in smaller data.  We
>> may want to store it compressed, or load it on demand.  We might even
>> have to make it compile-time optional for some use cases.
>> 
>
> ACK, understood.
>
> raw QAPI directory, including only the json files, is "only" 551.3 kB.
>
> I assume we can compile help text to something json (or json-like) and
> then compress it. Perhaps we could compile something like 
> qapi-help-introspect.json.tgz and load it on-demand from the QEMU
> binary when help text is requested.
>
> We could prototype under the experimental QMP command x-help, and
> limit it to sending help for just one command at a time to limit data
> transfer.
>
> The client could cache the information. (Against what kind of an
> identifier? Can QEMU report some kind of token that uniquely
> identifies its binary or uniquely identifies the set of QAPI commands
> it supports?)

I proposed something like it to permit QMP clients cache
query-qmp-schema output.  Libvirt didn't want it, so it never got beyond
the idea stage.

> This has the potential to exceed our capacity this summer, but a
> prototype experiment might be helpful to inform future work anyway.

Beware of the risk that comes with shiny stretch goals: loss of focus.
I believe this is actually this GSoC project's main risk.

Re: GSoC Intro - TUI interface for QMP

[Daniel P. Berrangé]

On Tue, Jun 08, 2021 at 05:01:28PM +0200, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
> [...]
> 
> > A challenge will be where to pull the help text from. The QEMU server
> > is not capable (today) of sending help information over the QMP socket
> > itself.
> >
> > We will need to implement a QMP command inside of QEMU directly that
> > is capable of delivering this information to the client so that it can 
> > render it.
> >
> > Since not all versions of QEMU will have this feature, the qmp-shell
> > will need to be able to gracefully deal with the lack of help text, 
> > displaying an error indicating that this version of QEMU does not have
> > help information compiled into it.
> 
> The doc text is bulky: my bld/docs/manual/interop/qemu-qmp-ref.html is
> 1.7 MiB and growing.  Less lavish markup results in smaller data.  We
> may want to store it compressed, or load it on demand.  We might even
> have to make it compile-time optional for some use cases.

From the POV of a TUI, I wouldn't think they necessarily need/want the
HTML format at all, because that means they now need an HTML -> text
conversion step.

IOW, an 'query-help' command might be better just sending back the
original raw docs from the .json files, and let the client then
decide how to format that. That ought to keep the data size down
as no formatting info would be included.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: GSoC Intro - TUI interface for QMP

[Daniel P. Berrangé]

On Tue, Jun 08, 2021 at 11:49:41AM -0400, John Snow wrote:
> On 6/8/21 11:01 AM, Markus Armbruster wrote:
> > John Snow <jsnow@redhat.com> writes:
> > 
> > [...]
> > 
> > > A challenge will be where to pull the help text from. The QEMU server
> > > is not capable (today) of sending help information over the QMP socket
> > > itself.
> > > 
> > > We will need to implement a QMP command inside of QEMU directly that
> > > is capable of delivering this information to the client so that it can
> > > render it.
> > > 
> > > Since not all versions of QEMU will have this feature, the qmp-shell
> > > will need to be able to gracefully deal with the lack of help text,
> > > displaying an error indicating that this version of QEMU does not have
> > > help information compiled into it.
> > 
> > The doc text is bulky: my bld/docs/manual/interop/qemu-qmp-ref.html is
> > 1.7 MiB and growing.  Less lavish markup results in smaller data.  We
> > may want to store it compressed, or load it on demand.  We might even
> > have to make it compile-time optional for some use cases.
> > 
> > 
> 
> ACK, understood.
> 
> raw QAPI directory, including only the json files, is "only" 551.3 kB.
> 
> I assume we can compile help text to something json (or json-like) and then
> compress it. Perhaps we could compile something like
> qapi-help-introspect.json.tgz and load it on-demand from the QEMU binary
> when help text is requested.

I think that's  overthinking things. Even a small QEMU install is 
10s of MB in size once you add up the qemu binary, firmware blobs
and depedancies. If we only keep the raw help text from the json,
we'll have < 1/2 MB, which is lost in the noise of the overall
deploymnt size. No need for compression or complex load-on-demand,
just keep it small and simple.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: GSoC Intro - TUI interface for QMP

[John Snow]

On 6/9/21 7:56 AM, Markus Armbruster wrote:
>> The client could cache the information. (Against what kind of an
>> identifier? Can QEMU report some kind of token that uniquely
>> identifies its binary or uniquely identifies the set of QAPI commands
>> it supports?)

> I proposed something like it to permit QMP clients cache
> query-qmp-schema output.  Libvirt didn't want it, so it never got beyond
> the idea stage.
> 

What ideas did you have for a cache key? We don't need to uniquely 
identify every instance or even every binary.

I suppose we could use an md5/sha1 checksum of the QMP introspection output?

>> This has the potential to exceed our capacity this summer, but a
>> prototype experiment might be helpful to inform future work anyway.
> Beware of the risk that comes with shiny stretch goals: loss of focus.
> I believe this is actually this GSoC project's main risk.

It is and I agree. I have been pushing Niteesh to complete the simplest 
possible prototype imaginable, but I believe he's identified having help 
text as something he'd really like to see, so I am investigating those 
concerns.

I do not think we'll actually be able to fully implement it start to 
finish, but it may be possible that we can implement a kind of "mockup" 
x-help command that has a few hardcoded things we can use to prototype 
the feature in the TUI.

I will keep scope creep in mind, we will pick and choose our battles. I 
am hell-bent on having *anything* checked into the tree by August, and I 
know that can be a longer process than we expect sometimes. I know this 
means keeping it small.

--js

Re: GSoC Intro - TUI interface for QMP

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 6/9/21 7:56 AM, Markus Armbruster wrote:
>>> The client could cache the information. (Against what kind of an
>>> identifier? Can QEMU report some kind of token that uniquely
>>> identifies its binary or uniquely identifies the set of QAPI commands
>>> it supports?)
>
>> I proposed something like it to permit QMP clients cache
>> query-qmp-schema output.  Libvirt didn't want it, so it never got beyond
>> the idea stage.
>> 
>
> What ideas did you have for a cache key? We don't need to uniquely
> identify every instance or even every binary.
>
> I suppose we could use an md5/sha1 checksum of the QMP introspection output?

commit 39a181581650f4d50f4445bc6276d9716cece050
Author: Markus Armbruster <armbru@redhat.com>
Date:   Wed Sep 16 13:06:28 2015 +0200

    qapi: New QMP command query-qmp-schema for QMP introspection

[...]

    New QMP command query-qmp-schema takes its return value from that
    variable.  Its reply is some 85KiBytes for me right now.

Has since grown to ~160KiB.
    
    If this turns out to be too much, we have a couple of options:
    
    * We can use shorter names in the JSON.  Not the QMP style.
    
    * Optionally return the sub-schema for commands and events given as
      arguments.
    
      Right now qmp_query_schema() sends the string literal computed by
      qmp-introspect.py.  To compute sub-schema at run time, we'd have to
      duplicate parts of qapi-introspect.py in C.  Unattractive.
    
    * Let clients cache the output of query-qmp-schema.
    
      It changes only on QEMU upgrades, i.e. rarely.  Provide a command
      query-qmp-schema-hash.  Clients can have a cache indexed by hash,
      and re-query the schema only when they don't have it cached.  Even
      simpler: put the hash in the QMP greeting.
    
    Signed-off-by: Markus Armbruster <armbru@redhat.com>
    Reviewed-by: Eric Blake <eblake@redhat.com>

Note this glosses over what exactly is hashed.  Back then, we generated
query-qmp-schema's output as a string, so we would have hashed that.
Today, we generate a QLitObject.  Less trivial to hash.  Quite feasible
all the same.

NB: Commit messages are love letters to your future self :)

[...]

Re: GSoC Intro - TUI interface for QMP

[John Snow]

On 6/9/21 8:07 AM, Daniel P. Berrangé wrote:
> I think that's  overthinking things. Even a small QEMU install is
> 10s of MB in size once you add up the qemu binary, firmware blobs
> and depedancies. If we only keep the raw help text from the json,
> we'll have < 1/2 MB, which is lost in the noise of the overall
> deploymnt size. No need for compression or complex load-on-demand,
> just keep it small and simple.
> 

Yes, you're right. Our first step should be to keep it small and simple 
and prove that it works.

We can worry about load on demand later if we still want it at that point.

--js

Re: GSoC Intro - TUI interface for QMP

[Vladimir Sementsov-Ogievskiy via]

24.05.2021 20:34, John Snow wrote:
> On 5/24/21 9:32 AM, Stefan Hajnoczi wrote:
>> On Sat, May 22, 2021 at 12:32:00AM +0530, Niteesh G. S. wrote:
> 
> Welcome Niteesh :) I look forward to working with you this summer.
> 
>>> By end of this summer, I would like to get a basic TUI with some desirable
>>> features working. Some of the features I would like to get working are
> 
> As a reminder to anyone reading this thread, the goal is to create a qmp-shell that functions more as a TUI, akin to mutt, irssi, or (my favorite example) mitmproxy. The idea is that there will be, at minimum, a history panel showing QMP messages that have occurred so far and a text entry panel for entering new commands.
> 
> This shell can then be augmented with various other features to facilitate testing, debugging, etc. One of the core upgrades over the existing qmp-shell will be the featuring of truly asynchronous events which will appear in the history panel without requiring the human user to press <enter> to allow them to display. This will use a new Asynchronous QMP library to facilitate this feature, bringing with it fixes over our current use of undocumented Python features abusing non-blocking sockets in the old QMP library.
> 
> My plan is to worry about implementing the very basics of the shell first, and then to add more features on as we feel comfortable with the basics. We can discuss what we consider to be the bare minimum for this project and lay out the feature requirements that define a successful minimum.
> 
>>> 1) Syntax checking
> 
> To a limited extent. I don't want to disallow the user from sending commands that are invalid in the event that we want to test the server's ability to cope with and reply to invalid commands.
> 
> However, if the syntax is malformed enough that we can't understand it to send it to the server, good error messages that point out what exactly went wrong are helpful.
> 
>>> 2) Syntax highlighting
>>> 3) Code completion
>>> 4) Logging
>>>
>>> I would like to hear some of the features you would like to have and also
>>> your
>>> advice's on implementation.
>>
>> Welcome Niteesh!
>>
>> It would be great to have an QMP API documentation viewer built into the
>> TUI. When you enter a command like 'migrate' it would automatically
>> display the associated QMP command documentation from qapi/*.json.
>>
> 
> Stefan's suggestion is an important one to me. The current qmp-shell does have an auto-complete list of available commands (press <tab> after connecting to see the list), but it offers no explanation of what the commands do, what their arguments are, etc.
> 
> Being able to get interactive help explaining the commands, their arguments, etc would be a massive usability improvement over the old shell.
> 
>> Stefan
>>
> Additionally, there's a few threads we've had before with people's stretch goals, dreams, nice-to-haves and so on. I owe you a bit of a compilation of those ideas. Here are some that I happen to remember.
> 
> (Keep in mind that not all of these have been discussed or vetted thoroughly on list, some may be vastly harder than others, and not all of them may be mutually compatible. There may even be objections to some of these. We will definitely not do every last thing on this list, and it is important we stay focused on the core task and get code merged this summer. However, some of these stretch goals might be fun and provide you with extreme leeway to investigate features that interest you personally, so I am listing them now.)
> 
> 
> 1. Utilize or otherwise re-implement readline's history feature, such that when opening qmp-shell, you can press up/down to recall previously issued commands. (The usage of 'readline' library itself may be complicated if we wish to use a TUI library like urwid. Some re-creation of features may be necessary.)
> 
> 2. Implement tab-complete for command names, command arguments, etc.
> 
> 3. Implement a FiSH-like suggestion mode where, without tab complete, the text entry window shows (in e.g. a gray font) the most likely auto-complete. You can press <tab> to accept this auto-completion.
> 
> 4. Add the ability to save a log file for a given session, recording timestamps, incoming messages and outgoing messages. Sessions could possibly be automatically saved and cycled such that the last 10 or so are kept, or maybe up 10MB, or some other kind of limit.
> 
> 5. Add the ability to load an old log file and "replay" it to a server. Optionally, the "replay" might be able to use the log timestamps to replay the commands with an identical timing. After a replay, it might be nice to be able to use a diff viewer of some kind to show differences between the original script and the replay.
> 
> 6. Add the ability to "attach" qmp-shell to an existing instance of a libvirt-managed QEMU and have qmp-shell behave in a "watch" mode, showing commands sent back and forth between QEMU and libvirt, to help facilitate live debugging of libvirt.
> 
> 7. Add the ability to "hide" certain events from the history view. By pressing a certain hotkey, we can enter a command mode and start filtering certain events. For example, we may wish to hide from view all JOB_STATUS_CHANGE events from view. Certain filtering modes may warrant their own explicit mode available directly in the shell, but we may wish to provide a more powerful filtering/scripting mechanism.
> 
> 8. Similar to above, add the ability to change the "verbosity" of events in the history panel live at runtime. For example:
> 
>    - Show the full QMP message, pretty-printed
>    - Show the full QMP message, condensed with no spaces.
>    - Show the raw byte sequence of each QMP message.
>    - Show an abbreviated summary; just the command name or event name.
>    - Show or hide timestamps
>    - Show outgoing commands in terms of what the user typed in qmp-shell.
>      (e.g., if we offer a simple syntax, show the simple-syntax version.)
> 
>    Being able to toggle between these at-will may be nice to help see
>    more or less information dynamically as the situation calls for it.
> 
> 
> 9. The ability to generate an iotest stub from the current session history or a subsection thereof. It's OK if the test isn't fully functional on its own, leaving some sections as stubs. Generating the commands, waiting for the commands to complete, and generating an '.out' could be a nice feature that helps qmp-shell users quickly write tests.
> 
> Some thoughts on this:
> 
> - iotests usually launch a VM to connect to. qmp-shell will not know the arguments QEMU was launched with. The generated test might attempt to launch QEMU with a minimum viable configuration, leaving other arguments as a stub marked #FIXME.
> 
> - The generated iotest may be able to automatically generate event waits. For instance, if a JOB_STATUS_CHANGE event occurs prior to the user entering the next command, the generated iotest could automatically insert a delay that waits for the same event.
> 
> - Generated tests should always end with a VM shutdown sequence, either explicitly (user sends a 'quit' command) or implicitly (The python iotest library calls vm.shutdown())
> 
> 
> 10. The ability to detect a disconnection and attempt to reconnect automatically, similar to how an IRC client would.
> 
> 11. The ability to define macros to perform multi-step, complex routines with a single command. Something like a ~/.config/qemu/qmp-shell.conf file could possibly define these macros. The ability to create/save/edit these macros directly from the shell could be nice.
> 
> 12. Integrating tools like Vladimir's block graph visualization tool. There are complex memory structures in QEMU that can be hard to understand quickly by reading large volumes of text. qmp-shell could be augmented to help render and visualize them quickly, facilitating debugging greatly.
> 

Thanks for noting this :)

Great that you've started this job! I'll share some my ideas in relation:

13. An ability to connect to running libvirt guest instead of qemu process. It's simple, and it's done in my scripts/render_block_graph, by additional small class LibvirtGuest, which realizes .command interface like QEMUMonitorProtocol.

Further without numbers and less concrete (and maybe less related to the considered project, but just to keep in mind)

== Language ==

I never liked typing Json things by hand. All these quotes, etc.. So it would be good to support (may be enabled by cmdline option) some dialects. Most simple is support Yaml: it's a lot less restrictive about quotes than Json.
I have a tool (I think I didn't publish it, by probably I should), which simplifies sending qmp commands to libvirt guests. And it implements own key-value syntax, which goes further than yaml and allows omitting not only quotes but also commas and colons, so instead of

virsh qemu-monitor-command vm1 '{"execute": "blockdev-add", "arguments": {"id": "disk", "driver": "qcow2", "cache": {"writeback": true, "direct": true}, "aio": "native", "discard": "unmap", "file": {"driver": "file", "filename": "/tmp/somedisk"}}}'

I can write

qmp vm1 blockdev-add id disk driver qcow2 cache {writeback true direct true} aio native discard unmap file {driver file filename /tmp/somedisk}


== Visualization ==

Hmm, as John said in 12 it would be good to integrate with some visualization. I'll say what idea I had around it:

We have script in qemu scripts/render_block_graph , which runs some qmp commands to get the structure of Qemu's block-layer (it's a complex graph structure) and uses graphviz to render a png image. I had an idea to make a simple http server, that provides a way in realtime in browser watch Qemu process block-layer.

And really, if we are going to visualize, I think trying to do it in terminal is a wrong way. Better to assist the terminal by web server (or may be, just make the terminal inside a web page?), and visualize in browser, as html/js/css already has everything we need for any kind of visualization.

Also, if you ever used browser internal debugger, you know about it's very kind feature of visualizing json objects, so in the output it is collapsed, and you can click by fields to open them.. That would be very nice for output of query-named-block-nodes command which is huge and hard to look through. And for all json objects in displayed history..


== Simple start of qmp-shell ==

Actually I don't use qmp-shell ;) Instead I just run qemu with "-qmp stdio" argument, because it's a lot simpler than remember how to run it with unix socket, and run qmp-shell on that unix socket in another terminal.

(and yes, to compose command in json, I use my "qmp" script which parses simple key-val language, and print command in json).

So, what I mean: it would be nice to have a possibility run _one_ command in _one_ terminal with all needed qemu arguments, to get into qmp terminal immediately, and have qemu stdout messages together with QAPI events. Like when I simply run qemu binary with "-qmp stdio" option.

-- 
Best regards,
Vladimir